[c-nsp] interfaces flapping QinQ and/or spanning tree

[Matt Buford]

On Tue, Sep 1, 2009 at 7:22 AM, Bruno Filipe <brun0_filipe at yahoo.com> wrote:

> Something Weird is happening to me...the setup is very simple (there's
> nothing fancy) but for some reason I do have my switch ports flapping all
> the time and due to the fact the spanning tree is busy converging whenever
> that happens causing lot of problems...
>


> Syslog OUTPUT
> Sep  1 11:37:37 cat1.lda2.ao.isp.net 2045: Sep  1 11:37:17.823 GMT+1:
> %SW_MATM-4-MACFLAP_NOTIF: Host 0023.3314.0420 in vlan 291 is flapping
> between port Fa0/9 and port Fa0/1
> Sep  1 11:37:37 cat1.lda2.isp.net 2046: Sep  1 11:37:17.932 GMT+1:
> %SW_MATM-4-MACFLAP_NOTIF: Host 0013.8fb8.cb9d in vlan 292 is flapping
> between port Fa0/9 and port Fa0/11
>
>
It isn't clear to me which switch on your diagram is logging this message.
 Is it a switch that is actually doing QinQ?  I have run into problems
before where I had a metro-Ethernet link provided to me by a provider that
used QinQ.  So, none of my switches were speaking QinQ.  The problem I ran
into is that my 6500 switches use the same MAC address for every "interface
vlan" on the switch.  I then had half of my spanning tree roots set to one
switch, and the other half set to the other switch.

This resulted in the same mac address being transmitted in both directions
around my redundant metro-e loop.  None of MY switches ever logged the "is
flapping between..." message, but my provider did get that message logged
constantly.  For me, it just looked like I kept losing connectivity because
whenever the mac address went one way, one half of the vlans worked.  When
it went the other way, the other half of the VLANs worked.

Here's an old diagram I prepared years ago explaining the situation:

http://www.overloaded.net/pics/qinq.gif

You won't run into this problem if a) you never reuse a mac address across
VLANs, or b) all your STP roots are the same, or c) you don't have redundant
links forming a loop that includes QinQ.

The average customer of a QinQ metro-e provider likely won't notice this
issue mostly because of "C".  I suspect most customers of metro-e providers
using QinQ probably aren't using the QinQ based link to form a redundant
loop.  In my case, I almost always need redundant bridging of all VLANs
between datacenters.

Our solution?  We now ask every provider that we might buy metro-e from if
they are using QinQ.  If they say yes, they are disqualified.
 Unfortunately, some say no but turn out to be using it after all.  It is
quite frustrating to spend lots of money and time trying to get a provider
to turn a circuit up in time for a big project only to find that once the
circuit is delivered, we can't turn up redundancy across it because the
circuit is unable to carry reused MAC addresses across different VLANs.