[c-nsp] Cat6509 CAM entries flapping

[James Sneeringer]

For those interested, problem solved. SPAN and tcpdump led me to it, thanks
Oli.

It's bizarre, and I don't fully understand it, but the source of the problem
was a misconfigured Windows 2003 server. Our VLAN10 contains a /16 network,
and this server had its netmask misconfigured as a /24, such that it did not
have any route to the hosts whose CAM entries were flapping, nor to its own
default gateway. (The servers are 10.2.1.x, and this particular server is
10.2.2.x.) The default gateway is in VLAN2, on the other side of the LD. All
I can figure is that the combination of errors somehow caused L2 traffic
involving this server to be misdirected. I can't explain it, but correcting
the server's netmask immediately caused the CAM flapping to stop.

I have a case open with TAC on this. If they can offer some coherent
explanation, I will pass it along. Thanks to everyone else who offered
suggestions.

-James


> -----Original Message-----
> From: Oliver Dewdney [mailto:oliver.dewdney at lbi.com] 
> Sent: Friday, March 09, 2007 1:13 PM
> To: 'Ge Moua'; 'James Sneeringer'; 'cisco-nsp at puck.nether.net'
> Subject: RE: [c-nsp] Cat6509 CAM entries flapping
> 
> I think a span/mirror port of port 6/11 and a packet capture tool like
> wireshark might be a good place to start. 
> 
> Oli Dewdney
> 
> -----Original Message-----
> From: Ge Moua [mailto:moua0100 at umn.edu] 
> Sent: 09 March 2007 16:09
> To: 'James Sneeringer'; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Cat6509 CAM entries flapping
> 
> 
> Some of the older CatOS had annoying IOS bugs that showed 
> these symptoms.
> I've ran into this in the past, the newer code is much more resilient
> (especially native IOS).  I'm here that you're running hybrid 
> IOS (CatOS on
> the switch, IOS on the router module).
> 
>  
> 
> 
> 
> :-)
> Regards,
> Ge Moua | Email: moua0100 at umn.edu
> 
> Network Design Engineer
> University of Minnesota | Networking & Telecommunications 
> Services 2218
> University Ave SE | Minneapolis, MN 55414-3029
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of James 
> Sneeringer
> Sent: Friday, March 09, 2007 7:41 AM
> To: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Cat6509 CAM entries flapping
> 
> Lincoln Dale (ltd) wrote: 
> > 
> > > Any suggestions on how to troubleshoot it?
> > 
> > the root cause is that you have a common MAC address 
> appearing in two
> > places.
> > 
> > if that server you have on port 6/11 the server behind the LD?
> 
> Port 6/11 is the LD.
> 
> > if it isn't, I suggest you trace WHY traffic is leaking where it
> > shouldn't be.
> 
> I don't know that it is leaking, and if it is there's no good 
> reason for it.
> Maybe this crude diagram will help, because I don't think I'm 
> explaining it
> very well.
> 
>     ----------          -----------
>     | Host 1 |----------|         | 4/2    ext ---------
>     ----------     6/42 |         |------------|       |
>                         | Cat6509 |            | LD430 |
>     ----------     6/45 |         |------------|       |
>     | Host 2 |----------|         | 6/11   int ---------
>     ----------          -----------
> 
> The LD430 does NAT. The external interface is in VLAN2, and 
> the internal
> interface is in VLAN10. Host 1 and Host 2 are also in VLAN10 
> on the inside.
> CatOS sees the MAC address for Host 1 flip flopping between 
> port 6/42 (the
> correct port) and 6/11 (the LD's port).
> 
> The only reasons I can think of for Host 1's MAC address to 
> show up on port
> 6/11 are:
> 
> 1) The LD is sending gratuitous ARPs and spoofing Host 1's 
> MAC address. As
> far as I know, LDs don't do this.
> 
> 2) Traffic from Host 1 is somehow entering the LD's external 
> interface, and
> is thus bridged to its internal interface. This is what I 
> meant by traffic
> being leaked. Host 1 is not on a trunk port and only sees 
> VLAN10, so I don't
> see how this should be possible.
> 
> > OR: investigate whether two servers have the same MAC address
> > (shouldn't happen, but alas some NIC manufacturers have made 
> > mistakes...).
> 
> We're looking into this as well. However, the problem is very recent,
> starting in the last week or so, and it's being exhibited for 
> multiple MAC
> addresses. If it were two server ports doing this, I'd 
> definitely be leaning
> in this direction, but with the LD involved it doesn't seem likely.
> 
> -James