[nsp] Cisco 3548/3550 MAC strangeness

[Michael Loftis]

Uhhhmmm...are these 400 windows systems, or other mixed systems?  This 
smells badly like a broadcast storm being started by hosts incorrectly 
responding to queries to the broadcast address(es) of their subnet.  Or 
misconfigured host(s) with the wrong idea of what the subnet is sending out 
broadcasts when they really intended to send unicast packets.

Say host A has 255.255.255.0 as it's mask and 10.1.1.0 as it's network when 
it is supposed to be 255.255.0.0 and 10.1.0.0 -- host  A sends a broadcast 
packet to 10.1.1.255 instead of ARPing and sending a unicast packet. 
Having this happen enough, along with hosts that incorrectly respond to 
this broadcast packet when it's incorrectly addressed can quickly fire up a 
nice broadcast storm.

--On Wednesday, February 12, 2003 10:52 PM -0500 Deepak Jain 
<deepak at ai.net> wrote:

>
> Hi,
>
> 	I am working with a customer who has the following server farm
> arrangement.
>
> 	About 400 servers connected to approximately 10 Cisco 3548 and 3550
> switches. No VLANs, no loops, no Spanning Tree, no command clusters. They
> each have about 20 IP addresses and share a single router. The switches
> link in a left-to-right fashion use SX-GBICs and uplink from the last
> switch by single-attach SX-GE.
>
> 	The problem is that there are unicast broadcast floods on the order of
> 100mb/s (at times, completely unpredictable) on many (not all) ports from
> the switch ports to each of the servers. The exact amount per server is
> not always the same, nor is the amount each switch is broadcasting.
>
> 	The machines are not rebooting, bridging or doing anything funky. They
> are simply there. With a default MAC-aging time of 4 hours and only 400
> (+1 router) MAC addresses to learn, I can't understand why so much
> broadcast traffic is being generated. Even when the MAC-address aging
> time is set to 14400 minutes there is no change in the behavior.
>
> 	If "port block unicast" is turned on all the FE ports, the floods (to the
> servers) stop, but the switches no longer report all the machines they are
> attached to "show mac" nor do they report these MACs to their neighboring
> switches even when direct connections are attempted. A clear ARP (on the
> router) solves the problem for anywhere from a few seconds to a few
> minutes. Even if the switch has learned the MAC address, unless
> continuous traffic is sent to it, it will lose the server, even if its
> directly attached to the switch.
>
> 	Cisco TAC's solution was to hard-code EACH server's MAC address into EACH
> switch via "mac-address secure xxxx.xxxx.xxxx f0/xx" which is cumbersome
> and feels silly.
>
> 	We have eliminated individual pieces of hardware as any switch of these
> series that are put in place repeat the behavior. IOS versions do not
> affect it either. VLANS would solve the problem, but the Customer is
> vehemently opposed to them, and I can't think of a reason why he needs
> them. We are only talking about a small amount of address space and no
> secondaries on the interfaces.
>
> 	Please give me an idea of what I need to look at to either solve the
> problem or prove why VLANs or some other solution would solve it. I talked
> to a non-Cisco engineer and they seemed to feel that its a problem at the
> ASIC-level of these units, but it seems so fundamental for essentially an
> enterprise switch, I have a hard time believing that either.
>
> Thanks in advance,
>
> DJ
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/