[nsp] Cisco 3548/3550 MAC strangeness

[Deepak Jain]

These are almost entirely modern Linux or BSD based systems. The two or
three Windows boxes on the network don't have anywhere near the traffic
levels (observed) to be responsible for even 1% of the broadcast storm, but
I can have them inspected for proper netmasks and segmented to see if the
symptoms disappear.

Thanks,

DJ

> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net]On Behalf Of Michael Loftis
> Sent: Thursday, February 13, 2003 1:34 AM
> To: deepak at ai.net; Cisco-Nsp at Puck. Nether. Net
> Subject: Re: [nsp] Cisco 3548/3550 MAC strangeness
>
>
> Uhhhmmm...are these 400 windows systems, or other mixed systems?  This
> smells badly like a broadcast storm being started by hosts incorrectly
> responding to queries to the broadcast address(es) of their subnet.  Or
> misconfigured host(s) with the wrong idea of what the subnet is
> sending out
> broadcasts when they really intended to send unicast packets.
>
> Say host A has 255.255.255.0 as it's mask and 10.1.1.0 as it's
> network when
> it is supposed to be 255.255.0.0 and 10.1.0.0 -- host  A sends a
> broadcast
> packet to 10.1.1.255 instead of ARPing and sending a unicast packet.
> Having this happen enough, along with hosts that incorrectly respond to
> this broadcast packet when it's incorrectly addressed can quickly
> fire up a
> nice broadcast storm.
>
> --On Wednesday, February 12, 2003 10:52 PM -0500 Deepak Jain
> <deepak at ai.net> wrote:
>
> >
> > Hi,
> >
> > 	I am working with a customer who has the following server farm
> > arrangement.
> >
> > 	About 400 servers connected to approximately 10 Cisco 3548 and 3550
> > switches. No VLANs, no loops, no Spanning Tree, no command
> clusters. They
> > each have about 20 IP addresses and share a single router. The switches
> > link in a left-to-right fashion use SX-GBICs and uplink from the last
> > switch by single-attach SX-GE.
> >
> > 	The problem is that there are unicast broadcast floods on
> the order of
> > 100mb/s (at times, completely unpredictable) on many (not all)
> ports from
> > the switch ports to each of the servers. The exact amount per server is
> > not always the same, nor is the amount each switch is broadcasting.
> >
> > 	The machines are not rebooting, bridging or doing anything
> funky. They
> > are simply there. With a default MAC-aging time of 4 hours and only 400
> > (+1 router) MAC addresses to learn, I can't understand why so much
> > broadcast traffic is being generated. Even when the MAC-address aging
> > time is set to 14400 minutes there is no change in the behavior.
> >
> > 	If "port block unicast" is turned on all the FE ports, the
> floods (to the
> > servers) stop, but the switches no longer report all the
> machines they are
> > attached to "show mac" nor do they report these MACs to their
> neighboring
> > switches even when direct connections are attempted. A clear ARP (on the
> > router) solves the problem for anywhere from a few seconds to a few
> > minutes. Even if the switch has learned the MAC address, unless
> > continuous traffic is sent to it, it will lose the server, even if its
> > directly attached to the switch.
> >
> > 	Cisco TAC's solution was to hard-code EACH server's MAC
> address into EACH
> > switch via "mac-address secure xxxx.xxxx.xxxx f0/xx" which is cumbersome
> > and feels silly.
> >
> > 	We have eliminated individual pieces of hardware as any
> switch of these
> > series that are put in place repeat the behavior. IOS versions do not
> > affect it either. VLANS would solve the problem, but the Customer is
> > vehemently opposed to them, and I can't think of a reason why he needs
> > them. We are only talking about a small amount of address space and no
> > secondaries on the interfaces.
> >
> > 	Please give me an idea of what I need to look at to either solve the
> > problem or prove why VLANs or some other solution would solve
> it. I talked
> > to a non-Cisco engineer and they seemed to feel that its a
> problem at the
> > ASIC-level of these units, but it seems so fundamental for
> essentially an
> > enterprise switch, I have a hard time believing that either.
> >
> > Thanks in advance,
> >
> > DJ
> >
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > http://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>