[nsp] Cisco 3548/3550 MAC strangeness

[Deepak Jain]

Hi,

	I am working with a customer who has the following server farm arrangement.

	About 400 servers connected to approximately 10 Cisco 3548 and 3550
switches. No VLANs, no loops, no Spanning Tree, no command clusters. They
each have about 20 IP addresses and share a single router. The switches link
in a left-to-right fashion use SX-GBICs and uplink from the last switch by
single-attach SX-GE.

	The problem is that there are unicast broadcast floods on the order of
100mb/s (at times, completely unpredictable) on many (not all) ports from
the switch ports to each of the servers. The exact amount per server is not
always the same, nor is the amount each switch is broadcasting.

	The machines are not rebooting, bridging or doing anything funky. They are
simply there. With a default MAC-aging time of 4 hours and only 400 (+1
router) MAC addresses to learn, I can't understand why so much broadcast
traffic is being generated. Even when the MAC-address aging time is set to
14400 minutes there is no change in the behavior.

	If "port block unicast" is turned on all the FE ports, the floods (to the
servers) stop, but the switches no longer report all the machines they are
attached to "show mac" nor do they report these MACs to their neighboring
switches even when direct connections are attempted. A clear ARP (on the
router) solves the problem for anywhere from a few seconds to a few minutes.
Even if the switch has learned the MAC address, unless continuous traffic is
sent to it, it will lose the server, even if its directly attached to the
switch.

	Cisco TAC's solution was to hard-code EACH server's MAC address into EACH
switch via "mac-address secure xxxx.xxxx.xxxx f0/xx" which is cumbersome and
feels silly.

	We have eliminated individual pieces of hardware as any switch of these
series that are put in place repeat the behavior. IOS versions do not affect
it either. VLANS would solve the problem, but the Customer is vehemently
opposed to them, and I can't think of a reason why he needs them. We are
only talking about a small amount of address space and no secondaries on the
interfaces.

	Please give me an idea of what I need to look at to either solve the
problem or prove why VLANs or some other solution would solve it. I talked
to a non-Cisco engineer and they seemed to feel that its a problem at the
ASIC-level of these units, but it seems so fundamental for essentially an
enterprise switch, I have a hard time believing that either.

Thanks in advance,

DJ