[c-nsp] Unicast traffic being sent to every port? Aging issue?

[Ray Van Dolson]

On Mon, Mar 22, 2010 at 07:03:36PM -0700, Ray Van Dolson wrote:
> We have two Dell PowerConnect M6220 switches (A1 and B1).  They are not
> cross-connected, but both have uplinks to the same subnet:
> 
>                       zfs1
>                      /
>                    +----+
>                    | A1 |---------|
>                    +----+     +-------+
>                               | Cisco |------- linux1
>                    +----+     +-------+
>                    | B1 |---------|
>                    +----+
>                     / \
>                   esx1 esx2
> 
> There's a host hanging off of A1 (zfs1) and several ESX hosts hanging
> off of B1 (esx1, esx2, etc).  There's a host linux1 hanging off the
> Cisco as well (actually many hosts, but for the sake of description
> 
> What's happening is, esx1/2 beging talking to zfs1.  All is well for a
> while... but at some point, zfs1's MAC address expires from the CAM on
> the switch (I guess that is what is happening).
> 
> At that point, the Cisco begins forwarding the unicast packets to all
> its ports.  The result -- linux1, and all other hosts see the packets.
> Occasionally, when we're dealing with a lot of traffic, this seriously
> impacts performance.
> 
> My question here is.. what is the _right_ way to deal with this?  This
> "flooding" can continue for many minutes at a time.. it isn't until an
> ARP reply eminates from zfs1 that the CAM table is populated again and
> the broadcasting stops.
> 
> I wonder if zfs1 would send back an ARP response quicker were it not
> behind an additional switch (the PowerConnect)... 

Well, I think I've nailed down the cause for this.

Probably if I'd more completely described things some of you woulda
pointed it out right away, but I was trying to keep the model
simplistic.

zfs1 is multi-homed.  Two interfaces on the same subnet.  Running
Solaris 10 with no special source based routing setup....

I probably don't need to go any further, but, suffice it to say,
packets destined for one interface on zfs1 were going "in" just fine,
but the replies were going out the other interface -- with a different
MAC address.

So obviously the switches eventually lose track of the "real" MAC
address and we get the symptoms I described.

Probably can be corrected with ipfilter in Solaris or changing our
infrastructure somewhat to handle this better.

Thanks all who replied -- it was good to learn about unicast storms!

Ray