[c-nsp] Issues with NIC teams on Unix and ESX servers

[Phil Mayers]

On Mon, 2007-09-03 at 23:12 +1000, Kamal Dissanayaka wrote:
> Hi,
> 
> We have recently migrated two of 6513 in our datacenter to Native IOS from
> catos. They were with catos 7 sup 2, msfc with 12.1  and now ios 12.2(18)sxf10.
> After the change we got strange issues with a Unix and ESX server.
> Sw1 and sw2  (6513 s) connected with etherchannel trunk and HSRP configured
> on them.
> 
> 1. ESX server ( see the diagram) has 6 VMware server and has three NIC
> configured a NIC team. two interfaces connected to sw1 and other connected
> to sw2. when all three interfaces are up servers 3,4 & 6 cannot reachable. I
> can see increasing output drops on sw1 interfaces. I can see the MAC
> addresses of unreachable servers learned through SW1 interfaces. SW1 has
> correct arp entries as well but servers doesnt have arp entries to default
> gateway. when I shut down the interfaces connected to sw1 everything works
> fine through the interface connected to sw2.

What happens if you just connect it to sw1 or sw2

You may be trying to do similar things to below, which won't work - see
my answer below

> 
> 2. With unix server it has two teamed network interfaces has high packet
> losses when ping to it. User complain slow performance. Here both interfaces
> of server shows same MAC addresses when I check on the switches, On SW1 unix
> server MAC address has learned through the directly connected interface. On
> SW2 switch flips the MAC of unix server through Etherchannel and Directly
> connected interfaces few times in minute. No spanning tree blocking or
> interface flaps occuring.

You can't just plug a "NIC teaming" (dumb phrase) into two separate
switches like that, when both NICs emit packets. It won't work - it
can't work.

You need one of:

 1. active/standby NIC teaming; standby NIC emits no traffic
 2. active/active with separate MAC addresses
 3. some kind of IP multipathing (not recommended)

> 
> has anybody in the list faced similar issues. any help is highly
> appreciated.

It sounds like you're causing FDB entries to flip-flop. As I say, this
won't work, you can't do it, and you'll need to find different solutions
to your problem. I would recommend active/standby NIC teaming.