[c-nsp] N7K with FEX - Port-Channel Load Distribution

[Oliver Garraux]

Hey,

Has anyone seen issues with the load balancing on port-channels going
to hosts connected to FEX's attached to Nexus 7k's?

We have a few hosts that are connected via port-channels to either a
2248 or 2232 FEX.  The FEX's are hanging off of a 7k.  This is working
as far as I can tell, in that all of the ports appear to be
operational in the port-channel.  On the switch though, I'm seeing
99.99% of the traffic being sent out of one of the links in the 4 port
port-channels (with "show port-channel traffic").  Some of the
port-channels show 100% of traffic over one link, a few are just
99.99% or something like 99.92%.  I'm seeing reasonably balanced stats
for received traffic.

One of the 7k's is using L3+L4 hashing, one is using just L3.  With
"show port-channel load-balance forwarding-path", I can see that if I
change one of the source IP's, the 7k says that it will choose a
different link in the port-channel.  I'm still waiting on one of the
server guys to let me know what IP's are connecting to these servers,
but it seems somewhat implausible to me that this is caused by there
just being a very small number of large flows that happen to get
hashed onto the same link.  I'm seeing this on all of the host facing
port-channels that we have off of the FEX's, and I see this
historically looking through our monitoring system, as well as right
now.

These are 7k's running 5.2(3a).  I'm noticing this on two different
7k's, with both 2248 and 2232 FEX's, and across 10+ different
port-channels to hosts.  I'm not seeing this on any of the routed L3
port-channels to other network devices though.  The load balancing on
the port-channels between the 7k and the FEX's themselves also looks
reasonable.

Thanks,

Oliver

-------------------------------------

Oliver Garraux
Check out my blog:  blog.garraux.net
Follow me on Twitter:  twitter.com/olivergarraux