[c-nsp] N7K with FEX - Port-Channel Load Distribution

[Tóth András]

Hi Oliver,

Verify the port-channel load-balance configuration and make sure it's set
to dest-ip-port or try out other options for more granular distribution of
traffic. The default load-balancing mode for Layer 3 interfaces is the
source and destination IP address. The load-balance configuration needs to
be done in the default VDC.

http://www.cisco.com/en/US/docs/switches/datacenter/sw/5_x/nx-os/interfaces/configuration/guide/if_portchannel.html

After changing the load-balance method to include L4 ports, you could run
multiple iterations of iperf which will by default use a random source port
so you can check if traffic is hashed to multiple links.

Best regards,
Andras


On Mon, Dec 17, 2012 at 12:09 AM, Oliver Garraux <oliver at g.garraux.net>wrote:

> Hey,
>
> Has anyone seen issues with the load balancing on port-channels going
> to hosts connected to FEX's attached to Nexus 7k's?
>
> We have a few hosts that are connected via port-channels to either a
> 2248 or 2232 FEX.  The FEX's are hanging off of a 7k.  This is working
> as far as I can tell, in that all of the ports appear to be
> operational in the port-channel.  On the switch though, I'm seeing
> 99.99% of the traffic being sent out of one of the links in the 4 port
> port-channels (with "show port-channel traffic").  Some of the
> port-channels show 100% of traffic over one link, a few are just
> 99.99% or something like 99.92%.  I'm seeing reasonably balanced stats
> for received traffic.
>
> One of the 7k's is using L3+L4 hashing, one is using just L3.  With
> "show port-channel load-balance forwarding-path", I can see that if I
> change one of the source IP's, the 7k says that it will choose a
> different link in the port-channel.  I'm still waiting on one of the
> server guys to let me know what IP's are connecting to these servers,
> but it seems somewhat implausible to me that this is caused by there
> just being a very small number of large flows that happen to get
> hashed onto the same link.  I'm seeing this on all of the host facing
> port-channels that we have off of the FEX's, and I see this
> historically looking through our monitoring system, as well as right
> now.
>
> These are 7k's running 5.2(3a).  I'm noticing this on two different
> 7k's, with both 2248 and 2232 FEX's, and across 10+ different
> port-channels to hosts.  I'm not seeing this on any of the routed L3
> port-channels to other network devices though.  The load balancing on
> the port-channels between the 7k and the FEX's themselves also looks
> reasonable.
>
> Thanks,
>
> Oliver
>
> -------------------------------------
>
> Oliver Garraux
> Check out my blog:  blog.garraux.net
> Follow me on Twitter:  twitter.com/olivergarraux
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>