[c-nsp] Understanding 10G line card oversubscription

[Mack McBride]

The 6708 has an odd port layout.
Pairs of ports connect up to an FPGA which has 16GB of bandwidth to the fabric asic.
The port pairs are 1,4;5,7;2,3; and 6,8
These pairs of ports can only send or receive 16GB in total.
The fabric asic has 20G and these combine two pairs: 1,4,5,7 and 2,4,6,8
Traffic between ports in these groups does not go over the fabric and is not counted against that BW.

The danger of packet loss is extremely dependent on burstiness of the traffic.
A general rule of thumb is over 80% of channel capacity is bad and will result in drops.
But that will not hold if the traffic is buffered or rate-limited somewhere else.

If the 20G fabric is congested 'really bad things' can happen. This is due to lost control traffic.
This is usually indicated by extremely erratic traffic and potentially even blade resets.
You will see input queue drops across the box if the fabric becomes congested on any blade.

The maximum you probably want to push in a single direction is 12.8G on a port pair and 16G on a port group.
Keep in mind communication between ports in a group is excluded in the group but included in the pair.

The 6716 is going to have similar limitations but I don't have a good document on how the port asics connect
to the fabric asic.  On the 6716 the ports are more normally ordered 1,2,3,4;5,6,7,8;9,10,11,12;13,14,15,16.
My guess (and it is only a guess) is that the quads of ports are connected down to the fabric asic via a single
FPGA and probably have the same 16G limit as the port pairs.  I can further speculate that ports 1-8 are on one
fabric channel (fabric 1) and ports 9-16 are on the other fabric channel (fabric 0).

If someone can verify the connection method and limitations on the 6716 it would be appreciated.

Mack McBride
Network Architect

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of D B
Sent: Sunday, March 20, 2011 2:13 PM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] Understanding 10G line card oversubscription

I'm looking for in-depth documentation on switch fabric oversubscription for
these two 10G line cards:
WS-X6708-10GE
WS-X6716-10GE

I'd like to understand its design and operation regarding these cards. Also,
how to identify/quantify instances where oversubscription is nearing/hitting
thresholds that will cause packet drops (SNMP?).
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/