[c-nsp] low throughput of 10GbE line

[Jirí Procházka]

Mack,

thank you for your help very much!

I've been searching about any other limitations besides 20Gbit on one ASIC 
and haven't found any.

It appeared that this was the core of the whole problem..

Te4/8 was pushing 7Gbps inbound
Te4/6 was pushing 8Gbps inbound

And that was the point where everything went bad.

I've swapped links to achieve better spread of the inbound traffic and now 
the router is handling over 56Gbps flawlessly.




Kind regards,


Jiri Prochazka


----- Original Message ----- 
From: "Mack McBride" <mack.mcbride at viawest.com>
To: "Nick Hilliard" <nick at inex.ie>; "Jirí Procházka" 
<jiri.prochazka at superhosting.cz>
Cc: <cisco-nsp at puck.nether.net>
Sent: Monday, March 15, 2010 10:38 PM
Subject: RE: [c-nsp] low throughput of 10GbE line


Those aggregates are wrong.
The port groups are arranged fairly oddly.

(fabric asic)
#show int Te7/1 capabilities | inc ASIC
  Ports-in-ASIC (Sub-port ASIC) : 1,4-5,7 (1)
#show int Te7/2 capabilities | inc ASIC
  Ports-in-ASIC (Sub-port ASIC) : 2-3,6,8 (2)

(FPGA pairs)
((1,4),(5,7)),((2,3),(6,8))

The inner pairs are 16Gbits.
The quadruples are 20Gbits.
It is easy to subscribe these blades incorrectly since the port
order is not what one would expect.

The transceiver having issues does indeed appear to be overheating.
That can cause all kinds of issues.

LR Mack McBride

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net 
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Nick Hilliard
Sent: Monday, March 15, 2010 3:17 PM
To: Jirí Procházka
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] low throughput of 10GbE line

On 15/03/2010 16:26, Jirí Procházka wrote:
> When traffic on this link reaches aproximately 6Gbps, latence to servers
> gets rapidly worse (about 100-150ms, about 2ms before)

the 6708 card has 200 megs of buffers per port.  doing the sums, this works
out at about 160ms of latency, assuming you're seeing a 10Gb microburst.
So at a superficial level, it looks like you're seeing packet drops because
of full buffers.

Also, you're running the card in oversubscription mode.  How much traffic
is te4/7 pushing?  I'd hazard a guess that you're running into
over-subscription problems on the blade.

I can't find the more detailed guide to the 6708 architecture on the cisco
web site, but there's a brief overview here:

> http://www.cisco.com/en/US/prod/collateral/switches/ps5718/ps708/prod_white_paper0900aecd80673385.html#wp9000681

while it's not going to give you exact details on tiny microbursts, I'd
consider installing RTG or YRTG with a 30 second poll interval, and monitor
all the ports on blade 4, along with the following aggregates.

te4/1 + te4/2
te4/3 + te4/4
te4/5 + te4/6
te4/7 + te4/8

te4/1 + te4/2 + te4/3 + te4/4
te4/5 + te4/6 + te4/7 + te4/8

Do you have a second 6708 blade?  You may need to consider running these
ports in non-oversubscribed mode.

Nick