[c-nsp] Exactly how bad is the 6704-10GE?

[Simon Lockhart]

All,

(This is vaguely related to my question earlier in the week about ASR capacity)

We use quite a few 6704-10GE blades on our network, and I'm seeing some
random congestion type issues. In some cases, I've made the problem go away
by shuffling ports between blades to spread the load, but I'm left wondering
exactly where the problems lie.

>From talking to people on IRC, etc, I'm told that the 6704 runs out of steam
around 24-26Gbps of throughput when handling imix traffic. I'm also told that
this is largely driven by pps, rather than bps.

If we take, for example, a 6504 on our network. It has a Sup2T in slot 1,
6704-10GE(CFC) in slot 2, 6724-SFP(CFC) in slot 3, and 6904-40G(DFC4) in slot 4.

I've got a 4*10G portchannel towards our core consisting of Te2/1, Te4/5, 
Te4/6 & Te4/8

Te2/3 and Te4/9 form a 2*10G portchannel towards an IXP

Te2/2 is a 10G link towards a transit provider.

The traffic profile on the 4*10G portchannel seems to max out at about 24Gbps.
I don't see any obvious packet drops or latency increase, just that the traffic
doesn't go any higher than that.

I suspect I'm hitting a limit on the 6704 which is causing this, but I can't
figure out what that limit is.

If I take a snapshot of the 3 active ports on the 6704 at peak time, I see:

Te2/1: In = 2.7Gbps/580kpps, Out = 5.7Gbps/613kpps
Te2/2: In = 7.0Gbps/865kpps, Out = 1.8Gbps/520kpps
Te2/3: In = 7.3Gbps/789kpps, Out = 2.5Gbps/666kpps

Summing that all up, I've got ~27Gbps of traffic flowing through the card, and
just over 4Mpps.

I also see this:

rtr#show fabric drop
  Polling interval for drop counters and timestamp is 1 in seconds 

  Packets dropped by fabric for different queues:
  Counters last cleared time: <22:54  08 Oct 14>
 slot    channel    Low-Q-drops                  High-Q-drops
    1          0              0                             0
    1          1              0                             0
    2          0          35759 @00:57 09Oct14              0
    2          1          76766 @00:57 09Oct14              0
    3          0              0                             0
    4          0            169 @00:56 09Oct14              0
    4          1              0                             0

So I seem to be seeing fabric drops on the 6704 slot, on both channels (but 
more on channel 1, which has ports Te2/1 and Te2/2 on it).

If I look at fabric utilisation, it doesn't say it's maxing out:

rtr#show fabric utilization detail
  Fabric utilization:     Ingress                    Egress
    Module  Chanl  Speed  rate  peak                 rate  peak               
    1       0        20G    0%    0%                   0%    0%               
    1       1        20G    0%    3% @19:53 08Oct14    0%    3% @19:53 08Oct14
    2       0        20G   27%   50% @22:14 08Oct14    5%   13% @22:13 08Oct14
    2       1        20G   33%   47% @00:33 09Oct14   23%   33% @23:09 08Oct14
    3       0        20G    0%    0%                   0%    0%               
    4       0        40G   11%   17% @22:30 08Oct14   26%   40% @00:02 09Oct14
    4       1        40G    0%    0%                   0%    0%               


So my questions...

1) For other people using the 6704-10GE blade, what sort of maximum throughput
   are you seeing? Have you managed to pinpoint what the limiting factor is?

2) What do the fabric drops really mean. My google-fu isn't helping a lot, and
   the command doesn't seem to be documented. Is there anything I can do to
   reduce the fabric drops? Why am I also seeing some on the 6904-40G slot,
   which should be a much more capable card.

Many thanks in advance,

Simon