[c-nsp] 6704 input drops at less-than-linerate

[Richard A Steenbergen]

So, I'm wondering if anybody else has seen this problem... I now have 
several 6704s on several different 6500/7600 boxes which seem to be 
experiencing input queue drops for no identifiable reason, and well short 
of their expected capacity (typically starting right around 7.5g).

For example, here is the interface I'm currently seeing:

  Input queue: 0/2000/3411219/0 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  30 second input rate 7400589000 bits/sec, 912629 packets/sec
  30 second output rate 4522675000 bits/sec, 936554 packets/sec

For some reason the show interface input queue line is double-counting the 
drops, if you look at show queueing interface you see a different value, 
but they are real drops:

  Packets dropped on Receive:

    queue     dropped  [cos-map]
    ---------------------------------------------
    1          1887650  [0 1 2 3 4 5 6 7 ]

I've already eliminated all the usual suspects, bus and fabric utilization 
are very low and not even close to being full, flow control is disabled, 
the problem happens in the same way even with mls qos completely disabled, 
and there should be no output queue congestion (i.e. nothing is full 
anywhere).

The last time this happened we tried moving the affected port to a 
dedicated 6704 with 3 unused ports, and saw the same drops with the same 
amount of traffic. This was an IPv4-only simple routed /30 interface, 
nothing fancy at all (1500 byte mtu, no filters, no service-policy, no 
qos, no rate-limits, etc). 

The only thing that seems vaguely suspicious is that I also have a small 
amount of output drops on some lower speed interfaces on other modules, 
even though nothing is full, no filters/policies/rate limits, etc. For 
example, here is a freshly counter-cleared box that is exhibiting the 
input queue drops on ten1/3:

Interface Resources
  Interface drops:
    Module    Total drops:    Tx            Rx      Highest drop port:  Tx  Rx
    1                          2       3628695                           1   3
    3                       7520          5562                           9   9
    4                       3414             0                          10   0
    7                        380             0                           2   0

None of those ports with the tx drops are full, but they are lower speed 
(1g and 100m). I'm out of ideas, and so is Cisco it seems. The TAC answer 
was basically "we don't know, this should be an output buffered platform, 
buy a 6708 it has bigger buffers". I've seen other 6704 cards doing line 
rate in/out on their interfaces just fine, so I know they CAN do it, but 
something about this configuration seems to be making it routinely die at 
7.5g. Obviously I know these are lan cards without large buffers, and 
obviously I don't expect 40g performance on a single card, but at this 
rate these cards are worthless for any high-speed role. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)