[c-nsp] FE ignored errors

[Gert Doering]

Hi,

On Mon, Dec 20, 2004 at 10:55:55AM -0500, Jon Lewis wrote:
> > We do also see them (on NPE-400/PA-FA-TX and on 7500/VIP2-50/PA-FE-TX).
> 
> How many are you seeing?  We're seeing bursts of thousands and wondering
> if it's impacting our network.  I was talking to someone online last night
> who suggested it could be MEMD starvation.

The values that we observe vary wildly.  This is for today, for a
7500/VIP2-50/PA-FE-TX  (+PA-POS-OC3SMI):

Cisco/fe6/0/0 [20.12. 1324-1329]: +128 input errors
Cisco/fe6/0/0 [20.12. 1334-1339]: +64 input errors
Cisco/fe6/0/0 [20.12. 1354-1359]: +154701 input errors
Cisco/fe6/0/0 [20.12. 1419-1424]: +243 input errors
Cisco/fe6/0/0 [20.12. 1529-1534]: +144 input errors
Cisco/fe6/0/0 [20.12. 1534-1539]: +84 input errors
Cisco/fe6/0/0 [20.12. 1624-1629]: +59 input errors
Cisco/fe6/0/0 [20.12. 1629-1634]: +217 input errors
Cisco/fe6/0/0 [20.12. 1639-1644]: +132 input errors

(the numbers are "increments in the total input error values, in the
specified 5-minute interval").

Overall, this router has accumulated:

  Last clearing of "show interface" counters 17w6d
...
     Received 13692161 broadcasts, 0 runts, 0 giants, 17 throttles
     656350 input errors, 0 CRC, 0 frame, 0 overrun, 653461 ignored
     0 watchdog, 0 multicast

5 minute average right now is 25 Mbit/s. in, 22 Mbit/s. out.  

Most traffic is dCEF switched.


In the same time (4 hours), only one of our NPE-400s accumulated errors:

Cisco-2/fe2/0 [20.12. 1354-1359]: +81154 input errors
Cisco-2/fe2/0 [20.12. 1554-1559]: +122 input errors

the time correlation (13:54-13:59) is what I'm basing my assumption
"something is hitting these boxes in a really nasty way" on (but it's
transit, not "traffic directed at the router").

> show control vip N accum confirms this.  We're dropping packets because
> the VIPs can't get buffers on the RSP MEMD, and at times are failing to RX
> buffer.

I'm not sure how to read the output of this command.  

"show contr vip 6 accum" prints lots of stuff, but this might be
the relevant part:

show vip accumulator from Slot 6:
Buffered RX packets by accumulator:
 Forward queue 0 : 446522 in, 2578 drops (0 paks, 0/24414/24414 bufs) 100000kbps
   No MEMD buf: 446522 in
      Limit drops  : 0 normal pak drops, 0 high prec pak drops
      Buffer drops : 2578 normal pak drops, 0 high prec pak drops
 Forward queue 8 : 29321 in, 0 drops (0 paks, 0/37841/37841 bufs) 155000kbps
   No MEMD buf: 29321 in
      Limit drops  : 0 normal pak drops, 0 high prec pak drops
      Buffer drops : 0 normal pak drops, 0 high prec pak drops

so it sounds like "same problem here".

> I've turned off the policy routing, and it made no difference.
> 
> Are you still setting hold-queue <N> <in|out> on your higher traffic
> interfaces?  If so, how high?

We do (1000 in, 500 out), but I'm not sure it made any difference - it's
hard to measure, as the effects *are* highly sporadic.

Just for completeness: the 7500 is running 12.0(27)S2, and the NPEs
are running 12.2(18)S (S3...S7).

gert
-- 
USENET is *not* the non-clickable part of WWW!
                                                           //www.muc.de/~gert/
Gert Doering - Munich, Germany                             gert at greenie.muc.de
fax: +49-89-35655025                        gert at net.informatik.tu-muenchen.de