[c-nsp] C720X NPE-G1 - interface errors + router freezes

[The Hawk]

Forgot to add a subject line last time...Sorry for the double post...some additional info has also been added inline.

Hello All,

We’re noticing a large number of input errors on multiple GigE interfaces on various C720X NPE-G1 routers.  These input errors match exactly with the number of overruns on those interfaces. I wouldn't be so concerned about the errors if other strange things were not happening on the routers. (such as frequent lockups and frequent eigrp drops).  At this point they go hand in hand however we need to pinpoint the origin of the problems and it hasn't been easy.

These are a couple strange things that we’ve noticed:

1.        this problem happens on more than 1 router (similar configuration on all routers)
2.       all routers affected began exhibiting these errors roughly around the same time (same early morning)
3.       All routers are G1 and ran the same IOS code (12.4.12)
4.       One of the routers was rebooted and upgraded to a different code however the problem still persists (12.4.21)
5.       we suspect it directly affects the functionality of the router as we see random lockups of the units (which may be the cause of the increased errors not the result of)
6.       All routers are connected to redundant switches and cabling/switches have been ruled out.

Below is a sample output of the show interface on one of the routers:

GigabitEthernet0/1 is up, line protocol is up
  Hardware is BCM1250 Internal MAC, address is 00b0.c2ee.b81b (bia 00b0.c2ee.b81b)
  Description: XXXXXXXX
  MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 2/255
  Encapsulation 802.1Q Virtual LAN, Vlan ID  1., loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s, media type is RJ45
  output flow-control is XON, input flow-control is XON
  ARP type: ARPA, ARP Timeout 04:00:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of 'show interface' counters never
  Input queue: 1/75/499528/97 (size/max/drops/flushes); Total output drops: 0
  Queueing strategy: fifo
  Output queue: 0/40 (size/max)
  5 minute input rate 8539000 bits/sec, 10996 packets/sec
  5 minute output rate 1905000 bits/sec, 807 packets/sec
     139728650 packets input, 10628478461 bytes, 92 no buffer
     Received 487174 broadcasts, 0 runts, 0 giants, 0 throttles
     171399 input errors, 0 CRC, 0 frame, 171399 overrun, 0 ignored
     0 watchdog, 1904930 multicast, 0 pause input
     0 input packets with dribble condition detected
     8125773 packets output, 2521690484 bytes, 0 underruns
     2 output errors, 0 collisions, 1 interface resets
     3842 unknown protocol drops
     0 babbles, 0 late collision, 0 deferred
     2 lost carrier, 0 no carrier, 0 pause output
     0 output buffer failures, 0 output buffers swapped out

Does anyone have any idea what might be going on?

I initially suspected an IOS bug (since all devices were affected) however after the upgrade the problem still persists.

Any help is appreciated.
Adrian
_________________________________________________________________