[c-nsp] C720X NPE-G1 - interface errors + router freezes

[The Hawk]

Hello Nick, Thanks for your suggestion.  I will try this one of these nights although I strongly believe that it will not solve this particular issue.  As I mentioned in the original post, we have multiple NPE-G1s that all started experiencing the same issue on the same day (early morning around 4:00AM).... I'm leaning towards some sort of attack that's happening on these routers to exploit a known vulnerability.   I was really hoping that the IOS upgrade would have fixed that but no luck there. Based on interface reports these GIGe Interfaces are not pushing more than 10 - 40Mb of traffic through them ... if it is some sort of attack, it must be using small packets or once again, a known vulnerability is exploited. Adrian> Subject: RE: [c-nsp] C720X NPE-G1 - interface errors + router freezes> Date: Fri, 5 Sep 2008 11:49:35 +1000> From: nick.geyer at eds.com> To: acidutu at hotmail.com; cisco-nsp at puck.nether.net> > I have seen issues like you mention creep up on NPE-G1's that have been> in service for a while. It all starts with a few input errors here and> there and progressively gets worse.> > Reseating the NPE seems to clear up all the issues and it starts> chugging along happily again. Possibly worth a try on one of your> routers to see if it makes a difference.> > Nick> > -----Original Message-----> From: cisco-nsp-bounces at puck.nether.net> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of The Hawk> Sent: Friday, 5 September 2008 11:12 AM> To: cisco-nsp at puck.nether.net> Subject: [c-nsp] C720X NPE-G1 - interface errors + router freezes> > > > Forgot to add a subject line last time...Sorry for the double> post...some additional info has also been added inline.> > Hello All,> > We're noticing a large number of input errors on multiple GigE> interfaces on various C720X NPE-G1 routers. These input errors match> exactly with the number of overruns on those interfaces. I wouldn't be> so concerned about the errors if other strange things were not happening> on the routers. (such as frequent lockups and frequent eigrp drops). At> this point they go hand in hand however we need to pinpoint the origin> of the problems and it hasn't been easy.> > These are a couple strange things that we've noticed:> > 1. this problem happens on more than 1 router (similar> configuration on all routers)> 2. all routers affected began exhibiting these errors roughly> around the same time (same early morning)> 3. All routers are G1 and ran the same IOS code (12.4.12)> 4. One of the routers was rebooted and upgraded to a different> code however the problem still persists (12.4.21)> 5. we suspect it directly affects the functionality of the router> as we see random lockups of the units (which may be the cause of the> increased errors not the result of)> 6. All routers are connected to redundant switches and> cabling/switches have been ruled out.> > Below is a sample output of the show interface on one of the routers:> > GigabitEthernet0/1 is up, line protocol is up> Hardware is BCM1250 Internal MAC, address is 00b0.c2ee.b81b (bia> 00b0.c2ee.b81b)> Description: XXXXXXXX> MTU 1500 bytes, BW 1000000 Kbit, DLY 10 usec,> reliability 255/255, txload 1/255, rxload 2/255> Encapsulation 802.1Q Virtual LAN, Vlan ID 1., loopback not set> Keepalive set (10 sec)> Full-duplex, 1000Mb/s, media type is RJ45> output flow-control is XON, input flow-control is XON> ARP type: ARPA, ARP Timeout 04:00:00> Last input 00:00:00, output 00:00:00, output hang never> Last clearing of 'show interface' counters never> Input queue: 1/75/499528/97 (size/max/drops/flushes); Total output> drops: 0> Queueing strategy: fifo> Output queue: 0/40 (size/max)> 5 minute input rate 8539000 bits/sec, 10996 packets/sec> 5 minute output rate 1905000 bits/sec, 807 packets/sec> 139728650 packets input, 10628478461 bytes, 92 no buffer> Received 487174 broadcasts, 0 runts, 0 giants, 0 throttles> 171399 input errors, 0 CRC, 0 frame, 171399 overrun, 0 ignored> 0 watchdog, 1904930 multicast, 0 pause input> 0 input packets with dribble condition detected> 8125773 packets output, 2521690484 bytes, 0 underruns> 2 output errors, 0 collisions, 1 interface resets> 3842 unknown protocol drops> 0 babbles, 0 late collision, 0 deferred> 2 lost carrier, 0 no carrier, 0 pause output> 0 output buffer failures, 0 output buffers swapped out> > Does anyone have any idea what might be going on?> > I initially suspected an IOS bug (since all devices were affected)> however after the upgrade the problem still persists.> > Any help is appreciated.> Adrian> _________________________________________________________________> > _______________________________________________> cisco-nsp mailing list cisco-nsp at puck.nether.net> https://puck.nether.net/mailman/listinfo/cisco-nsp> archive at http://puck.nether.net/pipermail/cisco-nsp/
_________________________________________________________________