[c-nsp] 7206vxr reloading with software forced crash?

[Francisco Rivas]

Hi all,

I've got a serious problem with one of my 7206vxr. It's reloading itself
several times, with a "Software forced crash". I've seen this behavior
before, and it won't happen too often, but this router in particular is
reloading at least twice a day. It doesn't have too much traffic (just
30-40Mb), and the CPU load is < 50%. 
 But, the main worry that I have about this, is that I've replaced the
chassis ENTIRELY, and the new chassis is giving me the same error. Look:


System was restarted by error - a Software forced crash, PC 0x606AED30
at 14:47:49 UTC Wed Aug 31 2005
7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Compiled Fri 08-Apr-05 13:48 by ccai
Image text-base: 0x60008BD4, data-base: 0x612C8000


This is the output of "sh region address 0x606AED30" on that chassis:

router#show region address 0x606AED30
Address 0x606AED30 is located physically in :

  Name  : text
  Class : IText
  Media : R/O
  Start : 0x60008BD4
  End   : 0x612C7413
  Size  : 0x012BE840



The old chassis was giving a similar error, not the same address, but it
was on the IText class too.
I've tried several IOS versions (12.2(14)S3, 12.2(14)S14, 12.2(18)S4)
and all of them are crashing with a similar error. Always a software
forced crash, always the same region address in the error (the region
varies from IOS to IOS, but with the same IOS, is the same region in
every crash), and always is a region on the IText class. And this is
with 2 totally different chassis (both with NPE300 / 256Mb RAM / 6 port
VXR Chassis).


The last part of the crashdump, just before the %Software-forced reload,
is this:


Aug 31 14:47:46 UTC: %SYS-3-OVERRUN: Block overrun at 20014AB8 (red zone
1F860000)
-Traceback= 60692A68 60695A04 606976D0 60697920
Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
 6770DFB4,6000011A 6770DB88,60C14048 6770DB88,60C0DED8 6770DB88,40000202
 69C13E0C,60E22530 64C26254,60E27F30 65360AA0,60E37E9C 69C13E0C,60E1F41C
Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
 69C13E0C,3000000E 6770DFB4,6000011A 6770DB88,60C12734 6770DB88,60C0DED8
 6770DB88,40000202 6770DFB4,6000011A 6770DB88,60C19E98 6601376C,60C25CA0
Aug 31 14:47:46 UTC: %SYS-6-BLKINFO: Corrupted redzone blk 20014AB8,
words 860, alloc 6063AAB8, InUse, dealloc 0, rfcnt 1
-Traceback= 60690550 60692A7C 60695A04 606976D0 60697920
Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AB8: 0xAB1234CD 0xFFFFFFFE
0x0 0x61584DD8
Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AC8: 0x6063AAB8 0x20015198
0x200143EC 0x8000035C
Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AD8: 0x1 0x0 0x0 0x0


Anyone can give me a hand on this?? I've already filtered ALL the
traffic going right to the IP address(es) of the router, but this is
still happening. I have full crashlogs and "sh stacks" and "sh tech"
available from the last crashes, so if anyone needs more info, I can
provide that without problems.
There are no changes on the config of the router since a couple of
months, so I can't blame that... anyone have any idea about this?


-- 
Francisco Rivas Catalan
Senior Network Engineer
IFX Networks
(56) 2 3744574
francisco.rivas at ifxnw.cl