[c-nsp] What to collect when router reload

[Rodney Dunn]

On Sun, May 22, 2005 at 09:30:37AM -0700, Yasser Aly wrote:
> Rodney,
>  
>  As a side note to this, how does core dump helps in analyzing hardware problems.
>

A coredump almost never helps isolate a hardware problem. They
are only really useful when it's a software problem and we need
more information about the state of variables when the crash
happens to load it in a debugger.

The crashinfo file is a mini corefile if you will.
It doesn't contain the full memory map of the router but only a small
subset. 

  
> Moreover, is it wise to make the core dump destination over a WAN link, or the router won't have enough time to ftp the core dump contents before crashing?
> 

No. It's best to do a crash to flash if you have the flash to hold the
core.

90+% of all crashes are able to be solved with:

a) A good description of the problem.
b) A great history of the router and any changes that were happening
   when it reloaded (AAA command logs, syslogs, console logs)
c) sh stack, crashinfo files

I never ask for a corefile unless using a-c fails to solve the problem
because I find too many people fail to get them correctly.

I always recommend capturing it on a local *net interface and not
over some WAN link.

 
> Regards,
> Yasser
> 
> Rodney Dunn <rodunn at cisco.com> wrote:
> You want the 'sh stack' after it reloads.
> 
> And why are you running an interim release on 12.2T?
> You should only run an interim as a last resort to get
> a bug fix while waiting for the next release to go on CCO
> and upgrade then.
> 
> Interims are not well tested so running them is a risk.
> 
> I don't know but check bootflash and see if the RPM
> saves a crashinfo file also.
> 
> Get that information to TAC.
> 
> On Sun, May 22, 2005 at 10:37:10AM +0200, Kim Onnel wrote:
> > Hi,
> > 
> > We have Cisco RPM PEs running 
> > 
> > Cisco Internetwork Operating System Software
> > IOS (tm) RPM Software (RPM-JS-M), Version 12.2(14.4)T, MAINTENANCE INTERIM 
> > SOFT
> > WARE
> > TAC Support: http://www.cisco.com/tac
> > Copyright (c) 1986-2003 by cisco Systems, Inc.
> > Compiled Mon 13-Jan-03 22:17 by ccai
> > Image text-base: 0x60008954, data-base: 0x61748000
> > 
> > ROM: System Bootstrap, Version 12.2(4r)T1, RELEASE SOFTWARE (fc1)
> > BOOTLDR: RPM Software (RPM-BOOT-M), Version 12.2(8)MC2b, EARLY DEPLOYMENT 
> > RELEAS
> > E SOFTWARE (fc3)
> > 
> > RPM-I uptime is 2 days, 19 hours, 5 minutes
> > System returned to ROM by error - a Software forced crash, PC 0x60377264
> > System restarted at 15:26:50 CAI Thu May 19 2005
> > Running default software
> > 
> > cisco RPM-PR (NPE400) processor with 491520K/32768K bytes of memory.
> > R7000 CPU at 350Mhz, Implementation 39, Rev 3.2, 256KB L2, 4096KB L3 Cache
> > Last reset from s/w peripheral
> > Bridging software.
> > X.25 software, Version 3.0.0.
> > SuperLAT software (copyright 1990 by Meridian Technology Corp).
> > TN3270 Emulation software.
> > 
> > 
> > The RPMs are probably pushing more than they can handle, what i can see is 
> > alot of self reloading reoccuring, alot of them comes back up with System 
> > returned to ROM by error - a Software forced crash
> > 
> > what i like to do is find out why they do that, if its a bug, or memory 
> > alloc. problem or a config. error that caused this to happen, how do i 
> > confirm what caused the reboot, if i need to open a TAC case, what do i need 
> > to collect?
> > 
> > I was advised to collect the result of 'sh stacks' before the RPM reloads, 
> > but i cant really guess, or should i just using linux scripts to keep 
> > collecting
> > 
> > advise ?
> > 
> > Regards
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/