[c-nsp] How Cisco TAC troubleshoots router/switch crash using the stack frames?

[Michael Loftis]

On Mon, Jun 3, 2013 at 4:53 AM, Martin T <m4rtntns at gmail.com> wrote:
> Hello,
>
> 1) Am I correct that IOS, similarly to Linux kernel, has a separate
> stack for each process and each time a process makes a function call,
> a stack frame is added to stack containing information about function,
> it's arguments and variables and removed from stack once function
> returns to caller? As there is a command "show stack <PID>" I guess
> there is a stack per process..

Sort of.  Except that everything is known in (classic) IOS - all code
and all addresses are known, so when the below happens, Cisco knows
exactly what function call was executed since each process shares all
the code address space.  IIRC some IOS really are actually one big
monolithic process, what it calls processes aren't really fully
independent processes but more like a task queue or similar.  And
since IOS doesn't dynamically load code, addresses (pc) are fully
known for a given version.   They can map back to source code lines
very easily.

>
>
> 2) For example if following information is logged to crashinfo file
> during the L3 switch crash right before the forced reload:
>
> Traceback: 110DFFB8 10B04E98 11A391D0 116F6428 1172171C 10A8FA34 10A86BB4
>
> Stack frames:
> Frame 1: pc=10B04E98 stack=20D0D670
> Frame 2: pc=11A391D0 stack=20D0D678
> Frame 3: pc=116F6428 stack=20D0D6B0
> Frame 4: pc=1172171C stack=20D0D730
> Frame 5: pc=10A8FA34 stack=20D0D7B0
> Frame 6: pc=10A86BB4 stack=20D0D7B8
>
>
> ..then how can TAC engineer trace back to certain function call? I
> understand that TAC engineers have probably access to IOS source code,
> symbol tables etc, but how how are the stack frames listed on
> "Traceback" line mapped to actual function calls and processes?
>
>
> regards,
> Martin
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/



--

"Genius might be described as a supreme capacity for getting its possessors
into trouble of all kinds."
-- Samuel Butler