[c-nsp] debugging stack corruption

[bill fumerola]

On Tue, Aug 19, 2008 at 10:41:05AM -0400, Rodney Dunn wrote:
> How are you getting this output?

ssh rtr1
en
sh stacks

> If you ssh/telnet to it and run the command do you get th esame output?

it is not signal noise (serial spew, ip corruption, etc).

> That's not stack corruption to me.

i'll try and profile the exec process, but i'm not so good w/ profiling
and tracing w/o at least symbols.

there is also the matter of the 30% solid EXEC process. however, the
switch that device is attached to (both in network and by serial via
rtr1:aux<>sw1:cons) is exhibiting the same behavior. it could be a feedback
loop on the serial connection, but i've tried turning all of that down
and still no relief. the jump occurred to both at the same time.

it could just be corruption in the display, but the CPU spike is what
made me investigate in the first place.

-- bill

> > rtr1#show stacks
> > Minimum process stacks:
> >  Free/Size   Name
[...]
> >  3360/6000
> > d^\ytd^[^P^Ld^\zTd^[`Dd^[I$d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
> > ,d^[mdd^\^Nld^\
> > dd^[	4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > ,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
> > ^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
> > \d^[^Add^[^Q\d^[^QLd^[
> > ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
> > ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
> > $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> > dd^[
> > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[w^Ld^[bLd^[L\d^[
> > Dd^[dld^[.<d^[Lld^\	td^\4d^\ld^^Td^\d^\	<d^\
> > ^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
> >  6856/9000
> > d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
> > ,d^[mdd^\^Nld^\
> > dd^[	4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > Minimum process stacks:
> >  Free/Size   Name
> > ,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
> > ^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
> > \d^[^Add^[^Q\d^[^QLd^[
> > ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
> > ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
> > $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> > dd^[
> > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[w^Ld^[bLd^[L\d^[
> > Dd^[dld^[.<d^[Lld^\	td^\4d^\ld^^Td^\d^\	<d^\
> > ^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
> > 10468/12000  HSRP (Standby)
> > 
> > Interrupt level stacks:
> > Level    Called Unused/Size  Name
> >   1  2648551315   6280/9000  Network interfaces
> >   2           0   9000/9000  DMA/Timer Interrupt
> >   3      185107   7472/9000  PA Management Int Handler
> >   4  1715750501   8444/9000  Console Uart
> >   5           0   9000/9000  OIR/Error Interrupt
> >   7  3207930022   8532/9000  NMI Interrupt Handler
> > 
> > Spurious interrupts: 233
> > rtr1#
> > 
> > and on a different router:
> > 
> > rtr1.chi#sh stacks
> > Minimum process stacks:
> >  Free/Size   Name
> > [....]
> >  3500/6000  
> >  7160/9000    5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
> > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > 5,< 5,<Vo^ATVo
> >  5316/6000   BGP Accepter