[c-nsp] debugging stack corruption
Rodney Dunn
rodunn at cisco.com
Wed Aug 20 09:34:10 EDT 2008
If you pull that cable on the aux/console do you still see the
saem thing?
On Tue, Aug 19, 2008 at 11:49:43AM -0700, bill fumerola wrote:
> On Tue, Aug 19, 2008 at 10:41:05AM -0400, Rodney Dunn wrote:
> > How are you getting this output?
>
> ssh rtr1
> en
> sh stacks
>
> > If you ssh/telnet to it and run the command do you get th esame output?
>
> it is not signal noise (serial spew, ip corruption, etc).
>
> > That's not stack corruption to me.
>
> i'll try and profile the exec process, but i'm not so good w/ profiling
> and tracing w/o at least symbols.
>
> there is also the matter of the 30% solid EXEC process. however, the
> switch that device is attached to (both in network and by serial via
> rtr1:aux<>sw1:cons) is exhibiting the same behavior. it could be a feedback
> loop on the serial connection, but i've tried turning all of that down
> and still no relief. the jump occurred to both at the same time.
>
> it could just be corruption in the display, but the CPU spike is what
> made me investigate in the first place.
>
> -- bill
>
> > > rtr1#show stacks
> > > Minimum process stacks:
> > > Free/Size Name
> [...]
> > > 3360/6000
> > > d^\ytd^[^P^Ld^\zTd^[`Dd^[I$d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
> > > ,d^[mdd^\^Nld^\
> > > dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> > > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > > ,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
> > > ^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
> > > \d^[^Add^[^Q\d^[^QLd^[
> > > ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
> > > ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
> > > $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> > > dd^[
> > > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> > > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[w^Ld^[bLd^[L\d^[
> > > Dd^[dld^[.<d^[Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> > > ^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
> > > 6856/9000
> > > d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
> > > ,d^[mdd^\^Nld^\
> > > dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> > > 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> > > ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> > > ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> > > Minimum process stacks:
> > > Free/Size Name
> > > ,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
> > > ^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
> > > \d^[^Add^[^Q\d^[^QLd^[
> > > ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
> > > ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
> > > $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> > > dd^[
> > > Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> > > 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> > > 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> > > ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> > > ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> > > <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[w^Ld^[bLd^[L\d^[
> > > Dd^[dld^[.<d^[Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> > > ^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
> > > 10468/12000 HSRP (Standby)
> > >
> > > Interrupt level stacks:
> > > Level Called Unused/Size Name
> > > 1 2648551315 6280/9000 Network interfaces
> > > 2 0 9000/9000 DMA/Timer Interrupt
> > > 3 185107 7472/9000 PA Management Int Handler
> > > 4 1715750501 8444/9000 Console Uart
> > > 5 0 9000/9000 OIR/Error Interrupt
> > > 7 3207930022 8532/9000 NMI Interrupt Handler
> > >
> > > Spurious interrupts: 233
> > > rtr1#
> > >
> > > and on a different router:
> > >
> > > rtr1.chi#sh stacks
> > > Minimum process stacks:
> > > Free/Size Name
> > > [....]
> > > 3500/6000
> > > 7160/9000 5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
> > > 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> > > 5,< 5,<Vo^ATVo
> > > 5316/6000 BGP Accepter
More information about the cisco-nsp
mailing list