[c-nsp] debugging stack corruption
Rodney Dunn
rodunn at cisco.com
Tue Aug 19 10:41:05 EDT 2008
How are you getting this output?
If you ssh/telnet to it and run the command do you get th esame output?
That's not stack corruption to me.
Rodney
On Mon, Aug 18, 2008 at 01:10:44PM -0700, bill fumerola wrote:
>
> anyone see anything like this. i assume only a reload will fix this:
>
> rtr1#sh proc cpu | e 0.0
> CPU utilization for five seconds: 33%/8%; one minute: 37%; five minutes:
> 35%
> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY Process
> 3 528125122320274973 22 23.35% 20.79% 20.97% 0 Exec
> 70 3616544001417549298 255 0.15% 0.11% 0.12% 0 IP Input
> 115 4851843096833738 0 0.15% 0.14% 0.15% 0 HQF Shaper Backg
> rtr1#
>
> nobody else is logged on, little to no amount of traffic is running
> through the aux/cons ports, but this is interesting:
>
> rtr1#show stacks
> Minimum process stacks:
> Free/Size Name
> 5676/6000 CDP BLOB
> 8640/9000 EM ED RF
> 11052/12000 Router Init
> 8676/9000 cdp init process
> 8348/12000 Init
> 5304/6000 RADIUS INITCONFIG
> 3616/6000 BGP Open
> 2264/3000 Rom Random Update Process
> 5616/6000 URPF stats
> 5316/6000 BGP Accepter
> 9248/12000 Exec
> 7176/12000 SSH Process
> 4264/6000 TFTP Read Process
> 4204/6000 MSDP Open
> 34540/36000 TCP Command
> 5236/7200 TTY Daemon
> 8496/9000 IP-EIGRP Router
> 3360/6000
> d^\ytd^[^P^Ld^\zTd^[`Dd^[I$d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
> ,d^[mdd^\^Nld^\
> dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> ,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
> ^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
> \d^[^Add^[^Q\d^[^QLd^[
> ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
> ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
> $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> dd^[
> Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[w^Ld^[bLd^[L\d^[
> Dd^[dld^[.<d^[Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> ^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
> 6856/9000
> d^\^[Td^[T^Dd^\y^Dd^\^P<d^[@Td^[p<d^\^B,d^\^Sld^\^@dd^\dtd^\f4d^\eDd^\g^Dd^\eTd^\c^Td^\f$d^\ddd^\g^Td^[e\d^[NTd^[)$d^[pLd^[|^\d^\
> ,d^[mdd^\^Nld^\
> dd^[ 4d^[Q<d^[^V^\d^\1dd^[1<d^[O4d^[|Dd^\^P<d^[^Ydd^[e\d^[)$d^[NTd^[
> 4d^[1^Dd^[`Td^[{td^[^E^\d^[m<d^[^Z^Ld^[8<d^[}^Dd^[j^\d^\^Q|d^[x^\d^[u^\d^\
> ,d^\^ALd^[jTd^[pLd^[|^\d^[~td^[^D,d^[RDd^ld^[x$d^[^^Dd^[ptd^[^Bld^[^QLd^[^Q\d^[
> ld^[zdd^\,$d^[ttd^[^Vdd^[iLd^[^X\d^[)4d^\34d^[v$d^[^VTd^\^Ptd^^\d^[{Dd^[R|d^\^Q^\d^[`^Ld^[]^Ld^\
> Minimum process stacks:
> Free/Size Name
> ,d^[^R^Dd^[^Fld^[\d^[b^Td^[^LDd^\^P^Dd^[^B4d^[^NLd^[^Y,d^[^Kdd^\
> ^\d^\^CDd^[s^Td^[^A^\d^[U,d^[j,d^[~^Dd^\^QDd^[Jtd^[~Ld^[|^Td^[,Dd^^\d^[rld^[R|d^[{Dd^[
> \d^[^Add^[^Q\d^[^QLd^[
> ld^[ttd^[zdd^\,$d^[^Vdd^[)4d^\34d^[wLd^[m,d^[^Z|d^[\,d^[g|d^[y|d^[^D<d^[x$d^[^^Dd^[
> ld^[^Bld^[RDd^[ptd^[^Q$d^[v4d^\^Ptd^[^VTd^[7$d^\1td^[P$d^[uTd^[^VTd^[zdd^[7$d^[z,d^[z^\d^[ytd^[@Td^[<^Dd^\,$d^\+Dd^\,4d^[^D<d^\`^Dd^[^VTd^[k4d^[P^Td^[a$d^[$d^[^V^\d^[^Utd^[mdd^[^Ytd^[|^Ld^[^L^Ld^\^ALd^[#^Dd^[e\d^[f^Dd^\^FTd^[^Pld^[^B|d^[n^\d^[d4d^[H|d^[^Rtd^[^N^Td^Td^[^Td^[{,d^[+dd^[`Td^[.^Dd^[s\d^[^ETd^[^Z^Ld^[
> $d^[YTd^\^L^Dd^[1^Dd^[^O^\d^[^PDd^[^L^\d^\
> dd^[
> Ld^[)$d^[#td^[1<d^[^E|d^[^_Ld^[KTd^[
> 4d^[^BDd^[yLd^[+,d^[^E^\d^\^S^Dd^[
> 4d^[y^Td^[^WDd^[l\d^[Y|d^\1^Dd^\0$d^\/Dd^\1dd^[{^Dd^[^SDd^[^LTd^[|^\d^[H4d^[pLd^[M<d^[.,d^[]ld^[Q<d^[U^\d^[~td^[l$d^[8<d^[6^Ld^[^F4d^[^Odd^\^O$d^[^K<d^[^N<d^[^K^Dd^[^W4d^[_,d^[p^Dd^[+^\d^[N,d^[$Td^[~^\d^[eLd^[NTd^[
> ,d^[xTd^[r4d^[u^\d^[n^Ld^[rDd^[p^Td^[{td^[~<d^\
> ,d^[}$d^[}^Dd^[P\d^[w|d^[mtd^[O4d^[{ld^[x\d^[?<d^[m<d^[
> <d^[o4d^[w<d^[y<d^[*<d^\^P<d^[u|d^[^Ydd^\^Pdd^[^Y<d^[D|d^\^P,d^[.td^\^Nld^\^N4d^[|Dd^[$^\d^[jTd^[q,d^[j^\d^[\Td^\^Q|d^[f,d^[^D,d^[gDd^[x^\d^[]4d^\&Dd^[w^Ld^[bLd^[L\d^[
> Dd^[dld^[.<d^[Lld^\ td^\4d^\ld^^Td^\d^\ <d^\
> ^Dd^Ld^$d^[,d^[dd^[^\d^[Td^\
> 10468/12000 HSRP (Standby)
>
> Interrupt level stacks:
> Level Called Unused/Size Name
> 1 2648551315 6280/9000 Network interfaces
> 2 0 9000/9000 DMA/Timer Interrupt
> 3 185107 7472/9000 PA Management Int Handler
> 4 1715750501 8444/9000 Console Uart
> 5 0 9000/9000 OIR/Error Interrupt
> 7 3207930022 8532/9000 NMI Interrupt Handler
>
> Spurious interrupts: 233
> rtr1#
>
> and on a different router:
>
> rtr1.chi#sh stacks
> Minimum process stacks:
> Free/Size Name
> [....]
> 3500/6000
> 7160/9000 5,<$/jDSw_h 5,< 5,< 5,< 5,< 5,< d(X d(X 5,< 5,< 5,< 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< d'X 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,<V7;@ 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^BT 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,< 5,< 5,< 5,< 5,<Q6mT 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,<#^Qz|#^Qy|#^Qy| 5,<#^Qx|#^Qx| 5,<%Dtx%Dtx%Dtx%Dtx%Dsx%Dsx%Dsx%Dsx 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,<%Dsx 5,< 5,< 5,<%Drx 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,<#^Qw|#^Qw|#^Qv| 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,<S3^R@ 5,< 5,< 5,<Vo^BT d'X d&X d&X d%X 5,< 5,< 5,<&S<<!A`t 5,< 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<#W:x#W9x
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<Vo^AT-4,4 5,<PN^P^T 5,<
> 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,< 5,<
> 5,< 5,<Vo^ATVo
> 5316/6000 BGP Accepter
> 10176/12000 Exec
>
> although that router doesn't display the same CPU symptoms.
>
> first router is running:
> Cisco IOS Software, 7301 Software (C7301-K91P-M), Version 12.2(31)SB11, RELEASE SOFTWARE (fc3)
> ROM: System Bootstrap, Version 12.3(4r)T4, RELEASE SOFTWARE (fc1)
> BOOTLDR: 7301 Software (C7301-BOOT-M), Version 12.3(26), RELEASE SOFTWARE (fc2)
>
> second router is running:
> Cisco IOS Software, 7301 Software (C7301-K91P-M), Version 12.2(31)SB12,
> RELEASE SOFTWARE (fc3)
>
>
> -- bill
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list