[c-nsp] 'show isis database' delayed crash on 15.0(1)M1
Bryan Fields
Bryan at bryanfields.net
Sat Feb 6 20:55:12 EST 2010
I was trouble shooing my network today and found a nasty little bug when some
one does 'show isis database' from exec mode on C181X Software
(C181X-ADVIPSERVICESK9-M), Version 15.0(1)M1, IOS.
After issuing the command you get the output of it, and some time in the next
30 sec the router crashes.
example:
LTRKAKHQR01-c1811w#sh isis database
IS-IS Level-1 Link State Database:
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
STPBFLGURT1.00-00 0x00005F0A 0x49A3 907 0/0/0
galaxydoor.00-00 0x0000200A 0x2DF0 900 0/0/0
LTRKAKHQR01-c1.00-00* 0x00000953 0x64C1 1099 0/0/0
TAMQFLTART1.00-00 0x00005859 0x1542 908 0/0/0
IS-IS Level-2 Link State Database:
LSPID LSP Seq Num LSP Checksum LSP Holdtime ATT/P/OL
STPBFLGURT1.00-00 0x000060A8 0x1149 914 0/0/0
galaxydoor.00-00 0x0000200F 0x645F 912 0/0/0
LTRKAKHQR01-c1.00-00* 0x00000991 0xF41F 916 0/0/0
TAMQFLTART1.00-00 0x00005926 0x83FD 913 0/0/0
LTRKAKHQR01-c1811w#term mon
LTRKAKHQR01-c1811w#sh clock
01:44:47.438 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:44:56.418 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:01.690 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:06.182 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:10.146 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:12.658 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
01:45:16.222 UTC Sun Feb 7 2010
LTRKAKHQR01-c1811w#sh clock
______BAM! Lockup at this point______
from the log output:
Feb 6 20:45:27 192.168.3.210 103: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:20
UTC: %SYS-3-CPUHOG: Task is running for (2000)msecs, more than (2000)msecs
(0/0),process = Check heaps.
Feb 6 20:45:27 192.168.3.210 104: LTRKAKHQR01-c1811w: -Traceback= 0x8007CCB0z
0x80B20C18z 0x80B22C8Cz 0x80B20EC8z 0x82050E18z 0x82052364z 0x82052770z
0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
Feb 6 20:45:27 192.168.3.210 105: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:22
UTC: %SYS-3-CPUHOG: Task is running for (4000)msecs, more than (2000)msecs
(0/0),process = Check heaps.
Feb 6 20:45:27 192.168.3.210 106: LTRKAKHQR01-c1811w: -Traceback= 0x8007CCB8z
0x80B20C18z 0x80B22528z 0x80B20EC8z 0x820500E4z 0x82050E54z 0x82052364z
0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
Feb 6 20:45:27 192.168.3.210 107: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-3-BADMAGIC: Corrupt block at 86AC28DC (magic 813E0508), -Traceback=
0x82052388z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
Feb 6 20:45:27 192.168.3.210 108: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-MTRACE: mallocfree: addr, pc
Feb 6 20:45:27 192.168.3.210 109: LTRKAKHQR01-c1811w: 86297A44,80BAE58C
86297A44,40000294 86CC08A0,80BAE570 86CC08A0,3000021E
Feb 6 20:45:27 192.168.3.210 110: LTRKAKHQR01-c1811w: 86DC8180,8154FFC0
866536C8,8154FE24 866536C8,8154FE24 866536C8,8154FE88
Feb 6 20:45:27 192.168.3.210 111: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-MTRACE: mallocfree: addr, pc
Feb 6 20:45:27 192.168.3.210 112: LTRKAKHQR01-c1811w: 8666F860,81569A98
866536C8,8154FE88 866536C8,8154FE24 866536C8,8154FE24
Feb 6 20:45:27 192.168.3.210 113: LTRKAKHQR01-c1811w: 866536C8,8154EE70
866536C8,8154EE70 866536C8,8154EE70 866536C8,8154EE70
Feb 6 20:45:27 192.168.3.210 114: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-BLKINFO: Corrupted magic value in in-use block blk 86AC28DC, words
6002, alloc 8012DAC4, InUse, dealloc FFFFFFFF, rfcnt 1, -Traceback=
0x82010150z 0x82052618z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz
0x80124418z
Feb 6 20:45:28 192.168.3.210 115: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-MEMDUMP: 0x86AC28DC: 0x813E0508 0x0 0x0 0x8364CCBC
Feb 6 20:45:28 192.168.3.210 116: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-MEMDUMP: 0x86AC28EC: 0x8012DAC4 0x86AC57F0 0x86AB9C1C 0x80001772
Feb 6 20:45:28 192.168.3.210 117: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-MEMDUMP: 0x86AC28FC: 0x1 0x86AC2980 0x15C 0x86D41800
Feb 6 20:45:28 192.168.3.210 118: LTRKAKHQR01-c1811w: Feb 7 2010 01:45:24
UTC: %SYS-6-STACKLOW: Stack for process Virtual Exec running low, 12/12000
Feb 6 20:45:40 192.168.3.250 1038: TAMQFLTART1: Feb 7 2010 01:45:39 UTC:
%LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed state to down
-------
Some times I'll get a crashinfo file, other times I will not.
>From a previous crash info:
-------------
CMD: 'sh isis database' 21:23:27 UTC Sat Feb 6 2010
validblock_diagnose, code = 2
current memory block, bp = 0x8700E0B0,
memorypool type is Processor
data check, ptr = 0x8700E0E0
next memory block, bp = 0x87010FC4,
memorypool type is Processor
data check, ptr = 0x87010FF4
previous memory block, bp = 0x870053DC,
memorypool type is Processor
data check, ptr = 0x8700540C
========= Dump bp = 0x8700E0B0 ======================
8700DFB0: 0 8700EAB0 FFFFFFFF 0 0 0 0 0
8700DFD0: 0 0 6347E519 0 8207070C D02688F2 6347E519 85F7C994
8700DFF0: 85F7C994 811AEC4C 8700E010 811AB16C D0D0D0D 245EBB78 D0D0D0D 867CD2F4
8700E010: 8700E040 813B2838 D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D 82070710
8700E030: 85F7C994 6347E519 875E3258 875E3214 8700E070 813B2B8C D0D0D0D 48822022
8700E050: 1 8700E2D8 0 0 8700E070 6347E519 8700E400 0
8700E070: 8700E0B0 813B4470 0 0 28822022 6347E519 0 0
8700E090: 0 0 6347E519 85F7C994 0 0 8700E400 0
8700E0B0: 8700E350 813E0508 0 0 8012DAC4 87010FC4 870053F0 80001772
8700E0D0: 1 0 8700E158 872550DC FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
8700E0F0: 0 0 FFFFFFFF FFFFFFFF FFFF FFFFFFFF FFFFFFFF FFFFFFFF
8700E110: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 0
8700E130: 2F 86427028 0 85F7C994 0 6347E519 6347E519 245EBB78
8700E150: 8700E2F0 867CD2F4 8700E1D8 811ABDE0 FFFFFFFF 6347E519 D02688F2 2F
8700E170: 0 0 0 C0 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
8700E190: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
=========
Feb 6 2010 21:24:09 UTC: %SYS-3-CPUHOG: Task is running for (2000)msecs, more
than (2000)msecs (1/1),process = Check heaps.
-Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22C8Cz 0x80B20EC8z 0x82050E18z
0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z Dump
bp->next = 0x87010FC4 ======================
87010EC4: 61780000 87010EF0 84228082 73796E74 A4CB80 7002FD0 87010F20 87010E70
87010EE4: 87010EF8 83EB0000 83EB0000 0 83EB0000 0 A4CB80 0
87010F04: 0 867F2054 0 0 86493648 87010FB0 80B77310 FFFFFF
87010F24: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
87010F44: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 2A 1 FFFFFFFF
87010F64: FFFFFFFF 0 0 0 0 0 0 0
87010F84: 0 0 0 0 0 0 0 0
87010FA4: 0 0 0 87010FB8 8012086C 0 80124418 FD0110DF
87010FC4: AB1234CD E40000 15F 873074D0 80B4FCA0 87015E18 8700E0C4 80002712
87010FE4: 1 8200EA4C 166 872550DC 0 0 87307494 0
87011004: 87307494 258 2C7 140018 2C1 0 0 0
87011024: 0 430000 83EC2BBC 41414120 536D616C 6C204368 756E0000 87011B6C
87011044: 87015E14 0 0 87011B70 87011B88 87011BA0 87011BB8 87011BD0
87011064: 87011BE8 87011C00 87011C18 87011C30 87011C48 87011C60 87011C78 87011C90
87011084: 87011CA8 87011CC0 87011CD8 87011CF0 87011D08 87011D20 87011D38 87011D50
870110A4: 87011D68 87011D80 87011D98 87011DB0 87011DC8 87011DE0 87011DF8 87011E10
========== Dump bp->previous = 0x870053F0 =====================
870052F0: 0 0 0 0 0 0 0 0
87005310: 0 FD0110DF AB1234CD FFFE0000 0 82FC74AC 81BDE144 87005390
87005330: 870052A8 80000024 1 0 1 850B5B2C 83824BA0 0
87005350: 0 0 1 0
Feb 6 2010 21:24:11 UTC: %SYS-3-CPUHOG: Task is running for (4000)msecs, more
than (2000)msecs (1/1),process = Check heaps.
-Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22528z 0x80B20EC8z 0x820500E4z
0x82050E54z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz
0x80124418z 0 0 0 90000
87005370: 1 870051FC 0 0 0 0 0 FD0110DF
87005390: AB1234CD FFFE0000 0 82FC74AC 81BD9178 870053DC 8700532C 8000000E
870053B0: 1 0 1 850B5B2C 1 0 0 0
870053D0: 0 0 FD0110DF AB1234CD 750000 75 83646E94 82C4EED4
870053F0: 8700E0B0 870053A4 4652 0 82C89068 7C 850B1410 DEADBEEF
87005410: 82C89068 0 D0D0D0D 83EC321C 83EC3218 D0D0D0D D0D0D0D D0D0D0D
87005430: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
87005450: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
87005470: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
87005490: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
870054B0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
870054D0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
============================================
Feb 6 2010 21:24:12 UTC: %SYS-3-BADMAGIC: Corrupt block at 8700E0B0 (magic
8700E350), -Traceback= 0x82052388z 0x82052770z 0x82055410z 0x820555CCz
0x8012086Cz 0x80124418z
Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
873068BC,80BAE58C 873068BC,40000294 859EC9B4,80BAE570 859EC9B4,3000021E
86EAC770,8154FFC0 86EA8998,8154FE24 86EA8998,8154FE24 859EC9B4,81540E3C
Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
859EC9B4,8153B354 859EC9B4,3000021E 86EA8998,8154FE88 86EAB098,81569A98
86EA8998,8154FE88 86EA8998,8154FE24 86EA8998,8154FE24 86EA8998,8154EE70
Feb 6 2010 21:24:12 UTC: %SYS-6-BLKINFO: Corrupted magic value in in-use
block blk 8700E0B0, words 6002, alloc 8012DAC4, InUse, dealloc FFFFFFFF, rfcnt
1, -Traceback= 0x82010150z 0x82052618z 0x82052770z 0x82055410z 0x820555CCz
0x8012086Cz 0x80124418z
Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0B0: 0x8700E350 0x813E0508
0x0 0x0
Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0C0: 0x8012DAC4 0x87010FC4
0x870053F0 0x80001772
Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0D0: 0x1 0x0 0x8700E158
0x872550DC
%Software-forced reload
21:24:12 UTC Sat Feb 6 2010: Unexpected exception to CPU: vector 1500, PC =
0x8011E220, LR = 0x8011E1E4
-Traceback= 0x8011E220z 0x8011E1E4z 0x82052770z 0x82055410z 0x820555CCz
0x8012086Cz 0x80124418z
CPU Register Context:
MSR = 0x02029220 CR = 0x28000042 CTR = 0x81F26400 XER = 0x00000000
R0 = 0x8011E1E4 R1 = 0x8511CBA8 R2 = 0xFFE97C10 R3 = 0x83FA9978
R4 = 0x82F869BC R5 = 0x00000000 R6 = 0x83970000 R7 = 0x82F60000
R8 = 0x02029220 R9 = 0x83AD0000 R10 = 0x00000000 R11 = 0x00000000
R12 = 0x24000088 R13 = 0xFFE994A8 R14 = 0x820554DC R15 = 0x00000000
R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
R20 = 0x00000000 R21 = 0x00000000 R22 = 0x83D60000 R23 = 0x83D60000
R24 = 0xAB1234AB R25 = 0xAB1234CD R26 = 0x83D60000 R27 = 0x85FBD91C
R28 = 0x00000000 R29 = 0x83647534 R30 = 0x83980000 R31 = 0x00000000
------
I've tried this on both 1811w's on my network and had the exact same problems.
Any one else seen this or know if it's a known bug? I've searched the cisco
site and cannot find a reference to this issue.
--
Bryan Fields
727-409-1194 - Voice
727-214-2508 - Fax
http://bryanfields.net
More information about the cisco-nsp
mailing list