[c-nsp] 'show isis database' delayed crash on 15.0(1)M1
Eninja
eninja at gmail.com
Sun Feb 7 18:00:29 EST 2010
Bryan,
Your box crashed because the memory got corrupted. This is a software
bug. Set it up for a core dump and send to bug manufacturer for
rectification.
/eninja
On Feb 7, 2010, at 2:55 AM, Bryan Fields <Bryan at bryanfields.net> wrote:
> I was trouble shooing my network today and found a nasty little bug
> when some
> one does 'show isis database' from exec mode on C181X Software
> (C181X-ADVIPSERVICESK9-M), Version 15.0(1)M1, IOS.
>
> After issuing the command you get the output of it, and some time in
> the next
> 30 sec the router crashes.
>
> example:
> LTRKAKHQR01-c1811w#sh isis database
>
> IS-IS Level-1 Link State Database:
> LSPID LSP Seq Num LSP Checksum LSP Holdtime
> ATT/P/OL
> STPBFLGURT1.00-00 0x00005F0A 0x49A3 907
> 0/0/0
> galaxydoor.00-00 0x0000200A 0x2DF0 900
> 0/0/0
> LTRKAKHQR01-c1.00-00* 0x00000953 0x64C1 1099
> 0/0/0
> TAMQFLTART1.00-00 0x00005859 0x1542 908
> 0/0/0
> IS-IS Level-2 Link State Database:
> LSPID LSP Seq Num LSP Checksum LSP Holdtime
> ATT/P/OL
> STPBFLGURT1.00-00 0x000060A8 0x1149 914
> 0/0/0
> galaxydoor.00-00 0x0000200F 0x645F 912
> 0/0/0
> LTRKAKHQR01-c1.00-00* 0x00000991 0xF41F 916
> 0/0/0
> TAMQFLTART1.00-00 0x00005926 0x83FD 913
> 0/0/0
> LTRKAKHQR01-c1811w#term mon
> LTRKAKHQR01-c1811w#sh clock
> 01:44:47.438 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
> 01:44:56.418 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
> 01:45:01.690 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
> 01:45:06.182 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
> 01:45:10.146 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
> 01:45:12.658 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
> 01:45:16.222 UTC Sun Feb 7 2010
> LTRKAKHQR01-c1811w#sh clock
>
> ______BAM! Lockup at this point______
>
> from the log output:
> Feb 6 20:45:27 192.168.3.210 103: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:20
> UTC: %SYS-3-CPUHOG: Task is running for (2000)msecs, more than (2000)
> msecs
> (0/0),process = Check heaps.
> Feb 6 20:45:27 192.168.3.210 104: LTRKAKHQR01-c1811w: -Traceback=
> 0x8007CCB0z
> 0x80B20C18z 0x80B22C8Cz 0x80B20EC8z 0x82050E18z 0x82052364z
> 0x82052770z
> 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
> Feb 6 20:45:27 192.168.3.210 105: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:22
> UTC: %SYS-3-CPUHOG: Task is running for (4000)msecs, more than (2000)
> msecs
> (0/0),process = Check heaps.
> Feb 6 20:45:27 192.168.3.210 106: LTRKAKHQR01-c1811w: -Traceback=
> 0x8007CCB8z
> 0x80B20C18z 0x80B22528z 0x80B20EC8z 0x820500E4z 0x82050E54z
> 0x82052364z
> 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz 0x80124418z
> Feb 6 20:45:27 192.168.3.210 107: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-3-BADMAGIC: Corrupt block at 86AC28DC (magic 813E0508), -
> Traceback=
> 0x82052388z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz
> 0x80124418z
> Feb 6 20:45:27 192.168.3.210 108: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> Feb 6 20:45:27 192.168.3.210 109: LTRKAKHQR01-c1811w:
> 86297A44,80BAE58C
> 86297A44,40000294 86CC08A0,80BAE570 86CC08A0,3000021E
> Feb 6 20:45:27 192.168.3.210 110: LTRKAKHQR01-c1811w:
> 86DC8180,8154FFC0
> 866536C8,8154FE24 866536C8,8154FE24 866536C8,8154FE88
> Feb 6 20:45:27 192.168.3.210 111: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> Feb 6 20:45:27 192.168.3.210 112: LTRKAKHQR01-c1811w:
> 8666F860,81569A98
> 866536C8,8154FE88 866536C8,8154FE24 866536C8,8154FE24
> Feb 6 20:45:27 192.168.3.210 113: LTRKAKHQR01-c1811w:
> 866536C8,8154EE70
> 866536C8,8154EE70 866536C8,8154EE70 866536C8,8154EE70
> Feb 6 20:45:27 192.168.3.210 114: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-BLKINFO: Corrupted magic value in in-use block blk
> 86AC28DC, words
> 6002, alloc 8012DAC4, InUse, dealloc FFFFFFFF, rfcnt 1, -Traceback=
> 0x82010150z 0x82052618z 0x82052770z 0x82055410z 0x820555CCz
> 0x8012086Cz
> 0x80124418z
> Feb 6 20:45:28 192.168.3.210 115: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-MEMDUMP: 0x86AC28DC: 0x813E0508 0x0 0x0 0x8364CCBC
> Feb 6 20:45:28 192.168.3.210 116: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-MEMDUMP: 0x86AC28EC: 0x8012DAC4 0x86AC57F0 0x86AB9C1C 0x80001772
> Feb 6 20:45:28 192.168.3.210 117: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-MEMDUMP: 0x86AC28FC: 0x1 0x86AC2980 0x15C 0x86D41800
> Feb 6 20:45:28 192.168.3.210 118: LTRKAKHQR01-c1811w: Feb 7 2010
> 01:45:24
> UTC: %SYS-6-STACKLOW: Stack for process Virtual Exec running low,
> 12/12000
> Feb 6 20:45:40 192.168.3.250 1038: TAMQFLTART1: Feb 7 2010
> 01:45:39 UTC:
> %LINEPROTO-5-UPDOWN: Line protocol on Interface Tunnel3, changed
> state to down
> -------
>
> Some times I'll get a crashinfo file, other times I will not.
>> From a previous crash info:
>
> -------------
> CMD: 'sh isis database' 21:23:27 UTC Sat Feb 6 2010
> validblock_diagnose, code = 2
>
> current memory block, bp = 0x8700E0B0,
> memorypool type is Processor
> data check, ptr = 0x8700E0E0
>
> next memory block, bp = 0x87010FC4,
> memorypool type is Processor
> data check, ptr = 0x87010FF4
>
> previous memory block, bp = 0x870053DC,
> memorypool type is Processor
> data check, ptr = 0x8700540C
> ========= Dump bp = 0x8700E0B0 ======================
>
> 8700DFB0: 0 8700EAB0 FFFFFFFF 0 0
> 0 0 0
> 8700DFD0: 0 0 6347E519 0 8207070C D02688F2
> 6347E519 85F7C994
> 8700DFF0: 85F7C994 811AEC4C 8700E010 811AB16C D0D0D0D 245EBB78
> D0D0D0D 867CD2F4
> 8700E010: 8700E040 813B2838 D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D 82070710
> 8700E030: 85F7C994 6347E519 875E3258 875E3214 8700E070 813B2B8C
> D0D0D0D 48822022
> 8700E050: 1 8700E2D8 0 0 8700E070 6347E519
> 8700E400 0
> 8700E070: 8700E0B0 813B4470 0 0 28822022
> 6347E519 0 0
> 8700E090: 0 0 6347E519 85F7C994 0 0
> 8700E400 0
>
> 8700E0B0: 8700E350 813E0508 0 0 8012DAC4 87010FC4
> 870053F0 80001772
> 8700E0D0: 1 0 8700E158 872550DC FFFFFFFF FFFFFFFF
> FFFFFFFF FFFFFFFF
> 8700E0F0: 0 0 FFFFFFFF FFFFFFFF FFFF FFFFFFFF
> FFFFFFFF FFFFFFFF
> 8700E110: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF 0
> 8700E130: 2F 86427028 0 85F7C994 0 6347E519
> 6347E519 245EBB78
> 8700E150: 8700E2F0 867CD2F4 8700E1D8 811ABDE0 FFFFFFFF 6347E519
> D02688F2 2F
> 8700E170: 0 0 0 C0 FFFFFFFF FFFFFFFF
> FFFFFFFF FFFFFFFF
> 8700E190: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF FFFFFFFF
> =========
> Feb 6 2010 21:24:09 UTC: %SYS-3-CPUHOG: Task is running for (2000)
> msecs, more
> than (2000)msecs (1/1),process = Check heaps.
> -Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22C8Cz 0x80B20EC8z
> 0x82050E18z
> 0x82052364z 0x82052770z 0x82055410z 0x820555CCz 0x8012086Cz
> 0x80124418z Dump
> bp->next = 0x87010FC4 ======================
>
> 87010EC4: 61780000 87010EF0 84228082 73796E74 A4CB80 7002FD0
> 87010F20 87010E70
> 87010EE4: 87010EF8 83EB0000 83EB0000 0 83EB0000 0
> A4CB80 0
> 87010F04: 0 867F2054 0 0 86493648 87010FB0
> 80B77310 FFFFFF
> 87010F24: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF FFFFFFFF
> 87010F44: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> 2A 1 FFFFFFFF
> 87010F64: FFFFFFFF 0 0 0 0
> 0 0 0
> 87010F84: 0 0 0 0 0
> 0 0 0
> 87010FA4: 0 0 0 87010FB8 8012086C 0 80124418
> FD0110DF
> 87010FC4: AB1234CD E40000 15F 873074D0 80B4FCA0 87015E18
> 8700E0C4 80002712
> 87010FE4: 1 8200EA4C 166 872550DC 0 0 87307494
> 0
> 87011004: 87307494 258 2C7 140018 2C1
> 0 0 0
> 87011024: 0 430000 83EC2BBC 41414120 536D616C 6C204368
> 756E0000 87011B6C
> 87011044: 87015E14 0 0 87011B70 87011B88 87011BA0
> 87011BB8 87011BD0
> 87011064: 87011BE8 87011C00 87011C18 87011C30 87011C48 87011C60
> 87011C78 87011C90
> 87011084: 87011CA8 87011CC0 87011CD8 87011CF0 87011D08 87011D20
> 87011D38 87011D50
> 870110A4: 87011D68 87011D80 87011D98 87011DB0 87011DC8 87011DE0
> 87011DF8 87011E10
> ========== Dump bp->previous = 0x870053F0 =====================
>
> 870052F0: 0 0 0 0 0
> 0 0 0
> 87005310: 0 FD0110DF AB1234CD FFFE0000 0 82FC74AC
> 81BDE144 87005390
> 87005330: 870052A8 80000024 1 0 1 850B5B2C
> 83824BA0 0
> 87005350: 0 0 1 0
> Feb 6 2010 21:24:11 UTC: %SYS-3-CPUHOG: Task is running for (4000)
> msecs, more
> than (2000)msecs (1/1),process = Check heaps.
> -Traceback= 0x8007CCB8z 0x80B20C18z 0x80B22528z 0x80B20EC8z
> 0x820500E4z
> 0x82050E54z 0x82052364z 0x82052770z 0x82055410z 0x820555CCz
> 0x8012086Cz
> 0x80124418z 0 0 0 90000
> 87005370: 1 870051FC 0 0 0
> 0 0 FD0110DF
> 87005390: AB1234CD FFFE0000 0 82FC74AC 81BD9178 870053DC
> 8700532C 8000000E
> 870053B0: 1 0 1 850B5B2C 1
> 0 0 0
> 870053D0: 0 0 FD0110DF AB1234CD 750000 75
> 83646E94 82C4EED4
> 870053F0: 8700E0B0 870053A4 4652 0 82C89068 7C
> 850B1410 DEADBEEF
>
> 87005410: 82C89068 0 D0D0D0D 83EC321C 83EC3218 D0D0D0D
> D0D0D0D D0D0D0D
> 87005430: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D D0D0D0D
> 87005450: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D D0D0D0D
> 87005470: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D D0D0D0D
> 87005490: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D D0D0D0D
> 870054B0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D D0D0D0D
> 870054D0: D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D D0D0D0D
> D0D0D0D D0D0D0D
> ============================================
>
> Feb 6 2010 21:24:12 UTC: %SYS-3-BADMAGIC: Corrupt block at 8700E0B0
> (magic
> 8700E350), -Traceback= 0x82052388z 0x82052770z 0x82055410z
> 0x820555CCz
> 0x8012086Cz 0x80124418z
> Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> 873068BC,80BAE58C 873068BC,40000294 859EC9B4,80BAE570
> 859EC9B4,3000021E
> 86EAC770,8154FFC0 86EA8998,8154FE24 86EA8998,8154FE24
> 859EC9B4,81540E3C
> Feb 6 2010 21:24:12 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> 859EC9B4,8153B354 859EC9B4,3000021E 86EA8998,8154FE88
> 86EAB098,81569A98
> 86EA8998,8154FE88 86EA8998,8154FE24 86EA8998,8154FE24
> 86EA8998,8154EE70
> Feb 6 2010 21:24:12 UTC: %SYS-6-BLKINFO: Corrupted magic value in
> in-use
> block blk 8700E0B0, words 6002, alloc 8012DAC4, InUse, dealloc
> FFFFFFFF, rfcnt
> 1, -Traceback= 0x82010150z 0x82052618z 0x82052770z 0x82055410z
> 0x820555CCz
> 0x8012086Cz 0x80124418z
> Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0B0: 0x8700E350
> 0x813E0508
> 0x0 0x0
> Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0C0: 0x8012DAC4
> 0x87010FC4
> 0x870053F0 0x80001772
> Feb 6 2010 21:24:12 UTC: %SYS-6-MEMDUMP: 0x8700E0D0: 0x1 0x0
> 0x8700E158
> 0x872550DC
>
> %Software-forced reload
>
>
> 21:24:12 UTC Sat Feb 6 2010: Unexpected exception to CPU: vector
> 1500, PC =
> 0x8011E220, LR = 0x8011E1E4
>
> -Traceback= 0x8011E220z 0x8011E1E4z 0x82052770z 0x82055410z
> 0x820555CCz
> 0x8012086Cz 0x80124418z
>
> CPU Register Context:
> MSR = 0x02029220 CR = 0x28000042 CTR = 0x81F26400 XER = 0x00000000
> R0 = 0x8011E1E4 R1 = 0x8511CBA8 R2 = 0xFFE97C10 R3 =
> 0x83FA9978
> R4 = 0x82F869BC R5 = 0x00000000 R6 = 0x83970000 R7 =
> 0x82F60000
> R8 = 0x02029220 R9 = 0x83AD0000 R10 = 0x00000000 R11 = 0x00000000
> R12 = 0x24000088 R13 = 0xFFE994A8 R14 = 0x820554DC R15 = 0x00000000
> R16 = 0x00000000 R17 = 0x00000000 R18 = 0x00000000 R19 = 0x00000000
> R20 = 0x00000000 R21 = 0x00000000 R22 = 0x83D60000 R23 =
> 0x83D60000
> R24 = 0xAB1234AB R25 = 0xAB1234CD R26 = 0x83D60000 R27 =
> 0x85FBD91C
> R28 = 0x00000000 R29 = 0x83647534 R30 = 0x83980000 R31 = 0x00000000
> ------
>
> I've tried this on both 1811w's on my network and had the exact same
> problems.
> Any one else seen this or know if it's a known bug? I've searched
> the cisco
> site and cannot find a reference to this issue.
>
> --
> Bryan Fields
>
> 727-409-1194 - Voice
> 727-214-2508 - Fax
> http://bryanfields.net
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list