[c-nsp] Help identifying bug
Dennis Peng
dpeng at cisco.com
Thu Oct 14 12:43:46 EDT 2004
It looks like a hardware problem to me since we crashed while trying
to fetch an instruction. If there is a history of similar crashes on
this box, I'd try to replace the DRAM first, then the NPE. If you have
a support contract for the box, you should probably open up a case
with TAC.
Dennis
james edwards [hackerwacker at cybermesa.com] wrote:
> Can someone give me opinions as to which bug I have hit here ? I uploaded
> the crashinfo
> to the bug tool and came up with CSCdx00728 at 85% (EIGRP, Other-AS routes
> are not promoted if current route gets worse in metric) but paging through
> the crashinfo I get the feeling this is BGP related.
>
>
> Oct 5 03:06:58.693 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.0.76 Down Peer
> closed the session
> Oct 5 03:07:05.861 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.0.76 Up
> Oct 5 03:50:12.206 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.0.78 Down BGP
> Notification sent
> Oct 5 03:50:12.206 MDT: %BGP-3-NOTIFICATION: sent to neighbor 65.19.0.78
> 4/0 (hold time expired) 0 bytes
> Oct 5 03:50:29.062 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.16.3 Down BGP
> Notification sent
> Oct 5 03:50:29.062 MDT: %BGP-3-NOTIFICATION: sent to neighbor 65.19.16.3
> 4/0 (hold time expired) 0 bytes
> Oct 5 03:55:29.754 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.0.78 Up
> Oct 5 03:56:10.664 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.16.3 Up
> Oct 5 03:59:47.025 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.0.80 Down Peer
> closed the session
> Oct 5 04:00:00.764 MDT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 101: Neighbor
> 65.19.32.2 (FastEthernet0/0) is down: holding time expired
> Oct 5 04:01:35.545 MDT: %DUAL-5-NBRCHANGE: IP-EIGRP(0) 101: Neighbor
> 65.19.32.2 (FastEthernet0/0) is up: new adjacency
> Oct 5 04:01:43.037 MDT: %BGP-5-ADJCHANGE: neighbor 65.19.0.80 Up
>
> Unexpected exception, CPU signal 10, PC = 0x607539F0
>
>
> -Traceback= 607539F0 60902D50
> $0 : 00000000, AT : 61FB7AB0, v0 : 00000000, v1 : 62CF53C0
> a0 : 0000000A, a1 : 00000018, a2 : 62387DA0, a3 : 00000000
> t0 : 60755BB0, t1 : 34008001, t2 : 00000014, t3 : 00000038
> t4 : 00008000, t5 : FFFF00FF, t6 : 4C4666CF, t7 : 62901930
> s0 : 00000000, s1 : 00010000, s2 : 00000000, s3 : FFFF0000
> s4 : 00000000, s5 : A8A80000, s6 : 62882720, s7 : 00000001
> t8 : 629019A4, t9 : FFFFFFFF, k0 : 36BA129B, k1 : 61FB4448
> gp : 62136E60, sp : 62419BE0, s8 : 00080000, ra : 60902D50
> EPC : 607539F0, ErrorEPC : 04000000, SREG : 34008003
> MDLO : CED9242C, MDHI : 00000F97, BadVaddr : 607539F0
> Cause 0000C018 (Code 0x6): Instruction Bus Error exception
>
>
> Unexpected exception, CPU signal 10, PC = 0x607539F0
>
>
> -Traceback= 607539F0 60902D50
> $0 : 00000000, AT : 61FB7AB0, v0 : 00000000, v1 : 62CF53C0
> a0 : 0000000A, a1 : 00000018, a2 : 62387DA0, a3 : 00000000
> t0 : 60755BB0, t1 : 34008001, t2 : 00000014, t3 : 00000038
> t4 : 00008000, t5 : FFFF00FF, t6 : 4C4666CF, t7 : 62901930
> s0 : 00000000, s1 : 00010000, s2 : 00000000, s3 : FFFF0000
> s4 : 00000000, s5 : A8A80000, s6 : 62882720, s7 : 00000001
> t8 : 629019A4, t9 : FFFFFFFF, k0 : 36BA129B, k1 : 61FB4448
> gp : 62136E60, sp : 62419BE0, s8 : 00080000, ra : 60902D50
> EPC : 607539F0, ErrorEPC : 04000000, SREG : 34008003
> MDLO : CED9242C, MDHI : 00000F97, BadVaddr : 607539F0
> Cause 0000C018 (Code 0x6): Instruction Bus Error exception
>
>
> === Start of Crashinfo Collection (05:03:57 MDT Tue Oct 5 2004) ===
>
> For image:
> Cisco Internetwork Operating System Software
> IOS (tm) 7200 Software (C7200-P-M), Version 12.2(18)S5, RELEASE SOFTWARE
> (fc1)
> Technical Support: http://www.cisco.com/techsupport
> Copyright (c) 1986-2004 by cisco Systems, Inc.
> Compiled Sat 08-May-04 10:43 by nmasa
>
>
>
> ========= Show Alignment =============================
>
>
>
> No alignment data has been recorded.
>
> No spurious memory references have been recorded.
>
>
> ========= Malloc and Free Traces=====================
>
> MallocFree Trace: ixmallocfree=0x3A ptr=0x623755E0
> 62375410: 65C0DFC8 20000038 65C0DFC8 60CBE094 65C0DFC8 60CC601C 63570138
> 60AEA854
> 62375430: 63572EC8 60AEA854 635738F0 60AEA854 65350480 60AEA854 63576228
> 60AEA854
> 62375450: 63576E90 60AEA854 65C0DFC8 20000038 65C0DFC8 60CBE094 65C0DFC8
> 60CC601C
> 62375470: 63570138 60AEA854 63572EC8 60AEA854 635738F0 60AEA854 65350480
> 60AEA854
> 62375490: 63576228 60AEA854 63576E90 60AEA854 63570138 60AEA854 63572EC8
> 60AEA854
> 623754B0: 635738F0 60AEA854 65350480 60AEA854 63576228 60AEA854 63576E90
> 60AEA854
> 623754D0: 65C0DFC8 20000038 65C0DFC8 60CBE094 65C0DFC8 60CC601C 65C0DFC8
> 20000038
> 623754F0: 65C0DFC8 60CBE094 65C0DFC8 60CC601C 65C0DFC8 20000038 65C0DFC8
> 60CBE094
> 62375510: 65C0DFC8 60CC601C 648A1A80 200003D0 648A1A80 6084BF20 63570138
> 60AEA854
> 62375530: 63572EC8 60AEA854 635738F0 60AEA854 65350480 60AEA854 63576228
> 60AEA854
> 62375550: 63576E90 60AEA854 63570138 60AEA854 63572EC8 60AEA854 635738F0
> 60AEA854
> 62375570: 65350480 60AEA854 63576228 60AEA854 63576E90 60AEA854 63570138
> 60AEA854
> 62375590: 63572EC8 60AEA854 635738F0 60AEA854 65350480 60AEA854 63576228
> 60AEA854
> 623755B0: 63576E90 60AEA854 63570138 60AEA854 63572EC8 60AEA854 635738F0
> 60AEA854
> 623755D0: 65350480 60AEA854 63576228 60AEA854 63576E90 60AEA854 63576228
> 60AEA854
> 623755F0: 63576E90 60AEA854 65C0DFC8 20000038 65C0DFC8 60CBE094 65C0DFC8
> 60CC601C
> ========== Stack Trace========
>
> -Traceback= 607539F0 60902D50
>
> ========= Context ======================
>
> 7200 Software (C7200-P-M), Version 12.2(18)S5, RELEASE SOFTWARE (fc1)
> Technical Support: http://www.cisco.com/techsupport
> Compiled Sat 08-May-04 10:43 by nmasa
> Signal = 10, Code = 0x18, Uptime 6w0d
> $0 : 00000000, AT : 61FB7AB0, v0 : 00000000, v1 : 62CF53C0
> a0 : 0000000A, a1 : 00000018, a2 : 62387DA0, a3 : 00000000
> t0 : 60755BB0, t1 : 34008001, t2 : 00000014, t3 : 00000038
> t4 : 00008000, t5 : FFFF00FF, t6 : 4C4666CF, t7 : 62901930
> s0 : 00000000, s1 : 00010000, s2 : 00000000, s3 : FFFF0000
> s4 : 00000000, s5 : A8A80000, s6 : 62882720, s7 : 00000001
> t8 : 629019A4, t9 : FFFFFFFF, k0 : 36BA129B, k1 : 61FB4448
> gp : 62136E60, sp : 62419BE0, s8 : 00080000, ra : 60902D50
> EPC : 607539F0, SREG : 34008003, Cause : 0000C018
>
> ========= Stack Dump
> ===========================================================
>
> Stack Frame Pointer in Context is 0x62419BE0 , at interrupt level 7
>
> 624197E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419800: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419820: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419840: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419860: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419880: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 624198A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 624198C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 624198E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419900: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419920: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419940: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419960: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419980: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 624199A0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 624199C0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 624199E0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419A00: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419A20: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419A40: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419A60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419A80: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419AA0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419AC0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419AE0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419B00: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419B20: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419B40: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419B60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF 0 DAE84A70 FFFFFFFF
> FFFFFFFF
> 62419B80: 0 DAE84A68 0 10000 0 607C5A6C FFFFFFFF
> FFFFFFFF
> 62419BA0: 62882720 62901EE8 60754C64 60754BD4 FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419BC0: 60753778 38 607548E4 FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419BE0: FFFFFFFF FFFFFFFF 0 62130000 FFFFFFFF CBE49F32 FFFFFFFF
> CBE5FA67
> 62419C00: 0 0 0 0 0 6 0
> F4240
> 62419C20: FFFFFFFF FFFFFFFF 0 61FB7820 0 61FB781C 0
> 61FB7818
> 62419C40: 0 61FB7814 0 61FB7810 0 61FB780C 0
> 61FB7808
> 62419C60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419C80: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419CA0: 0 61FB7868 0 0 0 62387DA0 0
> 38
> 62419CC0: FFFFFFFF FFFFFFFF 0 61FB77D8 FFFFFFFF FFFFFFFF 0
> 606C54C4
> 62419CE0: 34008003 FFFFFFFF 0 61FB7 0 8 FFFFFFFF
> FFFFFFFF
> 62419D00: FFFFFFFF FFFFFFFF 0 606C3B14 FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419D20: 0 0 FD0110DF 10DFFD01 AB1234CD FFFFFFFE 0
> 618D0F7C
> 62419D40: 61517D20 62419D80 624179DC 80000010 1 621EDBD8 726F6365
> 6564696E
> 62419D60: 0 0 0 0 0 0 FD0110DF
> 10DFFD01
> 62419D80: AB1234CD FFFFFFFE 0 618CF8B0 607578F8 6241C0E8 62419D44
> 8000119C
> 62419DA0: 1 621EDBD8 72207072 65666978 FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419DC0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419DE0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419E00: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419E20: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419E40: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419E60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419E80: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419EA0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419EC0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419EE0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419F00: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419F20: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419F40: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419F60: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419F80: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419FA0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
> 62419FC0: FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF FFFFFFFF
> FFFFFFFF
>
>
> ========= Process Level Info
> ===================================================
>
> ---- Current Process Block (at 0x629020A0) ----
>
> 62902070: AB1234CD 7A 629020A0 618D5BD8 6075C690 629022D8 628FFD1C
> 8000011C
> 62902090: 1 621EDBD8 D0D0D0D D0D0D0D 628FFD38 80007C68 60AF1CF4
> 6340D6B0
> 629020B0: 6150B234 607452AC 65A0B830 62390000 0 1 634DD3B8
> 60AEFE68
> 629020D0: 608A4AF4 62902038 0 607452F0 0 0 0
> 0
> 629020F0: 10100 10000 0 0 7A 0 0
> 0
> 62902110: 7F48 25C0B68 9710 0 0 0 0
> 2058
> 62902130: 0 0 0 DAE84994 0 DAE84A6C 7C42DC8
> BAA8F
> 62902150: 810A7 61A2E55C 4 F4 1C37 92D B38
> 2328
> 62902170: 2328 62420A48 0 0 0 0 0
> 0
> 62902190: 0 629020A0 62387A20 0 0 0 621DE408
> 0
> 629021B0: 0 0 420E 0 0 0 629021A0
> 0
> 629021D0: 0 0 4290 629020A0 0 0 0
> 0
> 629021F0: 629021A0 629020A0 0 0 142D2 0 0
> 0
> 62902210: 629021A0 629020A0 0 0 242D6 0 0
> 0
> 62902230: 0 0 0 62AA4760 0 0 0
> 0
> 62902250: 0 0 0 0 0 62AA47C0 62902250
> 62AA4768
> 62902270: 0 6238ABD0 0 0 0 0 0
> FFFFFFF
> 62902290: FFFFFFF 0 0 0 0 0 0
> 0
> 629022B0: 0 6356E870 0 0 0 BEEFCAFE
>
> ---- Partial decode of process block ----
>
> Pid 122: Process "BGP Scanner" stack 0x628FFD38 savedsp 0x80007C68
> Flags: analyze crashblock on_old_queue
> Status 0x00000000 Orig_ra 0x00000000 Routine 0x00000000 Signal 0
> Caller_pc 0x00000000 Callee_pc 0x00000000 Dbg_events 0x00000000 State 0
> Totmalloc 32584 Totfree 39586664 Totgetbuf 0
> Totretbuf 0 Edisms 0x0 Eparm 0x0
> Elapsed 0x7C42DC8 Ncalls 0xBAA8F Ngiveups 0x810A7
> Priority_q 4 Ticks_5s 244 Cpu_5sec 7223 Cpu_1min 2349
> Cpu_5min 2872 Stacksize 0x2328 Lowstack 0x2328
> Ttyptr 0x62420A48 Mem_holding 0x9710 Thrash_count 0
> Wakeup_reasons 0x0FFFFFFF Default_wakeup_reasons 0x0FFFFFFF
> Direct_wakeup_major 0x00000000 Direct_wakeup_minor 0x00000000
>
> Regs s0-s8,ra at last suspend; a0,a1,sp from proc creation, PC unused:
> a0: 60AF1CF4 a1: 6340D6B0 s0: 6150B234 s1: 607452AC s2: 65A0B830
> s3: 62390000 s4: 00000000 s5: 00000001 s6: 634DD3B8 s7: 60AEFE68
> s8: 608A4AF4 sp: 62902038 PC: 00000000 ra: 607452F0
>
>
>
>
> albcolo-router2#sho ver
> Cisco Internetwork Operating System Software
> IOS (tm) 7200 Software (C7200-P-M), Version 12.2(18)S5, RELEASE SOFTWARE
> (fc1)
> Technical Support: http://www.cisco.com/techsupport
> Copyright (c) 1986-2004 by cisco Systems, Inc.
> Compiled Sat 08-May-04 10:43 by nmasa
> Image text-base: 0x60008FE0, data-base: 0x6151E000
>
> ROM: System Bootstrap, Version 12.2(4r)B2, RELEASE SOFTWARE (fc2)
> BOOTLDR: 7200 Software (C7200-P-M), Version 12.2(18)S5, RELEASE SOFTWARE
> (fc1)
>
> albcolo-router2 uptime is 1 week, 1 day, 9 hours, 55 minutes
> System returned to ROM by bus error at PC 0x607539F0, address 0x607539F0 at
> 05:0
> 3:57 MDT Tue Oct 5 2004
> System restarted at 05:05:00 MDT Tue Oct 5 2004
> System image file is "slot0:c7200-p-mz.122-18.S5.bin"
>
> cisco 7206VXR (NPE400) processor (revision A) with 491520K/32768K bytes of
> memor
> y.
> Processor board ID 18285653
> R7000 CPU at 350Mhz, Implementation 39, Rev 3.3, 256KB L2 Cache
> 6 slot VXR midplane, Version 2.0
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list