[c-nsp] Cisco 6509 reboots on its own... again...

Youssef Bengelloun-Zahr youssef at 720.fr
Tue Jul 6 08:56:07 EDT 2010


Hello,

Small update on this one, got the crashfile info found this :


Cache error detected!
  CPO_ECC     (reg 26/0): 0x000000BE
  CPO_CACHERI (reg 27/0): 0xA0000000
  CP0_CAUSE   (reg 13/0): 0x00000400

Real cache error detected.  System will be halted.

Error: Primary data cache, fields: data,
Actual physical addr 0x00000000,
virtual address is imprecise.

 Imprecise Data Parity Error

 Imprecise Data Parity Error

 06:23:38 UTC Mon Jul 5 2010: Interrupt exception, CPU signal 20, PC =
0x40A94B8C



--------------------------------------------------------------------
   Possible software fault. Upon reccurence, please collect
   crashinfo, "show tech" and contact Cisco Technical Support.
--------------------------------------------------------------------


-Traceback= 425F30C8
$0 : 00000000, AT : 43B40000, v0 : 00000000, v1 : 5503EB98
a0 : 58CDD7E0, a1 : 502B6C64, a2 : C133A166, a3 : 00000000
t0 : 502B6C50, t1 : 502B6C24, t2 : 34008100, t3 : FFFF00FF
t4 : 425EB928, t5 : 00000000, t6 : 00000000, t7 : 47969878
s0 : FFFE0000, s1 : C1320000, s2 : 511DD9E0, s3 : C133A166
s4 : FFFFFF00, s5 : 00000008, s6 : 00000011, s7 : 511D999C
t8 : 502B6C3C, t9 : 00000000, k0 : 5536CD00, k1 : 411410B0
gp : 43B42D30, sp : 502B6BB0, s8 : 00000000, ra : 40A94B64
EPC  : 425F30C8, ErrorEPC : 40A94B8C, SREG     : 3400FF05
MDLO : 00010C20, MDHI     : 00000000, BadVaddr : 00000000
DATA_START : 0x43788D30
Cause 00000000 (Code 0x0): Interrupt exception


Any ideas ?

Thanks.

Y.



2010/7/6 krunal shah <krun.shah at gmail.com>

> There must be two crashinfo files for SP and RP and show tech-support. You
> need to collect it when you contact tech support.
>
> TAC usually has decoders from their developer to decode hex values in
> traceback.
>
>
> -Traceback= 41183348 41180F04 40DADF40 40FFA1CC 40FFA4D8 40752F58 40752F44
>
>
> Krunal
>
>
> On Mon, Jul 5, 2010 at 5:36 AM, Youssef Bengelloun-Zahr <youssef at 720.fr>wrote:
>
>> Hello,
>>
>> I have a c6509 with redundant SUP720-3BXL
>> (s72033-advipservicesk9_wan-mz.122-33.SXH2a.bin) that's rebooted on its
>> own
>> this morning. FYI, the same router reboot 3 weeks ago unexpectedly !
>>
>> Here is a trunkated output of the crashfile info :
>>
>> Jun 11 06:48:29.377: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO
>> mode
>> Jun 11 06:48:29.377: %SYS-SP-3-LOGGER_FLUSHING: System pausing to ensure
>> console debugging output.
>> Jun 11 06:48:29.377: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO
>> mode
>> Jun 11 06:48:29.569: %SYS-SP-3-LOGGER_FLUSHED: System was paused for
>> 00:00:00 to ensure console debugging output.
>> Jun 11 06:48:41.952: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 11 06:49:39.434: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off
>> for
>> the fabric in slot 5.
>> Jun 11 06:49:39.530: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric
>> Module in slot 5 became standby
>> Jun 11 06:49:42.850: %DIAG-SP-6-RUN_COMPLETE: Module 5: Running Complete
>> Diagnostics...
>> Jun 11 06:49:44.819: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online
>> Diagnostics
>> Jun 11 06:49:48.673: %OIR-SP-6-INSCARD: Card inserted in slot 5,
>> interfaces
>> are now online
>> Jun 11 09:53:37.178: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 11 13:02:59.715: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 11 13:04:16.254: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 14 09:00:28.800: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 14 09:05:08.864: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 17 08:35:59.058: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 17 08:39:58.941: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> CMD: 'sh mls cef summary ' 11:31:24 UTC Thu Jun 17 2010
>> CMD: 'exit' 11:31:25 UTC Thu Jun 17 2010
>> CMD: 'sh mls cef statistics ' 11:32:01 UTC Thu Jun 17 2010
>> CMD: 'sh mls cef maximum-routes ' 11:32:21 UTC Thu Jun 17 2010
>> CMD: 'sh mls cef rpf ' 11:33:07 UTC Thu Jun 17 2010
>> CMD: 'show mls acl inconsistency' 12:18:44 UTC Thu Jun 17 2010
>> Jun 21 08:14:58.161: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 22 08:15:53.784: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 22 11:56:07.044: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 22 11:58:40.637: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 23 11:01:20.484: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> Jun 23 12:31:21.556: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>> CMD: 'sh mls cef ' 21:30:10 UTC Sun Jun 27 2010
>> CMD: 'sh mls cef tcam hit ' 21:31:52 UTC Sun Jun 27 2010
>> Jun 29 11:51:04.876: %PFINIT-SP-5-CONFIG_SYNC: Sync'ing the startup
>> configuration to the standby Router.
>>
>> %Software-forced reload
>>
>>
>>  06:23:49 UTC Mon Jul 5 2010: Breakpoint exception, CPU signal 23, PC =
>> 0x41183348
>>
>>
>>
>> --------------------------------------------------------------------
>>   Possible software fault. Upon reccurence, please collect
>>   crashinfo, "show tech" and contact Cisco Technical Support.
>> --------------------------------------------------------------------
>>
>>
>> -Traceback= 41183348 41180F04 40DADF40 40FFA1CC 40FFA4D8 40752F58 40752F44
>> $0 : 00000000, AT : 1E020000, v0 : 43720000, v1 : 00000043
>> a0 : 447135B0, a1 : 00000043, a2 : 00000009, a3 : 00000000
>> t0 : 44C7494C, t1 : 44C74948, t2 : 44C74944, t3 : 44C74940
>> t4 : 44C7493C, t5 : 44C74938, t6 : 44C74934, t7 : 44C74930
>> s0 : 00000000, s1 : 41DF0000, s2 : 08FA84B0, s3 : 44C74AC0
>> s4 : 44C74AB8, s5 : 00000000, s6 : 00000000, s7 : 00000000
>> t8 : 44C7499C, t9 : 00000000, k0 : 470E1200, k1 : 40798CE0
>> gp : 41E591E0, sp : 44C74A20, s8 : 00000000, ra : 41180F04
>> EPC  : 41183348, ErrorEPC : 40947F88, SREG     : 3400FF03
>> MDLO : 333333E8, MDHI     : 000002D3, BadVaddr : 00000000
>> DATA_START : 0x41C420A0
>> Cause 00000024 (Code 0x9): Breakpoint exception
>>
>>
>> ========= Start of Crashinfo Collection (06:23:49 UTC Mon Jul 5 2010)
>> ==========
>> For image:
>> Cisco IOS Software, s72033_sp Software (s72033_sp-ADVIPSERVICESK9_WAN-M),
>> Version 12.2(33)SXH2a, RELEASE SOFTWARE (fc2)
>> Technical Support: http://www.cisco.com/techsupport
>> Copyright (c) 1986-2008 by Cisco Systems, Inc.
>> Compiled Fri 25-Apr-08 08:20 by prod_rel_team
>>
>>
>> ========= Show Alignment
>> =======================================================
>>
>>
>> No alignment data has been recorded.
>>
>> No spurious memory references have been recorded.
>>
>>
>> ========= Additional Subsystem Crashinfo
>> =======================================
>>
>> --------- show redundancy --------
>>
>> Switchovers this system has experienced          : 1
>> Last switchover reason                           : Active crashed.
>> Uptime since this supervisor switched to active  : 3 weeks, 2 days, 23
>> hours, 39 minutes
>> Total system uptime from reload                  : 13 weeks, 2 days, 19
>> hours, 5 minutes
>>
>> Standby is ready to take over
>>
>>
>> ========= Data Inconsistency Errors =========
>>
>> No data inconsistency errors have been recorded.
>>
>>
>> --------- show eobc --------
>>
>> Interface information:
>>    Interface EOBC0/0 (idb = 0x44A888B8)
>>    Hardware is Mistral EOBC (revision 5)
>>    Address is 0000.0600.0000 (bia 0000.0600.0000)
>>    Encap size         = 14         hardware status  = 0x210840
>>    IDB type           = 18         IDB state        = 4
>>    Encap type         = 0x1        Span encap size  = 0
>>    Error threshold    = 5000       Error count      = 0
>>
>> Counters:
>>    rxring             = 0x921DD00  rx ring entries       = 512
>>    rx_head            = 139        rx_tail               = 0
>>    inputs             = 150953935  rx_cumbytes           = 14190294763
>>    hw inputs          = 0          hw rx_cumbytes        = 0
>>    rx rate (bits/sec) = 41000      rx rate (packets/sec) = 53
>>    rx_buf_unavail     = 0          *rx input drops        = 4397*
>>    input broadcast    = 150        input resource        = 6815119
>>    input error        = 0          input giants          = 0
>>    *input crc          = 4397*       rx illegal length     = 0
>>    rxr eobc shadow    = 0x50C438F0 txr eobc shadow       = 0x44B94BCC
>>
>>    txring             = 0x921FD40  tx ring entries       = 0x200
>>    tx_head            = 297        tx_tail               = 297
>>    outputs            = 156727081  tx_cumbytes           = 26897233358
>>    hw outputs         = 0          hw tx_cumbytes        = 0
>>    tx rate (bits/sec) = 84000      tx rate (packets/sec) = 56
>>    tx_retry_error     = 2          tx_retry_count        = 276218
>>    tx_process_stopped = 0          tx total drops        = 0
>>
>> Mistral Registers
>>    soft_reset_cfg     = 0x000000   dma_buffer_size_reg   = 0x000000
>>    int_mask_hi        = 0x000076   int_mask_lo           = 0x7001AD8
>>    rxdscp_cnt         = 425        txdscp_cnt            = 0
>>    rxwork_dscp        = 0xEB20     txwork_dscp           = 0x688
>>    mistral_eobc_ds    = 0x44A897C4 mistral_dma_register  = 0x30000000
>>    mistral_glbl_reg   = 0x10020000
>>
>> Misc. Global Registers:
>>    global_cfg         = 0x20       mis_init_sts          = 0xF
>>    dimm_parm_cfg_hi   = 0x000003F6 dimm_parm_cfg_lo      = 0x42040F5A
>>    tm_init_size_cfg   = 0x8000
>>
>>
>> Here is the output of a show version :
>>
>> Cisco IOS Software, s72033_rp Software (s72033_rp-ADVIPSERVICESK9_WAN-M),
>> Version 12.2(33)SXH2a, RELEASE SOFTWARE (fc2)
>> Technical Support: http://www.cisco.com/techsupport
>> Copyright (c) 1986-2008 by Cisco Systems, Inc.
>> Compiled Fri 25-Apr-08 08:07 by prod_rel_team
>>
>> ROM: System Bootstrap, Version 12.2(17r)SX5, RELEASE SOFTWARE (fc1)
>>
>>  BB1.IX1 uptime is 13 weeks, 2 days, 22 hours, 1 minute
>> Uptime for this control processor is 3 weeks, 3 days, 2 hours, 26 minutes
>> Time since BB1.IX1 switched to active is 2 hours, 49 minutes
>> *System returned to ROM by Stateful Switchover at 07:42:25 UTC Wed May 20
>> 2009 (SP by reload)
>> *System restarted at 06:48:20 UTC Fri Jun 11 2010
>> System image file is
>> "bootdisk:s72033-advipservicesk9_wan-mz.122-33.SXH2a.bin"
>>
>>
>> This product contains cryptographic features and is subject to United
>> States and local country laws governing import, export, transfer and
>> use. Delivery of Cisco cryptographic products does not imply
>> third-party authority to import, export, distribute or use encryption.
>> Importers, exporters, distributors and users are responsible for
>> compliance with U.S. and local country laws. By using this product you
>> agree to comply with applicable laws and regulations. If you are unable
>> to comply with U.S. and local laws, return this product immediately.
>>
>> A summary of U.S. laws governing Cisco cryptographic products may be found
>> at:
>> http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
>>
>> If you require further assistance please contact us by sending email to
>> export at cisco.com.
>>
>> cisco WS-C6509 (R7000) processor (revision 2.0) with 983008K/65536K bytes
>> of
>> memory.
>> Processor board ID SCA043001KB
>> SR71000 CPU at 600Mhz, Implementation 0x504, Rev 1.2, 512KB L2 Cache
>> Last reset from s/w reset
>> 29 Virtual Ethernet interfaces
>> 96 FastEthernet interfaces
>> 124 Gigabit Ethernet interfaces
>> 1917K bytes of non-volatile configuration memory.
>> 8192K bytes of packet buffer memory.
>>
>> 65536K bytes of Flash internal SIMM (Sector size 512K).
>> Configuration register is 0x2102
>>
>>
>> I have noticed some EOBC input drops due to CRC. Would this be due to a
>> chassis default ? It's been running fine for more than two years now.
>>
>> I am running the IOS bug toolkit looking for a possible match with my
>> case.
>>
>> Thanks.
>>
>> Cheers.
>>
>> Y.
>>
>> --
>> Youssef BENGELLOUN-ZAHR ………………………………………………
>> Ingénieur Réseaux et Télécoms
>>
>>
>> Technopole de l'Aube  en Champagne - BP 601 - 10901 TROYES  Cedex 9
>> Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE
>> Tel                 +33 (0) 825 000 720
>> Tel. direct      +33 (0) 1 77 35 59 14
>> Tel. portable  +33 (0) 6 22 42 63 80
>> Email            ybz at 720.fr
>> ……………………………………………………………………………….....www.720.fr
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>>
>
>


-- 
Youssef BENGELLOUN-ZAHR ………………………………………………
Ingénieur Réseaux et Télécoms


Technopole de l'Aube  en Champagne - BP 601 - 10901 TROYES  Cedex 9
Agence Paris : 6, rue Charles Floquet - 92120 MONTROUGE
Tel                 +33 (0) 825 000 720
Tel. direct      +33 (0) 1 77 35 59 14
Tel. portable  +33 (0) 6 22 42 63 80
Email            ybz at 720.fr
……………………………………………………………………………….....www.720.fr


More information about the cisco-nsp mailing list