[c-nsp] Cat6500 PM_SCP-SP-2-LCP_FW_ERR_INFORM Bus Asic #0 out, of sync error

[Christina Klam]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Chris,

We had something similar happen here.  After replacing multiple modules,
we ended up replacing the chassis. Hopefully, you will not need to do that.

To determine if the problem is on just one module or many, run the
following commands on each card.  If it is the chassis, you will see
non-zero counts for ds_cbic_dbus_pktcrc on multiple cards.
===========================

Switch#remote log sw
Trying Switch ...
Entering CONSOLE for Switch
Type "^C^C^C" to end this session

Switch-sp#test lcp 1
% sending start...
% type commands to send to LCP.  type end when done.

LCP1 console# cntr 1 1 6
LCP1 console# cntr 1 1 6

error_cntrs_t for port 0, chan 0 (per [ch]an, else per asic:
ds_cbic_dbus_pktcrc           0x0000000000000007
ds_cbic_dbus_pktlen            0x0000000000000000
ds_cbic_rbus_crc              0x0000000000000000
ds_cbic_more_sz[ch]            0x0000000000000000
ds_cbic_less_sz               0x0000000000000000
ds_cbic_0_psel_cnt             0x00000000d9fb846f
ds_cbif_dbus_crc              0x0000000000000000

LCP CPXBIT> exit

Regards,
Christina

> Message: 3
> Date: Tue, 25 May 2010 08:11:51 -0500
> From: Chris Wopat <me at falz.net>
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] Cat6500 PM_SCP-SP-2-LCP_FW_ERR_INFORM Bus Asic #0 out
> 	of	sync error
> Message-ID:
> 	<AANLkTinLZPL6xoTcIJaxmwK1ABbHSjifX9aec8mS2Xii at mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
> 
> Hello, trying to determine if this is is an issue with the line card
> (WS-X6148-GE-TX) or the chassis (6509). Out of the blue once every 6
> months or so we get:
> 
> May 25 07:09:55.108 CDT: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 8 is
> experiencing the following error: Bus Asic #0 out of sync error
> May 25 07:10:13.107 CDT: %PM_SCP-SP-2-LCP_FW_ERR_INFORM: Module 8 is
> experiencing the following error: Bus Asic #0 out of sync error
> ...
> 
> After ~5 minutes of the above errors the card shuts down:
> 
> May 25 07:14:59.717 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet8/4,
> changed state to down
> May 25 07:14:59.717 CDT: %LINK-3-UPDOWN: Interface GigabitEthernet8/8,
> changed state to down
> May 25 07:14:59.717 CDT: %LINK-3-UPDOWN: Interface
> GigabitEthernet8/10, changed state to down
> <snip>
> May 25 07:14:59.921 CDT: %HA_EM-6-LOG: Mandatory.go_intlpbk.tcl: GOLD
> EEM TCL policy for TestIntPortLoopback
> May 25 07:14:59.625 CDT: %CONST_DIAG-SP-6-HM_TEST_SP_INFO:
> TestIntPortLoopback[8]: last_busy_percent[23%], Tx_Rate[68730],
> Rx_Rate[10475]
> May 25 07:14:59.625 CDT: %CONST_DIAG-SP-2-HM_MOD_RESET: Resetting
> Module 8 for software recovery, Reason: Failed TestIntPortLoopback
> May 25 07:14:59.625 CDT: %OIR-SP-3-PWRCYCLE: Card in module 8, is
> being power-cycled 'off (Diagnostic Failure)'
> May 25 07:14:59.761 CDT: %C6KPWR-SP-4-DISABLED: power to module in
> slot 8 set off (Diagnostic Failure)
> 
> At this point we have to re-seat the card. Currently running
> 12.2(33)SXI2a. Trying to determine which (or both) to replace- module
> or chassis. Thoughts?
> 
> --Chris

- -- 
Christina Klam
Network Administrator
Institute for Advanced Study
Email:  cklam at ias.edu

Einstein Drive          Telephone: 609-734-8154
Princeton, NJ 08540     Fax:  609-951-4418
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.9 (MingW32)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMAC8JAAoJEAdQ1gVazugU5H8H/iYGVSpnWtGPolerbfhFsNEn
jbJxK3tLqqYja8NCf2H2J2Z38faiUUqdnN+nUqjnb6PzzUu0eWG2r2IhDhEc7YFa
eW64a7/0mCjd00cVMLgEIWdPIonbD144r7m9OkAGL4YvzqeghT2vEnoEkKKeDHqC
PWHuwl34kckJTH+ybRSDT0e6uupl63JlmhpaqA2JSrGASDGCDZ8XFry2X8BSNHlm
E2sElFNzECFxrD7pc0PluzTdC+3ERM48qsFf9xSjWQQy6RcmCKMKvybSdAQ/hABc
P12jRWPyOPc7jcdVzuXUxJj5BSeqaRqjVHr81r3hh38AD6n/Wy3grucN2NteoG4=
=OQ1g
-----END PGP SIGNATURE-----