[nsp] RSP nightmare

[Nick Kraal]

Today our 7507 crashed big time. The following was the console output I
managed to capture in between being bollocked by the Boss.

Anyone out there understand this? And how would we troubleshoot this? I
understand that this is a %RSP 3 RESTART: interface [xxx] output
stuck/frozen/not transmitting issue. Funny thing under this pdf file from
Cisco under Troubleshooting, is to open a Cisco TAC case and capture the 'sh
tech' outout. And so I did.

Thanks in advance.

-nick/


Mar 23 12:03:07.923 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 0: IPC
Failure: timeout
Mar 23 12:03:08.523 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 1: IPC
Failure: timeout
Mar 23 12:03:09.127 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 4: IPC
Failure: timeout
Mar 23 12:03:18.819 MAL: %RSP-3-RESTART: interface FastEthernet1/1/0, not
transmitting
-Traceback= 403F5160 403FF660 40384884 40384870
Mar 23 12:03:18.819 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000008) failed (0x8010)
Mar 23 12:03:18.819 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000007) failed (0x8010)
Mar 23 12:03:18.819 MAL: %CBUS-3-ADDRFILTR: Interface FastEthernet1/1/0,
address filter write command failed, code 0x8010
-Traceback= 40443914 40444144 404442EC 40444EF0 403F523C 403FF660 40384884
40384870
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x0000FFFF) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000004) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000007) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000000) failed (0x8010)
Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000013) failed (0x8010)
Mar 23 12:03:18.831 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet1/1/0
failed to reset properly in cbus_pas_ethernet_reset(), code 0
x8010
Mar 23 12:03:18.835 MAL: %RSP-3-RESTART: interface FastEthernet1/1/1, not
transmitting
-Traceback= 403F5160 403FF660 40384884 40384870
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000008) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000007) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x0000FFFF) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000004) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000006) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000000) failed (0x8010)
Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
0x00000013) failed (0x8010)
Mar 23 12:03:18.847 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet1/1/1
failed to reset properly in cbus_pas_ethernet_reset(), code 0
x8010
Mar 23 12:03:18.847 MAL: %RSP-3-RESTART: interface FastEthernet4/1/0, not
transmitting
-Traceback= 403F5160 403FF660 40384884 40384870
Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
0x00000008) failed (0x8010)
Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
0x00000007) failed (0x8010)
Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
0x0000FFFF) failed (0x8010)
Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
0x00000100) failed (0x8010)
Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
0x00000004) failed (0x8010)
Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
0x00000006) failed (0x8010)
Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
0x00000000) failed (0x8010)
Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
0x00000013) failed (0x8010)
Mar 23 12:03:18.859 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet4/1/0
failed to reset properly in cbus_pas_ethernet_reset(), code 0
x8010