[nsp] RSP nightmare

[Tima Maryin]

Hello!


What message exactly did you see before FIBDISABLE?
What about free memory at this moment?

We're currently fighting with this: CSCee04316 .



Nick Kraal wrote:

> Today our 7507 crashed big time. The following was the console output I
> managed to capture in between being bollocked by the Boss.
> 
> Anyone out there understand this? And how would we troubleshoot this? I
> understand that this is a %RSP 3 RESTART: interface [xxx] output
> stuck/frozen/not transmitting issue. Funny thing under this pdf file from
> Cisco under Troubleshooting, is to open a Cisco TAC case and capture the 'sh
> tech' outout. And so I did.
> 
> Thanks in advance.
> 
> -nick/
> 
> 
> Mar 23 12:03:07.923 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 0: IPC
> Failure: timeout
> Mar 23 12:03:08.523 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 1: IPC
> Failure: timeout
> Mar 23 12:03:09.127 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 4: IPC
> Failure: timeout
> Mar 23 12:03:18.819 MAL: %RSP-3-RESTART: interface FastEthernet1/1/0, not
> transmitting
> -Traceback= 403F5160 403FF660 40384884 40384870
> Mar 23 12:03:18.819 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000008) failed (0x8010)
> Mar 23 12:03:18.819 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000007) failed (0x8010)
> Mar 23 12:03:18.819 MAL: %CBUS-3-ADDRFILTR: Interface FastEthernet1/1/0,
> address filter write command failed, code 0x8010
> -Traceback= 40443914 40444144 404442EC 40444EF0 403F523C 403FF660 40384884
> 40384870
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x0000FFFF) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000004) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000007) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000000) failed (0x8010)
> Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000013) failed (0x8010)
> Mar 23 12:03:18.831 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet1/1/0
> failed to reset properly in cbus_pas_ethernet_reset(), code 0
> x8010
> Mar 23 12:03:18.835 MAL: %RSP-3-RESTART: interface FastEthernet1/1/1, not
> transmitting
> -Traceback= 403F5160 403FF660 40384884 40384870
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000008) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000007) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x0000FFFF) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000004) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000006) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000000) failed (0x8010)
> Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> 0x00000013) failed (0x8010)
> Mar 23 12:03:18.847 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet1/1/1
> failed to reset properly in cbus_pas_ethernet_reset(), code 0
> x8010
> Mar 23 12:03:18.847 MAL: %RSP-3-RESTART: interface FastEthernet4/1/0, not
> transmitting
> -Traceback= 403F5160 403FF660 40384884 40384870
> Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> 0x00000008) failed (0x8010)
> Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> 0x00000007) failed (0x8010)
> Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> 0x0000FFFF) failed (0x8010)
> Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> 0x00000100) failed (0x8010)
> Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> 0x00000004) failed (0x8010)
> Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> 0x00000006) failed (0x8010)
> Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> 0x00000000) failed (0x8010)
> Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> 0x00000013) failed (0x8010)
> Mar 23 12:03:18.859 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet4/1/0
> failed to reset properly in cbus_pas_ethernet_reset(), code 0
> x8010