[nsp] RSP nightmare

[Nick Kraal]

Tima,

Sorry for the late reply.

[1] This was all we managed to capture via console before we were locked out
and the router died and lit up flashing like a Christmas tree.
[2] Not sure how to check this.

-nick/

----- Original Message ----- 
From: "Tima Maryin" <tima at rfn.ru>
To: "Nick Kraal" <nick at arc.net.my>
Cc: <cisco-nsp at puck.nether.net>
Sent: Wednesday, March 24, 2004 10:53 PM
Subject: Re: [nsp] RSP nightmare


> Hello!
>
>
> What message exactly did you see before FIBDISABLE?
> What about free memory at this moment?
>
> We're currently fighting with this: CSCee04316 .
>
>
>
> Nick Kraal wrote:
>
> > Today our 7507 crashed big time. The following was the console output I
> > managed to capture in between being bollocked by the Boss.
> >
> > Anyone out there understand this? And how would we troubleshoot this? I
> > understand that this is a %RSP 3 RESTART: interface [xxx] output
> > stuck/frozen/not transmitting issue. Funny thing under this pdf file
from
> > Cisco under Troubleshooting, is to open a Cisco TAC case and capture the
'sh
> > tech' outout. And so I did.
> >
> > Thanks in advance.
> >
> > -nick/
> >
> >
> > Mar 23 12:03:07.923 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 0: IPC
> > Failure: timeout
> > Mar 23 12:03:08.523 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 1: IPC
> > Failure: timeout
> > Mar 23 12:03:09.127 MAL: %FIB-3-FIBDISABLE: Fatal error, slot 4: IPC
> > Failure: timeout
> > Mar 23 12:03:18.819 MAL: %RSP-3-RESTART: interface FastEthernet1/1/0,
not
> > transmitting
> > -Traceback= 403F5160 403FF660 40384884 40384870
> > Mar 23 12:03:18.819 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000008) failed (0x8010)
> > Mar 23 12:03:18.819 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000007) failed (0x8010)
> > Mar 23 12:03:18.819 MAL: %CBUS-3-ADDRFILTR: Interface FastEthernet1/1/0,
> > address filter write command failed, code 0x8010
> > -Traceback= 40443914 40444144 404442EC 40444EF0 403F523C 403FF660
40384884
> > 40384870
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x0000FFFF) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000004) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000007) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000000) failed (0x8010)
> > Mar 23 12:03:18.823 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000013) failed (0x8010)
> > Mar 23 12:03:18.831 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet1/1/0
> > failed to reset properly in cbus_pas_ethernet_reset(), code 0
> > x8010
> > Mar 23 12:03:18.835 MAL: %RSP-3-RESTART: interface FastEthernet1/1/1,
not
> > transmitting
> > -Traceback= 403F5160 403FF660 40384884 40384870
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000008) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000007) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x0000FFFF) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000004) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000006) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000000) failed (0x8010)
> > Mar 23 12:03:18.835 MAL: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61
> > 0x00000013) failed (0x8010)
> > Mar 23 12:03:18.847 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet1/1/1
> > failed to reset properly in cbus_pas_ethernet_reset(), code 0
> > x8010
> > Mar 23 12:03:18.847 MAL: %RSP-3-RESTART: interface FastEthernet4/1/0,
not
> > transmitting
> > -Traceback= 403F5160 403FF660 40384884 40384870
> > Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> > 0x00000008) failed (0x8010)
> > Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> > 0x00000007) failed (0x8010)
> > Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> > 0x0000FFFF) failed (0x8010)
> > Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.847 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (36
> > 0x00000100) failed (0x8010)
> > Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> > 0x00000004) failed (0x8010)
> > Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> > 0x00000006) failed (0x8010)
> > Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> > 0x00000000) failed (0x8010)
> > Mar 23 12:03:18.851 MAL: %CBUS-3-CCBCMDFAIL1: Controller 4, cmd (61
> > 0x00000013) failed (0x8010)
> > Mar 23 12:03:18.859 MAL: %CBUS-4-RSETFAIL: Interface FastEthernet4/1/0
> > failed to reset properly in cbus_pas_ethernet_reset(), code 0
> > x8010
>