[c-nsp] Weird IOS problem
Sascha E. Pollok
nsp-list at pollok.net
Wed Jan 5 03:16:39 EST 2005
We saw this on a VIP2-50 with a single
PA-POS-PC3 on a 7507. Changing the slot
of the VIP did not help. Replacing the
VIP did. It is running for approx. 9
months now without ongoing failures.
Sascha
> I had a similar problem with (12.2(18)S7)on a 7507 rsp 4 and a vip4-50
> With a combination STM-1 FE Pas. I changed the vip and Pas with no change.
> Replacing the STM-1 with a E3 resolved everything and My router is now happy.
> The other 3 VIPs with different PA combinations were ok.
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Hank Nussbacher
> Sent: mercoledì 5 gennaio 2005 7.16
> To: cisco-nsp at puck.nether.net
> Subject: [c-nsp] Weird IOS problem
>
> This is a swing in the dark that perhaps someone else has seen this before. Our 7513 (w/ RSP16) running 12.2(18)S7 seems to be having a bad day. One particular VIP4-50 (slot1) started misbehaving and it causes all other VIPs to lose their CEF and for that particular VIP to take itself off-line!
>
> Log output:
> Jan 5 07:50:57: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FF30, slot 1, cmd code 2 -Traceback= 404F8334 404F8BF8 404EFF44 404ECD40 404CE974 4045C4A4 403D6B94 Jan 5 07:50:58: %DBUS-3-DBUSINTERR: Slot 1, Internal Error Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61 0x00000008) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000008) failed (0x8010) Jan 5 07:50:58: %CBUS-3-ADDRFILTR: Interface FastEthernet1/0/1, address filter write command failed, code 0x8010 -Traceback= 404FE1F8 404FEA40 404FECD4 40403F08 40651774 409C5234 409C588C
> 40A09674 409E5960 40420C1C 404DDDFC 404DDF80 404E4990 404CE86C Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x0000FFFF) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.22.1 on
> FastEthernet1/0/1 from 2WAY to DOWN, Neighbor Down: Interface down or detached Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.99.52 on
> FastEthernet1/0/1 from FULL to DOWN, Neighbor Down: Interface down or detached Jan 5 07:51:12: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 13012, status 0x0 -Traceback= 404DE264 404E5E64 404F4620 404F7E1C 404E49B4 404CE86C Jan 5 07:51:25: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 13036, status 0x0 -Traceback= 404DE264 404E5E64 404F7EA0 404E49B4 404CE86C Jan 5 07:51:38: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13036, status 0x0 -Traceback= 404DE264 404E5A78 404F7EB8 404E49B4 404CE86C Jan 5 07:51:51: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13028, status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404F7EE4
> 404E49B4 404CE86C
> Jan 5 07:52:02: %MDS-2-RP: MDFS is disabled on some line card(s). Use "show ip mds stats linecard" to view status and "clear ip mds linecard" to reset.
> Jan 5 07:54:28: %DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD, elapsed 20024, status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404F7EE4 404E49B4 404CE86C Jan 5 07:54:28: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40 cmd/data 0xB6 pos 4818686 Jan 5 07:54:28: %UCODE-3-LDFAIL: Unable to download ucode from system image in slot 1, trying rom ucode Jan 5 07:54:42: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 13064, status 0x0 -Traceback= 404DE264 404E1448 404E7BDC Jan 5 07:54:55: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13064, status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404E1450 404E7BDC Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal e!
> rror, slot/cpu 10/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: No window message, LC to RP IPC is non-operational .Jan 5 07:57:33: %DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD, elapsed 20012, status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404E1450 404E7BDC .Jan 5 07:57:33: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40 cmd/data 0xB6 pos 4818686 .Jan 5 07:57:33: %UCODE-3-LDFAIL: Unable to download ucode from system image in slot 1, trying rom ucode .Jan 5 07:57:33: %RSP-3-NOSTART: No microcode for VIP4-50 RM5271 card, s!
> lot 1 .Jan 5 07:57:34: %MDS-2-LC_FAILED_IPC_ACK: RP failed in getting
> Ack for IPC message of size 80 to LC in slot 1 with sequence 40250, error = retry queue flush .Jan 5 07:57:55: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 132.74.1.154 .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: keepalive failure TAU-gp1#sho cef line
> Slot/CPU MsgSent XDRSent Window LowQ MedQ HighQ Flags
> 2 5669 160596 LC wait 0 0 0 disabled
> 3 5669 160598 LC wait 0 0 0 disabled
> 4 5738 161622 LC wait 0 0 0 disabled
> 8 5742 161621 LC wait 0 0 0 disabled
> 9 5668 160577 LC wait 0 0 0 disabled
> 10 5668 160574 LC wait 0 0 0 disabled
> 11 5668 160577 LC wait 0 0 0 disabled
>
> VRF Default, version 311259, 153409 routes
> Slot/CPU Version CEF-XDR I/Fs State Flags
> 2 310726 157274 9 Active table-disabled
> 3 310726 157274 8 Active table-disabled
> 4 310726 157484 6 Active table-disabled
> 8 310726 157484 6 Active table-disabled
> 9 310726 157274 8 Active table-disabled
> 10 310726 157274 18 Active table-disabled
> 11 310726 157274 6 Active table-disabled
> TAU-gp1#show ip mds stats linecard
>
> Slot Status IPC(seq/max/window) Q(high/route) Reloads
> 1 disabled 40455/44286/3831 0/0 1
> 2 active 258 /4354 /4096 0/0 2
> 3 active 267 /4363 /4096 0/0 2
> 4 active 284 /4380 /4096 0/0 2
> 8 active 239 /4335 /4096 0/0 2
> 9 active 275 /4371 /4096 0/0 2
> 10 active 231 /2279 /2048 0/0 2
> 11 active 249 /4345 /4096 0/0 2
>
> Taking out VIP1 from the chassis and rebooting solves the problem. We have already replaced the VIP in slot1 yesterday when it first happened and life went on for about 12 hours before it happened again.
>
> Anyone seen anything like this? Cisco output intepreter was of no help since it indicated things that we don't run at all.
>
> Thanks,
> Hank
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list