[c-nsp] Weird IOS problem
Brian Turnbow
b.turnbow at twt.it
Wed Jan 5 03:09:22 EST 2005
I had a similar problem with (12.2(18)S7)on a 7507 rsp 4 and a vip4-50
With a combination STM-1 FE Pas. I changed the vip and Pas with no change.
Replacing the STM-1 with a E3 resolved everything and My router is now happy.
The other 3 VIPs with different PA combinations were ok.
-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Hank Nussbacher
Sent: mercoledì 5 gennaio 2005 7.16
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] Weird IOS problem
This is a swing in the dark that perhaps someone else has seen this before. Our 7513 (w/ RSP16) running 12.2(18)S7 seems to be having a bad day. One particular VIP4-50 (slot1) started misbehaving and it causes all other VIPs to lose their CEF and for that particular VIP to take itself off-line!
Log output:
Jan 5 07:50:57: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FF30, slot 1, cmd code 2 -Traceback= 404F8334 404F8BF8 404EFF44 404ECD40 404CE974 4045C4A4 403D6B94 Jan 5 07:50:58: %DBUS-3-DBUSINTERR: Slot 1, Internal Error Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61 0x00000008) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000008) failed (0x8010) Jan 5 07:50:58: %CBUS-3-ADDRFILTR: Interface FastEthernet1/0/1, address filter write command failed, code 0x8010 -Traceback= 404FE1F8 404FEA40 404FECD4 40403F08 40651774 409C5234 409C588C
40A09674 409E5960 40420C1C 404DDDFC 404DDF80 404E4990 404CE86C Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x0000FFFF) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.22.1 on
FastEthernet1/0/1 from 2WAY to DOWN, Neighbor Down: Interface down or detached Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.99.52 on
FastEthernet1/0/1 from FULL to DOWN, Neighbor Down: Interface down or detached Jan 5 07:51:12: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 13012, status 0x0 -Traceback= 404DE264 404E5E64 404F4620 404F7E1C 404E49B4 404CE86C Jan 5 07:51:25: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 13036, status 0x0 -Traceback= 404DE264 404E5E64 404F7EA0 404E49B4 404CE86C Jan 5 07:51:38: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13036, status 0x0 -Traceback= 404DE264 404E5A78 404F7EB8 404E49B4 404CE86C Jan 5 07:51:51: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13028, status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404F7EE4
404E49B4 404CE86C
Jan 5 07:52:02: %MDS-2-RP: MDFS is disabled on some line card(s). Use "show ip mds stats linecard" to view status and "clear ip mds linecard" to reset.
Jan 5 07:54:28: %DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD, elapsed 20024, status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404F7EE4 404E49B4 404CE86C Jan 5 07:54:28: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40 cmd/data 0xB6 pos 4818686 Jan 5 07:54:28: %UCODE-3-LDFAIL: Unable to download ucode from system image in slot 1, trying rom ucode Jan 5 07:54:42: %DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 13064, status 0x0 -Traceback= 404DE264 404E1448 404E7BDC Jan 5 07:54:55: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13064, status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404E1450 404E7BDC Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: No window message, LC to RP IPC is non-operational Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: No window message, LC to RP IPC is non-operational .Jan 5 07:57:33: %DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD, elapsed 20012, status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404E1450 404E7BDC .Jan 5 07:57:33: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40 cmd/data 0xB6 pos 4818686 .Jan 5 07:57:33: %UCODE-3-LDFAIL: Unable to download ucode from system image in slot 1, trying rom ucode .Jan 5 07:57:33: %RSP-3-NOSTART: No microcode for VIP4-50 RM5271 card, slot 1 .Jan 5 07:57:34: %MDS-2-LC_FAILED_IPC_ACK: RP failed in getting Ack for IPC message of size 80 to LC in slot 1 with sequence 40250, error = retry queue flush .Jan 5 07:57:55: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 132.74.1.154 .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: keepalive failure .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: keepalive failure TAU-gp1#sho cef line
Slot/CPU MsgSent XDRSent Window LowQ MedQ HighQ Flags
2 5669 160596 LC wait 0 0 0 disabled
3 5669 160598 LC wait 0 0 0 disabled
4 5738 161622 LC wait 0 0 0 disabled
8 5742 161621 LC wait 0 0 0 disabled
9 5668 160577 LC wait 0 0 0 disabled
10 5668 160574 LC wait 0 0 0 disabled
11 5668 160577 LC wait 0 0 0 disabled
VRF Default, version 311259, 153409 routes
Slot/CPU Version CEF-XDR I/Fs State Flags
2 310726 157274 9 Active table-disabled
3 310726 157274 8 Active table-disabled
4 310726 157484 6 Active table-disabled
8 310726 157484 6 Active table-disabled
9 310726 157274 8 Active table-disabled
10 310726 157274 18 Active table-disabled
11 310726 157274 6 Active table-disabled
TAU-gp1#show ip mds stats linecard
Slot Status IPC(seq/max/window) Q(high/route) Reloads
1 disabled 40455/44286/3831 0/0 1
2 active 258 /4354 /4096 0/0 2
3 active 267 /4363 /4096 0/0 2
4 active 284 /4380 /4096 0/0 2
8 active 239 /4335 /4096 0/0 2
9 active 275 /4371 /4096 0/0 2
10 active 231 /2279 /2048 0/0 2
11 active 249 /4345 /4096 0/0 2
Taking out VIP1 from the chassis and rebooting solves the problem. We have already replaced the VIP in slot1 yesterday when it first happened and life went on for about 12 hours before it happened again.
Anyone seen anything like this? Cisco output intepreter was of no help since it indicated things that we don't run at all.
Thanks,
Hank
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list