[c-nsp] Weird IOS problem
Hank Nussbacher
hank at mail.iucc.ac.il
Wed Jan 5 01:16:15 EST 2005
This is a swing in the dark that perhaps someone else has seen this
before. Our 7513 (w/ RSP16) running 12.2(18)S7 seems to be having a bad
day. One particular VIP4-50 (slot1) started misbehaving and it causes all
other VIPs to lose their CEF and for that particular VIP to take itself
off-line!
Log output:
Jan 5 07:50:57: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FF30, slot
1, cmd code 2
-Traceback= 404F8334 404F8BF8 404EFF44 404ECD40 404CE974 4045C4A4 403D6B94
Jan 5 07:50:58: %DBUS-3-DBUSINTERR: Slot 1, Internal Error
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61 0x00000008)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000008)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-ADDRFILTR: Interface FastEthernet1/0/1, address
filter write command failed, code 0x8010
-Traceback= 404FE1F8 404FEA40 404FECD4 40403F08 40651774 409C5234 409C588C
40A09674 409E5960 40420C1C 404DDDFC 404DDF80 404E4990 404CE86C
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x0000FFFF)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
failed (0x8010)
Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
failed (0x8010)
Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.22.1 on
FastEthernet1/0/1 from 2WAY to DOWN, Neighbor Down: Interface down or
detached
Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.99.52 on
FastEthernet1/0/1 from FULL to DOWN, Neighbor Down: Interface down or
detached
Jan 5 07:51:12: %DBUS-3-SW_NOTRDY: DBUS software not ready after
HARD_RESET, elapsed 13012, status 0x0
-Traceback= 404DE264 404E5E64 404F4620 404F7E1C 404E49B4 404CE86C
Jan 5 07:51:25: %DBUS-3-SW_NOTRDY: DBUS software not ready after
HARD_RESET, elapsed 13036, status 0x0
-Traceback= 404DE264 404E5E64 404F7EA0 404E49B4 404CE86C
Jan 5 07:51:38: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET,
elapsed 13036, status 0x0
-Traceback= 404DE264 404E5A78 404F7EB8 404E49B4 404CE86C
Jan 5 07:51:51: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET,
elapsed 13028, status 0x0
-Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404F7EE4
404E49B4 404CE86C
Jan 5 07:52:02: %MDS-2-RP: MDFS is disabled on some line card(s). Use
"show ip mds stats linecard" to view status and "clear ip mds linecard" to
reset.
Jan 5 07:54:28: %DBUS-3-SW_NOTRDY: DBUS software not ready after
DOWNLOAD, elapsed 20024, status 0x40
-Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404F7EE4 404E49B4
404CE86C
Jan 5 07:54:28: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40
cmd/data 0xB6 pos 4818686
Jan 5 07:54:28: %UCODE-3-LDFAIL: Unable to download ucode from system
image in slot 1, trying rom ucode
Jan 5 07:54:42: %DBUS-3-SW_NOTRDY: DBUS software not ready after
HARD_RESET, elapsed 13064, status 0x0
-Traceback= 404DE264 404E1448 404E7BDC
Jan 5 07:54:55: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET,
elapsed 13064, status 0x0
-Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404E1450
404E7BDC
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: No window
message, LC to RP IPC is non-operational
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window
message, LC to RP IPC is non-operational
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: No window
message, LC to RP IPC is non-operational
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: No window
message, LC to RP IPC is non-operational
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: No window
message, LC to RP IPC is non-operational
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: No window
message, LC to RP IPC is non-operational
Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: No window
message, LC to RP IPC is non-operational
.Jan 5 07:57:33: %DBUS-3-SW_NOTRDY: DBUS software not ready after
DOWNLOAD, elapsed 20012, status 0x40
-Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404E1450 404E7BDC
.Jan 5 07:57:33: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40
cmd/data 0xB6 pos 4818686
.Jan 5 07:57:33: %UCODE-3-LDFAIL: Unable to download ucode from system
image in slot 1, trying rom ucode
.Jan 5 07:57:33: %RSP-3-NOSTART: No microcode for VIP4-50 RM5271 card,
slot 1
.Jan 5 07:57:34: %MDS-2-LC_FAILED_IPC_ACK: RP failed in getting Ack for
IPC message of size 80 to LC in slot 1 with sequence 40250, error = retry
queue flush
.Jan 5 07:57:55: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req
from host 132.74.1.154
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: keepalive
failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: keepalive
failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: keepalive
failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: keepalive
failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: keepalive
failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: keepalive
failure
.Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: keepalive
failure
TAU-gp1#sho cef line
Slot/CPU MsgSent XDRSent Window LowQ MedQ HighQ Flags
2 5669 160596 LC wait 0 0 0 disabled
3 5669 160598 LC wait 0 0 0 disabled
4 5738 161622 LC wait 0 0 0 disabled
8 5742 161621 LC wait 0 0 0 disabled
9 5668 160577 LC wait 0 0 0 disabled
10 5668 160574 LC wait 0 0 0 disabled
11 5668 160577 LC wait 0 0 0 disabled
VRF Default, version 311259, 153409 routes
Slot/CPU Version CEF-XDR I/Fs State Flags
2 310726 157274 9 Active table-disabled
3 310726 157274 8 Active table-disabled
4 310726 157484 6 Active table-disabled
8 310726 157484 6 Active table-disabled
9 310726 157274 8 Active table-disabled
10 310726 157274 18 Active table-disabled
11 310726 157274 6 Active table-disabled
TAU-gp1#show ip mds stats linecard
Slot Status IPC(seq/max/window) Q(high/route) Reloads
1 disabled 40455/44286/3831 0/0 1
2 active 258 /4354 /4096 0/0 2
3 active 267 /4363 /4096 0/0 2
4 active 284 /4380 /4096 0/0 2
8 active 239 /4335 /4096 0/0 2
9 active 275 /4371 /4096 0/0 2
10 active 231 /2279 /2048 0/0 2
11 active 249 /4345 /4096 0/0 2
Taking out VIP1 from the chassis and rebooting solves the problem. We
have already replaced the VIP in slot1 yesterday when it first happened
and life went on for about 12 hours before it happened again.
Anyone seen anything like this? Cisco output intepreter was of no help
since it indicated things that we don't run at all.
Thanks,
Hank
More information about the cisco-nsp
mailing list