[c-nsp] Weird IOS problem
Oleksandr Pantus
alx at vsmu.vinnica.ua
Wed Jan 5 02:34:56 EST 2005
Hello !
7505, RSP4, VIP2-50.
After upgrading from S6 to S7, router stoped switching packets in dCEF.
Moreover, disabling a dCEF (downgrade to CEF or even to process
switching) would not help. I have to remove "ip cef distributed" statement
from startup-config and reload the router to fix this situation.
Downgrade to S6 would not help, and there is no sign of any problem
in logging. I have already replaced VIP with PA's thinking that
it might me a hardware problem with it - and nothing has changed.
Still no switching in dCEF.
The only thing to try is a replacement of RSP, but I have not it
at spares yet.
Now the router is working with plain CEF switching waiting for RSP
replacement.
On Wed, Jan 05, 2005 at 08:16:15AM +0200, Hank Nussbacher wrote:
> This is a swing in the dark that perhaps someone else has seen this
> before. Our 7513 (w/ RSP16) running 12.2(18)S7 seems to be having a bad
> day. One particular VIP4-50 (slot1) started misbehaving and it causes all
> other VIPs to lose their CEF and for that particular VIP to take itself
> off-line!
>
> Log output:
> Jan 5 07:50:57: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FF30, slot
> 1, cmd code 2
> -Traceback= 404F8334 404F8BF8 404EFF44 404ECD40 404CE974 4045C4A4 403D6B94
> Jan 5 07:50:58: %DBUS-3-DBUSINTERR: Slot 1, Internal Error
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61 0x00000008)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000008)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-ADDRFILTR: Interface FastEthernet1/0/1, address
> filter write command failed, code 0x8010
> -Traceback= 404FE1F8 404FEA40 404FECD4 40403F08 40651774 409C5234 409C588C
> 40A09674 409E5960 40420C1C 404DDDFC 404DDF80 404E4990 404CE86C
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x0000FFFF)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00000100)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
> failed (0x8010)
> Jan 5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333)
> failed (0x8010)
> Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.22.1 on
> FastEthernet1/0/1 from 2WAY to DOWN, Neighbor Down: Interface down or
> detached
> Jan 5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.99.52 on
> FastEthernet1/0/1 from FULL to DOWN, Neighbor Down: Interface down or
> detached
> Jan 5 07:51:12: %DBUS-3-SW_NOTRDY: DBUS software not ready after
> HARD_RESET, elapsed 13012, status 0x0
> -Traceback= 404DE264 404E5E64 404F4620 404F7E1C 404E49B4 404CE86C
> Jan 5 07:51:25: %DBUS-3-SW_NOTRDY: DBUS software not ready after
> HARD_RESET, elapsed 13036, status 0x0
> -Traceback= 404DE264 404E5E64 404F7EA0 404E49B4 404CE86C
> Jan 5 07:51:38: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET,
> elapsed 13036, status 0x0
> -Traceback= 404DE264 404E5A78 404F7EB8 404E49B4 404CE86C
> Jan 5 07:51:51: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET,
> elapsed 13028, status 0x0
> -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404F7EE4
> 404E49B4 404CE86C
> Jan 5 07:52:02: %MDS-2-RP: MDFS is disabled on some line card(s). Use
> "show ip mds stats linecard" to view status and "clear ip mds linecard" to
> reset.
> Jan 5 07:54:28: %DBUS-3-SW_NOTRDY: DBUS software not ready after
> DOWNLOAD, elapsed 20024, status 0x40
> -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404F7EE4 404E49B4
> 404CE86C
> Jan 5 07:54:28: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40
> cmd/data 0xB6 pos 4818686
> Jan 5 07:54:28: %UCODE-3-LDFAIL: Unable to download ucode from system
> image in slot 1, trying rom ucode
> Jan 5 07:54:42: %DBUS-3-SW_NOTRDY: DBUS software not ready after
> HARD_RESET, elapsed 13064, status 0x0
> -Traceback= 404DE264 404E1448 404E7BDC
> Jan 5 07:54:55: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET,
> elapsed 13064, status 0x0
> -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 404DCD90 404E1450
> 404E7BDC
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: No window
> message, LC to RP IPC is non-operational
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window
> message, LC to RP IPC is non-operational
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: No window
> message, LC to RP IPC is non-operational
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: No window
> message, LC to RP IPC is non-operational
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: No window
> message, LC to RP IPC is non-operational
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: No window
> message, LC to RP IPC is non-operational
> Jan 5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: No window
> message, LC to RP IPC is non-operational
> .Jan 5 07:57:33: %DBUS-3-SW_NOTRDY: DBUS software not ready after
> DOWNLOAD, elapsed 20012, status 0x40
> -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 404E1450 404E7BDC
> .Jan 5 07:57:33: %DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40
> cmd/data 0xB6 pos 4818686
> .Jan 5 07:57:33: %UCODE-3-LDFAIL: Unable to download ucode from system
> image in slot 1, trying rom ucode
> .Jan 5 07:57:33: %RSP-3-NOSTART: No microcode for VIP4-50 RM5271 card,
> slot 1
> .Jan 5 07:57:34: %MDS-2-LC_FAILED_IPC_ACK: RP failed in getting Ack for
> IPC message of size 80 to LC in slot 1 with sequence 40250, error = retry
> queue flush
> .Jan 5 07:57:55: %SNMP-3-AUTHFAIL: Authentication failure for SNMP req
> from host 132.74.1.154
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: keepalive
> failure
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: keepalive
> failure
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: keepalive
> failure
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: keepalive
> failure
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: keepalive
> failure
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: keepalive
> failure
> .Jan 5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: keepalive
> failure
> TAU-gp1#sho cef line
> Slot/CPU MsgSent XDRSent Window LowQ MedQ HighQ Flags
> 2 5669 160596 LC wait 0 0 0 disabled
> 3 5669 160598 LC wait 0 0 0 disabled
> 4 5738 161622 LC wait 0 0 0 disabled
> 8 5742 161621 LC wait 0 0 0 disabled
> 9 5668 160577 LC wait 0 0 0 disabled
> 10 5668 160574 LC wait 0 0 0 disabled
> 11 5668 160577 LC wait 0 0 0 disabled
>
> VRF Default, version 311259, 153409 routes
> Slot/CPU Version CEF-XDR I/Fs State Flags
> 2 310726 157274 9 Active table-disabled
> 3 310726 157274 8 Active table-disabled
> 4 310726 157484 6 Active table-disabled
> 8 310726 157484 6 Active table-disabled
> 9 310726 157274 8 Active table-disabled
> 10 310726 157274 18 Active table-disabled
> 11 310726 157274 6 Active table-disabled
> TAU-gp1#show ip mds stats linecard
>
> Slot Status IPC(seq/max/window) Q(high/route) Reloads
> 1 disabled 40455/44286/3831 0/0 1
> 2 active 258 /4354 /4096 0/0 2
> 3 active 267 /4363 /4096 0/0 2
> 4 active 284 /4380 /4096 0/0 2
> 8 active 239 /4335 /4096 0/0 2
> 9 active 275 /4371 /4096 0/0 2
> 10 active 231 /2279 /2048 0/0 2
> 11 active 249 /4345 /4096 0/0 2
>
> Taking out VIP1 from the chassis and rebooting solves the problem. We
> have already replaced the VIP in slot1 yesterday when it first happened
> and life went on for about 12 hours before it happened again.
>
> Anyone seen anything like this? Cisco output intepreter was of no help
> since it indicated things that we don't run at all.
>
> Thanks,
> Hank
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
--
S/Y,
Alexander, MD, nic-hdl: AJP1-UANIC
More information about the cisco-nsp
mailing list