[c-nsp] Weird IOS problem

Hank Nussbacher hank at mail.iucc.ac.il
Wed Jan 5 05:36:01 EST 2005 

At 09:09 AM 05-01-05 +0100, Brian Turnbow wrote:

Our VIP4-50 has a PA-2FE and PA-8T and we suspect the PA-2FE as being the 
problem.  Thing is it had been working for 3 years without a problem till 
now.  Weird.   -Hank

>  I had a similar problem with (12.2(18)S7)on a 7507 rsp 4 and a vip4-50
>With a combination STM-1 FE Pas. I changed the vip and Pas with no change.
>Replacing the STM-1 with a E3 resolved everything and My router is now happy.
>The other 3 VIPs with different PA combinations were ok.
>
>-----Original Message-----
>From: cisco-nsp-bounces at puck.nether.net 
>[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Hank Nussbacher
>Sent: mercoledì 5 gennaio 2005 7.16
>To: cisco-nsp at puck.nether.net
>Subject: [c-nsp] Weird IOS problem
>
>This is a swing in the dark that perhaps someone else has seen this 
>before.  Our 7513 (w/ RSP16) running 12.2(18)S7 seems to be having a bad 
>day.  One particular VIP4-50 (slot1) started misbehaving and it causes all 
>other VIPs to lose their CEF and for that particular VIP to take itself 
>off-line!
>
>Log output:
>Jan  5 07:50:57: %CBUS-3-CMDTIMEOUT: Cmd timed out, CCB 0xF800FF30, slot 
>1, cmd code 2 -Traceback= 404F8334 404F8BF8 404EFF44 404ECD40 404CE974 
>4045C4A4 403D6B94 Jan  5 07:50:58: %DBUS-3-DBUSINTERR: Slot 1, Internal 
>Error Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (61 
>0x00000008) failed (0x8010) Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: 
>Controller 1, cmd (36 0x00000008) failed (0x8010) Jan  5 07:50:58: 
>%CBUS-3-ADDRFILTR: Interface FastEthernet1/0/1, address filter write 
>command failed, code 0x8010 -Traceback= 404FE1F8 404FEA40 404FECD4 
>40403F08 40651774 409C5234 409C588C
>40A09674 409E5960 40420C1C 404DDDFC 404DDF80 404E4990 404CE86C Jan  5 
>07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x0000FFFF) failed 
>(0x8010) Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 
>0x00000100) failed (0x8010) Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: 
>Controller 1, cmd (36 0x00000100) failed (0x8010) Jan  5 07:50:58: 
>%CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) failed (0x8010) 
>Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd (36 0x00003333) 
>failed (0x8010) Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: Controller 1, cmd 
>(36 0x00003333) failed (0x8010) Jan  5 07:50:58: %CBUS-3-CCBCMDFAIL1: 
>Controller 1, cmd (36 0x00003333) failed (0x8010) Jan  5 07:50:59: 
>%OSPF-5-ADJCHG: Process 378, Nbr 192.114.22.1 on
>FastEthernet1/0/1 from 2WAY to DOWN, Neighbor Down: Interface down or 
>detached Jan  5 07:50:59: %OSPF-5-ADJCHG: Process 378, Nbr 192.114.99.52 on
>FastEthernet1/0/1 from FULL to DOWN, Neighbor Down: Interface down or 
>detached Jan  5 07:51:12: %DBUS-3-SW_NOTRDY: DBUS software not ready after 
>HARD_RESET, elapsed 13012, status 0x0 -Traceback= 404DE264 404E5E64 
>404F4620 404F7E1C 404E49B4 404CE86C Jan  5 07:51:25: %DBUS-3-SW_NOTRDY: 
>DBUS software not ready after HARD_RESET, elapsed 13036, status 0x0 
>-Traceback= 404DE264 404E5E64 404F7EA0 404E49B4 404CE86C Jan  5 07:51:38: 
>%DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13036, 
>status 0x0 -Traceback= 404DE264 404E5A78 404F7EB8 404E49B4 404CE86C Jan  5 
>07:51:51: %DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 
>13028, status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 
>404DCD90 404F7EE4
>404E49B4 404CE86C
>Jan  5 07:52:02: %MDS-2-RP: MDFS is disabled on some line card(s). Use 
>"show ip mds stats linecard" to view status and "clear ip mds linecard" to 
>reset.
>Jan  5 07:54:28: %DBUS-3-SW_NOTRDY: DBUS software not ready after 
>DOWNLOAD, elapsed 20024, status 0x40 -Traceback= 404DE264 404E3E60 
>404DC470 404DC8B4 404DCD90 404F7EE4 404E49B4 404CE86C Jan  5 07:54:28: 
>%DBUS-3-WCSLDERR: Slot 1, error loading WCS, status 0x40 cmd/data 0xB6 pos 
>4818686 Jan  5 07:54:28: %UCODE-3-LDFAIL: Unable to download ucode from 
>system image in slot 1, trying rom ucode Jan  5 07:54:42: 
>%DBUS-3-SW_NOTRDY: DBUS software not ready after HARD_RESET, elapsed 
>13064, status 0x0 -Traceback= 404DE264 404E1448 404E7BDC Jan  5 07:54:55: 
>%DBUS-3-SW_NOTRDY: DBUS software not ready after RESET, elapsed 13064, 
>status 0x0 -Traceback= 404DE264 404DEA78 404E3B44 404DC470 404DC8B4 
>404DCD90 404E1450 404E7BDC Jan  5 07:55:54: %FIB-3-FIBDISABLE: Fatal 
>error, slot/cpu 2/0: No window message, LC to RP IPC is non-operational 
>Jan  5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 4/0: No window 
>message, LC to RP IPC is non-operational Jan  5 07:55:54: 
>%FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: No window message, LC to RP 
>IPC is non-operational Jan  5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, 
>slot/cpu 3/0: No window message, LC to RP IPC is non-operational Jan  5 
>07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 8/0: No window message, 
>LC to RP IPC is non-operational Jan  5 07:55:54: %FIB-3-FIBDISABLE: Fatal 
>error, slot/cpu 11/0: No window message, LC to RP IPC is non-operational 
>Jan  5 07:55:54: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 9/0: No window 
>message, LC to RP IPC is non-operational .Jan  5 07:57:33: 
>%DBUS-3-SW_NOTRDY: DBUS software not ready after DOWNLOAD, elapsed 20012, 
>status 0x40 -Traceback= 404DE264 404E3E60 404DC470 404DC8B4 404DCD90 
>404E1450 404E7BDC .Jan  5 07:57:33: %DBUS-3-WCSLDERR: Slot 1, error 
>loading WCS, status 0x40 cmd/data 0xB6 pos 4818686 .Jan  5 07:57:33: 
>%UCODE-3-LDFAIL: Unable to download ucode from system image in slot 1, 
>trying rom ucode .Jan  5 07:57:33: %RSP-3-NOSTART: No microcode for 
>VIP4-50 RM5271 card, slot 1 .Jan  5 07:57:34: %MDS-2-LC_FAILED_IPC_ACK: RP 
>failed in getting Ack for IPC message of size 80 to LC in slot 1 with 
>sequence 40250, error = retry queue flush .Jan  5 07:57:55: 
>%SNMP-3-AUTHFAIL: Authentication failure for SNMP req from host 
>132.74.1.154 .Jan  5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 
>8/0: keepalive failure .Jan  5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, 
>slot/cpu 4/0: keepalive failure .Jan  5 08:00:46: %FIB-3-FIBDISABLE: Fatal 
>error, slot/cpu 9/0: keepalive failure .Jan  5 08:00:46: 
>%FIB-3-FIBDISABLE: Fatal error, slot/cpu 11/0: keepalive failure .Jan  5 
>08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 10/0: keepalive failure 
>.Jan  5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 2/0: keepalive 
>failure .Jan  5 08:00:46: %FIB-3-FIBDISABLE: Fatal error, slot/cpu 3/0: 
>keepalive failure TAU-gp1#sho cef line
>  Slot/CPU  MsgSent    XDRSent  Window   LowQ   MedQ  HighQ Flags
>  2            5669     160596 LC wait      0      0      0 disabled
>  3            5669     160598 LC wait      0      0      0 disabled
>  4            5738     161622 LC wait      0      0      0 disabled
>  8            5742     161621 LC wait      0      0      0 disabled
>  9            5668     160577 LC wait      0      0      0 disabled
>  10           5668     160574 LC wait      0      0      0 disabled
>  11           5668     160577 LC wait      0      0      0 disabled
>
>VRF Default, version 311259, 153409 routes
>  Slot/CPU Version    CEF-XDR    I/Fs State    Flags
>  2         310726     157274       9 Active   table-disabled
>  3         310726     157274       8 Active   table-disabled
>  4         310726     157484       6 Active   table-disabled
>  8         310726     157484       6 Active   table-disabled
>  9         310726     157274       8 Active   table-disabled
>  10        310726     157274      18 Active   table-disabled
>  11        310726     157274       6 Active   table-disabled
>TAU-gp1#show ip mds stats linecard
>
>Slot      Status    IPC(seq/max/window) Q(high/route)  Reloads
>  1        disabled  40455/44286/3831       0/0            1
>  2        active    258  /4354 /4096       0/0            2
>  3        active    267  /4363 /4096       0/0            2
>  4        active    284  /4380 /4096       0/0            2
>  8        active    239  /4335 /4096       0/0            2
>  9        active    275  /4371 /4096       0/0            2
>  10       active    231  /2279 /2048       0/0            2
>  11       active    249  /4345 /4096       0/0            2
>
>Taking out VIP1 from the chassis and rebooting solves the problem.  We 
>have already replaced the VIP in slot1 yesterday when it first happened 
>and life went on for about 12 hours before it happened again.
>
>Anyone seen anything like this?  Cisco output intepreter was of no help 
>since it indicated things that we don't run at all.
>
>Thanks,
>Hank
>_______________________________________________
>cisco-nsp mailing list  cisco-nsp at puck.nether.net 
>https://puck.nether.net/mailman/listinfo/cisco-nsp
>archive at http://puck.nether.net/pipermail/cisco-nsp/
>
>  +++++++++++++++++++++++++++++++++++++++++++
>  This Mail Was Scanned By Mail-seCure System
>  at the Tel-Aviv University CC.

More information about the cisco-nsp mailing list