[c-nsp] 7500 Crashes

Jason Berenson jason at pins.net
Sat Oct 15 12:57:45 EDT 2011


Greetings,

I've got a 7500 running with the following:

NAME: "7507z chassis, ID:76032590", DESCR: "7507z chassis"
PID: 4                 , VID: Hardware Version : 1.00 Board Revision : 
B0, SN: 76032590

NAME: "Line Card 0", DESCR: "Versatile Interface Processor (VIP4-80)"
PID: VIP4-80           , VID: Hardware Version : 2.04 Board Revision : 
A0, SN: 21008718

NAME: "Card Slot 0, Bay 0", DESCR: "Dual Port FastEthernet (RJ45)"
PID: PA-2FE-TX         , VID: Hardware Version : 1.0 Board Revision : 
B0, SN: MIC050800X2

NAME: "Card Slot 0, Bay 1", DESCR: "Serial T3+ Port Adapter"
PID: PA-T3+            , VID: Hardware Version : 1.0 Board Revision : 
B0, SN: 17830995

NAME: "Line Card 1", DESCR: "Versatile Interface Processor"
PID: VIP2-50           , VID: Hardware Version : 2.03 Board Revision : 
A0, SN: 17943843

NAME: "Card Slot 1, Bay 0", DESCR: "FastEthernet Port Adapter "
PID: PA-FE-TX-ISL      , VID: Hardware Version : 1.0 Board Revision : 
A0, SN: 06340347

NAME: "Card Slot 1, Bay 1", DESCR: "Dual Port FastEthernet (RJ45)"
PID: PA-2FE-TX         , VID: Hardware Version : 1.0 Board Revision : 
C0, SN: JAE070103WM

NAME: "RSP at Slot 3", DESCR: "R5000"
PID: RSP4              , VID: Hardware Version : 1.1 Board Revision : 
D0, SN: 11519637

NAME: "Line Card 4", DESCR: "Versatile Interface Processor (VIP4-80)"
PID: VIP4-80           , VID: Hardware Version : 2.04 Board Revision : 
B0, SN: 28099188

NAME: "Card Slot 4, Bay 0", DESCR: "Channelized DS3 - single wide, one port"
PID: PA-MC-T3          , VID: Hardware Version : 1.0 Board Revision : 
A0, SN: 17832515

NAME: "Card Slot 4, Bay 1", DESCR: "Channelized DS3 - single wide, one port"
PID: PA-MC-T3          , VID: Hardware Version : 1.0 Board Revision : 
A0, SN: 17827319

NAME: "Line Card 5", DESCR: "Versatile Interface Processor (VIP4-80)"
PID: VIP4-80           , VID: Hardware Version : 2.03 Board Revision : 
A0, SN: 25567743

NAME: "Card Slot 5, Bay 0", DESCR: "Channelized DS3 - single wide, one port"
PID: PA-MC-T3          , VID: Hardware Version : 1.0 Board Revision : 
A0, SN: 24501038

NAME: "Card Slot 5, Bay 1", DESCR: "Channelized DS3 - single wide, one port"
PID: PA-MC-T3          , VID: Hardware Version : 1.0 Board Revision : 
A0, SN: 15811547

NAME: "Line Card 6", DESCR: "Versatile Interface Processor (VIP6-80)"
PID: VIP6-80           , VID: Hardware Version : 2.01 Board Revision : 
A0, SN: 29452845

NAME: "Card Slot 6, Bay 0", DESCR: "GigabitEthernet Port Adapter"
PID: PA-GE             , VID: Hardware Version : 1.0 Board Revision : 
A1, SN: 18636663

The box keeps switching between the RSPs for some reason.  Here's the 
relevant logs:

*Oct 15 01:11:07.175: %HA-5-NOTICE: Suspending initialization pending 
bulk/config sync...
*Oct 15 01:11:07.251: %HA-5-NOTICE: Command line console will 
unavailable until init completes.
*Oct 15 01:11:07.351: %HA-5-NOTICE: Defaulting to hsa mode until sync 
completed and system restarted.
*Oct 15 01:11:07.363: %HA-5-SYNC_NOTICE: Bulk sync started.
*Oct 15 01:11:07.675: %HA-5-SYNC_NOTICE: Bulk sync completed.
*Oct 15 01:11:11.899: %HA-5-SYNC_NOTICE: Config sync started.
*Oct 15 01:11:11.975: %HA-5-NOTICE: Resuming initialization...
*Oct 14 20:11:16.987: %SYS-6-CLOCKUPDATE: System clock has been updated 
from 01:11:16 UTC Sat Oct 15 2011 to 20:11:16 EST Fri Oct 14 2011, 
configured from console by console.
*Oct 14 21:11:16.987: %SYS-6-CLOCKUPDATE: System clock has been updated 
from 20:11:16 EST Fri Oct 14 2011 to 21:11:16 EDT Fri Oct 14 2011, 
configured from console by console.
Oct 14 21:11:34.274: %HA-5-SYNC_NOTICE: Config sync completed.
Oct 14 21:11:34.926: %SYS-5-RESTART: System restarted --
Cisco IOS Software, RSP Software (RSP-JK9O3SV-M), Version 12.4(25), 
RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 21-Apr-09 19:28 by prod_rel_team
Oct 14 21:11:35.046: %SSH-5-ENABLED: SSH 1.99 has been enabled
Oct 14 21:11:39.930: %SNMP-5-COLDSTART: SNMP agent on host CS75-111.1 is 
undergoing a cold start
Oct 15 00:24:04.052: %HA-2-CUTOVER_NOTICE: Cutover initiated. Cease all 
console activity until system restarts.
Oct 15 00:24:04.052: %HA-2-CUTOVER_NOTICE: Do not add/remove RSPs or 
line cards until switchover completes.
Oct 15 00:24:04.052: %HA-2-CUTOVER_NOTICE: Deinitializing subsystems...
Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 0, 
interfaces disabled
Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 1, 
interfaces disabled
Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 4, 
interfaces disabled
Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 5, 
interfaces disabled
Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 6, 
interfaces disabled
Oct 15 00:24:04.148: Suspending service policy voice-policy-1024 on 
Multilink50021
Oct 15 00:24:04.164: %HA-2-CUTOVER_NOTICE: Reinitializing subsystems...
Oct 15 00:24:04.916: %HA-2-CUTOVER_NOTICE: System preparing to restart...
Oct 15 00:24:04.916: %HA-5-NOTICE: Resuming initialization...
Oct 15 00:24:05.056: %HA-2-CUTOVER_NOTICE: Cutover completed and system 
restarted.
Oct 15 00:24:05.056: %HA-2-CUTOVER_NOTICE: You may add/remove RSPs and 
line cards after interfaces are fully operational.
Oct 15 00:24:05.100: %HA-5-NOTICE: Standby (slave) will be restarted in 
approximately 5 minutes.
Oct 15 00:24:23.412: %RSP-3-SLAVECHANGE: Slave changed state from 
Non-participant to Slave
Oct 15 00:29:05.113: %RSP-3-SLAVECHANGE: Slave changed state from Slave 
to Non-participant
Oct 15 00:29:07.765: %RSP-3-SLAVECHANGE: Slave changed state from 
Non-participant to Slave
Oct 15 00:29:13.449: %RSP-3-SLAVECHANGE: Slave changed state from Slave 
to Non-participant
Oct 15 00:29:16.101: %RSP-3-SLAVECHANGE: Slave changed state from 
Non-participant to Slave
Oct 15 00:29:21.125: %RSP-3-SLAVECHANGE: Slave changed state from Slave 
to Non-participant
Oct 15 00:29:23.777: %RSP-3-SLAVECHANGE: Slave changed state from 
Non-participant to Slave
Oct 15 00:30:06.686: %HA-5-NOTICE: Standby (slave) configured to run HA 
image "disk0:rsp-jk9o3sv-mz.124-25.bin"
Oct 15 00:30:07.218: %HA-5-NOTICE: Loading standby (slave) image: 
"disk0:rsp-jk9o3sv-mz.124-25.bin"
Oct 15 00:31:28.690: %HA-5-MODE: Operating mode is hsa, configured mode 
is rpr-plus.
Oct 15 00:31:28.690: %HA-5-SYNC_NOTICE: Bulk sync started.
Oct 15 00:31:28.814: %HA-5-SYNC_NOTICE: Bulk sync completed.
Oct 15 00:31:35.647: %HA-5-SYNC_NOTICE: Config sync started.
Oct 15 00:31:54.067: %HA-5-SYNC_NOTICE: Config sync completed.
Oct 15 00:31:59.483: %HA-5-SYNC_NOTICE: Standby has restarted.
Oct 15 00:31:59.491: %HA-5-MODE: Operating mode is rpr-plus, configured 
mode is rpr-plus.

Here's the output of show redundancy:

Redundant System Information :
------------------------------
        Available system uptime = 21 hours, 14 minutes
Switchovers system experienced = 2
               Standby failures = 0
         Last switchover reason = active unit failed

                  Hardware Mode = Duplex
     Configured Redundancy Mode = rpr-plus
      Operating Redundancy Mode = rpr-plus
               Maintenance Mode = Disabled
                 Communications = Up

Current Processor Information :
-------------------------------
                Active Location = slot 3
         Current Software state = ACTIVE
        Uptime in current state = 12 hours, 28 minutes
                  Image Version = Cisco IOS Software, RSP Software 
(RSP-JK9O3SV-M), Version 12.4(25), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 21-Apr-09 19:28 by prod_rel_team
                           BOOT = disk0:rsp-jk9o3sv-mz.124-25.bin,1;
                    CONFIG_FILE =
                        BOOTLDR =
         Configuration register = 0x2102

Peer Processor Information :
----------------------------
               Standby Location = slot 2
         Current Software state = STANDBY COLD-CONFIG
        Uptime in current state = 12 hours, 21 minutes
                  Image Version = Cisco IOS Software, RSP Software 
(RSP-JK9O3SV-M), Version 12.4(25), RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2009 by Cisco Systems, Inc.
Compiled Tue 21-Apr-09 19:28 by prod_rel_team
                           BOOT = disk0:rsp-jk9o3sv-mz.124-25.bin,1;
                    CONFIG_FILE =
                        BOOTLDR =
         Configuration register = 0x2102

If anymore information is needed let me know and I'll post it.  This 
router keeps doing this every few hours.  Any help would be greatly 
appreciated.

Thanks,
Jason




More information about the cisco-nsp mailing list