[c-nsp] 7500 Crashes
Keegan Holley
keegan.holley at sungard.com
Sun Oct 16 12:40:04 EDT 2011
It looks like your logs start after the boot/switchover. Do you have a
syslog server? Anything interesting come in before these messages that may
have been wiped on boot? Does your show ver show a strange reboot reason or
a traceback? Do you have any thing setup for automated logins to this box?
Someone else mentioned an SSH bug. I think it was triggered when two ssh
sessions/sockets were started in quick succession IIRC.
2011/10/15 Jason Berenson <jason at pins.net>
> Greetings,
>
> I've got a 7500 running with the following:
>
> NAME: "7507z chassis, ID:76032590", DESCR: "7507z chassis"
> PID: 4 , VID: Hardware Version : 1.00 Board Revision : B0,
> SN: 76032590
>
> NAME: "Line Card 0", DESCR: "Versatile Interface Processor (VIP4-80)"
> PID: VIP4-80 , VID: Hardware Version : 2.04 Board Revision : A0,
> SN: 21008718
>
> NAME: "Card Slot 0, Bay 0", DESCR: "Dual Port FastEthernet (RJ45)"
> PID: PA-2FE-TX , VID: Hardware Version : 1.0 Board Revision : B0,
> SN: MIC050800X2
>
> NAME: "Card Slot 0, Bay 1", DESCR: "Serial T3+ Port Adapter"
> PID: PA-T3+ , VID: Hardware Version : 1.0 Board Revision : B0,
> SN: 17830995
>
> NAME: "Line Card 1", DESCR: "Versatile Interface Processor"
> PID: VIP2-50 , VID: Hardware Version : 2.03 Board Revision : A0,
> SN: 17943843
>
> NAME: "Card Slot 1, Bay 0", DESCR: "FastEthernet Port Adapter "
> PID: PA-FE-TX-ISL , VID: Hardware Version : 1.0 Board Revision : A0,
> SN: 06340347
>
> NAME: "Card Slot 1, Bay 1", DESCR: "Dual Port FastEthernet (RJ45)"
> PID: PA-2FE-TX , VID: Hardware Version : 1.0 Board Revision : C0,
> SN: JAE070103WM
>
> NAME: "RSP at Slot 3", DESCR: "R5000"
> PID: RSP4 , VID: Hardware Version : 1.1 Board Revision : D0,
> SN: 11519637
>
> NAME: "Line Card 4", DESCR: "Versatile Interface Processor (VIP4-80)"
> PID: VIP4-80 , VID: Hardware Version : 2.04 Board Revision : B0,
> SN: 28099188
>
> NAME: "Card Slot 4, Bay 0", DESCR: "Channelized DS3 - single wide, one
> port"
> PID: PA-MC-T3 , VID: Hardware Version : 1.0 Board Revision : A0,
> SN: 17832515
>
> NAME: "Card Slot 4, Bay 1", DESCR: "Channelized DS3 - single wide, one
> port"
> PID: PA-MC-T3 , VID: Hardware Version : 1.0 Board Revision : A0,
> SN: 17827319
>
> NAME: "Line Card 5", DESCR: "Versatile Interface Processor (VIP4-80)"
> PID: VIP4-80 , VID: Hardware Version : 2.03 Board Revision : A0,
> SN: 25567743
>
> NAME: "Card Slot 5, Bay 0", DESCR: "Channelized DS3 - single wide, one
> port"
> PID: PA-MC-T3 , VID: Hardware Version : 1.0 Board Revision : A0,
> SN: 24501038
>
> NAME: "Card Slot 5, Bay 1", DESCR: "Channelized DS3 - single wide, one
> port"
> PID: PA-MC-T3 , VID: Hardware Version : 1.0 Board Revision : A0,
> SN: 15811547
>
> NAME: "Line Card 6", DESCR: "Versatile Interface Processor (VIP6-80)"
> PID: VIP6-80 , VID: Hardware Version : 2.01 Board Revision : A0,
> SN: 29452845
>
> NAME: "Card Slot 6, Bay 0", DESCR: "GigabitEthernet Port Adapter"
> PID: PA-GE , VID: Hardware Version : 1.0 Board Revision : A1,
> SN: 18636663
>
> The box keeps switching between the RSPs for some reason. Here's the
> relevant logs:
>
> *Oct 15 01:11:07.175: %HA-5-NOTICE: Suspending initialization pending
> bulk/config sync...
> *Oct 15 01:11:07.251: %HA-5-NOTICE: Command line console will unavailable
> until init completes.
> *Oct 15 01:11:07.351: %HA-5-NOTICE: Defaulting to hsa mode until sync
> completed and system restarted.
> *Oct 15 01:11:07.363: %HA-5-SYNC_NOTICE: Bulk sync started.
> *Oct 15 01:11:07.675: %HA-5-SYNC_NOTICE: Bulk sync completed.
> *Oct 15 01:11:11.899: %HA-5-SYNC_NOTICE: Config sync started.
> *Oct 15 01:11:11.975: %HA-5-NOTICE: Resuming initialization...
> *Oct 14 20:11:16.987: %SYS-6-CLOCKUPDATE: System clock has been updated
> from 01:11:16 UTC Sat Oct 15 2011 to 20:11:16 EST Fri Oct 14 2011,
> configured from console by console.
> *Oct 14 21:11:16.987: %SYS-6-CLOCKUPDATE: System clock has been updated
> from 20:11:16 EST Fri Oct 14 2011 to 21:11:16 EDT Fri Oct 14 2011,
> configured from console by console.
> Oct 14 21:11:34.274: %HA-5-SYNC_NOTICE: Config sync completed.
> Oct 14 21:11:34.926: %SYS-5-RESTART: System restarted --
> Cisco IOS Software, RSP Software (RSP-JK9O3SV-M), Version 12.4(25), RELEASE
> SOFTWARE (fc2)
> Technical Support: http://www.cisco.com/**techsupport<http://www.cisco.com/techsupport>
> Copyright (c) 1986-2009 by Cisco Systems, Inc.
> Compiled Tue 21-Apr-09 19:28 by prod_rel_team
> Oct 14 21:11:35.046: %SSH-5-ENABLED: SSH 1.99 has been enabled
> Oct 14 21:11:39.930: %SNMP-5-COLDSTART: SNMP agent on host CS75-111.1 is
> undergoing a cold start
> Oct 15 00:24:04.052: %HA-2-CUTOVER_NOTICE: Cutover initiated. Cease all
> console activity until system restarts.
> Oct 15 00:24:04.052: %HA-2-CUTOVER_NOTICE: Do not add/remove RSPs or line
> cards until switchover completes.
> Oct 15 00:24:04.052: %HA-2-CUTOVER_NOTICE: Deinitializing subsystems...
> Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 0, interfaces
> disabled
> Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 1, interfaces
> disabled
> Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 4, interfaces
> disabled
> Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 5, interfaces
> disabled
> Oct 15 00:24:04.052: %OIR-6-REMCARD: Card removed from slot 6, interfaces
> disabled
> Oct 15 00:24:04.148: Suspending service policy voice-policy-1024 on
> Multilink50021
> Oct 15 00:24:04.164: %HA-2-CUTOVER_NOTICE: Reinitializing subsystems...
> Oct 15 00:24:04.916: %HA-2-CUTOVER_NOTICE: System preparing to restart...
> Oct 15 00:24:04.916: %HA-5-NOTICE: Resuming initialization...
> Oct 15 00:24:05.056: %HA-2-CUTOVER_NOTICE: Cutover completed and system
> restarted.
> Oct 15 00:24:05.056: %HA-2-CUTOVER_NOTICE: You may add/remove RSPs and line
> cards after interfaces are fully operational.
> Oct 15 00:24:05.100: %HA-5-NOTICE: Standby (slave) will be restarted in
> approximately 5 minutes.
> Oct 15 00:24:23.412: %RSP-3-SLAVECHANGE: Slave changed state from
> Non-participant to Slave
> Oct 15 00:29:05.113: %RSP-3-SLAVECHANGE: Slave changed state from Slave to
> Non-participant
> Oct 15 00:29:07.765: %RSP-3-SLAVECHANGE: Slave changed state from
> Non-participant to Slave
> Oct 15 00:29:13.449: %RSP-3-SLAVECHANGE: Slave changed state from Slave to
> Non-participant
> Oct 15 00:29:16.101: %RSP-3-SLAVECHANGE: Slave changed state from
> Non-participant to Slave
> Oct 15 00:29:21.125: %RSP-3-SLAVECHANGE: Slave changed state from Slave to
> Non-participant
> Oct 15 00:29:23.777: %RSP-3-SLAVECHANGE: Slave changed state from
> Non-participant to Slave
> Oct 15 00:30:06.686: %HA-5-NOTICE: Standby (slave) configured to run HA
> image "disk0:rsp-jk9o3sv-mz.124-25.**bin"
> Oct 15 00:30:07.218: %HA-5-NOTICE: Loading standby (slave) image:
> "disk0:rsp-jk9o3sv-mz.124-25.**bin"
> Oct 15 00:31:28.690: %HA-5-MODE: Operating mode is hsa, configured mode is
> rpr-plus.
> Oct 15 00:31:28.690: %HA-5-SYNC_NOTICE: Bulk sync started.
> Oct 15 00:31:28.814: %HA-5-SYNC_NOTICE: Bulk sync completed.
> Oct 15 00:31:35.647: %HA-5-SYNC_NOTICE: Config sync started.
> Oct 15 00:31:54.067: %HA-5-SYNC_NOTICE: Config sync completed.
> Oct 15 00:31:59.483: %HA-5-SYNC_NOTICE: Standby has restarted.
> Oct 15 00:31:59.491: %HA-5-MODE: Operating mode is rpr-plus, configured
> mode is rpr-plus.
>
> Here's the output of show redundancy:
>
> Redundant System Information :
> ------------------------------
> Available system uptime = 21 hours, 14 minutes
> Switchovers system experienced = 2
> Standby failures = 0
> Last switchover reason = active unit failed
>
> Hardware Mode = Duplex
> Configured Redundancy Mode = rpr-plus
> Operating Redundancy Mode = rpr-plus
> Maintenance Mode = Disabled
> Communications = Up
>
> Current Processor Information :
> ------------------------------**-
> Active Location = slot 3
> Current Software state = ACTIVE
> Uptime in current state = 12 hours, 28 minutes
> Image Version = Cisco IOS Software, RSP Software
> (RSP-JK9O3SV-M), Version 12.4(25), RELEASE SOFTWARE (fc2)
> Technical Support: http://www.cisco.com/**techsupport<http://www.cisco.com/techsupport>
> Copyright (c) 1986-2009 by Cisco Systems, Inc.
> Compiled Tue 21-Apr-09 19:28 by prod_rel_team
> BOOT = disk0:rsp-jk9o3sv-mz.124-25.**bin,1;
> CONFIG_FILE =
> BOOTLDR =
> Configuration register = 0x2102
>
> Peer Processor Information :
> ----------------------------
> Standby Location = slot 2
> Current Software state = STANDBY COLD-CONFIG
> Uptime in current state = 12 hours, 21 minutes
> Image Version = Cisco IOS Software, RSP Software
> (RSP-JK9O3SV-M), Version 12.4(25), RELEASE SOFTWARE (fc2)
> Technical Support: http://www.cisco.com/**techsupport<http://www.cisco.com/techsupport>
> Copyright (c) 1986-2009 by Cisco Systems, Inc.
> Compiled Tue 21-Apr-09 19:28 by prod_rel_team
> BOOT = disk0:rsp-jk9o3sv-mz.124-25.**bin,1;
> CONFIG_FILE =
> BOOTLDR =
> Configuration register = 0x2102
>
> If anymore information is needed let me know and I'll post it. This router
> keeps doing this every few hours. Any help would be greatly appreciated.
>
> Thanks,
> Jason
>
>
> ______________________________**_________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/**mailman/listinfo/cisco-nsp<https://puck.nether.net/mailman/listinfo/cisco-nsp>
> archive at http://puck.nether.net/**pipermail/cisco-nsp/<http://puck.nether.net/pipermail/cisco-nsp/>
>
>
More information about the cisco-nsp
mailing list