[c-nsp] Problem with 7507 ATM Crashing

Mon Jul 10 22:06:47 EDT 2006

Hey everyone,

	I have a 7507 - details below.  I did an upgrade of the IOS from:
rsp-ik9sv-mz.122-26.bin to rsp-ik9sv-mz.122-37.bin - which in all rights
would seem like a simple thing. no feature change or much but some
revisions.

	The reason we were upgrading is the ATM card was locking up, which
was fixable by a shut, no shut.  No idea what was actually causing it though
- no logs or anything.

	So I upgraded last night to 12.2(37) and then it all went to hell.
The logs showed:

0:00:50: %SYS-5-CONFIG_I: Configured from memory by console
00:02:31: %SCHED-3-THRASHING: Process thrashing on watched simple timer
(0x42914540).
-Process= "ATM PA Helper", ipl= 6, pid= 47
-Traceback= 403BFA3C 403BFDFC 404BC11C
00:02:31: %VIP2 R5K-3-MSG: slot0 VIP-3-LQR_IPC_FAIL:  IPC open failed
...
00:02:31: %VIP2 R5K-3-MSG: slot1 VIP-3-LQR_IPC_FAIL:  IPC open failed
00:02:31: %SYS-5-RESTART: System restarted --
Cisco Internetwork Operating System Software 
IOS (tm) RSP Software (RSP-IK9SV-M), Version 12.2(37), RELEASE SOFTWARE
(fc1)
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Thu 15-Jun-06 19:32 by pwade
00:02:31: %SNMP-5-COLDSTART: SNMP agent on host router is undergoing a cold
start
00:02:31: %SYS-6-BOOTTIME: Time taken to reboot after reload =  322 seconds
00:02:31: %SSH-5-ENABLED: SSH 1.5 has been enabled
00:02:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM0/1/0, changed
state to down
00:04:13: %SCHED-3-THRASHING: Process thrashing on watched simple timer
(0x42914540).
-Process= "ATM PA Helper", ipl= 6, pid= 47
-Traceback= 403BFA3C 403BFDFC 404BC11C
00:06:43: %SCHED-3-THRASHING: Process thrashing on watched simple timer
(0x42914540).
-Process= "ATM PA Helper", ipl= 6, pid= 47
-Traceback= 403BFA3C 403BFDFC 404BC11C
00:09:13: %SCHED-3-THRASHING: Process thrashing on watched simple timer
(0x42914540).
-Process= "ATM PA Helper", ipl= 6, pid= 47
-Traceback= 403BFA3C 403BFDFC 404BC11C
00:11:43: %SCHED-3-THRASHING: Process thrashing on watched simple timer
(0x42914540).
-Process= "ATM PA Helper", ipl= 6, pid= 47
-Traceback= 403BFA3C 403BFDFC 404BC11C

I downgraded back to 12.2(26) and it came back to its current working state.

Anyone know why such a trivial upgrade could cause such drastic problems?

.Skeeve

---

cisco RSP4 (R5000) processor with 262144K/2072K bytes of memory.
R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
Last reset from power-on
G.703/E1 software, Version 1.0.
G.703/JT2 software, Version 1.0.
X.25 software, Version 3.0.0.
Bridging software.
Chassis Interface.
2 VIP2 R5K controllers (3 FastEthernet)(1 ATM).
3 FastEthernet/IEEE 802.3 interface(s)
1 ATM network interface(s)
123K bytes of non-volatile configuration memory.

32768K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
16384K bytes of Flash PCMCIA card at slot 1 (Sector size 128K).
8192K bytes of Flash internal SIMM (Sector size 256K).

router#show diag
Slot 0:
        Physical slot 0, ~physical slot 0xF, logical slot 0, CBus 0
        Microcode Status 0x4
        Master Enable, LED, WCS Loaded
        Board is analyzed 
        Pending I/O Status: None
        EEPROM format version 1
        VIP2 R5K controller, HW rev 2.02, board revision C0
        Serial number: 12719540  Part number: 73-2167-05
        Test history: 0x00        RMA number: 00-00-00
        Flags: cisco 7000 board; 7500 compatible

        EEPROM contents (hex):
          0x20: 01 1E 02 02 00 C2 15 B4 49 08 77 05 00 00 00 00
          0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00

        Slot database information:
        Flags: 0x4      Insertion time: 0x2C3C (00:06:14 ago)

        Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM

        PA Bay 0 Information:
                Fast-Ethernet PA, 1 ports, 100BaseTX-ISL
                EEPROM format version 1
                HW rev 1.04, Board revision B0
                Serial number: 15805065  Part number: 73-1688-05 

        PA Bay 1 Information:
                ENHANCED ATM OC3 MM PA, 1 ports
                EEPROM format version 1
                HW rev 1.00, Board revision A0
                Serial number: 09590776  Part number: 73-2430-03 

        --Boot log begin--

Cisco Internetwork Operating System Software 
IOS (tm) VIP Software (SVIP-CDW-M), Version 12.2(37), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Thu 15-Jun-06 19:35 by pwade
Image text-base: 0x60010930, data-base: 0x60460000

        --Boot log end--

Slot 1:
        Physical slot 1, ~physical slot 0xE, logical slot 1, CBus 0
        Microcode Status 0x4
        Master Enable, LED, WCS Loaded
        Board is analyzed 
        Pending I/O Status: None
        EEPROM format version 1
        VIP2 R5K controller, HW rev 2.02, board revision D0
        Serial number: 17953867  Part number: 73-2167-05
        Test history: 0x00        RMA number: 00-00-00
        Flags: cisco 7000 board; 7500 compatible

        EEPROM contents (hex):
          0x20: 01 1E 02 02 01 11 F4 4B 49 08 77 05 00 00 00 00
          0x30: 68 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00

        Slot database information:
        Flags: 0x4      Insertion time: 0x2C3C (00:06:35 ago)

        Controller Memory Size: 128 MBytes DRAM, 4096 KBytes SRAM

        PA Bay 0 Information:
                Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
                EEPROM format version 1
                HW rev 1.01, Board revision A0
                Serial number: 14076764  Part number: 73-2618-02 

        --Boot log begin--

Cisco Internetwork Operating System Software 
IOS (tm) VIP Software (SVIP-CDW-M), Version 12.2(37), RELEASE SOFTWARE (fc1)
Copyright (c) 1986-2006 by cisco Systems, Inc.
Compiled Thu 15-Jun-06 19:35 by pwade
Image text-base: 0x60010930, data-base: 0x60460000

        --Boot log end--

Slot 3:   
        EEPROM format version 1
        Route/Switch Processor 4, HW rev 1.01, board revision D0
        Serial number: 11123251  Part number: 73-1689-03
        Test history: 0x00        RMA number: 00-00-00
        Flags: cisco 7000 board; 7500 compatible

        EEPROM contents (hex):
          0x20: 01 1A 01 01 00 A9 BA 33 49 06 99 03 00 00 00 00
          0x30: 68 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
Slot 31 (virtual):
        EEPROM format version 1
        Chassis Interface, HW rev 1.01, board revision C0
        Serial number: 12266020  Part number: 73-1306-02
        Test history: 0x00        RMA number: 00-00-00
        Flags: cisco 7000 board

        EEPROM contents (hex):
          0x20: 01 10 01 01 00 BB 2A 24 49 05 1A 02 00 00 00 00
          0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00

_______________________________________________________
Skeeve Stevens, RHCE     Email: skeeve at skeeve.org
Website: www.skeeve.org  - Telephone: (0414) 753 383
Address: P.O Box 1035, Epping, NSW, 1710, Australia

eIntellego - skeeve at eintellego.net - www.eintellego.net
_______________________________________________________
I'm a groove licked love child king of the verse
Si vis pacem, para bellum