[c-nsp] Problem with 7507 ATM Crashing
Rodney Dunn
rodunn at cisco.com
Tue Jul 11 13:56:55 EDT 2006
Skeeve,
After reading some more bugs...
This might be something else.
I see that if a VIP can't be communicated to from the RSP then
the ATM PA helper routine will be constantly trying to contact
the VIP for the ATM PA. That might be the trigger for the process
thrashing message and is a side item.
The root problem might be this:
> > 00:02:31: %VIP2 R5K-3-MSG: slot0 VIP-3-LQR_IPC_FAIL: IPC open failed
> > ...
> > 00:02:31: %VIP2 R5K-3-MSG: slot1 VIP-3-LQR_IPC_FAIL: IPC open failed
why IPC wasn't opened between the RSP and those two slots.
Is there anything unique about those two slots?
On Tue, Jul 11, 2006 at 01:52:34PM -0400, Rodney Dunn wrote:
> Skeeve,
>
> What does 'sh diag' say for that PA. I did a couple of quick bug
> searches and a couple of them pointed back to this hardware
> field notice:
>
> http://www.cisco.com/en/US/products/hw/routers/ps341/products_field_notice09186a0080178b6b.shtml
>
> I have no clue what the upgrade might have triggered it if that's
> relevant.
>
> Rodney
>
> On Tue, Jul 11, 2006 at 12:06:47PM +1000, Skeeve Stevens wrote:
> >
> > Hey everyone,
> >
> > I have a 7507 - details below. I did an upgrade of the IOS from:
> > rsp-ik9sv-mz.122-26.bin to rsp-ik9sv-mz.122-37.bin - which in all rights
> > would seem like a simple thing. no feature change or much but some
> > revisions.
> >
> > The reason we were upgrading is the ATM card was locking up, which
> > was fixable by a shut, no shut. No idea what was actually causing it though
> > - no logs or anything.
> >
> > So I upgraded last night to 12.2(37) and then it all went to hell.
> > The logs showed:
> >
> > 0:00:50: %SYS-5-CONFIG_I: Configured from memory by console
> > 00:02:31: %SCHED-3-THRASHING: Process thrashing on watched simple timer
> > (0x42914540).
> > -Process= "ATM PA Helper", ipl= 6, pid= 47
> > -Traceback= 403BFA3C 403BFDFC 404BC11C
> > 00:02:31: %VIP2 R5K-3-MSG: slot0 VIP-3-LQR_IPC_FAIL: IPC open failed
> > ...
> > 00:02:31: %VIP2 R5K-3-MSG: slot1 VIP-3-LQR_IPC_FAIL: IPC open failed
> > 00:02:31: %SYS-5-RESTART: System restarted --
> > Cisco Internetwork Operating System Software
> > IOS (tm) RSP Software (RSP-IK9SV-M), Version 12.2(37), RELEASE SOFTWARE
> > (fc1)
> > Copyright (c) 1986-2006 by cisco Systems, Inc.
> > Compiled Thu 15-Jun-06 19:32 by pwade
> > 00:02:31: %SNMP-5-COLDSTART: SNMP agent on host router is undergoing a cold
> > start
> > 00:02:31: %SYS-6-BOOTTIME: Time taken to reboot after reload = 322 seconds
> > 00:02:31: %SSH-5-ENABLED: SSH 1.5 has been enabled
> > 00:02:32: %LINEPROTO-5-UPDOWN: Line protocol on Interface ATM0/1/0, changed
> > state to down
> > 00:04:13: %SCHED-3-THRASHING: Process thrashing on watched simple timer
> > (0x42914540).
> > -Process= "ATM PA Helper", ipl= 6, pid= 47
> > -Traceback= 403BFA3C 403BFDFC 404BC11C
> > 00:06:43: %SCHED-3-THRASHING: Process thrashing on watched simple timer
> > (0x42914540).
> > -Process= "ATM PA Helper", ipl= 6, pid= 47
> > -Traceback= 403BFA3C 403BFDFC 404BC11C
> > 00:09:13: %SCHED-3-THRASHING: Process thrashing on watched simple timer
> > (0x42914540).
> > -Process= "ATM PA Helper", ipl= 6, pid= 47
> > -Traceback= 403BFA3C 403BFDFC 404BC11C
> > 00:11:43: %SCHED-3-THRASHING: Process thrashing on watched simple timer
> > (0x42914540).
> > -Process= "ATM PA Helper", ipl= 6, pid= 47
> > -Traceback= 403BFA3C 403BFDFC 404BC11C
> >
> >
> > I downgraded back to 12.2(26) and it came back to its current working state.
> >
> > Anyone know why such a trivial upgrade could cause such drastic problems?
> >
> > .Skeeve
> >
> >
> >
> > ---
> >
> > cisco RSP4 (R5000) processor with 262144K/2072K bytes of memory.
> > R5000 CPU at 200Mhz, Implementation 35, Rev 2.1, 512KB L2 Cache
> > Last reset from power-on
> > G.703/E1 software, Version 1.0.
> > G.703/JT2 software, Version 1.0.
> > X.25 software, Version 3.0.0.
> > Bridging software.
> > Chassis Interface.
> > 2 VIP2 R5K controllers (3 FastEthernet)(1 ATM).
> > 3 FastEthernet/IEEE 802.3 interface(s)
> > 1 ATM network interface(s)
> > 123K bytes of non-volatile configuration memory.
> >
> > 32768K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
> > 16384K bytes of Flash PCMCIA card at slot 1 (Sector size 128K).
> > 8192K bytes of Flash internal SIMM (Sector size 256K).
> >
> >
> > router#show diag
> > Slot 0:
> > Physical slot 0, ~physical slot 0xF, logical slot 0, CBus 0
> > Microcode Status 0x4
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > Pending I/O Status: None
> > EEPROM format version 1
> > VIP2 R5K controller, HW rev 2.02, board revision C0
> > Serial number: 12719540 Part number: 73-2167-05
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1E 02 02 00 C2 15 B4 49 08 77 05 00 00 00 00
> > 0x30: 60 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x2C3C (00:06:14 ago)
> >
> > Controller Memory Size: 128 MBytes DRAM, 8192 KBytes SRAM
> >
> > PA Bay 0 Information:
> > Fast-Ethernet PA, 1 ports, 100BaseTX-ISL
> > EEPROM format version 1
> > HW rev 1.04, Board revision B0
> > Serial number: 15805065 Part number: 73-1688-05
> >
> > PA Bay 1 Information:
> > ENHANCED ATM OC3 MM PA, 1 ports
> > EEPROM format version 1
> > HW rev 1.00, Board revision A0
> > Serial number: 09590776 Part number: 73-2430-03
> >
> > --Boot log begin--
> >
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-CDW-M), Version 12.2(37), RELEASE SOFTWARE (fc1)
> > Copyright (c) 1986-2006 by cisco Systems, Inc.
> > Compiled Thu 15-Jun-06 19:35 by pwade
> > Image text-base: 0x60010930, data-base: 0x60460000
> >
> >
> > --Boot log end--
> >
> > Slot 1:
> > Physical slot 1, ~physical slot 0xE, logical slot 1, CBus 0
> > Microcode Status 0x4
> > Master Enable, LED, WCS Loaded
> > Board is analyzed
> > Pending I/O Status: None
> > EEPROM format version 1
> > VIP2 R5K controller, HW rev 2.02, board revision D0
> > Serial number: 17953867 Part number: 73-2167-05
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1E 02 02 01 11 F4 4B 49 08 77 05 00 00 00 00
> > 0x30: 68 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> >
> > Slot database information:
> > Flags: 0x4 Insertion time: 0x2C3C (00:06:35 ago)
> >
> > Controller Memory Size: 128 MBytes DRAM, 4096 KBytes SRAM
> >
> > PA Bay 0 Information:
> > Fast-Ethernet PA, 2 ports, 100BaseTX-ISL
> > EEPROM format version 1
> > HW rev 1.01, Board revision A0
> > Serial number: 14076764 Part number: 73-2618-02
> >
> > --Boot log begin--
> >
> > Cisco Internetwork Operating System Software
> > IOS (tm) VIP Software (SVIP-CDW-M), Version 12.2(37), RELEASE SOFTWARE (fc1)
> > Copyright (c) 1986-2006 by cisco Systems, Inc.
> > Compiled Thu 15-Jun-06 19:35 by pwade
> > Image text-base: 0x60010930, data-base: 0x60460000
> >
> >
> > --Boot log end--
> >
> > Slot 3:
> > EEPROM format version 1
> > Route/Switch Processor 4, HW rev 1.01, board revision D0
> > Serial number: 11123251 Part number: 73-1689-03
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board; 7500 compatible
> >
> > EEPROM contents (hex):
> > 0x20: 01 1A 01 01 00 A9 BA 33 49 06 99 03 00 00 00 00
> > 0x30: 68 00 00 01 00 00 00 00 00 00 00 00 00 00 00 00
> > Slot 31 (virtual):
> > EEPROM format version 1
> > Chassis Interface, HW rev 1.01, board revision C0
> > Serial number: 12266020 Part number: 73-1306-02
> > Test history: 0x00 RMA number: 00-00-00
> > Flags: cisco 7000 board
> >
> > EEPROM contents (hex):
> > 0x20: 01 10 01 01 00 BB 2A 24 49 05 1A 02 00 00 00 00
> > 0x30: 60 00 00 00 FF 00 00 00 00 00 00 00 00 00 00 00
> >
> >
> > _______________________________________________________
> > Skeeve Stevens, RHCE Email: skeeve at skeeve.org
> > Website: www.skeeve.org - Telephone: (0414) 753 383
> > Address: P.O Box 1035, Epping, NSW, 1710, Australia
> >
> > eIntellego - skeeve at eintellego.net - www.eintellego.net
> > _______________________________________________________
> > I'm a groove licked love child king of the verse
> > Si vis pacem, para bellum
> >
> >
> >
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list