[c-nsp] problem with 7609
Piotr Chytla
pch at packetconsulting.pl
Wed Aug 4 03:40:02 EDT 2010
Hi,
Experts , I need advise , I've problem with my 7609-S with IOS 12.3(33) SRB1,
after applying 'ip policy route-map' to TenGig interfece, my RSP720 crashed
Hardware details :
Mod Ports Card Type Model Serial No.
--- ----- -------------------------------------- ------------------ -----------
1 0 4-subslot SPA Interface Processor-400 7600-SIP-400 JAEXXXXXXXX
2 4 CEF720 4 port 10-Gigabit Ethernet WS-X6704-10GE SALXXXXXXXX
3 48 CEF720 48 port 1000mb SFP WS-X6748-SFP SALXXXXXXXX
4 0 4-subslot SPA Interface Processor-400 7600-SIP-400 JAEXXXXXXXX
5 2 Route Switch Processor 720 (Active) RSP720-3CXL-GE JAEXXXXXXXX
7 48 CEF720 48 port 10/100/1000mb Ethernet WS-X6748-GE-TX SALXXXXXXXX
8 0 4-subslot SPA Interface Processor-400 7600-SIP-400 JAEXXXXXXXX
9 0 4-subslot SPA Interface Processor-400 7600-SIP-400 JAEXXXXXXXX
All four SIP-400 are disabled - 'PwrDown' state.
Interface are nothing fancy :
interface TenGigabitEthernet2/1
ip address X.X.248.202 255.255.255.252
no ip proxy-arp
no ip redirects
no ip unreachables
logging event link-status
end
And route-map ;
route-map RTR01-NH, permit, sequence 10
Match clauses:
ip address (access-lists): 102
Set clauses:
ip next-hop verify-availability X.X.32.34 10 track 1 [up]
Policy routing matches: 0 packets, 0 bytes
Extended IP access list 102
10 permit ip any X.X.120.0 0.0.0.127
Traffic with destination IP in access-list 102 , goes to other gateway . After applying
'ip policy route-map RTR01-NH' to Te2/1 RSP crashed . After switchover backup RSP crashed
after EARL tried to recover hardware problem .
Second crash :
Cisco IOS Software, c7600rsp72043_rp Software (c7600rsp72043_rp-ADVIPSERVICESK9-M), Version 12.2(33)SR
C2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Thu 18-Sep-08 03:16 by prod_rel_team
*Jul 20 15:56:26.628: %SSH-5-ENABLED: SSH 2.0 has been enabled
*Jul 20 13:55:44.575: %SYS-SP-3-LOGGER_FLUSHED: System was paused for 00:00:00 to ensure console debug
ging output.
Firmware compiled 07-Jul-08 00:53 by integ Build [100]
*Jul 20 13:56:16.711: %SPANTREE-SP-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jul 20 13:56:16.903: SP: SP: Currently running ROMMON from S (Gold) region
*Jul 20 13:56:21.902: %C7600_PWR-SP-4-PSCOMBINEDMODE: power supplies set to combined mode.
*Jul 20 13:56:26.448: %SYS-SP-5-RESTART: System restarted --
Cisco IOS Software, c7600rsp72043_sp Software (c7600rsp72043_sp-ADVIPSERVICESK9-M), Version 12.2(33)SR
C2, RELEASE SOFTWARE (fc2)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2008 by Cisco Systems, Inc.
Compiled Thu 18-Sep-08 03:46 by prod_rel_team
*Jul 20 13:56:27.504: %OIR-SP-6-INSPS: Power supply inserted in slot 1
*Jul 20 13:56:27.508: %C7600_PWR-SP-4-PSOK: power supply 1 turned on.
*Jul 20 13:56:27.568: %OIR-SP-6-INSPS: Power supply inserted in slot 2
*Jul 20 13:56:27.572: %C7600_PWR-SP-4-PSOK: power supply 2 turned on.
*Jul 20 13:56:30.160: %C7600_PWR-SP-4-DISABLED: power to module in slot 1 set off (admin request)
*Jul 20 13:56:31.200: %C7600_PWR-SP-4-DISABLED: power to module in slot 4 set off (admin request)
*Jul 20 15:56:34.884: %DIAG-SP-6-RUN_MINIMUM: Module 5: Running Minimal Diagnostics...
*Jul 20 15:56:39.652: %OIR-6-REMCARD: Card removed from slot 1, interfaces disabled
*Jul 20 15:56:39.660: %SPA_OIR-6-OFFLINECARD: SPA (SPA-1XOC48POS/RPR) offline in subslot 1/0
*Jul 20 15:56:39.660: %OIR-6-REMCARD: Card removed from slot 4, interfaces disabled
*Jul 20 15:56:39.664: %SPA_OIR-6-OFFLINECARD: SPA (SPA-1XOC48POS/RPR) offline in subslot 4/0
*Jul 20 15:56:39.664: %OIR-6-REMCARD: Card removed from slot 8, interfaces disabled
*Jul 20 15:56:39.668: %SPA_OIR-6-OFFLINECARD: SPA (SPA-1XOC48POS/RPR) offline in subslot 8/0
*Jul 20 15:56:39.668: %OIR-6-REMCARD: Card removed from slot 9, interfaces disabled
*Jul 20 15:56:39.668: %SPA_OIR-6-OFFLINECARD: SPA (SPA-1XOC48POS/RPR) offline in subslot 9/0
*Jul 20 15:56:39.312: %DIAG-SP-6-DIAG_OK: Module 5: Passed Online Diagnostics
*Jul 20 15:56:40.088: %OIR-SP-6-INSCARD: Card inserted in slot 5, interfaces are now online
*Jul 20 15:56:57.504: %PFREDUN-SP-6-ACTIVE: Standby initializing for SSO mode
[..]
*Jul 20 15:57:31.817: %FABRIC-SP-5-CLEAR_BLOCK: Clear block option is off for the fabric in slot 6.
*Jul 20 15:57:31.901: %FABRIC-SP-5-FABRIC_MODULE_BACKUP: The Switch Fabric Module in slot 6 became
+standby
*Jul 20 15:57:32.653: %DIAG-SP-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
*Jul 20 15:57:33.173: %DIAG-SP-6-DIAG_OK: Module 6: Passed Online Diagnostics
*Jul 20 15:57:34.593: %OIR-SP-6-INSCARD: Card inserted in slot 6, interfaces are now online
*Jul 20 15:57:58.997: %DIAG-SP-6-RUN_MINIMUM: Module 7: Running Minimal Diagnostics...
*Jul 20 15:58:07.981: %DIAG-SP-6-RUN_MINIMUM: Module 2: Running Minimal Diagnostics...
*Jul 20 15:58:19.982: %DIAG-SP-6-RUN_MINIMUM: Module 3: Running Minimal Diagnostics...
*Jul 20 15:58:28.322: %LINK-3-UPDOWN: Interface GigabitEthernet7/2, changed state to down
*Jul 20 15:58:27.358: %DIAG-SP-6-DIAG_OK: Module 7: Passed Online Diagnostics
*Jul 20 15:58:29.522: %LINK-3-UPDOWN: Interface GigabitEthernet7/5, changed state to down
*Jul 20 13:57:23.760: %SPANTREE-SP-STDBY-5-EXTENDED_SYSID: Extended SysId enabled for type vlan
*Jul 20 13:57:23.952: SP-STDBY: SP: Currently running ROMMON from S (Gold) region
*Jul 20 13:57:30.688: %DIAG-SP-STDBY-6-RUN_MINIMUM: Module 6: Running Minimal Diagnostics...
*Jul 20 13:57:31.600: %DIAG-SP-STDBY-6-DIAG_OK: Module 6: Passed Online Diagnostics
[..]
*Jul 20 15:58:41.254: %HA_CONFIG_SYNC-6-BULK_CFGSYNC_SUCCEED: Bulk Sync succeeded
*Jul 20 15:58:40.454: %OIR-SP-6-INSCARD: Card inserted in slot 7, interfaces are now online
*Jul 20 15:58:41.602: %EARL-SP-2-SWITCH_BUS_IDLE: Switching bus is idle for 2 seconds. The card grant
is 5
*Jul 20 15:58:41.078: %PFREDUN-SP-STDBY-6-STANDBY: Ready for SSO mode
*Jul 20 15:58:46.450: %EARL-SP-2-SWITCH_BUS_IDLE: Switching bus is idle for 2 seconds. The card grant
is 5
*Jul 20 15:58:48.762: %EARL-SP-2-SWITCH_BUS_IDLE: Switching bus is idle for 4 seconds. The card grant
is 5
*Jul 20 15:58:51.074: %EARL-SP-2-SWITCH_BUS_IDLE: Switching bus is idle for 6 seconds. The card grant
is 5
[..]
*Jul 20 15:59:05.134: %C7600_PWR-SP-4-DISABLED: power to module in slot 6 set off (Switching Bus Idle)
*Jul 20 15:59:05.142: %EARL-SP-2-SWITCH_BUS_IDLE: Switching bus is idle for 18 seconds. The card grant
is 5
*Jul 20 15:59:05.158: %CONST_ISSU-SP-3-CONST_MTU_NOT_ENOUGH: ISSU Cat6k LTL Client(6015): Requested bu
ffer size (1292) is greater than the max MTU size (0)
*Jul 20 15:59:05.158: %CONST_ISSU-SP-3-CONST_MTU_NOT_ENOUGH: ISSU Cat6k LTL Client(6015): Requested bu
[..]
*Jul 20 15:59:05.466: %CONST_ISSU-SP-3-CONST_MTU_NOT_ENOUGH: ISSU Cat6k LTL Client(6015): Requested bu
ffer size (172) is greater than the max MTU size (0)
*Jul 20 15:59:07.482: %CPU_MONITOR-3-PEER_EXCEPTION: CPU_MONITOR peer has failed due to exception , re
set by [5/0] buffer size (1292) is greater than the max MTU size (0)
I've temporaly solved this by removing backup RSP720 ( Mod 6 ) , I suppose that backup RSP is broken
or have some hardware problem but first crash was after applying 'ip policy ' . There is some known
problem with policy routing on 7609 with SRB1? If someone can take a look to my crashinfo files
I can send it personaly.
/pch
--
Dyslexia bug unpatched since 1977 ...
exploit has been leaked to the underground.
More information about the cisco-nsp
mailing list