[c-nsp] 7206vxr reloading with software forced crash?
Francisco Rivas
frivas at lanparty.cl
Wed Aug 31 15:50:24 EDT 2005
Thanks for all the responses off-list (and on-list too) that I've
received. I've already opened a TAC with all the info, but still I'm
looking for extra advice :)
This is the actual config of the router, and the "show ver" output. Part
of the config is trimmed (for safety/personal reasons), so excuse me if
I have a mistake there.
------------------ show version ------------------
Cisco Internetwork Operating System Software
IOS (tm) 7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE
SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by cisco Systems, Inc.
Compiled Fri 08-Apr-05 13:48 by ccai
Image text-base: 0x60008BD4, data-base: 0x612C8000
ROM: System Bootstrap, Version 12.0(19990210:195103) [12.0XE 105],
DEVELOPMENT SOFTWARE
BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.2(14)S14, RELEASE
SOFTWARE (fc1)
C7206X2 uptime is 9 minutes
System returned to ROM by error - a Software forced crash, PC 0x606AED30
at 14:47:49 UTC Wed Aug 31 2005
System restarted at 14:49:33 UTC Wed Aug 31 2005
System image file is "slot0:c7200-p-mz.122-14.S14.bin"
cisco 7206VXR (NPE300) processor (revision D) with 229376K/65536K bytes
of memory.
Processor board ID 20393948
R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2 Cache
6 slot VXR midplane, Version 2.0
Last reset from power-on
X.25 software, Version 3.0.0.
Bridging software.
1 FastEthernet/IEEE 802.3 interface(s)
2 Serial network interface(s)
125K bytes of non-volatile configuration memory.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
4096K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x2102
------------------ show running-config ------------------
Building configuration...
Current configuration : 19488 bytes
!
! No configuration change since last restart
!
version 12.2
no service pad
service tcp-keepalives-in
service tcp-keepalives-out
service timestamps debug datetime localtime show-timezone
service timestamps log datetime localtime show-timezone
service password-encryption
no service dhcp
!
hostname C7206X2
!
boot system slot0:c7200-p-mz.122-14.S14.bin
boot bootldr slot0:c7200-kboot-mz.122-14.S14.bin
logging snmp-authfail
logging queue-limit 100
no logging console
aaa new-model
aaa authentication banner ^CCC
Unauthorized Access Prohibited!
^C
aaa authentication fail-message ^CCC
Failed Login!
^C
aaa authentication login default group radius local
aaa authorization exec default group radius local
aaa accounting commands 7 default start-stop group tacacs+
aaa accounting commands 15 default start-stop group tacacs+
enable secret 5 <removed>
enable password 7 <removed>
!
username admin password 7 <removed>
username operator privilege 7 password 7 <removed>
ip subnet-zero
ip flow-cache timeout inactive 300
ip flow-cache timeout active 1
ip cef
!
!
!
no ip domain-lookup
ip domain-name blah
ip name-server x.x.x.x
ip name-server x.x.x.x
ip name-server x.x.x.x
!
no ip bootp server
ip vrf VRF1
rd xxxxx:999000016
route-target export xxxxx:999000016
route-target import xxxxx:999000016
!
ip vrf VRF2
rd xxxxx:999000010
route-target export xxxxx:999000010
route-target import xxxxx:999000010
!
ip vrf VRF3
rd xxxxx:200440004
export map VRF-INT
route-target export xxxxx:200440004
route-target import xxxxx:200440004
route-target import xxxxx:999000001
!
ip vrf VRF4
rd xxxxx:999002000
route-target export xxxxx:999002000
route-target import xxxxx:999002000
!
clns routing
mpls label protocol ldp
mpls ldp logging neighbor-changes
mpls traffic-eng tunnels
no tag-switching ip propagate-ttl
no tag-switching advertise-tags
tag-switching advertise-tags for LOOPBACKS
tag-switching tdp router-id Loopback1
!
class-map match-any QoS-OUT-PREMIUM
match ip precedence 5
match ip dscp cs5
match ip dscp ef
match mpls experimental 5
class-map match-any QoS-OUT-SILVER
match ip precedence 3
match ip dscp cs3
match ip dscp af31
match ip dscp af32
match ip dscp af33
match mpls experimental 3
class-map match-any QoS-OUT-GOLD
match ip precedence 4
match ip precedence 6
match ip dscp cs4
match ip dscp af41
match ip dscp af42
match ip dscp af43
match ip dscp cs6
match mpls experimental 4
match mpls experimental 6
!
!
policy-map QoS-OUT-S5/0
class QoS-OUT-PREMIUM
priority 13200
class QoS-OUT-GOLD
bandwidth 15400
queue-limit 256
class QoS-OUT-SILVER
bandwidth 11000
queue-limit 128
class class-default
fair-queue
policy-map QoS-OUT-F0/0
class QoS-OUT-PREMIUM
priority 30000
class QoS-OUT-GOLD
bandwidth 35000
queue-limit 256
class QoS-OUT-SILVER
bandwidth 25000
queue-limit 128
class class-default
fair-queue
!
call rsvp-sync
!
!
!
!
!
!
interface Loopback1
description Loopback for BGP
ip address x.x.x.x 255.255.255.255
ip access-group CONTROL-PLANE in
ip router isis
!
interface FastEthernet0/0
description 8006x1_Fa1/2_To Backbone
ip address y.y.y.y 255.255.255.0
ip access-group CONTROL-PLANE in
no ip redirects
no ip unreachables
no ip proxy-arp
ip router isis
rate-limit input access-group 1 1000000 192000 192000 conform-action
set-prec-transmit 4 exceed-action set-prec-transmit 4
ip route-cache flow
ip ospf priority 10
no ip mroute-cache
load-interval 30
duplex full
mpls netflow egress
mpls traffic-eng tunnels
tag-switching mtu 1530
tag-switching ip
max-reserved-bandwidth 100
service-policy output QoS-OUT-F0/0
ip rsvp bandwidth 1000
!
interface Serial6/0
no ip address
ip route-cache flow
load-interval 30
shutdown
mpls netflow egress
mpls traffic-eng tunnels
tag-switching mtu 1530
tag-switching ip
max-reserved-bandwidth 100
service-policy output QoS-OUT-S5/0
dsu bandwidth 44210
framing c-bit
cablelength 10
serial restart-delay 0
ip rsvp bandwidth 1000
!
interface Serial6/1
description DS3 Uplink
ip address x.x.x.x 255.255.255.252
ip access-group CONTROL-PLANE in
no ip redirects
no ip unreachables
ip router isis
ip route-cache flow
load-interval 30
mpls netflow egress
tag-switching mtu 1530
tag-switching ip
max-reserved-bandwidth 100
service-policy output QoS-OUT-S5/0
dsu bandwidth 44210
framing c-bit
cablelength 10
serial restart-delay 0
ip rsvp bandwidth 1000
!
router ospf 1
router-id x.x.x.x
log-adjacency-changes
redistribute connected subnets
redistribute static subnets
network x.x.x.x 0.0.0.255 area 0
default-information originate always
!
router isis
net xx.xxxxx.xxxxxx.xxxxx
is-type level-2-only
metric-style wide level-2
mpls traffic-eng router-id Loopback1
mpls traffic-eng level-2
!
!
router bgp XXXXX
no bgp default ipv4-unicast
bgp log-neighbor-changes
neighbor iBGP-RR-CLIENTS peer-group
neighbor iBGP-RR-CLIENTS remote-as XXXXX
neighbor iBGP-RR-CLIENTS update-source Loopback1
neighbor iBGP-US peer-group
neighbor iBGP-US remote-as XXXXX
neighbor iBGP-US description Uplink BGP
neighbor iBGP-US update-source Loopback1
neighbor iBGP-RR peer-group
neighbor iBGP-RR remote-as xxxxx
neighbor iBGP-RR update-source Loopback1
neighbor eBGP-BACKUP peer-group
neighbor eBGP-BACKUP remote-as YYYYY
neighbor eBGP-BACKUP description Backup IP
neighbor x.x.x.x peer-group iBGP-US
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP
neighbor x.x.x.x peer-group iBGP
neighbor x.x.x.x peer-group eBGP-BACKUP
!
!
address-family ipv4
neighbor iBGP-RR-CLIENTS activate
neighbor iBGP-RR-CLIENTS route-reflector-client
neighbor iBGP-RR-CLIENTS next-hop-self
neighbor iBGP-RR-CLIENTS send-community both
neighbor iBGP-RR-CLIENTS route-map NO-ROUTES out
neighbor iBGP-US activate
neighbor iBGP-US next-hop-self
neighbor iBGP-US send-community both
neighbor iBGP-US route-map iBGP-US-IN in
neighbor iBGP-US route-map iBGP-US-OUT out
neighbor iBGP-RR activate
neighbor iBGP-RR route-reflector-client
neighbor iBGP-RR next-hop-self
neighbor iBGP-RR send-community both
neighbor eBGP-BACKUP activate
neighbor eBGP-BACKUP prefix-list eBGP-OUT out
neighbor eBGP-BACKUP route-map eBGP-IN in
neighbor eBGP-BACKUP route-map eBGP-OUT out
neighbor eBGP-BACKUP maximum-prefix 10
neighbor x.x.x.x peer-group iBGP-US
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR
neighbor x.x.x.x peer-group iBGP-RR
neighbor x.x.x.x peer-group eBGP-BACKUP
no auto-summary
no synchronization
network x.x.x.x route-map ORIGINATE
network x.x.x.x mask 255.255.224.0 route-map ORIGINATE
network x.x.x.x mask 255.255.224.0 route-map ORIGINATE
exit-address-family
!
address-family ipv4 multicast
no auto-summary
exit-address-family
!
address-family vpnv4
neighbor iBGP-RR-CLIENTS activate
neighbor iBGP-RR-CLIENTS route-reflector-client
neighbor iBGP-RR-CLIENTS next-hop-self
neighbor iBGP-RR-CLIENTS send-community both
neighbor iBGP-US activate
neighbor iBGP-US send-community both
neighbor iBGP-RR activate
neighbor iBGP-RR route-reflector-client
neighbor iBGP-RR send-community extended
neighbor x.x.x.x peer-group iBGP-US
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
neighbor x.x.x.x peer-group iBGP-RR
neighbor x.x.x.x peer-group iBGP-RR
no auto-summary
exit-address-family
!
address-family ipv4 vrf VRF3
no auto-summary
no synchronization
exit-address-family
!
address-family ipv4 vrf VRF2
redistribute connected
redistribute static
default-information originate
no auto-summary
no synchronization
exit-address-family
!
address-family ipv4 vrf VRF1
redistribute connected
redistribute static
no auto-summary
no synchronization
exit-address-family
!
address-family ipv4 vrf VRF4
redistribute connected
redistribute static
no auto-summary
no synchronization
exit-address-family
!
ip classless
<several static routes>
ip flow-export source Loopback1
ip flow-export version 5 origin-as
ip flow-export destination x.x.x.x 2055
no ip http server
ip bgp-community new-format
<several community-lists>
!
!
ip prefix-list eBGP-OUT description Deny all prefixes more specific
than /24
ip prefix-list eBGP-OUT seq 10 permit 0.0.0.0/0 le 24
ip prefix-list eBGP-OUT seq 99 deny 0.0.0.0/0 le 32
!
!
<several ip access-lists>
ip radius source-interface Loopback1
!
logging facility local5
logging source-interface Loopback1
logging x.x.x.x
<several access-lists>
no cdp run
!
route-map eBGP-IN permit 10
set local-preference 200
set as-path prepend last-as 5
!
route-map NO-ROUTES deny 5
!
route-map iBGP-US-IN permit 10
!
route-map iBGP-US-IN permit 20
!
route-map ORIGINATE permit 10
set local-preference 103
set community xxxxx:xxx
!
route-map iBGP-RR-CLIENTS-OUT deny 10
!
route-map C7206x1-OUT permit 5
!
route-map NO_EXPORT permit 10
set community no-export
!
route-map eBGP-OUT permit 10
match community XXXXXX
set as-path prepend xxxxx xxxxx xxxxx
!
route-map iBGP-US permit 10
match community xxxxxx xxxxxxx
!
route-map iBGP-OUT deny 20
!
!
snmp-server community <removed> RO 99
snmp-server community <removed> RW 99
snmp-server community <removed> RW
snmp-server community <removed> RO
snmp-server ifindex persist
snmp-server trap-source Loopback1
snmp-server location IDC
snmp-server contact NOC
snmp-server chassis-id xxxxxxx
snmp-server enable traps snmp linkdown linkup coldstart warmstart
snmp-server enable traps tty
snmp-server enable traps atm pvc
snmp-server enable traps envmon fan shutdown supply temperature
snmp-server enable traps bgp
snmp-server enable traps frame-relay
snmp-server enable traps mpls ldp session-up session-down
!
tftp-server slot0:c7200-k91p-mz.122-18.S4.bin
tacacs-server host x.x.x.x key <removed>
tacacs-server directed-request
radius-server host x.x.x.x auth-port 1645 acct-port 1646
radius-server key 7 <removed>
!
dial-peer cor custom
!
!
!
!
line con 0
stopbits 1
line aux 0
modem InOut
no exec
transport input all
stopbits 1
line vty 0 4
access-class ACCESO-TELNET in
exec-timeout 5 0
password 7 <removed>
transport preferred none
line vty 5 15
access-class ACCESO-TELNET in
exec-timeout 5 0
transport preferred none
!
!
exception crashinfo file slot0:crashinfo
scheduler allocate 4000 200
ntp clock-period 17179995
ntp update-calendar
ntp server x.x.x.x
end
Thanks to all for the help.
Regards,
Francisco Rivas Catalan
Senior Network Engineer
IFX Networks
(56) 2 3744574
francisco.rivas at ifxnw.cl
On Wed, 2005-08-31 at 12:33 -0400, Francisco Rivas wrote:
> Hi all,
>
> I've got a serious problem with one of my 7206vxr. It's reloading itself
> several times, with a "Software forced crash". I've seen this behavior
> before, and it won't happen too often, but this router in particular is
> reloading at least twice a day. It doesn't have too much traffic (just
> 30-40Mb), and the CPU load is < 50%.
> But, the main worry that I have about this, is that I've replaced the
> chassis ENTIRELY, and the new chassis is giving me the same error. Look:
>
>
> System was restarted by error - a Software forced crash, PC 0x606AED30
> at 14:47:49 UTC Wed Aug 31 2005
> 7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE SOFTWARE (fc1)
> Technical Support: http://www.cisco.com/techsupport
> Compiled Fri 08-Apr-05 13:48 by ccai
> Image text-base: 0x60008BD4, data-base: 0x612C8000
>
>
> This is the output of "sh region address 0x606AED30" on that chassis:
>
> router#show region address 0x606AED30
> Address 0x606AED30 is located physically in :
>
> Name : text
> Class : IText
> Media : R/O
> Start : 0x60008BD4
> End : 0x612C7413
> Size : 0x012BE840
>
>
>
> The old chassis was giving a similar error, not the same address, but it
> was on the IText class too.
> I've tried several IOS versions (12.2(14)S3, 12.2(14)S14, 12.2(18)S4)
> and all of them are crashing with a similar error. Always a software
> forced crash, always the same region address in the error (the region
> varies from IOS to IOS, but with the same IOS, is the same region in
> every crash), and always is a region on the IText class. And this is
> with 2 totally different chassis (both with NPE300 / 256Mb RAM / 6 port
> VXR Chassis).
>
>
> The last part of the crashdump, just before the %Software-forced reload,
> is this:
>
>
> Aug 31 14:47:46 UTC: %SYS-3-OVERRUN: Block overrun at 20014AB8 (red zone
> 1F860000)
> -Traceback= 60692A68 60695A04 606976D0 60697920
> Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> 6770DFB4,6000011A 6770DB88,60C14048 6770DB88,60C0DED8 6770DB88,40000202
> 69C13E0C,60E22530 64C26254,60E27F30 65360AA0,60E37E9C 69C13E0C,60E1F41C
> Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> 69C13E0C,3000000E 6770DFB4,6000011A 6770DB88,60C12734 6770DB88,60C0DED8
> 6770DB88,40000202 6770DFB4,6000011A 6770DB88,60C19E98 6601376C,60C25CA0
> Aug 31 14:47:46 UTC: %SYS-6-BLKINFO: Corrupted redzone blk 20014AB8,
> words 860, alloc 6063AAB8, InUse, dealloc 0, rfcnt 1
> -Traceback= 60690550 60692A7C 60695A04 606976D0 60697920
> Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AB8: 0xAB1234CD 0xFFFFFFFE
> 0x0 0x61584DD8
> Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AC8: 0x6063AAB8 0x20015198
> 0x200143EC 0x8000035C
> Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AD8: 0x1 0x0 0x0 0x0
>
>
> Anyone can give me a hand on this?? I've already filtered ALL the
> traffic going right to the IP address(es) of the router, but this is
> still happening. I have full crashlogs and "sh stacks" and "sh tech"
> available from the last crashes, so if anyone needs more info, I can
> provide that without problems.
> There are no changes on the config of the router since a couple of
> months, so I can't blame that... anyone have any idea about this?
>
More information about the cisco-nsp
mailing list