[c-nsp] 7206vxr reloading with software forced crash?
Rodney Dunn
rodunn at cisco.com
Wed Aug 31 16:02:11 EDT 2005
As Bruce said it's memory corruption.
I don't have a 12.2(14)S14 image loaded handy but
see if you can turn on:
scheduler heapcheck process memory processor checktype all
if not you may have to do:
scheduler heapcheck process (this may take a considerable amount
of CPU). That's why we tried to put in the more granular
options in the first command to limit the checks between
context switches. The idea is you catch the memory corruption
earlier so what you see in the crashinfo file is more representative
of the layout exactly when the corruption happens.
You should try turning that on and capturing a couple of crashinfo
files and attach those to the case. That's the best you can
do in order to solve it.
Rodney
On Wed, Aug 31, 2005 at 03:50:24PM -0400, Francisco Rivas wrote:
> Thanks for all the responses off-list (and on-list too) that I've
> received. I've already opened a TAC with all the info, but still I'm
> looking for extra advice :)
>
> This is the actual config of the router, and the "show ver" output. Part
> of the config is trimmed (for safety/personal reasons), so excuse me if
> I have a mistake there.
>
>
> ------------------ show version ------------------
>
> Cisco Internetwork Operating System Software
> IOS (tm) 7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE
> SOFTWARE (fc1)
> Technical Support: http://www.cisco.com/techsupport
> Copyright (c) 1986-2005 by cisco Systems, Inc.
> Compiled Fri 08-Apr-05 13:48 by ccai
> Image text-base: 0x60008BD4, data-base: 0x612C8000
>
> ROM: System Bootstrap, Version 12.0(19990210:195103) [12.0XE 105],
> DEVELOPMENT SOFTWARE
> BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.2(14)S14, RELEASE
> SOFTWARE (fc1)
>
> C7206X2 uptime is 9 minutes
> System returned to ROM by error - a Software forced crash, PC 0x606AED30
> at 14:47:49 UTC Wed Aug 31 2005
> System restarted at 14:49:33 UTC Wed Aug 31 2005
> System image file is "slot0:c7200-p-mz.122-14.S14.bin"
>
> cisco 7206VXR (NPE300) processor (revision D) with 229376K/65536K bytes
> of memory.
> Processor board ID 20393948
> R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2 Cache
> 6 slot VXR midplane, Version 2.0
>
> Last reset from power-on
> X.25 software, Version 3.0.0.
> Bridging software.
> 1 FastEthernet/IEEE 802.3 interface(s)
> 2 Serial network interface(s)
> 125K bytes of non-volatile configuration memory.
>
> 20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
> 4096K bytes of Flash internal SIMM (Sector size 256K).
> Configuration register is 0x2102
>
>
> ------------------ show running-config ------------------
>
>
>
> Building configuration...
>
> Current configuration : 19488 bytes
> !
> ! No configuration change since last restart
> !
> version 12.2
> no service pad
> service tcp-keepalives-in
> service tcp-keepalives-out
> service timestamps debug datetime localtime show-timezone
> service timestamps log datetime localtime show-timezone
> service password-encryption
> no service dhcp
> !
> hostname C7206X2
> !
> boot system slot0:c7200-p-mz.122-14.S14.bin
> boot bootldr slot0:c7200-kboot-mz.122-14.S14.bin
> logging snmp-authfail
> logging queue-limit 100
> no logging console
> aaa new-model
> aaa authentication banner ^CCC
> Unauthorized Access Prohibited!
>
> ^C
> aaa authentication fail-message ^CCC
> Failed Login!
> ^C
> aaa authentication login default group radius local
> aaa authorization exec default group radius local
> aaa accounting commands 7 default start-stop group tacacs+
> aaa accounting commands 15 default start-stop group tacacs+
> enable secret 5 <removed>
> enable password 7 <removed>
> !
> username admin password 7 <removed>
> username operator privilege 7 password 7 <removed>
> ip subnet-zero
> ip flow-cache timeout inactive 300
> ip flow-cache timeout active 1
> ip cef
> !
> !
> !
> no ip domain-lookup
> ip domain-name blah
> ip name-server x.x.x.x
> ip name-server x.x.x.x
> ip name-server x.x.x.x
> !
> no ip bootp server
> ip vrf VRF1
> rd xxxxx:999000016
> route-target export xxxxx:999000016
> route-target import xxxxx:999000016
> !
> ip vrf VRF2
> rd xxxxx:999000010
> route-target export xxxxx:999000010
> route-target import xxxxx:999000010
> !
> ip vrf VRF3
> rd xxxxx:200440004
> export map VRF-INT
> route-target export xxxxx:200440004
> route-target import xxxxx:200440004
> route-target import xxxxx:999000001
> !
> ip vrf VRF4
> rd xxxxx:999002000
> route-target export xxxxx:999002000
> route-target import xxxxx:999002000
> !
> clns routing
> mpls label protocol ldp
> mpls ldp logging neighbor-changes
> mpls traffic-eng tunnels
> no tag-switching ip propagate-ttl
> no tag-switching advertise-tags
> tag-switching advertise-tags for LOOPBACKS
> tag-switching tdp router-id Loopback1
> !
> class-map match-any QoS-OUT-PREMIUM
> match ip precedence 5
> match ip dscp cs5
> match ip dscp ef
> match mpls experimental 5
> class-map match-any QoS-OUT-SILVER
> match ip precedence 3
> match ip dscp cs3
> match ip dscp af31
> match ip dscp af32
> match ip dscp af33
> match mpls experimental 3
> class-map match-any QoS-OUT-GOLD
> match ip precedence 4
> match ip precedence 6
> match ip dscp cs4
> match ip dscp af41
> match ip dscp af42
> match ip dscp af43
> match ip dscp cs6
> match mpls experimental 4
> match mpls experimental 6
> !
> !
> policy-map QoS-OUT-S5/0
> class QoS-OUT-PREMIUM
> priority 13200
> class QoS-OUT-GOLD
> bandwidth 15400
> queue-limit 256
> class QoS-OUT-SILVER
> bandwidth 11000
> queue-limit 128
> class class-default
> fair-queue
> policy-map QoS-OUT-F0/0
> class QoS-OUT-PREMIUM
> priority 30000
> class QoS-OUT-GOLD
> bandwidth 35000
> queue-limit 256
> class QoS-OUT-SILVER
> bandwidth 25000
> queue-limit 128
> class class-default
> fair-queue
> !
> call rsvp-sync
> !
> !
> !
> !
> !
> !
> interface Loopback1
> description Loopback for BGP
> ip address x.x.x.x 255.255.255.255
> ip access-group CONTROL-PLANE in
> ip router isis
> !
> interface FastEthernet0/0
> description 8006x1_Fa1/2_To Backbone
> ip address y.y.y.y 255.255.255.0
> ip access-group CONTROL-PLANE in
> no ip redirects
> no ip unreachables
> no ip proxy-arp
> ip router isis
> rate-limit input access-group 1 1000000 192000 192000 conform-action
> set-prec-transmit 4 exceed-action set-prec-transmit 4
> ip route-cache flow
> ip ospf priority 10
> no ip mroute-cache
> load-interval 30
> duplex full
> mpls netflow egress
> mpls traffic-eng tunnels
> tag-switching mtu 1530
> tag-switching ip
> max-reserved-bandwidth 100
> service-policy output QoS-OUT-F0/0
> ip rsvp bandwidth 1000
> !
> interface Serial6/0
> no ip address
> ip route-cache flow
> load-interval 30
> shutdown
> mpls netflow egress
> mpls traffic-eng tunnels
> tag-switching mtu 1530
> tag-switching ip
> max-reserved-bandwidth 100
> service-policy output QoS-OUT-S5/0
> dsu bandwidth 44210
> framing c-bit
> cablelength 10
> serial restart-delay 0
> ip rsvp bandwidth 1000
> !
> interface Serial6/1
> description DS3 Uplink
> ip address x.x.x.x 255.255.255.252
> ip access-group CONTROL-PLANE in
> no ip redirects
> no ip unreachables
> ip router isis
> ip route-cache flow
> load-interval 30
> mpls netflow egress
> tag-switching mtu 1530
> tag-switching ip
> max-reserved-bandwidth 100
> service-policy output QoS-OUT-S5/0
> dsu bandwidth 44210
> framing c-bit
> cablelength 10
> serial restart-delay 0
> ip rsvp bandwidth 1000
> !
> router ospf 1
> router-id x.x.x.x
> log-adjacency-changes
> redistribute connected subnets
> redistribute static subnets
> network x.x.x.x 0.0.0.255 area 0
> default-information originate always
> !
> router isis
> net xx.xxxxx.xxxxxx.xxxxx
> is-type level-2-only
> metric-style wide level-2
> mpls traffic-eng router-id Loopback1
> mpls traffic-eng level-2
> !
> !
> router bgp XXXXX
> no bgp default ipv4-unicast
> bgp log-neighbor-changes
> neighbor iBGP-RR-CLIENTS peer-group
> neighbor iBGP-RR-CLIENTS remote-as XXXXX
> neighbor iBGP-RR-CLIENTS update-source Loopback1
> neighbor iBGP-US peer-group
> neighbor iBGP-US remote-as XXXXX
> neighbor iBGP-US description Uplink BGP
> neighbor iBGP-US update-source Loopback1
> neighbor iBGP-RR peer-group
> neighbor iBGP-RR remote-as xxxxx
> neighbor iBGP-RR update-source Loopback1
> neighbor eBGP-BACKUP peer-group
> neighbor eBGP-BACKUP remote-as YYYYY
> neighbor eBGP-BACKUP description Backup IP
> neighbor x.x.x.x peer-group iBGP-US
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP
> neighbor x.x.x.x peer-group iBGP
> neighbor x.x.x.x peer-group eBGP-BACKUP
> !
> !
> address-family ipv4
> neighbor iBGP-RR-CLIENTS activate
> neighbor iBGP-RR-CLIENTS route-reflector-client
> neighbor iBGP-RR-CLIENTS next-hop-self
> neighbor iBGP-RR-CLIENTS send-community both
> neighbor iBGP-RR-CLIENTS route-map NO-ROUTES out
> neighbor iBGP-US activate
> neighbor iBGP-US next-hop-self
> neighbor iBGP-US send-community both
> neighbor iBGP-US route-map iBGP-US-IN in
> neighbor iBGP-US route-map iBGP-US-OUT out
> neighbor iBGP-RR activate
> neighbor iBGP-RR route-reflector-client
> neighbor iBGP-RR next-hop-self
> neighbor iBGP-RR send-community both
> neighbor eBGP-BACKUP activate
> neighbor eBGP-BACKUP prefix-list eBGP-OUT out
> neighbor eBGP-BACKUP route-map eBGP-IN in
> neighbor eBGP-BACKUP route-map eBGP-OUT out
> neighbor eBGP-BACKUP maximum-prefix 10
> neighbor x.x.x.x peer-group iBGP-US
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR
> neighbor x.x.x.x peer-group iBGP-RR
> neighbor x.x.x.x peer-group eBGP-BACKUP
> no auto-summary
> no synchronization
> network x.x.x.x route-map ORIGINATE
> network x.x.x.x mask 255.255.224.0 route-map ORIGINATE
> network x.x.x.x mask 255.255.224.0 route-map ORIGINATE
> exit-address-family
> !
> address-family ipv4 multicast
> no auto-summary
> exit-address-family
> !
> address-family vpnv4
> neighbor iBGP-RR-CLIENTS activate
> neighbor iBGP-RR-CLIENTS route-reflector-client
> neighbor iBGP-RR-CLIENTS next-hop-self
> neighbor iBGP-RR-CLIENTS send-community both
> neighbor iBGP-US activate
> neighbor iBGP-US send-community both
> neighbor iBGP-RR activate
> neighbor iBGP-RR route-reflector-client
> neighbor iBGP-RR send-community extended
> neighbor x.x.x.x peer-group iBGP-US
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> neighbor x.x.x.x peer-group iBGP-RR
> neighbor x.x.x.x peer-group iBGP-RR
> no auto-summary
> exit-address-family
> !
> address-family ipv4 vrf VRF3
> no auto-summary
> no synchronization
> exit-address-family
> !
> address-family ipv4 vrf VRF2
> redistribute connected
> redistribute static
> default-information originate
> no auto-summary
> no synchronization
> exit-address-family
> !
> address-family ipv4 vrf VRF1
> redistribute connected
> redistribute static
> no auto-summary
> no synchronization
> exit-address-family
> !
> address-family ipv4 vrf VRF4
> redistribute connected
> redistribute static
> no auto-summary
> no synchronization
> exit-address-family
> !
> ip classless
> <several static routes>
> ip flow-export source Loopback1
> ip flow-export version 5 origin-as
> ip flow-export destination x.x.x.x 2055
> no ip http server
> ip bgp-community new-format
> <several community-lists>
> !
> !
>
> ip prefix-list eBGP-OUT description Deny all prefixes more specific
> than /24
> ip prefix-list eBGP-OUT seq 10 permit 0.0.0.0/0 le 24
> ip prefix-list eBGP-OUT seq 99 deny 0.0.0.0/0 le 32
> !
> !
> <several ip access-lists>
> ip radius source-interface Loopback1
> !
> logging facility local5
> logging source-interface Loopback1
> logging x.x.x.x
> <several access-lists>
> no cdp run
> !
> route-map eBGP-IN permit 10
> set local-preference 200
> set as-path prepend last-as 5
> !
> route-map NO-ROUTES deny 5
> !
> route-map iBGP-US-IN permit 10
> !
> route-map iBGP-US-IN permit 20
> !
> route-map ORIGINATE permit 10
> set local-preference 103
> set community xxxxx:xxx
> !
> route-map iBGP-RR-CLIENTS-OUT deny 10
> !
> route-map C7206x1-OUT permit 5
> !
> route-map NO_EXPORT permit 10
> set community no-export
> !
> route-map eBGP-OUT permit 10
> match community XXXXXX
> set as-path prepend xxxxx xxxxx xxxxx
> !
> route-map iBGP-US permit 10
> match community xxxxxx xxxxxxx
> !
> route-map iBGP-OUT deny 20
> !
> !
> snmp-server community <removed> RO 99
> snmp-server community <removed> RW 99
> snmp-server community <removed> RW
> snmp-server community <removed> RO
> snmp-server ifindex persist
> snmp-server trap-source Loopback1
> snmp-server location IDC
> snmp-server contact NOC
> snmp-server chassis-id xxxxxxx
> snmp-server enable traps snmp linkdown linkup coldstart warmstart
> snmp-server enable traps tty
> snmp-server enable traps atm pvc
> snmp-server enable traps envmon fan shutdown supply temperature
> snmp-server enable traps bgp
> snmp-server enable traps frame-relay
> snmp-server enable traps mpls ldp session-up session-down
> !
> tftp-server slot0:c7200-k91p-mz.122-18.S4.bin
> tacacs-server host x.x.x.x key <removed>
> tacacs-server directed-request
> radius-server host x.x.x.x auth-port 1645 acct-port 1646
> radius-server key 7 <removed>
> !
> dial-peer cor custom
> !
> !
> !
> !
> line con 0
> stopbits 1
> line aux 0
> modem InOut
> no exec
> transport input all
> stopbits 1
> line vty 0 4
> access-class ACCESO-TELNET in
> exec-timeout 5 0
> password 7 <removed>
> transport preferred none
> line vty 5 15
> access-class ACCESO-TELNET in
> exec-timeout 5 0
> transport preferred none
> !
> !
> exception crashinfo file slot0:crashinfo
> scheduler allocate 4000 200
> ntp clock-period 17179995
> ntp update-calendar
> ntp server x.x.x.x
> end
>
>
> Thanks to all for the help.
>
>
> Regards,
>
>
> Francisco Rivas Catalan
> Senior Network Engineer
> IFX Networks
> (56) 2 3744574
> francisco.rivas at ifxnw.cl
>
>
>
>
> On Wed, 2005-08-31 at 12:33 -0400, Francisco Rivas wrote:
> > Hi all,
> >
> > I've got a serious problem with one of my 7206vxr. It's reloading itself
> > several times, with a "Software forced crash". I've seen this behavior
> > before, and it won't happen too often, but this router in particular is
> > reloading at least twice a day. It doesn't have too much traffic (just
> > 30-40Mb), and the CPU load is < 50%.
> > But, the main worry that I have about this, is that I've replaced the
> > chassis ENTIRELY, and the new chassis is giving me the same error. Look:
> >
> >
> > System was restarted by error - a Software forced crash, PC 0x606AED30
> > at 14:47:49 UTC Wed Aug 31 2005
> > 7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE SOFTWARE (fc1)
> > Technical Support: http://www.cisco.com/techsupport
> > Compiled Fri 08-Apr-05 13:48 by ccai
> > Image text-base: 0x60008BD4, data-base: 0x612C8000
> >
> >
> > This is the output of "sh region address 0x606AED30" on that chassis:
> >
> > router#show region address 0x606AED30
> > Address 0x606AED30 is located physically in :
> >
> > Name : text
> > Class : IText
> > Media : R/O
> > Start : 0x60008BD4
> > End : 0x612C7413
> > Size : 0x012BE840
> >
> >
> >
> > The old chassis was giving a similar error, not the same address, but it
> > was on the IText class too.
> > I've tried several IOS versions (12.2(14)S3, 12.2(14)S14, 12.2(18)S4)
> > and all of them are crashing with a similar error. Always a software
> > forced crash, always the same region address in the error (the region
> > varies from IOS to IOS, but with the same IOS, is the same region in
> > every crash), and always is a region on the IText class. And this is
> > with 2 totally different chassis (both with NPE300 / 256Mb RAM / 6 port
> > VXR Chassis).
> >
> >
> > The last part of the crashdump, just before the %Software-forced reload,
> > is this:
> >
> >
> > Aug 31 14:47:46 UTC: %SYS-3-OVERRUN: Block overrun at 20014AB8 (red zone
> > 1F860000)
> > -Traceback= 60692A68 60695A04 606976D0 60697920
> > Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> > 6770DFB4,6000011A 6770DB88,60C14048 6770DB88,60C0DED8 6770DB88,40000202
> > 69C13E0C,60E22530 64C26254,60E27F30 65360AA0,60E37E9C 69C13E0C,60E1F41C
> > Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> > 69C13E0C,3000000E 6770DFB4,6000011A 6770DB88,60C12734 6770DB88,60C0DED8
> > 6770DB88,40000202 6770DFB4,6000011A 6770DB88,60C19E98 6601376C,60C25CA0
> > Aug 31 14:47:46 UTC: %SYS-6-BLKINFO: Corrupted redzone blk 20014AB8,
> > words 860, alloc 6063AAB8, InUse, dealloc 0, rfcnt 1
> > -Traceback= 60690550 60692A7C 60695A04 606976D0 60697920
> > Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AB8: 0xAB1234CD 0xFFFFFFFE
> > 0x0 0x61584DD8
> > Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AC8: 0x6063AAB8 0x20015198
> > 0x200143EC 0x8000035C
> > Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AD8: 0x1 0x0 0x0 0x0
> >
> >
> > Anyone can give me a hand on this?? I've already filtered ALL the
> > traffic going right to the IP address(es) of the router, but this is
> > still happening. I have full crashlogs and "sh stacks" and "sh tech"
> > available from the last crashes, so if anyone needs more info, I can
> > provide that without problems.
> > There are no changes on the config of the router since a couple of
> > months, so I can't blame that... anyone have any idea about this?
> >
>
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list