[c-nsp] 7206vxr reloading with software forced crash?
Francisco Rivas
frivas at lanparty.cl
Thu Sep 1 15:35:20 EDT 2005
Well, my router has crashed again. I've got the crashinfo file, and I've
uploaded it to the TAC. If anyone want to take a look at the file,
please let me know.
thanks a lot!!
Francisco Rivas C.
On Wed, 2005-08-31 at 16:53 -0400, Francisco Rivas wrote:
> Thanks for the response Rodney. I've added the "scheduler heapcheck
> process memory processor checktype all" to the config, so for now I'm
> waiting for the next crash...
>
>
> regards,
>
> Francisco Rivas C.
>
>
>
> On Wed, 2005-08-31 at 16:02 -0400, Rodney Dunn wrote:
> > As Bruce said it's memory corruption.
> >
> > I don't have a 12.2(14)S14 image loaded handy but
> > see if you can turn on:
> >
> > scheduler heapcheck process memory processor checktype all
> >
> > if not you may have to do:
> >
> > scheduler heapcheck process (this may take a considerable amount
> > of CPU). That's why we tried to put in the more granular
> > options in the first command to limit the checks between
> > context switches. The idea is you catch the memory corruption
> > earlier so what you see in the crashinfo file is more representative
> > of the layout exactly when the corruption happens.
> >
> > You should try turning that on and capturing a couple of crashinfo
> > files and attach those to the case. That's the best you can
> > do in order to solve it.
> >
> > Rodney
> >
> >
> >
> >
> > On Wed, Aug 31, 2005 at 03:50:24PM -0400, Francisco Rivas wrote:
> > > Thanks for all the responses off-list (and on-list too) that I've
> > > received. I've already opened a TAC with all the info, but still I'm
> > > looking for extra advice :)
> > >
> > > This is the actual config of the router, and the "show ver" output. Part
> > > of the config is trimmed (for safety/personal reasons), so excuse me if
> > > I have a mistake there.
> > >
> > >
> > > ------------------ show version ------------------
> > >
> > > Cisco Internetwork Operating System Software
> > > IOS (tm) 7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE
> > > SOFTWARE (fc1)
> > > Technical Support: http://www.cisco.com/techsupport
> > > Copyright (c) 1986-2005 by cisco Systems, Inc.
> > > Compiled Fri 08-Apr-05 13:48 by ccai
> > > Image text-base: 0x60008BD4, data-base: 0x612C8000
> > >
> > > ROM: System Bootstrap, Version 12.0(19990210:195103) [12.0XE 105],
> > > DEVELOPMENT SOFTWARE
> > > BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.2(14)S14, RELEASE
> > > SOFTWARE (fc1)
> > >
> > > C7206X2 uptime is 9 minutes
> > > System returned to ROM by error - a Software forced crash, PC 0x606AED30
> > > at 14:47:49 UTC Wed Aug 31 2005
> > > System restarted at 14:49:33 UTC Wed Aug 31 2005
> > > System image file is "slot0:c7200-p-mz.122-14.S14.bin"
> > >
> > > cisco 7206VXR (NPE300) processor (revision D) with 229376K/65536K bytes
> > > of memory.
> > > Processor board ID 20393948
> > > R7000 CPU at 262Mhz, Implementation 39, Rev 2.1, 256KB L2 Cache
> > > 6 slot VXR midplane, Version 2.0
> > >
> > > Last reset from power-on
> > > X.25 software, Version 3.0.0.
> > > Bridging software.
> > > 1 FastEthernet/IEEE 802.3 interface(s)
> > > 2 Serial network interface(s)
> > > 125K bytes of non-volatile configuration memory.
> > >
> > > 20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
> > > 4096K bytes of Flash internal SIMM (Sector size 256K).
> > > Configuration register is 0x2102
> > >
> > >
> > > ------------------ show running-config ------------------
> > >
> > >
> > >
> > > Building configuration...
> > >
> > > Current configuration : 19488 bytes
> > > !
> > > ! No configuration change since last restart
> > > !
> > > version 12.2
> > > no service pad
> > > service tcp-keepalives-in
> > > service tcp-keepalives-out
> > > service timestamps debug datetime localtime show-timezone
> > > service timestamps log datetime localtime show-timezone
> > > service password-encryption
> > > no service dhcp
> > > !
> > > hostname C7206X2
> > > !
> > > boot system slot0:c7200-p-mz.122-14.S14.bin
> > > boot bootldr slot0:c7200-kboot-mz.122-14.S14.bin
> > > logging snmp-authfail
> > > logging queue-limit 100
> > > no logging console
> > > aaa new-model
> > > aaa authentication banner ^CCC
> > > Unauthorized Access Prohibited!
> > >
> > > ^C
> > > aaa authentication fail-message ^CCC
> > > Failed Login!
> > > ^C
> > > aaa authentication login default group radius local
> > > aaa authorization exec default group radius local
> > > aaa accounting commands 7 default start-stop group tacacs+
> > > aaa accounting commands 15 default start-stop group tacacs+
> > > enable secret 5 <removed>
> > > enable password 7 <removed>
> > > !
> > > username admin password 7 <removed>
> > > username operator privilege 7 password 7 <removed>
> > > ip subnet-zero
> > > ip flow-cache timeout inactive 300
> > > ip flow-cache timeout active 1
> > > ip cef
> > > !
> > > !
> > > !
> > > no ip domain-lookup
> > > ip domain-name blah
> > > ip name-server x.x.x.x
> > > ip name-server x.x.x.x
> > > ip name-server x.x.x.x
> > > !
> > > no ip bootp server
> > > ip vrf VRF1
> > > rd xxxxx:999000016
> > > route-target export xxxxx:999000016
> > > route-target import xxxxx:999000016
> > > !
> > > ip vrf VRF2
> > > rd xxxxx:999000010
> > > route-target export xxxxx:999000010
> > > route-target import xxxxx:999000010
> > > !
> > > ip vrf VRF3
> > > rd xxxxx:200440004
> > > export map VRF-INT
> > > route-target export xxxxx:200440004
> > > route-target import xxxxx:200440004
> > > route-target import xxxxx:999000001
> > > !
> > > ip vrf VRF4
> > > rd xxxxx:999002000
> > > route-target export xxxxx:999002000
> > > route-target import xxxxx:999002000
> > > !
> > > clns routing
> > > mpls label protocol ldp
> > > mpls ldp logging neighbor-changes
> > > mpls traffic-eng tunnels
> > > no tag-switching ip propagate-ttl
> > > no tag-switching advertise-tags
> > > tag-switching advertise-tags for LOOPBACKS
> > > tag-switching tdp router-id Loopback1
> > > !
> > > class-map match-any QoS-OUT-PREMIUM
> > > match ip precedence 5
> > > match ip dscp cs5
> > > match ip dscp ef
> > > match mpls experimental 5
> > > class-map match-any QoS-OUT-SILVER
> > > match ip precedence 3
> > > match ip dscp cs3
> > > match ip dscp af31
> > > match ip dscp af32
> > > match ip dscp af33
> > > match mpls experimental 3
> > > class-map match-any QoS-OUT-GOLD
> > > match ip precedence 4
> > > match ip precedence 6
> > > match ip dscp cs4
> > > match ip dscp af41
> > > match ip dscp af42
> > > match ip dscp af43
> > > match ip dscp cs6
> > > match mpls experimental 4
> > > match mpls experimental 6
> > > !
> > > !
> > > policy-map QoS-OUT-S5/0
> > > class QoS-OUT-PREMIUM
> > > priority 13200
> > > class QoS-OUT-GOLD
> > > bandwidth 15400
> > > queue-limit 256
> > > class QoS-OUT-SILVER
> > > bandwidth 11000
> > > queue-limit 128
> > > class class-default
> > > fair-queue
> > > policy-map QoS-OUT-F0/0
> > > class QoS-OUT-PREMIUM
> > > priority 30000
> > > class QoS-OUT-GOLD
> > > bandwidth 35000
> > > queue-limit 256
> > > class QoS-OUT-SILVER
> > > bandwidth 25000
> > > queue-limit 128
> > > class class-default
> > > fair-queue
> > > !
> > > call rsvp-sync
> > > !
> > > !
> > > !
> > > !
> > > !
> > > !
> > > interface Loopback1
> > > description Loopback for BGP
> > > ip address x.x.x.x 255.255.255.255
> > > ip access-group CONTROL-PLANE in
> > > ip router isis
> > > !
> > > interface FastEthernet0/0
> > > description 8006x1_Fa1/2_To Backbone
> > > ip address y.y.y.y 255.255.255.0
> > > ip access-group CONTROL-PLANE in
> > > no ip redirects
> > > no ip unreachables
> > > no ip proxy-arp
> > > ip router isis
> > > rate-limit input access-group 1 1000000 192000 192000 conform-action
> > > set-prec-transmit 4 exceed-action set-prec-transmit 4
> > > ip route-cache flow
> > > ip ospf priority 10
> > > no ip mroute-cache
> > > load-interval 30
> > > duplex full
> > > mpls netflow egress
> > > mpls traffic-eng tunnels
> > > tag-switching mtu 1530
> > > tag-switching ip
> > > max-reserved-bandwidth 100
> > > service-policy output QoS-OUT-F0/0
> > > ip rsvp bandwidth 1000
> > > !
> > > interface Serial6/0
> > > no ip address
> > > ip route-cache flow
> > > load-interval 30
> > > shutdown
> > > mpls netflow egress
> > > mpls traffic-eng tunnels
> > > tag-switching mtu 1530
> > > tag-switching ip
> > > max-reserved-bandwidth 100
> > > service-policy output QoS-OUT-S5/0
> > > dsu bandwidth 44210
> > > framing c-bit
> > > cablelength 10
> > > serial restart-delay 0
> > > ip rsvp bandwidth 1000
> > > !
> > > interface Serial6/1
> > > description DS3 Uplink
> > > ip address x.x.x.x 255.255.255.252
> > > ip access-group CONTROL-PLANE in
> > > no ip redirects
> > > no ip unreachables
> > > ip router isis
> > > ip route-cache flow
> > > load-interval 30
> > > mpls netflow egress
> > > tag-switching mtu 1530
> > > tag-switching ip
> > > max-reserved-bandwidth 100
> > > service-policy output QoS-OUT-S5/0
> > > dsu bandwidth 44210
> > > framing c-bit
> > > cablelength 10
> > > serial restart-delay 0
> > > ip rsvp bandwidth 1000
> > > !
> > > router ospf 1
> > > router-id x.x.x.x
> > > log-adjacency-changes
> > > redistribute connected subnets
> > > redistribute static subnets
> > > network x.x.x.x 0.0.0.255 area 0
> > > default-information originate always
> > > !
> > > router isis
> > > net xx.xxxxx.xxxxxx.xxxxx
> > > is-type level-2-only
> > > metric-style wide level-2
> > > mpls traffic-eng router-id Loopback1
> > > mpls traffic-eng level-2
> > > !
> > > !
> > > router bgp XXXXX
> > > no bgp default ipv4-unicast
> > > bgp log-neighbor-changes
> > > neighbor iBGP-RR-CLIENTS peer-group
> > > neighbor iBGP-RR-CLIENTS remote-as XXXXX
> > > neighbor iBGP-RR-CLIENTS update-source Loopback1
> > > neighbor iBGP-US peer-group
> > > neighbor iBGP-US remote-as XXXXX
> > > neighbor iBGP-US description Uplink BGP
> > > neighbor iBGP-US update-source Loopback1
> > > neighbor iBGP-RR peer-group
> > > neighbor iBGP-RR remote-as xxxxx
> > > neighbor iBGP-RR update-source Loopback1
> > > neighbor eBGP-BACKUP peer-group
> > > neighbor eBGP-BACKUP remote-as YYYYY
> > > neighbor eBGP-BACKUP description Backup IP
> > > neighbor x.x.x.x peer-group iBGP-US
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP
> > > neighbor x.x.x.x peer-group iBGP
> > > neighbor x.x.x.x peer-group eBGP-BACKUP
> > > !
> > > !
> > > address-family ipv4
> > > neighbor iBGP-RR-CLIENTS activate
> > > neighbor iBGP-RR-CLIENTS route-reflector-client
> > > neighbor iBGP-RR-CLIENTS next-hop-self
> > > neighbor iBGP-RR-CLIENTS send-community both
> > > neighbor iBGP-RR-CLIENTS route-map NO-ROUTES out
> > > neighbor iBGP-US activate
> > > neighbor iBGP-US next-hop-self
> > > neighbor iBGP-US send-community both
> > > neighbor iBGP-US route-map iBGP-US-IN in
> > > neighbor iBGP-US route-map iBGP-US-OUT out
> > > neighbor iBGP-RR activate
> > > neighbor iBGP-RR route-reflector-client
> > > neighbor iBGP-RR next-hop-self
> > > neighbor iBGP-RR send-community both
> > > neighbor eBGP-BACKUP activate
> > > neighbor eBGP-BACKUP prefix-list eBGP-OUT out
> > > neighbor eBGP-BACKUP route-map eBGP-IN in
> > > neighbor eBGP-BACKUP route-map eBGP-OUT out
> > > neighbor eBGP-BACKUP maximum-prefix 10
> > > neighbor x.x.x.x peer-group iBGP-US
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR
> > > neighbor x.x.x.x peer-group iBGP-RR
> > > neighbor x.x.x.x peer-group eBGP-BACKUP
> > > no auto-summary
> > > no synchronization
> > > network x.x.x.x route-map ORIGINATE
> > > network x.x.x.x mask 255.255.224.0 route-map ORIGINATE
> > > network x.x.x.x mask 255.255.224.0 route-map ORIGINATE
> > > exit-address-family
> > > !
> > > address-family ipv4 multicast
> > > no auto-summary
> > > exit-address-family
> > > !
> > > address-family vpnv4
> > > neighbor iBGP-RR-CLIENTS activate
> > > neighbor iBGP-RR-CLIENTS route-reflector-client
> > > neighbor iBGP-RR-CLIENTS next-hop-self
> > > neighbor iBGP-RR-CLIENTS send-community both
> > > neighbor iBGP-US activate
> > > neighbor iBGP-US send-community both
> > > neighbor iBGP-RR activate
> > > neighbor iBGP-RR route-reflector-client
> > > neighbor iBGP-RR send-community extended
> > > neighbor x.x.x.x peer-group iBGP-US
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR-CLIENTS
> > > neighbor x.x.x.x peer-group iBGP-RR
> > > neighbor x.x.x.x peer-group iBGP-RR
> > > no auto-summary
> > > exit-address-family
> > > !
> > > address-family ipv4 vrf VRF3
> > > no auto-summary
> > > no synchronization
> > > exit-address-family
> > > !
> > > address-family ipv4 vrf VRF2
> > > redistribute connected
> > > redistribute static
> > > default-information originate
> > > no auto-summary
> > > no synchronization
> > > exit-address-family
> > > !
> > > address-family ipv4 vrf VRF1
> > > redistribute connected
> > > redistribute static
> > > no auto-summary
> > > no synchronization
> > > exit-address-family
> > > !
> > > address-family ipv4 vrf VRF4
> > > redistribute connected
> > > redistribute static
> > > no auto-summary
> > > no synchronization
> > > exit-address-family
> > > !
> > > ip classless
> > > <several static routes>
> > > ip flow-export source Loopback1
> > > ip flow-export version 5 origin-as
> > > ip flow-export destination x.x.x.x 2055
> > > no ip http server
> > > ip bgp-community new-format
> > > <several community-lists>
> > > !
> > > !
> > >
> > > ip prefix-list eBGP-OUT description Deny all prefixes more specific
> > > than /24
> > > ip prefix-list eBGP-OUT seq 10 permit 0.0.0.0/0 le 24
> > > ip prefix-list eBGP-OUT seq 99 deny 0.0.0.0/0 le 32
> > > !
> > > !
> > > <several ip access-lists>
> > > ip radius source-interface Loopback1
> > > !
> > > logging facility local5
> > > logging source-interface Loopback1
> > > logging x.x.x.x
> > > <several access-lists>
> > > no cdp run
> > > !
> > > route-map eBGP-IN permit 10
> > > set local-preference 200
> > > set as-path prepend last-as 5
> > > !
> > > route-map NO-ROUTES deny 5
> > > !
> > > route-map iBGP-US-IN permit 10
> > > !
> > > route-map iBGP-US-IN permit 20
> > > !
> > > route-map ORIGINATE permit 10
> > > set local-preference 103
> > > set community xxxxx:xxx
> > > !
> > > route-map iBGP-RR-CLIENTS-OUT deny 10
> > > !
> > > route-map C7206x1-OUT permit 5
> > > !
> > > route-map NO_EXPORT permit 10
> > > set community no-export
> > > !
> > > route-map eBGP-OUT permit 10
> > > match community XXXXXX
> > > set as-path prepend xxxxx xxxxx xxxxx
> > > !
> > > route-map iBGP-US permit 10
> > > match community xxxxxx xxxxxxx
> > > !
> > > route-map iBGP-OUT deny 20
> > > !
> > > !
> > > snmp-server community <removed> RO 99
> > > snmp-server community <removed> RW 99
> > > snmp-server community <removed> RW
> > > snmp-server community <removed> RO
> > > snmp-server ifindex persist
> > > snmp-server trap-source Loopback1
> > > snmp-server location IDC
> > > snmp-server contact NOC
> > > snmp-server chassis-id xxxxxxx
> > > snmp-server enable traps snmp linkdown linkup coldstart warmstart
> > > snmp-server enable traps tty
> > > snmp-server enable traps atm pvc
> > > snmp-server enable traps envmon fan shutdown supply temperature
> > > snmp-server enable traps bgp
> > > snmp-server enable traps frame-relay
> > > snmp-server enable traps mpls ldp session-up session-down
> > > !
> > > tftp-server slot0:c7200-k91p-mz.122-18.S4.bin
> > > tacacs-server host x.x.x.x key <removed>
> > > tacacs-server directed-request
> > > radius-server host x.x.x.x auth-port 1645 acct-port 1646
> > > radius-server key 7 <removed>
> > > !
> > > dial-peer cor custom
> > > !
> > > !
> > > !
> > > !
> > > line con 0
> > > stopbits 1
> > > line aux 0
> > > modem InOut
> > > no exec
> > > transport input all
> > > stopbits 1
> > > line vty 0 4
> > > access-class ACCESO-TELNET in
> > > exec-timeout 5 0
> > > password 7 <removed>
> > > transport preferred none
> > > line vty 5 15
> > > access-class ACCESO-TELNET in
> > > exec-timeout 5 0
> > > transport preferred none
> > > !
> > > !
> > > exception crashinfo file slot0:crashinfo
> > > scheduler allocate 4000 200
> > > ntp clock-period 17179995
> > > ntp update-calendar
> > > ntp server x.x.x.x
> > > end
> > >
> > >
> > > Thanks to all for the help.
> > >
> > >
> > > Regards,
> > >
> > >
> > > Francisco Rivas Catalan
> > > Senior Network Engineer
> > > IFX Networks
> > > (56) 2 3744574
> > > francisco.rivas at ifxnw.cl
> > >
> > >
> > >
> > >
> > > On Wed, 2005-08-31 at 12:33 -0400, Francisco Rivas wrote:
> > > > Hi all,
> > > >
> > > > I've got a serious problem with one of my 7206vxr. It's reloading itself
> > > > several times, with a "Software forced crash". I've seen this behavior
> > > > before, and it won't happen too often, but this router in particular is
> > > > reloading at least twice a day. It doesn't have too much traffic (just
> > > > 30-40Mb), and the CPU load is < 50%.
> > > > But, the main worry that I have about this, is that I've replaced the
> > > > chassis ENTIRELY, and the new chassis is giving me the same error. Look:
> > > >
> > > >
> > > > System was restarted by error - a Software forced crash, PC 0x606AED30
> > > > at 14:47:49 UTC Wed Aug 31 2005
> > > > 7200 Software (C7200-P-M), Version 12.2(14)S14, RELEASE SOFTWARE (fc1)
> > > > Technical Support: http://www.cisco.com/techsupport
> > > > Compiled Fri 08-Apr-05 13:48 by ccai
> > > > Image text-base: 0x60008BD4, data-base: 0x612C8000
> > > >
> > > >
> > > > This is the output of "sh region address 0x606AED30" on that chassis:
> > > >
> > > > router#show region address 0x606AED30
> > > > Address 0x606AED30 is located physically in :
> > > >
> > > > Name : text
> > > > Class : IText
> > > > Media : R/O
> > > > Start : 0x60008BD4
> > > > End : 0x612C7413
> > > > Size : 0x012BE840
> > > >
> > > >
> > > >
> > > > The old chassis was giving a similar error, not the same address, but it
> > > > was on the IText class too.
> > > > I've tried several IOS versions (12.2(14)S3, 12.2(14)S14, 12.2(18)S4)
> > > > and all of them are crashing with a similar error. Always a software
> > > > forced crash, always the same region address in the error (the region
> > > > varies from IOS to IOS, but with the same IOS, is the same region in
> > > > every crash), and always is a region on the IText class. And this is
> > > > with 2 totally different chassis (both with NPE300 / 256Mb RAM / 6 port
> > > > VXR Chassis).
> > > >
> > > >
> > > > The last part of the crashdump, just before the %Software-forced reload,
> > > > is this:
> > > >
> > > >
> > > > Aug 31 14:47:46 UTC: %SYS-3-OVERRUN: Block overrun at 20014AB8 (red zone
> > > > 1F860000)
> > > > -Traceback= 60692A68 60695A04 606976D0 60697920
> > > > Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> > > > 6770DFB4,6000011A 6770DB88,60C14048 6770DB88,60C0DED8 6770DB88,40000202
> > > > 69C13E0C,60E22530 64C26254,60E27F30 65360AA0,60E37E9C 69C13E0C,60E1F41C
> > > > Aug 31 14:47:46 UTC: %SYS-6-MTRACE: mallocfree: addr, pc
> > > > 69C13E0C,3000000E 6770DFB4,6000011A 6770DB88,60C12734 6770DB88,60C0DED8
> > > > 6770DB88,40000202 6770DFB4,6000011A 6770DB88,60C19E98 6601376C,60C25CA0
> > > > Aug 31 14:47:46 UTC: %SYS-6-BLKINFO: Corrupted redzone blk 20014AB8,
> > > > words 860, alloc 6063AAB8, InUse, dealloc 0, rfcnt 1
> > > > -Traceback= 60690550 60692A7C 60695A04 606976D0 60697920
> > > > Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AB8: 0xAB1234CD 0xFFFFFFFE
> > > > 0x0 0x61584DD8
> > > > Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AC8: 0x6063AAB8 0x20015198
> > > > 0x200143EC 0x8000035C
> > > > Aug 31 14:47:46 UTC: %SYS-6-MEMDUMP: 0x20014AD8: 0x1 0x0 0x0 0x0
> > > >
> > > >
> > > > Anyone can give me a hand on this?? I've already filtered ALL the
> > > > traffic going right to the IP address(es) of the router, but this is
> > > > still happening. I have full crashlogs and "sh stacks" and "sh tech"
> > > > available from the last crashes, so if anyone needs more info, I can
> > > > provide that without problems.
> > > > There are no changes on the config of the router since a couple of
> > > > months, so I can't blame that... anyone have any idea about this?
> > > >
> > >
> > >
> > > _______________________________________________
> > > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
More information about the cisco-nsp
mailing list