[c-nsp] Cisco 7206 VXR hangs
Scott Lambert
lambert at lambertfam.org
Wed Jan 10 06:20:26 EST 2007
I have a Cisco 7204VXR with NPE-G1 that has been hanging on me at one to
three week intervals.
The box is doing DSL aggregation as well as being our core router. We
have a handful of T1s on it. Both DSL and Internet are on the same ATM
OC3 interface.
The box had been rock solid from July through October in the current
hardware and software configuration before I tried some load reducing
configuration changes. The last hardware change was an upgrade from
NPE-300 which had been working for years to the NPE-G1 which required
the IOS upgrade to 12.2(28) SB2.
I did two config changes Oct 4 because the telco was trying to tell me
my ATM throughput problems were due to CPU load on the box. The CPU
load dropped by half. We went from 50% to 25% CPU utilization. The ATM
problems remained. The telco eventually found a provisioning error and
fixed the ATM issues.
I enabled route-cache cef on my PPPo{E|A} virtual-templates and
increased the small, middle, and big buffers' permanent settings.
@@ -172,6 +172,10 @@ controller T1 4/7
linecode b8zs
channel-group 0 timeslots 1-24
!
+buffers small permanent 700
+buffers middle permanent 700
+buffers big permanent 400
+!
bba-group pppoe global
virtual-template 3
sessions per-vc limit 1024
@@ -8322,8 +8326,6 @@ interface Serial4/7:0
interface Virtual-Template1
description PPPoA Template
ip unnumbered Loopback0
- no ip route-cache cef
- no ip route-cache
ip ospf database-filter all out
peer default ip address pool dsl
ppp authentication pap callin
@@ -8333,8 +8335,6 @@ interface Virtual-Template3
mtu 1492
ip unnumbered Loopback0
ip mtu 1492
- no ip route-cache cef
- no ip route-cache
ip ospf database-filter all out
no logging event link-status
peer default ip address pool dsl
November 4th, we had our first lockup during pretty much the slowest
day of the week and off-peak hours at that. There were no log entries
because the syslog server was broken and there was no response on the
serial console. A power-cycle brought it right back up. Everything
appears to work normally after the power-cycle. I crossed my fingers
and hoped the cause was "cosmic ray".
Two weeks and one day later, the same thing happenned. That night I
brought the IOS up to the current level 12.2(28) SB5.
Three weeks later, another lockup. We ordered RAM and a spare ATM OC3
card. They have arrived but not been installed yet.
Tonight, a week later, it happened again. I have now fixed my syslog
problems and enabled logging to the console for warning level and above
messages.
The CPU, temperature, line error rate, and bandwidth MRTG graphs are
normal leading up to the hangs.
Are the above config statements known to be dangerous with 12.2(28)SB#?
If its not a known IOS bug, is there more likely hardware culprit I
should replace first? What else do I need to be doing to track this
problem down?
Cisco IOS Software, 7200 Software (C7200-IK91S-M), Version 12.2(28)SB5, RELEASE
SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2006 by Cisco Systems, Inc.
Compiled Mon 02-Oct-06 22:14 by richv
ROM: System Bootstrap, Version 12.3(4r)T1, RELEASE SOFTWARE (fc1)
BOOTLDR: 7200 Software (C7200-KBOOT-M), Version 12.2(4)BW2, EARLY DEPLOYMENT REL
EASE SOFTWARE (fc1)
router-7204 uptime is 7 hours, 54 minutes
System returned to ROM by power-on
System restarted at 02:42:15 UTC Wed Jan 10 2007
System image file is "disk2:c7200-ik91s-mz.122-28.SB5.bin"
This product contains cryptographic features and is subject to United
States and local country laws governing import, export, transfer and
use. Delivery of Cisco cryptographic products does not imply
third-party authority to import, export, distribute or use encryption.
Importers, exporters, distributors and users are responsible for
compliance with U.S. and local country laws. By using this product you
agree to comply with applicable laws and regulations. If you are unable
to comply with U.S. and local laws, return this product immediately.
A summary of U.S. laws governing Cisco cryptographic products may be found at:
http://www.cisco.com/wwl/export/crypto/tool/stqrg.html
If you require further assistance please contact us by sending email to
export at cisco.com.
Cisco 7204VXR (NPE-G1) processor (revision B) with 229376K/32768K bytes of memor
y.
Processor board ID 21276969
SB-1 CPU at 700Mhz, Implementation 1025, Rev 0.2, 512KB L2 Cache
4 slot VXR midplane, Version 2.1
Last reset from power-on
PCI bus mb1 (Slots 1, 3 and 5) has a capacity of 600 bandwidth points.
Current configuration on bus mb1 has a total of 300 bandwidth points.
This configuration is within the PCI bus capacity and is supported.
PCI bus mb2 (Slots 2, 4 and 6) has a capacity of 600 bandwidth points.
Current configuration on bus mb2 has a total of 0 bandwidth points.
This configuration is within the PCI bus capacity and is supported.
Please refer to the following document "Cisco 7200 Series Port
Adaptor Hardware Configuration Guidelines" on CCO <www.cisco.com>,
for c7200 bandwidth points oversubscription/usage guidelines.
1 FastEthernet interface
3 Gigabit Ethernet interfaces
8 Serial interfaces
1 ATM interface
8 Channelized T1/PRI ports
509K bytes of NVRAM.
20480K bytes of Flash PCMCIA card at slot 0 (Sector size 128K).
62592K bytes of ATA PCMCIA card at slot 2 (Sector size 512 bytes).
16384K bytes of Flash internal SIMM (Sector size 256K).
Configuration register is 0x2102
ATM1/0 is up, line protocol is up
Hardware is ENHANCED ATM PA, address is 0004.9b68.741c (bia 0004.9b68.741c)
MTU 4470 bytes, sub MTU 4470, BW 149760 Kbit, DLY 80 usec,
reliability 255/255, txload 12/255, rxload 14/255
Encapsulation ATM, loopback not set
Encapsulation(s): AAL5, PVC mode
4095 maximum active VCs, 1666 current VCCs
VC Auto Creation Disabled.
VC idle disconnect time: 300 seconds
1 carrier transitions
Last input 00:00:00, output 00:00:00, output hang never
Last clearing of "show interface" counters never
Input queue: 0/75/0/12869 (size/max/drops/flushes); Total output drops: 699
Queueing strategy: Per VC Queueing
5 minute input rate 8464000 bits/sec, 3353 packets/sec
5 minute output rate 7418000 bits/sec, 3138 packets/sec
125441557 packets input, 1857627093 bytes, 0 no buffer
Received 0 broadcasts (481 IP multicast)
0 runts, 0 giants, 0 throttles
3709 input errors, 5859 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort
117849345 packets output, 760064429 bytes, 0 underruns
0 output errors, 0 collisions, 0 interface resets
0 output buffer failures, 0 output buffers swapped out
Buffer elements:
499 in free list (500 max allowed)
43875283 hits, 0 misses, 0 created
Public buffer pools:
Small buffers, 104 bytes (total 700, permanent 700, peak 724 @ 08:34:59):
666 in free list (80 min, 850 max allowed)
20505460 hits, 39 misses, 24 trims, 24 created
15 failures (0 no memory)
Middle buffers, 600 bytes (total 700, permanent 700, peak 711 @ 08:34:59):
697 in free list (80 min, 850 max allowed)
5096893 hits, 9 misses, 11 trims, 11 created
0 failures (0 no memory)
Big buffers, 1536 bytes (total 400, permanent 400):
398 in free list (20 min, 450 max allowed)
3125790 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
VeryBig buffers, 4520 bytes (total 10, permanent 10):
9 in free list (0 min, 100 max allowed)
1 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Large buffers, 5024 bytes (total 0, permanent 0):
0 in free list (0 min, 10 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Huge buffers, 18024 bytes (total 0, permanent 0):
0 in free list (0 min, 4 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Interface buffer pools:
IPC buffers, 4096 bytes (total 2, permanent 2):
1 in free list (1 min, 8 max allowed)
1 hits, 0 fallbacks, 0 trims, 0 created
0 failures (0 no memory)
Header pools:
Header buffers, 0 bytes (total 511, permanent 256, peak 511 @ 08:35:33):
255 in free list (256 min, 1024 max allowed)
171 hits, 85 misses, 0 trims, 255 created
0 failures (0 no memory)
256 max cache size, 256 in cache
135051 hits in cache, 0 misses in cache
Particle Clones:
1024 clones, 6115 hits, 0 misses
Public particle pools:
F/S buffers, 128 bytes (total 512, permanent 512):
0 in free list (0 min, 512 max allowed)
512 hits, 0 misses
512 max cache size, 512 in cache
6115 hits in cache, 0 misses in cache
Normal buffers, 512 bytes (total 2048, permanent 2048):
2048 in free list (1024 min, 4096 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
Private particle pools:
HQF buffers, 0 bytes (total 128, permanent 128):
128 in free list (0 min, 128 max allowed)
0 hits, 0 misses, 0 trims, 0 created
0 failures (0 no memory)
GigabitEthernet0/1 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/2 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
GigabitEthernet0/3 buffers, 512 bytes (total 1000, permanent 1000):
0 in free list (0 min, 1000 max allowed)
1000 hits, 0 fallbacks
1000 max cache size, 872 in cache
128 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
FastEthernet0/0 buffers, 512 bytes (total 400, permanent 400):
0 in free list (0 min, 400 max allowed)
400 hits, 0 fallbacks
400 max cache size, 266 in cache
53409240 hits in cache, 0 misses in cache
14 buffer threshold, 0 threshold transitions
ATM1/0 buffers, 512 bytes (total 4000, permanent 4000):
0 in free list (0 min, 4000 max allowed)
4000 hits, 1 misses
T1 4/0 buffers, 512 bytes (total 768, permanent 768):
0 in free list (0 min, 768 max allowed)
768 hits, 0 fallbacks
768 max cache size, 512 in cache
8457568 hits in cache, 0 misses in cache
10 buffer threshold, 0 threshold transitions
--
Scott Lambert KC5MLE Unix SysAdmin
lambert at lambertfam.org
More information about the cisco-nsp
mailing list