[c-nsp] Cisco 1760 - 12.4(1c) - CPU utilization creep

Fri May 2 21:57:05 EDT 2008

Greetings,

We have been experiencing a problem that we believe is something going
haywire with a 1760 router (show ver is below).

Essentially, over time (one to three weeks) the router's CPU utilization
creeps up and begins spiking at 90 to 100% until, if left untreated via
RELOAD it gets progressively worse until pegging at 100% making even a
telnet visit to type the word "reload" a 10 minute lesson in patience.  The
problem first becomes noticed by the end user when they start having
problems with their 30 or so VoIP phones which are more sensitive to network
issues.

Running nearly idle this evening (processing around 50kbps) the cpu was
still showing 18 to 22% [if memory serves] and after reload it basically
flatlined in the expected 1-3% range.

Whack-a-mole fix it attempts have included:
Loading a different IOS version onto the router:
  I believe 12.3(18) [c1700-k9o3sy7-mz.123-18.bin] was the original and it
is still available in flash 
  but we now show it loading 12.4(1c) [see below] after the IOS change.
Replacing the router itself once when the IOS change didn't help.
Packet traces taken on the LAN side of the network do not indicate any
significant broadcast or other suspect traffic.

The following line is at the affected location in the SHOW VER output but
not on the stable router's SHOW VER output: "ROM: C1700 Software
(C1700-K9O3SY7-M), Version 12.3(18), RELEASE SOFTWARE (fc3)"

Any help or advice is greatly appreciated!

SHOW VERSION output

Cisco IOS Software, C1700 Software (C1700-IPBASE-M), Version 12.4(1c),
RELEASE SOFTWARE (fc1)
Technical Support:  <http://www.cisco.com/techsupport>
http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Wed 26-Oct-05 06:46 by evmiller

ROM: System Bootstrap, Version 12.2(7r)XM2, RELEASE SOFTWARE (fc1)
ROM: C1700 Software (C1700-K9O3SY7-M), Version 12.3(18), RELEASE SOFTWARE
(fc3)

********** uptime is 9 minutes
System returned to ROM by reload at 00:10:01 UTC Sat May 3 2008
System restarted at 00:14:05 UTC Sat May 3 2008
System image file is "flash:c1700-ipbase-mz.124-1c.bin"

Cisco 1760 (MPC860P) processor (revision 0x851) with 55706K/9830K bytes of
memory.
Processor board ID FOC09512DZZ (2976419533), with hardware revision 0000
MPC860P processor: part number 5, mask 2
1 FastEthernet interface
1 Serial interface
WIC T1-DSU
32K bytes of NVRAM.
32768K bytes of processor board System flash (Read/Write)

Configuration register is 0x2102

COPY OF CONFIGURATION (EDITED TO GET G RATING) ;-)
note: configuration mostly provided by customer's VAR whom we are working
with on the problem.

Using 2816 out of 29688 bytes
version 12.4
service timestamps debug uptime
service timestamps log uptime
service password-encryption
!
hostname xxxxxxxxxxxxx
!
boot-start-marker
boot system flash c1700-ipbase-mz.124-1c.bin
boot-end-marker
!
no logging buffered
enable password 7 xxxxxxxxxxxxxxx
!
no aaa new-model
!
resource policy
!
memory-size iomem 15
mmi polling-interval 60
no mmi auto-configure
no mmi pvc
mmi snmp-timeout 180
ip subnet-zero
ip cef
!
!
no ip dhcp use vrf connected
ip dhcp excluded-address 10.100.1.1
!
ip dhcp pool Phones
   network 10.100.1.0 255.255.255.0
   default-router 10.100.1.1
   option 66 ascii "xxxxxxxxxxxx"
   dns-server xxxxxxxxxxxx xxxxxxxxxxxx
   lease 60
!
!
ip domain name xxxxxxxxxxxxxx
ip name-server xxxxxxxxxxxx
ip name-server xxxxxxxxxxxx
!
!
class-map match-all VOIP-CONTROL
 match access-group 182
 match access-group 102
class-map match-all VOIP-RTP
 match access-group 181
 match access-group 102
!
!
policy-map QOS-Policy-1.5M-Internet
 class VOIP-RTP
  priority 1000
 class VOIP-CONTROL
  bandwidth 64
 class class-default
  fair-queue
!
!
!
interface FastEthernet0/0
 description Connection to LAN
 no ip address
 speed 100
 full-duplex
!
interface FastEthernet0/0.1
 description Private data network VLAN
!
interface FastEthernet0/0.2
 description Voice network VLAN
 encapsulation dot1Q 2
 ip address 10.100.1.1 255.255.255.0
 ip nat inside
 no snmp trap link-status
!
interface FastEthernet0/0.3
 description Public address VLAN
 encapsulation dot1Q 3
 ip address xxxxxxxxxxxxxxx 255.255.255.248
 no snmp trap link-status
!
interface Serial0/0
 description T1 Uplink
 bandwidth 1536
 ip address xxxxxxxxxxxx 255.255.255.252
 ip nat outside
 service-policy output QOS-Policy-1.5M-Internet
!
ip classless
ip route 0.0.0.0 0.0.0.0 Serial0/0
!
no ip http server
ip nat translation udp-timeout 602
ip nat inside source list 10 interface Serial0/0 overload
ip nat inside source static tcp 10.100.1.15 80 interface Serial0/0 33000
!
access-list 10 permit 10.100.1.0 0.0.0.255
access-list 102 permit udp 10.100.1.0 0.0.0.255 any
access-list 102 permit udp any 10.100.1.0 0.0.0.255
access-list 181 permit udp any any dscp ef
access-list 182 permit udp any any dscp af31
snmp-server community xxxxxxxxxxxx RO
!
control-plane
!
!
ntp clock-period 17208021
ntp server xxxxxxxxxxxx prefer
end