[c-nsp] Troubleshooting high processing on router

Joost greene joost.greene at gmail.com
Sun Aug 7 05:59:08 EDT 2005


Hi,

I am trying to troubleshoot why a router has high cpu usage, from
output shown below, it appears that IP Input is the beast,

I know that this box is pushing alot of different types of
traffic(MPLS, BGP,...,almost 500 Virtual-Access/Virtual-Template and
SVIs)

 Its an RPM (equivalent to 7204 router)

My question is how do i troubleshoot high IP input, i looked at the
below output of 'sh ip traffic, cant make sense of the numbers, i am
trying to look at number of broadcasts compared to normal packets
numbers but i wouldnt know if its much or normal.

Cisco IOS Software, RPM Software (RPM-JS-M), Version 12.3(11)T5,
RELEASE SOFTWARE (fc1)
Technical Support: http://www.cisco.com/techsupport
Copyright (c) 1986-2005 by Cisco Systems, Inc.
Compiled Sat 02-Apr-05 14:57 by yiyan

ROM: System Bootstrap, Version 12.1(20001003:080040)
[swtools-rommon400 102], DEVELOPMENT SOFTWARE
BOOTLDR: RPM Software (RPM-BOOT-M), Version 12.2(4)T,  RELEASE SOFTWARE (fc1)

 uptime is 2 weeks, 2 days, 7 hours, 17 minutes
System returned to ROM by reload at 04:33:28 CAI Fri Jul 22 2005
System restarted at 04:34:26 CAI Fri Jul 22 2005
Running default software
Last reload reason: Reload command


Cisco RPM-PR (NPE400) processor with 491520K/32768K bytes of memory.
R7000 CPU at 350MHz, Implementation 39, Rev 3.2, 256KB L2, 4096KB L3 Cache
Last reset from s/w peripheral
2 FastEthernet interfaces
1 ATM interface
125K bytes of NVRAM.




CPU utilization for five seconds: 44%/30%; one minute: 45%; five minutes: 43%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
  57   2226091361581548486        140 13.83% 13.00% 12.34%   0 IP Input
  32     3442980   1431013       2405  0.15%  0.28%  0.25%   0 Per-Second Jobs
  45      619476   2259111        274  0.07%  0.04%  0.02%   0 CEF process
  40      366308   6764173         54  0.07%  0.01%  0.00%   0 ATM OAM Input
 164     1044272   2393821        436  0.07%  0.06%  0.07%   0 RIP Send
  38       99488    145992        681  0.07%  0.00%  0.00%   0 ATM Periodic
  50     1026996    288610       3558  0.07%  0.06%  0.07%   0 Compute load avg
 105      598792   8684878         68  0.07%  0.02%  0.01%   0 Tag Input
   8           0         2          0  0.00%  0.00%  0.00%   0 ATM Idle Timer
   9          40     23473          1  0.00%  0.00%  0.00%   0 IPC Dynamic Cach
   7           0         2          0  0.00%  0.00%  0.00%   0 Serial Backgroun
  12        4548   1406529          3  0.00%  0.00%  0.00%   0 IPC Deferred Por
   6           0         2          0  0.00%  0.00%  0.00%   0 Timers
  10       29520  43553511          0  0.00%  0.00%  0.00%   0 RADIUS
  11        4436   1406526          3  0.00%  0.00%  0.00%   0 IPC Periodic Tim
   5        1352      3142        430  0.00%  0.00%  0.00%   0 Pool Manager
  17           0         1          0  0.00%  0.00%  0.00%   0 Crash writer
  18       14304   1406522         10  0.00%  0.00%  0.00%   0 EnvMon
  19           0         1          0  0.00%  0.00%  0.00%   0 OIR Handler
  20         700     28788         24  0.00%  0.00%  0.00%   0 ARP Input
  13         616     10194         60  0.00%  0.00%  0.00%   0 IPC Seat Manager
  14           0         1          0  0.00%  0.00%  0.00%   0 IPC BackPressure
   2         776    281645          2  0.00%  0.00%  0.00%   0 Load Meter
  24           0         1          0  0.00%  0.00%  0.00%   0 SERIAL A'detect
  15           4      4348          0  0.00%  0.00%  0.00%   0 AAA high-capacit
  26           0         2          0  0.00%  0.00%  0.00%   0 Dialer event
  16           0         1          0  0.00%  0.00%  0.00%   0 Policy Manager
  28           4         2       2000  0.00%  0.00%  0.00%   0 Critical Bkgnd
   3           0        13          0  0.00%  0.00%  0.00%   0 OSPF Hello 99
  30         864      3791        227  0.00%  0.00%  0.00%   0 Logger
  31       15096   1406510         10  0.00%  0.00%  0.00%   0 TTY Background
  21          40       927         43  0.00%  0.00%  0.00%   0 DDR Timers
  33       11520   1403194          8  0.00%  0.00%  0.00%   0 RPM Hardware Dia
  34        7244   1403194          5  0.00%  0.00%  0.00%   0 RPM Software Dia
  35           0         1          0  0.00%  0.00%  0.00%   0 CSP Timer
  36           0         1          0  0.00%  0.00%  0.00%   0 RPM TEST
  37           0         1          0  0.00%  0.00%  0.00%   0 Coproc Event Pro
   4     1987000    168132      11818  0.00%  0.09%  0.07%   0 Check heaps
  39        6628     41483        159  0.00%  0.00%  0.00%   0 ATM ARP INPUT
  22           0         2          0  0.00%  0.00%  0.00%   0 Entity MIB API
  23        1056   1396612          0  0.00%  0.00%  0.00%   0 MPLS HC Counter
  42        3192   1407251          2  0.00%  0.00%  0.00%   0 ATM PA Helper

RPM1-Maadi-I#sh ip traffic
IP statistics:
  Rcvd:  686065560 total, 10112838 local destination
         10750 format errors, 0 checksum errors, 1271416 bad hop count
         0 unknown protocol, 2330835 not a gateway
         0 security failures, 0 bad options, 827 with options
  Opts:  0 end, 0 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 0 record route
         0 stream ID, 0 strict source route, 827 alert, 0 cipso, 0 ump
         0 other
  Frags: 13710 reassembled, 102 timeouts, 0 couldn't reassemble
         13554 fragmented, 5515 couldn't fragment
  Bcast: 458932 received, 327605 sent
  Mcast: 2633985 received, 15778165 sent
  Sent:  14984677 generated, 664821480 forwarded
  Drop:  3118222 encapsulation failed, 899 unresolved, 1 no adjacency
         2185541 no route, 0 unicast RPF, 0 forced drop
         0 options denied
  Drop:  0 packets with source IP address zero
  Drop:  0 packets with internal loop back IP address

ICMP statistics:
  Rcvd: 4 format errors, 1 checksum errors, 0 redirects, 145 unreachable
        699239 echo, 52036 echo reply, 0 mask requests, 0 mask replies, 0 quench
        0 parameter, 0 timestamp, 0 info request, 0 other
        0 irdp solicitations, 0 irdp advertisements
  Sent: 21 redirects, 796819 unreachable, 53607 echo, 699239 echo reply
        0 mask requests, 0 mask replies, 0 quench, 0 timestamp
        0 info reply, 3516273 time exceeded, 0 parameter problem
        0 irdp solicitations, 0 irdp advertisements

UDP statistics:
  Rcvd: 2837629 total, 3 checksum errors, 2361837 no port
  Sent: 11300834 total, 0 forwarded broadcasts

TCP statistics:
  Rcvd: 6191695 total, 2030 checksum errors, 206 no port
  Sent: 564078 total

BGP statistics:
  Rcvd: 2030894 total, 34 opens, 0 notifications, 1593141 updates
        437621 keepalives, 98 route-refresh, 0 unrecognized
  Sent: 437768 total, 34 opens, 15 notifications, 0 updates
        437668 keepalives, 51 route-refresh

OSPF statistics:
  Rcvd: 328169 total, 0 checksum errors
        286801 hello, 17 database desc, 1 link state req
        39777 link state updates, 1573 link state acks

  Sent: 1858925 total
        1974490 hello, 15 database desc, 6 link state req
        3649 link state updates, 21436 link state acks

IP-EIGRP statistics:
  Rcvd: 0 total
  Sent: 0 total

PIMv2 statistics: Sent/Received
  Total: 0/0, 0 checksum errors, 0 format errors
  Registers: 0/0 (0 non-rp, 0 non-sm-group), Register Stops: 0/0,  Hellos: 0/0
  Join/Prunes: 0/0, Asserts: 0/0, grafts: 0/0
  Bootstraps: 0/0, Candidate_RP_Advertisements: 0/0
  Queue drops: 0
  State-Refresh: 0/0

IGMP statistics: Sent/Received
  Total: 0/0, Format errors: 0/0, Checksum errors: 0/0
  Host Queries: 0/0, Host Reports: 0/0, Host Leaves: 0/0
  DVMRP: 0/0, PIM: 0/0
  Queue drops: 0

ARP statistics:
  Rcvd: 1529 requests, 385 replies, 0 reverse, 0 other
  Sent: 8457 requests, 7841 replies (144 proxy), 0 reverse



More information about the cisco-nsp mailing list