[cisco-bba] High CPU utilization on Cisco 7206VXR seems to be limiting performance of higher-speed users

[Frank Bulk]

We've had two complaints from 2 Mbps customers that they aren't getting
their contracted bandwidth.  I went to one of them and confirmed that it's
mixed, getting only up to 1.5 Mbps at times.  Half of our customers run at
128/128 kbps, another 40% at 1024/256 kbps, and the remaining at 2048/384
kbps.

We have a Cisco 7206VXR with an NPE400 with 491520K/32768K bytes of memory
running c7200-is-mz.122-26.bin.  The processor is running at about 60%, up
from 40% a year ago.  I believe that the CPU has something do with the
performance.  

========
Router#sh proc cpu | exc 0.00.*0.00
CPU utilization for five seconds: 61%/27%; one minute: 62%; five minutes:
60%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   3    15913764  57964253        274  0.16%  0.04%  0.04%   0 PPP auth
   4   176627692  11860384      14892  0.00%  0.41%  0.43%   0 Check heaps
  15  1228555796 497123198       2471  2.51%  3.14%  3.51%   0 ARP Input
  16    67532280   9516840       7096  0.00%  0.14%  0.16%   0 HC Counter
Timer
  22   417429972  77856792       5361  0.32%  0.75%  0.80%   0 Net
Background
  40  14789027243609760351          0 23.24% 23.18% 23.30%   0 IP Input
  41    12942320   4757230       2720  0.08%  0.05%  0.06%   0 CDP Protocol
  49    53911324  50794855       1061  0.00%  0.07%  0.07%   0 IP Background
  63   169456380  62895851       2694  0.16%  0.29%  0.31%   0 CEF process
  94   309167136   7797068      39652  0.89%  0.79%  0.80%   0 Compute load
avg
 103   676779760  45869622      14754  1.02%  3.72%  3.99%   0 PPPOE
discovery
 113   210967168 765640753        275  0.16%  0.18%  0.19%   0 PPP manager
 126   488852524  15943470      30661  4.37%  2.72%  2.79%   0 VTEMPLATE
Backgr
 127          40       152        263  0.08%  0.03%  0.00%   2 Virtual Exec
Router#
========

I followed the advice on Cisco's web pages on troubleshooting IP Input CPU
load on Friday but nothing I tried seemed to make a difference.  

Our DSL customers come in on two OC3's and we have some FTTH customers
coming in on Fa0/0.  Our main Ethernet interface, Fa0/1, does have quite a
few drops and flushes, but you can see the loads are low and cacti reports
interface utilization of about 10 to 15 Mbps.
========
FastEthernet0/1 is up, line protocol is up
  Hardware is i82543 (Livengood), address is 000d.6633.dc06 (bia
000d.6633.dc06)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 12/255, rxload 56/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 00:05:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/4135105/302954774 (size/max/drops/flushes); Total output
drops: 10256 
                    ^^^^^^^ ^^^^^^^^^           ^^^^^ ^^^^^^^
^^^^^
========
Router#show interfaces switching
<snipped out the FastEthernet0/0>
FastEthernet0/1 
          Throttle count     916697
        Drops         RP    4135723         SP          0
  SPD Flushes       Fast  303102861        SSE          0
  SPD Aggress       Fast          0
 SPD Priority     Inputs   58974374      Drops          0

     Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
        Other    Process   18572355 1146224998    3926592  235595520
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
           IP    Process 3017811923 1959581225 2355715836 3994592527
            Cache misses          0
                    Fast 4013202896  604980241 3965937813 3361756168
               Auton/SSE          0          0          0          0
          ARP    Process  823012431 2140444865   38144362 2441239168
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
          CDP    Process     654814  270489385     655922  347750478
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
 PPP over ATM    Process          0          0          9        540
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
         MSCP    Process          0          0   31363715 2007277760
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
========
A complete 'show interfaces switching' and 'show ip traffic', one after
another, can be found zipped up in this file:
http://www.mtcnet.net/~fbulk/show_interfaces_switching_ip_traffic.zip

I also dropped the two ACLs we have on our Ethernet interface and it didn't
make a difference.
 
We have about 2013 active PPPoA connections and 35 PPPoE connections.

We haven't changed code or anything for 2+ years, before my time.  We are
cutting some customers over from PPPoA coming in on an ATM interface to
PPPoE on Fa0/0 as we convert them to a FTTH installation, but that's perhaps
been 25 connections or so.
 
I would welcome any ideas anyone has.  I would like to avoid upgrading to a
G1 or G2 if I could.
 
Kind regards,
 
Frank