[cisco-bba] High CPU utilization on Cisco 7206VXR seems to belimiting performance of higher-speed users

Frank Bulk frnkblk at iname.com
Mon Apr 23 14:22:48 EDT 2007


With Oli's most gracious assistance I was able to bring the Cisco 7206VXR
down from the 60-75% range to about 35%.

I discovered that when running a 'sh ip interface' that most of my
Virtual-Access interfaces were using TCP header compression  Oli informed me
that 'ip tcp header compression' triggers process switching of the traffic.
Although it was not explicitly configured on my 7206VXR, our RADIUS server
was returning the "Van-Jacobson Header Compression" attribute (inherited
from our dialup config, and alluded to here:
http://puck.nether.net/pipermail/cisco-bba/2004-December/000348.html).  I
had a colleague a lot smarter than me remove that attribute from the RADIUS
responses for requests coming from the 7206VXR.  That brought things down to
45%, or a savings of 15 to 30%.

But Oli was still not satisfied and noticed in my config that I was using
'ip igmp join-group' rather than 'ip igmp static-group'.  I changed those
and saved myself another 10%, bringing IP Input down to sub 5% levels.

Thanks, Oli!

Kind regards,

Frank

-----Original Message-----
From: cisco-bba-bounces at puck.nether.net
[mailto:cisco-bba-bounces at puck.nether.net] On Behalf Of Frank Bulk
Sent: Thursday, April 12, 2007 1:49 PM
To: cisco-bba at puck.nether.net
Subject: [cisco-bba] High CPU utilization on Cisco 7206VXR seems to
belimiting performance of higher-speed users


We've had two complaints from 2 Mbps customers that they aren't getting
their contracted bandwidth.  I went to one of them and confirmed that it's
mixed, getting only up to 1.5 Mbps at times.  Half of our customers run at
128/128 kbps, another 40% at 1024/256 kbps, and the remaining at 2048/384
kbps.

We have a Cisco 7206VXR with an NPE400 with 491520K/32768K bytes of memory
running c7200-is-mz.122-26.bin.  The processor is running at about 60%, up
from 40% a year ago.  I believe that the CPU has something do with the
performance.  

========
Router#sh proc cpu | exc 0.00.*0.00
CPU utilization for five seconds: 61%/27%; one minute: 62%; five minutes:
60%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
   3    15913764  57964253        274  0.16%  0.04%  0.04%   0 PPP auth
   4   176627692  11860384      14892  0.00%  0.41%  0.43%   0 Check heaps
  15  1228555796 497123198       2471  2.51%  3.14%  3.51%   0 ARP Input
  16    67532280   9516840       7096  0.00%  0.14%  0.16%   0 HC Counter
Timer
  22   417429972  77856792       5361  0.32%  0.75%  0.80%   0 Net
Background
  40  14789027243609760351          0 23.24% 23.18% 23.30%   0 IP Input
  41    12942320   4757230       2720  0.08%  0.05%  0.06%   0 CDP Protocol
  49    53911324  50794855       1061  0.00%  0.07%  0.07%   0 IP Background
  63   169456380  62895851       2694  0.16%  0.29%  0.31%   0 CEF process
  94   309167136   7797068      39652  0.89%  0.79%  0.80%   0 Compute load
avg
 103   676779760  45869622      14754  1.02%  3.72%  3.99%   0 PPPOE
discovery
 113   210967168 765640753        275  0.16%  0.18%  0.19%   0 PPP manager
 126   488852524  15943470      30661  4.37%  2.72%  2.79%   0 VTEMPLATE
Backgr
 127          40       152        263  0.08%  0.03%  0.00%   2 Virtual Exec
Router#
========

I followed the advice on Cisco's web pages on troubleshooting IP Input CPU
load on Friday but nothing I tried seemed to make a difference.  

Our DSL customers come in on two OC3's and we have some FTTH customers
coming in on Fa0/0.  Our main Ethernet interface, Fa0/1, does have quite a
few drops and flushes, but you can see the loads are low and cacti reports
interface utilization of about 10 to 15 Mbps.
========
FastEthernet0/1 is up, line protocol is up
  Hardware is i82543 (Livengood), address is 000d.6633.dc06 (bia
000d.6633.dc06)
  MTU 1500 bytes, BW 100000 Kbit, DLY 100 usec,
     reliability 255/255, txload 12/255, rxload 56/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 100Mb/s, 100BaseTX/FX
  ARP type: ARPA, ARP Timeout 00:05:00
  Last input 00:00:00, output 00:00:00, output hang never
  Last clearing of "show interface" counters never
  Input queue: 0/75/4135105/302954774 (size/max/drops/flushes); Total output
drops: 10256 
                    ^^^^^^^ ^^^^^^^^^           ^^^^^ ^^^^^^^
^^^^^
========
Router#show interfaces switching
<snipped out the FastEthernet0/0>
FastEthernet0/1 
          Throttle count     916697
        Drops         RP    4135723         SP          0
  SPD Flushes       Fast  303102861        SSE          0
  SPD Aggress       Fast          0
 SPD Priority     Inputs   58974374      Drops          0

     Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out
        Other    Process   18572355 1146224998    3926592  235595520
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
           IP    Process 3017811923 1959581225 2355715836 3994592527
            Cache misses          0
                    Fast 4013202896  604980241 3965937813 3361756168
               Auton/SSE          0          0          0          0
          ARP    Process  823012431 2140444865   38144362 2441239168
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
          CDP    Process     654814  270489385     655922  347750478
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
 PPP over ATM    Process          0          0          9        540
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
         MSCP    Process          0          0   31363715 2007277760
            Cache misses          0
                    Fast          0          0          0          0
               Auton/SSE          0          0          0          0
========
A complete 'show interfaces switching' and 'show ip traffic', one after
another, can be found zipped up in this file:
http://www.mtcnet.net/~fbulk/show_interfaces_switching_ip_traffic.zip

I also dropped the two ACLs we have on our Ethernet interface and it didn't
make a difference.
 
We have about 2013 active PPPoA connections and 35 PPPoE connections.

We haven't changed code or anything for 2+ years, before my time.  We are
cutting some customers over from PPPoA coming in on an ATM interface to
PPPoE on Fa0/0 as we convert them to a FTTH installation, but that's perhaps
been 25 connections or so.
 
I would welcome any ideas anyone has.  I would like to avoid upgrading to a
G1 or G2 if I could.
 
Kind regards,
 
Frank

_______________________________________________
cisco-bba mailing list
cisco-bba at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-bba



More information about the cisco-bba mailing list