[c-nsp] cpu spike every minute

[Matlock, Kenneth L]

Ok, found the bug ID:

 

CSCsr12976

 

Ken Matlock
Network Analyst
Exempla Healthcare
(303) 467-4671
matlockk at exempla.org

________________________________

From: Matlock, Kenneth L 
Sent: Tuesday, January 11, 2011 6:43 AM
To: Holemans Wim; cisco-nsp at puck.nether.net
Subject: RE: [c-nsp] cpu spike every minute

 

We had the same thing with the same version of Modular code. (Which is
the reason why we no longer run Modular)

 

I'll have to dig up the bug ID, but basically it's a small memory leak
that's triggered by 'show run' or 'write term'.

 

If you have dual supervisors then swap to the other one and the problem
will go away for a while.

 

Do you use Rancid or the like to archive configs? That's what was
triggering it on our side.

 

Either way, Cisco removed Safe Harbor certification from that code
release, so I'd recomend upgrading that code anyway.

 

When I get into the office I'll find the particular bug ID.

 

Ken Matlock

matlockk at exempla.org

Network Analyst

Exempla Healthcare

 

________________________________

From: cisco-nsp-bounces at puck.nether.net on behalf of Holemans Wim
Sent: Tue 1/11/2011 5:51 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] cpu spike every minute

We are seeing a cpu spike (and corresponding icmp respons latency) every
minute on one of our 65XX.
It is a   6506-E with Sup32-8G running IOS version
ipbasek9-vz.122-18.SXF6.
I checked al our mgmt processes (snmp requests, arp table copies,...)
but found nothing that could lead to this behavior.
Normal icmp respons times (seen from our mgmt station) are a couple msec
but every minute this rises till 1000 ms and more.

We also see a spike in cpu usage at the same time and I tried to
determine what process uses all this cpu and got the following result :
Normal : cpu around 11%
ldus220#sh proc cpu | excl 0.0
CPU utilization for five seconds: 11%; one minute: 13%; five minutes:
14%
PID       5Sec    1Min     5Min Process
1         0.1%    2.7%     3.7% kernel
12312     7.6%    7.0%     7.4% ios-base
12329     1.8%    1.6%     1.2% tcp.proc
12330     0.6%    0.5%     0.5% udp.proc
12331     0.1%    0.2%     0.2% iprouting.iosproc
12332     0.1%    0.1%     0.1% cdp2.iosproc

Hit : cpu above 40%, caused by kernel process
ldus220#sh proc cpu | excl 0.0
CPU utilization for five seconds: 46%; one minute: 16%; five minutes:
14%
PID       5Sec    1Min     5Min Process
1        35.6%    6.3%     4.5% kernel
12312     7.1%    7.1%     7.4% ios-base
12329     1.0%    1.5%     1.2% tcp.proc
12330     0.3%    0.5%     0.5% udp.proc
12331     0.1%    0.2%     0.2% iprouting.iosproc
12332     0.1%    0.1%     0.1% cdp2.iosproc

Using cpu detail, I can see it is process id 17 that is hit but I don't
have a clue what this process does. How can I find what this process
does and if it is internal kernel housekeeping that is causing this or
an external cause ?
ldus220#sh proc cpu detail | excl 0.0
CPU utilization for five seconds: 46%; one minute: 21%; five minutes:
18%
PID/TID   5Sec    1Min     5Min Process             Prio  STATE
CPU
1        33.9%    5.7%     4.3% kernel
26d20h
       1  54.2%   79.5%    81.6% [idle thread]          0  Ready
1355d
     17  33.9%    5.7%     4.2%                       10  Running
6d07h
12312     7.3%    9.8%     9.1% ios-base
106d

Any info or pointer to info would be appreciated.

Wim Holemans
Network Services
University of Antwerp

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/