[c-nsp] 7206VXR Big CPU Increase After 12.4 Upgrade

[Mark Taylor]

Justin,

Thanks for the suggestions, on each point:

>> This is looking very much like cpu could potentially increase more 
>> quickly
>> than traffic. The jump in cpu between images is beginning to concern me a
>> little. Has anybody else experienced this? Is it normal?
>
> I can't speak specifically to the resource footprint of IOS 12.4, but a
> few things you should look at would be:
>
> 1. your logs - is anything interesting being logged that could yield clues
> to your increased CPU utilization?  You may need to turn the logging
> severity level down to 'debugging' if it isn't there already.  If you
> don't do so, you probably want to enable logging of OSPF adjacency and BGP
> neighbor changes.  Flapping here could indicate a lower-level problem like
> an unstable interface, but the flapping can certainly tear up some CPU
> cycles...

We log these type of events and there aren't any flapping sessions, 
everything appears to be stable. However since the image upgrade there have 
been a 3 or 4 of these per hour, always for Gig0/3
:
%IF-4-BACKWARD_COUNTERS: Corrected for backward rxtx_errors counters (6 -> 
4) on GigabitEthernet0/3

This didn't happen before under 12.3. I'm not sure if it is indicative of a 
particular issue or not yet. I found this explanation in an old doc for 
12.0:

"The interface specified in the message has a packet counter that has 
decreased in number. This condition can occur if a packet is counted and 
then dropped. This event was detected and corrected."

It does say contact TAC if it persists.

> 2. Does a 'show proc cpu' show one or two specific processes chewing up
> inordinate amounts of CPU time?  If you see lots of time being dedicated
> to the IP Input process, I'd double-check that CEF is enabled  everywhere
> that it should be.  'show proc cpu hist' may also reveal some interesting
> information.

router#sh processes cpu sorted
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 171      594544      3809     156089  5.81%  0.73%  0.55%   0 BGP Scanner
  37      131660     12717      10353  0.32%  0.16%  0.15%   0 Net 
Background
  41      352508     44419       7935  0.32%  0.31%  0.32%   0 Per-Second 
Jobs
  56      250412    879171        284  0.16%  0.17%  0.16%   0 IP Input
 170      133388    176097        757  0.16%  0.10%  0.09%   0 BGP I/O
 169      159044    469384        338  0.08%  0.14%  0.15%   0 BGP Router
 153        3164   1360731          2  0.08%  0.08%  0.08%   0 PPP manager
  25      116700    458778        254  0.08%  0.08%  0.08%   0 ARP Input
  80        1504    121532         12  0.08%  0.03%  0.02%   0 TCP Timer
  22          36     44180          0  0.08%  0.00%  0.00%   0 IPC Deferred 
Por

I think CEF is enabled everywhere, there isn't anything more specific on any 
interface config that indicates otherwise.

We have turned off netflow export now, removing the interface "ip 
route-cache flow" and other ip "flow-export" commands. This has saved around 
7-8 percent cpu, so we now have around a 10 percent increase above 12.3, but 
now without netflow.

> 3. If you're running flow switching, is there anything in the flow cache
> that indicates traffic hitting the router that could jack up the CPU?
> 4. Look at 'show mem sum' to see if perhaps you're running into a memor
> leak or something?  If a router gets very low on memory, it may try to
> cannibalize some processes (moost notably CEF) to conserve memory.

We have 512Mb in this box. Still 222Mb free and stable on our snmp graphing, 
so looks ok.

> 5. Did the upgrade to 12.4 inadvertantly enable something in your config
> that you didn't want, or disable something you did want?

Nothing obvious that I can see from the running config.

> This isn't everything, but it should be enough to get you started.

Again, thanks for the suggestions, I've also sent a copy of the config 
across to Rodney.
Mark.