[c-nsp] Cat6500 with ION - CPU spiking after long uptime?

[Will Hargrave]

Hi all,

I have 4x Cat6500/Sup720 running (packed) modular IOS - 12.2(18)SXF9.
Uptime is about 5 months. Nothing special in terms of config - layer-two
switching only, STP, 67xx series linecards etc.

I've noticed in the past few weeks that the control plane latency has
begun to spike heavily on a regular cycle of 60 seconds. By 'spiking
heavily' I mean ICMP latency going to >1sec for several seconds. This is
beginning to cause operational problems (SNMP times out, etc).

All the switches on the network do this on a regular cycle of 60
seconds, but of course at different times throughout the cycle, which
would indicate to me a software bug rather than something actually
happening on the network. Transit traffic is unaffected.

Examination of 'show proc cpu hist' is no use, I suspect something in
ION stops this showing the true state. The 'kernel' process seems to be
the one CPU spiking:

sw#show proc cpu | excl 0.0
CPU utilization for five seconds: 86%; one minute: 30%; five minutes: 23%
PID       5Sec    1Min     5Min Process
1        85.3%   26.3%    20.3% kernel
16408     0.7%    2.4%     2.1% ios-base
16425     0.2%    0.5%     0.3% tcp.proc

It seems to only affect management traffic but i'm worried if it gets
worse it may start to make things like STP break.
.

Anyone else seen something similar? I have a TAC case open.

At the moment, I'm considering just going to non-modular IOS, but I was
hoping to actually use the modularity features in the future. I'm quite
reluctant to make the jump to SXH at this stage.

Will