[f-nsp] 100% cpu utilization for 1-2 minutes every 30 minutes

[Eric A Louie]

I have a problem with one 2402 in my network.

For a few days, it will work fine, with less than 5% CPU utilization


Then, slowly, every 30 minutes, the CPU utilization will increase for a minute or two, then drop back to 5%

Eventually, we end up with 100% CPU spikes that cripple the router

I've replaced it and still see the same behavior.

When I reboot the router, it behaves for a short time (sometimes 3-5 days, sometimes 3-5 hours), and the behavior resumes.

We're running OSPF with two neighbors and I see this sequence of log messages on the target router sometimes before the cpu spikes, but there's not a 100% correlation between this and the spikes


Jun 20 08:29:25 ncy-sd-base1 OSPF: nbr state changed, rid 192.168.254.32, nbr addr 10.250.243.241, nbr rid 192.168.254.24, state full, triggered by ExchangeDone 
Jun 20 08:29:25 ncy-sd-base1 OSPF: nbr state changed, rid 192.168.254.32, nbr addr 10.250.198.233, nbr rid 192.168.254.25, state full, triggered by ExchangeDone 
Jun 20 08:29:24 ncy-sd-base1 OSPF: nbr state changed, rid 192.168.254.32, nbr addr 10.250.243.241, nbr rid 192.168.254.24, state full, triggered by SeqNumberMismatch: INIT bit set for database packet at FULL state 
Jun 20 08:29:24 ncy-sd-base1 OSPF: nbr state changed, rid 192.168.254.32, nbr addr 10.250.198.233, nbr rid 192.168.254.25, state full, triggered by SeqNumberMismatch: INIT bit set for database packet at FULL state 
Jun 20 08:26:31 ncy-sd-base1 OSPF: nbr state changed, rid 192.168.254.32, nbr addr 10.250.243.241, nbr rid 192.168.254.24, state initializing, triggered by 1-WayReceived 
Jun 20 08:26:31 ncy-sd-base1 OSPF: nbr state changed, rid 192.168.254.32, nbr addr 10.250.198.233, nbr rid 192.168.254.25, state initializing, triggered by 1-WayReceived 


On the adjacent neighbor, I see these messages
Jun 20 08:29:23 lsm-sd-base1 OSPF: interface state changed, rid 192.168.254.24, intf addr 10.250.243.241, state backup designated router 
Jun 20 08:29:21 lsm-sd-base1 OSPF: interface state changed, rid 192.168.254.24, intf addr 10.250.243.241, state designated router 
Jun 20 08:26:44 lsm-sd-base1 OSPF: interface state changed, rid 192.168.254.24, intf addr 10.250.243.241, state backup designated router 
Jun 20 08:26:26 lsm-sd-base1 OSPF: interface state changed, rid 192.168.254.24, intf addr 10.250.243.241, state designated router 


My NOC is turning off monitoring from one of our tools tonight to see if that's what's causing the problem.  Otherwise, does anyone have a good guess of an event that happens every 30 minutes that might cause a CPU spike?
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/foundry-nsp/attachments/20130620/38ff6554/attachment.html>