[nsp] CEF scanner eating all CPU

[jlewis at lewis.org]

I upgraded several 3640 DSL agg routers last night from 
c3640-js-mz.122-12.bin to c3640-js-mz.122-17a.bin.  Configs are all fairly 
similar...T1 IMA cards taking ATM T1's from the ILEC for mostly PPPoA DSL 
customers, some of the routers have some bridged DSL customers as well.
All are MPLS VPN PE routers and use BGP to exchange vrf routes.

One particular router has CEF scanner eating all available CPU time.  
After the upgrade, CPU utilization shot up from about 15% to 99%, stayed 
there for about 8 hours, dropped to normal levels (15-20%) for 4 hours, 
then shot back up to 99%.

Nothing in the config other than boot variable was changed when going from
122-12 to 122-17a.  There's plenty of free memory (75mb).  What could
cause CEF Scanner to do this?

hmm...while typing this message, it just went back to normal again.

I've tried clearing cef, turning cef off briefly, but when reenabled, it 
went back to 99% cpu.

Here's what it looks like when we're at 99% cpu.
#sh proc cpu | incl CEF
  93        4524       993       4555  0.32%  0.30%  0.32%   0 CEF process
 113      610180      8837      69048 64.48% 63.94% 61.92%   0 CEF Scanner

And right now...
#sh proc cpu | incl CEF
  93        7124      2378       2995  0.32%  0.25%  0.28%   0 CEF process       
 113      828288     12033      68834  0.00%  0.39% 18.57%   0 CEF Scanner      

The odd thing is, even when it was at 99% cpu, you wouldn't have guessed 
it from CLI responsiveness.

----------------------------------------------------------------------
 Jon Lewis *jlewis at lewis.org*|  I route
 System Administrator        |  therefore you are
 Atlantic Net                |  
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________