[nsp] Re: CEF scanner eating all CPU

[jlewis at lewis.org]

On Tue, 22 Jul 2003, Siva Valliappan wrote:

>    the CEF Scanner process usually runs when it needs to compute a
> new L2 adjacency because of a MAC rewrite or something along those
> lines.  so things to look at is if you have something that is
> constantly changing your L2 address for a link/address/etc.  e.g.
> do you have a default route?  is it pointed at an ip address that
> is constantly changing MAC addresses, etc.

The one doing this right now does have a default route pointing to the 
other IP in the /30 configured on its ethernet interface.  It used to be 
directly connected via crossover cable to an ethernet port on a 7206, but 
it was recently moved to a switch port on a 2924xl as a VLAN (so we didn't 
have to renumber the devices after removing the direct connection).  I 
don't think that has anything to do with it as that rearranging was done a 
week or so before the IOS upgrade with no ill effects and our other 3640 
that's done this has had no such changes.

> you might want to configure
> 
> conf t
>  ip cef table event-log
> end
> 
> and then issue a
> 
> "show ip cef event"
> "show ip cef event detail"

I added that config when I received your message.  Yesterday, late 
afternoon, this router went back to 99% CPU, 

gsvlflma-3640#sh proc cpu | incl CEF
  93      582004    405754       1434  0.16%  0.38%  0.37%   0 CEF process
 113    76737540   1718008      44667 80.19% 76.77% 77.92%   0 CEF Scanner
gsvlflma-3640#sh ip cef event det
CEF table events (storage for 0 events, 0 events recorded)

It's been running 99% CPU for the past 18 hours or so.  Nobody seems to 
have noticed/complained.  CLI interactive response is fine.

CPU utilization for five seconds: 99%/17%; one minute: 99%; five minutes: 99%

> it might possibly be CSCdz59236.  but hard to tell without some
> indepth troubleshooting.  you may want to consider opening a TAC
> case so that a TAC engineer can work with you in running CPU profiling
> to see what the CEF Scanner process is actually doing.  this will
> help us figure out if this is a known issue or something new.

I guess I'll try opening a case later today.  The unit does use lots of 
virtual templates / virtual interfaces...but VTEMPLATE Backgr hardly using 
any CPU...just CEF Scanner.

I just noticed I am getting frequent messages like these:

Jul 24 12:24:14: %TFIB-7-SCANSABORTED: TFIB scan not completing.  MAC 
string updated.
Jul 24 12:24:45: %TFIB-7-SCANSABORTED: TFIB scan not completing.  MAC 
string updated.
Jul 24 12:25:16: %TFIB-7-SCANSABORTED: TFIB scan not completing.  MAC 
string updated.


----------------------------------------------------------------------
 Jon Lewis *jlewis at lewis.org*|  I route
 System Administrator        |  therefore you are
 Atlantic Net                |  
_________ http://www.lewis.org/~jlewis/pgp for PGP public key_________