[c-nsp] CEF Scanner eating CPU in Supervisor 720
Rodney Dunn
rodunn at cisco.com
Fri Jun 9 08:39:21 EDT 2006
On Fri, Jun 09, 2006 at 08:56:47AM +0200, Peter Salanki wrote:
> -----BEGIN PGP SIGNED MESSAGE-----
> Hash: SHA1
>
> I think the problem s not at all MPLS related. I did a perl hack that
> paresed the output of sh ip cef event new and added static arp on the
> hosts which flapped rapidly, 7 IPs had abnormal activity. The load is
> now down to a more acceptable level of 20% avg. I could remove the
> statics and isable MPLS on the core facing interfaces just to make
> sure that MPLS has nothing to do with it if you want.
I hate asking for things from production networks but if you could
do that without too much trouble that would be a good data point
to have.
Also, those macs are they on the downstream interfaces (non mpls
enabled interfaces)?
Are there any routes resolving through those arp/mac's on those interfaces?
How are the arps changing?
Do you have any
> case about this, and/or any plans of "fixing" it?
I need to understand a little more about what the root problem is first.
I don't like the
> thought of directly connected kiddies being able to drain all cpu on
> my (imo. not cheap) sup720-3bxl by just stealing eachothers IP
> addresses.
Help me understand a bit more about what is actually going on to trigger
it and we'll see what we can do.
I'm not a l2 person but isn't there stuff about securing macs on ports, etc.?
>
> 9 jun 2006 kl. 00.32 skrev Rodney Dunn:
>
> > One trick is you can do a 'sh ip cef ev new' and do it over and
> > over. See which ones are flapping.
> >
> > How many routes do you have?
> >
> > Can you turn off logging to the console: no logg con
> >
> > and run a couple of mpls debugs and let's see what that says:
> >
> > debug mpls lfib cef
> > debug mpls lfib enc
> >
> > Set the lot to a couple of meg.
> >
> > Rodney
> >
> > On Thu, Jun 08, 2006 at 11:13:02PM +0200, Peter Salanki wrote:
> >> The CEF Scanner is now eating almost all CPU :/
> >>
> >> The events table doesn't look any particular to me,
> >> --SNAP--
> >>
> >> +00:00:00.000: 81.170.148.226/32 ADJ (Vl4001) update
> >> [OK]
> >> +00:00:00.024: 195.178.160.138/32 ADJ (Vl19) update
> >> [OK]
> >> +00:00:00.052: 81.170.138.13/32 ADJ (Vl604) update
> >> [OK]
> >> +00:00:00.232: 81.170.152.129/32 ADJ (Vl4003) update
> >> [OK]
> >> +00:00:00.240: 81.170.148.118/32 ADJ (Vl4001) update
> >> [OK]
> >> +00:00:00.304: 81.170.149.246/32 ADJ (Vl4001) update
> >> [OK]
> >> +00:00:00.320: 81.170.152.50/32 ADJ (Vl4003) update
> >> [OK]
> >> +00:00:00.380: 81.170.154.117/32 ADJ (Vl4004) update
> >> [OK]
> >> +00:00:00.388: 213.136.56.90/32 ADJ (Vl39) update
> >> [OK]
> >> +00:00:00.400: 81.170.136.79/32 ADJ (Vl504) update
> >> [OK]
> >> +00:00:00.416: 195.178.160.173/32 ADJ (Vl19) update
> >> [OK]
> >> +00:00:00.512: 81.170.164.163/32 ADJ (Vl4009) update
> >> [OK]
> >> +00:00:00.728: 81.170.130.75/32 ADJ (Vl204) update
> >> [OK]
> >> +00:00:00.736: [Default] 199.3.108.0/24 NBD modified
> >> [OK]
> >> +00:00:00.736: [Default] 199.3.109.0/24 NBD modified
> >> [OK]
> >> +00:00:00.820: 195.178.186.24/32 ADJ (Vl666) update
> >> [OK]
> >> +00:00:00.832: 81.170.160.3/32 ADJ (Vl4007) update
> >> [OK]
> >> +00:00:00.868: 81.170.164.33/32 ADJ (Vl4009) update
> >> [OK]
> >> +00:00:00.944: 81.170.132.159/32 ADJ (Vl304) update
> >> [OK]
> >> +00:00:00.952: 81.170.128.77/32 ADJ (Vl104) update
> >> [OK]
> >> +00:00:01.008: 81.170.149.246/32 ADJ (Vl4001) update
> >> [OK]
> >> +00:00:01.128: 194.68.123.141/32 ADJ (Vl15) update
> >> [OK]
> >> --More--
> >>
> >>
> >> 8 jun 2006 kl. 19.40 skrev Rodney Dunn:
> >>
> >>> Are you running MPLS on the box?
> >>>
> >>> Check the sh ip cef event outut and see if you have a /32 ADJ
> >>> for a mac constantly changing. That's the most common trigger
> >>> I've seen for the scanner running high.
> >>>
> >>> You are forcing CEF to constantly reresolve prefixes.
> >>>
> >>> Rodney
> >>>
> >>> On Thu, Jun 08, 2006 at 02:23:22PM +0200, Peter Salanki wrote:
> >>>> -----BEGIN PGP SIGNED MESSAGE-----
> >>>> Hash: SHA1
> >>>>
> >>>> Hello,
> >>>>
> >>>> Process "CEF Scanner" is eating average 60% of the CPU on one of my
> >>>> Sup720-3BXL. This leads to snmp responses being delayed and full
> >>>> BGP
> >>>> updates taking a long time. I have not seen this on any of my other
> >>>> sup720s. What differs this box from the rest is that this one has a
> >>>> lot of directly connected hosts ~10 SVIs with 300 hosts each
> >>>> (on /23
> >>>> subnets). I have tried setting arp timeout to 1200 on those SVIs,
> >>>> which resulted in a small CPU utilization decrease. What can I
> >>>> do to
> >>>> calm down the CEF Scanner? I'm running 12.2(18)SXF4.
> >>>>
> >>>> CPU utilization for five seconds: 44%/4%; one minute: 38%; five
> >>>> minutes: 38%
> >>>> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY
> >>>> Process
> >>>> 119 103495040 719635 143819 35.40% 23.87% 21.54% 0 CEF
> >>>> Scanner
> >>>>
> >>>> Sincerely
> >>>>
> >>>> Peter Salanki
> >>>> Chief Network Engineer
> >>>> Bahnhof AB (AS8473)
> >>>> www.bahnhof.se
> >>>> Office: +46855577132
> >>>> Cell: +46709174932
> >>>>
> >>>>
> >>>> -----BEGIN PGP SIGNATURE-----
> >>>> Version: GnuPG v1.4.2.2 (Darwin)
> >>>>
> >>>> iD8DBQFEiBa7iQKhdiFGiogRAr9aAJ9W+rryMPcg5qnAYrYTU9jbRg8PFgCdHDA3
> >>>> QjIpm/Yk7kuf4VjZN5MqDq8=
> >>>> =O029
> >>>> -----END PGP SIGNATURE-----
> >>>> _______________________________________________
> >>>> cisco-nsp mailing list cisco-nsp at puck.nether.net
> >>>> https://puck.nether.net/mailman/listinfo/cisco-nsp
> >>>> archive at http://puck.nether.net/pipermail/cisco-nsp/
> >>
> >> Med vänliga hälsningar
> >>
> >> Peter Salanki
> >> Nätansvarig
> >> Bahnhof AB (AS8473)
> >> www.bahnhof.se
> >> Kontor: +46855577132
> >> Mobil: +46709174932
> >>
>
> Sincerely
>
> Peter Salanki
> Chief Network Engineer
> Bahnhof AB (AS8473)
> www.bahnhof.se
> Office: +46855577132
> Cell: +46709174932
>
>
> -----BEGIN PGP SIGNATURE-----
> Version: GnuPG v1.4.2.2 (Darwin)
>
> iD8DBQFEiRuviQKhdiFGiogRApBTAJ9wZqTm+iAVcO4AgccM7OUvfCjlyACgltZ3
> Mzw94W8HEF7+RGBjmObwqXc=
> =EAim
> -----END PGP SIGNATURE-----
More information about the cisco-nsp
mailing list