Re: [nsp] MSFC CPU Utilization Pegged at 99%....but a sh proc cpu doesnot reveal anything past 1%

From: Charles Spurgeon (ces@argus.gw.utexas.edu)
Date: Tue Feb 13 2001 - 10:45:04 EST


On Fri, Feb 02, 2001 at 05:03:36PM -0600, Don Wester wrote:
> I would not call them major problems, but I have some issues.
>
> Running 5.5(2) with a fairly large amount of mls cache entries (60K -
> 105K) depending on the location. I believe the cache tables tend to
> get corrupted after a couple of weeks. I do a 'sh mls' and it can
> take up to five minutes to get a display back and the numbers don't
> add up once it is received. We have done all the necessary tweeking,
> destination based entries, fast aging, etc, the tables are just that
> large.

Beware of 5.5(2) code. It has serious CPU overload bugs that occur
when there are high numbers of MLS entries with NDE enabled. Also,
once the CPU hits the roof it can get stuck there and stay pegged even
when the MLS/NDE load drops.

It sounds like you have encountered these bugs since it takes 5 min to
get displays and MLS entries don't add up. That's exactly what we saw
when the bugs hit. We also saw some routing failures, probably due to
broken entries in the MLS cache. Check "show proc" on your switch to
see what the load is. The "L3 Aging" process will be nailed if the bug
is hitting.

Assuming the CPU is not stuck at high load, you should be able to get
the load to drop by shutting off NDE. That will keep your net running
until you can replace the code.

BugIDs: CSCdr10379 (high CPU load under L3 aging and NDE), CSCds51525
(CPU stuck at high loads)

We're now running 5.5(4b) code which contains fixes for the CPU
overload due to high numbers of MLS entries and NDE, and for the CPU
stuck at high load condition. Tests in our lab showed a huge
improvement in handling high MLS/NDE loads in this code. Instead of
going to 100%, the CPU now loafs along at a few percent.

-Charles

Charles E. Spurgeon / UTnet
UT Austin Telecommunications and Internet Services
c.spurgeon@mail.utexas.edu / 512.475.9265



This archive was generated by hypermail 2b29 : Sun Aug 04 2002 - 04:12:29 EDT