[nsp] Cisco 2500 Traffic Limit and high cpu utilization.

[Church, Chuck]

'Sh int stat' will give you a nice breakdown of packets being fast versus process switched on the interfaces.  A well-performing router without ACLs or NAT can see maybe 99% or more packets being fast switched.  The RED you have configured may be affecting it though.  I'd make it a priority to get CEF running correctly, so maybe you're hitting a bug that's been fixed in a later version, like Terry said.  Check the buffers to see if you're getting many misses and/or trims.  If you are using ACLs, try re-arranging the entries to make most of the hits occur at the top of the ACL.  Routing or policy routing bad packets to null might be a better option than an interface ACL.  From what I've read, a 2500 will process switch at most 800 pps, and fast switch about 4000 pps.  Your 28% IP input value with about 475 pps passing through the router indicates about 224 pps being process switched.  That's a pretty high percentage.  What's the whole config look like?

Chuck Church
Lead Design Engineer
CCIE #8776, MCNE, MCSE
Wam!Net Government Services - Design & Implementation Team
13665 Dulles Technology Dr. Ste 250
Herndon, VA 20171
Office: 703-480-2569
Cell: 703-819-3495
cchurch at wamnetgov.com
PGP key: http://pgp.mit.edu:11371/pks/lookup?op=index&search=cchurch%40wamnetgov.com

> -----Original Message-----
> From: Terry Baranski [mailto:tbaranski at mail.com]
> Sent: Wednesday, March 17, 2004 7:03 PM
> To: 'Mehmet Ali Suzen'
> Cc: cisco-nsp at puck.nether.net
> Subject: RE: [nsp] Cisco 2500 Traffic Limit and high cpu utilization.
> 
> 
> > Dear List,
> > I would like to address the recent issue again. (I was a way 
> > for a week or so.)
> > Last IoS upgrade has been done last summer. So it might be the case.
> > Here is a specific real time example When CPU usage is as follows;
> > router1#show proc cpu | include CPU
> > CPU utilization for five seconds: 81%/39%; one minute: 77%; 
> > five minutes: 79%
> > router1#show proc cpu | include IP Input
> >   21   362809868  30390192      11938 28.04% 26.10% 25.32%   
> > 0 IP Input
> 
> So the process-switched packets are using around 28% of the CPU,
> interrupts (usually fast- or CEF-switched packets) are using 39%, and
> the rest (~14%) is used by 1 or more additional processes 
> (which should
> show up in 'show proc cpu').  
> 
> > At that point traffic on the ethernet :
> >  Queueing strategy: random early detection(RED)
> >   5 minute input rate 424000 bits/sec, 249 packets/sec
> >   5 minute output rate 923000 bits/sec, 223 packets/sec
> > And traffic on the serial interface 
> > Queueing strategy: random early detection(RED)
> >   5 minute input rate 703000 bits/sec, 127 packets/sec
> >   5 minute output rate 138000 bits/sec, 107 packets/sec
> 
> These numbers are potentially more telling.  If all the traffic on the
> serial interface goes through the Ethernet interface as well, 
> the router
> is pushing around 1.35Mbits of traffic.  This is lower than the
> theoretical numbers I gave originally but with all the 
> features you have
> enabled I wouldn't be surprised if you're near the router's 
> limits.  If
> the traffic that's being processed-switched could be 
> fast-switched, you
> might get some benefit, but I'm not aware of a command that 
> can indicate
> why certain packets are being punted to the process switching path. (A
> "show interface switching" will give you an idea of how many 
> packets are
> affected.)  Are there any packets coming in and going out the same
> interface?  If so, "ip route-cache same-interface" may help.
> 
> > Serial has a HDSL connection of 1 Mb/s bandwith total.  We have
> > a LAN and dialup connections. At this point network slows down even
> > tho we didn't fullfill the bandwitdh. 
> 
> The above output indicates that you're using 70% of the serial link's
> input bandwidth and 80% of the router's CPU.  There's 
> breathing room in
> both these numbers so it's not clear why things are slowing 
> down at this
> point.  But we should clarify exactly what you mean by "slowing down".
> There's not a lot of spare bandwidth available per the above 
> output, and
> the burstiness of Internet traffic can make 5 minute averages
> misleadingly low anyway, so it could simply be link 
> saturation (or more
> precisely, near-saturation) that's the primary cause of the problem.
> (Though it doesn't look like you'll get much more bandwidth out of the
> router anyway without getting CEF working or making some 
> other change.)
> 
> That being said, it's near impossible for us to diagnose why 
> CEF brings
> your traffic to a halt -- some things just aren't meant to be
> troubleshooted over a mailing list.  TAC is your best bet if 
> you have a
> contract; if not, upgrading IOS to a known stable version 
> (others on the
> list may have specific recommendations depending on what 
> you're running
> now) is probably the way to go.  But as I mentioned before, 
> in your case
> it's not a guarantee that CEF will significantly increase the 
> amount of
> throughput your router can handle.  But I think a link upgrade is
> advisable regardless -- 70% average sustained usage is at or 
> above what
> I'd consider "time to upgrade", and I think most here would agree with
> that.  
> 
> (As for your other message, ACKs and PSHs are perfectly normal in TCP.
> Every packet except the initial SYN is going to have the ACK flag set,
> and the PSH flag is also very common.)
> 
> -Terry
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>