[nsp] Cisco 2500 Traffic Limit and high cpu utilization.

[Terry Baranski]

> Dear List,
> I would like to address the recent issue again. (I was a way 
> for a week or so.)
> Last IoS upgrade has been done last summer. So it might be the case.
> Here is a specific real time example When CPU usage is as follows;
> router1#show proc cpu | include CPU
> CPU utilization for five seconds: 81%/39%; one minute: 77%; 
> five minutes: 79%
> router1#show proc cpu | include IP Input
>   21   362809868  30390192      11938 28.04% 26.10% 25.32%   
> 0 IP Input

So the process-switched packets are using around 28% of the CPU,
interrupts (usually fast- or CEF-switched packets) are using 39%, and
the rest (~14%) is used by 1 or more additional processes (which should
show up in 'show proc cpu').  

> At that point traffic on the ethernet :
>  Queueing strategy: random early detection(RED)
>   5 minute input rate 424000 bits/sec, 249 packets/sec
>   5 minute output rate 923000 bits/sec, 223 packets/sec
> And traffic on the serial interface 
> Queueing strategy: random early detection(RED)
>   5 minute input rate 703000 bits/sec, 127 packets/sec
>   5 minute output rate 138000 bits/sec, 107 packets/sec

These numbers are potentially more telling.  If all the traffic on the
serial interface goes through the Ethernet interface as well, the router
is pushing around 1.35Mbits of traffic.  This is lower than the
theoretical numbers I gave originally but with all the features you have
enabled I wouldn't be surprised if you're near the router's limits.  If
the traffic that's being processed-switched could be fast-switched, you
might get some benefit, but I'm not aware of a command that can indicate
why certain packets are being punted to the process switching path. (A
"show interface switching" will give you an idea of how many packets are
affected.)  Are there any packets coming in and going out the same
interface?  If so, "ip route-cache same-interface" may help.

> Serial has a HDSL connection of 1 Mb/s bandwith total.  We have
> a LAN and dialup connections. At this point network slows down even
> tho we didn't fullfill the bandwitdh. 

The above output indicates that you're using 70% of the serial link's
input bandwidth and 80% of the router's CPU.  There's breathing room in
both these numbers so it's not clear why things are slowing down at this
point.  But we should clarify exactly what you mean by "slowing down".
There's not a lot of spare bandwidth available per the above output, and
the burstiness of Internet traffic can make 5 minute averages
misleadingly low anyway, so it could simply be link saturation (or more
precisely, near-saturation) that's the primary cause of the problem.
(Though it doesn't look like you'll get much more bandwidth out of the
router anyway without getting CEF working or making some other change.)

That being said, it's near impossible for us to diagnose why CEF brings
your traffic to a halt -- some things just aren't meant to be
troubleshooted over a mailing list.  TAC is your best bet if you have a
contract; if not, upgrading IOS to a known stable version (others on the
list may have specific recommendations depending on what you're running
now) is probably the way to go.  But as I mentioned before, in your case
it's not a guarantee that CEF will significantly increase the amount of
throughput your router can handle.  But I think a link upgrade is
advisable regardless -- 70% average sustained usage is at or above what
I'd consider "time to upgrade", and I think most here would agree with
that.  

(As for your other message, ACKs and PSHs are perfectly normal in TCP.
Every packet except the initial SYN is going to have the ACK flag set,
and the PSH flag is also very common.)

-Terry