[c-nsp] 3550 High CPU - nothing in proc cpu

[Hector Herrera]

On Sun, Nov 22, 2009 at 4:01 PM, e ninja <eninja at gmail.com> wrote:
> Hector,
>
> It is interesting that the cisco article tells you how to profile your cpu
> but not how to interpret the results ;-)
>
> There is only one way to interpret the results - contact Cisco to report the
> abnormality. They will have to decode the address/es using the symbol files
> for your device software which will reveal the culprit function/s. It should
> be pretty straight forward to isolate cause and rectify thereafter.

I did receive an email from someone at Cisco offering to look up the
functions.  Thank you  :-)  I can't wait to see the outcome.

> FYI, seeing CPU spikes to X% during high traffic is not abnormal for most
> non-distributed platforms that are groaning under an inappropriate switching
> algorithm or overload.
>
> Out of curiosity, is 40% cpu utilization above your benchmarked baseline? If
> no, ignore. Also, any alignment corrections? device#sh align

Your question made me go back and review my notes.  CPU load appears
to be directly correlated to the amount of traffic on the switch.  At
50Mbps the cpu load is 40%, at 200Mbps the load is 100%.  At 20Mbps
the load (currently) is 10%

I wonder if expecting the 3550-12t platform to handle more than
200Mbps is too much to ask?  The specs indicate it's capable of
17Mpps.  According to the logs, at 200Mbps (with the 100% cpu load)
the router was forwarding 45Kpps, much less than the advertised
capacity.

Perhaps it is a bad design on my part.

I learned that the 3550-12t has three forwarding engines, one for each
set of four interfaces (0/1 to 0/4, 0/5 to 0/8 and 0/9 to 0/12)

With that in mind, I configured a VRF with four routed interfaces (0/1
to 0/4).  0/3 is a BGP interface.  0/4 is the LAN.  0/1 and 0/2 are
configured in a load-balancing static default route.  The forwarding
engine is configured to use per-destination load-balancing.

If I understand it correctly, Cisco's load-balancing in
per-destination mode has an initial cost when the destination is not
present in the routing table, but once it is there, CEF takes care of
the forwarding.  Since the traffic on the network is stream based
(Live video streams), with very few new destinations (less than 500
per hour), but a constant stream of packets which should be handled by
CEF.

So I'm still at a loss ... Should I expect better performance from the
3550-12t or am I trying to squeeze blood out of stones?

Hector