[c-nsp] 3550 High CPU - nothing in proc cpu

[Hector Herrera]

Thank you for your responses.

I collected the commands to run the next time the cpu utilization
spikes.  I did manage to capture the output of 'show cef
not-cef-switched' and it shows a very large number under the
"unsupported" column.  All the other columns are zero.

Reading on the list archives I found a few commands to diagnose the
"unsupported" column and according to the output, it appears that it's
caused by TTL-expired being send to the cpu for processing.  Does this
mean that the hardware can't handle the TTL expired load or that
TTL-expired messages are strictly a software process on this hardware
(3550-12t)?

If I have such a large number of TTL-expired messages, does that mean
I have a routing loop somewhere?  If so, I have three uplink
interfaces, how do I find out which interface is causing the punts?

Here is the output from the commands I ran:

van-hc16-423-router#show ip cef switching stat

       Reason                          Drop       Punt  Punt2Host
RP LES No route                           0          0         37
RP LES Packet destined for us             0     273716          0
RP LES No adjacency                    8587          0          0
RP LES TTL expired                        0          0    1676276
RP LES Unclassified reason                1          0          0
RP LES Neighbor resolution req       210055          3          0
RP LES Total                         218643     273719    1676313

All    Total                         218643     273719    1676313
van-hc16-423-router#show ip cef switching stat feature
IPv4 CEF input features:
       Feature                Drop    Consume       Punt  Punt2Host Gave route
Total                            0          0          0          0          0

IPv4 CEF output features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF post-encap features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF for us features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF punt features:
       Feature                Drop    Consume       Punt  Punt2Host    New i/f
Total                            0          0          0          0          0

IPv4 CEF local features:
       Feature                Drop    Consume       Punt  Punt2Host Gave route
Total                            0          0          0          0          0
van-hc16-423-router#sh ip arp summ
16 IP ARP entries, with 0 of them incomplete
van-hc16-423-router#sh sdm prefer
 The current template is the routing extended-match template.
 The selected template optimizes the resources in
 the switch to support this level of features for
 16 routed interfaces and 1K VLANs.

 number of unicast mac addresses:   6K
 number of igmp groups:             6K
 number of qos aces:                1K
 number of security aces:           1K
 number of unicast routes:          12K
 number of multicast routes:        6K

van-hc16-423-router#sh ip route summary
IP routing table name is Default-IP-Routing-Table(0)
IP routing table maximum-paths is 32
Route Source    Networks    Subnets     Overhead    Memory (bytes)
connected       0           1           64          152
static          0           0           0           0
bgp 4280        0           0           0           0
  External: 0 Internal: 0 Local: 0
internal        1                                   1172
Total           1           1           64          1324
van-hc16-423-router#sh ip route vrf PublicRouter sum
van-hc16-423-router#sh ip route vrf PublicRouter summary
IP routing table name is PublicRouter(1)
IP routing table maximum-paths is 32
Route Source    Networks    Subnets     Overhead    Memory (bytes)
connected       0           4           256         608
static          1           0           128         152
bgp 4280        1274        1134        154112      367036
  External: 2408 Internal: 0 Local: 0
internal        66                                  77352
Total           1341        1138        154496      445148
van-hc16-423-router#


On Sat, Nov 14, 2009 at 6:59 PM, Harald Firing Karlsen
<maillist at thelan.no> wrote:
> Hector Herrera wrote:
>>
>> During a high network usage event, the cpu load increased to 90%
>> sustained, while a 'show processes cpu' did not reveal any culprits.
>> I suspected IP Input may be consuming a high amount of cpu, but it was
>> only at 2.7%
>>
>> The 3550 is working as a L3 router with two static entries for the
>> default gw (for load balancing on our uplink).
>>
>> Traffic levels at the time of the high cpu usage were ~120Mbps.
>>
>> I also examined broadcast packet counts and traffic destined for the
>> router itself.  They also did not reveal anything out of the ordinary.
>>
>> Do you have any suggestions on what I should be looking at to
>> determine the source of the high cpu usage?
>>
>
> What did the topmost line in the "show processes cpu" say? At the five
> second average you got two values; one is for interrupts and the other is
> for process cpu usage. My guess is you was seing a lot of interrupts which
> means traffic was punted to the CPU. Take a look at some of the other
> threads on c-nsp to find out what kind of traffic was being punted ("show
> cef not-cef-switched" is a good start).
>
> Hope this was helpfull
>
> --
> Harald Firing Karlsen
>



-- 
Hector Herrera
President
Pier Programming Services Ltd.