[c-nsp] 3750 CPU Usage; TCAM Exhaustion?

[Carl Jones]

Hi all,

I'm running 3x 3750G-24 in a stack. I'm seeing high CPU usage e.g.:

CPU utilization for five seconds: 69%/24%; one minute: 63%; five minutes: 74%
 PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
 201    50885085   5144152       9891 17.41% 17.60% 16.68%   0 Spanning Tree
  73     9841381   3782242       2601  9.26%  7.11%  6.23%   0 HLFM address lea
 134     6355962    267005      23804  1.75%  0.93%  0.95%   0 HL3U bkgrd proce
 301     3115451    273365      11396  1.43%  1.13%  0.96%   0 CEF: IPv4 proces
  60     5452829     80160      68024  1.27%  1.78%  1.81%   0 Adjust Regions
 192     2715297   3543754        766  1.27%  0.95%  1.15%   0 IP Input
 133     3143317   3476055        904  0.95%  0.84%  1.08%   0 Hulc LED Process
   9     2401841   2057237       1167  0.95%  0.71%  0.74%   0 ARP Input
 153     1479640    198328       7460  0.63%  0.58%  0.55%   0 PI MATM Aging Pr
  96      785363    313609       2504  0.63%  0.32%  0.31%   0 hpm counter proc
 197     2776025   4243834        654  0.47%  0.63%  0.75%   0 ADJ resolve proc
 142      378847    243608       1555  0.15%  0.13%  0.13%   0 HRPC qos request
  92      348283   1281519        271  0.15%  0.12%  0.14%   0 hpm main process
 141      353779     41162       8594  0.15%  0.10%  0.14%   0 HQM Stack Proces

23 ports are configured as trunks (to 2950/3550/2960s). They show
normal CPU utilization. Enabling spanning tree debugging shows nothing
out of the ordinary (just regular BPDUs). They are all attached to the
first switch (nothing in use on the other two).

There are ~80 VLANs that terminate on the stack and two routed interfaces.

Currently I see:

core-dal#sh platform tcam utilization

CAM Utilization for ASIC# 0                      Max            Used
                                             Masks/Values    Masks/values

 Unicast mac addresses:                        400/3200        373/2911
 IPv4 IGMP groups + multicast routes:          144/1152          6/26
 IPv4 unicast directly-connected routes:       400/3200        373/2911
 IPv4 unicast indirectly-connected routes:    1040/8320        114/848
 IPv4 policy based routing aces:               384/512           1/2
 IPv4 qos aces:                                768/768         324/324
 IPv4 security aces:                          1024/1024         31/31

core-dal#sh ip arp sum
7222 IP ARP entries, with 1011 of them incomplete

Currently using the routing template. Unfortunately that did not seem
to help with the CPU usage (nor did 'no ip unreachables' on our
VLANs).

core-dal#sh sdm prefer
 The current template is "desktop routing" template.

Using a fairly recent IOS on them:
*    1 28    WS-C3750G-24TS     12.2(50)SE2           C3750-IPSERVICESK9-M

I suspect I may be seeing TCAM exhaustion. Any suggestions on how I
can confirm or avoid that?

Regards,
Carl