[c-nsp] 3750 CPU Usage; TCAM Exhaustion?

[Michael Schedrin]

2009/8/5 Carl Jones <carl at outerloop.net>

> Hi all,
>
> I'm running 3x 3750G-24 in a stack. I'm seeing high CPU usage e.g.:
>
> CPU utilization for five seconds: 69%/24%; one minute: 63%; five minutes:
> 74%
>  PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY Process
>  201    50885085   5144152       9891 17.41% 17.60% 16.68%   0 Spanning
> Tree
>  73     9841381   3782242       2601  9.26%  7.11%  6.23%   0 HLFM address
> lea
>  134     6355962    267005      23804  1.75%  0.93%  0.95%   0 HL3U bkgrd
> proce
>  301     3115451    273365      11396  1.43%  1.13%  0.96%   0 CEF: IPv4
> proces
>  60     5452829     80160      68024  1.27%  1.78%  1.81%   0 Adjust
> Regions
>  192     2715297   3543754        766  1.27%  0.95%  1.15%   0 IP Input
>  133     3143317   3476055        904  0.95%  0.84%  1.08%   0 Hulc LED
> Process
>   9     2401841   2057237       1167  0.95%  0.71%  0.74%   0 ARP Input
>  153     1479640    198328       7460  0.63%  0.58%  0.55%   0 PI MATM
> Aging Pr
>  96      785363    313609       2504  0.63%  0.32%  0.31%   0 hpm counter
> proc
>  197     2776025   4243834        654  0.47%  0.63%  0.75%   0 ADJ resolve
> proc
>  142      378847    243608       1555  0.15%  0.13%  0.13%   0 HRPC qos
> request
>  92      348283   1281519        271  0.15%  0.12%  0.14%   0 hpm main
> process
>  141      353779     41162       8594  0.15%  0.10%  0.14%   0 HQM Stack
> Proces
>
> 23 ports are configured as trunks (to 2950/3550/2960s). They show
> normal CPU utilization. Enabling spanning tree debugging shows nothing
> out of the ordinary (just regular BPDUs). They are all attached to the
> first switch (nothing in use on the other two).
>
> There are ~80 VLANs that terminate on the stack and two routed interfaces.
>
> Currently I see:
>
> core-dal#sh platform tcam utilization
>
> CAM Utilization for ASIC# 0                      Max            Used
>                                             Masks/Values    Masks/values
>
>  Unicast mac addresses:                        400/3200        373/2911

Look at "sh mac address-table count" Check "Total Mac Address Space
Available:"
3750 hat a table for 6000 mac adresses. If you stack 3*3750, this bundle
will also have table of 6000.

>
>  IPv4 IGMP groups + multicast routes:          144/1152          6/26
>  IPv4 unicast directly-connected routes:       400/3200        373/2911
>  IPv4 unicast indirectly-connected routes:    1040/8320        114/848
>  IPv4 policy based routing aces:               384/512           1/2
>  IPv4 qos aces:                                768/768         324/324
>  IPv4 security aces:                          1024/1024         31/31
>
> core-dal#sh ip arp sum
> 7222 IP ARP entries, with 1011 of them incomplete

Yes this proves information about lack of size of mac-address-table.

>
>
> Currently using the routing template. Unfortunately that did not seem
> to help with the CPU usage (nor did 'no ip unreachables' on our
> VLANs).
>
> core-dal#sh sdm prefer
>  The current template is "desktop routing" template.
>
> Using a fairly recent IOS on them:
> *    1 28    WS-C3750G-24TS     12.2(50)SE2           C3750-IPSERVICESK9-M
>
> I suspect I may be seeing TCAM exhaustion. Any suggestions on how I
> can confirm or avoid that?

You have two ways to avoid the problem. First - change sdm to "vlan". Second
- disassemble stack and make every switch use it's own mac-address-table and
tcam. You will summary have 6000*3 mac addresses table.

>
>
> Regards,
> Carl
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>