[c-nsp] 3550 high cpu & process switched traffic

Church, Chuck cchurch at multimax.com
Fri Aug 18 15:07:16 EDT 2006


The SDM indicates a maximum of 8K routes.  But your route table shows
almost 11,000 subnets.  That would explain a lot of CPU-forwarding,
wouldn't it? 


Chuck Church
Network Engineer
CCIE #8776, MCNE, MCSE
Multimax, Inc.
Enterprise Network Engineering
Home Office - 864-335-9473 
Cell - 864-266-3978
cchurch at multimax.com

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Tassos
Chatzithomaoglou
Sent: Friday, August 18, 2006 2:57 PM
To: Arie Vayner (avayner)
Cc: cisco-nsp
Subject: Re: [c-nsp] 3550 high cpu & process switched traffic

Here they are:

3550#sh tcam inacl 1 stat
Ingress ACL TCAM#1: Number of active labels: 11
Ingress ACL TCAM#1: Number of masks   allocated:   78, available:  130
Ingress ACL TCAM#1: Number of entries allocated:  189, available: 1475

3550#sh tcam outacl 1 stat
Egress ACL TCAM#1: Number of active labels: 4
Egress ACL TCAM#1: Number of masks   allocated:   15, available:  193
Egress ACL TCAM#1: Number of entries allocated:   35, available: 1629

3550#sh tcam pbr 1 stat
PBR TCAM#1: Number of active labels: 0
PBR TCAM#1: Number of masks   allocated:    0
PBR TCAM#1: Number of entries allocated:    0

3550#sh tcam qos 1 stat
QoS TCAM#1: Number of active labels: 5
QoS TCAM#1: Number of masks   allocated:    5, available:  411
QoS TCAM#1: Number of entries allocated:   16, available: 3312


3550#sh sdm prefer
  The current template is the default template.
  The selected template optimizes the resources in
  the switch to support this level of features for
  8 routed interfaces and 1K VLANs.

  number of unicast mac addresses:   5K
  number of igmp groups:             1K
  number of qos aces:                1K
  number of security aces:           1K
  number of unicast routes:          8K
  number of multicast routes:        1K


Arie Vayner (avayner) wrote on 18/8/2006 21:53:
> Sorry:
> 
> sh tcam inacl 1 stat
> sh tcam outacl 1 stat
> sh tcam pbr 1 stat
> sh tcam qos 1 stat
> 
> And 
> 
> show sdm prefer
> 
> Arie
> 
> -----Original Message-----
> From: Arie Vayner (avayner) 
> Sent: Friday, August 18, 2006 21:51 PM
> To: 'Tassos Chatzithomaoglou'
> Cc: cisco-nsp
> Subject: RE: [c-nsp] 3550 high cpu & process switched traffic
> 
> Tassos,
> 
> Can you please send:
> 
> sh tcam inacl 1 statistics
> show sdm prefer
> 
> Thanks
> Arie 
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Tassos
> Chatzithomaoglou
> Sent: Friday, August 18, 2006 21:43 PM
> To: Clinton Work
> Cc: cisco-nsp
> Subject: Re: [c-nsp] 3550 high cpu & process switched traffic
> 
> Hi Clinton,
> 
> Check my inline comments for the outputs you requested...
> 
> Clinton Work wrote on 18/8/2006 21:23:
>> I have seen this issue many times with the 3550s.  Your ACLs are 
>> probably too large / complicated and they are over-running the 3550 
>> hardware TCAM.
>>
>> Other possible causes:
>> - Too many routes
> 
> 3550#sh ip cef sum
> IP CEF with switching (Table Version 2714880), flags=0x0
>    11194 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 3
>    11197 leaves, 683 nodes, 2233168 bytes, 2714439 inserts, 2703242
> invalidations
>    1 load sharing elements, 336 bytes, 1 references
>    universal per-destination load sharing algorithm, id BAF66A0D
>    2(0) CEF resets, 446 revisions of existing leaves
>    Resolution Timer: Exponential (currently 1s, peak 1s)
>    444 in-place/0 aborted modifications
>    refcounts:  196815 leaf, 175104 node
> 
>    Table epoch: 0 (11197 entries at this epoch)
> 
> Adjacency Table has 58 adjacencies
> 
> 3550#sh ip route sum
> IP routing table name is Default-IP-Routing-Table(0)
> Route Source    Networks    Subnets     Overhead    Memory (bytes)
> connected       0           12          912         1920
> static          1           6           804         1120
> ospf xxx        192         10889       709248      1776020
>    Intra-area: 0 Inter-area: 188 External-1: 23 External-2: 10870
>    NSSA External-1: 0 NSSA External-2: 0
> internal        489                                 577020
> Total           682         10907       710964      2356080
> 
> 3550#sh l3tcam shadow
> L3 TCAM: total 72 bit entries = 9216, used entries = 8774
> L3 TCAM: total 144 bit entries = 4608, used entries = 0
> 
>> - Too many ARP entries. Note, ARP entry uses one IP CEF routing
entry.
>>
> 
> "sh arp" displays around 75 entries.
> 
>> Please see:
>>
>> Understand and Configure the Switching Database Manager on Catalyst 
>> 3550 Series Switches:
>> http://www.cisco.com/warp/public/473/145.html
>>
>>
>> Check the output from these commands:
>> show controller cpu
> 
> 3550#sh contr cpu
> stp packets : 17395785 retrieved, 0 dropped, 0 errors ram access
packets
> : 88713983 retrieved, 0 dropped, 0 errors routing protocol packets :
> 75142574 retrieved, 0 dropped, 0 errors forwarding packets : 0
> retrieved, 0 dropped, 0 errors routing packets : 3323805100 retrieved,
0
> dropped, 0 errors
> L2 protocol packets : 1491847 retrieved, 0 dropped, 0 errors igmp
> snooping protocol packets : 6113814 retrieved, 0 dropped, 0 errors
> queue7 : 0 retrieved, 0 dropped, 0 errors icmp redirect packets : 0
> retrieved, 0 dropped, 0 errors icmp unreachable packets : 0 retrieved,
0
> dropped, 0 errors logging packets : 0 retrieved, 0 dropped, 0 errors
> addr learning packets : 0 retrieved, 0 dropped, 0 errors rpffail
packets
> : 0 retrieved, 0 dropped, 0 errors
> queue13 : 50 retrieved, 0 dropped, 0 errors
> queue14 : 0 retrieved, 0 dropped, 0 errors
> queue15 : 0 retrieved, 0 dropped, 0 errors
> 
>> show access-lists hardware counters
>>
> 
> 3550#sh access-lists hardware counters
> Input Drops:             0 matches (0 bytes)
> Output Drops:            340637 matches (22547288 bytes)
> Input Forwarded:         78500081633 matches (38770143343719 bytes)
> Output Forwarded:        57823143638 matches (29808587886766 bytes)
> Input Bridge Only:       3310978 matches (239194398 bytes)
> Bridge and Route in CPU: 0 matches (0 bytes)
> Route in CPU:            13764993 matches (1259523078 bytes)
> 
> after 2 mins:
> 
> 3550#sh access-lists hardware counters
> Input Drops:             0 matches (0 bytes)
> Output Drops:            340639 matches (22547416 bytes)
> Input Forwarded:         78502363549 matches (38771255482430 bytes)
> Output Forwarded:        57824317525 matches (29809415168851 bytes)
> Input Bridge Only:       3311091 matches (239201856 bytes)
> Bridge and Route in CPU: 0 matches (0 bytes)
> Route in CPU:            13765101 matches (1259535021 bytes)
> 
> 
>> Try removing and re-applying your ACLs and see if the following 
>> message is logged:
>> show logging | inc FM-3-UNLOADING
>>
> 
> I tried it, but i didn't got any error message.
> 
>> You can look at the "show tcam inacl | outacl" commands, but they are

>> useless for troubleshooting TCAM utilization issues. In my opinion, 
>> the show tcam commands should show you total requested TCAM size in 
>> addition to the TCAM space currently used. Reporting TCAM utilization

>> only for the ACLs that fit into the TCAM isn't very useful!
>>
> 
> 
> 3550#sh tcam inacl 1 stat
> Ingress ACL TCAM#1: Number of active labels: 11
> Ingress ACL TCAM#1: Number of masks   allocated:   78, available:  130
> Ingress ACL TCAM#1: Number of entries allocated:  189, available: 1475
> 
> 3550#sh tcam outacl 1 stat
> Egress ACL TCAM#1: Number of active labels: 4
> Egress ACL TCAM#1: Number of masks   allocated:   15, available:  193
> Egress ACL TCAM#1: Number of entries allocated:   35, available: 1629
> 
>> 3550#show tcam inacl 1 stat
>> Ingress ACL TCAM#1: Number of active labels: 5
>> Ingress ACL TCAM#1: Number of masks   allocated:   46, available:
370
>> Ingress ACL TCAM#1: Number of entries allocated:   74, available:
3254
>>
>>
>>
>> Tassos Chatzithomaoglou wrote:
>>> 3550#sh proc cpu | exc 0.00
>>> CPU utilization for five seconds: 93%/91%; one minute: 93%; five
>>> minutes: 93%
>>>   PID Runtime(ms)   Invoked      uSecs   5Sec   1Min   5Min TTY
> Process
>>>    20     4544896 123670508         36  0.57%  0.63%  0.62%   0
Vegas
> 
>>> LED Proces
>>>    27    30048240  14764148       2035  0.40%  0.48%  0.45%   0
Vegas
> 
>>> Statistics
>>>    33     4701728  12413685        378  0.08%  0.06%  0.07%   0
> L3MD_STAT
>>>    35      919348  69239506         13  0.08%  0.07%  0.08%   0
> VegasPM
>>>    36    11606324   9735382       1192  0.08%  0.14%  0.15%   0 
>>> VUR_MGR bg proce
>>>    39    30030680 110922911        270  0.40%  0.40%  0.40%   0 IP
> Input
>>>    90      813456   7361206        110  0.08%  0.04%  0.04%   0 CEF 
>>> process
>>>    99    13504708  28875655        467  0.24%  0.10%  0.15%   0 OSPF

>>> Router
>>>
>>> Of course, there isn't any debug running in the background...
>>>
>>> omar parihuana wrote on 18/8/2006 20:42:
>>>> Hi,
>>>>
>>>> There is something wrong.. I have a 3550 with near 100SVI and my
CPU
> 
>>>> process not reach 20%
>>>>
>>
> 

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/



More information about the cisco-nsp mailing list