[c-nsp] 3550 high cpu & process switched traffic
Arie Vayner (avayner)
avayner at cisco.com
Fri Aug 18 15:04:59 EDT 2006
Tassos,
My guess would be that you are overloading the resources of this switch.
First, you have too many routes on this switch. It is not built for such
a large routing table. Can you reduce its size?
Second, I think you might be running out of ACL masks in the TCAM.
I would take the following action plan:
1. change the SDM to "routing" : "sdm prefer routing" (requires a
reload)
2. remove the ACL's from all the interfaces
3. reduce the routing table
Then start adding the ACLs one by one, always monitoring the "show tcam
(in/out)acl stat" output to make sure it does not fill up.
Another option to make it more efficient (I am not 100% sure if its
relevant for 3550, but worth a try) is to reuse the same ACLs on
different vlan interfaces. Try making the ACLs common, and use the same
ACLs on several interfaces.
You need to remember this is a HW platform, which requires all features
to fit into TCAM or else it would process switch it all...
Arie
-----Original Message-----
From: Tassos Chatzithomaoglou [mailto:achatz at forthnet.gr]
Sent: Friday, August 18, 2006 21:53 PM
To: Arie Vayner (avayner)
Cc: cisco-nsp
Subject: Re: [c-nsp] 3550 high cpu & process switched traffic
Hi Arie,
Below you can find the outputs you requested.
Arie Vayner (avayner) wrote on 18/8/2006 21:50:
> Tassos,
>
> Can you please send:
>
> sh tcam inacl 1 statistics
3550#sh tcam inacl 1 statistics
Ingress ACL TCAM#1: Number of active labels: 11
Ingress ACL TCAM#1: Number of masks allocated: 78, available: 130
Ingress ACL TCAM#1: Number of entries allocated: 189, available: 1475
> show sdm prefer
3550#sh sdm prefer
The current template is the default template.
The selected template optimizes the resources in
the switch to support this level of features for
8 routed interfaces and 1K VLANs.
number of unicast mac addresses: 5K
number of igmp groups: 1K
number of qos aces: 1K
number of security aces: 1K
number of unicast routes: 8K
number of multicast routes: 1K
>
> Thanks
> Arie
>
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Tassos
> Chatzithomaoglou
> Sent: Friday, August 18, 2006 21:43 PM
> To: Clinton Work
> Cc: cisco-nsp
> Subject: Re: [c-nsp] 3550 high cpu & process switched traffic
>
> Hi Clinton,
>
> Check my inline comments for the outputs you requested...
>
> Clinton Work wrote on 18/8/2006 21:23:
>> I have seen this issue many times with the 3550s. Your ACLs are
>> probably too large / complicated and they are over-running the 3550
>> hardware TCAM.
>>
>> Other possible causes:
>> - Too many routes
>
> 3550#sh ip cef sum
> IP CEF with switching (Table Version 2714880), flags=0x0
> 11194 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 3
> 11197 leaves, 683 nodes, 2233168 bytes, 2714439 inserts, 2703242
> invalidations
> 1 load sharing elements, 336 bytes, 1 references
> universal per-destination load sharing algorithm, id BAF66A0D
> 2(0) CEF resets, 446 revisions of existing leaves
> Resolution Timer: Exponential (currently 1s, peak 1s)
> 444 in-place/0 aborted modifications
> refcounts: 196815 leaf, 175104 node
>
> Table epoch: 0 (11197 entries at this epoch)
>
> Adjacency Table has 58 adjacencies
>
> 3550#sh ip route sum
> IP routing table name is Default-IP-Routing-Table(0)
> Route Source Networks Subnets Overhead Memory (bytes)
> connected 0 12 912 1920
> static 1 6 804 1120
> ospf xxx 192 10889 709248 1776020
> Intra-area: 0 Inter-area: 188 External-1: 23 External-2: 10870
> NSSA External-1: 0 NSSA External-2: 0
> internal 489 577020
> Total 682 10907 710964 2356080
>
> 3550#sh l3tcam shadow
> L3 TCAM: total 72 bit entries = 9216, used entries = 8774
> L3 TCAM: total 144 bit entries = 4608, used entries = 0
>
>> - Too many ARP entries. Note, ARP entry uses one IP CEF routing
entry.
>>
>
> "sh arp" displays around 75 entries.
>
>> Please see:
>>
>> Understand and Configure the Switching Database Manager on Catalyst
>> 3550 Series Switches:
>> http://www.cisco.com/warp/public/473/145.html
>>
>>
>> Check the output from these commands:
>> show controller cpu
>
> 3550#sh contr cpu
> stp packets : 17395785 retrieved, 0 dropped, 0 errors ram access
> packets
> : 88713983 retrieved, 0 dropped, 0 errors routing protocol packets :
> 75142574 retrieved, 0 dropped, 0 errors forwarding packets : 0
> retrieved, 0 dropped, 0 errors routing packets : 3323805100 retrieved,
> 0 dropped, 0 errors
> L2 protocol packets : 1491847 retrieved, 0 dropped, 0 errors igmp
> snooping protocol packets : 6113814 retrieved, 0 dropped, 0 errors
> queue7 : 0 retrieved, 0 dropped, 0 errors icmp redirect packets : 0
> retrieved, 0 dropped, 0 errors icmp unreachable packets : 0 retrieved,
> 0 dropped, 0 errors logging packets : 0 retrieved, 0 dropped, 0 errors
> addr learning packets : 0 retrieved, 0 dropped, 0 errors rpffail
> packets
> : 0 retrieved, 0 dropped, 0 errors
> queue13 : 50 retrieved, 0 dropped, 0 errors
> queue14 : 0 retrieved, 0 dropped, 0 errors
> queue15 : 0 retrieved, 0 dropped, 0 errors
>
>> show access-lists hardware counters
>>
>
> 3550#sh access-lists hardware counters
> Input Drops: 0 matches (0 bytes)
> Output Drops: 340637 matches (22547288 bytes)
> Input Forwarded: 78500081633 matches (38770143343719 bytes)
> Output Forwarded: 57823143638 matches (29808587886766 bytes)
> Input Bridge Only: 3310978 matches (239194398 bytes)
> Bridge and Route in CPU: 0 matches (0 bytes)
> Route in CPU: 13764993 matches (1259523078 bytes)
>
> after 2 mins:
>
> 3550#sh access-lists hardware counters
> Input Drops: 0 matches (0 bytes)
> Output Drops: 340639 matches (22547416 bytes)
> Input Forwarded: 78502363549 matches (38771255482430 bytes)
> Output Forwarded: 57824317525 matches (29809415168851 bytes)
> Input Bridge Only: 3311091 matches (239201856 bytes)
> Bridge and Route in CPU: 0 matches (0 bytes)
> Route in CPU: 13765101 matches (1259535021 bytes)
>
>
>> Try removing and re-applying your ACLs and see if the following
>> message is logged:
>> show logging | inc FM-3-UNLOADING
>>
>
> I tried it, but i didn't got any error message.
>
>> You can look at the "show tcam inacl | outacl" commands, but they are
>> useless for troubleshooting TCAM utilization issues. In my opinion,
>> the show tcam commands should show you total requested TCAM size in
>> addition to the TCAM space currently used. Reporting TCAM utilization
>> only for the ACLs that fit into the TCAM isn't very useful!
>>
>
>
> 3550#sh tcam inacl 1 stat
> Ingress ACL TCAM#1: Number of active labels: 11
> Ingress ACL TCAM#1: Number of masks allocated: 78, available: 130
> Ingress ACL TCAM#1: Number of entries allocated: 189, available: 1475
>
> 3550#sh tcam outacl 1 stat
> Egress ACL TCAM#1: Number of active labels: 4
> Egress ACL TCAM#1: Number of masks allocated: 15, available: 193
> Egress ACL TCAM#1: Number of entries allocated: 35, available: 1629
>
>> 3550#show tcam inacl 1 stat
>> Ingress ACL TCAM#1: Number of active labels: 5
>> Ingress ACL TCAM#1: Number of masks allocated: 46, available:
370
>> Ingress ACL TCAM#1: Number of entries allocated: 74, available:
3254
>>
>>
>>
>> Tassos Chatzithomaoglou wrote:
>>> 3550#sh proc cpu | exc 0.00
>>> CPU utilization for five seconds: 93%/91%; one minute: 93%; five
>>> minutes: 93%
>>> PID Runtime(ms) Invoked uSecs 5Sec 1Min 5Min TTY
> Process
>>> 20 4544896 123670508 36 0.57% 0.63% 0.62% 0
Vegas
>
>>> LED Proces
>>> 27 30048240 14764148 2035 0.40% 0.48% 0.45% 0
Vegas
>
>>> Statistics
>>> 33 4701728 12413685 378 0.08% 0.06% 0.07% 0
> L3MD_STAT
>>> 35 919348 69239506 13 0.08% 0.07% 0.08% 0
> VegasPM
>>> 36 11606324 9735382 1192 0.08% 0.14% 0.15% 0
>>> VUR_MGR bg proce
>>> 39 30030680 110922911 270 0.40% 0.40% 0.40% 0 IP
> Input
>>> 90 813456 7361206 110 0.08% 0.04% 0.04% 0 CEF
>>> process
>>> 99 13504708 28875655 467 0.24% 0.10% 0.15% 0 OSPF
>>> Router
>>>
>>> Of course, there isn't any debug running in the background...
>>>
>>> omar parihuana wrote on 18/8/2006 20:42:
>>>> Hi,
>>>>
>>>> There is something wrong.. I have a 3550 with near 100SVI and my
>>>> CPU
>
>>>> process not reach 20%
>>>>
>>
>
> --
> ***************************************
> Tassos Chatzithomaoglou
> Network Design & Development Department
> FORTHnet S.A.
> <achatz at forthnet.gr>
> ***************************************
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>
--
***************************************
Tassos Chatzithomaoglou
Network Design & Development Department
FORTHnet S.A.
<achatz at forthnet.gr>
***************************************
More information about the cisco-nsp
mailing list