[c-nsp] 3550 high cpu & process switched traffic
Shaikh, Nasir
Nasir.Shaikh at atosorigin.com
Mon Aug 21 06:29:22 EDT 2006
Tassos,
When you say that the problem is solved do you mean that CPU utilisation is back to normal? Or do you now infact see egress packets being fast-switched?
Triggered by your issue I checked my 3550s (aroung 33x 3550-12T and 2x 3550-12G) and to my surprise all of them 0 bytes being fastswitched (egress). My CPU just hovers around an average 5%.
Following are outputs from one of the 3550-12Ts.
Anyone care to shed some light on this matter?
#sh int stats
Interface Vlan1 is disabled
GigabitEthernet0/1
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1945212 309923497 1501108 162558113
Route cache 224617 77770360 0 0
Total 2169829 387693857 1501108 162558113
GigabitEthernet0/2
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 576315 43999531 703709 60875753
Route cache 3712 244412 0 0
Total 580027 44243943 703709 60875753
GigabitEthernet0/3
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 431308 34380754 658008 57739434
Route cache 4 288 0 0
Total 431312 34381042 658008 57739434
GigabitEthernet0/4
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 365635 44848540 590677 68343424
Route cache 3 390 0 0
Total 365638 44848930 590677 68343424
Interface GigabitEthernet0/5 is disabled
Interface GigabitEthernet0/6 is disabled
GigabitEthernet0/7
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 497370 58089632 310649 30463140
Route cache 5484 1312003 0 0
Total 502854 59401635 310649 30463140
GigabitEthernet0/8
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 430610 34198707 663504 58936494
Route cache 0 0 0 0
Total 430610 34198707 663504 58936494
GigabitEthernet0/9
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 1263043 89708329 1604673 109117115
Route cache 0 0 0 0
Total 1263043 89708329 1604673 109117115
GigabitEthernet0/10
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 389 24896 227332 25184384
Route cache 0 0 0 0
Total 389 24896 227332 25184384
Interface GigabitEthernet0/11 is disabled
Interface GigabitEthernet0/12 is disabled
Loopback0
Switching path Pkts In Chars In Pkts Out Chars Out
Processor 0 0 420475 25228500
Route cache 0 0 0 0
Total 0 0 420475 25228500
#sh proc cpu
CPU utilization for five seconds: 1%/0%; one minute: 1%; five minutes: 1%
#sh ip route summ
IP routing table name is Default-IP-Routing-Table(0)
Route Source Networks Subnets Overhead Memory (bytes)
connected 0 9 576 1440
static 0 0 0 0
eigrp 2130 28 1070 70336 175680
internal 142 167560
Total 170 1079 70912 344680
#sh sdm prefer
The current template is the default template.
The selected template optimizes the resources in
the switch to support this level of features for
16 routed interfaces and 1K VLANs.
number of unicast mac addresses: 6K
number of igmp groups: 6K
number of qos aces: 2K
number of security aces: 2K
number of unicast routes: 12K
number of multicast routes: 6K
#sh l3tcam shadow
L3 TCAM: total 72 bit entries = 18432, used entries = 1216
L3 TCAM: total 144 bit entries = 9216, used entries = 0
#sh ip cef summ
IP CEF with switching (Table Version 8591), flags=0x0
1193 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 0
1196 leaves, 536 nodes, 720152 bytes, 8555 inserts, 7359 invalidations
1 load sharing elements, 336 bytes, 1 references
universal per-destination load sharing algorithm, id 2B515256
2(0) CEF resets, 40 revisions of existing leaves
Resolution Timer: Exponential (currently 1s, peak 1s)
39 in-place/0 aborted modifications
refcounts: 139328 leaf, 137472 node
Table epoch: 0 (1196 entries at this epoch)
Adjacency Table has 61 adjacencies
bsipsg01#sh contr cpu
stp packets : 1943432 retrieved, 0 dropped, 0 errors
ram access packets : 34793921 retrieved, 0 dropped, 0 errors
routing protocol packets : 2417872 retrieved, 0 dropped, 0 errors
forwarding packets : 0 retrieved, 0 dropped, 0 errors
routing packets : 3324671 retrieved, 0 dropped, 0 errors
L2 protocol packets : 259277 retrieved, 0 dropped, 0 errors
igmp snooping protocol packets : 0 retrieved, 0 dropped, 0 errors
queue7 : 0 retrieved, 0 dropped, 0 errors
icmp redirect packets : 1857 retrieved, 0 dropped, 0 errors
icmp unreachable packets : 308 retrieved, 0 dropped, 0 errors
logging packets : 0 retrieved, 0 dropped, 0 errors
addr learning packets : 0 retrieved, 0 dropped, 0 errors
rpffail packets : 0 retrieved, 0 dropped, 0 errors
queue13 : 106 retrieved, 0 dropped, 0 errors
queue14 : 0 retrieved, 0 dropped, 0 errors
queue15 : 0 retrieved, 0 dropped, 0 errors
RAM Access:
13510182 sends 34793921 read replies 24678 write replies
13510179 completed 2 retries 0 failures
0 nomem 0 nobuffers 0 errors
0 expedite toggles 0 fa-lost 0 fa-passives
bsipsg01#sh tcam inacl 1 stat
Ingress ACL TCAM#1: Number of active labels: 7
Ingress ACL TCAM#1: Number of masks allocated: 46, available: 370
Ingress ACL TCAM#1: Number of entries allocated: 137, available: 3191
bsipsg01#sh tcam outacl 1 stat
Egress ACL TCAM#1: Number of active labels: 2
Egress ACL TCAM#1: Number of masks allocated: 6, available: 410
Egress ACL TCAM#1: Number of entries allocated: 5, available: 3323
bsipsg01#sh tcam pbr 1 stat
PBR TCAM#1: Number of active labels: 0
PBR TCAM#1: Number of masks allocated: 0
PBR TCAM#1: Number of entries allocated: 0
bsipsg01#sh tcam qos 1 stat
QoS TCAM#1: Number of active labels: 1
QoS TCAM#1: Number of masks allocated: 4, available: 820
QoS TCAM#1: Number of entries allocated: 3, available: 6589
regards
Nash
-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net]On Behalf Of Tassos Chatzithomaoglou
Sent: zaterdag 19 augustus 2006 12:19
To: cisco-nsp
Subject: Re: [c-nsp] 3550 high cpu & process switched traffic
Ok, i disabled all incoming ospf routing updates through a distribute list and the problem seems to
be solved for the moment (in order to be sure i'll have to wait until the traffic reaches its max
again). So it -most probably- should be the routing table size that caused such a high cpu load.
Now comes the interesting part:
I had opened a case with cisco tac some months ago about the same problem. We had followed almost
the same steps like here and had come to the conclusion that it should be some special traffic
traversing the router causing it to be processed by the cpu. So we needed to capture a sample of
traffic through sniffing and examine it. Of course that capturing isn't very easy when the hardware
is located somewhere away. So the case ended there.
According to tac, although they had seen both "sh ip cef sum" & "sh ip route sum", the following
output was the one that actually showed there was no problem with the routing table size:
3550#sh l3tcam shadow
L3 TCAM: total 72 bit entries = 9216, used entries = 8774
L3 TCAM: total 144 bit entries = 4608, used entries = 0
What was even more strange is that we had another 3550 with exactly the same routing table but only
5% cpu load. And the main difference between these 2 3550's was just the amount of traffic passing
through. This 3550 had around 170Mbps max (95% cpu) while the other had around 30Mbps max (5% cpu).
When this 3550's traffic was decreasing (~80Mbps), cpu load was falling too (40%). I guess that was
another cause that made tac believe there was something "strange" in the traffic of this 3550.
I still wonder what might be the relationship between routing table size and amount of traffic, so
the same routing table size with different amounts of traffic is causing different cpu loads on a
3550. According to another cisco engineer, "generally" a 3550 should be able to forward in L3
whatever it's able to do in L2 with no apparent difference in cpu load.
Yesterday, Clinton's quote "You will see some due to IP options that can't be processed in the TCAM
IP forwarding" made me search through "sh ip traffic" where you can see the number of packets with
options (i wonder how tac didn't think of that; they were looking for redirects/unreachables only).
Since this number was very small, it probably shouldn't be the kind of traffic causing the high cpu
load. I'm not trying to reduce tac's job, but sometimes i believe they provide "hard to implement"
solutions (remote sniffing being one of them).
Thanks a lot guys for all your help.
Regards,
Tassos
Tassos Chatzithomaoglou wrote on 19/8/2006 12:11 p?:
> Thanks a lot Clinton and everyone else for your answers...
>
> Since i would like first to test a quick "solution" for the routing size
> problem before i find a permanent solution for decreasing its size, if i
> use a in distribute-list under ospf which denies everything (so
> everything goes out through the default gateway), will that make the
> routing table shorter?
>
> Regards,
> Tassos
>
> Clinton Work wrote on 18/8/2006 10:03 ??:
>>
>> Your problem is too many routes for the default SDM template. The
>> default template is limited to 8K Unicast routes. If you change the
>> SDM template to routing you can probably fit the 11K routes into the
>> TCAM. The routing template is 24K for the 3550-12G and 16K routes for
>> the regular 3550s.
>>
>> Please check "show sdm prefer".
>>
>> 3550#show sdm prefer
>> The current template is the default template.
>> The selected template optimizes the resources in
>> the switch to support this level of features for
>> 16 routed interfaces and 1K VLANs.
>>
>> number of unicast mac addresses: 6K
>> number of igmp groups: 6K
>> number of qos aces: 2K
>> number of security aces: 2K
>> number of unicast routes: 12K
>> number of multicast routes: 6K
>>
>>
>> Note, seeing a high pps of routing packets under "show controller cpu"
>> is bad. It means that the IP packets are being punted to the CPU for
>> forwarding. You will see some due to IP options that can't be
>> processed in the TCAM IP forwarding.
>>
>> 3550#sh contr cpu
>> routing packets : 3323805100 retrieved, 0 dropped, 0 errors
>>
>> Example from a 3550 routing about 50Mbps of Internet traffic and the
>> CPU is at 2%. Each of these commands is a couple of seconds apart.
>> The router is forwarding about 20,000 pps of Internet traffic.
>>
>>
>> 3550#show controllers cpu-interface | inc routing
>> routing protocol packets : 21492556 retrieved, 0 dropped
>> routing packets : 297482820 retrieved, 0 dropped
>>
>> 3550#show controllers cpu-interface | inc routing
>> routing protocol packets : 21492557 retrieved, 0 dropped
>> routing packets : 297482826 retrieved, 0 dropped
>>
>> 3550#show controllers cpu-interface | inc routing
>> routing protocol packets : 21492557 retrieved, 0 dropped
>> routing packets : 297482831 retrieved, 0 dropped
>>
>> 3550#show controllers cpu-interface | inc routing
>> routing protocol packets : 21492557 retrieved, 0 dropped
>> routing packets : 297482837 retrieved, 0 dropped
>>
>> 3550#show controllers cpu-interface | inc routing
>> routing protocol packets : 21492557 retrieved, 0 dropped
>> routing packets : 297482845 retrieved, 0 dropped
>>
>>
>>
>>
>> Tassos Chatzithomaoglou wrote:
>>>
>>> 3550#sh ip cef sum
>>> IP CEF with switching (Table Version 2714880), flags=0x0
>>> 11194 routes, 0 reresolve, 0 unresolved (0 old, 0 new), peak 3
>>> 11197 leaves, 683 nodes, 2233168 bytes, 2714439 inserts, 2703242
>>> invalidations
>>> 1 load sharing elements, 336 bytes, 1 references
>>> universal per-destination load sharing algorithm, id BAF66A0D
>>> 2(0) CEF resets, 446 revisions of existing leaves
>>> Resolution Timer: Exponential (currently 1s, peak 1s)
>>> 444 in-place/0 aborted modifications
>>> refcounts: 196815 leaf, 175104 node
>>>
>>> Table epoch: 0 (11197 entries at this epoch)
>>
>>
_______________________________________________
cisco-nsp mailing list cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: Disclaimer.txt
Url: https://puck.nether.net/pipermail/cisco-nsp/attachments/20060821/bb771858/attachment-0001.txt
More information about the cisco-nsp
mailing list