[c-nsp] 3550 CPU Usage & IPSec

[randal k]

Oli,
Another good idea. This switch does some Q-in-Q service, and its MTU
is 1530 everywhere; unfortunately it is virtually fragment free:

IP statistics:
  Rcvd:  2218345267 total, 62765867 local destination
         52 format errors, 33 checksum errors, 16655618 bad hop count
         0 unknown protocol, 17690 not a gateway
         0 security failures, 0 bad options, 58045 with options
  Opts:  329 end, 35 nop, 0 basic security, 0 loose source route
         0 timestamp, 0 extended security, 3 record route
         0 stream ID, 0 strict source route, 57716 alert, 0 cipso, 0 ump
         0 other
  Frags: 40 reassembled, 46 timeouts, 0 couldn't reassemble
         78 fragmented, 0 couldn't fragment
  Bcast: 14256448 received, 40677 sent
  Mcast: 7140931 received, 817787 sent
  Sent:  57670558 generated, 1091620153 forwarded
  Drop:  41037866 encapsulation failed, 0 unresolved, 0 no adjacency
         3872 no route, 0 unicast RPF, 706317 forced drop
         0 options denied, 0 source IP address zero

Since I'm plum out of ideas, I've already scheduled a time to switch
this customer over to a different 3550 to see if the problem persists
or follows him. I'll definitely post back with results.

Cheers,
Randal

On Sun, Nov 23, 2008 at 6:59 AM, Oliver Boehmer (oboehmer)
<oboehmer at cisco.com> wrote:
> I would check for fragmentation, as suggested by someone earlier in the
> thread. I didn't check, but I would suspect the 3550 doing fragmentation
> in "software" (i.e. within the interrupt context). How are your MTUs on
> your core interface up to (and including) the 3550?
> Check "show ip traffic", fragmentations should show up there..
>
>        oli
>
> randal k <> wrote on Friday, November 21, 2008 23:18:
>
>> Burton,
>> There is already ~150mbps of other traffic flowing through this
>> switch, all of which generates approximately zero CPU, which is how it
>> looks for 11 other active 3550s, all pushing hundreds of mbps; they're
>> extremely good at high pps layer 3 with very little CPU usage. Yes,
>> cef is on everywhere.
>>
>> The thing that draws the attention here is that it is the only 3550 in
>> our network that has more than 1-2% CPU. Of all of the customers
>> attached to this switch, his is the only port whose graph is an exact
>> match for the CPU usage, and his traffic is overwhelmingly IPSec. I
>> guess I could move him to a different 3550 distribution switch and see
>> if the problem follows.
>>
>> Thanks for your continued input -
>> Randal
>>
>>
>>
>>
>> On Fri, Nov 21, 2008 at 11:17 AM, Burton Windle <bwindle at fint.org>
>> wrote:
>>> I could be very wrong here, but I'm thought that if the usage is in
>>> the interrupt, then the CPU usage is just because of the volume of
>>> traffic, not the contents. But don't quote me on that.
>>>
>>> Easy way to test would be to push a similar volume of non-IPSec
>>> traffic and see what the CPU does.
>>>
>>>
>>> --
>>> Burton Windle                           bwindle at fint.org
>>>
>>>
>>> On Fri, 21 Nov 2008, randal k wrote:
>>>
>>>> Excuse my typo, my original answer of "IP Input" was completely
>>>> wrong, since it's pretty easy to get them confused. I'm looking at
>>>> it now and it's purely Interrupt traffic.
>>>>
>>>> dist03.cos01#show proc cpu
>>>> CPU utilization for five seconds: 26%/24%; one minute: 25%; five
>>>> minutes: 26%
>>>>
>>>> No, I'm not running anything on the 3550, it's purely a packet
>>>> pusher. It is a 3550-12T, and hanging off of it is the customer's
>>>> 3560g-24TS
>>>> and VPN3000. All of the tunnels terminate on the Concentrator - the
>>>> 3550 just does some basic layer3 forwarding and has no features.
>>>>
>>>> Net -- 7206edge -- 6509core --- 3550dist ---
>>>> 3560customer/VPN3000customer
>>>>
>>>> That's why I find it a little bit odd that just forwarding IPSec
>>>> packets (not originating/terminating them) is hitting the CPU.
>>>>
>>>> Randal
>>>>
>>>
>> _______________________________________________
>> cisco-nsp mailing list  cisco-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/cisco-nsp
>> archive at http://puck.nether.net/pipermail/cisco-nsp/
>