[c-nsp] Brief CPU spikes on 6500 Sup 720

[Aaron Riemer]

Hi Phil,

Answers below:

1) IOS - s72033-advipservicesk9_wan-mz.122-18.SXF17a.bin
2) HSRP configured between two core 6509's. SVI is VLAN1 (I know don't ask)
trunked between the cores via 10G. Only ports in VLAN1 on one core switch
are impacted and seeing the flooding.
3) Building floor switches connect to both cores (Routed and running EIGRP)
4) Spanning Tree Below:

Core1:
spanning-tree mode pvst
spanning-tree vlan 1-199,336,503-930 priority 16384

Core2:
spanning-tree mode pvst
spanning-tree vlan 1-199,336,503-930 priority 0

5) No rate limiting or CoPP configured. We are seeing drops even when the
CPU is not hitting 100% (most likely due to ASIC oversubscription).  
6) Source of traffic is unknown at this stage. Will turn to wireshark
tomorrow.
7) I don't believe there are any L2 loops. If spanning-tree was an issue I
would think the CPU would gradually hit 100% and stay there.

We are seeing output drops on interfaces and oversubscription of ASICs as a
result of this flooding which I think is the main culprit for the brief
connectivity outages. Is there a way similar to CoPP to protect the ASICs to
ensure they are never 100% utilised? Egress shaping on all suspect ports?


Thanks,

Aaron.


-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of JC Cockburn
Sent: Wednesday, 14 July 2010 8:03 PM
To: 'Phil Mayers'
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
Importance: High

Hi Phil,
I had a problem like this last year on 6500's.
It was related to bug: CSCsk23521
Basically a server in our datacenter used multicast addresses in the range
allocated for BPDU's, and this just killed the SP (100% CPU...).

If you do a "remote command switch sh proc cpu" on the 6500 you can see if
the SP CPU is under fire...

Cheers
JC

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net
[mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Phil Mayers
Sent: Wednesday, July 14, 2010 1:41 PM
To: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720

On 14/07/10 11:30, Aaron Riemer wrote:
> Hi Group,
>
>
>
> We are having trouble with unicast flooding on a particular VLAN and
> associated ports and as a result brief spikes in CPU usage on one of our
> 6509 core switches.
>
>
>
> ARP and MAC timeouts are set to default and we haven't had problems with
> this in the past. The problem is I believe this is causing brief 100%
spikes
> within the SP or RP and as a result brief connectivity outages.

Which is it? SP or RP?

>
>
>
> We have narrowed down the source of the unicast flooding but we need to
know
> why it is occurring.

Rather more info required I think.

  * IOS version
  * Config of ports & SVIs in question
  * Nature of downstream devices (if any)
  * spanning tree config (if any)
  * rough idea of the size of the ARP & MAC tables
  * Any MLS rate-limit or CoPP config
  * Nature of the source of the unicast-flooded traffic
  * Any possibility of loops in the network?

> Has anyone experienced this in the past? Could unicast flooding over
> multiple interfaces account for this kind of behaviour?

Anything punted to the CPU at high rate could cause this kind of thing. 
That's why MLS limiters and CoPP are important on this platform, even 
with all their limitations.
_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: RP CPU.txt
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20100714/ab471918/attachment-0003.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: sh module.txt
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20100714/ab471918/attachment-0004.txt>
-------------- next part --------------
An embedded and charset-unspecified text was scrubbed...
Name: SP CPU.txt
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20100714/ab471918/attachment-0005.txt>