[c-nsp] Brief CPU spikes on 6500 Sup 720

[Matthew Huff]

Since you are running HSRP,I'm willing to bet it's a asymmetrical routing with aging timeout causing a unicast flooding.

If you make the arging timeout >= to the arp timeout it might fix your problem:

mac-address-table aging-time 14400

http://www.ciscopress.com/articles/article.asp?p=336872

----
Matthew Huff       | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com  | Phone: 914-460-4039
aim: matthewbhuff  | Fax:   914-460-4139



> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Aaron
> Riemer
> Sent: Wednesday, July 14, 2010 9:46 AM
> To: 'JC Cockburn'; 'Phil Mayers'
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> 
> Hi Phil,
> 
> Answers below:
> 
> 1) IOS - s72033-advipservicesk9_wan-mz.122-18.SXF17a.bin
> 2) HSRP configured between two core 6509's. SVI is VLAN1 (I know don't ask)
> trunked between the cores via 10G. Only ports in VLAN1 on one core switch
> are impacted and seeing the flooding.
> 3) Building floor switches connect to both cores (Routed and running EIGRP)
> 4) Spanning Tree Below:
> 
> Core1:
> spanning-tree mode pvst
> spanning-tree vlan 1-199,336,503-930 priority 16384
> 
> Core2:
> spanning-tree mode pvst
> spanning-tree vlan 1-199,336,503-930 priority 0
> 
> 5) No rate limiting or CoPP configured. We are seeing drops even when the
> CPU is not hitting 100% (most likely due to ASIC oversubscription).
> 6) Source of traffic is unknown at this stage. Will turn to wireshark
> tomorrow.
> 7) I don't believe there are any L2 loops. If spanning-tree was an issue I
> would think the CPU would gradually hit 100% and stay there.
> 
> We are seeing output drops on interfaces and oversubscription of ASICs as a
> result of this flooding which I think is the main culprit for the brief
> connectivity outages. Is there a way similar to CoPP to protect the ASICs to
> ensure they are never 100% utilised? Egress shaping on all suspect ports?
> 
> 
> Thanks,
> 
> Aaron.
> 
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of JC Cockburn
> Sent: Wednesday, 14 July 2010 8:03 PM
> To: 'Phil Mayers'
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> Importance: High
> 
> Hi Phil,
> I had a problem like this last year on 6500's.
> It was related to bug: CSCsk23521
> Basically a server in our datacenter used multicast addresses in the range
> allocated for BPDU's, and this just killed the SP (100% CPU...).
> 
> If you do a "remote command switch sh proc cpu" on the 6500 you can see if
> the SP CPU is under fire...
> 
> Cheers
> JC
> 
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net
> [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Phil Mayers
> Sent: Wednesday, July 14, 2010 1:41 PM
> To: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> 
> On 14/07/10 11:30, Aaron Riemer wrote:
> > Hi Group,
> >
> >
> >
> > We are having trouble with unicast flooding on a particular VLAN and
> > associated ports and as a result brief spikes in CPU usage on one of our
> > 6509 core switches.
> >
> >
> >
> > ARP and MAC timeouts are set to default and we haven't had problems with
> > this in the past. The problem is I believe this is causing brief 100%
> spikes
> > within the SP or RP and as a result brief connectivity outages.
> 
> Which is it? SP or RP?
> 
> >
> >
> >
> > We have narrowed down the source of the unicast flooding but we need to
> know
> > why it is occurring.
> 
> Rather more info required I think.
> 
>   * IOS version
>   * Config of ports & SVIs in question
>   * Nature of downstream devices (if any)
>   * spanning tree config (if any)
>   * rough idea of the size of the ARP & MAC tables
>   * Any MLS rate-limit or CoPP config
>   * Nature of the source of the unicast-flooded traffic
>   * Any possibility of loops in the network?
> 
> > Has anyone experienced this in the past? Could unicast flooding over
> > multiple interfaces account for this kind of behaviour?
> 
> Anything punted to the CPU at high rate could cause this kind of thing.
> That's why MLS limiters and CoPP are important on this platform, even
> with all their limitations.
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/