[c-nsp] Brief CPU spikes on 6500 Sup 720

Wed Jul 14 11:04:51 EDT 2010

That problem is described here:
http://www.cisco.com/en/US/docs/solutions/Enterprise/Campus/HA_campus_DG/hacampusdg.html#wp1108782

If you're not using a load-balancing first hop redundancy mechanism (GLBP), the issue can be avoided by ensuring:
- L2 interconnect between the L3 switches
- The VLAN's STP root and Primary router are on the same switch

Even if you are using GLBP, you should still probably have the L2 link between L3 switches.  It prevents weird L2 failure scenarios, and keeps your access switches from doing transit work.

> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-
> bounces at puck.nether.net] On Behalf Of Benjamin Lovell
> Sent: Wednesday, July 14, 2010 10:38 AM
> To: Aaron Riemer
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> 
> Most of the time we see problems like this it is caused by asymmetric
> routing. The ECPM return path leads to the standby switch which does
> not know the DMAC as all traffic was processed by the active switch.
> This is usually fixed by increasing the MAC table timers to match the
> ARP timers so that standby will keep MAC in table long enough until the
> next time ARP comes around.
> 
> -Ben
> 
> On Jul 14, 2010, at 9:46 AM, Aaron Riemer wrote:
> 
> > Hi Phil,
> >
> > Answers below:
> >
> > 1) IOS - s72033-advipservicesk9_wan-mz.122-18.SXF17a.bin
> > 2) HSRP configured between two core 6509's. SVI is VLAN1 (I know
> don't ask)
> > trunked between the cores via 10G. Only ports in VLAN1 on one core
> switch
> > are impacted and seeing the flooding.
> > 3) Building floor switches connect to both cores (Routed and running
> EIGRP)
> > 4) Spanning Tree Below:
> >
> > Core1:
> > spanning-tree mode pvst
> > spanning-tree vlan 1-199,336,503-930 priority 16384
> >
> > Core2:
> > spanning-tree mode pvst
> > spanning-tree vlan 1-199,336,503-930 priority 0
> >
> > 5) No rate limiting or CoPP configured. We are seeing drops even when
> the
> > CPU is not hitting 100% (most likely due to ASIC oversubscription).
> > 6) Source of traffic is unknown at this stage. Will turn to wireshark
> > tomorrow.
> > 7) I don't believe there are any L2 loops. If spanning-tree was an
> issue I
> > would think the CPU would gradually hit 100% and stay there.
> >
> > We are seeing output drops on interfaces and oversubscription of
> ASICs as a
> > result of this flooding which I think is the main culprit for the
> brief
> > connectivity outages. Is there a way similar to CoPP to protect the
> ASICs to
> > ensure they are never 100% utilised? Egress shaping on all suspect
> ports?
> >
> >
> > Thanks,
> >
> > Aaron.
> >
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of JC Cockburn
> > Sent: Wednesday, 14 July 2010 8:03 PM
> > To: 'Phil Mayers'
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> > Importance: High
> >
> > Hi Phil,
> > I had a problem like this last year on 6500's.
> > It was related to bug: CSCsk23521
> > Basically a server in our datacenter used multicast addresses in the
> range
> > allocated for BPDU's, and this just killed the SP (100% CPU...).
> >
> > If you do a "remote command switch sh proc cpu" on the 6500 you can
> see if
> > the SP CPU is under fire...
> >
> > Cheers
> > JC
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Phil Mayers
> > Sent: Wednesday, July 14, 2010 1:41 PM
> > To: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> >
> > On 14/07/10 11:30, Aaron Riemer wrote:
> >> Hi Group,
> >>
> >>
> >>
> >> We are having trouble with unicast flooding on a particular VLAN and
> >> associated ports and as a result brief spikes in CPU usage on one of
> our
> >> 6509 core switches.
> >>
> >>
> >>
> >> ARP and MAC timeouts are set to default and we haven't had problems
> with
> >> this in the past. The problem is I believe this is causing brief
> 100%
> > spikes
> >> within the SP or RP and as a result brief connectivity outages.
> >
> > Which is it? SP or RP?
> >
> >>
> >>
> >>
> >> We have narrowed down the source of the unicast flooding but we need
> to
> > know
> >> why it is occurring.
> >
> > Rather more info required I think.
> >
> >  * IOS version
> >  * Config of ports & SVIs in question
> >  * Nature of downstream devices (if any)
> >  * spanning tree config (if any)
> >  * rough idea of the size of the ARP & MAC tables
> >  * Any MLS rate-limit or CoPP config
> >  * Nature of the source of the unicast-flooded traffic
> >  * Any possibility of loops in the network?
> >
> >> Has anyone experienced this in the past? Could unicast flooding over
> >> multiple interfaces account for this kind of behaviour?
> >
> > Anything punted to the CPU at high rate could cause this kind of
> thing.
> > That's why MLS limiters and CoPP are important on this platform, even
> > with all their limitations.
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> > _______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > <RP CPU.txt><sh module.txt><SP
> CPU.txt>_______________________________________________
> > cisco-nsp mailing list  cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> 
> 
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/