[c-nsp] Brief CPU spikes on 6500 Sup 720
Matthew Huff
mhuff at ox.com
Wed Jul 14 11:57:10 EDT 2010
HSRP is an "inbound" next hop routing protocol. It creates a virtual Ip that tracks with a virtual MAC. It has no bearing on outbound traffic, only inbound. With HSRP it is very easy to create asymmetrical routing.
For example, consider a firewall connected to a VLAN with two 6500 switches
(10.1.2.2) 6500-A (10.1.1.1)
PC --> HSRP (10.1.2.3) (10.1.1.3) (HSRP) --> Firewall (10.1.1.4)
(10.1.2.4) (10.1.2.1) 6500-B (10.1.1.2)
PC outbound traffic will go out via 6500-A, return traffic will come in via 6500-B
With no priority set with HSRP, the active interface is chosen by the "higher" IP.
With dynamic routing and HSRP, it's even easier to create asymetrical routing.
----
Matthew Huff | One Manhattanville Rd
OTA Management LLC | Purchase, NY 10577
http://www.ox.com | Phone: 914-460-4039
aim: matthewbhuff | Fax: 914-460-4139
> -----Original Message-----
> From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Aaron
> Riemer
> Sent: Wednesday, July 14, 2010 10:59 AM
> To: 'Benjamin Lovell'
> Cc: cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
>
> Forgive my ignorance. What is ECPM??
>
> Shouldn't all routed traffic be handled by the active HSRP node?
>
> -----Original Message-----
> From: Benjamin Lovell [mailto:belovell at cisco.com]
> Sent: Wednesday, 14 July 2010 10:38 PM
> To: Aaron Riemer
> Cc: 'JC Cockburn'; 'Phil Mayers'; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
>
> Most of the time we see problems like this it is caused by asymmetric
> routing. The ECPM return path leads to the standby switch which does not
> know the DMAC as all traffic was processed by the active switch. This is
> usually fixed by increasing the MAC table timers to match the ARP timers so
> that standby will keep MAC in table long enough until the next time ARP
> comes around.
>
> -Ben
>
> On Jul 14, 2010, at 9:46 AM, Aaron Riemer wrote:
>
> > Hi Phil,
> >
> > Answers below:
> >
> > 1) IOS - s72033-advipservicesk9_wan-mz.122-18.SXF17a.bin
> > 2) HSRP configured between two core 6509's. SVI is VLAN1 (I know don't
> ask)
> > trunked between the cores via 10G. Only ports in VLAN1 on one core switch
> > are impacted and seeing the flooding.
> > 3) Building floor switches connect to both cores (Routed and running
> EIGRP)
> > 4) Spanning Tree Below:
> >
> > Core1:
> > spanning-tree mode pvst
> > spanning-tree vlan 1-199,336,503-930 priority 16384
> >
> > Core2:
> > spanning-tree mode pvst
> > spanning-tree vlan 1-199,336,503-930 priority 0
> >
> > 5) No rate limiting or CoPP configured. We are seeing drops even when the
> > CPU is not hitting 100% (most likely due to ASIC oversubscription).
> > 6) Source of traffic is unknown at this stage. Will turn to wireshark
> > tomorrow.
> > 7) I don't believe there are any L2 loops. If spanning-tree was an issue I
> > would think the CPU would gradually hit 100% and stay there.
> >
> > We are seeing output drops on interfaces and oversubscription of ASICs as
> a
> > result of this flooding which I think is the main culprit for the brief
> > connectivity outages. Is there a way similar to CoPP to protect the ASICs
> to
> > ensure they are never 100% utilised? Egress shaping on all suspect ports?
> >
> >
> > Thanks,
> >
> > Aaron.
> >
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of JC Cockburn
> > Sent: Wednesday, 14 July 2010 8:03 PM
> > To: 'Phil Mayers'
> > Cc: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> > Importance: High
> >
> > Hi Phil,
> > I had a problem like this last year on 6500's.
> > It was related to bug: CSCsk23521
> > Basically a server in our datacenter used multicast addresses in the range
> > allocated for BPDU's, and this just killed the SP (100% CPU...).
> >
> > If you do a "remote command switch sh proc cpu" on the 6500 you can see if
> > the SP CPU is under fire...
> >
> > Cheers
> > JC
> >
> > -----Original Message-----
> > From: cisco-nsp-bounces at puck.nether.net
> > [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Phil Mayers
> > Sent: Wednesday, July 14, 2010 1:41 PM
> > To: cisco-nsp at puck.nether.net
> > Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
> >
> > On 14/07/10 11:30, Aaron Riemer wrote:
> >> Hi Group,
> >>
> >>
> >>
> >> We are having trouble with unicast flooding on a particular VLAN and
> >> associated ports and as a result brief spikes in CPU usage on one of our
> >> 6509 core switches.
> >>
> >>
> >>
> >> ARP and MAC timeouts are set to default and we haven't had problems with
> >> this in the past. The problem is I believe this is causing brief 100%
> > spikes
> >> within the SP or RP and as a result brief connectivity outages.
> >
> > Which is it? SP or RP?
> >
> >>
> >>
> >>
> >> We have narrowed down the source of the unicast flooding but we need to
> > know
> >> why it is occurring.
> >
> > Rather more info required I think.
> >
> > * IOS version
> > * Config of ports & SVIs in question
> > * Nature of downstream devices (if any)
> > * spanning tree config (if any)
> > * rough idea of the size of the ARP & MAC tables
> > * Any MLS rate-limit or CoPP config
> > * Nature of the source of the unicast-flooded traffic
> > * Any possibility of loops in the network?
> >
> >> Has anyone experienced this in the past? Could unicast flooding over
> >> multiple interfaces account for this kind of behaviour?
> >
> > Anything punted to the CPU at high rate could cause this kind of thing.
> > That's why MLS limiters and CoPP are important on this platform, even
> > with all their limitations.
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> >
> > _______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
> > <RP CPU.txt><sh module.txt><SP
> CPU.txt>_______________________________________________
> > cisco-nsp mailing list cisco-nsp at puck.nether.net
> > https://puck.nether.net/mailman/listinfo/cisco-nsp
> > archive at http://puck.nether.net/pipermail/cisco-nsp/
>
> _______________________________________________
> cisco-nsp mailing list cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
More information about the cisco-nsp
mailing list