[c-nsp] Brief CPU spikes on 6500 Sup 720

[Aaron Riemer]

Hi Lee,

Putting a static mac entry in stops the flooding for that particular unicast
destination. Therefore the MAC must be disappearing from the switches CAM
table.

Enabled SNMP traps and MAC-notifications and this brought another issue to
my attention. There is a huge amount of mac-flapping going on (not for this
host) but our ESX hosts that have vmnics trunking to both our cores.

The VM guys are sending traffic for each VM host out both connected vmnic's
causing the MAC to be learnt on the vmnic port and the trunk port between
the core switches hence the flapping.

Could this be contributing to the problem and possibly explain why MAC
addresses are removed from the CAM table? No TCN's are noted so this surely
isn't the reason. MAC age is set to 4 hours same as default ARP timeout..

I would prefer the VM team dedicated a NIC for their VM's to eliminate this
kind of behaviour.

Has anyone else experienced this?

Cheers,

Aaron.

-----Original Message-----
From: Lee [mailto:ler762 at gmail.com] 
Sent: Friday, 16 July 2010 8:09 AM
To: Aaron Riemer
Cc: cisco-nsp at puck.nether.net
Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720

>   3) Packet captures confirmed ...

Does a "sh mac-add add [destination mac address from the capture]" on
the flooding switch show that mac address?  For the correct vlan?  On
the Sup as well as all DFC equipped cards?

Can you ping the source and destination from the core switch?  Does
that stop the unicast flooding?

Regards,
Lee


On 7/15/10, Aaron Riemer <ariemer at amnet.net.au> wrote:
> Guys,
>
> We are still seeing these floods. They occur when our backups run and it
> doesn't matter if the traffic is routed into the vlan or switched within
the
> same vlan as the same problem occurs.
>
> Therefore:
>   1) asymmetric routing cannot be the problem for the L2 switched traffic
>   2) No TCN's are occurring within the VLAN STP (PVST).
>   3) Packet captures confirmed that all ports on VLAN1 on one core were
> receiving the unicast traffic destined for a host on this vlan.
>   4) From the packet captures we were able to deduce source / dest MAC
> addresses confirming that the destination MAC was in fact not aging out
and
> could be seen clear as day in the MAC table.
>
> Had TAC on the line troubleshooting it for a couple of hours tonight and
> still no closer to solving the problem. Mac-age time out is set to the
same
> as the arp timeout (4 hours)
>
> I don't believe the issue is linked to the CPU spikes but rather the
output
> drops that we are seeing oversubscribing our 6548 line cards.
>
> Anyone have any further ideas?
>
> Thanks,
>
> Aaron.
>
> -----Original Message-----
> From: Phil Mayers [mailto:p.mayers at imperial.ac.uk]
> Sent: Wednesday, 14 July 2010 11:14 PM
> To: Aaron Riemer
> Cc: 'Matthew Huff'; 'JC Cockburn'; cisco-nsp at puck.nether.net
> Subject: Re: [c-nsp] Brief CPU spikes on 6500 Sup 720
>
> On 14/07/10 15:51, Aaron Riemer wrote:
>> Yes i have read all about unicast flooding:
>>
>> Can occur by:
>>
>> 1) Asymmetric routing
>> 2) Spanning Tree TCN
>> 3) MAC aging out
>>
>> I cannot see any TCN's or Asymmetric routing so i think we may have to
>> adjust the mac aging as you suggested.
>
> If you're running HSRP, the standby node has a route for the subnet as
> well as the active e.g. traffic might flow out through the active, and
> back through the standby as follows:
>
> host
>   |
>   active-standby
>   |        |
> (cloud)   ^
>   |        |
>   router --/
>   |
> target
>
> from   host->active->router-target
> return target->router->standby->host
>
> ...if "standby" has an ARP entry (default 4 hours) but not MAC table
> entry (default 5 minutes) it will unknown-unicast-flood the return
> traffic. If it's a lot of traffic, that will burn a lot of bandwidth...
>
> Whether this will happen will depend on your routing topology.
>
>>
>> I am just trying to work out why the hell this has only just started
>> occurring!
>
> Well, without knowing the source of the traffic it's impossible to tell.
>
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>