[c-nsp] 2 to 3% packet loss on single VLAN on LAG interface

[Frank Bulk]

Thanks for the feedback offline. 

I spent some time on this in the last 36 hours.  Unfortunately turning off
port mirroring didn't help.  What I did notice, when adding more test
points, was that I was seeing 9 to 12 Mbps of traffic egressing a port on
the Brocade that had nothing but a SOHO router.  When I sniffed it it was
clear that it was unicast flooding.  I checked and re-checked, but at ~5000
MAC addresses we weren't exhausting the Brocade's tables nor the 7609's.  I
adjust MAC and ARP aging times down to shorter values (so that the 7609
wouldn't even forward the traffic if the ARP table for that IP was empty),
but also no change.

What did reduce the packet loss of almost zero and the reduced the unicast
flooding to ~50 kbps was shutting down the LAG member on the 6704 that was
the busier of the two 6704's (it's busier because the card also handles most
of our incoming Internet traffic).  If I unshut the LAG member the unicast
flooding and packet loss resumed.

So is this a symptom of some resources on the 7600 or the on the 6704?
Should I put the 10G port that handles the Internet traffic LAG member on
the same ASIC?

Frank

-----Original Message-----
From: cisco-nsp [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of
Frank Bulk
Sent: Wednesday, July 23, 2014 2:19 AM
To: cisco-nsp at puck.nether.net
Subject: [c-nsp] 2 to 3% packet loss on single VLAN on LAG interface

Looking for some tribal knowledge from this group -- we have a Cisco 7609-S
running IOS 15.2(4)S5 with RSP720C's and DFC3C's on all our line cards.  We
have two WS-X6704-10GE cards that move around 7 Gbps (that's a total of in
and out) at peak times between their 7 active interfaces.  Some of those
Gbps are mirrored traffic.

What our customers are seeing and what we have confirmed is 2 to 3% ICMP
packet loss for traffic in a certain VLAN that flow in/out of the 10G
cross-card port channel facing a Brocade ICX6610 stack.  This is packet loss
pinging *through* the router, not ICMP traffic terminating hitting the
router.  

e.g. 
server--1G---Cisco7609==10G==VLAN A==Brocade---CPE  (packet loss)
server--1G---Cisco7609==10G==VLAN B==Brocade---CMTS (no packet loss)
server--1G---Cisco7609==1G===VLAN A==access-gear---CPE (no packet loss)
laptop==access gear==10G==VLAN A==Brocade---CPE (no packet loss)
laptop==access gear---CPE (no packet loss)

That VLAN has about 4500 to 5500 active MAC addresses.  We can ping an IP
address in that certain VLANs that flow in/out some other 1G port-channel on
a WS-X6748-GE-TX and there is no packet loss.  Some of other VLANs I've
tested that flow in/out of the 10G cross-card port channel also don't have
packet loss.

So what would cause (what appears to be) a certain VLAN to drop some traffic
when flowing across a 10G port-channel and not a 1G port-channel?  I know
the 6704 has some performance limitations -- how do I measure that on the
box?

Regards,

Frank Bulk

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/