[c-nsp] FW: 2960S drops/packet loss
John Elliot
johnelliot67 at hotmail.com
Thu Dec 22 18:32:41 EST 2011
Re-forward - My original reply seems to have gone missing...
Thanks - Fairly certain we are seeing actual drops (And we are seeing packet-loss across this link)
Gi1/0/17 is mapped to asic 0/20
Gi1/0/17 17 17 17 0/20 1 17 17 local Yes Yes
Port-asic Port Drop Statistics - Summary
========================================
Port 20 TxQueue Drop Stats: 308277833
And majority appear to be in Queue 1:
(The formatting will probably be screwed up by Hotmail.... apologies)
Port 20 TxQueue Drop Statistics Queue 0 Weight 0 Frames 3 Weight 1 Frames 0 Weight 2 Frames 0 Queue 1 Weight 0 Frames 308240408 Weight 1 Frames 458 Weight 2 Frames 0 Queue 2 Weight 0 Frames 37898 Weight 1 Frames 0 Weight 2 Frames 0 Queue 3 Weight 0 Frames 91 Weight 1 Frames 0 Weight 2 Frames 0 Queue 4 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 5 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 6 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0 Queue 7 Weight 0 Frames 0 Weight 1 Frames 0 Weight 2 Frames 0
So hoping someone can suggest a few tweaks to reduce these drops(And also (more importantly!) the packet-loss)
Note - We have other ports on this switch doing far more traffic, but not seeing any loss/drops on these....so I believe it is due to the bursty traffic specific to this trunk port
Cheers
From: tlusty at wayfair.com
To: cisco-nsp at puck.nether.net; johnelliot67 at hotmail.com; andriy.bilous at gmail.com
Date: Thu, 22 Dec 2011 09:08:17 -0500
Subject: Re: [c-nsp] 2960S drops/packet loss
We’ve run into the same/similar output drop bug (technically a “different” bug, with ID CSCtq86186, but it’s really the same thing). We’re seeing it on a 3750X with 12.2(58)SE. TAC also recommended that we upgrade to 15.2 to fix the issue, but as Andriy noted, it hasn’t been released yet. When I asked them about this, they said that it was actually fixed in 15.0.1(SE1), which was released Dec 2nd. We haven’t tested that assertion out yet; we’ve decided to hold off on putting the “fresh out of the oven” IOS on our gear, but if someone else has, and has a good or bad experience we’d be all ears. And just to reiterate what Andriy said, use command ”sh platform port-asic stats drop” to get the true numbers as “sh int <blah>” can’t be trusted. Ensure there are actually drops happening, so you don’t end up chasing a ghost :) -Tom Tom Lusty
Senior Systems Engineer Wayfair
177 Huntington Ave, Suite 6000
Boston, MA 02115
P: 617.502.7026
tlusty at wayfair.com -----Original Message-----
Date: Thu, 22 Dec 2011 13:09:24 +0100From: Andriy Bilous <andriy.bilous at gmail.com>To: John Elliot <johnelliot67 at hotmail.com>Cc: cisco-nsp <cisco-nsp at puck.nether.net>Subject: Re: [c-nsp] 2960S drops/packet lossMessage-ID: <CAOnSrb9LO8YffjjwHOPW09SKK+gpKqjWWBK-UFZVtxAivphAdA at mail.gmail.com>Content-Type: text/plain; charset=ISO-8859-1 There is a bug in most of SEs for all stackable models, which makes drop statistics unusable (see CSCso81660 for example - there are lots of BugIDs with same diagnostics and keep in mind "Fixed-in is a lie).Reported numbers aren't realistic and often go both ways - increase and decrease in what appears to be a random manner. sh platform port-asic stats drop should report right value though. So, before you start tweaking, you may want to look further into it and just wait for the sane IOS release. Our SE reported it's fixed in 15.2, which isn't available for download yet. On Thu, Dec 22, 2011 at 4:41 AM, John Elliot <johnelliot67 at hotmail.com> wrote:> > Hi Guys,> > Have a pair of 2960's in a stack, one port(trunk) connects to another DC and we are seeing ~5% packet-lossand large output drops to this DC.> > > #sh interfaces gigabitEthernet 1/0/17 counters errors> > Port ? ? ? ?Align-Err ? ? FCS-Err ? ?Xmit-Err ? ? Rcv-Err ?UnderSize > ?OutDiscards> Gi1/0/17 ? ? ? ? ? ?0 ? ? ? ? ? 0 ? ? ? ? ? 0 ? ? ? ? ? 0 ? ? ? ? ?0 ? > ? ? 182867> > GigabitEthernet1/0/17 is up, line protocol is up (connected) ?Hardware > is Gigabit Ethernet, address is a0cf.5b87.ec11 (bia a0cf.5b87.ec11) > ?Description: QinQ_to_DC2 ?MTU 1998 bytes, BW 100000 Kbit, DLY 100 > usec, ? ? reliability 255/255, txload 41/255, rxload 23/255 > ?Encapsulation ARPA, loopback not set ?Keepalive set (10 sec) > ?Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX ?input > flow-control is off, output flow-control is unsupported ?ARP type: > ARPA, ARP Timeout 04:00:00 ?Last input 6d13h, output 00:00:00, output > hang never ?Last clearing of "show interface" counters 04:02:15 ?Input > queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 183592 > ?Queueing strategy: fifo ?Output queue: 0/40 (size/max) ?30 second > input rate 9047000 bits/sec, 2075 packets/sec ?30 second output rate > 16324000 bits/sec, 2309 packets/sec> > > As you can see, 30sec rate isnt excessive, but as the drops are outdiscards it would appear we are getting hit by the small buffers/microburst issue.> > Done a bit of reasearch, and as we have mls qos configured(Need to as we have to trust dscp markings), we needto look at "tweaking" the buffer allocations on the switch to hopefully mitigate these drops.> > There appears to be a range of recommendations when it comes to these tweaks - Hoping someone has some suggestions on what to set with "mls qos queue-set output" to alleviate the drops?(start conservative, then apply more aggressive if needed)....and also, does adjusting the buffers require an outage window?> > Our traffic is primarily backup(replication which is very bursty), and > Internet> > Thanks in advance.> _______________________________________________> cisco-nsp mailing list ?cisco-nsp at puck.nether.net > https://puck.nether.net/mailman/listinfo/cisco-nsp> archive at http://puck.nether.net/pipermail/cisco-nsp/ ------------------------------ _______________________________________________cisco-nsp mailing listcisco-nsp at puck.nether.nethttps://puck.nether.net/mailman/listinfo/cisco-nsp
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20111222/b1c61fa3/attachment.html>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.gif
Type: image/gif
Size: 3960 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20111222/b1c61fa3/attachment.gif>
More information about the cisco-nsp
mailing list