<html>
<head>
<style><!--
.hmmessage P
{
margin:0px;
padding:0px
}
body.hmmessage
{
font-size: 10pt;
font-family:Tahoma
}
--></style></head>
<body class='hmmessage'><div dir='ltr'>
<div>Re-forward - My original reply seems to have gone missing...<br><br>
<meta http-equiv="Content-Type" content="text/html; charset=unicode">
<meta name="Generator" content="Microsoft SafeHTML">
<style>
.ExternalClass .ecxhmmessage P
{padding:0px;}
.ExternalClass body.ecxhmmessage
{font-size:10pt;font-family:Tahoma;}
</style>
<div dir="ltr">
<div><br></div><div>Thanks - Fairly certain we are seeing actual drops (And we are seeing packet-loss across this link)</div><div><br></div><div><br></div><div>Gi1/0/17 is mapped to asic 0/20</div><div><br></div><div><div><br></div><div>Gi1/0/17 17 17 17 0/20 1 17 17 local Yes Yes</div></div><div><br></div><div><br></div><div><div>Port-asic Port Drop Statistics - Summary</div><div><br></div><div>========================================</div></div><div><br></div><div><br></div><div> Port 20 TxQueue Drop Stats: 308277833</div><div><br></div><div><br></div><div>And majority appear to be in Queue 1:</div><div><br></div><div><br></div><div>(The formatting will probably be screwed up by Hotmail.... apologies)</div><div><br></div><div><br></div><div><div> Port 20 TxQueue Drop Statistics</div><div> Queue 0</div><div> Weight 0 Frames 3</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div><div> Queue 1</div><div> Weight 0 Frames 308240408</div><div> Weight 1 Frames 458</div><div> Weight 2 Frames 0</div><div> Queue 2</div><div> Weight 0 Frames 37898</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div><div> Queue 3</div><div> Weight 0 Frames 91</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div><div> Queue 4</div><div> Weight 0 Frames 0</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div><div> Queue 5</div><div> Weight 0 Frames 0</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div><div> Queue 6</div><div> Weight 0 Frames 0</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div><div> Queue 7</div><div> Weight 0 Frames 0</div><div> Weight 1 Frames 0</div><div> Weight 2 Frames 0</div></div><div><br></div><div><br></div><div><br></div>So hoping someone can suggest a few tweaks to reduce these drops(And also (more importantly!) the packet-loss)<div><br></div><div><br></div><div>Note - We have other ports on this switch doing far more traffic, but not seeing any loss/drops on these....so I believe it is due to the bursty traffic specific to this trunk port</div><div><br></div><div><br></div><div><br></div><div>Cheers</div><div><br><div><br><br><div><div id="ecxSkyDrivePlaceholder"></div><hr id="ecxstopSpelling">From: tlusty@wayfair.com<br>To: cisco-nsp@puck.nether.net; johnelliot67@hotmail.com; andriy.bilous@gmail.com<br>Date: Thu, 22 Dec 2011 09:08:17 -0500<br>Subject: Re: [c-nsp] 2960S drops/packet loss<br><br>
<style>
.ExternalClass .ecxshape
{;}
</style><style>
.ExternalClass p.ecxMsoNormal, .ExternalClass li.ecxMsoNormal, .ExternalClass div.ecxMsoNormal
{margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri","sans-serif";}
.ExternalClass a:link, .ExternalClass span.ecxMsoHyperlink
{color:blue;text-decoration:underline;}
.ExternalClass a:visited, .ExternalClass span.ecxMsoHyperlinkFollowed
{color:purple;text-decoration:underline;}
.ExternalClass p.ecxMsoPlainText, .ExternalClass li.ecxMsoPlainText, .ExternalClass div.ecxMsoPlainText
{margin-bottom:.0001pt;font-size:11.0pt;font-family:"Calibri","sans-serif";}
.ExternalClass p.ecxMsoAcetate, .ExternalClass li.ecxMsoAcetate, .ExternalClass div.ecxMsoAcetate
{margin-bottom:.0001pt;font-size:8.0pt;font-family:"Tahoma","sans-serif";}
.ExternalClass span.ecxEmailStyle17
{font-family:"Calibri","sans-serif";color:windowtext;}
.ExternalClass span.ecxBalloonTextChar
{font-family:"Tahoma","sans-serif";}
.ExternalClass span.ecxPlainTextChar
{font-family:"Calibri","sans-serif";}
.ExternalClass .ecxMsoChpDefault
{font-family:"Calibri","sans-serif";}
@page WordSection1
{size:8.5in 11.0in;}
.ExternalClass div.ecxWordSection1
{page:WordSection1;}
</style><div class="ecxWordSection1"><p class="ecxMsoNormal">We’ve run into the same/similar output drop bug (technically a “different” bug, with ID CSCtq86186, but it’s really the same thing). We’re seeing it on a 3750X with 12.2(58)SE. TAC also recommended that we upgrade to 15.2 to fix the issue, but as Andriy noted, it hasn’t been released yet. When I asked them about this, they said that it was actually fixed in 15.0.1(SE1), which was released Dec 2<sup>nd</sup>. We haven’t tested that assertion out yet; we’ve decided to hold off on putting the “fresh out of the oven” IOS on our gear, but if someone else has, and has a good or bad experience we’d be all ears.</p><p class="ecxMsoNormal"> </p><p class="ecxMsoNormal">And just to reiterate what Andriy said, use command ”sh platform port-asic stats drop” to get the true numbers as “sh int <blah>” can’t be trusted. Ensure there are actually drops happening, so you don’t end up chasing a ghost :) </p><p class="ecxMsoNormal"> </p><p class="ecxMsoNormal">-Tom</p><p class="ecxMsoNormal"> </p><table class="ecxMsoNormalTable" border="0" cellspacing="0" cellpadding="0"><tbody><tr><td style="padding:0in 0in 3.75pt 0in"><p class="ecxMsoNormal" style="margin-right:15.0pt"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5A2149;text-transform:uppercase">Tom Lusty</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#464646"><br>Senior Systems Engineer </span></p></td><td rowspan="2" style="border:none;border-left:solid #6D6D6D 1.0pt;padding:0in 0in 0in 15.0pt"><p class="ecxMsoNormal"><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#464646"><img width="130" height="55" id="ecx_x0000_i1025" src="cid:image001.gif@01CCC085.5B026A10" alt="Description: wayfair inc"></span><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#464646"></span></p></td></tr><tr><td style="padding:0in 33.75pt 0in 0in"><p class="ecxMsoNormal"><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5A2149;text-transform:uppercase">Wayfair</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#464646"><br>177 Huntington Ave, Suite 6000<br>Boston, MA 02115<br></span><b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#5A2149;text-transform:uppercase">P</span></b><span style="font-size:8.5pt;font-family:"Arial","sans-serif";color:#464646">: 617.502.7026 <br><a href="mailto:tlusty@wayfair.com"><span style="color:blue">tlusty@wayfair.com</span></a> </span></p></td></tr></tbody></table><p class="ecxMsoNormal"> </p><p class="ecxMsoPlainText"> </p><p class="ecxMsoNormal">-----Original Message-----<br>Date: Thu, 22 Dec 2011 13:09:24 +0100</p><p class="ecxMsoPlainText">From: Andriy Bilous <andriy.bilous@gmail.com></p><p class="ecxMsoPlainText">To: John Elliot <johnelliot67@hotmail.com></p><p class="ecxMsoPlainText">Cc: cisco-nsp <cisco-nsp@puck.nether.net></p><p class="ecxMsoPlainText">Subject: Re: [c-nsp] 2960S drops/packet loss</p><p class="ecxMsoPlainText">Message-ID:</p><p class="ecxMsoPlainText"> <CAOnSrb9LO8YffjjwHOPW09SKK+gpKqjWWBK-UFZVtxAivphAdA@mail.gmail.com></p><p class="ecxMsoPlainText">Content-Type: text/plain; charset=ISO-8859-1</p><p class="ecxMsoPlainText"> </p><p class="ecxMsoPlainText">There is a bug in most of SEs for all stackable models, which makes drop statistics unusable (see CSCso81660 for example - there are lots of BugIDs with same diagnostics and keep in mind "Fixed-in is a lie).</p><p class="ecxMsoPlainText">Reported numbers aren't realistic and often go both ways - increase and decrease in what appears to be a random manner. sh platform port-asic stats drop should report right value though. So, before you start tweaking, you may want to look further into it and just wait for the sane IOS release. Our SE reported it's fixed in 15.2, which isn't available for download yet.</p><p class="ecxMsoPlainText"> </p><p class="ecxMsoPlainText">On Thu, Dec 22, 2011 at 4:41 AM, John Elliot <<a href="mailto:johnelliot67@hotmail.com">johnelliot67@hotmail.com</a>> wrote:</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> Hi Guys,</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> Have a pair of 2960's in a stack, one port(trunk) connects to another DC and we are seeing ~5% packet-lossand large output drops to this DC.</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> #sh interfaces gigabitEthernet 1/0/17 counters errors</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> Port ? ? ? ?Align-Err ? ? FCS-Err ? ?Xmit-Err ? ? Rcv-Err ?UnderSize </p><p class="ecxMsoPlainText">> ?OutDiscards</p><p class="ecxMsoPlainText">> Gi1/0/17 ? ? ? ? ? ?0 ? ? ? ? ? 0 ? ? ? ? ? 0 ? ? ? ? ? 0 ? ? ? ? ?0 ? </p><p class="ecxMsoPlainText">> ? ? 182867</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> GigabitEthernet1/0/17 is up, line protocol is up (connected) ?Hardware </p><p class="ecxMsoPlainText">> is Gigabit Ethernet, address is a0cf.5b87.ec11 (bia a0cf.5b87.ec11) </p><p class="ecxMsoPlainText">> ?Description: QinQ_to_DC2 ?MTU 1998 bytes, BW 100000 Kbit, DLY 100 </p><p class="ecxMsoPlainText">> usec, ? ? reliability 255/255, txload 41/255, rxload 23/255 </p><p class="ecxMsoPlainText">> ?Encapsulation ARPA, loopback not set ?Keepalive set (10 sec) </p><p class="ecxMsoPlainText">> ?Full-duplex, 100Mb/s, media type is 10/100/1000BaseTX ?input </p><p class="ecxMsoPlainText">> flow-control is off, output flow-control is unsupported ?ARP type: </p><p class="ecxMsoPlainText">> ARPA, ARP Timeout 04:00:00 ?Last input 6d13h, output 00:00:00, output </p><p class="ecxMsoPlainText">> hang never ?Last clearing of "show interface" counters 04:02:15 ?Input </p><p class="ecxMsoPlainText">> queue: 0/75/0/0 (size/max/drops/flushes); Total output drops: 183592 </p><p class="ecxMsoPlainText">> ?Queueing strategy: fifo ?Output queue: 0/40 (size/max) ?30 second </p><p class="ecxMsoPlainText">> input rate 9047000 bits/sec, 2075 packets/sec ?30 second output rate </p><p class="ecxMsoPlainText">> 16324000 bits/sec, 2309 packets/sec</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> As you can see, 30sec rate isnt excessive, but as the drops are outdiscards it would appear we are getting hit by the small buffers/microburst issue.</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> Done a bit of reasearch, and as we have mls qos configured(Need to as we have to trust dscp markings), we needto look at "tweaking" the buffer allocations on the switch to hopefully mitigate these drops.</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> There appears to be a range of recommendations when it comes to these tweaks - Hoping someone has some suggestions on what to set with "mls qos queue-set output" to alleviate the drops?(start conservative, then apply more aggressive if needed)....and also, does adjusting the buffers require an outage window?</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> Our traffic is primarily backup(replication which is very bursty), and </p><p class="ecxMsoPlainText">> Internet</p><p class="ecxMsoPlainText">> </p><p class="ecxMsoPlainText">> Thanks in advance.</p><p class="ecxMsoPlainText">> _______________________________________________</p><p class="ecxMsoPlainText">> cisco-nsp mailing list <a href="mailto:?cisco-nsp@puck.nether.net">?cisco-nsp@puck.nether.net</a> </p><p class="ecxMsoPlainText">> <a href="https://puck.nether.net/mailman/listinfo/cisco-nsp" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-nsp</a></p><p class="ecxMsoPlainText">> archive at <a href="http://puck.nether.net/pipermail/cisco-nsp/" target="_blank">http://puck.nether.net/pipermail/cisco-nsp/</a></p><p class="ecxMsoPlainText"> </p><p class="ecxMsoPlainText"> </p><p class="ecxMsoPlainText"> </p><p class="ecxMsoPlainText">------------------------------</p><p class="ecxMsoPlainText"> </p><p class="ecxMsoPlainText">_______________________________________________</p><p class="ecxMsoPlainText">cisco-nsp mailing list</p><p class="ecxMsoPlainText"><a href="mailto:cisco-nsp@puck.nether.net">cisco-nsp@puck.nether.net</a></p><p class="ecxMsoPlainText"><a href="https://puck.nether.net/mailman/listinfo/cisco-nsp" target="_blank">https://puck.nether.net/mailman/listinfo/cisco-nsp</a></p><p class="ecxMsoNormal"> </p></div></div></div></div> </div></div> </div></body>
</html>