<html><head><meta http-equiv="Content-Type" content="text/html charset=utf-8"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">Over the years we’ve seen odd issues where one of the switch-fabric-links will “wigout” and some of the data moving between cards will get corrupted. When this happens we power cycle each switch fab one at a time using this process:<div class="">1) Shutdown SFM #3</div><div class="">2) Wait 1 minute</div><div class="">3) Power SFM #3 on again</div><div class="">4) Verify all SFM links are up to SFM#3</div><div class="">5) Wait 1 minute</div><div class="">6) Perform steps 1-5 for SFM #2</div><div class="">7) Perform steps 1-5 for SFM #3</div><div class=""><br class=""></div><div class="">Not sure you’re seeing the same issue that we see but the “SFM Dance” (as we call it) is a once-every-four-months thing somewhere across our 16 XMR4000 boxes. It can be done with little to no impact if you are patient verify status before moving to the next SFM.</div><div class=""><br class=""></div><div class=""><br class=""><div><blockquote type="cite" class=""><div class="">On Feb 13, 2015, at 11:41 AM, <a href="mailto:nethub@gmail.com" class="">nethub@gmail.com</a> wrote:</div><br class="Apple-interchange-newline"><div class=""><div class="WordSection1" style="page: WordSection1; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);"><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class="">We have three switch fabrics installed, all are under 1% utilized.<o:p class=""></o:p></span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class=""> </span></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class=""> </span></div><div class=""><div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif; color: windowtext;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; color: windowtext;" class=""><span class="Apple-converted-space"> </span>Jeroen Wunnink | Hibernia Networks [<a href="mailto:jeroen.wunnink@atrato.com" style="color: purple; text-decoration: underline;" class="">mailto:jeroen.wunnink@atrato.com</a>]<span class="Apple-converted-space"> </span><br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, February 13, 2015 12:27 PM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:nethub@gmail.com" style="color: purple; text-decoration: underline;" class="">nethub@gmail.com</a>; 'Jeroen Wunnink | Hibernia Networks'<br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [f-nsp] MLX throughput issues<o:p class=""></o:p></span></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><o:p class=""> </o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">How many switchfabrics do you have in that MLX and how high is the utilization on them<br class=""><br class="">On 13/02/15 18:12,<span class="Apple-converted-space"> </span><a href="mailto:nethub@gmail.com" style="color: purple; text-decoration: underline;" class="">nethub@gmail.com</a><span class="Apple-converted-space"> </span>wrote:<o:p class=""></o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite"><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class="">We also tested with a spare Quanta LB4M we have and are seeing about the same speeds as we are seeing with the FLS648 (around 20MB/s or 160Mbps).</span><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class=""> </span><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class="">I also reduced the number of routes we are accepting down to about 189K and that did not make a difference.</span><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class=""> </span><o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="color: rgb(31, 73, 125);" class=""> </span><o:p class=""></o:p></div><div class=""><div style="border-style: solid none none; border-top-color: rgb(181, 196, 223); border-top-width: 1pt; padding: 3pt 0in 0in;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><b class=""><span style="font-size: 10pt; font-family: Tahoma, sans-serif; color: windowtext;" class="">From:</span></b><span style="font-size: 10pt; font-family: Tahoma, sans-serif; color: windowtext;" class=""><span class="Apple-converted-space"> </span>foundry-nsp [<a href="mailto:foundry-nsp-bounces@puck.nether.net" style="color: purple; text-decoration: underline;" class="">mailto:foundry-nsp-bounces@puck.nether.net</a>]<span class="Apple-converted-space"> </span><b class="">On Behalf Of<span class="Apple-converted-space"> </span></b>Jeroen Wunnink | Hibernia Networks<br class=""><b class="">Sent:</b><span class="Apple-converted-space"> </span>Friday, February 13, 2015 3:35 AM<br class=""><b class="">To:</b><span class="Apple-converted-space"> </span><a href="mailto:foundry-nsp@puck.nether.net" style="color: purple; text-decoration: underline;" class="">foundry-nsp@puck.nether.net</a><br class=""><b class="">Subject:</b><span class="Apple-converted-space"> </span>Re: [f-nsp] MLX throughput issues</span><o:p class=""></o:p></div></div></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div class=""><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The FLS switches do something weird with packets. I've noticed they somehow interfere with changing the MSS window size dynamically, resulting in destinations further away having very poor speed results compared to destinations close by.<span class="Apple-converted-space"> </span><br class=""><br class="">We got rid of those a while ago.<br class=""><br class=""><br class="">On 12/02/15 17:37,<span class="Apple-converted-space"> </span><a href="mailto:nethub@gmail.com" style="color: purple; text-decoration: underline;" class="">nethub@gmail.com</a><span class="Apple-converted-space"> </span>wrote:<o:p class=""></o:p></div></div><blockquote style="margin-top: 5pt; margin-bottom: 5pt;" class="" type="cite"><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">We are having a strange issue on our MLX running code 5.6.00c. We are encountering some throughput issues that seem to be randomly impacting specific networks.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">We use the MLX to handle both external BGP and internal VLAN routing. Each FLS648 is used for Layer 2 VLANs only.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">From a server connected by 1 Gbps uplink to a Foundry FLS648 switch, which is then connected to the MLX on a 10 Gbps port, running a speed test to an external network is getting 20MB/s.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Connecting the same server directly to the MLX is getting 70MB/s.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Connecting the same server to one of my customer's Juniper EX3200 (which BGP peers with the MLX) also gets 70MB/s.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Testing to another external network, all three scenarios get 110MB/s.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The path to both test network locations goes through the same IP transit provider.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">We are running NI-MLX-MR with 2GB RAM, NI-MLX-10Gx4 connect to the Foundry FLS648 by XFP-10G-LR, NI-MLX-1Gx20-GC was used for directly connecting the server. A separate NI-MLX-10Gx4 connects to our upstream BGP providers. Customer’s Juniper EX3200 connects to the same NI-MLX-10Gx4 as the FLS648. We take default routes plus full tables from three providers by BGP, but filter out most of the routes.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">The fiber and optics on everything look fine. CPU usage is less than 10% on the MLX and all line cards and CPU usage at 1% on the FLS648. ARP table on the MLX is about 12K, and BGP table is about 308K routes.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""> <o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class="">Any assistance would be appreciated. I suspect there is a setting that we’re missing on the MLX that is causing this issue.<o:p class=""></o:p></div><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: 'Times New Roman , serif', serif;" class=""><br class=""><br class=""><br class=""><br class=""></span><o:p class=""></o:p></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">_______________________________________________<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">foundry-nsp mailing list<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a href="mailto:foundry-nsp@puck.nether.net" style="color: purple; text-decoration: underline;" class="">foundry-nsp@puck.nether.net</a><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a href="http://puck.nether.net/mailman/listinfo/foundry-nsp" style="color: purple; text-decoration: underline;" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a><o:p class=""></o:p></pre></blockquote><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: 'Times New Roman , serif', serif;" class=""><br class=""><br class=""><br class=""><br class=""></span><o:p class=""></o:p></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">-- <o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""> <o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">Jeroen Wunnink<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">IP NOC Manager - Hibernia Networks<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 <o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a href="mailto:jeroen.wunnink@hibernianetworks.com" style="color: purple; text-decoration: underline;" class="">jeroen.wunnink@hibernianetworks.com</a><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a href="http://www.hibernianetworks.com/" style="color: purple; text-decoration: underline;" class="">www.hibernianetworks.com</a><o:p class=""></o:p></pre></blockquote><div style="margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri, sans-serif;" class=""><span style="font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><br class=""><br class=""><br class=""><o:p class=""></o:p></span></div><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">-- <o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><o:p class=""> </o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">Jeroen Wunnink<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">IP NOC Manager - Hibernia Networks<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">Main numbers (Ext: 1011): USA +1.908.516.4200 | UK +44.1704.322.300 <o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class="">Netherlands +31.208.200.622 | 24/7 IP NOC Phone: +31.20.82.00.623<o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a href="mailto:jeroen.wunnink@hibernianetworks.com" style="color: purple; text-decoration: underline;" class="">jeroen.wunnink@hibernianetworks.com</a><o:p class=""></o:p></pre><pre style="margin: 0in 0in 0.0001pt; font-size: 10pt; font-family: 'Courier New';" class=""><a href="http://www.hibernianetworks.com/" style="color: purple; text-decoration: underline;" class="">www.hibernianetworks.com</a><o:p class=""></o:p></pre></div><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255); float: none; display: inline !important;" class="">foundry-nsp mailing list</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><a href="mailto:foundry-nsp@puck.nether.net" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">foundry-nsp@puck.nether.net</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class=""><a href="http://puck.nether.net/mailman/listinfo/foundry-nsp" style="color: purple; text-decoration: underline; font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; background-color: rgb(255, 255, 255);" class="">http://puck.nether.net/mailman/listinfo/foundry-nsp</a></div></blockquote></div><br class=""></div></body></html>