<p dir="ltr">Remember (and maybe someone can link to the trace route interpretation PDF from Nanog) that if the packet loss doesn't continue end to end then its not likely a real issue as the forwarding plane isn't loosing packets. The loss on a single hop either points to limits on ICMP responses on the control plane, or congestion on the forwarding plane.</p>
<div class="gmail_quote">On Feb 28, 2014 5:09 PM, "jay lark" <<a href="mailto:jay@jaylark.com">jay@jaylark.com</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">We’ve also see increased packet loss and latency over the past week or so from our London Level3 office to our Verizon offices in NY and LA. Routes to our
ATT office in Chicago have been okay for the most part.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Here’s a current trace from London to LAX. Heavy packet loss in London. It was in Paris before that
</span><span style="font-size:11.0pt;font-family:Wingdings;color:#1f497d">J</span><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> My traceroute [v0.80]<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">lon-nagios (0.0.0.0) Fri Feb 28 21:52:35 2014<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Resolver: Received error response 2. (server failure)er of fields quit<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> Packets Pings<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Host Loss% Snt Last Avg Best Wrst StDev<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">1. xxxxxxxxxxxxxxxxxxxxxxxxxxx 0.0% 468 0.3 0.3 0.2 4.0 0.3<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">2. xxxxxxxxxxxxxxx 0.0% 468 0.9 2.3 0.8 79.1 7.7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">3. <a href="http://ae-0-11.edge3.london2.level3.net" target="_blank">ae-0-11.edge3.london2.level3.net</a> 61.8% 468 1.1 1.6 1.0 18.0 2.5<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">4. <a href="http://ae-3-3.ebr1.paris1.level3.net" target="_blank">ae-3-3.ebr1.paris1.level3.net</a> 0.0% 468 109.9 171.8 103.0 246.0 26.9<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">5. <a href="http://ae-91-91.csw4.paris1.level3.net" target="_blank">ae-91-91.csw4.paris1.level3.net</a> 0.2% 468 111.7 170.4 99.6 241.8 27.0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">6. <a href="http://ae-92-92.ebr2.paris1.level3.net" target="_blank">ae-92-92.ebr2.paris1.level3.net</a> 0.0% 468 121.8 170.1 95.8 236.3 27.0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">7. <a href="http://ae-41-41.ebr2.washington1.level3.net" target="_blank">ae-41-41.ebr2.washington1.level3.net</a> 0.0% 467 118.5 171.4 95.7 241.1 27.1<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">8. <a href="http://ae-45-45.ebr2.washington12.level3.net" target="_blank">ae-45-45.ebr2.washington12.level3.net</a> 0.0% 467 113.4 171.8 94.9 243.5 27.0<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">9. <a href="http://ae-204-3604.edge1.washington12.level3.net" target="_blank">ae-204-3604.edge1.washington12.level3.net</a> 0.0% 467 107.7 171.7 95.4 250.7 27.8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> <a href="http://ae-201-3601.edge1.washington12.level3.net" target="_blank">ae-201-3601.edge1.washington12.level3.net</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> <a href="http://ae-203-3603.edge1.washington12.level3.net" target="_blank">ae-203-3603.edge1.washington12.level3.net</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> <a href="http://ae-202-3602.edge1.washington12.level3.net" target="_blank">ae-202-3602.edge1.washington12.level3.net</a><u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">10. <a href="tel:208.51.134.89" value="+12085113489" target="_blank">208.51.134.89</a> 0.0% 467 144.0 233.5 124.8 454.1 47.8<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">11. <a href="http://po1-20g.ar6.dca3.gblx.net" target="_blank">po1-20g.ar6.dca3.gblx.net</a> 0.0% 467 148.8 226.8 125.8 448.9 37.7<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">12. <a href="http://uunet-2.ar3.dca3.gblx.net" target="_blank">uunet-2.ar3.dca3.gblx.net</a> 0.0% 467 146.1 225.9 126.5 319.5 32.9<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">13. ???<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">14. tengige0-6-0-2.gw5.lax7.alter 0.0% 467 203.1 290.3 203.1 370.6 31.4<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">15. <a href="http://whitehousepost-gw.customer.alter.net" target="_blank">whitehousepost-gw.customer.alter.net</a> 0.0% 467 210.6 289.2 200.1 368.3 30.6<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><a name="1447a8b7f1a1a261__MailEndCompose"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></a></p>
<div>
<div style="border:none;border-top:solid #e1e1e1 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""> Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>]
<b>On Behalf Of </b><a href="mailto:alexzarris@yahoo.com" target="_blank">alexzarris@yahoo.com</a><br>
<b>Sent:</b> Tuesday, February 25, 2014 11:00 AM<br>
<b>To:</b> 'Clayton Zekelman'; 'Michael Morrison'; '<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>'<br>
<b>Subject:</b> Re: [outages] Level3 Routing...<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p>We’ve been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes:
<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>“Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative
troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral.”<u></u><u></u></p>
<p><u></u> <u></u></p>
<p>However, we’re still having internet issues.<u></u><u></u></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif""><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif"">Alex<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [<a href="mailto:outages-bounces@outages.org" target="_blank">mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Clayton Zekelman<br>
<b>Sent:</b> Tuesday, February 25, 2014 7:04 AM<br>
<b>To:</b> Michael Morrison; <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Level3 Routing...<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
That's what we told our customer... by the time they replied, the issue had gone away.<br>
<br>
At 09:57 AM 25/02/2014, Michael Morrison wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">Content-Language: en-US<br>
Content-Type: multipart/alternative;<br>
boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"<br>
<br>
As always, trace routes are helpful. We’ve been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.<br>
<br>
<b>From:</b> Outages [<a href="mailto:outages-bounces@outages.org" target="_blank"> mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Clayton Zekelman<br>
<b>Sent:</b> Tuesday, February 25, 2014 6:39 AM<br>
<b>To:</b> *Network; Luke Rockwell; Petter Bruland<br>
<b>Cc:</b> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Level3 Routing...<br>
<br>
<br>
<br>
We had a report from a customer earlier this morning, but they advised the problem went away.<br>
<br>
We connect to Level3 at Southfield MI.<br>
<br>
At 09:35 AM 25/02/2014, *Network wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Content-Language: en-US<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Content-Type: multipart/alternative;<u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Latency issue with Level 3 here in Detroit area
<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Anton<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">From: Outages [<a href="mailto:outages-bounces@outages.org" target="_blank">
</a><a href="mailto:outages-bounces@outages.org" target="_blank">mailto:outages-bounces@outages.org</a>] On Behalf Of Luke Rockwell<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Sent: Tuesday, February 18, 2014 12:24 PM<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">To: Petter Bruland<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Cc: <a href="mailto:Outages@outages.org" target="_blank">
Outages@outages.org</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Subject: Re: [outages] Level3 Routing...<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
nothing on the level 3 network events page I have access to<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Luke<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland <<a href="mailto:Petter.Bruland@allegiantair.com" target="_blank"> Petter.Bruland@allegiantair.com</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Not seeing any issues with Level3 in Las Vegas.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">-Petter<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">From: Outages [<a href="mailto:outages-bounces@outages.org" target="_blank">
</a><a href="mailto:outages-bounces@outages.org" target="_blank">mailto:outages-bounces@outages.org</a>] On Behalf Of Nick Olsen<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Sent: Tuesday, February 18, 2014 6:04 AM<u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
To: Morgan Miskell; <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Subject: Re: [outages] Level3 Routing...<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Not seeing anything homed off of Level 3 Orlando.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Traceroutes to a few common places look fine.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">~474815 Routes from 3356.<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Nick Olsen<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Network Operations <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">(855) FLSPEED x106<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<div style="margin-left:.5in">
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
</div>
<p class="MsoNormal" style="margin-left:.5in">From: "Morgan Miskell" <<a href="mailto:morgan.miskell@caro.net" target="_blank">morgan.miskell@caro.net</a> ><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Sent: Tuesday, February 18, 2014 9:00 AM<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">To: "<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>><u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
Subject: [outages] Level3 Routing...<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Anyone seeing Level3 routing issues? We just observed a massive drop in<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">routes in the BGP feeds and seeing some strange ICMP issues along their<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">backbone in terms of traceroutes along their backbone. They might be<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">adding some filters based on the recent DDOS attacks, I can ping<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">interfaces but traceroutes go to hell...<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">-- <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Morgan A. Miskell<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">CaroNet Data Centers<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="tel:704-643-8330%20x206" value="+17046438330" target="_blank">704-643-8330 x206</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">----------------------------------------------------------------------------<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">The information contained in this e-mail is confidential and is intended<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">only for the named recipient(s). If you are not the intended recipient<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">you must not copy, distribute, or take any action or reliance on it. If<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">you have received this e-mail in error, please notify the sender. Any<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">unauthorized disclosure of the information contained in this e-mail is<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">strictly prohibited.<u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
----------------------------------------------------------------------------<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">_______________________________________________<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Outages mailing list<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">_______________________________________________<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Outages mailing list<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"> <u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">_______________________________________________<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in">Outages mailing list<u></u><u></u></p>
<p class="MsoNormal" style="margin-left:.5in"><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><u></u><u></u></p>
<p class="MsoNormal" style="margin-right:0in;margin-bottom:12.0pt;margin-left:.5in">
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
<p class="MsoNormal">---<br>
<br>
Clayton Zekelman<br>
Managed Network Systems Inc. (MNSi)<br>
3363 Tecumseh Rd. E<br>
Windsor, Ontario<br>
N8W 1H4<br>
<br>
tel. <a href="tel:519-985-8410" value="+15199858410" target="_blank">519-985-8410</a><br>
fax. <a href="tel:519-985-8409" value="+15199858409" target="_blank">519-985-8409</a> <br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
<p>---<br>
<br>
Clayton Zekelman<br>
Managed Network Systems Inc. (MNSi)<br>
3363 Tecumseh Rd. E<br>
Windsor, Ontario<br>
N8W 1H4<br>
<br>
tel. <a href="tel:519-985-8410" value="+15199858410" target="_blank">519-985-8410</a><br>
fax. <a href="tel:519-985-8409" value="+15199858409" target="_blank">519-985-8409</a> <u></u><u></u></p>
</div>
</div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div>