<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks Tim and Darrell, these updates are priceless for us, since the provider we use to connect to Level3 still seems to be totally clueless about the issue.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Would you mind sharing the Level3 master ticket # so we can raise their awareness?<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>If you receive any further updates, especially confirmation of issue resolution, it would be wonderful if you post that to this list, too.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>Thanks again.<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'> Outages [mailto:outages-bounces@outages.org] <b>On Behalf Of </b>Tim Glen<br><b>Sent:</b> Saturday, October 19, 2013 7:20 PM<br><b>To:</b> Darrell Hyde<br><b>Cc:</b> OUTAGES Mailing List<br><b>Subject:</b> Re: [outages] Road Runner/Level 3 broked?<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Hardware failure and apparently the part that was sent to fix was also broke... This is from the most recent update to the ticket.<o:p></o:p></p></div><div><p class=MsoNormal><br>The Transport NOC has reported that the field technician completed rerouting traffic; however, some services failed to restore. The replacement circuit pack is still in transit to the site, and an emergency maintenance window has been scheduled for <a href="x-apple-data-detectors://3">04:00 GMT</a> to perform the replacement. Once the circuit pack is replaced, customers will be reverted to the original route. All services will be impacted while traffic is being routed back. The next update will be sent once it has been confirmed that the maintenance activity has commenced, or should any additional information become available.<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>On Oct 19, 2013, at 12:55 PM, "Darrell Hyde" <<a href="mailto:DHyde@hosting.com">DHyde@hosting.com</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt;-webkit-text-size-adjust: auto'><div><p class=MsoNormal>Were they able to provide any detail with respect to root cause?<br>________________________________________<br>From: Outages [<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>] On Behalf Of Darren Schreiber [<a href="mailto:darren@2600hz.com">darren@2600hz.com</a>]<br>Sent: Saturday, October 19, 2013 12:01 PM<br>To: Tim Glen; 'Sajal Kayan'; OUTAGES Mailing List<br>Subject: Re: [outages] Road Runner/Level 3 broked?<br><br>Level3 just called me back for the ticket I filed. Tech was very knowledgable. Is aware of the issue (obviously) and has attached my ticket to the master event ticket so that I can see realtime updates.<br><br>Appears the ETA is within the next 2 hours.<br><br>From: Darren Schreiber <<a href="mailto:darren@2600hz.com">darren@2600hz.com</a><<a href="mailto:darren@2600hz.com">mailto:darren@2600hz.com</a>>><br>Date: Saturday, October 19, 2013 7:03 AM<br>To: Tim Glen <<a href="mailto:tglen@asicentral.com">tglen@asicentral.com</a><<a href="mailto:tglen@asicentral.com">mailto:tglen@asicentral.com</a>>>, 'Sajal Kayan' <<a href="mailto:sajal83@gmail.com">sajal83@gmail.com</a><<a href="mailto:sajal83@gmail.com">mailto:sajal83@gmail.com</a>>>, OUTAGES Mailing List <<a href="mailto:Outages@outages.org">Outages@outages.org</a><<a href="mailto:Outages@outages.org">mailto:Outages@outages.org</a>>><br>Subject: Re: [outages] Road Runner/Level 3 broked?<br><br>I can confirm the same from our datacenter in NY. Tracepath goes in a loop…<br><br>From: Tim Glen <<a href="mailto:tglen@asicentral.com">tglen@asicentral.com</a><<a href="mailto:tglen@asicentral.com">mailto:tglen@asicentral.com</a>>><br>Date: Saturday, October 19, 2013 6:59 AM<br>To: 'Sajal Kayan' <<a href="mailto:sajal83@gmail.com">sajal83@gmail.com</a><<a href="mailto:sajal83@gmail.com">mailto:sajal83@gmail.com</a>>>, OUTAGES Mailing List <<a href="mailto:Outages@outages.org">Outages@outages.org</a><<a href="mailto:Outages@outages.org">mailto:Outages@outages.org</a>>><br>Subject: Re: [outages] Road Runner/Level 3 broked?<br><br>I just opened a ticket with L3.<br><br>I've been having packet loss since 9:07AM today over my L3 circuit.<br><br>I am in Philly and L3's customer access router is experiencing high packet loss.<br><br><br><br>~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>Timothy Glen<br>Sr. Network Engineer<br>Advertising Specialty Institute | Promoting Success<br><br><a href="mailto:tglen@asicentral.com">tglen@asicentral.com</a><<a href="mailto:tglen@asicentral.com">mailto:tglen@asicentral.com</a>><br>tel: (215) 953-3692<br>fax: (215) 953-3759<br>Think before you print.<br><br><br><br>________________________________<br>From: Outages [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>] On Behalf Of Sajal Kayan<br>Sent: Saturday, October 19, 2013 9:52 AM<br>To: OUTAGES Mailing List<br>Subject: [outages] Road Runner/Level 3 broked?<br><br>Using road runner in Brooklyn NY.<br><br>Cant access certain sites ( Facebook, twitter, citibank) . All of them have one thing in common, they go thru level3.<br><br>(FYI full name of hop 7 is xe-4-2-0.<a href="http://edge4.frankfurt1.level3.net">edge4.frankfurt1.level3.net</a><<a href="http://xe-4-2-0.edge4.frankfurt1.level3.net">http://xe-4-2-0.edge4.frankfurt1.level3.net</a>> )<br><br>sajal@sajal-laptop:~$ mtr --report <a href="http://www.citi.com">www.citi.com</a><<a href="http://www.citi.com">http://www.citi.com</a>><br>HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev<br> 1.|-- 192.168.1.1 0.0% 10 1.2 110.3 1.0 603.9 230.5<br> 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 37.9 207.9 17.0 1418. 444.2<br> 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 9.8 93.3 9.4 471.9 170.3<br> 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 12.6 78.3 10.1 402.2 142.2<br> 5.|-- 107.14.19.22 0.0% 10 13.8 64.6 11.5 331.0 112.6<br> 6.|-- ae-0-0.pr0.nyc30.tbone.rr 0.0% 10 13.8 50.9 11.4 257.6 82.3<br> 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 13.4 35.0 9.1 186.5 55.8<br> 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0<br>sajal@sajal-laptop:~$<br><br><br>sajal@sajal-laptop:~$ mtr --report <a href="http://www.twitter.com">www.twitter.com</a><<a href="http://www.twitter.com">http://www.twitter.com</a>><br>HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev<br> 1.|-- 192.168.1.1 0.0% 10 1.2 2.9 1.0 9.5 2.8<br> 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 151.6 184.3 75.8 529.8 137.6<br> 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 252.6 160.8 64.7 299.7 87.2<br> 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 243.5 166.4 64.1 345.8 88.3<br> 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 246.7 189.2 88.6 309.1 75.7<br> 6.|-- 107.14.17.169 0.0% 10 256.6 175.3 102.6 256.6 60.5<br> 7.|-- xe-4-2-0.edge4.frankfurt1 0.0% 10 258.6 179.6 95.4 302.2 69.1<br> 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0<br>sajal@sajal-laptop:~$<br><br>sajal@sajal-laptop:~$ mtr --report <a href="http://www.facebook.com">www.facebook.com</a><<a href="http://www.facebook.com">http://www.facebook.com</a>><br>HOST: sajal-laptop Loss% Snt Last Avg Best Wrst StDev<br> 1.|-- 192.168.1.1 0.0% 10 4.0 3.1 1.1 8.5 2.8<br> 2.|-- cpe-24-90-104-1.nyc.res.r 0.0% 10 225.9 163.7 97.8 252.8 57.0<br> 3.|-- tge-0-0-0-1.nycmnyr-rtr2. 0.0% 10 298.0 168.6 69.3 298.0 79.0<br> 4.|-- agg23.nyquny9101r.nyc.rr. 0.0% 10 292.5 155.0 90.8 292.5 71.0<br> 5.|-- ae-3-0.cr0.nyc20.tbone.rr 0.0% 10 239.0 148.6 68.5 248.6 63.2<br> 6.|-- 107.14.17.169 0.0% 10 268.1 148.7 73.0 268.1 64.9<br> 7.|-- xe-4-2-0.edge4.frankfurt1 10.0% 10 235.1 130.2 66.2 235.1 49.2<br> 8.|-- ??? 100.0 10 0.0 0.0 0.0 0.0 0.0<br>sajal@sajal-laptop:~$<br><br>-Sajal<o:p></o:p></p></div></blockquote></div></body></html>