<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 14 (filtered medium)"><!--[if !mso]><style>v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style><![endif]--><style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"Plain Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.PlainTextChar
{mso-style-name:"Plain Text Char";
mso-style-priority:99;
mso-style-link:"Plain Text";
font-family:"Calibri","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoPlainText>We’ve been having issues in San Jose the past two days on both L3 MPLS and Internet circuits. MPLS was resolved yesterday with these notes: <o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>“Event The IP NOC advised that troubleshooting is ongoing with the equipment vendor to further isolate the point of failure. Additionally, it has been advised that voice services are impacted and calls are failing network wide. Collaborative troubleshooting efforts between the IP and Voice NOC and the equipment vendor will continue in an effort to expedite service restoral.”<o:p></o:p></p><p class=MsoPlainText><o:p> </o:p></p><p class=MsoPlainText>However, we’re still having internet issues.<o:p></o:p></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>Alex<o:p></o:p></span></p><p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'><o:p> </o:p></span></p><div><div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> Outages [mailto:outages-bounces@outages.org] <b>On Behalf Of </b>Clayton Zekelman<br><b>Sent:</b> Tuesday, February 25, 2014 7:04 AM<br><b>To:</b> Michael Morrison; Outages@outages.org<br><b>Subject:</b> Re: [outages] Level3 Routing...<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal><br>That's what we told our customer... by the time they replied, the issue had gone away.<br><br>At 09:57 AM 25/02/2014, Michael Morrison wrote:<br><br><o:p></o:p></p><p class=MsoNormal style='margin-bottom:12.0pt'>Content-Language: en-US<br>Content-Type: multipart/alternative;<br> boundary="_000_0693cfd728744391a68dbd0e93123a64EX01wcilocal_"<br><br>As always, trace routes are helpful. We’ve been having intermittent issue with Level 3 routes for the past couple weeks, most issue for us seem to be when going from Level 3 to Global Crossing routes.<br> <br><b>From:</b> Outages [<a href="mailto:outages-bounces@outages.org"> mailto:outages-bounces@outages.org</a>] <b>On Behalf Of </b>Clayton Zekelman<br><b>Sent:</b> Tuesday, February 25, 2014 6:39 AM<br><b>To:</b> *Network; Luke Rockwell; Petter Bruland<br><b>Cc:</b> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br><b>Subject:</b> Re: [outages] Level3 Routing...<br> <br><br><br>We had a report from a customer earlier this morning, but they advised the problem went away.<br><br>We connect to Level3 at Southfield MI.<br><br>At 09:35 AM 25/02/2014, *Network wrote:<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Content-Language: en-US<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Content-Type: multipart/alternative;<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'> boundary="_000_391AAB48E84BB24288BED210F379E1DC4DEA4C02NOCMAILDAG2Inte_"<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Latency issue with Level 3 here in Detroit area <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Anton<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>From: Outages [<a href="mailto:outages-bounces@outages.org"> </a><a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>] On Behalf Of Luke Rockwell<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Sent: Tuesday, February 18, 2014 12:24 PM<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>To: Petter Bruland<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Cc: <a href="mailto:Outages@outages.org">Outages@outages.org</a><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Subject: Re: [outages] Level3 Routing...<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'>nothing on the level 3 network events page I have access to<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Luke<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>On Tue, Feb 18, 2014 at 9:17 AM, Petter Bruland <<a href="mailto:Petter.Bruland@allegiantair.com"> Petter.Bruland@allegiantair.com</a>> wrote:<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Not seeing any issues with Level3 in Las Vegas.<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>-Petter<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>From: Outages [<a href="mailto:outages-bounces@outages.org"> </a><a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>] On Behalf Of Nick Olsen<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Sent: Tuesday, February 18, 2014 6:04 AM<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'>To: Morgan Miskell; <a href="mailto:Outages@outages.org">Outages@outages.org</a><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Subject: Re: [outages] Level3 Routing...<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Not seeing anything homed off of Level 3 Orlando.<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Traceroutes to a few common places look fine.<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>~474815 Routes from 3356.<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Nick Olsen<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Network Operations <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>(855) FLSPEED x106<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><div class=MsoNormal align=center style='margin-left:.5in;text-align:center'><hr size=2 width="100%" align=center></div><p class=MsoNormal style='margin-left:.5in'>From: "Morgan Miskell" <<a href="mailto:morgan.miskell@caro.net">morgan.miskell@caro.net</a> ><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Sent: Tuesday, February 18, 2014 9:00 AM<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>To: "<a href="mailto:Outages@outages.org">Outages@outages.org</a>" <<a href="mailto:outages@outages.org">outages@outages.org</a>><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'>Subject: [outages] Level3 Routing...<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Anyone seeing Level3 routing issues? We just observed a massive drop in<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>routes in the BGP feeds and seeing some strange ICMP issues along their<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>backbone in terms of traceroutes along their backbone. They might be<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>adding some filters based on the recent DDOS attacks, I can ping<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>interfaces but traceroutes go to hell...<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>-- <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Morgan A. Miskell<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>CaroNet Data Centers<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>704-643-8330 x206<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>----------------------------------------------------------------------------<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>The information contained in this e-mail is confidential and is intended<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>only for the named recipient(s). If you are not the intended recipient<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>you must not copy, distribute, or take any action or reliance on it. If<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>you have received this e-mail in error, please notify the sender. Any<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>unauthorized disclosure of the information contained in this e-mail is<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>strictly prohibited.<o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'>----------------------------------------------------------------------------<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>_______________________________________________<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Outages mailing list<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><a href="mailto:Outages@outages.org">Outages@outages.org</a><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>_______________________________________________<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Outages mailing list<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><a href="mailto:Outages@outages.org">Outages@outages.org</a><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'> <o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>_______________________________________________<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'>Outages mailing list<o:p></o:p></p><p class=MsoNormal style='margin-left:.5in'><a href="mailto:Outages@outages.org">Outages@outages.org</a><o:p></o:p></p><p class=MsoNormal style='mso-margin-top-alt:0in;margin-right:0in;margin-bottom:12.0pt;margin-left:.5in'><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p><p class=MsoNormal>---<br><br>Clayton Zekelman<br>Managed Network Systems Inc. (MNSi)<br>3363 Tecumseh Rd. E<br>Windsor, Ontario<br>N8W 1H4<br><br>tel. 519-985-8410<br>fax. 519-985-8409 <br>_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@outages.org">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p><p>---<br><br>Clayton Zekelman<br>Managed Network Systems Inc. (MNSi)<br>3363 Tecumseh Rd. E<br>Windsor, Ontario<br>N8W 1H4<br><br>tel. 519-985-8410<br>fax. 519-985-8409 <o:p></o:p></p></div></body></html>