<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p
{mso-style-priority:99;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
pre
{mso-style-priority:99;
mso-style-link:"HTML Preformatted Char";
margin:0in;
margin-bottom:.0001pt;
font-size:10.0pt;
font-family:"Courier New";}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.HTMLPreformattedChar
{mso-style-name:"HTML Preformatted Char";
mso-style-priority:99;
mso-style-link:"HTML Preformatted";
font-family:Consolas;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
span.EmailStyle22
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle24
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Not trying to beat a dead horse, but any ongoing issues with Level 3 ? Screenshot
<a href="http://i.imgur.com/VuO9nrh.png">http://i.imgur.com/VuO9nrh.png</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><a href="http://www1.internetpulse.net/">http://www1.internetpulse.net/</a> shows packet loss<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Bobin<o:p></o:p></span></p>
</div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<div>
<div style="border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [mailto:outages-bounces@outages.org]
<b>On Behalf Of </b>Bobin Joseph via Outages<br>
<b>Sent:</b> Friday, August 28, 2015 6:57 AM<br>
<b>To:</b> outages@outages.org<br>
<b>Subject:</b> Re: [outages] Level3/TW Telecom issues?<o:p></o:p></span></p>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">So that is how they lowered the traffic on the cards in Chicago.<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Thanks,<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D">Bobin<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Rafael Possamai via Outages<br>
<b>Sent:</b> Friday, August 28, 2015 6:26 AM<br>
<b>To:</b> Ryan Brooks<br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Level3/TW Telecom issues?<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">August 27th, 2015: the day everyone routed around L3<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Fri, Aug 28, 2015 at 1:27 AM, Ryan Brooks via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">All my tickets got closed w/o confirmation. Good thing they turned down the tcp-disruption settings / "perform adjustments on the links". It is also very disconcerting that ICMP and IPSec seemed to do better during this capacity problem;
perception is important w/ traffic engineering, but not so much with NOC mad-libs.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">Simultaneous to the below update, another notice indicated that OSP completed splice work which fixed the problem. So we're double-fixed now.<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Aug 28, 2015, at 1:13 AM, Ryan Gelobter <<a href="mailto:ryan.g@atwgpc.net" target="_blank">ryan.g@atwgpc.net</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p>Tonight..we did nothing and waited for everyone to go to bed.<o:p></o:p></p>
<p>08/28/2015 3:08 AM GMT<br>
Event Conclusion Summary<o:p></o:p></p>
<p>Start: August 27, 2015 13:20 GMT<br>
Stop: August 28, 2015 00:00 GMT<o:p></o:p></p>
<p>Root Cause: A protocol issue impacted IP services in multiple markets.<br>
Fix Action: Adjustments were made to clear the errors.<o:p></o:p></p>
<p>Summary:<br>
The IP NOC began investigating the root cause with Tier III Technical Support. It was reported that the issue was causing packet loss for customers. Operations Engineering teams were engaged, and Field Services were dispatched to a site in Chicago, IL to assist
with investigations. Troubleshooting identified a protocol issue, and Operations Engineering worked with Tier III Technical Support to perform adjustments on the links. It was confirmed that the errors cleared. The traffic load was also lowered on cards in
Chicago to alleviate any further issues. Should any additional impact be experienced, please contact the Level 3 Technical Service Center.<o:p></o:p></p>
<div>
<p class="MsoNormal">On Aug 27, 2015 8:07 PM, "Ryan Brooks via Outages" <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Yesterday, fiber cut. Today, "Internet Protocol" problems. Tonight, capacity management?<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Aug 27, 2015, at 8:01 PM, Ryan Gelobter via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<p class="MsoNormal"><strong><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333;background:#F3F3F3">08/27/2015 11:16 PM GMT</span></strong><o:p></o:p></p>
<pre style="margin-bottom:6.75pt;line-height:13.5pt;background:#F3F3F3"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Multiple cards have been ordered, and will be installed on equipment in Chicago, IL in an effort to balance the traffic load through the site and alleviate the issue. Field Services are currently en route to the site with the cards, and are expected to arrive shortly.<o:p></o:p></span></pre>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Aug 27, 2015 at 5:02 PM, Ryan Gelobter <<a href="mailto:ryan.g@atwgpc.net" target="_blank">ryan.g@atwgpc.net</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">Latest update.....<br>
<strong><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333;background:#F3F3F3">08/27/2015 9:49 PM GMT</span></strong><o:p></o:p></p>
<pre style="margin-bottom:6.75pt;line-height:13.5pt;background:#F3F3F3"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Tier III Technical Support and Operations Engineering teams continue investigating the issue identified with the Internet Protocols. Restoration efforts are ongoing; however, there is currently no estimated restoral time.<o:p></o:p></span></pre>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Aug 27, 2015 at 3:14 PM, Ryan Brooks <<a href="mailto:rbrooks@stack41.com" target="_blank">rbrooks@stack41.com</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal">ICMP blazes through at times, but TCP completely falls apart.<o:p></o:p></p>
<div>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<blockquote style="margin-top:5.0pt;margin-bottom:5.0pt">
<div>
<p class="MsoNormal">On Aug 27, 2015, at 3:13 PM, Ryan Gelobter via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<div>
<pre style="margin-bottom:6.75pt;line-height:13.5pt;background:#F3F3F3;border-radius:4px;white-space:pre-wrap;word-wrap:break-word;max-width:800px;word-spacing:0px"><strong><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333;background:#F3F3F3">08/27/2015 8:05 PM GMT</span></strong><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333;background:#F3F3F3"><br></span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Level 3 Tier III and Operations Engineering teams have identified Internet Protocols dropping, affecting customer services. Restoration efforts are in progress, but an estimated time of restoral is not available at this time.<o:p></o:p></span></pre>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Aug 27, 2015 at 3:06 PM, Ryan Gelobter <<a href="mailto:ryan.g@atwgpc.net" target="_blank">ryan.g@atwgpc.net</a>> wrote:<o:p></o:p></p>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt">If you have access to the Level3 portal you should see ticket #9639047 under Network Events now.<o:p></o:p></p>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><br>
<b><span style="background:#F3F3F3">Event Summary</span></b><br>
<span style="background:#F3F3F3">IP Network Event ~ Multiple Markets</span><b><br>
<br>
08/27/2015 6:36 PM GMT</b><o:p></o:p></span></p>
<pre style="margin-bottom:6.75pt;line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">IP and Transport Tier III, Operations Engineering and Field Services continue collaboratively working the issue.<o:p></o:p></span></pre>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">08/27/2015 4:59 PM GMT</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p></o:p></span></p>
<pre style="margin-bottom:6.75pt;line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">Operations Engineering is engaged and Field Services is on site in Chicago, IL investigating the issue.<o:p></o:p></span></pre>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">08/27/2015 4:38 PM GMT</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p></o:p></span></p>
<pre style="margin-bottom:6.75pt;line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">The engineers are currently migrating traffic in efforts of restoring services while troubleshooting continues. Field Services is being dispatched to a Chicago, IL site to assist.<o:p></o:p></span></pre>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p> </o:p></span></p>
<p class="MsoNormal" style="line-height:11.25pt;background:#F3F3F3"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">08/27/2015 4:21 PM GMT</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333"><o:p></o:p></span></p>
<pre style="margin-bottom:6.75pt;line-height:13.5pt"><span style="font-size:9.0pt;font-family:"Arial","sans-serif";color:#333333">IP services are affected across multiple markets and the root cause is currently under investigation. The IP NOC and IP and Transport Tier III are actively troubleshooting and working to isolate the cause. The engineers have detected peering issues which are resulting in packet loss for customers. Please be advised that updates will be provided at minimum of hourly unless otherwise noted.<o:p></o:p></span></pre>
<div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Thu, Aug 27, 2015 at 2:59 PM, Eric Henson via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<o:p></o:p></p>
<p class="MsoNormal" style="margin-bottom:12.0pt">We had to shut down our Level3 circuit in the DFW area, had issues from 1406CST until we shut down our circuit.<br>
<br>
<br>
-----Original Message-----<br>
From: Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>] On Behalf Of jungle Boogie via Outages<br>
Sent: Thursday, August 27, 2015 2:56 PM<br>
To: Bobin Joseph<br>
Cc: <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
Subject: Re: [outages] Level3/TW Telecom issues?<o:p></o:p></p>
<div>
<div>
<p class="MsoNormal">On 27 August 2015 at 12:48, Bobin Joseph via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br>
> Anybody else having complete outage on Level3? We lost complete<br>
> internet in DFW and SAN<br>
><br>
<br>
We were not down but significant routing issues in Las Vegas.<br>
<br>
> Thanks,<br>
> Bobin<br>
><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<p class="MsoNormal">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</div>
</div>
</blockquote>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</div>
</div>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><o:p></o:p></p>
</div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</div>
</body>
</html>