<html><head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii"></head><body dir="auto"><div style="-webkit-text-size-adjust: auto; ">It appears to have been a new issue, but possibly related to the previous one. Level(3) reported the following this morning:</div><div style="-webkit-text-size-adjust: auto; "><br></div><div><font color="#000000"><span style="-webkit-text-size-adjust: auto; background-color: rgba(255, 255, 255, 0);"><a href="x-apple-data-detectors://1" x-apple-data-detectors="true" x-apple-data-detectors-type="calendar-event" x-apple-data-detectors-result="1">13:53 GMT</a> - The IP NOC reports that protocol drops following a scheduled maintenance, GCR <a href="tel:7622319" x-apple-data-detectors="true" x-apple-data-detectors-type="telephone" x-apple-data-detectors-result="2">7622319</a>, in Detroit, MI impacted IP services. The issue existed on a connection between two of the routers in Detroit, MI. The IP NOC has disabled that connection to allow traffic to flow through alternate routes and eliminate the protocol drops. The IP NOC will continue investigating the faulty connection in an attempt to determine the root cause and restore the connection for service. At this time, the IP NOC reports that services are up and stable</span></font><br><br><span style="-webkit-text-size-adjust: auto;">Sent from my iPhone</span></div><div style="-webkit-text-size-adjust: auto; "><br>On Mar 8, 2014, at 6:36 AM, "Mitch Patterson" <<a href="mailto:mitpatterson@gmail.com">mitpatterson@gmail.com</a>> wrote:<br><br></div><blockquote type="cite" style="-webkit-text-size-adjust: auto; "><div><p dir="ltr">They were doing emergency maintance likely related to the other night</p>
<div class="gmail_quote">On Mar 8, 2014 6:34 AM, "Greg Smythe" <<a href="mailto:greg@thesmythes.org">greg@thesmythes.org</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div lang="EN-US" link="blue" vlink="purple">
<div>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">As I type this I am seeing Level3 issues in the Detroit area. Been going on since about 4:30am Eastern. My co-lo provider had to disable peering to get us back
and running.<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">-=-=-=-=-=-=-=-=-=-=--=-=-<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Greg Smythe<u></u><u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"><u></u> <u></u></span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [mailto:<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Mitch Patterson<br>
<b>Sent:</b> Friday, March 07, 2014 12:51 AM<br>
<b>To:</b> Bill Wichers<br>
<b>Cc:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Level 3 Detroit issues again<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Just got this<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">*** CASCADED EXTERNAL NOTES 07-Mar-2014 05:48:37 GMT From CASE: 7621845 - Event The Operations Engineering team has isolated the cause of the packet loss failure. A change in parameters
on the Detroit's network router has been initiated to restore normal operations.<u></u><u></u></p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Mar 7, 2014 at 12:41 AM, Mitch Patterson <<a href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>> wrote:<u></u><u></u></p>
<div>
<p class="MsoNormal">Just got this<u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"><b><span style="font-size:9.0pt;font-family:"Arial","sans-serif";background:#f0f0f0">03/07/2014 05:06 AM GMT</span></b><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><br>
<br>
</span><u></u><u></u></p>
<pre style="line-height:12.95pt;background:#f0f0f0"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><u></u> <u></u></span></pre>
<pre style="line-height:12.95pt;background:#f0f0f0"><span style="font-size:9.0pt;font-family:"Arial","sans-serif""><u></u> <u></u></span></pre>
<pre style="line-height:12.95pt;background:#f0f0f0"><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">*** CASCADED EXTERNAL NOTES 07-Mar-2014 05:06:43 GMT From CASE: 7621845 - Event<u></u><u></u></span></pre>
<pre style="line-height:12.95pt;background:#f0f0f0"><span style="font-size:9.0pt;font-family:"Arial","sans-serif"">The Technical Services Center (TSC) detected a packet loss condition on a network router in Detroit, MI. The Operations Engineering team has been engaged on the issue and they are currently analyzing the packet loss condition to determine the cause of the failure.<u></u><u></u></span></pre>
</div>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Fri, Mar 7, 2014 at 12:01 AM, Mitch Patterson <<a href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>> wrote:<u></u><u></u></p>
<div>
<p class="MsoNormal">Do you know are the circuits you are on legacy GLBX? from what i'm told by our guy who handles the colo the legacy GLBX ones are fine. but the other ones are not.<u></u><u></u></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><u></u> <u></u></p>
<div>
<p class="MsoNormal">On Thu, Mar 6, 2014 at 11:59 PM, Bill Wichers <<a href="mailto:billw@waveform.net" target="_blank">billw@waveform.net</a>> wrote:<u></u><u></u></p>
<p class="MsoNormal">We connect into them in Detroit and we don't see this. We aren't in their colo though.<br>
<br>
-Bill<br>
<br>
Sent from my iPhone<u></u><u></u></p>
<div>
<p class="MsoNormal"><br>
On Mar 6, 2014, at 11:44 PM, "Mitch Patterson" <<a href="mailto:mitpatterson@gmail.com" target="_blank">mitpatterson@gmail.com</a>> wrote:<br>
<br>
> We are seeing over 20% packet loss on our circuits at their colo in Southfield. They are telling me its been refereed to their IP core team as its a high level issue. Anyone else seeing this?<u></u><u></u></p>
</div>
<p class="MsoNormal">> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><u></u><u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div>
</div>
</blockquote></div>
</div></blockquote></body></html>