<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<style type="text/css" style="display:none"><!-- p { margin-top: 0px; margin-bottom: 0px; } @font-face { font-family: "Cambria Math"; } @font-face { font-family: Calibri; } @font-face { font-family: Consolas; } p.MsoNormal, li.MsoNormal, div.MsoNormal { margin: 0in 0in 0.0001pt; font-size: 11pt; font-family: Calibri; } a:link, span.MsoHyperlink { color: blue; text-decoration: underline; } a:visited, span.MsoHyperlinkFollowed { color: purple; text-decoration: underline; } span.EmailStyle17 { font-family: Calibri; color: windowtext; } span.EmailStyle18 { font-family: Calibri; color: windowtext; } span.EmailStyle19 { font-family: Calibri; color: windowtext; } span.msoIns { text-decoration: underline; color: teal; } .MsoChpDefault { font-size: 10pt; } @page WordSection1 { margin: 1in; } div.WordSection1 { }--></style>
</head>
<body dir="ltr" style="font-size:12pt;color:#000000;background-color:#FFFFFF;font-family:Calibri,Arial,Helvetica,sans-serif;">
<p>Got this couple hours ago.<br>
</p>
<p><br>
</p>
<p><span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">*** CASCADED EXTERNAL NOTES 25-Aug-2016 23:03:17 GMT From CASE: 11364554 - Event</span><br style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">
<span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">Event Conclusion Summary</span><br style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">
<span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">Outage Start: August 25, 2016 20:00 GMT</span><br style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">
<span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">Outage Stop: August 25, 2016 21:50 GMT</span><br style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">
<span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">Root Cause: An unknown issue was causing a link to bounce in San Diego, CA, impacting
IP services.</span><br style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">
<span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">Fix Action: The IP NOC shutdown the bouncing link to resolve the issue and return
services to a stable state.</span><br style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">
<span style="color: rgb(33, 33, 33); font-family: "Segoe UI", "Segoe WP", "Segoe UI WPC", Tahoma, Arial, sans-serif; font-size: 13.3333px; background-color: rgb(255, 255, 255);">Reason for Outage (RFO) Summary: The IP NOC reported that an unknown issue was
causing a link to bounce in San Diego, CA, impacting IP services. The IP NOC shutdown the bouncing link to resolve the issue and return services to a stable state. Tier III Technical Support and the equipment vendor were engaged for further investigation into
the root cause. This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure. If additional issues are
experienced, please contact the Level 3 Technical Service Center at 1-877-4LEVEL3.</span><br>
</p>
<p><br>
</p>
<p>Thanks,<br>
</p>
<p>Bobin<br>
</p>
<div style="color: rgb(33, 33, 33);">
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Outages <outages-bounces@outages.org> on behalf of Eric Kuhnke via Outages <outages@outages.org><br>
<b>Sent:</b> Thursday, August 25, 2016 3:20 PM<br>
<b>To:</b> outages@outages.org<br>
<b>Subject:</b> Re: [outages] Level3 circuit issues in SoCal</font>
<div> </div>
</div>
<div>
<p>It is interesting to see that the reliability of an IP transit provider sometimes has no direct correlation with their pricing structure for various commitments in $/MRC for a given value of Gbps. No matter what the marketing materials/sales say about how
awesome their network is.<br>
</p>
<p>In my experience, in a particular metro area, buying transit from a well-peered medium sized regional tier 2/3 size carrier
<i>where you personally know the people who have enable on the core routers</i> can result in a more resilient network.<br>
</p>
<br>
<div class="moz-cite-prefix">On 8/25/16 3:15 PM, David Hubbard via Outages wrote:<br>
</div>
<blockquote type="cite">
<meta name="Title" content="">
<meta name="Keywords" content="">
<meta name="Generator" content="Microsoft Word 15 (filtered
medium)">
<style>
<!--
@font-face
{font-family:"Cambria Math"}
@font-face
{font-family:Calibri}
@font-face
{font-family:Consolas}
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:Calibri}
a:link, span.MsoHyperlink
{color:blue;
text-decoration:underline}
a:visited, span.MsoHyperlinkFollowed
{color:purple;
text-decoration:underline}
span.EmailStyle17
{font-family:Calibri;
color:windowtext}
span.EmailStyle18
{font-family:Calibri;
color:windowtext}
span.EmailStyle19
{font-family:Calibri;
color:windowtext}
span.msoIns
{text-decoration:underline;
color:teal}
.MsoChpDefault
{font-size:10.0pt}
@page WordSection1
{margin:1.0in 1.0in 1.0in 1.0in}
div.WordSection1
{}
-->
</style>
<div class="WordSection1">
<p class="MsoNormal">Just got this:</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.0pt; font-family:Consolas">*** CASCADED EXTERNAL NOTES 25-Aug-2016 22:11:34 GMT From CASE: 11364554 - Event</span></p>
<p class="MsoNormal" style="text-autospace:none"><span style="font-size:12.0pt; font-family:Consolas">The IP NOC reported that an unknown issue was causing a link to bounce in San Diego, CA, impacting IP services. The IP NOC shutdown the bouncing link to resolve
the issue and return services to a stable state. Tier III Technical Support and the equipment vendor were engaged for further investigation into the root cause. The IP NOC is continuing to monitor and a final notification will be sent when stability has been
confirmed.</span></p>
<p class="MsoNormal"> </p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">But yeah, I feel the same way. Their network has been an absolute mess since they started trying to merge the legacy Twtelecom network together; it’s been really bad as of late, and any redundancy we had via the other provider is now gone
as we have some locations with only L3 / TW circuits; both have issues at the same time now if it’s routing related. Ugh</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">David</p>
<p class="MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF
1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Matt Hoppes
<a class="moz-txt-link-rfc2396E" href="mailto:mattlists@rivervalleyinternet.net">
<mattlists@rivervalleyinternet.net></a><br>
<b>Date: </b>Thursday, August 25, 2016 at 6:11 PM<br>
<b>To: </b>David Hubbard <a class="moz-txt-link-rfc2396E" href="mailto:dhubbard@dino.hostasaurus.com">
<dhubbard@dino.hostasaurus.com></a><br>
<b>Cc: </b>Bobin Joseph <a class="moz-txt-link-rfc2396E" href="mailto:Bobin.Joseph@mitchell.com">
<Bobin.Joseph@mitchell.com></a>, <a class="moz-txt-link-rfc2396E" href="mailto:outages@outages.org">
"outages@outages.org"</a> <a class="moz-txt-link-rfc2396E" href="mailto:outages@outages.org">
<outages@outages.org></a><br>
<b>Subject: </b>Re: [outages] Level3 circuit issues in SoCal</span><span style="font-size:12.0pt; color:black"></span></p>
</div>
<div>
<p class="MsoNormal"><span style=""> </span></p>
</div>
<div>
<div>
<div>
<p class="MsoNormal">What in the world is going on with the Level3 network over the last two weeks? It's falling apart. </p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
On Aug 25, 2016, at 17:46, David Hubbard via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:</p>
</div>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p class="MsoNormal">Same; San Diego area fiber connection. Does not seem to be affecting our legacy TW phone circuits. Happened about an hour ago and has just started again. Was about to open a ticket.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">David</p>
<p class="MsoNormal"> </p>
<div style="border:none; border-top:solid #B5C4DF
1.0pt; padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="color:black">From: </span></b><span style="color:black">Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>> on behalf of Bobin Joseph via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
<b>Reply-To: </b>Bobin Joseph <<a href="mailto:Bobin.Joseph@mitchell.com">Bobin.Joseph@mitchell.com</a>><br>
<b>Date: </b>Thursday, August 25, 2016 at 5:38 PM<br>
<b>To: </b>"<a href="mailto:outages@outages.org">outages@outages.org</a>" <<a href="mailto:outages@outages.org">outages@outages.org</a>><br>
<b>Subject: </b>[outages] Level3 circuit issues in SoCal</span></p>
</div>
<div>
<p class="MsoNormal"><span style="font-family:"Times New Roman""> </span></p>
</div>
<div>
<div>
<p class="MsoNormal">Are anybody else experiencing issues with Level3 in the Southern California region?</p>
<p class="MsoNormal">We are seeing constant intermittent outages on our internet and VPLS circuit.</p>
<p class="MsoNormal"> </p>
<p class="MsoNormal">Thanks,</p>
<p class="MsoNormal">Bobin </p>
<p class="MsoNormal"> </p>
</div>
</div>
</div>
</blockquote>
<blockquote style="margin-top:5.0pt; margin-bottom:5.0pt">
<div>
<p class="MsoNormal"><span style="">_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a></span></p>
</div>
</blockquote>
</div>
</div>
</div>
<br>
<fieldset class="mimeAttachmentHeader"></fieldset> <br>
<pre>_______________________________________________
Outages mailing list
<a class="moz-txt-link-abbreviated" href="mailto:Outages@outages.org">Outages@outages.org</a>
<a class="moz-txt-link-freetext" href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a>
</pre>
</blockquote>
<br>
</div>
</div>
</body>
</html>