<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=us-ascii">
<meta name=Generator content="Microsoft Word 12 (filtered medium)">
<!--[if !mso]>
<style>
v\:* {behavior:url(#default#VML);}
o\:* {behavior:url(#default#VML);}
w\:* {behavior:url(#default#VML);}
.shape {behavior:url(#default#VML);}
</style>
<![endif]-->
<title>Re: [outages] Denver Level3 problems?</title>
<style>
<!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page Section1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.Section1
{page:Section1;}
-->
</style>
<!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang=EN-US link=blue vlink=purple>
<div class=Section1>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'>They have told us that were trying to isolate some dirty circuits
from an etherchannel bundle and that went way wrong… In the Denver area… data
center/POP .. whatever.. Level 3 rocks…<o:p></o:p></span></p>
<p class=MsoNormal><span style='font-size:11.0pt;font-family:"Calibri","sans-serif";
color:#1F497D'><o:p> </o:p></span></p>
<div>
<div style='border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in 0in 0in'>
<p class=MsoNormal><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>
outages-bounces@outages.org [mailto:outages-bounces@outages.org] <b>On Behalf Of
</b>Bradley Bopp<br>
<b>Sent:</b> Tuesday, May 18, 2010 15:04<br>
<b>To:</b> Joseph Jackson<br>
<b>Cc:</b> outages@outages.org<br>
<b>Subject:</b> Re: [outages] Denver Level3 problems?<o:p></o:p></span></p>
</div>
</div>
<p class=MsoNormal><o:p> </o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif"'>An internal source has informed me there
was an issue with a Data Center in the Denver area in which something “blew up”
and this caused a chain effect with anything going through Denver to any other
pop in the USA. <br>
<br>
The problem since has been resolved and was for a very short period.<br>
<br>
Brad<br>
<br>
<br>
<br>
On 5/18/10 2:32 PM, "Joseph Jackson" <<a
href="jjackson@aninetworks.net">jjackson@aninetworks.net</a>> wrote:</span><o:p></o:p></p>
<p class=MsoNormal style='margin-bottom:12.0pt'><span style='font-size:11.0pt;
font-family:"Calibri","sans-serif";color:#1F497D'>We also saw issues with
customer end points that took level 3 around the same time but in NYC – LA and
DFW POPS.<br>
<br>
Joseph<br>
<br>
<br>
</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br>
</span><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a
href="outages-bounces@outages.org">outages-bounces@outages.org</a> [<a
href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Sean Clark<br>
<b>Sent:</b> Tuesday, May 18, 2010 11:53 AM<br>
<b>To:</b> Jayson Baker; randal k<br>
<b>Cc:</b> <a href="outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Denver Level3 problems?<br>
</span><br>
<span style='font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1F497D'>We
had many customer complaints and all our IP SLA tests spiked between 1110
and 1135 est for Level 3 in our Denver market. But we also saw the
problems in DAL on our level 3 peer (SLA test as well). Some event happened in
the Denver area on level 3, we have opened a ticket with them and they reported
to us they have had a lot of customer call in for the same thing. They have not
given us an RFO yet…<br>
<br>
<br>
</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br>
</span><b><span style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'>From:</span></b><span
style='font-size:10.0pt;font-family:"Tahoma","sans-serif"'> <a
href="outages-bounces@outages.org">outages-bounces@outages.org</a> [<a
href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Jayson Baker<br>
<b>Sent:</b> Tuesday, May 18, 2010 12:31<br>
<b>To:</b> randal k<br>
<b>Cc:</b> <a href="outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Denver Level3 problems?<br>
</span><br>
We've been having issues staying connected to Google Chat for the last hour or
so.<br>
<span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br>
</span>And it just so happens we're connected through you. :-)<br>
<span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br>
</span><br>
Best Wishes,<br>
Jayson A. Baker<br>
Peak Internet<br>
<span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br>
</span>On Tue, May 18, 2010 at 9:51 AM, randal k <<a
href="rkohutek@gmail.com">rkohutek@gmail.com</a>> wrote:<br>
Did anybody else see Denver Level3 service take a hit this morning? Our logs
indicated nothing, but we saw various blackholing all over their network for
~20 minutes, starting around 9:15am MST.<br>
<br>
Everybody have a great day -<br>
<span style='color:#888888'>Randal<br>
</span><br>
_______________________________________________<br>
Outages mailing list<br>
<a href="Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a><br>
<span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><o:p></o:p></span></p>
<div class=MsoNormal align=center style='text-align:center'><span
style='font-size:11.0pt;font-family:"Calibri","sans-serif"'>
<hr size=3 width="95%" align=center>
</span></div>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'>_______________________________________________<br>
Outages mailing list<br>
<a href="Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a></span><o:p></o:p></p>
<p class=MsoNormal><span style='font-size:10.0pt;font-family:Consolas'><br>
</span><span style='font-size:11.0pt;font-family:"Calibri","sans-serif"'><br>
Bradley D. Bopp<br>
Director Of Engineering<br>
<a href="http://www.nationalnet.com">http://www.nationalnet.com</a><br>
<a href="http://as22384.peeringdb.net">http://as22384.peeringdb.net</a><br>
<br>
NationalNet is committed to the highest level of Customer Service available in
the Web Hosting business. If you have any questions, comments or concerns
feel free to contact us at 678-247-7000 ext 1 (or toll-free, 888-4-NATNET).<br>
</span><o:p></o:p></p>
</div>
</body>
</html>