<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>This had wide impact, including MPLS customers and Internet service across the globe.</div><div><br></div><div>Our alerts started around 03:00 ET and the last issue cleared shortly before 09:00 AM ET today. AT&T stated a 3 hour outage window with no official rfo yet.</div>
<div><br></div><div>The numbers I saw stated over 400 routers were rebooted to recover.<br><br><div>- Chris</div>Sent from my mobile device</div><div><br>On Jan 13, 2014, at 11:21 AM, Evan Rhoades <<a href="mailto:evan@exigent.com">evan@exigent.com</a>> wrote:<br>
<br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Tahoma;
panose-1:2 11 6 4 3 5 4 4 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman","serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.MsoAcetate, li.MsoAcetate, div.MsoAcetate
{mso-style-priority:99;
mso-style-link:"Balloon Text Char";
margin:0in;
margin-bottom:.0001pt;
font-size:8.0pt;
font-family:"Tahoma","sans-serif";}
span.hoenzb
{mso-style-name:hoenzb;}
span.EmailStyle18
{mso-style-type:personal;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri","sans-serif";
color:#1F497D;}
span.BalloonTextChar
{mso-style-name:"Balloon Text Char";
mso-style-priority:99;
mso-style-link:"Balloon Text";
font-family:"Tahoma","sans-serif";}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">Earlier today we had a site down in the 12.250.x.x block, it came back up around 8am PST.
</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<div>
<div style="border:none;border-top:solid #b5c4df 1.0pt;padding:3.0pt 0in 0in 0in">
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Eric Henson<br>
<b>Sent:</b> Monday, January 13, 2014 7:53 AM<br>
<b>To:</b> Damian Guppy; staticsafe<br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Reports of AT&T outage(s)</span></p>
</div>
</div>
<p class="MsoNormal"> </p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">According to that reddit thread, ATT U-Verse can’t connect to anything that uses AWS, such as Netflix.</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">One post says this:
</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d">“Work in a NOC; from what I understand ATT had a code bug that caused memory problems on their edge routers. Had to reboot about 270 of them. Its supposed to
be resolved now”</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal" style="line-height:150%"><span style="font-size:11.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c">--
<b><br>
</b></span><span style="font-size:11.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#6f8cc0">ERIC HENSON</span><span style="font-family:"Arial","sans-serif";color:#78a22f"><br>
</span><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c">Solutions Architect for Systems Organization</span></p>
<p class="MsoNormal" style="line-height:150%"><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c">PFSweb |
</span><span style="font-size:11.0pt;line-height:150%;font-family:"Calibri","sans-serif";color:#1f497d"><a href="http://www.pfsweb.com/"><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c">www.pfsweb.com</span></a></span><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c"><br>
</span><b><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#6f8cc0">p:</span></b><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#6f8cc0">
</span><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c">972.881.2900 x3104<br>
</span><b><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#6f8cc0">m:</span></b><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#6f8cc0">
</span><span style="font-size:8.0pt;line-height:150%;font-family:"Arial","sans-serif";color:#59595c">972.948.3424</span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri","sans-serif";color:#1f497d"> </span></p>
<p class="MsoNormal"><b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif"">From:</span></b><span style="font-size:10.0pt;font-family:"Tahoma","sans-serif""> Outages [<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>]
<b>On Behalf Of </b>Damian Guppy<br>
<b>Sent:</b> Monday, January 13, 2014 9:44 AM<br>
<b>To:</b> staticsafe<br>
<b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br>
<b>Subject:</b> Re: [outages] Reports of AT&T outage(s)</span></p>
<p class="MsoNormal"> </p>
<div>
<p class="MsoNormal">I'm noticing a few sites down, possibly due to DDoS?</p>
<div>
<p class="MsoNormal"> </p>
</div>
<div>
<p class="MsoNormal">--Damian</p>
</div>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12.0pt"> </p>
<div>
<p class="MsoNormal">On Mon, Jan 13, 2014 at 11:27 PM, staticsafe <<a href="mailto:me@staticsafe.ca" target="_blank">me@staticsafe.ca</a>> wrote:</p>
<p class="MsoNormal"><a href="http://www.reddit.com/r/sysadmin/comments/1v3fs1/att_outage/" target="_blank">http://www.reddit.com/r/sysadmin/comments/1v3fs1/att_outage/</a><br>
<a href="http://downdetector.com/status/att/map" target="_blank">http://downdetector.com/status/att/map</a><br>
<span class="hoenzb"><span style="color:#888888">--</span></span><span style="color:#888888"><br>
<span class="hoenzb">staticsafe</span><br>
<br>
<span class="hoenzb">_______________________________________________</span><br>
<span class="hoenzb">Outages mailing list</span><br>
<span class="hoenzb"><a href="mailto:Outages@outages.org">Outages@outages.org</a></span><br>
<span class="hoenzb"><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a></span></span></p>
</div>
<p class="MsoNormal"> </p>
</div>
<p class="MsoNormal"><br>
------------------------<br>
This email was scanned by BitDefender.</p>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Outages mailing list</span><br><span><a href="mailto:Outages@outages.org">Outages@outages.org</a></span><br>
<span><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a></span><br></div></blockquote></body></html>