<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
</head>
<body dir="auto">
<div>Thats from September 2010, a bit old i say.<br>
<br>
</div>
<div><br>
<div><br>
<br>
Erik Sundberg
<div>Sr. Network Engineer</div>
<div>Nitel</div>
<div>1101 West Lake Street,6th Fl</div>
<div>Chicago, IL 60607</div>
<div>Desk: 773-661-5532</div>
<div>Cell: 708-710-7419</div>
<div>NOC 24/7: 866-892-0915</div>
<div>Email: <a href="mailto:esundberg@nitelusa.com">esundberg@nitelusa.com</a></div>
<div><a href="http://www.nitelusa.com">http://www.nitelusa.com</a></div>
</div>
On Sep 29, 2015, at 12:50 PM, Scott Voll via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br>
<br>
</div>
<blockquote type="cite">
<div>
<div dir="ltr"><a href="https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919">https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919</a><br>
<div><br>
</div>
<div>Guess it's already out there.</div>
<div><br>
</div>
<div>Scott</div>
<div><br>
</div>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages
<span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hey Rafael -
<div><br>
</div>
<div>I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and
see if we can get something posted.</div>
</div>
<div class="gmail_extra"><br clear="all">
<div>
<div>William Collier-Byrd<br>
<a href="mailto:will@collier-byrd.net" target="_blank">will@collier-byrd.net</a></div>
</div>
<br>
<div class="gmail_quote">On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <span dir="ltr">
<<a href="mailto:rafael@e2wsolutions.com" target="_blank">rafael@e2wsolutions.com</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="ltr">Hi Will,
<div><br>
</div>
<div>Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a
series of cascading problems due to human error. Was it an SDN bad config push?</div>
<div>
<div class="h5">
<div class="gmail_extra"><br clear="all">
<div>
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div>
<div dir="ltr">
<div style="font-size:12.8000001907349px"><br>
</div>
<div style="font-size:12.8000001907349px"><br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
<br>
<div class="gmail_quote">On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages
<span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>I am starting to feel like a broken record, but we're aware and working on it. <br>
<br>
<div>--<br>
Will Collier-Byrd</div>
<br>
</div>
<div>
<div><br>
<br>
<br>
<div class="gmail_quote">On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages"
<span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="3D"ltr"">
<div dir="ltr">Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit. It is that same message.<br>
</div>
<div class="gmail_extra"><br>
<div class="gmail_quote">On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages <span dir="ltr">
<<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">Yup....getting the same here, <br>
<div>
<div><br>
<p><span><span>To ensure prompt service, please use "Reply All" when responding </span>
<br>
<br>
<span>Robert Colon | </span><span>TCSA Tier 3 | </span><span>EthoStream LLC | </span>
<span>P: <a href="tel:%28877%29%20282-2519%20x2114" value="+18772822519" target="_blank">
(877) 282-2519 x2114</a> | </span><span>F: <a href="tel:%28414%29%20258-8307" value="+14142588307" target="_blank">
(414) 258-8307</a> | </span><span>20800 Swenson Drive, Suite 175 Waukesha, WI 53186 |
</span><span><a href="http://www.telkonet.com/" target="_blank"></a><a href="http://www.telkonet.com" target="_blank">www.telkonet.com</a> |
</span><span><a href="http://www.ethostream.com/" target="_blank">www.ethostream.com</a> |
</span><span><a href="https://twitter.com/telkonet" target="_blank">@Telkonet</a>
</span></span></p>
<br>
</div>
</div>
<span>
<div>On 9/28/2015 2:23 PM, James Hartig via Outages wrote:<br>
</div>
</span><span>
<blockquote type="cite">
<div dir="ltr">Anyone else getting:
<div><br>
<div>
<div>Sorry, something went wrong.</div>
<div>We're working on it and we'll get it fixed as soon as we can.</div>
<div><br>
</div>
<div>when you go to Facebook?</div>
<div>--</div>
<div>James Hartig</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset> <br>
<pre>_______________________________________________
Outages mailing list
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a>
</pre>
</blockquote>
<br>
</span></div>
<br>
<hr>
<p style="font-size:1.3em;font-family:Arial,Helvetica,sans-serif"><a href="http://www.telkonet.com" target="_blank"><img src="http://www.telkonet.com/images/signature/telkonetlogo.gif"><img src="http://www.telkonet.com/images/signature/telkosig.gif" style="font-size:1.3em"></a><a href="http://www.ethostream.com" target="_blank"><img src="http://www.ethostream.com/images/signature/ethosig.gif" style="font-size:1.3em"></a></p>
<p><font face="Arial" color="#808080" size="1">This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person
responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at
<a href="tel:%28414%29%20223-0473" value="+14142230473" target="_blank">(414) 223-0473</a>.</font></p>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
</div>
</div>
</div>
</div>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</div>
</blockquote>
</div>
<br>
</div>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
</blockquote>
</div>
<br>
</div>
</div>
</blockquote>
<blockquote type="cite">
<div><span>_______________________________________________</span><br>
<span>Outages mailing list</span><br>
<span><a href="mailto:Outages@outages.org">Outages@outages.org</a></span><br>
<span><a href="https://puck.nether.net/mailman/listinfo/outages">https://puck.nether.net/mailman/listinfo/outages</a></span><br>
</div>
</blockquote>
<br>
<hr>
<font face="Arial" color="Gray" size="1"><br>
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering
it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify
the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.<br>
</font>
</body>
</html>