<div dir="ltr"><a href="https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919">https://www.facebook.com/notes/facebook-engineering/more-details-on-todays-outage/431441338919</a><br><div><br></div><div>Guess it's already out there.</div><div><br></div><div>Scott</div><div><br></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Sep 29, 2015 at 9:36 AM, William Byrd via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hey Rafael -<div><br></div><div>I am sure it would make a very interesting post mortem post, and I will ask if anyone would be willing to put something together to share externally. Right now I can't share any details, so I'll just have to apologize (again) for the inconvenience and see if we can get something posted.</div></div><div class="gmail_extra"><br clear="all"><div><div>William Collier-Byrd<br><a href="mailto:will@collier-byrd.net" target="_blank">will@collier-byrd.net</a></div></div>
<br><div class="gmail_quote">On Tue, Sep 29, 2015 at 8:25 AM, Rafael Possamai <span dir="ltr"><<a href="mailto:rafael@e2wsolutions.com" target="_blank">rafael@e2wsolutions.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Hi Will,<div><br></div><div>Are you able to disclose any high level details of what happened? It's just interesting to know how such a large infrastructure, with assumed redundancies, can go down completely. I know AWS had a similar outage, which lasted even longer, but it was a series of cascading problems due to human error. Was it an SDN bad config push?</div><div><div class="h5"><div class="gmail_extra"><br clear="all"><div><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div style="font-size:12.8000001907349px"><br></div><div style="font-size:12.8000001907349px"><br></div></div></div></div></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Mon, Sep 28, 2015 at 2:59 PM, William Collier-Byrd via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div>I am starting to feel like a broken record, but we're aware and working on it. <br><br><div>--<br>Will Collier-Byrd</div><br></div><div><div><br><br><br>
<div class="gmail_quote">On Mon, Sep 28, 2015 at 12:56 PM -0700, "Mike Phipps via Outages" <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div dir="3D"ltr"">
<div dir="ltr">Confirmed on T-Mobile LTE and Comcast Business Class in metro Detroit. It is that same message.<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Sep 28, 2015 at 3:27 PM, Bob Colon via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div bgcolor="#FFFFFF" text="#000000">
Yup....getting the same here, <br>
<div>
<div>
<br>
<p> <span> <span>To
ensure prompt service, please use "Reply All" when
responding </span> <br>
<br>
<span>Robert Colon | </span> <span>TCSA Tier 3 | </span> <span>EthoStream
LLC | </span> <span>P: <a href="tel:%28877%29%20282-2519%20x2114" value="+18772822519" target="_blank">(877) 282-2519 x2114</a>
| </span> <span>F: <a href="tel:%28414%29%20258-8307" value="+14142588307" target="_blank">(414) 258-8307</a> | </span>
<span>20800 Swenson Drive, Suite 175 Waukesha,
WI 53186 | </span> <span><a href="http://www.telkonet.com/" target="_blank"></a><a href="http://www.telkonet.com" target="_blank">www.telkonet.com</a> | </span>
<span><a href="http://www.ethostream.com/" target="_blank">www.ethostream.com</a>
| </span> <span><a href="https://twitter.com/telkonet" target="_blank">@Telkonet</a> </span>
</span> </p>
<br>
</div>
</div><span>
<div>On 9/28/2015 2:23 PM, James Hartig via
Outages wrote:<br>
</div>
</span><span><blockquote type="cite">
<div dir="ltr">Anyone else getting:
<div><br>
<div>
<div>Sorry, something went wrong.</div>
<div>We're working on it and we'll get it fixed as soon as
we can.</div>
<div><br>
</div>
<div>when you go to Facebook?</div>
<div>--</div>
<div>James Hartig</div>
</div>
</div>
</div>
<br>
<fieldset></fieldset>
<br>
<pre>_______________________________________________
Outages mailing list
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a>
</pre>
</blockquote>
<br>
</span></div>
<br>
<hr><p style="font-size:1.3em;font-family:Arial,Helvetica,sans-serif"><a href="http://www.telkonet.com" target="_blank"><img src="http://www.telkonet.com/images/signature/telkonetlogo.gif"><img src="http://www.telkonet.com/images/signature/telkosig.gif" style="font-size:1.3em"></a><a href="http://www.ethostream.com" target="_blank"><img src="http://www.ethostream.com/images/signature/ethosig.gif" style="font-size:1.3em"></a></p>
<p><font face="Arial" color="#808080" size="1">This e-mail and any files transmitted with it are private and confidential and are solely for the use of the addressee. It may contain material which is legally privileged. If you are not the addressee or the person responsible for delivering to the addressee, be advised that you have received this e-mail in error and that any use of it is strictly prohibited. If you have received this e-mail in error please contact us at <a href="tel:%28414%29%20223-0473" value="+14142230473" target="_blank">(414) 223-0473</a>.</font></p><br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div>
</div>
</blockquote>
</div>
</div></div></div><br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div></div></div></div>
</blockquote></div><br></div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div>