<html><head><meta http-equiv="content-type" content="text/html; charset=utf-8"></head><body dir="auto"><div>The outage was abrupt, but the recovery came in stages. Read traffic first, followed by write traffic ~1.5 hours later. That makes me think a power problem, or automation gone awry. We always blame the network team, but that rings hollow to me here.</div><div id="AppleMailSignature"><br></div><div id="AppleMailSignature">On strategy, I am fully behind prioritization of read traffic recovery over write traffic. That's evolving over time, but is still true for most use cases.</div><div id="AppleMailSignature"><br></div><div id="AppleMailSignature">For those saying "who cares," you may not understand the number of blended integrated systems out there in this age. This took down a huge number of correlated services, and it shouldn't have. We need looser coupling.</div><div id="AppleMailSignature"><br></div><div id="AppleMailSignature">- Mike Christian</div><div id="AppleMailSignature"><br><br>Sent from my iPad</div><div><br>On Mar 1, 2017, at 11:25 AM, Chapman, Brad (NBCUniversal) <<a href="mailto:Brad.Chapman@nbcuni.com">Brad.Chapman@nbcuni.com</a>> wrote:<br><br></div><blockquote type="cite"><div>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:12.0pt;
font-family:"Times New Roman",serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:#1F497D;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
<div class="WordSection1">
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">“…lots of services affected…”<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">Well, that was pretty obvious from the dashboard yesterday:
<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><a href="https://i.imgur.com/xTec0Bn.png">https://i.imgur.com/xTec0Bn.png</a><o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D">-Brad<o:p></o:p></span></p>
<p class="MsoNormal"><span style="font-size:11.0pt;font-family:"Calibri",sans-serif;color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif">From:</span></b><span style="font-size:11.0pt;font-family:"Calibri",sans-serif"> Outages-discussion [<a href="mailto:outages-discussion-bounces@outages.org">mailto:outages-discussion-bounces@outages.org</a>]
<b>On Behalf Of </b>Kevin Blackham<br>
<b>Sent:</b> Wednesday, March 1, 2017 11:17 AM<br>
<b>To:</b> Bob Strecansky <<a href="mailto:bob@mailchimp.com">bob@mailchimp.com</a>><br>
<b>Cc:</b> <a href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a><br>
<b>Subject:</b> Re: [Outages-discussion] S3 Outages Postmortem<o:p></o:p></span></p>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">I have some insights, but I'm under NDA. This was big enough I expect some public disclosure (my words).<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">I can tell you we observed lots of services affected, not just S3. EBS was jacking up IO all over the place, and many machines didn't even ping. SES was quite broken, as was autoscaling. One might conclude it was a network problem.<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
<div>
<p class="MsoNormal">On Mar 1, 2017 12:09, "Bob Strecansky" <<a href="mailto:bob@mailchimp.com">bob@mailchimp.com</a>> wrote:<o:p></o:p></p>
<blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-right:0in">
<div>
<p class="MsoNormal">Has anyone heard anything about why S3 was down for 5 hours yesterday? Usually Amazon doesn't post postmortems, and i'm curious as to what happened.<o:p></o:p></p>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Thanks,<o:p></o:p></p>
</div>
<div>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
<div>
<p class="MsoNormal">Bob Strecansky<o:p></o:p></p>
</div>
</div>
<div>
<p class="MsoNormal">-- <o:p></o:p></p>
</div>
<div>
<div>
<p class="MsoNormal">Thanks,<br>
<br>
-B<o:p></o:p></p>
</div>
</div>
<p class="MsoNormal" style="margin-bottom:12.0pt"><br>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org">Outages-discussion@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages-discussion" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><o:p></o:p></p>
</blockquote>
</div>
</div>
</div>
</div></blockquote><blockquote type="cite"><div><span>_______________________________________________</span><br><span>Outages-discussion mailing list</span><br><span><a href="mailto:Outages-discussion@outages.org">Outages-discussion@outages.org</a></span><br><span><a href="https://puck.nether.net/mailman/listinfo/outages-discussion">https://puck.nether.net/mailman/listinfo/outages-discussion</a></span><br></div></blockquote></body></html>