[Outages-discussion] S3 Outages Postmortem

[Chapman, Brad (NBCUniversal)]

“…lots of services affected…”

Well, that was pretty obvious from the dashboard yesterday:

https://i.imgur.com/xTec0Bn.png

-Brad

From: Outages-discussion [mailto:outages-discussion-bounces at outages.org] On Behalf Of Kevin Blackham
Sent: Wednesday, March 1, 2017 11:17 AM
To: Bob Strecansky <bob at mailchimp.com>
Cc: outages-discussion at outages.org
Subject: Re: [Outages-discussion] S3 Outages Postmortem

I have some insights, but I'm under NDA. This was big enough I expect some public disclosure (my words).

I can tell you we observed lots of services affected, not just S3. EBS was jacking up IO all over the place, and many machines didn't even ping. SES was quite broken, as was autoscaling. One might conclude it was a network problem.

On Mar 1, 2017 12:09, "Bob Strecansky" <bob at mailchimp.com<mailto:bob at mailchimp.com>> wrote:
Has anyone heard anything about why S3 was down for 5 hours yesterday?  Usually Amazon doesn't post postmortems, and i'm curious as to what happened.

Thanks,

Bob Strecansky
--
Thanks,

-B

_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org<mailto:Outages-discussion at outages.org>
https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20170301/51b349c2/attachment.html>