[Outages-discussion] On the subject of root cause analyses and postmortems...

[Chapman, Brad (NBCUniversal)]

Howdy Jay,

In light of today’s Facebook outage, I thought I’d inquire as to how your new book about complex failure analysis is coming along.

Below is the cover art I proposed at the start of the pandemic, with some new owls as requested.  :-)

Cheers,
Brad

[A group of people in garment  Description automatically generated with low confidence][A picture containing text, bird of prey, bird, owl  Description automatically generated]



From: Outages-discussion <outages-discussion-bounces at outages.org> on behalf of "Chapman, Brad (NBCUniversal)" <Brad.Chapman at nbcuni.com>
Date: Sunday, March 22, 2020 at 8:52 PM
To: Jay Ashworth <jra at baylink.com>, Gary Josack <gary at byoteki.com>, "Patrick W. Gilmore" <patrick at ianai.net>
Cc: OUTAGES Discussion List <outages-discussion at outages.org>
Subject: Re: [Outages-discussion] [EXTERNAL] Re: [outages] Discord: Sever outages and Increased API errors

Hmm, you have a point.  Maybe the three wise guys are a more appropriate animal metaphor...




From: Jay Ashworth <jra at baylink.com>
Date: Sunday, March 22, 2020 at 8:52 AM
To: "Chapman, Brad (NBCUniversal)" <Brad.Chapman at nbcuni.com>, Gary Josack <gary at byoteki.com>, Gary Josack via Outages <outages at outages.org>, "Patrick W. Gilmore" <patrick at ianai.net>
Subject: Re: [EXTERNAL] Re: [outages] Discord: Sever outages and Increased API errors

Yeah, but that's an owl, with a 360-degree neck. Shouldn't it have been facing away from the camera? :-)
On March 22, 2020 2:03:04 AM EDT, "Chapman, Brad (NBCUniversal)" <Brad.Chapman at nbcuni.com> wrote:
Whenever you decide to publish such a book, I’ve got your title and cover art ready...

Put me down for a pre-order.

-Brad

[A close up of a bird  Description automatically generated]



From: Outages <outages-bounces at outages.org> on behalf of Jay Ashworth via Outages <outages at outages.org>
Reply-To: Jay Ashworth <jra at baylink.com>
Date: Friday, March 20, 2020 at 9:07 PM
To: Gary Josack <gary at byoteki.com>, Gary Josack via Outages <outages at outages.org>, "Patrick W. Gilmore" <patrick at ianai.net>
Cc: outages <outages at outages.org>
Subject: [EXTERNAL] Re: [outages] Discord: Sever outages and Increased API errors

I am sufficiently interested that I'm seriously considering writing a book on this topic actually. That was a very nice write-up. Do I correctly understand, then, that the root cause was the broken golang library function / routine?

(For the moment, we will pass on the topic of the irony of having to come up with an alternate name for your own language because otherwise people won't be able to find things about it... in the search engine you run that you wrote it for :-) Way to go Google.)
On March 20, 2020 11:54:08 PM EDT, Gary Josack via Outages <outages at outages.org> wrote:
If anyone is interested we published a post mortem on our status page for this incident. https://status.discordapp.com/incidents/62gt9cgjwdgf<https://urldefense.com/v3/__https:/status.discordapp.com/incidents/62gt9cgjwdgf__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIv4M9w-XQ$>

On Mon, Mar 16, 2020 at 2:49 PM Patrick W. Gilmore via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Discord seems to be having issues. It is showing on Down Detector. The splash page says:
[https://files.slack.com/files-pri/T2F7AJRMY-FV5C2QE4A/image.png]
[cid:170fb3917e5a2da56b91]

The Server Status page shows:
[cid:170fb3917e5774fd4222]

Mostly FYI.

--
TTFN,
patrick

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!9IlozgcLDbWntVNJYRLbLSjlkY1RnEa2iRitGoMKZBTwtilpUibKlSbLxIs82I34TA$>

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211005/ca460083/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 61320 bytes
Desc: image001.png
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211005/ca460083/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 81589 bytes
Desc: image002.png
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211005/ca460083/attachment-0003.png>