[outages] After-Action Report: Facebook/Meta outage

[Aaron Hope]

We are not renewing Splunk.  We have migrated to Azure Sentinel for logging and monitoring.

Aaron
________________________________
From: Outages <outages-bounces at outages.org> on behalf of Jay Ashworth via Outages <outages at outages.org>
Sent: Tuesday, March 5, 2024 2:59 PM
To: Ross Tajvar <ross at tajvar.io>
Cc: outages at outages.org <outages at outages.org>
Subject: Re: [outages] After-Action Report: Facebook/Meta outage

Yeah, but I shouldn't have gotten bitten. I'm supposed to be holding myself to a higher standard, as a moderator of this particular group.


On March 5, 2024 2:58:00 PM EST, Ross Tajvar <ross at tajvar.io> wrote:
Indeed - this happens every time facebook has a major outage, because they stopped putting out posts explaining what happened and because people tend not to look too closely before spreading links that look plausible.

On Tue, Mar 5, 2024 at 2:56 PM Jay R. Ashworth via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Well, when I pulled up the link from Ross's mobile version, it had comments
under it from 10/22; if those people didn't know where to go either... :-}

----- Original Message -----
> From: "Damian Menscher" <damian at google.com<mailto:damian at google.com>>
> To: "jra" <jra at baylink.com<mailto:jra at baylink.com>>
> Cc: outages at outages.org<mailto:outages at outages.org>
> Sent: Tuesday, March 5, 2024 2:16:33 PM
> Subject: Re: [outages] After-Action Report: Facebook/Meta outage

> It clearly *is* from 2010:
> https://engineering.fb.com/2010/09/23/uncategorized/more-details-on-today-s-o
>
> Damian
>
> On Tue, Mar 5, 2024 at 11:13 AM Jay R. Ashworth via Outages <
> outages at outages.org<mailto:outages at outages.org>> wrote:
>
>> Well, it seems equivocal whether that's about *this* outage
>> rather than an earlier one -- though it's clearly not from 2010.
>>
>> More to follow.
>>
>> Cheers,
>> -- jra
>>
>> ----- Original Message -----
>> > From: "Jay R. Ashworth via Outages" <outages at outages.org<mailto:outages at outages.org>>
>> > To: outages at outages.org<mailto:outages at outages.org>
>> > Sent: Tuesday, March 5, 2024 1:57:24 PM
>> > Subject: [outages] After-Action Report: Facebook/Meta outage
>>
>> > [ I can't set the header from Zimbra, but please, Reply-To: -discuss ]
>> >
>> > Here's a link to FB's AAR about this morning's 70-minuteish outage.
>> >
>> > It's been pointed out to me that this affected other services as well, if
>> > they are an OAuth client, and you chose to use FB's OAuth service to
>> create
>> > your account there.
>> >
>> >
>> https://m.facebook.com/nt/screen/?params=%7B%22note_id%22%3A10158791436142200%7D&path=%2Fnotes%2Fnote%2F
>> >
>> > [ Hat-tip: Jorge Amodio ]
>> >
>> > Cheers,
>> > -- jra
>> >
>> > --
>> > Jay R. Ashworth                  Baylink
>> jra at baylink.com<mailto:jra at baylink.com>
>> > Designer                     The Things I Think
>>  RFC 2100
>> > Ashworth & Associates       http://www.bcp38.info          2000 Land
>> Rover DII
>> > St Petersburg FL USA      BCP38: Ask For It By Name!           +1 727
>> 647 1274 <(727)%20647-1274>
>> > _______________________________________________
>> > Outages mailing list
>> > Outages at outages.org<mailto:Outages at outages.org>
>> > https://puck.nether.net/mailman/listinfo/outages
>>
>> --
>> Jay R. Ashworth                  Baylink
>> jra at baylink.com<mailto:jra at baylink.com>
>> Designer                     The Things I Think                       RFC
>> 2100
>> Ashworth & Associates       http://www.bcp38.info          2000 Land
>> Rover DII
>> St Petersburg FL USA      BCP38: Ask For It By Name!           +1 727 647
>> 1274 <(727)%20647-1274>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org<mailto:Outages at outages.org>
>> https://puck.nether.net/mailman/listinfo/outages

--
Jay R. Ashworth                  Baylink                       jra at baylink.com<mailto:jra at baylink.com>
Designer                     The Things I Think                       RFC 2100
Ashworth & Associates       http://www.bcp38.info          2000 Land Rover DII
St Petersburg FL USA      BCP38: Ask For It By Name!           +1 727 647 1274
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
WARNING CONCERNING CONFIDENTIALITY This email message, including any attachments, is for the sole use by its intended recipient or recipients and may contain confidential and privileged information. Any unauthorized review, use, disclosure, or distribution is prohibited. If you are not the intended recipient of this message, please immediately notify the sender and delete the original message without making a copy.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20240305/08e5c223/attachment.htm>