[Outages-discussion] [EXTERNAL] Re: [outages] Facebook Down

[Jeffrey Ollie]

https://www.reddit.com/r/sysadmin/comments/q181fv/looks_like_facebook_is_down/

I think keeping the comments secret is no longer possible.

On Mon, Oct 4, 2021 at 1:17 PM Chapman, Brad (NBCUniversal) <
Brad.Chapman at nbcuni.com> wrote:

> Here's a partial comment chain from the reddit /r/sysadmin forum, started
> by a user who claims to be a part of the "recovery and investigation team"
> at Facebook.
>
> Also... in the last few seconds this person just deleted their comments so
> he probably doesn't want this shared publicly.  Keep it in the mailing
> list, please.
>
> =============
>
> (fb person)
>
> Update 1640 UTC:
>
> As many of you know, DNS for FB services has been affected and this is
> likely a symptom of the actual issue, and that's that BGP peering with
> Facebook peering routers has gone down, very likely due to a configuration
> change that went into effect shortly before the outages happened (started
> roughly 1540 UTC).
>
> There are people now trying to gain access to the peering routers to
> implement fixes, but the people with physical access is separate from the
> people with knowledge of how to actually authenticate to the systems and
> people who know what to actually do, so there is now a logistical challenge
> with getting all that knowledge unified.
>
> Part of this is also due to lower staffing in data centers due to pandemic
> measures.
>
> > (reply) "That implies access was lost that wasn't planned... was this
> malicious?"
>
> >> (fb person) No discussion that I'm aware of yet that is considering a
> threat/attack vector.  I believe the original change was 'automatic' (as in
> configuration done via a web interface). However, now that connection to
> the outside world is down, remote access to those tools don't exist
> anymore, so the emergency procedure is to gain physical access to the
> peering routers and do all the configuration locally.
>
> >>> (reply) "Even in the biggest of organizations, they still have to wait
> for somebody to race down to the datacenter and plug his laptop into a
> router."
>
> =============
>
>
> On 10/4/21, 10:36 AM, "Outages on behalf of Jay Weekley via Outages" <
> outages-bounces at outages.org on behalf of outages at outages.org> wrote:
>
>     Same in North Alabama.  The one time in my history of Facebook that I
>     need to get in touch with someone urgently.
>
>     Matt Hoppes via Outages wrote:
>     > Is anyone else seeing Facebook down?   Seems to be offline here in
>     > Pennsylvania.. started with error messages now DNS errors.
>     >
>     > Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
>     > _______________________________________________
>     > Outages mailing list
>     > Outages at outages.org
>     >
> https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2qvc9T3xA$
>     >
>
>     --
>     *Jay Weekley*
>     *Cyber Broadband
>     *
>
>     --
>     This email has been checked for viruses by AVG.
>
> https://urldefense.com/v3/__https://www.avg.com__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2rqgIkaYA$
>
>     _______________________________________________
>     Outages mailing list
>     Outages at outages.org
>
> https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2qvc9T3xA$
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>


-- 
Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211004/3e13babc/attachment.htm>