[Outages-discussion] [EXTERNAL] Re: [outages] Facebook Down

[Chapman, Brad (NBCUniversal)]

Here's a partial comment chain from the reddit /r/sysadmin forum, started by a user who claims to be a part of the "recovery and investigation team" at Facebook.  

Also... in the last few seconds this person just deleted their comments so he probably doesn't want this shared publicly.  Keep it in the mailing list, please.

=============

(fb person)

Update 1640 UTC:

As many of you know, DNS for FB services has been affected and this is likely a symptom of the actual issue, and that's that BGP peering with Facebook peering routers has gone down, very likely due to a configuration change that went into effect shortly before the outages happened (started roughly 1540 UTC).

There are people now trying to gain access to the peering routers to implement fixes, but the people with physical access is separate from the people with knowledge of how to actually authenticate to the systems and people who know what to actually do, so there is now a logistical challenge with getting all that knowledge unified.

Part of this is also due to lower staffing in data centers due to pandemic measures.

> (reply) "That implies access was lost that wasn't planned... was this malicious?"

>> (fb person) No discussion that I'm aware of yet that is considering a threat/attack vector.  I believe the original change was 'automatic' (as in configuration done via a web interface). However, now that connection to the outside world is down, remote access to those tools don't exist anymore, so the emergency procedure is to gain physical access to the peering routers and do all the configuration locally.

>>> (reply) "Even in the biggest of organizations, they still have to wait for somebody to race down to the datacenter and plug his laptop into a router."

=============


On 10/4/21, 10:36 AM, "Outages on behalf of Jay Weekley via Outages" <outages-bounces at outages.org on behalf of outages at outages.org> wrote:

    Same in North Alabama.  The one time in my history of Facebook that I 
    need to get in touch with someone urgently.

    Matt Hoppes via Outages wrote:
    > Is anyone else seeing Facebook down?   Seems to be offline here in 
    > Pennsylvania.. started with error messages now DNS errors.
    >
    > Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
    > _______________________________________________
    > Outages mailing list
    > Outages at outages.org
    > https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2qvc9T3xA$ 
    >

    -- 
    *Jay Weekley*
    *Cyber Broadband
    *

    -- 
    This email has been checked for viruses by AVG.
    https://urldefense.com/v3/__https://www.avg.com__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2rqgIkaYA$ 

    _______________________________________________
    Outages mailing list
    Outages at outages.org
    https://urldefense.com/v3/__https://puck.nether.net/mailman/listinfo/outages__;!!PIZeeW5wscynRQ!-ez0CoXRkIas2K_7V9zsmtyhcS3qpE53x25S5lj-IHR4cg3IVSY5KGcHp2qvc9T3xA$