[Outages-discussion] [outages] Facebook Down

[Chad Austin]

Someone must have gone a long time before they finally learned the 
importance of the reload-in command.

-Chad

On 10/4/2021 1:28 PM, Matt Hoppes wrote:
> That's bad... if I read that properly.. they've locked themselves out 
> of their routers and are unable to get in without physically touching 
> each one.
>
> On 10/4/21 1:00 PM, Eric Henson wrote:
>> Someone on the Facebook Incident response team is posting updates to 
>> Reddit.
>>
>> https://www.reddit.com/user/ramenporn/comments/
>>
>> “As many of you know, DNS for FB services has been affected and this 
>> is likely a symptom of the actual issue, and that's that *BGP peering 
>> with Facebook peering routers has gone down*, very likely due to a 
>> configuration change that went into effect shortly before the outages 
>> happened (started roughly 1540 UTC).
>>
>> There are people now trying to gain access to the peering routers to 
>> implement fixes, but the people with physical access is separate from 
>> the people with knowledge of how to actually authenticate to the 
>> systems and people who know what to actually do, so there is now a 
>> logistical challenge with getting all that knowledge unified.
>>
>> Part of this is also due to lower staffing in data centers due to 
>> pandemic measures.”
>>
>> --------
>>
>> *Eric Henson*
>>
>> Senior Manager, IT Operations
>>
>> PFSweb, Inc.
>>
>> *o*:  972.685.1387
>>
>> *m:*972.948.3424
>>
>> www.pfsweb.com
>>
>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Link IP 
>> via Outages
>> *Sent:* Monday, October 4, 2021 10:51 AM
>> *To:* Matt Hoppes <mattlists at rivervalleyinternet.net>
>> *Cc:* outages at outages.org
>> *Subject:* Re: [outages] Facebook Down
>>
>> *CAUTION:*This email originated from outside of the organization. Do 
>> not click links or open attachments unless you recognize the sender 
>> and know the content is safe.
>>
>> Down here in  the UK as well
>>
>> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages 
>> <outages at outages.org <mailto:outages at outages.org>> wrote:
>>
>>     Is anyone else seeing Facebook down?   Seems to be offline here in
>>     Pennsylvania.. started with error messages now DNS errors.
>>
>>     Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
>>     _______________________________________________
>>     Outages mailing list
>>     Outages at outages.org <mailto:Outages at outages.org>
>>     https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> -- 
>>
>> *Lance Davis*
>>
>> *Link IP Networks Limited*
>> Delta House,
>> Wavell Road,
>> Wythenshawe,
>> M22 5QZ
>>
>> Office Tel: 0161 813 1870
>>
>> Mobile Tel: 07703 196010
>>
>> Email: lance at linkipnetworks.co.uk <mailto:lance at linkipnetworks.co.uk>
>>
>> Link IP Networks Limited is a company registered in England and Wales 
>> with registration number 11881958 and registered address Delta House, 
>> Wavell Road, Wythenshawe, M22 5QZ.
>>
>> This e-mail and any files transmitted with it are confidential and/or 
>> legally privileged and intended solely for the use of the individual 
>> or entity to whom they are addressed. If you are not the addressee 
>> indicated in this e-mail (or responsible for the delivery of the e 
>> mail to such person) be advised that you have received this e-mail in 
>> error, and that any use, dissemination, forwarding, printing, or 
>> copying of this e-mail is strictly prohibited. If you have received 
>> this e-mail in error please notify the sender by reply e-mail and 
>> then immediately permanently delete it and all copies from your 
>> system. Whilst we maintain virus checks, we accept no liability for 
>> viruses introduced from this e-mail.
>>
>> üPlease consider the *environment* before printing this e-mail
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion