[Outages-discussion] [outages] Facebook Down

[Matt Hoppes]

That's bad... if I read that properly.. they've locked themselves out of 
their routers and are unable to get in without physically touching each one.

On 10/4/21 1:00 PM, Eric Henson wrote:
> Someone on the Facebook Incident response team is posting updates to Reddit.
> 
> https://www.reddit.com/user/ramenporn/comments/
> 
> “As many of you know, DNS for FB services has been affected and this is 
> likely a symptom of the actual issue, and that's that *BGP peering with 
> Facebook peering routers has gone down*, very likely due to a 
> configuration change that went into effect shortly before the outages 
> happened (started roughly 1540 UTC).
> 
> There are people now trying to gain access to the peering routers to 
> implement fixes, but the people with physical access is separate from 
> the people with knowledge of how to actually authenticate to the systems 
> and people who know what to actually do, so there is now a logistical 
> challenge with getting all that knowledge unified.
> 
> Part of this is also due to lower staffing in data centers due to 
> pandemic measures.”
> 
> --------
> 
> *Eric Henson*
> 
> Senior Manager, IT Operations
> 
> PFSweb, Inc.
> 
> *o*:  972.685.1387
> 
> *m:*972.948.3424
> 
> www.pfsweb.com
> 
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Link IP via 
> Outages
> *Sent:* Monday, October 4, 2021 10:51 AM
> *To:* Matt Hoppes <mattlists at rivervalleyinternet.net>
> *Cc:* outages at outages.org
> *Subject:* Re: [outages] Facebook Down
> 
> *CAUTION:*This email originated from outside of the organization. Do not 
> click links or open attachments unless you recognize the sender and know 
> the content is safe.
> 
> Down here in  the UK as well
> 
> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages 
> <outages at outages.org <mailto:outages at outages.org>> wrote:
> 
>     Is anyone else seeing Facebook down?   Seems to be offline here in
>     Pennsylvania.. started with error messages now DNS errors.
> 
>     Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
>     _______________________________________________
>     Outages mailing list
>     Outages at outages.org <mailto:Outages at outages.org>
>     https://puck.nether.net/mailman/listinfo/outages
> 
> 
> -- 
> 
> *Lance Davis*
> 
> *Link IP Networks Limited*
> Delta House,
> Wavell Road,
> Wythenshawe,
> M22 5QZ
> 
> Office Tel: 0161 813 1870
> 
> Mobile Tel: 07703 196010
> 
> Email: lance at linkipnetworks.co.uk <mailto:lance at linkipnetworks.co.uk>
> 
> Link IP Networks Limited is a company registered in England and Wales 
> with registration number 11881958 and registered address Delta House, 
> Wavell Road, Wythenshawe, M22 5QZ.
> 
> This e-mail and any files transmitted with it are confidential and/or 
> legally privileged and intended solely for the use of the individual or 
> entity to whom they are addressed. If you are not the addressee 
> indicated in this e-mail (or responsible for the delivery of the e mail 
> to such person) be advised that you have received this e-mail in error, 
> and that any use, dissemination, forwarding, printing, or copying of 
> this e-mail is strictly prohibited. If you have received this e-mail in 
> error please notify the sender by reply e-mail and then immediately 
> permanently delete it and all copies from your system. Whilst we 
> maintain virus checks, we accept no liability for viruses introduced 
> from this e-mail.
> 
> üPlease consider the *environment* before printing this e-mail
>