[Outages-discussion] [outages] Facebook Down

Mon Oct 4 19:07:20 EDT 2021

Or maybe commit confirmed
________________________________
From: Outages-discussion <outages-discussion-bounces at outages.org> on behalf of Chad Austin <me at chadaustin.org>
Sent: Tuesday, October 5, 2021 9:39:50 AM
To: outages-discussion at outages.org <outages-discussion at outages.org>
Subject: Re: [Outages-discussion] [outages] Facebook Down

[EXTERNAL EMAIL]

Someone must have gone a long time before they finally learned the
importance of the reload-in command.

-Chad

On 10/4/2021 1:28 PM, Matt Hoppes wrote:
> That's bad... if I read that properly.. they've locked themselves out
> of their routers and are unable to get in without physically touching
> each one.
>
> On 10/4/21 1:00 PM, Eric Henson wrote:
>> Someone on the Facebook Incident response team is posting updates to
>> Reddit.
>>
>> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0
>>
>> “As many of you know, DNS for FB services has been affected and this
>> is likely a symptom of the actual issue, and that's that *BGP peering
>> with Facebook peering routers has gone down*, very likely due to a
>> configuration change that went into effect shortly before the outages
>> happened (started roughly 1540 UTC).
>>
>> There are people now trying to gain access to the peering routers to
>> implement fixes, but the people with physical access is separate from
>> the people with knowledge of how to actually authenticate to the
>> systems and people who know what to actually do, so there is now a
>> logistical challenge with getting all that knowledge unified.
>>
>> Part of this is also due to lower staffing in data centers due to
>> pandemic measures.”
>>
>> --------
>>
>> *Eric Henson*
>>
>> Senior Manager, IT Operations
>>
>> PFSweb, Inc.
>>
>> *o*:  972.685.1387
>>
>> *m:*972.948.3424
>>
>> https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0
>>
>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Link IP
>> via Outages
>> *Sent:* Monday, October 4, 2021 10:51 AM
>> *To:* Matt Hoppes <mattlists at rivervalleyinternet.net>
>> *Cc:* outages at outages.org
>> *Subject:* Re: [outages] Facebook Down
>>
>> *CAUTION:*This email originated from outside of the organization. Do
>> not click links or open attachments unless you recognize the sender
>> and know the content is safe.
>>
>> Down here in  the UK as well
>>
>> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages
>> <outages at outages.org <mailto:outages at outages.org>> wrote:
>>
>>     Is anyone else seeing Facebook down?   Seems to be offline here in
>>     Pennsylvania.. started with error messages now DNS errors.
>>
>>     Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
>>     _______________________________________________
>>     Outages mailing list
>>     Outages at outages.org <mailto:Outages at outages.org>
>>     https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0
>>
>>
>> --
>>
>> *Lance Davis*
>>
>> *Link IP Networks Limited*
>> Delta House,
>> Wavell Road,
>> Wythenshawe,
>> M22 5QZ
>>
>> Office Tel: 0161 813 1870
>>
>> Mobile Tel: 07703 196010
>>
>> Email: lance at linkipnetworks.co.uk <mailto:lance at linkipnetworks.co.uk>
>>
>> Link IP Networks Limited is a company registered in England and Wales
>> with registration number 11881958 and registered address Delta House,
>> Wavell Road, Wythenshawe, M22 5QZ.
>>
>> This e-mail and any files transmitted with it are confidential and/or
>> legally privileged and intended solely for the use of the individual
>> or entity to whom they are addressed. If you are not the addressee
>> indicated in this e-mail (or responsible for the delivery of the e
>> mail to such person) be advised that you have received this e-mail in
>> error, and that any use, dissemination, forwarding, printing, or
>> copying of this e-mail is strictly prohibited. If you have received
>> this e-mail in error please notify the sender by reply e-mail and
>> then immediately permanently delete it and all copies from your
>> system. Whilst we maintain virus checks, we accept no liability for
>> viruses introduced from this e-mail.
>>
>> üPlease consider the *environment* before printing this e-mail
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0
_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211004/d30b8da4/attachment-0001.htm>