[Outages-discussion] [outages] Facebook Down

[Chad Austin]

It looks like they learned their lesson, but are now applying it to 
reddit comments.

-Chad

On 10/4/2021 7:07 PM, James Liakos wrote:
> Or maybe commit confirmed
> ------------------------------------------------------------------------
> *From:* Outages-discussion <outages-discussion-bounces at outages.org> on 
> behalf of Chad Austin <me at chadaustin.org>
> *Sent:* Tuesday, October 5, 2021 9:39:50 AM
> *To:* outages-discussion at outages.org <outages-discussion at outages.org>
> *Subject:* Re: [Outages-discussion] [outages] Facebook Down
> [EXTERNAL EMAIL]
>
> Someone must have gone a long time before they finally learned the
> importance of the reload-in command.
>
> -Chad
>
> On 10/4/2021 1:28 PM, Matt Hoppes wrote:
> > That's bad... if I read that properly.. they've locked themselves out
> > of their routers and are unable to get in without physically touching
> > each one.
> >
> > On 10/4/21 1:00 PM, Eric Henson wrote:
> >> Someone on the Facebook Incident response team is posting updates to
> >> Reddit.
> >>
> >> 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0 
> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0>
> >>
> >> “As many of you know, DNS for FB services has been affected and this
> >> is likely a symptom of the actual issue, and that's that *BGP peering
> >> with Facebook peering routers has gone down*, very likely due to a
> >> configuration change that went into effect shortly before the outages
> >> happened (started roughly 1540 UTC).
> >>
> >> There are people now trying to gain access to the peering routers to
> >> implement fixes, but the people with physical access is separate from
> >> the people with knowledge of how to actually authenticate to the
> >> systems and people who know what to actually do, so there is now a
> >> logistical challenge with getting all that knowledge unified.
> >>
> >> Part of this is also due to lower staffing in data centers due to
> >> pandemic measures.”
> >>
> >> --------
> >>
> >> *Eric Henson*
> >>
> >> Senior Manager, IT Operations
> >>
> >> PFSweb, Inc.
> >>
> >> *o*:  972.685.1387
> >>
> >> *m:*972.948.3424
> >>
> >> 
> https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0 
> <https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0>
> >>
> >> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Link IP
> >> via Outages
> >> *Sent:* Monday, October 4, 2021 10:51 AM
> >> *To:* Matt Hoppes <mattlists at rivervalleyinternet.net>
> >> *Cc:* outages at outages.org
> >> *Subject:* Re: [outages] Facebook Down
> >>
> >> *CAUTION:*This email originated from outside of the organization. Do
> >> not click links or open attachments unless you recognize the sender
> >> and know the content is safe.
> >>
> >> Down here in  the UK as well
> >>
> >> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages
> >> <outages at outages.org <mailto:outages at outages.org 
> <mailto:outages at outages.org>>> wrote:
> >>
> >>     Is anyone else seeing Facebook down?   Seems to be offline here in
> >>     Pennsylvania.. started with error messages now DNS errors.
> >>
> >>     Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.
> >> _______________________________________________
> >>     Outages mailing list
> >>     Outages at outages.org <mailto:Outages at outages.org 
> <mailto:Outages at outages.org>>
> >> 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0 
> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0>
> >>
> >>
> >> --
> >>
> >> *Lance Davis*
> >>
> >> *Link IP Networks Limited*
> >> Delta House,
> >> Wavell Road,
> >> Wythenshawe,
> >> M22 5QZ
> >>
> >> Office Tel: 0161 813 1870
> >>
> >> Mobile Tel: 07703 196010
> >>
> >> Email: lance at linkipnetworks.co.uk 
> <mailto:lance at linkipnetworks.co.uk <mailto:lance at linkipnetworks.co.uk>>
> >>
> >> Link IP Networks Limited is a company registered in England and Wales
> >> with registration number 11881958 and registered address Delta House,
> >> Wavell Road, Wythenshawe, M22 5QZ.
> >>
> >> This e-mail and any files transmitted with it are confidential and/or
> >> legally privileged and intended solely for the use of the individual
> >> or entity to whom they are addressed. If you are not the addressee
> >> indicated in this e-mail (or responsible for the delivery of the e
> >> mail to such person) be advised that you have received this e-mail in
> >> error, and that any use, dissemination, forwarding, printing, or
> >> copying of this e-mail is strictly prohibited. If you have received
> >> this e-mail in error please notify the sender by reply e-mail and
> >> then immediately permanently delete it and all copies from your
> >> system. Whilst we maintain virus checks, we accept no liability for
> >> viruses introduced from this e-mail.
> >>
> >> üPlease consider the *environment* before printing this e-mail
> >>
> > _______________________________________________
> > Outages-discussion mailing list
> > Outages-discussion at outages.org
> > 
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0 
> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0 
> <https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211004/0bf6d340/attachment-0001.htm>