<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
</head>
<body>
<div dir="ltr">
<div></div>
<div>
<div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color: rgb(255, 255, 255);">
Or maybe commit confirmed </div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" style="font-size:11pt" color="#000000"><b>From:</b> Outages-discussion <outages-discussion-bounces@outages.org> on behalf of Chad Austin <me@chadaustin.org><br>
<b>Sent:</b> Tuesday, October 5, 2021 9:39:50 AM<br>
<b>To:</b> outages-discussion@outages.org <outages-discussion@outages.org><br>
<b>Subject:</b> Re: [Outages-discussion] [outages] Facebook Down</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt;">
<div class="PlainText">[EXTERNAL EMAIL]<br>
<br>
Someone must have gone a long time before they finally learned the<br>
importance of the reload-in command.<br>
<br>
-Chad<br>
<br>
On 10/4/2021 1:28 PM, Matt Hoppes wrote:<br>
> That's bad... if I read that properly.. they've locked themselves out<br>
> of their routers and are unable to get in without physically touching<br>
> each one.<br>
><br>
> On 10/4/21 1:00 PM, Eric Henson wrote:<br>
>> Someone on the Facebook Incident response team is posting updates to<br>
>> Reddit.<br>
>><br>
>> <a href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0">
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0</a><br>
>><br>
>> “As many of you know, DNS for FB services has been affected and this<br>
>> is likely a symptom of the actual issue, and that's that *BGP peering<br>
>> with Facebook peering routers has gone down*, very likely due to a<br>
>> configuration change that went into effect shortly before the outages<br>
>> happened (started roughly 1540 UTC).<br>
>><br>
>> There are people now trying to gain access to the peering routers to<br>
>> implement fixes, but the people with physical access is separate from<br>
>> the people with knowledge of how to actually authenticate to the<br>
>> systems and people who know what to actually do, so there is now a<br>
>> logistical challenge with getting all that knowledge unified.<br>
>><br>
>> Part of this is also due to lower staffing in data centers due to<br>
>> pandemic measures.”<br>
>><br>
>> --------<br>
>><br>
>> *Eric Henson*<br>
>><br>
>> Senior Manager, IT Operations<br>
>><br>
>> PFSweb, Inc.<br>
>><br>
>> *o*: 972.685.1387<br>
>><br>
>> *m:*972.948.3424<br>
>><br>
>> <a href="https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0">
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0</a><br>
>><br>
>> *From:* Outages <outages-bounces@outages.org> *On Behalf Of *Link IP<br>
>> via Outages<br>
>> *Sent:* Monday, October 4, 2021 10:51 AM<br>
>> *To:* Matt Hoppes <mattlists@rivervalleyinternet.net><br>
>> *Cc:* outages@outages.org<br>
>> *Subject:* Re: [outages] Facebook Down<br>
>><br>
>> *CAUTION:*This email originated from outside of the organization. Do<br>
>> not click links or open attachments unless you recognize the sender<br>
>> and know the content is safe.<br>
>><br>
>> Down here in the UK as well<br>
>><br>
>> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages<br>
>> <outages@outages.org <<a href="mailto:outages@outages.org">mailto:outages@outages.org</a>>> wrote:<br>
>><br>
>> Is anyone else seeing Facebook down? Seems to be offline here in<br>
>> Pennsylvania.. started with error messages now DNS errors.<br>
>><br>
>> Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.<br>
>> _______________________________________________<br>
>> Outages mailing list<br>
>> Outages@outages.org <<a href="mailto:Outages@outages.org">mailto:Outages@outages.org</a>><br>
>> <a href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0">
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0</a><br>
>><br>
>><br>
>> --<br>
>><br>
>> *Lance Davis*<br>
>><br>
>> *Link IP Networks Limited*<br>
>> Delta House,<br>
>> Wavell Road,<br>
>> Wythenshawe,<br>
>> M22 5QZ<br>
>><br>
>> Office Tel: 0161 813 1870<br>
>><br>
>> Mobile Tel: 07703 196010<br>
>><br>
>> Email: lance@linkipnetworks.co.uk <<a href="mailto:lance@linkipnetworks.co.uk">mailto:lance@linkipnetworks.co.uk</a>><br>
>><br>
>> Link IP Networks Limited is a company registered in England and Wales<br>
>> with registration number 11881958 and registered address Delta House,<br>
>> Wavell Road, Wythenshawe, M22 5QZ.<br>
>><br>
>> This e-mail and any files transmitted with it are confidential and/or<br>
>> legally privileged and intended solely for the use of the individual<br>
>> or entity to whom they are addressed. If you are not the addressee<br>
>> indicated in this e-mail (or responsible for the delivery of the e<br>
>> mail to such person) be advised that you have received this e-mail in<br>
>> error, and that any use, dissemination, forwarding, printing, or<br>
>> copying of this e-mail is strictly prohibited. If you have received<br>
>> this e-mail in error please notify the sender by reply e-mail and<br>
>> then immediately permanently delete it and all copies from your<br>
>> system. Whilst we maintain virus checks, we accept no liability for<br>
>> viruses introduced from this e-mail.<br>
>><br>
>> üPlease consider the *environment* before printing this e-mail<br>
>><br>
> _______________________________________________<br>
> Outages-discussion mailing list<br>
> Outages-discussion@outages.org<br>
> <a href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0">
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0</a><br>
_______________________________________________<br>
Outages-discussion mailing list<br>
Outages-discussion@outages.org<br>
<a href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0">https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0</a><br>
</div>
</span></font></div>
</body>
</html>