<html>
<head>
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
</head>
<body>
<p>It looks like they learned their lesson, but are now applying it
to reddit comments. <br>
</p>
<p>-Chad<br>
</p>
<div class="moz-cite-prefix">On 10/4/2021 7:07 PM, James Liakos
wrote:<br>
</div>
<blockquote type="cite"
cite="mid:SYBPR01MB5999CF091CA5FD5F25648430ECAE9@SYBPR01MB5999.ausprd01.prod.outlook.com">
<meta http-equiv="Content-Type" content="text/html;
charset=windows-1252">
<div dir="ltr">
<div>
<div>
<div dir="ltr" style="color: rgb(0, 0, 0); background-color:
rgb(255, 255, 255);">
Or maybe commit confirmed </div>
</div>
</div>
</div>
<hr style="display:inline-block;width:98%" tabindex="-1">
<div id="divRplyFwdMsg" dir="ltr"><font style="font-size:11pt"
face="Calibri, sans-serif" color="#000000"><b>From:</b>
Outages-discussion
<a class="moz-txt-link-rfc2396E" href="mailto:outages-discussion-bounces@outages.org"><outages-discussion-bounces@outages.org></a> on behalf of
Chad Austin <a class="moz-txt-link-rfc2396E" href="mailto:me@chadaustin.org"><me@chadaustin.org></a><br>
<b>Sent:</b> Tuesday, October 5, 2021 9:39:50 AM<br>
<b>To:</b> <a class="moz-txt-link-abbreviated" href="mailto:outages-discussion@outages.org">outages-discussion@outages.org</a>
<a class="moz-txt-link-rfc2396E" href="mailto:outages-discussion@outages.org"><outages-discussion@outages.org></a><br>
<b>Subject:</b> Re: [Outages-discussion] [outages] Facebook
Down</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span
style="font-size:11pt;">
<div class="PlainText">[EXTERNAL EMAIL]<br>
<br>
Someone must have gone a long time before they finally
learned the<br>
importance of the reload-in command.<br>
<br>
-Chad<br>
<br>
On 10/4/2021 1:28 PM, Matt Hoppes wrote:<br>
> That's bad... if I read that properly.. they've
locked themselves out<br>
> of their routers and are unable to get in without
physically touching<br>
> each one.<br>
><br>
> On 10/4/21 1:00 PM, Eric Henson wrote:<br>
>> Someone on the Facebook Incident response team is
posting updates to<br>
>> Reddit.<br>
>><br>
>> <a
href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0"
moz-do-not-send="true">
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.reddit.com%2Fuser%2Framenporn%2Fcomments%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044900064%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=Pv3C6OqnOTsOZYrsBB0mDc%2BkE7T6tOc149SwLvuQzS8%3D&reserved=0</a><br>
>><br>
>> “As many of you know, DNS for FB services has
been affected and this<br>
>> is likely a symptom of the actual issue, and
that's that *BGP peering<br>
>> with Facebook peering routers has gone down*,
very likely due to a<br>
>> configuration change that went into effect
shortly before the outages<br>
>> happened (started roughly 1540 UTC).<br>
>><br>
>> There are people now trying to gain access to the
peering routers to<br>
>> implement fixes, but the people with physical
access is separate from<br>
>> the people with knowledge of how to actually
authenticate to the<br>
>> systems and people who know what to actually do,
so there is now a<br>
>> logistical challenge with getting all that
knowledge unified.<br>
>><br>
>> Part of this is also due to lower staffing in
data centers due to<br>
>> pandemic measures.”<br>
>><br>
>> --------<br>
>><br>
>> *Eric Henson*<br>
>><br>
>> Senior Manager, IT Operations<br>
>><br>
>> PFSweb, Inc.<br>
>><br>
>> *o*: 972.685.1387<br>
>><br>
>> *m:*972.948.3424<br>
>><br>
>> <a
href="https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0"
moz-do-not-send="true">
https://aus01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.pfsweb.com%2F&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=qfgtHba0DojZNkvmOF7u2sYjhxGfv3APRWn4k85oCUs%3D&reserved=0</a><br>
>><br>
>> *From:* Outages
<a class="moz-txt-link-rfc2396E" href="mailto:outages-bounces@outages.org"><outages-bounces@outages.org></a> *On Behalf Of *Link IP<br>
>> via Outages<br>
>> *Sent:* Monday, October 4, 2021 10:51 AM<br>
>> *To:* Matt Hoppes
<a class="moz-txt-link-rfc2396E" href="mailto:mattlists@rivervalleyinternet.net"><mattlists@rivervalleyinternet.net></a><br>
>> *Cc:* <a class="moz-txt-link-abbreviated" href="mailto:outages@outages.org">outages@outages.org</a><br>
>> *Subject:* Re: [outages] Facebook Down<br>
>><br>
>> *CAUTION:*This email originated from outside of
the organization. Do<br>
>> not click links or open attachments unless you
recognize the sender<br>
>> and know the content is safe.<br>
>><br>
>> Down here in the UK as well<br>
>><br>
>> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via
Outages<br>
>> <<a class="moz-txt-link-abbreviated" href="mailto:outages@outages.org">outages@outages.org</a> <<a
href="mailto:outages@outages.org" moz-do-not-send="true">mailto:outages@outages.org</a>>>
wrote:<br>
>><br>
>> Is anyone else seeing Facebook down? Seems
to be offline here in<br>
>> Pennsylvania.. started with error messages
now DNS errors.<br>
>><br>
>> Seeing lots of errors on Instagram,
QuickBooks, Whatsapp, Amazon.<br>
>>
_______________________________________________<br>
>> Outages mailing list<br>
>> <a class="moz-txt-link-abbreviated" href="mailto:Outages@outages.org">Outages@outages.org</a> <<a
href="mailto:Outages@outages.org" moz-do-not-send="true">mailto:Outages@outages.org</a>><br>
>> <a
href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0"
moz-do-not-send="true">
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=4gVfA0Tii2zABZR1ugNztgsYarea8ytRrKKbh7llQJ8%3D&reserved=0</a><br>
>><br>
>><br>
>> --<br>
>><br>
>> *Lance Davis*<br>
>><br>
>> *Link IP Networks Limited*<br>
>> Delta House,<br>
>> Wavell Road,<br>
>> Wythenshawe,<br>
>> M22 5QZ<br>
>><br>
>> Office Tel: 0161 813 1870<br>
>><br>
>> Mobile Tel: 07703 196010<br>
>><br>
>> Email: <a class="moz-txt-link-abbreviated" href="mailto:lance@linkipnetworks.co.uk">lance@linkipnetworks.co.uk</a> <<a
href="mailto:lance@linkipnetworks.co.uk"
moz-do-not-send="true">mailto:lance@linkipnetworks.co.uk</a>><br>
>><br>
>> Link IP Networks Limited is a company registered
in England and Wales<br>
>> with registration number 11881958 and registered
address Delta House,<br>
>> Wavell Road, Wythenshawe, M22 5QZ.<br>
>><br>
>> This e-mail and any files transmitted with it are
confidential and/or<br>
>> legally privileged and intended solely for the
use of the individual<br>
>> or entity to whom they are addressed. If you are
not the addressee<br>
>> indicated in this e-mail (or responsible for the
delivery of the e<br>
>> mail to such person) be advised that you have
received this e-mail in<br>
>> error, and that any use, dissemination,
forwarding, printing, or<br>
>> copying of this e-mail is strictly prohibited. If
you have received<br>
>> this e-mail in error please notify the sender by
reply e-mail and<br>
>> then immediately permanently delete it and all
copies from your<br>
>> system. Whilst we maintain virus checks, we
accept no liability for<br>
>> viruses introduced from this e-mail.<br>
>><br>
>> üPlease consider the *environment* before
printing this e-mail<br>
>><br>
> _______________________________________________<br>
> Outages-discussion mailing list<br>
> <a class="moz-txt-link-abbreviated" href="mailto:Outages-discussion@outages.org">Outages-discussion@outages.org</a><br>
> <a
href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0"
moz-do-not-send="true">
https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0</a><br>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a class="moz-txt-link-abbreviated" href="mailto:Outages-discussion@outages.org">Outages-discussion@outages.org</a><br>
<a
href="https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0"
moz-do-not-send="true">https://aus01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpuck.nether.net%2Fmailman%2Flistinfo%2Foutages-discussion&data=04%7C01%7Cjames%40liakos.com.au%7Cc59c6e5114ba4dafc6de08d98789c15b%7C6ad8265a93b14b7990a74df444e5977e%7C0%7C0%7C637689849044910026%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=dlDwjgre0MHXq%2F499A8BS69ICSJSKDiXmwNfXcKUWIU%3D&reserved=0</a><br>
</div>
</span></font></div>
</blockquote>
</body>
</html>