<div dir="ltr">That Reddit comment was deleted, and shortly after I discovered that the whole account was deleted.<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Oct 4, 2021 at 12:52 PM Matt Hoppes <<a href="mailto:mattlists@rivervalleyinternet.net">mattlists@rivervalleyinternet.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">That's bad... if I read that properly.. they've locked themselves out of <br>
their routers and are unable to get in without physically touching each one.<br>
<br>
On 10/4/21 1:00 PM, Eric Henson wrote:<br>
> Someone on the Facebook Incident response team is posting updates to Reddit.<br>
> <br>
> <a href="https://www.reddit.com/user/ramenporn/comments/" rel="noreferrer" target="_blank">https://www.reddit.com/user/ramenporn/comments/</a><br>
> <br>
> “As many of you know, DNS for FB services has been affected and this is <br>
> likely a symptom of the actual issue, and that's that *BGP peering with <br>
> Facebook peering routers has gone down*, very likely due to a <br>
> configuration change that went into effect shortly before the outages <br>
> happened (started roughly 1540 UTC).<br>
> <br>
> There are people now trying to gain access to the peering routers to <br>
> implement fixes, but the people with physical access is separate from <br>
> the people with knowledge of how to actually authenticate to the systems <br>
> and people who know what to actually do, so there is now a logistical <br>
> challenge with getting all that knowledge unified.<br>
> <br>
> Part of this is also due to lower staffing in data centers due to <br>
> pandemic measures.”<br>
> <br>
> --------<br>
> <br>
> *Eric Henson*<br>
> <br>
> Senior Manager, IT Operations<br>
> <br>
> PFSweb, Inc.<br>
> <br>
> *o*: 972.685.1387<br>
> <br>
> *m:*972.948.3424<br>
> <br>
> <a href="http://www.pfsweb.com" rel="noreferrer" target="_blank">www.pfsweb.com</a><br>
> <br>
> *From:* Outages <<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>> *On Behalf Of *Link IP via <br>
> Outages<br>
> *Sent:* Monday, October 4, 2021 10:51 AM<br>
> *To:* Matt Hoppes <<a href="mailto:mattlists@rivervalleyinternet.net" target="_blank">mattlists@rivervalleyinternet.net</a>><br>
> *Cc:* <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
> *Subject:* Re: [outages] Facebook Down<br>
> <br>
> *CAUTION:*This email originated from outside of the organization. Do not <br>
> click links or open attachments unless you recognize the sender and know <br>
> the content is safe.<br>
> <br>
> Down here in the UK as well<br>
> <br>
> On Mon, 4 Oct 2021 at 16:48, Matt Hoppes via Outages <br>
> <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> <mailto:<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>>> wrote:<br>
> <br>
> Is anyone else seeing Facebook down? Seems to be offline here in<br>
> Pennsylvania.. started with error messages now DNS errors.<br>
> <br>
> Seeing lots of errors on Instagram, QuickBooks, Whatsapp, Amazon.<br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a> <mailto:<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a>><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
> <br>
> <br>
> -- <br>
> <br>
> *Lance Davis*<br>
> <br>
> *Link IP Networks Limited*<br>
> Delta House,<br>
> Wavell Road,<br>
> Wythenshawe,<br>
> M22 5QZ<br>
> <br>
> Office Tel: 0161 813 1870<br>
> <br>
> Mobile Tel: 07703 196010<br>
> <br>
> Email: <a href="mailto:lance@linkipnetworks.co.uk" target="_blank">lance@linkipnetworks.co.uk</a> <mailto:<a href="mailto:lance@linkipnetworks.co.uk" target="_blank">lance@linkipnetworks.co.uk</a>><br>
> <br>
> Link IP Networks Limited is a company registered in England and Wales <br>
> with registration number 11881958 and registered address Delta House, <br>
> Wavell Road, Wythenshawe, M22 5QZ.<br>
> <br>
> This e-mail and any files transmitted with it are confidential and/or <br>
> legally privileged and intended solely for the use of the individual or <br>
> entity to whom they are addressed. If you are not the addressee <br>
> indicated in this e-mail (or responsible for the delivery of the e mail <br>
> to such person) be advised that you have received this e-mail in error, <br>
> and that any use, dissemination, forwarding, printing, or copying of <br>
> this e-mail is strictly prohibited. If you have received this e-mail in <br>
> error please notify the sender by reply e-mail and then immediately <br>
> permanently delete it and all copies from your system. Whilst we <br>
> maintain virus checks, we accept no liability for viruses introduced <br>
> from this e-mail.<br>
> <br>
> üPlease consider the *environment* before printing this e-mail<br>
> <br>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div>