[Outages-discussion] FB Outage AAR I - Engineering Posts Pabulum

[Bryan Holloway]

These days it's next to impossible to get a POTS line run to a cabinet 
in a major datacenter.

On the other hand, FB runs their own, so I suppose anything's possible 
when you have more money than god.


On 10/5/21 9:38 PM, Anthony Hoppe wrote:
> If that's the case, it's a bit of an oversight.  If you're depending on DNS for OOB access, you'd want OOB DNS servers available too, heh...
> 
> Maybe they can go back to the golden days of sticking a modem & POTS line on the console port of their routers/switches.  Or get super fancy and deploy terminal servers at each datacenter to conserve on phone lines :-D.
> 
> 
> 
> ----- Original Message -----
> From: "George Metz" <george.metz at gmail.com>
> To: "Ross Tajvar" <ross at tajvar.io>
> Cc: "Outages List" <outages-discussion at outages.org>
> Sent: Tuesday, October 5, 2021 12:12:23 PM
> Subject: Re: [Outages-discussion] FB Outage AAR I - Engineering Posts Pabulum
> 
> If I had to guess (because I was wondering about that too), it was
> because they didn't have the IPs of their out-of-band stuff available
> and were expecting their DNS to be able to answer... and the DNS
> servers functionally shut themselves off.
> 
> On Tue, Oct 5, 2021 at 2:14 PM Ross Tajvar <ross at tajvar.io> wrote:
>>
>> "Our primary and out-of-band network access was down"
>>
>> Sounds like someone doesn't know what "out-of-band" means.
>>
>> On Tue, Oct 5, 2021, 2:04 PM Jay R. Ashworth <jra at baylink.com> wrote:
>>>
>>> This doesn't say anything we don't already know, except where it conflicts
>>> with things we already know.  But it's fun to watch, ain't it?  ;-)
>>>
>>>    https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
>>>
>>> Cheers,
>>> -- jra
>>>
>>> --
>>> Jay R. Ashworth                  Baylink                       jra at baylink.com
>>> Designer                     The Things I Think                       RFC 2100
>>> Ashworth & Associates       http://www.bcp38.info          2000 Land Rover DII
>>> St Petersburg FL USA      BCP38: Ask For It By Name!           +1 727 647 1274
>>> _______________________________________________
>>> Outages-discussion mailing list
>>> Outages-discussion at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages-discussion
>>
>> _______________________________________________
>> Outages-discussion mailing list
>> Outages-discussion at outages.org
>> https://puck.nether.net/mailman/listinfo/outages-discussion
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>