[Outages-discussion] FB Outage AAR I - Engineering Posts Pabulum

[Rusty Dekema]

There are terminal servers available that use cellular (e.g. LTE)
networks for their outside connectivity, too. I don't know what cell
signal availability is like on the floor of a major datacenter, but if
Facebook owns the facility, they could certainly run an antenna cable
between a terminal server and the roof.

-Rusty

On Tue, Oct 5, 2021 at 4:15 PM Bryan Holloway <bryan at shout.net> wrote:
>
> These days it's next to impossible to get a POTS line run to a cabinet
> in a major datacenter.
>
> On the other hand, FB runs their own, so I suppose anything's possible
> when you have more money than god.
>
>
> On 10/5/21 9:38 PM, Anthony Hoppe wrote:
> > If that's the case, it's a bit of an oversight.  If you're depending on DNS for OOB access, you'd want OOB DNS servers available too, heh...
> >
> > Maybe they can go back to the golden days of sticking a modem & POTS line on the console port of their routers/switches.  Or get super fancy and deploy terminal servers at each datacenter to conserve on phone lines :-D.
> >
> >
> >
> > ----- Original Message -----
> > From: "George Metz" <george.metz at gmail.com>
> > To: "Ross Tajvar" <ross at tajvar.io>
> > Cc: "Outages List" <outages-discussion at outages.org>
> > Sent: Tuesday, October 5, 2021 12:12:23 PM
> > Subject: Re: [Outages-discussion] FB Outage AAR I - Engineering Posts Pabulum
> >
> > If I had to guess (because I was wondering about that too), it was
> > because they didn't have the IPs of their out-of-band stuff available
> > and were expecting their DNS to be able to answer... and the DNS
> > servers functionally shut themselves off.
> >
> > On Tue, Oct 5, 2021 at 2:14 PM Ross Tajvar <ross at tajvar.io> wrote:
> >>
> >> "Our primary and out-of-band network access was down"
> >>
> >> Sounds like someone doesn't know what "out-of-band" means.
> >>
> >> On Tue, Oct 5, 2021, 2:04 PM Jay R. Ashworth <jra at baylink.com> wrote:
> >>>
> >>> This doesn't say anything we don't already know, except where it conflicts
> >>> with things we already know.  But it's fun to watch, ain't it?  ;-)
> >>>
> >>>    https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/
> >>>
> >>> Cheers,
> >>> -- jra
> >>>
> >>> --
> >>> Jay R. Ashworth                  Baylink                       jra at baylink.com
> >>> Designer                     The Things I Think                       RFC 2100
> >>> Ashworth & Associates       http://www.bcp38.info          2000 Land Rover DII
> >>> St Petersburg FL USA      BCP38: Ask For It By Name!           +1 727 647 1274
> >>> _______________________________________________
> >>> Outages-discussion mailing list
> >>> Outages-discussion at outages.org
> >>> https://puck.nether.net/mailman/listinfo/outages-discussion
> >>
> >> _______________________________________________
> >> Outages-discussion mailing list
> >> Outages-discussion at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages-discussion
> > _______________________________________________
> > Outages-discussion mailing list
> > Outages-discussion at outages.org
> > https://puck.nether.net/mailman/listinfo/outages-discussion
> > _______________________________________________
> > Outages-discussion mailing list
> > Outages-discussion at outages.org
> > https://puck.nether.net/mailman/listinfo/outages-discussion
> >
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion