[Outages-discussion] FB Outage AAR I - Engineering Posts Pabulum

[Matthew Petach]

On Tue, Oct 5, 2021 at 10:19 PM Ross Tajvar <ross at tajvar.io> wrote:

> That works too, depending on the size of your infrastructure and the size
> of your organization. If hundreds of people across the world are changing
> things all the time, it might not work so well...
>

Ah.

I wasn't worried about the hundreds of thousands of servers
in datacenters that were user-facing.  I was worried about the
terminal servers and out of band routers, the authentication
servers, the hidden master DNS servers--the crucial bits you
really needed to be able to bootstrap in case of disaster, the
things that everything else depended upon in order to work.

Those are the systems and IP addresses that go into the binder;
not the rest of the servers and systems that are touched by
hundreds of people around the world all the time.

As a corollary to that--if hundreds of people around the world
are changing your out of band infrastructure all the time,
it's probably not really out of band anymore.   ^_^;

Matt



>
> On Wed, Oct 6, 2021, 1:17 AM Matthew Petach <matt at petach.org> wrote:
>
>>
>>
>> On Tue, Oct 5, 2021, 15:59 Ross Tajvar <ross at tajvar.io> wrote:
>>
>>> There are ways to be secure without "running it yourself".
>>> For example, if the US government can trust AWS, surely so can Facebook?
>>> Especially for something like "a backup copy of documentation".
>>>
>>> On Tue, Oct 5, 2021 at 6:45 PM Tim Dobson <lists at tdobson.net> wrote:
>>>
>>>> On 05/10/2021 21:07, Mike Bolitho wrote:
>>>> > That's definitely possible but what team doesn't have their
>>>> management
>>>> > IPs somewhere other than DNS? That seems crazy to me.
>>>>
>>>> Like stored in an internal information management system in your own IP
>>>> space? Whoops.
>>>>
>>>> I'd imagine that storing that sort of information outside of facebook
>>>> could have contravened their own policies on data security.
>>>>
>>>
>>
>> I've always been partial to a binder with printouts of all vital IPs for
>> infrastructure and disaster recovery, updated monthly with apologies to to
>> the trees.
>>
>> Matt
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20211006/9e0351f4/attachment.htm>