[Outages-discussion] [outages] Google network issues

[Geoff Wolf AB3LS]

The RFO reminds me so much of the worst outage AWS has had thus far. Very
similar circumstances. Human error causing a larger than intended removal
of system capacity from production. For AWS it resulted in their S3 storage
system being down for hours in the middle of a business day and impacted
basically all of their cloud products and broke a ton of websites using S3
storage as a backend.

Good read: https://aws.amazon.com/message/41926/

Geoff

On Tue, Jun 4, 2019 at 10:19 PM Charles Sprickman <spork at bway.net> wrote:

>
> > On Jun 4, 2019, at 9:47 PM, Seth Mattinen <sethm at rollernet.us> wrote:
> >
> > On 6/4/19 5:51 PM, Matt Hoppes wrote:
> >> Who makes critical changes in the middle of a day on Sunday to core
> infrastructure?
> >
> >
> > A lot of people that run networks that don't have an "off peak" where
> it's always going to be someone's daytime in some timezone.
>
> In this case though, this was a known userbase - northeast US, eastern
> time zone, no?  I don’t believe the post-mortem said they rolled out these
> changes anywhere else.
>
> Charles
>
> > _______________________________________________
> > Outages-discussion mailing list
> > Outages-discussion at outages.org
> > https://puck.nether.net/mailman/listinfo/outages-discussion
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-- 
Geoffrey Wolf
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20190605/9238aaac/attachment.htm>