[Outages-discussion] [outages] Google network issues

[Mara Wallace]

Considering the outcome of the maint it was the perfect time - if they had
failed on a Monday there would have been mass panic.  Grumpy sysops are
much more preferable.


On Tue, Jun 4, 2019 at 21:44 Matt Hoppes <mattlists at rivervalleyinternet.net>
wrote:

> But 2am on a Sunday?
>
> On 6/4/19 8:52 PM, Neal Lawson wrote:
> > Well google does, and many org’s I have worked for as well prefer
> > weekend changes ….
> >
> >
> >> On Jun 4, 2019, at 5:51 PM, Matt Hoppes
> >> <mattlists at rivervalleyinternet.net
> >> <mailto:mattlists at rivervalleyinternet.net>> wrote:
> >>
> >> Who makes critical changes in the middle of a day on Sunday to core
> >> infrastructure?
> >>
> >> On Jun 4, 2019, at 2:49 PM, Charles Sprickman <spork at bway.net
> >> <mailto:spork at bway.net>> wrote:
> >>
> >>> I guess "the cloud" doesn’t have a separate management network. :)
> >>>
> >>> ——
> >>> Once alerted, engineering teams quickly identified the cause of the
> >>> network congestion, but the same network congestion which was
> >>> creating service degradation also slowed the engineering teams’
> >>> ability to restore the correct configurations, prolonging the outage.
> >>> ——
> >>>
> >>>> On Jun 4, 2019, at 2:24 PM, Frank Bulk <frnkblk at iname.com
> >>>> <mailto:frnkblk at iname.com>> wrote:
> >>>>
> >>>> RFO:
> https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption
> >>>> Frank
> >>>> *From:*Outages <outages-bounces at outages.org
> >>>> <mailto:outages-bounces at outages.org>>*On Behalf Of*Kevin Day via
> Outages
> >>>> *Sent:*Sunday, June 02, 2019 3:27 PM
> >>>> *To:*outages at outages.org <mailto:outages at outages.org>
> >>>> *Subject:*[outages] Google network issues
> >>>> Starting at 2019-06-02 12:53 (US/Pacific), Google is reporting a
> >>>> network issue affecting their own products as well as "throughout
> >>>> all Google Cloud Platform, this is affecting all products currently".
> >>>> https://status.cloud.google.com/incident/cloud-networking/19009
> >>>> Anecdotally this appears to have caused intermittent problems to
> >>>> complete outages for:
> >>>> * Gmail
> >>>> * Youtube
> >>>> * Google's public DNS (8.8.8.8)
> >>>> * Google products like Nest
> >>>> * Non-google sites apparently hosted in Google's cloud products such
> >>>> as Snapchat, Vimeo, etc.
> >>>> _______________________________________________
> >>>> Outages-discussion mailing list
> >>>> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org
> >
> >>>> https://puck.nether.net/mailman/listinfo/outages-discussion
> >>>
> >>> _______________________________________________
> >>> Outages-discussion mailing list
> >>> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org>
> >>> https://puck.nether.net/mailman/listinfo/outages-discussion
> >> _______________________________________________
> >> Outages-discussion mailing list
> >> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org>
> >> https://puck.nether.net/mailman/listinfo/outages-discussion
> >
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20190604/ea7e3bed/attachment.htm>