[Outages-discussion] [outages] Google network issues

[frnkblk at iname.com]

Remember, it was supposed to impact a smaller number of systems … if that had worked, they would have rolled it back/reverted it. Google is always testing small/incremental chanages.

 

Frank 

 

From: Matt Hoppes <mattlists at rivervalleyinternet.net> 
Sent: Tuesday, June 4, 2019 7:51 PM
To: Charles Sprickman <spork at bway.net>
Cc: Frank Bulk <frnkblk at iname.com>; outages-discussion at outages.org
Subject: Re: [Outages-discussion] [outages] Google network issues

 

Who makes critical changes in the middle of a day on Sunday to core infrastructure?


On Jun 4, 2019, at 2:49 PM, Charles Sprickman <spork at bway.net <mailto:spork at bway.net> > wrote:

I guess "the cloud" doesn’t have a separate management network. :)

——

Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage. 

—— 

On Jun 4, 2019, at 2:24 PM, Frank Bulk <frnkblk at iname.com <mailto:frnkblk at iname.com> > wrote:

 

RFO:  <https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption> https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption

 

Frank

 

From: Outages < <mailto:outages-bounces at outages.org> outages-bounces at outages.org> On Behalf Of Kevin Day via Outages
Sent: Sunday, June 02, 2019 3:27 PM
To:  <mailto:outages at outages.org> outages at outages.org
Subject: [outages] Google network issues

 

Starting at 2019-06-02 12:53 (US/Pacific), Google is reporting a network issue affecting their own products as well as "throughout all Google Cloud Platform, this is affecting all products currently".

 

 <https://status.cloud.google.com/incident/cloud-networking/19009> https://status.cloud.google.com/incident/cloud-networking/19009

 

 

Anecdotally this appears to have caused intermittent problems to complete outages for:

 

* Gmail

* Youtube

* Google's public DNS (8.8.8.8)

* Google products like Nest

* Non-google sites apparently hosted in Google's cloud products such as Snapchat, Vimeo, etc.

 

 

 

 

_______________________________________________
Outages-discussion mailing list
 <mailto:Outages-discussion at outages.org> Outages-discussion at outages.org
 <https://puck.nether.net/mailman/listinfo/outages-discussion> https://puck.nether.net/mailman/listinfo/outages-discussion

 

_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org <mailto:Outages-discussion at outages.org> 
https://puck.nether.net/mailman/listinfo/outages-discussion

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20190605/51a732b4/attachment.htm>