[Outages-discussion] [outages] Google network issues

[Matt Hoppes]

But 2am on a Sunday?

On 6/4/19 8:52 PM, Neal Lawson wrote:
> Well google does, and many org’s I have worked for as well prefer 
> weekend changes ….
> 
> 
>> On Jun 4, 2019, at 5:51 PM, Matt Hoppes 
>> <mattlists at rivervalleyinternet.net 
>> <mailto:mattlists at rivervalleyinternet.net>> wrote:
>>
>> Who makes critical changes in the middle of a day on Sunday to core 
>> infrastructure?
>>
>> On Jun 4, 2019, at 2:49 PM, Charles Sprickman <spork at bway.net 
>> <mailto:spork at bway.net>> wrote:
>>
>>> I guess "the cloud" doesn’t have a separate management network. :)
>>>
>>> ——
>>> Once alerted, engineering teams quickly identified the cause of the 
>>> network congestion, but the same network congestion which was 
>>> creating service degradation also slowed the engineering teams’ 
>>> ability to restore the correct configurations, prolonging the outage.
>>> ——
>>>
>>>> On Jun 4, 2019, at 2:24 PM, Frank Bulk <frnkblk at iname.com 
>>>> <mailto:frnkblk at iname.com>> wrote:
>>>>
>>>> RFO:https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption
>>>> Frank
>>>> *From:*Outages <outages-bounces at outages.org 
>>>> <mailto:outages-bounces at outages.org>>*On Behalf Of*Kevin Day via Outages
>>>> *Sent:*Sunday, June 02, 2019 3:27 PM
>>>> *To:*outages at outages.org <mailto:outages at outages.org>
>>>> *Subject:*[outages] Google network issues
>>>> Starting at 2019-06-02 12:53 (US/Pacific), Google is reporting a 
>>>> network issue affecting their own products as well as "throughout 
>>>> all Google Cloud Platform, this is affecting all products currently".
>>>> https://status.cloud.google.com/incident/cloud-networking/19009
>>>> Anecdotally this appears to have caused intermittent problems to 
>>>> complete outages for:
>>>> * Gmail
>>>> * Youtube
>>>> * Google's public DNS (8.8.8.8)
>>>> * Google products like Nest
>>>> * Non-google sites apparently hosted in Google's cloud products such 
>>>> as Snapchat, Vimeo, etc.
>>>> _______________________________________________
>>>> Outages-discussion mailing list
>>>> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org>
>>>> https://puck.nether.net/mailman/listinfo/outages-discussion
>>>
>>> _______________________________________________
>>> Outages-discussion mailing list
>>> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org>
>>> https://puck.nether.net/mailman/listinfo/outages-discussion
>> _______________________________________________
>> Outages-discussion mailing list
>> Outages-discussion at outages.org <mailto:Outages-discussion at outages.org>
>> https://puck.nether.net/mailman/listinfo/outages-discussion
>