<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=Windows-1252">
<meta name="Generator" content="Microsoft Exchange Server">
<!-- converted from text --><style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left: #800000 2px solid; } --></style>
</head>
<body>
<div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.0pt; line-height:1.3; color:#1F497D">
<div>If you're a 911 answering center, your best time is probably early Sunday morning, after the bar-time LE/EMS rush, but before everyone starts getting out of church. <br>
<br>
If you're a corporation with offices sprinkled around the globe and customers who need support 24x7, there's no good time... But we go with Friday evening in to early Saturday morning US Central Time because it hits where the most quiet times coincide in the
cultures/countries we work with.<br>
<br>
It's all a matter of perspective. <br>
<br>
</div>
<div><br>
</div>
<div id="x_signature-x" class="x_signature_editor" style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12.0pt; color:#1F497D">
Sent from my phone; disregard quality of message.<br>
</div>
</div>
<div id="x_quoted_header" style="clear:both">
<hr style="border:none; height:1px; color:#E1E1E1; background-color:#E1E1E1">
<div style="border:none; padding:3.0pt 0cm 0cm 0cm"><span style="font-size:11.0pt; font-family:'Calibri','sans-serif'"><b>From:</b> Matt Hoppes <mattlists@rivervalleyinternet.net><br>
<b>Sent:</b> Tuesday, June 4, 2019 20:44<br>
<b>To:</b> Neal Lawson<br>
<b>Cc:</b> outages-discussion@outages.org<br>
<b>Subject:</b> Re: [Outages-discussion] [outages] Google network issues<br>
</span></div>
</div>
<br type="attribution">
</div>
<font size="2"><span style="font-size:10pt;">
<div class="PlainText">External Mail. Careful of links / attachments. Submit Helpdesk if unsure.<br>
<br>
<br>
But 2am on a Sunday?<br>
<br>
On 6/4/19 8:52 PM, Neal Lawson wrote:<br>
> Well google does, and many org’s I have worked for as well prefer<br>
> weekend changes ….<br>
><br>
><br>
>> On Jun 4, 2019, at 5:51 PM, Matt Hoppes<br>
>> <mattlists@rivervalleyinternet.net<br>
>> <<a href="mailto:mattlists@rivervalleyinternet.net">mailto:mattlists@rivervalleyinternet.net</a>>> wrote:<br>
>><br>
>> Who makes critical changes in the middle of a day on Sunday to core<br>
>> infrastructure?<br>
>><br>
>> On Jun 4, 2019, at 2:49 PM, Charles Sprickman <spork@bway.net<br>
>> <<a href="mailto:spork@bway.net">mailto:spork@bway.net</a>>> wrote:<br>
>><br>
>>> I guess "the cloud" doesn’t have a separate management network. :)<br>
>>><br>
>>> ——<br>
>>> Once alerted, engineering teams quickly identified the cause of the<br>
>>> network congestion, but the same network congestion which was<br>
>>> creating service degradation also slowed the engineering teams’<br>
>>> ability to restore the correct configurations, prolonging the outage.<br>
>>> ——<br>
>>><br>
>>>> On Jun 4, 2019, at 2:24 PM, Frank Bulk <frnkblk@iname.com<br>
>>>> <<a href="mailto:frnkblk@iname.com">mailto:frnkblk@iname.com</a>>> wrote:<br>
>>>><br>
>>>> RFO:https://urldefense.proofpoint.com/v2/url?u=https-3A__cloud.google.com_blog_topics_inside-2Dgoogle-2Dcloud_an-2Dupdate-2Don-2Dsundays-2Dservice-2Ddisruption&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=FwBJD9RBfsyKpaTY1sy3VdBN-JfQNqleXR_BRkG9k-c&e=<br>
>>>> Frank<br>
>>>> *From:*Outages <outages-bounces@outages.org<br>
>>>> <<a href="mailto:outages-bounces@outages.org">mailto:outages-bounces@outages.org</a>>>*On Behalf Of*Kevin Day via Outages<br>
>>>> *Sent:*Sunday, June 02, 2019 3:27 PM<br>
>>>> *To:*outages@outages.org <<a href="mailto:outages@outages.org">mailto:outages@outages.org</a>><br>
>>>> *Subject:*[outages] Google network issues<br>
>>>> Starting at 2019-06-02 12:53 (US/Pacific), Google is reporting a<br>
>>>> network issue affecting their own products as well as "throughout<br>
>>>> all Google Cloud Platform, this is affecting all products currently".<br>
>>>> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__status.cloud.google.com_incident_cloud-2Dnetworking_19009&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=mdc1caZeLDbtv7vsWgl5kKWPcBEhV2tatc9TnH09ZUI&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__status.cloud.google.com_incident_cloud-2Dnetworking_19009&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=mdc1caZeLDbtv7vsWgl5kKWPcBEhV2tatc9TnH09ZUI&e=</a><br>
>>>> Anecdotally this appears to have caused intermittent problems to<br>
>>>> complete outages for:<br>
>>>> * Gmail<br>
>>>> * Youtube<br>
>>>> * Google's public DNS (8.8.8.8)<br>
>>>> * Google products like Nest<br>
>>>> * Non-google sites apparently hosted in Google's cloud products such<br>
>>>> as Snapchat, Vimeo, etc.<br>
>>>> _______________________________________________<br>
>>>> Outages-discussion mailing list<br>
>>>> Outages-discussion@outages.org <<a href="mailto:Outages-discussion@outages.org">mailto:Outages-discussion@outages.org</a>><br>
>>>> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=</a><br>
>>><br>
>>> _______________________________________________<br>
>>> Outages-discussion mailing list<br>
>>> Outages-discussion@outages.org <<a href="mailto:Outages-discussion@outages.org">mailto:Outages-discussion@outages.org</a>><br>
>>> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=</a><br>
>> _______________________________________________<br>
>> Outages-discussion mailing list<br>
>> Outages-discussion@outages.org <<a href="mailto:Outages-discussion@outages.org">mailto:Outages-discussion@outages.org</a>><br>
>> <a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=">
https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=</a><br>
><br>
_______________________________________________<br>
Outages-discussion mailing list<br>
Outages-discussion@outages.org<br>
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=">https://urldefense.proofpoint.com/v2/url?u=https-3A__puck.nether.net_mailman_listinfo_outages-2Ddiscussion&d=DwIGaQ&c=hmKldkTQNOmajg2omeC5PQ&r=M1K-203ZNyxaobL1crVDnyAzABRAVHowr_bA7QoeIkQ&m=gqZSgIELLm131ly75tMjNcvrZXWOy-orJeCJTzetiPw&s=9xEv71HCptRQaQ-cm9dtP3IS9hsm3XLZbUINAenZKdA&e=</a><br>
</div>
</span></font>
</body>
</html>