<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Helvetica;
panose-1:2 11 6 4 2 2 2 2 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0in;
mso-margin-bottom-alt:auto;
margin-left:0in;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
span.apple-converted-space
{mso-style-name:apple-converted-space;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link=blue vlink=purple><div class=WordSection1><p class=MsoNormal>Remember, it was supposed to impact a smaller number of systems … if that had worked, they would have rolled it back/reverted it. Google is always testing small/incremental chanages.<o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><p class=MsoNormal>Frank <o:p></o:p></p><p class=MsoNormal><o:p> </o:p></p><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in'><p class=MsoNormal><b>From:</b> Matt Hoppes <mattlists@rivervalleyinternet.net> <br><b>Sent:</b> Tuesday, June 4, 2019 7:51 PM<br><b>To:</b> Charles Sprickman <spork@bway.net><br><b>Cc:</b> Frank Bulk <frnkblk@iname.com>; outages-discussion@outages.org<br><b>Subject:</b> Re: [Outages-discussion] [outages] Google network issues<o:p></o:p></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><p class=MsoNormal>Who makes critical changes in the middle of a day on Sunday to core infrastructure?<o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'><br>On Jun 4, 2019, at 2:49 PM, Charles Sprickman <<a href="mailto:spork@bway.net">spork@bway.net</a>> wrote:<o:p></o:p></p></div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>I guess "the cloud" doesn’t have a separate management network. :)<br><br>——<o:p></o:p></p><div><p class=MsoNormal>Once alerted, engineering teams quickly identified the cause of the network congestion, but the same network congestion which was creating service degradation also slowed the engineering teams’ ability to restore the correct configurations, prolonging the outage. <o:p></o:p></p></div><div><p class=MsoNormal style='margin-bottom:12.0pt'>—— <o:p></o:p></p><div><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>On Jun 4, 2019, at 2:24 PM, Frank Bulk <<a href="mailto:frnkblk@iname.com">frnkblk@iname.com</a>> wrote:<o:p></o:p></p></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>RFO:<span class=apple-converted-space> </span><a href="https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption"><span style='color:purple'>https://cloud.google.com/blog/topics/inside-google-cloud/an-update-on-sundays-service-disruption</span></a><o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal>Frank<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><p class=MsoNormal><b>From:</b><span class=apple-converted-space> </span>Outages <<a href="mailto:outages-bounces@outages.org"><span style='color:purple'>outages-bounces@outages.org</span></a>><span class=apple-converted-space> </span><b>On Behalf Of<span class=apple-converted-space> </span></b>Kevin Day via Outages<br><b>Sent:</b><span class=apple-converted-space> </span>Sunday, June 02, 2019 3:27 PM<br><b>To:</b><span class=apple-converted-space> </span><a href="mailto:outages@outages.org"><span style='color:purple'>outages@outages.org</span></a><br><b>Subject:</b><span class=apple-converted-space> </span>[outages] Google network issues<o:p></o:p></p></div><div><p class=MsoNormal> <o:p></o:p></p></div><div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>Starting at 2019-06-02 12:53 (US/Pacific), Google is reporting a network issue affecting their own products as well as "throughout all Google Cloud Platform, this is affecting all products currently".</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'><a href="https://status.cloud.google.com/incident/cloud-networking/19009"><span style='color:purple'>https://status.cloud.google.com/incident/cloud-networking/19009</span></a></span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>Anecdotally this appears to have caused intermittent problems to complete outages for:</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>* Gmail</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>* Youtube</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>* Google's public DNS (8.8.8.8)</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>* Google products like Nest</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'>* Non-google sites apparently hosted in Google's cloud products such as Snapchat, Vimeo, etc.</span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><div><p class=MsoNormal><span style='font-size:9.0pt;font-family:"Helvetica",sans-serif'> </span><o:p></o:p></p></div></div><div><p class=MsoNormal> <o:p></o:p></p></div></div><p class=MsoNormal><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif'>_______________________________________________<br>Outages-discussion mailing list<br></span><a href="mailto:Outages-discussion@outages.org"><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:purple'>Outages-discussion@outages.org</span></a><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif'><br></span><a href="https://puck.nether.net/mailman/listinfo/outages-discussion"><span style='font-size:10.5pt;font-family:"Helvetica",sans-serif;color:purple'>https://puck.nether.net/mailman/listinfo/outages-discussion</span></a><o:p></o:p></p></div></blockquote></div><p class=MsoNormal><o:p> </o:p></p></div></div></blockquote><blockquote style='margin-top:5.0pt;margin-bottom:5.0pt'><div><p class=MsoNormal>_______________________________________________<br>Outages-discussion mailing list<br><a href="mailto:Outages-discussion@outages.org">Outages-discussion@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages-discussion">https://puck.nether.net/mailman/listinfo/outages-discussion</a><o:p></o:p></p></div></blockquote></div></body></html>