<div dir="ltr">This info pertains to <a href="http://accounts.google.com">accounts.google.com</a>:<div><br></div><div>Errors started around noon MST 12-Nov.</div><div><img src="https://mail.google.com/mail/u/0/?ui=2&ik=c8cbc4d38b&view=att&th=1586a6ba4415daac&attid=0.1&disp=safe&realattid=ii_ivk5r1350_1586a6ba571de3e9&zw" width="562" height="179"><br><br></div><div>We didn't start actively troubleshooting until the morning of the 14th. <a href="http://accounts.google.com">accounts.google.com</a> was resolving to <span style="font-size:12.8px">216.58.219.237 from our DC and our corporate office. We were seeing constant curl timeouts (well over 50%) and mtr showed tons of packet loss at the last two hops (anywhere from 4%-60%). An offsite user found that they were resolving <a href="http://accounts.google.com">accounts.google.com</a> to </span><span style="font-size:12.8px">216.58.219.205 and they were not seeing the same curl timeouts. Setup some host file hacks on a few hosts, and didn't see any more errors from those hosts. The </span><span style="font-size:12.8px">216.58.219.237</span><span style="font-size:12.8px"> endpoint is still problematic today.</span></div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><br><div>----------</div><div>Tim Korn</div><div>Network Ninja</div><div><br></div></div></div></div>
<br><div class="gmail_quote">On Tue, Nov 15, 2016 at 4:50 PM, Laurie Denness <span dir="ltr"><<a href="mailto:ldenness@etsy.com" target="_blank">ldenness@etsy.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Note this affecting Gmail and adwords heavily too, if you're in the NYC metro, see if you can reproduce:<div><br></div><div><div>[box:~] $ time curl -4 -vv <a href="http://adwords.google.com" target="_blank">http://adwords.google.com</a></div><div>* About to connect() to <a href="http://adwords.google.com" target="_blank">adwords.google.com</a> port 80 (#0)</div><div>* Trying 74.125.29.100...</div></div><div><hang for 1 minute> </div><div><br></div><div><br></div><div>tcpdump:</div><div><div>23:47:02.402713 IP (tos 0x0, ttl 64, id 6873, offset 0, flags [DF], proto TCP (6), length 60)</div><div> box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum 0x738d (incorrect -> 0x4660), seq 570037774, win 29200, options [mss 1460,sackOK,TS val 1067194520 ecr 0,nop,wscale 9], length 0</div><div>23:47:06.410714 IP (tos 0x0, ttl 64, id 6874, offset 0, flags [DF], proto TCP (6), length 60)</div><div> box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum 0x738d (incorrect -> 0x36b8), seq 570037774, win 29200, options [mss 1460,sackOK,TS val 1067198528 ecr 0,nop,wscale 9], length 0</div><div>23:47:14.426726 IP (tos 0x0, ttl 64, id 6875, offset 0, flags [DF], proto TCP (6), length 60)</div><div> box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum 0x738d (incorrect -> 0x1768), seq 570037774, win 29200, options [mss 1460,sackOK,TS val 1067206544 ecr 0,nop,wscale 9], length 0</div></div><div><br></div><div>I'd think we were going nuts, except we're seeing from many independent networks and looks like there's a flurry of NY based people reporting on <a href="http://downdetector.com/status/google/map/" target="_blank">http://downdetector.com/<wbr>status/google/map/</a></div><div><br></div><div>Definitely seems NYC based only, although I cannot replicate on Verizon FiOS in Manhattan. </div></div><div class="gmail_extra"><br><div class="gmail_quote"><div><div class="h5">On 15 November 2016 at 18:38, Tim Korn via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="h5"><div dir="ltr">We've been having issues reaching google since last Friday. We're trying from at least four different ISPs (Zayo, Cogent, TimeWarner, Verizon FiOS) and all show the same issues intermittently. <div><br></div><div>We're having issues with the google apps suite, <a href="http://accounts.google.com" target="_blank">accounts.google.com</a>, and <a href="http://www.google.com" target="_blank">www.google.com</a>. <br><div><br></div><div><br></div><div><font size="4">From our corporate office:</font></div><div><div>[bam01:~] $ time curl -4 -vv <a href="http://www.google.com" target="_blank">http://www.google.com</a></div><div>* Rebuilt URL to: <a href="http://www.google.com/" target="_blank">http://www.google.com/</a></div><div>* Trying 74.125.21.103...</div><div>* connect to 74.125.21.103 port 80 failed: Connection refused</div><div>* Trying 74.125.21.99...</div><div>* Connected to <a href="http://www.google.com" target="_blank">www.google.com</a> (74.125.21.99) port 80 (#0)</div><div>> GET / HTTP/1.1</div><div>> Host: <a href="http://www.google.com" target="_blank">www.google.com</a></div><div>> User-Agent: curl/7.43.0</div><div>> Accept: */*</div><div>></div><div>< HTTP/1.1 200 OK</div><div><snip></div><div><b>real 1m6.523s (1 minute connect time)</b></div><div><br></div><div><br></div><div><font size="4">From Datacenter:</font></div><div>[dingy01.ny2:~] $ time curl -4 -vv <a href="http://www.google.com" target="_blank">http://www.google.com</a></div><div>* About to connect() to <a href="http://www.google.com" target="_blank">www.google.com</a> port 80 (#0)</div><div>* Trying 216.58.193.132...</div><div>* Connected to <a href="http://www.google.com" target="_blank">www.google.com</a> (216.58.193.132) port 80 (#0)</div><div>> GET / HTTP/1.1</div><div>> User-Agent: curl/7.29.0</div><div>> Host: <a href="http://www.google.com" target="_blank">www.google.com</a></div><div>> Accept: */*</div><div>></div><div>< HTTP/1.1 200 OK</div><div><snip></div><div><b>real 1m3.189s (took 1 minute to connect)</b></div></div><div><div><div class="m_8257528468601426990m_-3584236057204636984gmail_signature"><div dir="ltr"><br></div><div dir="ltr"><div dir="ltr" style="font-size:12.8px">[ding02.ny5:~] $ curl -4 -k <a href="https://accounts.google.com/" target="_blank">https://accounts.google.com</a></div><div dir="ltr" style="font-size:12.8px"><div dir="ltr"><b>curl: (7) couldn't connect to host</b></div></div><div dir="ltr" style="font-size:12.8px"><div dir="ltr">[ding02.ny5:~] $ curl -4 -k <a href="https://accounts.google.com/" target="_blank">https://accounts.google.com</a></div><div dir="ltr"><b>curl: (7) couldn't connect to host</b></div><div dir="ltr">[ding02.ny5:~] $ curl -4 -k <a href="https://accounts.google.com/" target="_blank">https://accounts.google.com</a></div><div dir="ltr"><HTML></div><div dir="ltr"><HEAD></div><div dir="ltr"><TITLE>Moved Temporarily</TITLE></div><div dir="ltr"></HEAD></div><div dir="ltr"><BODY BGCOLOR="#FFFFFF" TEXT="#000000"></div><div dir="ltr"><H1>Moved Temporarily</H1></div><div dir="ltr">The document has moved <A HREF="<a href="https://accounts.google.com/ManageAccount" target="_blank">https://accounts.google.<wbr>com/ManageAccount</a>">here</A>.</div><div dir="ltr"></BODY></div><div dir="ltr"></HTML></div><div dir="ltr"><br></div><div dir="ltr"><br></div></div><div>----------</div><div>Tim Korn</div><div><br></div></div></div></div>
</div></div></div>
<br></div></div><span class="">______________________________<wbr>_________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailma<wbr>n/listinfo/outages</a><br>
<br></span></blockquote></div><br></div>
</blockquote></div><br></div>