[outages] Google Connectivity Issues in NYC

Tue Nov 15 19:10:48 EST 2016

This info pertains to accounts.google.com:

Errors started around noon MST 12-Nov.

We didn't start actively troubleshooting until the morning of the 14th.
accounts.google.com was resolving to 216.58.219.237 from our DC and our
corporate office.  We were seeing constant curl timeouts (well over 50%)
and mtr showed tons of packet loss at the last two hops (anywhere from
4%-60%).  An offsite user found that they were resolving accounts.google.com
to 216.58.219.205 and they were not seeing the same curl timeouts.  Setup
some host file hacks on a few hosts, and didn't see any more errors from
those hosts.  The 216.58.219.237 endpoint is still problematic today.

----------
Tim Korn
Network Ninja

On Tue, Nov 15, 2016 at 4:50 PM, Laurie Denness <ldenness at etsy.com> wrote:

> Note this affecting Gmail and adwords heavily too, if you're in the NYC
> metro, see if you can reproduce:
>
> [box:~] $ time curl -4 -vv http://adwords.google.com
> * About to connect() to adwords.google.com port 80 (#0)
> *   Trying 74.125.29.100...
> <hang for 1 minute>
>
>
> tcpdump:
> 23:47:02.402713 IP (tos 0x0, ttl 64, id 6873, offset 0, flags [DF], proto
> TCP (6), length 60)
>     box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum
> 0x738d (incorrect -> 0x4660), seq 570037774, win 29200, options [mss
> 1460,sackOK,TS val 1067194520 ecr 0,nop,wscale 9], length 0
> 23:47:06.410714 IP (tos 0x0, ttl 64, id 6874, offset 0, flags [DF], proto
> TCP (6), length 60)
>     box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum
> 0x738d (incorrect -> 0x36b8), seq 570037774, win 29200, options [mss
> 1460,sackOK,TS val 1067198528 ecr 0,nop,wscale 9], length 0
> 23:47:14.426726 IP (tos 0x0, ttl 64, id 6875, offset 0, flags [DF], proto
> TCP (6), length 60)
>     box.dc.etsy.com.35746 > qg-in-f113.1e100.net.http: Flags [S], cksum
> 0x738d (incorrect -> 0x1768), seq 570037774, win 29200, options [mss
> 1460,sackOK,TS val 1067206544 ecr 0,nop,wscale 9], length 0
>
> I'd think we were going nuts, except we're seeing from many independent
> networks and looks like there's a flurry of NY based people reporting on
> http://downdetector.com/status/google/map/
>
> Definitely seems NYC based only, although I cannot replicate on Verizon
> FiOS in Manhattan.
>
> On 15 November 2016 at 18:38, Tim Korn via Outages <outages at outages.org>
> wrote:
>
>> We've been having issues reaching google since last Friday.  We're trying
>> from at least four different ISPs (Zayo, Cogent, TimeWarner, Verizon FiOS)
>> and all show the same issues intermittently.
>>
>> We're having issues with the google apps suite, accounts.google.com, and
>> www.google.com.
>>
>>
>> From our corporate office:
>> [bam01:~] $ time curl -4 -vv http://www.google.com
>> * Rebuilt URL to: http://www.google.com/
>> *   Trying 74.125.21.103...
>> * connect to 74.125.21.103 port 80 failed: Connection refused
>> *   Trying 74.125.21.99...
>> * Connected to www.google.com (74.125.21.99) port 80 (#0)
>> > GET / HTTP/1.1
>> > Host: www.google.com
>> > User-Agent: curl/7.43.0
>> > Accept: */*
>> >
>> < HTTP/1.1 200 OK
>> <snip>
>> *real    1m6.523s (1 minute connect time)*
>>
>>
>> From Datacenter:
>> [dingy01.ny2:~] $ time curl -4 -vv http://www.google.com
>> * About to connect() to www.google.com port 80 (#0)
>> *   Trying 216.58.193.132...
>> * Connected to www.google.com (216.58.193.132) port 80 (#0)
>> > GET / HTTP/1.1
>> > User-Agent: curl/7.29.0
>> > Host: www.google.com
>> > Accept: */*
>> >
>> < HTTP/1.1 200 OK
>> <snip>
>> *real    1m3.189s    (took 1 minute to connect)*
>>
>> [ding02.ny5:~] $ curl -4 -k https://accounts.google.com
>> *curl: (7) couldn't connect to host*
>> [ding02.ny5:~] $ curl -4 -k https://accounts.google.com
>> *curl: (7) couldn't connect to host*
>> [ding02.ny5:~] $ curl -4 -k https://accounts.google.com
>> <HTML>
>> <HEAD>
>> <TITLE>Moved Temporarily</TITLE>
>> </HEAD>
>> <BODY BGCOLOR="#FFFFFF" TEXT="#000000">
>> <H1>Moved Temporarily</H1>
>> The document has moved <A HREF="https://accounts.google.com/ManageAccount
>> ">here</A>.
>> </BODY>
>> </HTML>
>>
>>
>> ----------
>> Tim Korn
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20161115/85433a86/attachment.htm>