[Outages-discussion] [EXTERNAL] Re: Question about the Google “sorry” page...

[Chapman, Brad (NBCUniversal)]

It’s a network that has no direct route to the Internet and all web traffic *must* traverse the proxy.

-Brad

From: Damian Menscher <damian at google.com>
Date: Sunday, June 28, 2020 at 10:05 PM
To: "Chapman, Brad (NBCUniversal)" <Brad.Chapman at nbcuni.com>
Cc: "outages-discussion at outages.org" <outages-discussion at outages.org>
Subject: Re: [EXTERNAL] Re: [Outages-discussion] Question about the Google “sorry” page...

Two cautions regarding proxies:
  - as mentioned before, make sure you don't have an open proxy, which might be abused
  - if you're proxying only some traffic (eg, for content filtering, etc), then be sure all Google traffic gets proxied out the same IP.  We sometimes see weirdness when some requests go through the proxy, but other requests go directly from the (home) IP.  This can cause problems, for example the captcha exemption may fail due to the IP mis-match.

Damian

On Sun, Jun 28, 2020 at 9:55 PM Chapman, Brad (NBCUniversal) <Brad.Chapman at nbcuni.com<mailto:Brad.Chapman at nbcuni.com>> wrote:
Interesting; thanks.

Would you expect to see this behavior in an environment where a proxy server is used to funnel traffic to the Internet and clients have to use a PAC file or WPAD?
—Sent from my iPhone


On Jun 28, 2020, at 9:34 PM, Damian Menscher <damian at google.com<mailto:damian at google.com>> wrote:
Blocking occurs when automated searching is detected, not simply due to the total volume of requests from a single IP.  As such, there is no option for an exception.

To "solve" this, we recommend you minimize the number of users sharing an IP.  The easiest method is with IPv6, since then each user can have their own /64 (our abuse systems don't look deeper than that).  If you're stuck with IPv4, separate your corporate-managed machines from the guest wifi (which is harder to control), and try to give different groups of users their own NAT IP (by building or floor, etc).  That way when there's a problem you'll have fewer users impacted, and a smaller list of suspects.

If you want to start digging into the reasons why your IP might have been blocked, the most common reasons for getting blocked (mostly for websearch) include (in no particular order):
  - malware that proxies abuse for criminals
  - browser extensions that automate searching
  - misconfigured browsers that have anomalous behavior
  - corporate proxies that are open for abuse
  - users installing "P2P VPN" software, which is also abused

Damian
--
Damian Menscher :: Security Reliability Engineer :: Google :: AS15169

On Sun, Jun 28, 2020 at 4:57 PM Chapman, Brad (NBCUniversal) <Brad.Chapman at nbcuni.com<mailto:Brad.Chapman at nbcuni.com>> wrote:
Greetings Outages-Discussion,

I hope you are all having a pleasant Sunday afternoon / evening with no P1 / SevA / 4-alarm fires caused by a violation of Read-only Friday.

Given the number of sysadmins and telecom / network engineers on this list, I am guessing that we have seen (or been asked to explain) the Google “Sorry” page.

Occasionally, our company gets a burst of calls about this issue, until the lockout expires on Google’s side.   We manage >50,000 computers so even short lockouts can generate dozens of calls.

Has anyone ever approached Google’s NOC team to request an exemption from the Sorry page for their busiest external IP addresses? Or, if not a blanket exemption, to request an increase in the threshold before it is tripped?

Hope you’re all staying safe.

Cheers,
Brad Chapman
NBCUniversal

—Sent from my iPhone
_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org<mailto:Outages-discussion at outages.org>
https://puck.nether.net/mailman/listinfo/outages-discussion<https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages-discussion__;!!PIZeeW5wscynRQ!-T5SokgIYLbWPeqRO4boP4fHxQbHaOHVW5G6FNDQ4sI2cVgFNtCDeAvOwaP5eN4PNg$>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20200629/e30c374c/attachment-0001.htm>