<div><div dir="auto">When I dealt with this a few employers ago it meant running some reports from the proxy logs hunting who had the most traffic to google and one of the top 10 had a RealPlayer browser plugin that was misbehaving as Damian noted is a popular reason.</div><div dir="auto"><br></div><div dir="auto">That or someone playing around with scripting requests :)</div><div dir="auto"><br></div><div dir="auto"><br></div><div dir="auto">Zach</div></div><div dir="auto"><br></div><div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jun 29, 2020 at 1:25 AM Chapman, Brad (NBCUniversal) <<a href="mailto:Brad.Chapman@nbcuni.com">Brad.Chapman@nbcuni.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;padding-left:1ex;border-left-color:rgb(204,204,204)">
<div lang="EN-US" link="blue" vlink="purple">
<div class="m_7225003523452863785WordSection1">
<p class="MsoNormal">It’s a network that has no direct route to the Internet and all web traffic *<b>must</b>* traverse the proxy.
</p></div></div><div lang="EN-US" link="blue" vlink="purple"><div class="m_7225003523452863785WordSection1"><p class="MsoNormal"><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">-Brad<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div style="border-style:solid none none;border-top-width:1pt;padding:3pt 0in 0in;border-top-color:rgb(181,196,223)">
<p class="MsoNormal"><b><span style="font-size:12pt;color:black">From: </span></b><span style="font-size:12pt;color:black">Damian Menscher <<a href="mailto:damian@google.com" target="_blank">damian@google.com</a>><br>
<b>Date: </b>Sunday, June 28, 2020 at 10:05 PM<br>
<b>To: </b>"Chapman, Brad (NBCUniversal)" <<a href="mailto:Brad.Chapman@nbcuni.com" target="_blank">Brad.Chapman@nbcuni.com</a>><br>
<b>Cc: </b>"<a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a>" <<a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a>><br>
<b>Subject: </b>Re: [EXTERNAL] Re: [Outages-discussion] Question about the Google “sorry” page...<u></u><u></u></span></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Two cautions regarding proxies:<u></u><u></u></p>
<div>
<p class="MsoNormal"> - as mentioned before, make sure you don't have an open proxy, which might be abused<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - if you're proxying only some traffic (eg, for content filtering, etc), then be sure all Google traffic gets proxied out the same IP. We sometimes see weirdness when some requests go through the proxy, but other requests go directly
from the (home) IP. This can cause problems, for example the captcha exemption may fail due to the IP mis-match.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Damian<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jun 28, 2020 at 9:55 PM Chapman, Brad (NBCUniversal) <<a href="mailto:Brad.Chapman@nbcuni.com" target="_blank">Brad.Chapman@nbcuni.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in;border-left-color:rgb(204,204,204)">
<div>
<p class="MsoNormal">Interesting; thanks. <u></u><u></u></p>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">Would you expect to see this behavior in an environment where a proxy server is used to funnel traffic to the Internet and clients have to use a PAC file or WPAD?<u></u><u></u></p>
<div>
<p class="MsoNormal">—Sent from my iPhone<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><br>
<br>
<u></u><u></u></p>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<p class="MsoNormal" style="margin-bottom:12pt">On Jun 28, 2020, at 9:34 PM, Damian Menscher <<a href="mailto:damian@google.com" target="_blank">damian@google.com</a>> wrote:<u></u><u></u></p>
</blockquote>
</div>
<blockquote style="margin-top:5pt;margin-bottom:5pt">
<div>
<div>
<div>
<p class="MsoNormal">Blocking occurs when automated searching is detected, not simply due to the total volume of requests from a single IP. As such, there is no option for an exception.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">To "solve" this, we recommend you minimize the number of users sharing an IP. The easiest method is with IPv6, since then each user can have their own /64 (our abuse systems don't look deeper than that). If you're stuck with IPv4, separate
your corporate-managed machines from the guest wifi (which is harder to control), and try to give different groups of users their own NAT IP (by building or floor, etc). That way when there's a problem you'll have fewer users impacted, and a smaller list
of suspects.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">If you want to start digging into the reasons why your IP might have been blocked, the most common reasons for getting blocked (mostly for websearch) include (in no particular order):<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - malware that proxies abuse for criminals<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - browser extensions that automate searching<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - misconfigured browsers that have anomalous behavior<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - corporate proxies that are open for abuse<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"> - users installing "P2P VPN" software, which is also abused<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">Damian<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">-- <u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Damian Menscher :: Security Reliability Engineer :: Google :: AS15169<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Sun, Jun 28, 2020 at 4:57 PM Chapman, Brad (NBCUniversal) <<a href="mailto:Brad.Chapman@nbcuni.com" target="_blank">Brad.Chapman@nbcuni.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-style:none none none solid;border-left-width:1pt;padding:0in 0in 0in 6pt;margin-left:4.8pt;margin-right:0in;border-left-color:rgb(204,204,204)">
<p class="MsoNormal">Greetings Outages-Discussion,<br>
<br>
I hope you are all having a pleasant Sunday afternoon / evening with no P1 / SevA / 4-alarm fires caused by a violation of Read-only Friday.
<br>
<br>
Given the number of sysadmins and telecom / network engineers on this list, I am guessing that we have seen (or been asked to explain) the Google “Sorry” page.<br>
<br>
Occasionally, our company gets a burst of calls about this issue, until the lockout expires on Google’s side. We manage >50,000 computers so even short lockouts can generate dozens of calls.
<br>
<br>
Has anyone ever approached Google’s NOC team to request an exemption from the Sorry page for their busiest external IP addresses? Or, if not a blanket exemption, to request an increase in the threshold before it is tripped?<br>
<br>
Hope you’re all staying safe.<br>
<br>
Cheers,<br>
Brad Chapman<br>
NBCUniversal<br>
<br>
—Sent from my iPhone<br>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br>
<a href="https://urldefense.com/v3/__https:/puck.nether.net/mailman/listinfo/outages-discussion__;!!PIZeeW5wscynRQ!-T5SokgIYLbWPeqRO4boP4fHxQbHaOHVW5G6FNDQ4sI2cVgFNtCDeAvOwaP5eN4PNg$" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><u></u><u></u></p>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</blockquote>
</div>
</div>
</div>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br>
</blockquote></div></div>