[Outages-discussion] Question about the Google “sorry” page...

[Matthew Petach]

At $PREVIOUS_EMPLOYER, we had good success with increasing the size of our
external NAT pools, spreading the requests over a larger swath of IP space.

Have you looked at the flow stats coming from the external side of your NAT
devices to see how many connections per second each IP in the pool is
making to Google, and correlating that with which client machines on the
inside are making the requests, to make sure there's no
inappropriate software running that may be generating more requests than
you would otherwise anticipate?

Sometimes the sorry page is a reasonable canary in the coal mine,
indicating someone's doing something internally that they shouldn't really
be doing.

Best of luck!

Matt


On Sun, Jun 28, 2020 at 4:55 PM Chapman, Brad (NBCUniversal) <
Brad.Chapman at nbcuni.com> wrote:

> Greetings Outages-Discussion,
>
> I hope you are all having a pleasant Sunday afternoon / evening with no P1
> / SevA / 4-alarm fires caused by a violation of Read-only Friday.
>
> Given the number of sysadmins and telecom / network engineers on this
> list, I am guessing that we have seen (or been asked to explain) the Google
> “Sorry” page.
>
> Occasionally, our company gets a burst of calls about this issue, until
> the lockout expires on Google’s side.   We manage >50,000 computers so even
> short lockouts can generate dozens of calls.
>
> Has anyone ever approached Google’s NOC team to request an exemption from
> the Sorry page for their busiest external IP addresses? Or, if not a
> blanket exemption, to request an increase in the threshold before it is
> tripped?
>
> Hope you’re all staying safe.
>
> Cheers,
> Brad Chapman
> NBCUniversal
>
> —Sent from my iPhone
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20200628/47b77e17/attachment.htm>