[Outages-discussion] [outages] Google Down?

[Jay Hennigan]

[Moved to -discussion]

On 12/10/19 14:01, Matt Hoppes via Outages wrote:
> Ok. But here’s where I’m confused. I know believe the Google issue we 
> experienced was related to the Internet2 issue (we peer with them).
>
> So where does the line get drawn?  Google WAS down for us, and 
> potentially others who route that traffic through Internet2.

Several lines got crossed IMNSHO.

* Your post was very non-specific. "As of 3:25pm Eastern we've lost 
access to all things Google." That's it. Who is "we" in terms of 
location, source AS, carrier, etc.?

* You did no independent troubleshooting before reporting it to the 
list. IPv4, IPv6, both? Is it DNS related - does Google resolve? Where 
does traceroute stop? Is the target AS/subnet in your routing table? The 
issue began at 3:25 Eastern and your post was at 3:30.

* You didn't try to determine the scope of the issue. Can you reach 
Google from your cell phone? (Hint: Turn off your local wi-fi first.) 
Even consumer-level tools like downdetector.com or downfor.io can be 
very useful. Do you run any graphing/monitoring tools? The problem 
wasn't Google, it was Internet2. A quick glance at a dashboard of 
traffic graphs to your transit peers might have shown a significant 
change in traffic to/from Internet2. This should be a clue.

Report to the list *AFTER* you've done your troubleshooting, not before. 
Be thorough in your report. Even if you can't reach Internet2 to verify 
(they're up to their necks in alarms), that's a very valid data point.

"There seems to be a problem with Internet2. Their helpdesk gives 
fast-busy and their status page is unreachable. We peer with them to get 
to Google which is no longer responsive. Traffic graphs to Internet2 
dropped suddenly at 20:25 UTC. We're located in BF, Egypt. Can anyone 
confirm?" -- That would be a useful post to outages.

The first step in troubleshooting anything is to gather facts. *Reading* 
the outages list is a way to gather facts. Basic observation of the 
nature and scope of the problem are others. *Posting* to the list may be 
a means of gathering additional facts confirming your theory, but only 
once you've actually developed a theory based on all of the other tools 
at your disposal. It can also be very useful in providing facts to the 
other subscribers, but garbage in results in garbage out.


-- 
Jay Hennigan - jay at west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV