[outages] Godaddy / Premium DNS outage?

[Naveen Nathan]

> On 09/11/2012 01:53 PM, francis.daigneault-t/WHVqD6nWw at public.gmane.org wrote:
> > It's the easy answer to avoid admitting they where DDOS, when you 
> > can't explain something or you don't want to admit something blame 
> > it on "corruption" The outages start and stop at the time predicted
> > by the hacker who "admit" it.. how could that happen.. coincidence
> > ?
> 
> What if 'corruption' in this case meant 'tampering'?  Is it possible
> that their routing tables were corrupted either by injection or
> someone compromising a router and reconfiguring it?
> 
> It's been a few years since I last heard about route injection (at
> DefCon, wasn't it?) so I don't know whether or not it's widely
> possible, or at least plausible in the case of GoDaddy.

I don't think this is the case. I've done my own (rudimentary)
analysis, and their routers were unaffected as far as the BGP DFZ
is concerned (check http://bgplay.routeviews.org or http://bgp.potaroo.net
against their route announcements and you won't see any unusual
activity). Normally, NANOG tends to be quick on routing issues if
they affect the Internet, but everyone seemed to be more focused
on the DNS issues at the time. Some actually claiming connectivity
(albeit intermittent) by IP.

Going from what they've said, chatter on the Internet (and on this
list), and the conjectures this is what I've concluded:

If it were a BGP issue, it would have to be an incredibly nasty iBGP problem. The nature of the problem would've had to shown intermittent connectivity as there are posted traceroutes (under this subject thread) during the outage showing reachability to GoDaddy. Having said that, to take 5-12 hours to recover doesn't really sound accurate at all, even if it was a misconfiguration. Diagnostics and active monitoring software which undoubtedly they have in place can usually pinpoint the problem. But supposing they don't have their act together at all, it's still a more convenient excuse than the alternatives.

If it was a security breach, this will surmount to more negative publicity, media outlets spreading information, and basically scaring off their customer base. They also participate in the browser CA oligarchy, so I imagine people would think twice before purchasing an SSL certificate from GoDaddy.

If it was a DoS/DDoS attack, they're basically admitting their capacity and equipment is inadequate. Competitors will target their customer base claiming they're DDoS resilient (which for the most part is BS marketing), and they will probably lose a hearty chunk of customers.

Lastly, GoDaddy may have no understanding or comprehension of the events that occured during the outage, outside of what we already know. This could be plausible assuming they weren't retaining logs of all their network devices, or did not have enough information (such as storing netflow data during a DoS/DDoS).

I'm not going to argue that spreading disinformation is ethical or unethical given the circumstances, but I assume it would be the safest position for the company. Also given that two of their employees, in this list, are quick to promise they wouldn't spread disinformation, it just makes everything seem more suspicious. Also, every announcement is quick to denounce that the outage is due to a security breach or DoS/DDoS attack. This just seems to be a tactic to save face, and to avoid blowback from media outlets that could pick it up, sensationalize it, and result in more negative press.

At least by claiming it was their fault either intentionally or unintentionally, they can at least regain customer confidence and goodwill, dish out a bunch of service credit, and everyone can forget about it. But, from all the discussions so far, and what they've actually said, nothing adds up. In any case, the best bet is to wait for more information.

- Naveen