[outages] Godaddy / Premium DNS outage?

[Tony]

Mike,

My two cents here, anyone in Operations who does not understand that s$%t
happens and that any company can be taken down by a set of circumstances
that they were not prepared for either does not really "work" in Operations
or is just harping on this to take a shot at GoDaddy. I have seen "perfect
storm" circumstances that set off long term outages that no one could have
predicted. I respect that fact that you spoke out on this list and
appreciate  what you could share and based on past and present experience
believe that your teams did a good the public announcement and your
comments on this list.

Thank You

Tony Pasqualini
Ops Director

On Fri, Oct 5, 2012 at 11:49 AM, <mdob at godaddy.com> wrote:

> Obviously the explanation of the incident had to be consumed by the
> general public, however we encountered an unknown bug that was found which
> started the domino effect. Aside from this group, that level of detail
> wouldn't be understood by a majority of the recipients.
>
> With that said, please feel free to take this off list with Jason or
> Myself.
>
> Mike Dob
> Manager, Network Engineering
>
>
>
> -------- Original Message--------
> Subject:: Re: [outages] Godaddy / Premium DNS outage?
> From: ryanL <ryan.landry at gmail.com>
> Date: Oct 5, 2012 11:32 AM
> To:: "Jason LeBlanc" <jleblanc at godaddy.com>
> CC: outages at outages.org
>
> We encountered a new bug when we executed our change.
>
>
> -------- Original Message--------
> Subject:: Re: [outages] Godaddy / Premium DNS outage?
> From: ryanL <ryan.landry at gmail.com>
> Date: Oct 5, 2012 11:32 AM
> To:: "Jason LeBlanc" <jleblanc at godaddy.com>
> CC: outages at outages.org
>
> so, redistributed eBGP into IGP? how does one 210x their routing table? i
> appreciate the openness, but it doesn't sound like a "perfect storm" to me.
> it sounds like someone screwed up.
>
>
> On Fri, Oct 5, 2012 at 10:18 AM, Jason LeBlanc <jleblanc at godaddy.com>
> wrote:
>
> > I made a commitment to get everyone on here the RFO.  Its presented in
> > layman's terms:
> >
> >
> http://inside.godaddy.com/inside-story-happened-godaddy-com-sept-10-2012/
> >
> > Regards,
> >
> > Jason LeBlanc
> > Network Engineering
> > (480)505-8800 x7202
> >
> >
> > Please contact my direct supervisor mdob at godaddy.com with any feedback.
> >
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20121005/d52b49b0/attachment.htm>