[outages] Level 3 down in Atlanta

[William R. Lorenz]

On Thu, 22 Oct 2009, George Herbert wrote:

> On Thu, Oct 22, 2009 at 7:03 PM, Jay R. Ashworth <jra at baylink.com> wrote:

>>> Level 3 has a single router or switch handling packets at a major POP? 
>>> I doubt this, but the outage is confirmation something bad happened. 
>>> That said: where's the redundancy, and why didn't it kick in?

>> Oh; you're *always* asking that.

> The RFO that went out somewhat after he asked that was more useful... 
> N=2 redundancy was in place.  However, when primary had hardware 
> failure, secondary had (unknown / unstated) software, config, or 
> hardware failure that hadn't been detected or checked, and it didn't

I'm not in Atlanta but from what was mentioned on the list, it was a soft 
failure which is why the other routers didn't failover w/ HSRP or whatnot:

   https://puck.nether.net/pipermail/outages/2009-October/001607.html
   https://puck.nether.net/pipermail/outages/2009-October/001608.html

The real question should be why nobody powered down that device the first 
or second time, considering it didn't failover properly the first time.

   https://puck.nether.net/pipermail/outages/2009-October/001600.html
   https://puck.nether.net/pipermail/outages/2009-October/001611.html

These things happen from time-to-time -- that's the Internet.

-- 
William R. Lorenz