[Outages-discussion] [outages] Verizon west coast outage?

[Jeremy Chadwick]

On Fri, Oct 02, 2009 at 06:16:01PM -0500, Larry Sheldon wrote:
> Jeremy Chadwick wrote (on "outages"):
> >On Fri, Oct 02, 2009 at 04:55:30PM -0500, Forestal, Andre Jr. wrote:
> >>When routers have problems, they are designed to report that they are
> >>sick. Internet traffic is rerouted to adjacent routers automatically and
> >>sent around the trouble spot. In this case, that didn't happen. The
> >>router went into a hung state and did not appear to the rest of the
> >>network as though it was having problems.
> >
> >It appears BFD is a protocol foreign to Verizon.  And why didn't hold
> >timers expire on associated iBGP peers?  Questions, questions...
> 
> I was thinking, in the old days (I worked "Telegraph", the "L", "N",
> and "R" carrier) we would have all been fired if we didn't have a
> restoration route up-and-running in about 4 minutes.
> 
> 40 minutes?!!?  Are you sh*tting me?

What bothers me even more than the outage duration is their implication
that they took the same router which just misbehaved and *put it back
into the network mesh* so they could "watch it".  This adds further fuel
to the "where's the redundancy?" fire.

-- 
| Jeremy Chadwick                                   jdc at parodius.com |
| Parodius Networking                       http://www.parodius.com/ |
| UNIX Systems Administrator                  Mountain View, CA, USA |
| Making life hard for others since 1977.              PGP: 4BD6C0CB |