[VoiceOps] Tampa / Orlando Verizon Business Voice Services Outage

[Paul Timmins]

On Nov 20, 2012, at 22:32 , Gabriel Gunderson <gabe at gundy.org> wrote:
> 
> Hehe... here we go (just when you think you've planned for everything):
> 
> http://www2.tbo.com/news/breaking-news/2012/nov/20/2/verizon-phone-problems-blamed-on-rodents-ar-572335/


I am always astounded by how Verizon Business can be totally caught off guard by fiber cuts.  Are they really building their networks with unprotected capacity, or are they monitoring/designing their backbone so poorly that they are either affected by supposedly diverse paths traveling through common conduit, or are losing service because they don't monitor simplex operation on their network and don't dispatch until they lose the protect side of the ring too.

The perpeptual joke around here (and I'm recounting this from memory, as it was a few years back) was when we got an industry notification from Verizon (former Bell Atlantic) notifying CLECs of some sort of OC-192 outage in one of the eastern states saying:

"ring took several unexpected protection switches, lost service for 1 minute, but returned to service without error. No investigation performed."

Then the next day:
"OC-192 outage, between City X and City Y. No ETTR. Technicians and splicing crews are enroute."
And about 4 hours later:
"OC-192 outage cleared. Cable had fallen onto the roadway and vehicles were driving over it" (or similar "if we gave a crap yesterday this could have been prevented or fixed in the maintenance window" sort of reason)

-Paul