[VoiceOps] Tampa / Orlando Verizon Business Voice Services Outage

[Paul Timmins]

On Nov 20, 2012, at 23:11 , Paul Timmins <paul at timmins.net> wrote:

> 
> On Nov 20, 2012, at 22:32 , Gabriel Gunderson <gabe at gundy.org> wrote:
>> 
>> Hehe... here we go (just when you think you've planned for everything):
>> 
>> http://www2.tbo.com/news/breaking-news/2012/nov/20/2/verizon-phone-problems-blamed-on-rodents-ar-572335/
> 
> 
> I am always astounded by how Verizon Business can be totally caught off guard by fiber cuts.  Are they really building their networks with unprotected capacity, or are they monitoring/designing their backbone so poorly that they are either affected by supposedly diverse paths traveling through common conduit, or are losing service because they don't monitor simplex operation on their network and don't dispatch until they lose the protect side of the ring too.
> 
> The perpeptual joke around here (and I'm recounting this from memory, as it was a few years back) was when we got an industry notification from Verizon (former Bell Atlantic) notifying CLECs of some sort of OC-192 outage in one of the eastern states saying:
> 
> "ring took several unexpected protection switches, lost service for 1 minute, but returned to service without error. No investigation performed."
> 
> Then the next day:
> "OC-192 outage, between City X and City Y. No ETTR. Technicians and splicing crews are enroute."
> And about 4 hours later:
> "OC-192 outage cleared. Cable had fallen onto the roadway and vehicles were driving over it" (or similar "if we gave a crap yesterday this could have been prevented or fixed in the maintenance window" sort of reason)

Hah, actually, found it. It was actually an OC-192 ring going into simplex in Pennsylvania. Certain details removed to protect the guilty:

Email on 11/18/2008:

[11/18/2008 9:10:50 AM CT] 
Cause of Event:  Unknown. 
Corrective Action: The trouble cleared prior to intervention. 
    
[11/18/2008 6:19:50 AM CT]  
OC192 in simplex.  Investigation in progress.  There is no estimated time of restoral.  This is a non-service affecting event. 

Later that evening:

[11/18/2008 9:23:58 PM CT] 
OC192 in simplex.  Investigation in progress.  There is no estimated time of restoral.  This is a non-service affecting event.