[outages] Level 3 Fiber Cut in Cincinnati, OH

Thu Feb 4 10:21:34 EST 2016

Yup, I've got a ticket open, linked to the master ticket as well...here's
the latest...which I read as the notional equivalent of a bunch of folks
standing around scratching their heads saying, "But it should be working!"
 ;)

In all seriousness, I'm sure they're working diligently, and I've got
redundant connectivity, so I'm not dead in the water and can laugh about
it a bit.

*** CASCADED EXTERNAL NOTES 04-Feb-2016 14:25:32 GMT From CASE: 10329498 -
Event
Level 3 Field Services are dispatching to a site closer to the damage area
to further isolate.  The local provider has a temporary fiber ran around
the damage.  They are currently investigating the splice plan for any
discrepancies. The splice crew remains onsite awaiting the investigation
of the splice plan.

On Thu, February 4, 2016 09:25, Hyoun Kim via Outages wrote:
> Yep.  We're still down here too.  Sorry for the lack of updates.  Was
> putting out other fires, but you're the first to respond in relation to
> this one.  Here is the latest as of 1 hour ago and the past couple prior:
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 13:25:53 GMT From CASE: 10329498
> -
> Event Field Services have arrived on site to test the fiber to assist the
> local provider with further isolating the issue.
>
>
>
> 02/04/2016 12:28:09 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 12:27:45 GMT From CASE: 10329498
> -
> Event
>
>
> Field Services have been engaged to dispatch and test the fiber to assist
>  the local provider with further isolating the issue.  An ETA of 13:30
> GMT
> has been provided for Field Services to arrive on site
>
>
>
>
>
> 02/04/2016 11:47:45 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 11:47:21 GMT From CASE: 10329498
> -
> Event
>
>
> The Transport NOC has informed that further investigations have revealed
> alarms that have not cleared and some customers remain impacted.  The
> local provider is being contacted for further investigations.
>
>
>
>
>
> 02/04/2016 11:30:30 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 11:30:04 GMT From CASE: 10329498
> -
> Event
>
>
> The Transport NOC has verified that the local provider has completed
> splicing and alarms have cleared.  The Technical Service Center has been
> engaged to verify service restoral.
>
>
>
>
>
> 02/04/2016 10:52:10 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 10:51:47 GMT From CASE: 10329498
> -
> Event
>
>
> The local provider has informed that splicing is taking longer than
> expected.  Splicing is anticipated to be complete within the next 20
> minutes.
>
>
>
>
>
> 02/04/2016 10:42:25 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 10:42:02 GMT From CASE: 10329498
> -
> Event
>
>
> The Transport NOC has informed that a few alarms remain.  The local
> provider is being contacted for an update.
>
>
>
>
>
> 02/04/2016 10:20:04 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 10:19:33 GMT From CASE: 10329498
> -
> Event
>
>
> Splicing efforts by the local provider continue.  An ETTR of 10:30 GMT
> remains unchanged at this time.
>
>
>
>
>
> 02/04/2016 08:42:45 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 08:42:25 GMT From CASE: 10329498
> -
> Event
>
>
> Splicing efforts continue to restore impacted customers. An ETTR of 10:30
>  GMT remains unchanged at this time.
>
>
>
>
>
>
> 02/04/2016 06:33:29 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 06:33:07 GMT From CASE: 10329498
> -
> Event
>
>
> As splicing efforts continue by the local provider, some alarms are
> beginning to clear.  An updated ETTR is currently unavailable.
>
>
>
>
>
> 02/04/2016 05:21:57 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 05:21:29 GMT From CASE: 10329498
> -
> Event
>
>
> Splicing remains ongoing.  An ETTR of 06:30 GMT remains unchanged at this
>  time.
>
>
>
>
>
> 02/04/2016 04:29:41 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 04:29:11 GMT From CASE: 10329498
> -
> Event
>
>
> Splicing is still ongoing at this time, and alarms are expected to start
> clearing at approximately 06:30 GMT.
>
>
>
>
>
> 02/04/2016 02:37:52 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 02:37:27 GMT From CASE: 10329498
> -
> Event
>
>
> The local provider advised that splicing is ongoing at this time, an
> estimated restoral time has not been provided.
>
>
>
>
>
> 02/04/2016 01:39:34 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 01:39:10 GMT From CASE: 10329498
> -
> Event
>
>
> Correction: The local provider advised that boring is taking longer than
> expected. In an effort to expedite service restoral Field Services will
> splice a temporary cable around the damaged area. Splicing is expected to
>  commence within the hour.
>
>
>
>
>
> 02/04/2016 01:02:50 GMT -
>
>
> *** CASCADED EXTERNAL NOTES 04-Feb-2016 01:02:22 GMT From CASE: 10329498
> -
> Event
>
>
> Field Services have advised that boring is taking longer than expected.
> In
> an effort to expedite service restoral Field Services will splice a
> temporary cable around the damaged area. Splicing is expected to commence
>  within the hour.
>
> On Thu, Feb 4, 2016 at 7:47 AM, Jeff McAdams via Outages <
> outages at outages.org> wrote:
>
>> Just a quick update on this.  We're approaching 24 hours and still some
>>  customers (such as myself) down.
>>
>>
>>
>> On Wed, February 3, 2016 11:51, Hyoun Kim via Outages wrote:
>>
>>> Update:
>>>
>>>
>>>
>>> *** CASCADED EXTERNAL NOTES 03-Feb-2016 16:49:03 GMT From CASE:
>>> 10329498
>>> -
>>> Event The local provider has located a 120 count cable that was
>>> damaged
>> by
>>> a contractor using boring equipment to install a new gas line in the
>>> area. The local provider has engaged construction and splicing crews
>>> to assist with the repairs; however no ETA or ETTR is currently
>>> available.
>>>
>>> On Wed, Feb 3, 2016 at 10:03 AM, Hyoun Kim <faluzure at gmail.com>
>>> wrote:
>>>
>>>
>>>
>>>> My circuit from Nashville, TN to Chambersburg, PA went down around
>>>> 8:40
>>>> AM
>>>> CST.  When I opened a ticket w/ Level 3, they mentioned there is a
>>>> major event that occurred exactly at 8:40 AM on their end as well.
>>>>
>>>> The automated email system reported:
>>>>
>>>>
>>>>
>>>> 02/03/2016 15:24:42 GMT - The Transport NOC has reported that
>>>> suspected fiber cut in Cincinnati, OH is impacting unprotected
>>>> Transport
>>>> services. Field Services have been engaged to investigate further.
>>>> Please be advised
>>>> that updates for this event will be relayed hourly unless otherwise
>>>> noted.
>>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>>
>>
>>
>> --
>> Jeff
>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>

-- 
Jeff