[outages] L3/CenturyLink - Cuts in NC and FL

[Alex Lembesis]

Rough week...

We have two diverse paths on unprotected services between PA and FL that are currently down.  CenturyLink is reporting fiber cuts on both.


First Cut....



03/22/2019 10:45:28 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 10:42:19 GMT From CASE: 16079534 - Event Splice repair has commenced at the second damage site and is ongoing at this time. The field crew is experiencing some delays due to water in the excavation area that must be routinely pumped out of the work area for fiber repairs to be made. An ETR remains unavailable.

03/22/2019 09:32:10 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 09:28:55 GMT From CASE: 16079534 - Event

Splicing is complete at the first point of damage. Approximately 1000 ft of replacement cable is currently being pulled into a manhole and laid into proper placement for splice repair to commence at the next damage site.

03/22/2019 08:22:38 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 08:19:27 GMT From CASE: 16079534 - Event

Splice repair of the affected fiber is ongoing at this time. An ETR remains unavailable.

03/22/2019 07:18:03 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 07:14:49 GMT From CASE: 16079534 - Event

Repair of the damaged fiber span has commenced. An estimated time of restoral (ETR) is currently unavailable.

03/22/2019 06:15:28 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 06:12:18 GMT From CASE: 16079534 - Event

The local providers field crew has exposed the affected span. The 98-count fiber was found to be damaged in multiple places. Analysis of the damage is currently underway to determine the best course of repair.

03/22/2019 05:01:02 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 04:57:51 GMT From CASE: 16079534 - Event

Excavation efforts to expose the affected span are ongoing at this time.

03/22/2019 04:01:26 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 03:58:05 GMT From CASE: 16079534 - Event

The local provider reports that the construction crew continues to excavate the terrain. Personnel have advised that fiber span is deeper than expected which is prolonging the exposure of the fiber cable.

03/22/2019 03:05:49 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 03:02:29 GMT From CASE: 16079534 - Event

The local provider reported that construction crews continue to excavate the terrain. The Transport NOC has escalated with the local provider as an effort to obtain more information.

03/22/2019 02:06:28 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 02:03:08 GMT From CASE: 16079534 - Event

The Transport NOC has been engaged the local provider for a status update. As such, more information will be provided momentarily.

03/22/2019 01:12:38 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 01:09:11 GMT From CASE: 16079534 - Event

The local provider reports that a construction crew is on site and is currently excavating both ends of the fiber damage. Once excavation is completed, two new splice enclosures along with replacement cabling will be installed.

03/22/2019 00:29:42 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 00:26:23 GMT From CASE: 16079534 - Event

The local provider reports that the ETA of the splicing crew remains unavailable at this time.

03/21/2019 23:42:24 GMT -

*** CASCADED EXTERNAL NOTES 21-Mar-2019 23:39:16 GMT From CASE: 16079534 - Event

The Transport NOC reports that the local provider has engaged and dispatched a splicing crew to the fault location to begin the repairs. However, an ETA for splicing personnel to be on-site is currently unavailable.

03/21/2019 22:45:37 GMT -

*** CASCADED EXTERNAL NOTES 21-Mar-2019 22:42:25 GMT From CASE: 16079534 - Event

The local provider reports that the field crew arrive at the site in Dunn, NC and identified a fiber cut that was caused a construction crew who was performing a directional boring operation. Onsite personnel at this time are currently assessing the fiber damage further to unveil the extent of the repair required.

03/21/2019 21:48:14 GMT -

*** CASCADED EXTERNAL NOTES 21-Mar-2019 21:45:08 GMT From CASE: 16079534 - Event

The local provider reports that additional fiber testing has further isolated the fault towards Dunn, NC. As such, the local providers field crew has dispatch to Dunn, NC to continue isolation; however, an ETA is currently unavailable.

03/21/2019 20:47:47 GMT -

*** CASCADED EXTERNAL NOTES 21-Mar-2019 20:44:30 GMT From CASE: 16079534 - Event

The local provider reports that the field crew has arrived at a site located in Benson, NC and additional light level fiber testing is underway.

03/21/2019 19:46:12 GMT -

*** CASCADED EXTERNAL NOTES 21-Mar-2019 19:43:03 GMT From CASE: 16079534 - Event

The Transport NOC reports that the local provider has isolated a potential fiber cut on their network. The local provider has engaged and dispatched field crews to suspected fault location for onsite isolation; however, an ETA is currently unavailable.

03/21/2019 19:02:30 GMT -

*** CASCADED EXTERNAL NOTES 21-Mar-2019 18:47:59 GMT From CASE: 16079534 - Event

The Transport NOC was engaged, and initial investigation suggested a fiber cut was impacting unprotected services. As such, Field Operations were engaged, and personnel have completed initial light level fiber testing. The test results isolated a fault approximately 14,000ft from the testing facility towards a local provider's network. At this time, the Transport NOC has engaged the associated local provider to investigate their network for any potential faults.





Second Cut...



03/22/2019 14:32:01 GMT -

*** CASCADED EXTERNAL NOTES 22-Mar-2019 14:21:09 GMT From CASE: 16085394 - Event Field Operations tested the fiber and it appears that the issue a partial fiber damage 60 miles from the testing site. Field Operations are enroute to the other side of the span to test from that direction and investigate for good spare fibers to migrate services. Field Operations gave an estimated time of arrival to the other site of 17:00 GMT. The next update will be provided when Field Operations arrives on site or more information becomes available.

03/22/2019 13:49:08 GMT - 03/22/2019 13:17:19 GMT - The Transport NOC engaged Field Operations who traveled to the site and is setting up to begin performing light level measurements to determine the location of the fiber issue.

03/22/2019 12:56:16 GMT - On March 22, 2019, at 12:37 GMT, CenturyLink identified a service impact between Bradenton and Tampa, FL. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.




This message is intended solely for the designated recipient(s). It may contain confidential or proprietary information and may be subject to attorney-client privilege or other confidentiality protections. If you are not a designated recipient you may not review, copy or distribute this message. If you receive this in error, please notify the sender by reply e-mail and delete this message. Thank you.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20190322/d6e80b00/attachment.htm>