[outages] Centurylink Fiber cut in Phoenix

Wed Jul 18 11:52:20 EDT 2018

Is anyone still seeing issues? The cut was restored, but we are still
seeing layer 3 issues.

On Wed, Jul 18, 2018, 10:48 AM Mike Bolitho via Outages <outages at outages.org>
wrote:

> *** EXTERNAL NOTES 18-Jul-2018 08:35:17 GMT
> Event Conclusion Summary
>
> Outage Start: July 17 2018 15:29 GMT
>
> Outage Stop: July 18 2018 06:55 GMT
>
>
>
> Root Cause: A fiber cut on a buried fiber cable in Phoenix, AZ was
> impacting unprotected services.
>
> Fix action: The fiber was spliced to restore.
>
> Summary: The Transport NOC reported that a suspected fiber cut in Phoenix,
> AZ was impacting unprotected transport services. Field Operations were
> engaged to investigate further and identified the fault condition 28.43
> miles from the site. Field Operations and the Outside Plant team dispatched
> to site and isolated a 96-count buried fiber cable that has been cut by
> boring activities occurring in the area. The cable was buried 14 feet deep
> and was within 25 feet of a railroad line. As such, the railroad line
> provider would not allow excavations activities to commence until all
> necessary shoring supplies were on site. Once the shoring was in place, the
> excavation team will began digging a trench to access the damaged fiber and
> perform the repairs. In addition, local utility providers were on site
> marking existing utility lines in the area to prevent impacts during the
> repairs The trench needed to be approximately 50 feet in length and at
> least 15 feet deep to access the fiber and raise it for splicing repairs to
> commence. Field Operations reported 10 of the 12 conduits had been
> destroyed. Excavation was completed, slack was pulled and the fiber
> prepped, tested, and then spliced. Services were confirmed restored by the
> Repair Center with impacted customers.
>
> This service impact has concluded; however, there may be additional
> analysis and discovery that occurs as the incident is reviewed by NOC
> management. Any available updates will be relayed upon event ticket closure
> within the next 72 hours. At that time, a customer satisfaction survey link
> may be available. We strive to provide thorough communications containing
> the available information during a service disruption. Please let us know
> if the updates you received during this event were satisfactory.
>
>
> - Mike Bolitho
>
> On Tue, Jul 17, 2018 at 6:10 PM, Murali Medikonda via Outages <
> outages at outages.org> wrote:
>
>> Curious if anyone has any further updates on this fiber-cut.. I
>> appreciate it
>>
>>
>> On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <
>> outages at outages.org> wrote:
>>
>>> Thanks, Joe. Below is the most recent update we received a bit ago:
>>>
>>> *** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event
>>>
>>> Field Operations have reported that the Construction team is en route to
>>> the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has
>>> been provided. In addition, local utility providers are currently on site
>>> marking existing utility lines in the area to prevent impacts during the
>>> repairs. Once on site, the Construction team will commence with installing
>>> the necessary shoring around the damage site. It is estimated that the
>>> shoring activities will be completed at 23:00 GMT and excavation activities
>>> will commence at that time. The Transport NOC has reported that there are
>>> currently no reroute opportunities. The next update will be relayed once
>>> the shoring has been completed or additional information becomes available.
>>>
>>> Regards,
>>>
>>> Cory
>>>
>>> On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <
>>> outages at outages.org> wrote:
>>>
>>>> We have  P2P link between Dallas and San Diego that went down. It is
>>>> also impacting our connectivity to Dallas over our other CL links.
>>>> Centurylink sent out this notification. Sending it out in case others are
>>>> having similar issues.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 07/17/2018 18:31:54 GMT -
>>>>
>>>> *** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE:
>>>> 14896924 - Event Field Operations have identified a 96-count buried fiber
>>>> cable that has been cut by boring activities occurring in the area. The
>>>> cable is buried 14 feet deep and is within 25 feet of a railroad line. As
>>>> such, the railroad line provider will not allow excavations activities to
>>>> commence until all necessary shoring supplies are on site.  The supplies
>>>> are being obtained at this time, however, the ETA to the site is not yet
>>>> known.  Once the shoring is in place, the excavation team will begin
>>>> digging a trench to access the damaged fiber and perform the repairs.  The
>>>> trench will need to be approximately 50 feet in length and at least 15 feet
>>>> deep to access the fiber and raise it for splicing repairs to commence.
>>>> The initial activities are expected to take approximately two hours. As
>>>> such the next update will be relayed at 20:30 GMT or as additional
>>>> information becomes available.  This issue has been escalated internally to
>>>> the Executive level proactively.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 07/17/2018 17:38:50 GMT -
>>>>
>>>> *** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE:
>>>> 14896924 - Event
>>>>
>>>> Field Operations and the OSPE team have arrived at the site and have
>>>> identified a location where they believe a buried fiber cable has been
>>>> damaged by construction activities occurring in the area. Additional
>>>> assessments are ongoing at this time.  A fiber splicing repair team has
>>>> been engaged to assist with the repairs once the damaged fiber has been
>>>> identified, however the ETA to the site is not yet known
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> 07/17/2018 16:42:52 GMT -
>>>>
>>>> *** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE:
>>>> 14896924 - Event
>>>>
>>>> Field Operations have performed light measurement readings from the
>>>> CenturyLink Gateway in Phoenix and have identified the fault condition
>>>> 28.43 miles from the site.  The Outside Plant Engineer (OSPE) has been
>>>> engaged to assist with further isolation.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Bobin Joseph
>>>>
>>>>
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>>
>>> _______________________________________________
>>> Outages mailing list
>>> Outages at outages.org
>>> https://puck.nether.net/mailman/listinfo/outages
>>>
>>>
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20180718/ea980e37/attachment.htm>