[outages] Centurylink Fiber cut in Phoenix

Wed Jul 18 13:05:29 EDT 2018

Our VPLS and P2P’s are back up and running fine. This is between TX and SoCal.

Thanks,
Bobin Joseph

From: Outages <outages-bounces at outages.org> On Behalf Of Cory Sell via Outages
Sent: Wednesday, July 18, 2018 8:52 AM
To: Mike Bolitho <mikebolitho at gmail.com>
Cc: outages at outages.org
Subject: Re: [outages] Centurylink Fiber cut in Phoenix

Is anyone still seeing issues? The cut was restored, but we are still seeing layer 3 issues.
On Wed, Jul 18, 2018, 10:48 AM Mike Bolitho via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
*** EXTERNAL NOTES 18-Jul-2018 08:35:17 GMT
Event Conclusion Summary

Outage Start: July 17 2018 15:29 GMT

Outage Stop: July 18 2018 06:55 GMT

Root Cause: A fiber cut on a buried fiber cable in Phoenix, AZ was impacting unprotected services.

Fix action: The fiber was spliced to restore.

Summary: The Transport NOC reported that a suspected fiber cut in Phoenix, AZ was impacting unprotected transport services. Field Operations were engaged to investigate further and identified the fault condition 28.43 miles from the site. Field Operations and the Outside Plant team dispatched to site and isolated a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable was buried 14 feet deep and was within 25 feet of a railroad line. As such, the railroad line provider would not allow excavations activities to commence until all necessary shoring supplies were on site. Once the shoring was in place, the excavation team will began digging a trench to access the damaged fiber and perform the repairs. In addition, local utility providers were on site marking existing utility lines in the area to prevent impacts during the repairs The trench needed to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. Field Operations reported 10 of the 12 conduits had been destroyed. Excavation was completed, slack was pulled and the fiber prepped, tested, and then spliced. Services were confirmed restored by the Repair Center with impacted customers.

This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory.

- Mike Bolitho

On Tue, Jul 17, 2018 at 6:10 PM, Murali Medikonda via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Curious if anyone has any further updates on this fiber-cut.. I appreciate it

On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
Thanks, Joe. Below is the most recent update we received a bit ago:

*** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event

Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available.

Regards,

Cory
On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
We have  P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links.  Centurylink sent out this notification. Sending it out in case others are having similar issues.

07/17/2018 18:31:54 GMT -

*** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site.  The supplies are being obtained at this time, however, the ETA to the site is not yet known.  Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs.  The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence.  The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available.  This issue has been escalated internally to the Executive level proactively.

07/17/2018 17:38:50 GMT -

*** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event

Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing at this time.  A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known

07/17/2018 16:42:52 GMT -

*** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event

Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site.  The Outside Plant Engineer (OSPE) has been engaged to assist with further isolation.

Thanks,
Bobin Joseph

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20180718/bbad28be/attachment.htm>