Is anyone still seeing issues? The cut was restored, but we are still seeing layer 3 issues. <br><br><div class="gmail_quote"><div dir="ltr">On Wed, Jul 18, 2018, 10:48 AM Mike Bolitho via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div class="gmail_default"><div class="gmail_default"><font face="arial, helvetica, sans-serif">*** EXTERNAL NOTES 18-Jul-2018 08:35:17 GMT</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Event Conclusion Summary</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Outage Start: July 17 2018 15:29 GMT </font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Outage Stop: July 18 2018 06:55 GMT</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Root Cause: A fiber cut on a buried fiber cable in Phoenix, AZ was impacting unprotected services. </font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Fix action: The fiber was spliced to restore. </font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">Summary: The Transport NOC reported that a suspected fiber cut in Phoenix, AZ was impacting unprotected transport services. Field Operations were engaged to investigate further and identified the fault condition 28.43 miles from the site. Field Operations and the Outside Plant team dispatched to site and isolated a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable was buried 14 feet deep and was within 25 feet of a railroad line. As such, the railroad line provider would not allow excavations activities to commence until all necessary shoring supplies were on site. Once the shoring was in place, the excavation team will began digging a trench to access the damaged fiber and perform the repairs. In addition, local utility providers were on site marking existing utility lines in the area to prevent impacts during the repairs The trench needed to be approximately 50 feet in length and at least 15 feet deep to access the fiber and raise it for splicing repairs to commence. Field Operations reported 10 of the 12 conduits had been destroyed. Excavation was completed, slack was pulled and the fiber prepped, tested, and then spliced. Services were confirmed restored by the Repair Center with impacted customers.</font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif"><br></font></div><div class="gmail_default"><font face="arial, helvetica, sans-serif">This service impact has concluded; however, there may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory.</font></div></div><div class="gmail_extra"></div></div><div dir="ltr"><div class="gmail_extra"><br clear="all"><div><div class="m_3619991687817532019m_8566619125628648070m_7071494032579538183gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><font color="#000000" face="arial, helvetica, sans-serif"><br>- Mike Bolitho</font></div></div></div></div></div><div dir="ltr"><div class="gmail_extra">
<br><div class="gmail_quote">On Tue, Jul 17, 2018 at 6:10 PM, Murali Medikonda via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Curious if anyone has any further updates on this fiber-cut.. I appreciate it<div class="gmail_extra"><div><div class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><span><div dir="ltr" style="margin-left:0pt"><span><div dir="ltr" style="margin-left:0pt"><br></div></span></div></span></div></div></div></div></div></div></div></div>
<br><div class="gmail_quote">On Tue, Jul 17, 2018 at 3:57 PM, Cory Sell via Outages <span dir="ltr"><<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Thanks, Joe. Below is the most recent update we received a bit ago:<div><br></div><div>*** CASCADED EXTERNAL NOTES 17-Jul-2018 19:50:18 GMT - Event<br><br>Field Operations have reported that the Construction team is en route to the site with backhoes and the shoring supplies. An ETA of 20:00 GMT has been provided. In addition, local utility providers are currently on site marking existing utility lines in the area to prevent impacts during the repairs. Once on site, the Construction team will commence with installing the necessary shoring around the damage site. It is estimated that the shoring activities will be completed at 23:00 GMT and excavation activities will commence at that time. The Transport NOC has reported that there are currently no reroute opportunities. The next update will be relayed once the shoring has been completed or additional information becomes available.</div><div><br></div><div>Regards, </div><div><br></div><div>Cory<br><br><div class="gmail_quote"><div><div class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929h5"><div dir="ltr">On Tue, Jul 17, 2018, 1:39 PM Bobin Joseph via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div></div></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div><div class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929h5">
<div lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257WordSection1">
<p class="MsoNormal">We have P2P link between Dallas and San Diego that went down. It is also impacting our connectivity to Dallas over our other CL links. Centurylink sent out this notification. Sending it out in case others are having similar issues.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">07/17/2018 18:31:54 GMT -<u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">*** CASCADED EXTERNAL NOTES 17-Jul-2018 18:30:14 GMT From CASE: 14896924 - Event Field Operations have identified a 96-count buried fiber cable that has been cut by boring activities occurring in the area. The cable is buried 14 feet
deep and is within 25 feet of a railroad line. As such, the railroad line provider will not allow excavations activities to commence until all necessary shoring supplies are on site. The supplies are being obtained at this time, however, the ETA to the site
is not yet known. Once the shoring is in place, the excavation team will begin digging a trench to access the damaged fiber and perform the repairs. The trench will need to be approximately 50 feet in length and at least 15 feet deep to access the fiber
and raise it for splicing repairs to commence. The initial activities are expected to take approximately two hours. As such the next update will be relayed at 20:30 GMT or as additional information becomes available. This issue has been escalated internally
to the Executive level proactively.<u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText"><u></u> <u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText"><u></u> <u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">07/17/2018 17:38:50 GMT - <u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">*** CASCADED EXTERNAL NOTES 17-Jul-2018 17:36:55 GMT From CASE: 14896924 - Event<u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">Field Operations and the OSPE team have arrived at the site and have identified a location where they believe a buried fiber cable has been damaged by construction activities occurring in the area. Additional assessments are ongoing
at this time. A fiber splicing repair team has been engaged to assist with the repairs once the damaged fiber has been identified, however the ETA to the site is not yet known<u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText"><u></u> <u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText"><u></u> <u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">07/17/2018 16:42:52 GMT - <u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">*** CASCADED EXTERNAL NOTES 17-Jul-2018 16:40:40 GMT From CASE: 14896924 - Event<u></u><u></u></p>
<p class="m_3619991687817532019m_8566619125628648070m_7071494032579538183m_-1586448802408746929m_-5449654654591850741m_-441379614279350257MsoPlainText">Field Operations have performed light measurement readings from the CenturyLink Gateway in Phoenix and have identified the fault condition 28.43 miles from the site. The Outside Plant Engineer (OSPE) has been engaged to assist with
further isolation.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks,<u></u><u></u></p>
<p class="MsoNormal">Bobin Joseph<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
</div></div></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div></div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div></div>
<br>_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br></blockquote></div><br></div></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>