<div dir="ltr"><font size="2"><span style="font-size:11pt">*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:45:17 GMT From CASE: 16976512 - Event<br>
*Event Conclusion Summary*<br>
<br>
Outage Start: August 23, 2019 14:44 GMT<br>
Outage Stop: August 23, 2019 17:19 GMT<br>
<br>
Root Cause: A fiber cut caused multiple isolated routers in Des Moines, IA that was impacting IP services.<br>
Fix Action: Impacted services were migrated to a diverse fiber path.<br>
<br>
Summary: On August 23, 2019 at 14:44 GMT, CenturyLink identified a
service impact in Des Moines, IA and the IP NOC began investigating. It
was advised that multiple routers had become isolating causing impact
to IP services. Field Operations were dispatched to the site of the
isolated routers but confirmed power and were unable to identify any
issues. The Transport NOC was then engaged and identified a suspected
fiber issue. Field Operations were dispatched for initial light level
testing. Initial light level testing confirmed the existence of a fiber
cut in the area, and Field Operations advised there were multiple
instances of constructions works ongoing in the area. Field Operations
began traversing the span and taking additional light level readings
from manholes along the span. Field Operations later identified that
equipment laying rock in the area had damaged multiple cables. Efforts
to identify the extent of the damage and formulate a repair plan
remained ongo<br>
ing.<br>
<br>
While efforts to identify the fiber damage were ongoing, services were
migrated to a diverse fiber path to restore all services. At this time
all services have been restored but are currently in a Loss of
Redundancy state while the fiber repairs remain ongoing. The Transport
and IP NOC will continue to monitor services while the fiber repairs
remain ongoing to ensure stability. A final notification will be
provided momentarily.<br>
<br>
This service impact has concluded. Due to the nature of this outage, it
may be necessary to reset your services locally at your equipment. If
after that action has been performed a service issue prevails, please
contact the CenturyLink Repair Center for troubleshooting assistance.
There may be additional analysis and discovery that occurs as the
incident is reviewed by NOC management. Any available updates will be
relayed upon event ticket closure within the next 72 hours. At that
time, a customer satisfaction survey link may be available. We strive to
provide thorough communications containing the available information
during a service disruption. Please let us know if the updates you
received during this event were satisfactory.</span></font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 23, 2019 at 1:36 PM Jeffrey Ollie <<a href="mailto:jeff@ocjtech.us">jeff@ocjtech.us</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font size="2"><span style="font-size:11pt">*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 - Event<br>
Field Operations on site have identified that equipment laying rock in
the area had damaged multiple cables. Efforts to identify the extent of
the damage and formulate a repair plan are ongoing.<br>
<br>
While efforts to identify the fiber damage were ongoing, services were
migrated to a diverse fiber path to restore all services. At this time
all services have been restored but are currently in a Loss of
Redundancy state while the fiber repairs remain ongoing. The Transport
and IP NOC will continue to monitor services while the fiber repairs
remain ongoing to ensure stability. A final notification will be
provided momentarily.</span></font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 23, 2019 at 12:52 PM Jeffrey Ollie <<a href="mailto:jeff@ocjtech.us" target="_blank">jeff@ocjtech.us</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font size="2"><span style="font-size:11pt">*** CASCADED EXTERNAL NOTES 23-Aug-2019 17:23:30 GMT From CASE: 16976512 - Event<br></span></font><div><font size="2"><span style="font-size:11pt">
The Transport NOC has advised that Field Operations continue to work to
isolate the fiber damage. Additional light level testing was performed
along the span, and Field Operations have advised the damage is
suspected to be located 125 feet from the latest
testing location.<br></span></font></div><div><br></div><div><font size="2"><span style="font-size:11pt">08/23/2019 16:37:04 GMT -<br>
*** CASCADED EXTERNAL NOTES 23-Aug-2019 16:34:24 GMT From CASE: 16976512 - Event<br>
Field Operations is working to traverse the fiber span in an effort to
identify the fiber damage. It has been advised that there are multiple
areas of construction being performed along the span and Field
Operations are actively investigating those areas.</span></font><br><font size="2"><span style="font-size:11pt"></span></font></div><div><font size="2"><span style="font-size:11pt">
</span></font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 23, 2019 at 10:56 AM Jeffrey Ollie <<a href="mailto:jeff@ocjtech.us" target="_blank">jeff@ocjtech.us</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><font size="2"><span style="font-size:11pt">*** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event<br>
The Transport NOC have dispatched Field Operations to two sites in Des
Moines, IA and have taken light level readings indicating a suspected
fiber issue is present. Construction efforts in the area have been
identified in the area, and Field Operations are
dispatching to that area to further isolate the suspected damage.<br>
</span></font></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <<a href="mailto:jeff@ocjtech.us" target="_blank">jeff@ocjtech.us</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div>I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area.</div><div><br></div><div><font size="2"><span style="font-size:11pt">08/23/2019 15:30:46 GMT -
The IP NOC is investigating an issue that is causing multiple routers to
be isolated in Des Moines, IA that is impacting IP services. Field
Operations are on site and have advised that there is power to the site
and no immediate issues observed with the equipment. The Transport NOC
is being engaged to investigate the transport portion of the network.<br>
<br>
08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink
identified a service impact in Des Moines, IA. The NOC is engaged and
investigating in order to isolate the cause. Please be advised that
updates for this event will be relayed at a minimum of hourly unless
otherwise noted. The information conveyed hereafter is associated to
live troubleshooting efforts and details may evolve as the discovery
process evolves through to service resolution, ticket closure, or post
incident review.</span></font></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <<a href="mailto:jmaslak@antelope.net" target="_blank">jmaslak@antelope.net</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Not national. 3356 and 209 are up in Denver.</div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr">Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?<br clear="all"><div><br>-- <br><div dir="ltr" class="gmail-m_-6885538384080823287gmail-m_-7219065020882392377gmail-m_5283940228212563372gmail-m_2079661070588533800gmail-m_1854100968199420050gmail-m_-22736917884062621gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div></div></div>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</blockquote></div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-6885538384080823287gmail-m_-7219065020882392377gmail-m_5283940228212563372gmail-m_2079661070588533800gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-6885538384080823287gmail-m_-7219065020882392377gmail-m_5283940228212563372gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-6885538384080823287gmail-m_-7219065020882392377gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail-m_-6885538384080823287gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature">Jeff Ollie<br>The majestik møøse is one of the mäni interesting furry animals in Sweden.</div>