[Outages-discussion] [outages] CenturyLink Internet Outage in Iowa, possibly Nationally?

Fri Aug 23 17:25:45 EDT 2019

Thanks for all the updates.

New stories here:

https://www.kcci.com/article/thousands-report-centurylink-service-outage/28799106

https://whoradio.iheart.com/content/2019-08-23-massive-century-link-outages-in-des-moines-central-iowa-over/

Frank

From: Outages <outages-bounces at outages.org> On Behalf Of Jeffrey Ollie via Outages
Sent: Friday, August 23, 2019 1:58 PM
To: Outages at outages.org (outages at outages.org) <Outages at outages.org>
Subject: Re: [outages] CenturyLink Internet Outage in Iowa, possibly Nationally?

*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:45:17 GMT From CASE: 16976512 - Event
*Event Conclusion Summary*

Outage Start: August 23, 2019 14:44 GMT
Outage Stop: August 23, 2019 17:19 GMT

Root Cause: A fiber cut caused multiple isolated routers in Des Moines, IA that was impacting IP services.
Fix Action: Impacted services were migrated to a diverse fiber path.

Summary: On August 23, 2019 at 14:44 GMT, CenturyLink identified a service impact in Des Moines, IA and the IP NOC began investigating.  It was advised that multiple routers had become isolating causing impact to IP services.  Field Operations were dispatched to the site of the isolated routers but confirmed power and were unable to identify any issues.  The Transport NOC was then engaged and identified a suspected fiber issue.  Field Operations were dispatched for initial light level testing.  Initial light level testing confirmed the existence of a fiber cut in the area, and Field Operations advised there were multiple instances of constructions works ongoing in the area.  Field Operations began traversing the span and taking additional light level readings from manholes along the span.  Field Operations later identified that equipment laying rock in the area had damaged multiple cables.  Efforts to identify the extent of the damage and formulate a repair plan remained ongo
 ing.

While efforts to identify the fiber damage were ongoing, services were migrated to a diverse fiber path to restore all services.  At this time all services have been restored but are currently in a Loss of Redundancy state while the fiber repairs remain ongoing.  The Transport and IP NOC will continue to monitor services while the fiber repairs remain ongoing to ensure stability.  A final notification will be provided momentarily.

This service impact has concluded. Due to the nature of this outage, it may be necessary to reset your services locally at your equipment.  If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.  There may be additional analysis and discovery that occurs as the incident is reviewed by NOC management. Any available updates will be relayed upon event ticket closure within the next 72 hours. At that time, a customer satisfaction survey link may be available. We strive to provide thorough communications containing the available information during a service disruption. Please let us know if the updates you received during this event were satisfactory.

On Fri, Aug 23, 2019 at 1:36 PM Jeffrey Ollie <jeff at ocjtech.us <mailto:jeff at ocjtech.us> > wrote:

*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 - Event
Field Operations on site have identified that equipment laying rock in the area had damaged multiple cables.  Efforts to identify the extent of the damage and formulate a repair plan are ongoing.

While efforts to identify the fiber damage were ongoing, services were migrated to a diverse fiber path to restore all services.  At this time all services have been restored but are currently in a Loss of Redundancy state while the fiber repairs remain ongoing.  The Transport and IP NOC will continue to monitor services while the fiber repairs remain ongoing to ensure stability.  A final notification will be provided momentarily.

On Fri, Aug

23, 2019 at 12:52 PM Jeffrey Ollie <jeff at ocjtech.us <mailto:jeff at ocjtech.us> > wrote:

*** CASCADED EXTERNAL NOTES 23-Aug-2019 17:23:30 GMT From CASE: 16976512 - Event

The Transport NOC has advised that Field Operations continue to work to isolate the fiber damage.  Additional light level testing was performed along the span, and Field Operations have advised the damage is suspected to be located 125 feet from the latest testing location.

08/23/2019 16:37:04 GMT -
*** CASCADED EXTERNAL NOTES 23-Aug-2019 16:34:24 GMT From CASE: 16976512 - Event
Field Operations is working to traverse the fiber span in an effort to identify the fiber damage.  It has been advised that there are multiple areas of construction being performed along the span and Field Operations are actively investigating those areas.

On Fri, Aug 23, 2019 at 10:56 AM Jeffrey Ollie <jeff at ocjtech.us <mailto:jeff at ocjtech.us> > wrote:

*** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512 - Event
The Transport NOC have dispatched Field Operations to two sites in Des Moines, IA and have taken light level readings indicating a suspected fiber issue is present.  Construction efforts in the area have been identified in the area, and Field Operations are dispatching to that area to further isolate the suspected damage.

On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff at ocjtech.us <mailto:jeff at ocjtech.us> > wrote:

I finally made it through the queue and was told that there's a possible fiber cut in the Altoona, Iowa area.

08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is causing multiple routers to be isolated in Des Moines, IA that is impacting IP services.  Field Operations are on site and have advised that there is power to the site and no immediate issues observed with the equipment.  The Transport NOC is being engaged to investigate the transport portion of the network.

08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink identified a service impact in Des Moines, IA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review.

On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak at antelope.net <mailto:jmaslak at antelope.net> > wrote:

Not national.  3356 and 209 are up in Denver.

On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages <outages at outages.org <mailto:outages at outages.org> > wrote:

Our CenturyLink services here in Iowa are down. Ethernet carriers are up, BGP sessions are up but we're not getting any BGP routes except 150 odd internal CenturyLink routes. Been waiting on hold with CL support for a while. Is anyone else seeing the same issue?

-- 