[outages] CenturyLink Internet Outage in Iowa, possibly Nationally?

Fri Aug 23 14:57:41 EDT 2019

*** CASCADED EXTERNAL NOTES 23-Aug-2019 18:45:17 GMT From CASE: 16976512 -
Event
*Event Conclusion Summary*

Outage Start: August 23, 2019 14:44 GMT
Outage Stop: August 23, 2019 17:19 GMT

Root Cause: A fiber cut caused multiple isolated routers in Des Moines, IA
that was impacting IP services.
Fix Action: Impacted services were migrated to a diverse fiber path.

Summary: On August 23, 2019 at 14:44 GMT, CenturyLink identified a service
impact in Des Moines, IA and the IP NOC began investigating.  It was
advised that multiple routers had become isolating causing impact to IP
services.  Field Operations were dispatched to the site of the isolated
routers but confirmed power and were unable to identify any issues.  The
Transport NOC was then engaged and identified a suspected fiber issue.
Field Operations were dispatched for initial light level testing.  Initial
light level testing confirmed the existence of a fiber cut in the area, and
Field Operations advised there were multiple instances of constructions
works ongoing in the area.  Field Operations began traversing the span and
taking additional light level readings from manholes along the span.  Field
Operations later identified that equipment laying rock in the area had
damaged multiple cables.  Efforts to identify the extent of the damage and
formulate a repair plan remained ongo
 ing.

While efforts to identify the fiber damage were ongoing, services were
migrated to a diverse fiber path to restore all services.  At this time all
services have been restored but are currently in a Loss of Redundancy state
while the fiber repairs remain ongoing.  The Transport and IP NOC will
continue to monitor services while the fiber repairs remain ongoing to
ensure stability.  A final notification will be provided momentarily.

This service impact has concluded. Due to the nature of this outage, it may
be necessary to reset your services locally at your equipment.  If after
that action has been performed a service issue prevails, please contact the
CenturyLink Repair Center for troubleshooting assistance.  There may be
additional analysis and discovery that occurs as the incident is reviewed
by NOC management. Any available updates will be relayed upon event ticket
closure within the next 72 hours. At that time, a customer satisfaction
survey link may be available. We strive to provide thorough communications
containing the available information during a service disruption. Please
let us know if the updates you received during this event were satisfactory.

On Fri, Aug 23, 2019 at 1:36 PM Jeffrey Ollie <jeff at ocjtech.us> wrote:

> *** CASCADED EXTERNAL NOTES 23-Aug-2019 18:09:40 GMT From CASE: 16976512 -
> Event
> Field Operations on site have identified that equipment laying rock in the
> area had damaged multiple cables.  Efforts to identify the extent of the
> damage and formulate a repair plan are ongoing.
>
> While efforts to identify the fiber damage were ongoing, services were
> migrated to a diverse fiber path to restore all services.  At this time all
> services have been restored but are currently in a Loss of Redundancy state
> while the fiber repairs remain ongoing.  The Transport and IP NOC will
> continue to monitor services while the fiber repairs remain ongoing to
> ensure stability.  A final notification will be provided momentarily.
>
> On Fri, Aug 23, 2019 at 12:52 PM Jeffrey Ollie <jeff at ocjtech.us> wrote:
>
>> *** CASCADED EXTERNAL NOTES 23-Aug-2019 17:23:30 GMT From CASE: 16976512
>> - Event
>> The Transport NOC has advised that Field Operations continue to work to
>> isolate the fiber damage.  Additional light level testing was performed
>> along the span, and Field Operations have advised the damage is suspected
>> to be located 125 feet from the latest testing location.
>>
>> 08/23/2019 16:37:04 GMT -
>> *** CASCADED EXTERNAL NOTES 23-Aug-2019 16:34:24 GMT From CASE: 16976512
>> - Event
>> Field Operations is working to traverse the fiber span in an effort to
>> identify the fiber damage.  It has been advised that there are multiple
>> areas of construction being performed along the span and Field Operations
>> are actively investigating those areas.
>>
>> On Fri, Aug 23, 2019 at 10:56 AM Jeffrey Ollie <jeff at ocjtech.us> wrote:
>>
>>> *** CASCADED EXTERNAL NOTES 23-Aug-2019 15:50:26 GMT From CASE: 16976512
>>> - Event
>>> The Transport NOC have dispatched Field Operations to two sites in Des
>>> Moines, IA and have taken light level readings indicating a suspected fiber
>>> issue is present.  Construction efforts in the area have been identified in
>>> the area, and Field Operations are dispatching to that area to further
>>> isolate the suspected damage.
>>>
>>> On Fri, Aug 23, 2019 at 10:39 AM Jeffrey Ollie <jeff at ocjtech.us> wrote:
>>>
>>>> I finally made it through the queue and was told that there's a
>>>> possible fiber cut in the Altoona, Iowa area.
>>>>
>>>> 08/23/2019 15:30:46 GMT - The IP NOC is investigating an issue that is
>>>> causing multiple routers to be isolated in Des Moines, IA that is impacting
>>>> IP services.  Field Operations are on site and have advised that there is
>>>> power to the site and no immediate issues observed with the equipment.  The
>>>> Transport NOC is being engaged to investigate the transport portion of the
>>>> network.
>>>>
>>>> 08/23/2019 15:18:36 GMT - On August 23, 2019 at 15:04 GMT, CenturyLink
>>>> identified a service impact in Des Moines, IA. The NOC is engaged and
>>>> investigating in order to isolate the cause. Please be advised that updates
>>>> for this event will be relayed at a minimum of hourly unless otherwise
>>>> noted. The information conveyed hereafter is associated to live
>>>> troubleshooting efforts and details may evolve as the discovery process
>>>> evolves through to service resolution, ticket closure, or post incident
>>>> review.
>>>>
>>>> On Fri, Aug 23, 2019 at 10:31 AM Joelle Maslak <jmaslak at antelope.net>
>>>> wrote:
>>>>
>>>>> Not national.  3356 and 209 are up in Denver.
>>>>>
>>>>> On Fri, Aug 23, 2019 at 9:27 AM Jeffrey Ollie via Outages <
>>>>> outages at outages.org> wrote:
>>>>>
>>>>>> Our CenturyLink services here in Iowa are down. Ethernet carriers are
>>>>>> up, BGP sessions are up but we're not getting any BGP routes except 150 odd
>>>>>> internal CenturyLink routes. Been waiting on hold with CL support for a
>>>>>> while. Is anyone else seeing the same issue?
>>>>>>
>>>>>> --
>>>>>> Jeff Ollie
>>>>>> The majestik møøse is one of the mäni interesting furry animals in
>>>>>> Sweden.
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Jeff Ollie
>>>> The majestik møøse is one of the mäni interesting furry animals in
>>>> Sweden.
>>>>
>>>
>>>
>>> --
>>> Jeff Ollie
>>> The majestik møøse is one of the mäni interesting furry animals in
>>> Sweden.
>>>
>>
>>
>> --
>> Jeff Ollie
>> The majestik møøse is one of the mäni interesting furry animals in Sweden.
>>
>
>
> --
> Jeff Ollie
> The majestik møøse is one of the mäni interesting furry animals in Sweden.
>

-- 
Jeff Ollie
The majestik møøse is one of the mäni interesting furry animals in Sweden.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20190823/5efd6c3f/attachment.htm>