[outages] Centurylink Packet Loss - Socal

Tue Apr 21 13:53:37 EDT 2020

Latest update...

04/21/2020 16:58:13 GMT - IP NOC reports teams at the first cut site continue working to pull slack which is being slowed by having to pump out a manhole. Prep work is in progress on one side of the cable. Teams at the second cut site have identified the damage caused by construction excavation. A repair team is on site assessing the situation to begin repairs.

________________________________
From: Outages <outages-bounces at outages.org> on behalf of Bobin Joseph via Outages <outages at outages.org>
Sent: Tuesday, April 21, 2020 10:22 AM
To: outages at outages.org <outages at outages.org>; Eric Flanery <eflanery at fsr.com>
Subject: Re: [outages] Centurylink Packet Loss - Socal

Got this update from CL.

04/21/2020 16:07:37 GMT -
*** CASCADED EXTERNAL NOTES 21-Apr-2020 16:07:34 GMT From CASE: 18573666 - Event
IP NOC reports that services are impacted by multiple fiber cuts that are impacting the CenturyLink network. Field Operations and all necessary repair personnel have arrived to the first fiber cut location and are currently working to pull in the slack on the fiber span. Approximately 3,000 feet of fiber will be required to pulled in.

04/21/2020 16:05:08 GMT - 04/21/2020 15:34:02 GMT - On April 21 at  13:55 GMT, CenturyLink identified a service impact in multiple markets impacting IP services. As this network fault is impacting multiple customers, the event has increased visibility with CenturyLink leadership. As such, client trouble ticket associated to this fault have been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting efforts and details may evolve as the discovery process evolves through to service resolution, ticket closure, or post incident review.

Bobin Joseph   Senior Network Engineer
Direct +1 858-380-3052
www.lytx.com<https://www.lytx.com/en-us>

[Lytx Logo]<https://www.lytx.com/en-us>
[LinkedIn]<https://www.linkedin.com/company/lytxinc>
[imageFacebook]<https://www.facebook.com/LytxInc>
[imageTwitter]<https://twitter.com/lytx>
[imageYouTube]<https://www.youtube.com/user/LytxInc>

-----

CONFIDENTIALITY NOTICE: This e-mail and any attachments contain information from Lytx, Inc. and/or its affiliates, and are intended solely for the use of the named recipient(s). This e-mail may contain confidential information of Lytx and its customers. Any dissemination of this e-mail by anyone other than an intended recipient is strictly prohibited. If you are not a named recipient, you are prohibited from any further viewing of the e-mail or any attachments or from making any use of the e-mail or attachments. If you believe you have received this e-mail in error, notify the sender immediately and permanently delete the e-mail, any attachments, and all copies thereof from any drives or storage media and destroy any printouts of the e-mail or attachments.

________________________________
From: Outages <outages-bounces at outages.org> on behalf of Eric Flanery via Outages <outages at outages.org>
Sent: Tuesday, April 21, 2020 8:57 AM
To: outages at outages.org <outages at outages.org>
Subject: Re: [outages] Centurylink Packet Loss - Socal

We are also seeing ~10% loss and ~100ms extra latency on CTL; between Spokane, WA and DC. It does appear to be occurring on the old L3 network.

--Eric

-----Original Message-----
From: Outages <outages-bounces at outages.org> On Behalf Of Douglas K. Rand via Outages
Sent: Tuesday, April 21, 2020 8:43 AM
To: outages at outages.org
Subject: ADV: Re: [outages] Centurylink Packet Loss - Socal

We have been seeing CL packet loss since around 9:30am CDT. Mostly on their Level3 network, seems to be between Atlanta and Minneapolis for us. Perhaps a broader CL/L3 issue?

On 4/21/20 10:36 AM, Samuel Jackson via Outages wrote:
> Anyone experiencing CL internet packet loss in the San Diego area? We
> started seeing a spike around 7am PDT. Pings to multiple remote sites
> show around 40% loss.
>
> Downdetector also shows some activity.
>
> Thanks,
> Bobin
> /*ATTENTION: This email is from an external source. Always use caution
> when opening attachments or clicking links from unknown senders or
> when receiving unexpected emails.*/
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200421/451406fb/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 120042112224901195.png
Type: image/png
Size: 1278 bytes
Desc: 120042112224901195.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20200421/451406fb/attachment.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 120042112224901395.png
Type: image/png
Size: 15436 bytes
Desc: 120042112224901395.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20200421/451406fb/attachment-0001.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 120042112224901595.png
Type: image/png
Size: 20433 bytes
Desc: 120042112224901595.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20200421/451406fb/attachment-0002.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 120042112224901795.png
Type: image/png
Size: 15101 bytes
Desc: 120042112224901795.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20200421/451406fb/attachment-0003.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 120042112224901995.png
Type: image/png
Size: 15327 bytes
Desc: 120042112224901995.png
URL: <https://puck.nether.net/pipermail/outages/attachments/20200421/451406fb/attachment-0004.png>