[Outages-discussion] CenturyLink Outages this morning

[Biddle, Josh]

Thanks.  This is one for the books I guess.  I just got in to work a few hours ago so I am playing catch up.  It sounds to me like they still don't know what happened.  In that thread I see multiple reports of circuits going into loopback.  I'm guessing this is CL's way of trying to segregate parts of their core infrastructure in attempt to kill whatever is broadcasting and reconverge their network.  It sounds like they still have not figured out the root cause.

Anyone have any thoughts or updates?


From: frnkblk at iname.com <frnkblk at iname.com>
Sent: Thursday, December 27, 2018 9:10 PM
To: Biddle, Josh <JBiddle at ntst.com>
Subject: RE: [Outages-discussion] CenturyLink Outages this morning

This Reddit post is probably the best/closes to what you're asking for: https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecnsbct/<https://urldefense.proofpoint.com/v2/url?u=https-3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=>

Frank

From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Biddle, Josh
Sent: Thursday, December 27, 2018 6:34 PM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

Anyone have root cause of Century Link outage and ETTR?


From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 6:42 PM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

My IPv6 access to www.qwest.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-DS_zFzzr3sACarc7Az1k7o8&e=> and www.centurylink.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=> has been stable since 3:29 pm U.S. Central.

downdetector.com has flatlined for a while, but still not close to zero, and based on what I see in the reddit thread, the issue is not resolved.

Frank

From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 9:02 AM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

Thanks for sharing.  IPv6 access to www.qwest.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=> and www.centurylink.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=> has been intermittent since 5:00 am U.S. Central.

Frank

From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] CenturyLink Outages this morning


We are seeing a lot of centurylink circuits and services down this morning.



Centurylink Control Center portal comes up but auth failing unable to login



Waves down or bouncing

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)

ENNI Down in Denver (Down)



Still tying to open tickets with them there portal is not working and stuck on hold right now.



________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com<mailto:compliance at NTST.com> immediately and permanently delete this email and any attachments.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com immediately and permanently delete this email and any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181228/82eabc35/attachment-0001.html>