<div dir="ltr"><div dir="ltr">They're definitely lost. Last troubleshoot was a card issue in Denver which didn't change anything. <div><br></div><div>These are the updates from the ticket we have with them if anyone isn't getting these. </div><div><br></div><div><div>------------------------------------------------------------------------------------------------------------------------</div><div><br></div></div><div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-28 02:02:14 GMT - Once the card was removed in Denver, CO it was confirmed that there was no significant improvement. Additional packet captures, and logs will be pulled from the device with the card removed to further isolate the root cause. The Equipment vendor continues to work with CenturyLink Field Operations at multiple sites to remove the secondary communication channel tunnel across the network until full visibility can be restored. The equipment vendor has identified a number of additional nodes that visibility has been restored to, and their engineers are currently working to apply the necessary filter to each of the reachable nodes. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-28 01:03:11 GMT - Following the review of the logs and packet captures, the Equipment Vendor驴s Tier IV Support team has identified a suspected card issue in Denver, CO.驴驴Field Operations has arrived on site and are working in cooperation with the Equipment Vendor to remove the card. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-28 00:01:53 GMT - The Equipment Vendor is currently reviewing the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root cause. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 22:58:43 GMT - Multiple teams continue work to pull additional logs and packet captures on devices that have had visibility restored, which will be scrutinized during root cause analysis. The Tier IV Equipment Vendor Technical Support team in conjunction with Field Operations are working to remove a secondary communication channel tunnel across the network until visibility can be restored. The Equipment Vendor Support team has dispatched their Field Operations team to the site in Chicago, IL and has been obtaining data directly from the equipment. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 21:36:58 GMT - It has been advised that visibility has been restored to both the Chicago, IL and Atlanta, GA sites. Engineering and Tier IV Equipment Vendor Technical Support are currently working to obtain additional logs from devices across multiple sites including Chicago and Atlanta to further isolate the root cause. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 20:19:36 GMT - Tier IV Equipment Vendor Technical Support continues to work with CenturyLink Field Operations and Engineering to restore visibility and apply the filter to devices in Atlanta, GA and Chicago, IL. While those efforts are ongoing additional logs have been pulled from the devices in Kansas City, MO and New Orleans, LA following the restoral of visibility and the necessary filter application to obtain additional pertinent information now that the device is remotely accessible. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 19:30:01 GMT - Spoke with Sean advised we were seeing system reestored he verified and advised his other circuit is showing restored at the same time as this circuit Related to nationwide outage. adding to parent ticket Sean request we leave this ticket open until they are handsoff please when you are available could you send us your traceroutes thank you </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 19:17:01 GMT - Efforts to regain visibility to sites in Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been restored the filter will be applied to limit communication traffic between sites which was causing CPU spikes that in turn prevented the devices from functioning properly. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 18:13:33 GMT - The equipment vendor驴s Tier IV Technical Support team continued to investigate the equipment logs to further assist with isolation. Field Operations were dispatched to multiple sites to investigate equipment in Kansas City, MO, New Orleans, LA, as well as Atlanta, GA. It has been advised that a controller card has been stabilized in New Orleans, LA restoring visibility to the equipment to allow additional investigations to continue. A filter was then applied to equipment in Kansas City, MO that further alleviated the impact observed. Investigations remain ongoing to further restore network services. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 15:46:56 GMT - Following the isolation of a node in San Antonio, TX that alleviated some of the impact the necessary teams have shifted troubleshooting focus to additional nodes experiencing issues. A node in Atlanta, GA as well as a site in New Orleans, LA are currently being investigated. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 14:44:42 GMT - Field Operations dispatched to various locations to troubleshoot cooperatively with the equipment vendor. During cooperative troubleshooting, a device in San Antonio, TX was seeming to broadcast traffic consuming capacity and impacting other nodes in the network. The node was isolated from the network, which appears to have alleviated some impact; however, troubleshooting efforts continue to restore all impacted services. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 13:50:05 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in various locations. The NOC engaged to begin investigations. The NOC engaged Tier IV Vendor Support to assist in troubleshooting and fault isolation efforts. The NOC engaged Field Operations to cooperatively troubleshoot. Field Operations arrived on site in Denver, CO. Field Operations dispatched additional technicians to Kansas City, MO and Omaha, NE. The NOC is continuing to cooperatively troubleshoot with Tier IV Vendor Support for a site in San Antonio, TX. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 13:14:12 GMT - Field Operations has arrived on site at the Denver location. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 12:49:55 GMT - The NOC has engaged Tier IV Vendor Support to assist in troubleshooting efforts. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 12:31:34 GMT - Field Operations has dispatched a second technician to another site to assist with troubleshooting and fault isolation efforts. An ETA of 13:30 GMT has been provided. </span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px">2018-12-27 12:21:46 GMT - The NOC has engaged Field Operations to assist in troubleshooting and fault isolation efforts. An ETA of 13:15 GMT has been provided.</span></div><div><span style="color:rgb(0,0,0);font-size:12px"><br></span></div><div><span style="color:rgb(0,0,0);font-size:12px"> 2018-12-27 11:57:40 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.</span></div><input name="virtru-metadata" type="hidden" value="{"email-policy":{"state":"closed","expirationUnit":"days","disableCopyPaste":false,"disablePrint":false,"disableForwarding":false,"enableNoauth":false,"expires":false,"isManaged":false},"attachments":{},"compose-id":"57","compose-window":{"secure":false}}"></div></div><br><div class="gmail_quote" style=""><div dir="ltr">On Thu, Dec 27, 2018 at 6:59 PM Biddle, Josh <<a href="mailto:JBiddle@ntst.com">JBiddle@ntst.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">
<div lang="EN-US">
<div class="gmail-m_-6669067582017170190WordSection1">
<p class="MsoNormal">Thanks. This is one for the books I guess. I just got in to work a few hours ago so I am playing catch up. It sounds to me like they still don’t know what happened. In that thread I see multiple reports of circuits going into loopback.
I’m guessing this is CL’s way of trying to segregate parts of their core infrastructure in attempt to kill whatever is broadcasting and reconverge their network. It sounds like they still have not figured out the root cause.
<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<p class="MsoNormal">Anyone have any thoughts or updates?<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> <a href="mailto:frnkblk@iname.com" target="_blank">frnkblk@iname.com</a> <<a href="mailto:frnkblk@iname.com" target="_blank">frnkblk@iname.com</a>> <br>
<b>Sent:</b> Thursday, December 27, 2018 9:10 PM<br>
<b>To:</b> Biddle, Josh <<a href="mailto:JBiddle@ntst.com" target="_blank">JBiddle@ntst.com</a>><br>
<b>Subject:</b> RE: [Outages-discussion] CenturyLink Outages this morning<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">This Reddit post is probably the best/closes to what you’re asking for:
<a href="https://urldefense.proofpoint.com/v2/url?u=https-3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=" target="_blank">
https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecnsbct/</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Frank <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <<a href="mailto:outages-discussion-bounces@outages.org" target="_blank">outages-discussion-bounces@outages.org</a>>
<b>On Behalf Of </b>Biddle, Josh<br>
<b>Sent:</b> Thursday, December 27, 2018 6:34 PM<br>
<b>To:</b> <a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a><br>
<b>Subject:</b> Re: [Outages-discussion] CenturyLink Outages this morning<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><a name="m_-6669067582017170190_OLE_LINK6">Anyone have root cause of Century Link outage and ETTR?</a><u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <<a href="mailto:outages-discussion-bounces@outages.org" target="_blank">outages-discussion-bounces@outages.org</a>>
<b>On Behalf Of </b>Frank Bulk<br>
<b>Sent:</b> Thursday, December 27, 2018 6:42 PM<br>
<b>To:</b> <a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a><br>
<b>Subject:</b> Re: [Outages-discussion] CenturyLink Outages this morning<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">My IPv6 access to <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-DS_zFzzr3sACarc7Az1k7o8&e=" target="_blank">
www.qwest.com</a> and <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=" target="_blank">
www.centurylink.com</a> has been stable since 3:29 pm U.S. Central.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal"><a href="http://downdetector.com" target="_blank">downdetector.com</a> has flatlined for a while, but still not close to zero, and based on what I see in the reddit thread, the issue is not resolved.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Frank <u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages-discussion <<a href="mailto:outages-discussion-bounces@outages.org" target="_blank">outages-discussion-bounces@outages.org</a>>
<b>On Behalf Of </b>Frank Bulk<br>
<b>Sent:</b> Thursday, December 27, 2018 9:02 AM<br>
<b>To:</b> <a href="mailto:outages-discussion@outages.org" target="_blank">outages-discussion@outages.org</a><br>
<b>Subject:</b> Re: [Outages-discussion] CenturyLink Outages this morning<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Thanks for sharing. IPv6 access to <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=" target="_blank">
www.qwest.com</a> and <a href="https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=" target="_blank">
www.centurylink.com</a> has been intermittent since 5:00 am U.S. Central.<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<p class="MsoNormal">Frank<u></u><u></u></p>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div style="border-right:none;border-bottom:none;border-left:none;border-top:1pt solid rgb(225,225,225);padding:3pt 0in 0in">
<p class="MsoNormal"><b>From:</b> Outages <<a href="mailto:outages-bounces@outages.org" target="_blank">outages-bounces@outages.org</a>>
<b>On Behalf Of </b>Erik Sundberg via Outages<br>
<b>Sent:</b> Thursday, December 27, 2018 8:45 AM<br>
<b>To:</b> <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a><br>
<b>Subject:</b> [outages] CenturyLink Outages this morning<u></u><u></u></p>
</div>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div id="gmail-m_-6669067582017170190divtagdefaultwrapper">
<p><span style="font-size:12pt;color:black">We are seeing a lot of centurylink circuits and services down this morning.<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
<p><span style="font-size:12pt;color:black">Centurylink Control Center portal comes up but auth failing unable to login<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
<p><span style="font-size:12pt;color:black">Waves down or bouncing <u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black">Denver - Seattle (Bouncing)<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black">Denver - Chicago (Bouncing)<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black">New York - Los Angeles (Down)<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black">Chicago - Atlanta (Down)<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black"><br>
ENNI Down in Denver (Down)<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
<p><span style="font-size:12pt;color:black">Still tying to open tickets with them there portal is not working and stuck on hold right now.<u></u><u></u></span></p>
<p><span style="font-size:12pt;color:black"><u></u> <u></u></span></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div class="MsoNormal" align="center" style="text-align:center">
<hr size="2" width="100%" align="center">
</div>
<p class="MsoNormal"><span style="font-size:7.5pt;font-family:Arial,sans-serif;color:gray"><br>
CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering
it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify
the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.</span><u></u><u></u></p>
<p class="MsoNormal">This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient,
or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you
have received this transmission in error, please email <a href="mailto:compliance@NTST.com" target="_blank">
compliance@NTST.com</a> immediately and permanently delete this email and any attachments.
<u></u><u></u></p>
</div>
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee
or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received
this transmission in error, please email compliance@NTST.com immediately and permanently delete this email and any attachments.
</div>
_______________________________________________<br>
Outages-discussion mailing list<br>
<a href="mailto:Outages-discussion@outages.org" target="_blank">Outages-discussion@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages-discussion" rel="noreferrer" target="_blank">https://puck.nether.net/mailman/listinfo/outages-discussion</a><br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><table align="left" cellpadding="0" cellspacing="0" width="100%" style="font-size:13.3333px;font-family:Times"><tbody><tr><td></td></tr></tbody></table><table style="font-family:Times;padding-top:4px;border-top:1px solid rgb(242,207,115)"><tbody><tr><td><span style="font-family:helvetica,arial,sans-serif;font-size:13px;line-height:1.3;color:rgb(37,39,41)"><strong>WeWork | Erik Wooding</strong> <br>Manager of Network Engineering, US & Canada West, Latin America<span style="font-family:"SF Optimized",system-ui,-apple-system,system-ui,".SFNSText-Regular",sans-serif;color:rgb(75,79,86);font-size:12px;letter-spacing:-0.12px"> </span><br>O: 917-810-9345 <br><a href="http://www.wework.com" style="color:rgb(37,39,41)" target="_blank">wework.com</a> <br><br><span style="color:rgb(242,207,115);font-weight:bold">Create Your Life's Work</span> <br><br><span style="line-height:1.3;color:rgb(153,153,153)">Get rewarded for good ideas and good people! <br>Apply for funding at <a href="http://creatorawards.wework.com/" style="color:rgb(153,153,153);font-weight:bold" target="_blank">creatorawards.wework.com</a> or <br>help grow the community at <a href="http://refer.wework.com" style="color:rgb(153,153,153);font-weight:bold" target="_blank">refer.wework.com</a></span></span></td></tr></tbody></table><br></div></div></div></div></div></div></div></div></div></div></div>