<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/xhtml; charset=utf-8">
</head>
<body>
<div style="font-family:sans-serif"><div style="white-space:normal"><p dir="auto">Stable,</p>
<p dir="auto">latest statement (Flowspec):</p>
<p dir="auto">Summary: On August 30, 2020 10:04 GMT, CenturyLink identified an issue to be affecting users across multiple markets. The IP Network Operations Center (NOC) was engaged, and initial research identified that an offending flowspec announcement prevented Border Gateway Protocol (BGP) from establishing across multiple elements throughout the CenturyLink Network. The IP NOC deployed a global configuration change to block the offending flowspec announcement, which allowed BGP to begin to correctly establish. As the change propagated through the network, the IP NOC observed all associated service affecting alarms clearing and services returning to a stable state.</p>
<p dir="auto">On 30 Aug 2020, at 18:33, Tim Choe via Outages wrote:</p>
</div>
<blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><div id="CFCA4555-C1F7-43D4-9966-2C140E8086E3"><div lang="EN-US" link="blue" vlink="purple"><div style="page:WordSection1"><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D">According to Centurylink tweet, their service has been restored. Can someone help me confirm that this is true before I start shifting the traffic back?</span></p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D"> </span></p><div><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><b>From:</b> Outages [mailto:outages-bounces@outages.org] <b>On Behalf Of </b>Stephen Flynn via Outages<br><b>Sent:</b> Sunday, August 30, 2020 8:41 AM<br><b>To:</b> outages@outages.org<br><b>Subject:</b> Re: [outages] 3556 problems</p></div></div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D">Latest update from CenturyLink</span></p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D"> </span></p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>Notes History:</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D"> </span></p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D"> </span></p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D"> </span></p><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:12pt'><span style="color:black">Regards,<br><br>Stephen Flynn<br>Atlantic.Net<br>Direct: (321) 206-1390<br><a href="mailto:sflynn@staff.atlantic.net">sflynn@staff.atlantic.net</a><br><a href="http://www.atlantic.net/">www.atlantic.net</a></span><span style="font-size:12.0pt;color:black"></span></p></div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style="color:#1F497D"> </span></p><div><div style="border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0in 0in 0in"><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><b>From:</b> Outages <<a href="mailto:outages-bounces@outages.org">outages-bounces@outages.org</a>> <b>On Behalf Of </b>Dylan Ebner via Outages<br><b>Sent:</b> Sunday, August 30, 2020 11:00 AM<br><b>To:</b> Michael McDonald <<a href="mailto:mmcdonald@liquidweb.com">mmcdonald@liquidweb.com</a>><br><b>Cc:</b> <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> Re: [outages] 3556 problems</p></div></div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><table border="1" cellspacing="0" cellpadding="0" align="left" width="100%" style="width:100.0%"><tr><td style="background:#B04800;padding:5.25pt 1.5pt 5.25pt 1.5pt"><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-element:frame; mso-element-anchor-horizontal:column; mso-element-anchor-vertical:paragraph; mso-element-frame-hspace:1.8pt; mso-element-wrap:around; mso-height-rule:exactly'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'> </span></p></td><td width="100%" style="width:100.0%;background:#FFE878;padding:5.25pt 3.75pt 5.25pt 11.25pt;word-wrap: break-word"><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-element:frame; mso-element-anchor-horizontal:column; mso-element-anchor-vertical:paragraph; mso-element-frame-hspace:1.8pt; mso-element-wrap:around; mso-height-rule:exactly'><strong><span style='font-size:12.0pt;font-family:"Calibri",sans-serif;color:#B04800'>CAUTION:</span></strong><span style="font-size:12.0pt;color:#B04800"> This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.</span></p></div></td></tr></table><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><span style='font-size:12.0pt;font-family:"Times New Roman",serif'> </span></p><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>Here is their latest update:</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>Next update by: 2020-08-30 15:50 GMT</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>This notification is sent from an unmonitored email account. Please click the link below to reply:</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'><a href="mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A">mailto:SMC@CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A</a></p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>Notes History:</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear. </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt; mso-style-link:"Plain Text Char"; mso-style-priority:99'>The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><b>From:</b> Michael McDonald <<a href="mailto:mmcdonald@liquidweb.com">mmcdonald@liquidweb.com</a>> <br><b>Sent:</b> Sunday, August 30, 2020 9:56 AM<br><b>To:</b> Dylan Ebner <<a href="mailto:dylan.ebner@crlmed.com">dylan.ebner@crlmed.com</a>><br><b>Cc:</b> Jay Ashworth <<a href="mailto:jra@baylink.com">jra@baylink.com</a>>; <a href="mailto:outages@outages.org">outages@outages.org</a><br><b>Subject:</b> Re: [outages] 3556 problems</p><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!</p></div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p><div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <<a href="mailto:outages@outages.org">outages@outages.org</a>> wrote:</p></div><blockquote style="border:none;border-left:solid #CCCCCC 1.0pt;padding:0in 0in 0in 6.0pt;margin-left:4.8pt;margin-top:5.0pt;margin-right:0in;margin-bottom:5.0pt"><div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403</p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'><br>08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.</p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>-------- Original message --------</p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>From: Jay Ashworth via Outages <<a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a>> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>Date: 8/30/20 9:36 AM (GMT-06:00) </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>To: <a href="mailto:outages@outages.org" target="_blank">outages@outages.org</a> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>Subject: [outages] 3556 problems </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'> </p></div><div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.<br>-- <br>Sent from my Android device with K-9 Mail. Please excuse my brevity.</p></div></div><p style='font-family:"Calibri", sans-serif; font-size:11pt; margin:0; margin-bottom:0.0001pt'>_______________________________________________<br>Outages mailing list<br><a href="mailto:Outages@outages.org" target="_blank">Outages@outages.org</a><br><a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a></p></blockquote></div></div></div></div></div></blockquote>
<div style="white-space:normal"><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px">
</blockquote><blockquote style="border-left:2px solid #777; color:#777; margin:0 0 5px; padding-left:5px"><p dir="auto">_______________________________________________<br>
Outages mailing list<br>
Outages@outages.org<br>
<a href="https://puck.nether.net/mailman/listinfo/outages" style="color:#777">https://puck.nether.net/mailman/listinfo/outages</a></p>
</blockquote></div>
</div>
</body>
</html>