[outages] 3556 problems

Sun Aug 30 12:57:15 EDT 2020

“Can someone help me confirm that this is true before I start shifting the traffic back?”

I won’t be activating our links until this evenings maintenance window.

Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn at staff.atlantic.net<mailto:sflynn at staff.atlantic.net>
www.atlantic.net<http://www.atlantic.net/>

From: Outages <outages-bounces at outages.org> On Behalf Of Jason Kuehl via Outages
Sent: Sunday, August 30, 2020 12:51 PM
To: Josh Luthman <josh at imaginenetworksllc.com>
Cc: OUTAGES Mailing List <outages at outages.org>
Subject: Re: [outages] 3556 problems

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

I just shifted back and everything looks normal. I'm in Waltham MA.

On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
We were effected by this in reaching the Google SAS.  It's been working without problems for 100 minutes.
Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
According to Centurylink tweet, their service has been restored.  Can someone help me confirm that this is true before I start shifting the traffic back?

From: Outages [mailto:outages-bounces at outages.org<mailto:outages-bounces at outages.org>] On Behalf Of Stephen Flynn via Outages
Sent: Sunday, August 30, 2020 8:41 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] 3556 problems

Latest update from CenturyLink

Notes History:

08/30/2020 15:31:25 GMT - The IP NOC continues to observe service affecting alarms clearing as the configuration that was deployed continues to propagate through the affected devices. Once affected services have been confirmed restored and stable, a final notification will be provided.

08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.

Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.

08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated  investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.

08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.

08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.

08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

Regards,

Stephen Flynn
Atlantic.Net
Direct: (321) 206-1390
sflynn at staff.atlantic.net<mailto:sflynn at staff.atlantic.net>
www.atlantic.net<http://www.atlantic.net/>

From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Dylan Ebner via Outages
Sent: Sunday, August 30, 2020 11:00 AM
To: Michael McDonald <mmcdonald at liquidweb.com<mailto:mmcdonald at liquidweb.com>>
Cc: outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] 3556 problems

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.

Here is their latest update:

The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.

Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.

Next update by: 2020-08-30 15:50 GMT

This notification is sent from an unmonitored email account.  Please click the link below to reply:

mailto:SMC at CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A

Notes History:

08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations Engineering team confirmed a routing issue to be preventing BGP sessions from establishing correctly. A configuration adjustment was deployed at a high level, and sessions began to re-establish with stability. As the change propagates through the affected devices, service affecting alarms continue to clear.

Due to the nature of this outage, it may be necessary to reset your services locally at your equipment, or manually reset your BGP session. If after that action has been performed a service issue prevails, please contact the CenturyLink Repair Center for troubleshooting assistance.

08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated  investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.

08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated investigation and troubleshooting efforts remain ongoing this time.

08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated investigations to isolate and troubleshoot the fault at this time.

08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink identified a market wide. As this network fault is impacting multiple clients, the event has increased visibility with CenturyLink leadership. As such, client trouble tickets associated to this fault have been automatically escalated to higher priority.

The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

From: Michael McDonald <mmcdonald at liquidweb.com<mailto:mmcdonald at liquidweb.com>>
Sent: Sunday, August 30, 2020 9:56 AM
To: Dylan Ebner <dylan.ebner at crlmed.com<mailto:dylan.ebner at crlmed.com>>
Cc: Jay Ashworth <jra at baylink.com<mailto:jra at baylink.com>>; outages at outages.org<mailto:outages at outages.org>
Subject: Re: [outages] 3556 problems

Not on that ticket, so thanks a ton for the update! Execs are asking questions about what's going on and it's nice to give them feedback!

On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
For those who aren't on the master CL ticket, here is the latest from CL. Ticket number: 19545403

08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway Protocol (BGP) issues causing service impacts across multiple markets. Cooperative escalated investigation efforts with CenturyLink leadership are underway to isolate and troubleshoot the issue and expedite service restoral.

-------- Original message --------
From: Jay Ashworth via Outages <outages at outages.org<mailto:outages at outages.org>>
Date: 8/30/20 9:36 AM (GMT-06:00)
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] 3556 problems

Okay, it's pretty clear that they're having trouble. :-) Let's limit it now to actual tickets and reports of clearance.
--
Sent from my Android device with K-9 Mail. Please excuse my brevity.
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages

--
Sincerely,

Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl at gmail.com<mailto:jason.w.kuehl at gmail.com>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200830/a2229bdf/attachment.htm>