[outages] 3556 problems

Sun Aug 30 12:41:16 EDT 2020

We were effected by this in reaching the Google SAS.  It's been working
without problems for 100 minutes.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages at outages.org>
wrote:

> According to Centurylink tweet, their service has been restored.  Can
> someone help me confirm that this is true before I start shifting the
> traffic back?
>
>
>
> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Stephen
> Flynn via Outages
> *Sent:* Sunday, August 30, 2020 8:41 AM
> *To:* outages at outages.org
> *Subject:* Re: [outages] 3556 problems
>
>
>
> Latest update from CenturyLink
>
>
>
> Notes History:
>
>
>
> 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service
> affecting alarms clearing as the configuration that was deployed continues
> to propagate through the affected devices. Once affected services have been
> confirmed restored and stable, a final notification will be provided.
>
>
>
> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations
> Engineering team confirmed a routing issue to be preventing BGP sessions
> from establishing correctly. A configuration adjustment was deployed at a
> high level, and sessions began to re-establish with stability. As the
> change propagates through the affected devices, service affecting alarms
> continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP session. If
> after that action has been performed a service issue prevails, please
> contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated  investigation efforts with CenturyLink leadership
> are underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
> investigation and troubleshooting efforts remain ongoing this time.
>
>
>
> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
> investigations to isolate and troubleshoot the fault at this time.
>
>
>
> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
> identified a market wide. As this network fault is impacting multiple
> clients, the event has increased visibility with CenturyLink leadership. As
> such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
>
>
>
> The NOC is engaged and investigating in order to isolate the cause. Please
> be advised that updates for this event will be relayed at a minimum of
> hourly unless otherwise noted. The information conveyed hereafter is
> associated to live troubleshooting effort and as the discovery process
> evolves through to service resolution, ticket closure, or post incident
> review, details may evolve.
>
>
>
>
>
>
>
> Regards,
>
> Stephen Flynn
> Atlantic.Net
> Direct: (321) 206-1390
> sflynn at staff.atlantic.net
> www.atlantic.net
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Dylan Ebner
> via Outages
> *Sent:* Sunday, August 30, 2020 11:00 AM
> *To:* Michael McDonald <mmcdonald at liquidweb.com>
> *Cc:* outages at outages.org
> *Subject:* Re: [outages] 3556 problems
>
>
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
>
>
> Here is their latest update:
>
>
>
>
>
> The IP NOC with the assistance of the Operations Engineering team
> confirmed a routing issue to be preventing BGP sessions from establishing
> correctly. A configuration adjustment was deployed at a high level, and
> sessions began to re-establish with stability. As the change propagates
> through the affected devices, service affecting alarms continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP session. If
> after that action has been performed a service issue prevails, please
> contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
> Next update by: 2020-08-30 15:50 GMT
>
>
>
> This notification is sent from an unmonitored email account.  Please click
> the link below to reply:
>
>
>
>
> mailto:SMC at CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A
> <SMC at CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
>
>
>
> Notes History:
>
>
>
> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the Operations
> Engineering team confirmed a routing issue to be preventing BGP sessions
> from establishing correctly. A configuration adjustment was deployed at a
> high level, and sessions began to re-establish with stability. As the
> change propagates through the affected devices, service affecting alarms
> continue to clear.
>
>
>
> Due to the nature of this outage, it may be necessary to reset your
> services locally at your equipment, or manually reset your BGP session. If
> after that action has been performed a service issue prevails, please
> contact the CenturyLink Repair Center for troubleshooting assistance.
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated  investigation efforts with CenturyLink leadership
> are underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
> investigation and troubleshooting efforts remain ongoing this time.
>
>
>
> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
> investigations to isolate and troubleshoot the fault at this time.
>
>
>
> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
> identified a market wide. As this network fault is impacting multiple
> clients, the event has increased visibility with CenturyLink leadership. As
> such, client trouble tickets associated to this fault have been
> automatically escalated to higher priority.
>
>
>
> The NOC is engaged and investigating in order to isolate the cause. Please
> be advised that updates for this event will be relayed at a minimum of
> hourly unless otherwise noted. The information conveyed hereafter is
> associated to live troubleshooting effort and as the discovery process
> evolves through to service resolution, ticket closure, or post incident
> review, details may evolve.
>
>
>
>
>
> *From:* Michael McDonald <mmcdonald at liquidweb.com>
> *Sent:* Sunday, August 30, 2020 9:56 AM
> *To:* Dylan Ebner <dylan.ebner at crlmed.com>
> *Cc:* Jay Ashworth <jra at baylink.com>; outages at outages.org
> *Subject:* Re: [outages] 3556 problems
>
>
>
> Not on that ticket, so thanks a ton for the update! Execs are asking
> questions about what's going on and it's nice to give them feedback!
>
>
>
> On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <
> outages at outages.org> wrote:
>
> For those who aren't on the master CL ticket, here is the latest from CL.
> Ticket number: 19545403
>
>
>
>
>
>
> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
> Protocol (BGP) issues causing service impacts across multiple markets.
> Cooperative escalated investigation efforts with CenturyLink leadership are
> underway to isolate and troubleshoot the issue and expedite service
> restoral.
>
>
>
>
>
>
>
> -------- Original message --------
>
> From: Jay Ashworth via Outages <outages at outages.org>
>
> Date: 8/30/20 9:36 AM (GMT-06:00)
>
> To: outages at outages.org
>
> Subject: [outages] 3556 problems
>
>
>
> Okay, it's pretty clear that they're having trouble. :-) Let's limit it
> now to actual tickets and reports of clearance.
> --
> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200830/7cd7c0d3/attachment.htm>