[outages] 3556 problems
Jason Kuehl
jason.w.kuehl at gmail.com
Sun Aug 30 12:51:01 EDT 2020
I just shifted back and everything looks normal. I'm in Waltham MA.
On Sun, Aug 30, 2020 at 12:44 PM Josh Luthman via Outages <
outages at outages.org> wrote:
> We were effected by this in reaching the Google SAS. It's been working
> without problems for 100 minutes.
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
> On Sun, Aug 30, 2020, 12:38 PM Tim Choe via Outages <outages at outages.org>
> wrote:
>
>> According to Centurylink tweet, their service has been restored. Can
>> someone help me confirm that this is true before I start shifting the
>> traffic back?
>>
>>
>>
>> *From:* Outages [mailto:outages-bounces at outages.org] *On Behalf Of *Stephen
>> Flynn via Outages
>> *Sent:* Sunday, August 30, 2020 8:41 AM
>> *To:* outages at outages.org
>> *Subject:* Re: [outages] 3556 problems
>>
>>
>>
>> Latest update from CenturyLink
>>
>>
>>
>> Notes History:
>>
>>
>>
>> 08/30/2020 15:31:25 GMT - The IP NOC continues to observe service
>> affecting alarms clearing as the configuration that was deployed continues
>> to propagate through the affected devices. Once affected services have been
>> confirmed restored and stable, a final notification will be provided.
>>
>>
>>
>> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
>> Operations Engineering team confirmed a routing issue to be preventing BGP
>> sessions from establishing correctly. A configuration adjustment was
>> deployed at a high level, and sessions began to re-establish with
>> stability. As the change propagates through the affected devices, service
>> affecting alarms continue to clear.
>>
>>
>>
>> Due to the nature of this outage, it may be necessary to reset your
>> services locally at your equipment, or manually reset your BGP session. If
>> after that action has been performed a service issue prevails, please
>> contact the CenturyLink Repair Center for troubleshooting assistance.
>>
>>
>>
>> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
>> Protocol (BGP) issues causing service impacts across multiple markets.
>> Cooperative escalated investigation efforts with CenturyLink leadership
>> are underway to isolate and troubleshoot the issue and expedite service
>> restoral.
>>
>>
>>
>> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
>> investigation and troubleshooting efforts remain ongoing this time.
>>
>>
>>
>> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
>> investigations to isolate and troubleshoot the fault at this time.
>>
>>
>>
>> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
>> identified a market wide. As this network fault is impacting multiple
>> clients, the event has increased visibility with CenturyLink leadership. As
>> such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>>
>>
>>
>> The NOC is engaged and investigating in order to isolate the cause.
>> Please be advised that updates for this event will be relayed at a minimum
>> of hourly unless otherwise noted. The information conveyed hereafter is
>> associated to live troubleshooting effort and as the discovery process
>> evolves through to service resolution, ticket closure, or post incident
>> review, details may evolve.
>>
>>
>>
>>
>>
>>
>>
>> Regards,
>>
>> Stephen Flynn
>> Atlantic.Net
>> Direct: (321) 206-1390
>> sflynn at staff.atlantic.net
>> www.atlantic.net
>>
>>
>>
>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Dylan Ebner
>> via Outages
>> *Sent:* Sunday, August 30, 2020 11:00 AM
>> *To:* Michael McDonald <mmcdonald at liquidweb.com>
>> *Cc:* outages at outages.org
>> *Subject:* Re: [outages] 3556 problems
>>
>>
>>
>>
>>
>> *CAUTION:* This email originated from outside of the organization. Do
>> not click links or open attachments unless you recognize the sender and
>> know the content is safe.
>>
>>
>>
>> Here is their latest update:
>>
>>
>>
>>
>>
>> The IP NOC with the assistance of the Operations Engineering team
>> confirmed a routing issue to be preventing BGP sessions from establishing
>> correctly. A configuration adjustment was deployed at a high level, and
>> sessions began to re-establish with stability. As the change propagates
>> through the affected devices, service affecting alarms continue to clear.
>>
>>
>>
>> Due to the nature of this outage, it may be necessary to reset your
>> services locally at your equipment, or manually reset your BGP session. If
>> after that action has been performed a service issue prevails, please
>> contact the CenturyLink Repair Center for troubleshooting assistance.
>>
>>
>>
>> Next update by: 2020-08-30 15:50 GMT
>>
>>
>>
>> This notification is sent from an unmonitored email account. Please
>> click the link below to reply:
>>
>>
>>
>>
>> mailto:SMC at CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A
>> <SMC at CenturyLink.com?subject=RE%3A%20CenturyLink%20Ticket%20#:%2019545403,&body=Customer%20Reply%3A%0D%0A>
>>
>>
>>
>> Notes History:
>>
>>
>>
>> 08/30/2020 14:46:29 GMT - The IP NOC with the assistance of the
>> Operations Engineering team confirmed a routing issue to be preventing BGP
>> sessions from establishing correctly. A configuration adjustment was
>> deployed at a high level, and sessions began to re-establish with
>> stability. As the change propagates through the affected devices, service
>> affecting alarms continue to clear.
>>
>>
>>
>> Due to the nature of this outage, it may be necessary to reset your
>> services locally at your equipment, or manually reset your BGP session. If
>> after that action has been performed a service issue prevails, please
>> contact the CenturyLink Repair Center for troubleshooting assistance.
>>
>>
>>
>> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
>> Protocol (BGP) issues causing service impacts across multiple markets.
>> Cooperative escalated investigation efforts with CenturyLink leadership
>> are underway to isolate and troubleshoot the issue and expedite service
>> restoral.
>>
>>
>>
>> 08/30/2020 12:14:09 GMT - The IP NOC advises cooperative escalated
>> investigation and troubleshooting efforts remain ongoing this time.
>>
>>
>>
>> 08/30/2020 11:36:30 GMT - The IP NOC is engaged in cooperative escalated
>> investigations to isolate and troubleshoot the fault at this time.
>>
>>
>>
>> 08/30/2020 10:48:38 GMT - On August 30, 2020 10:04 GMT, CenturyLink
>> identified a market wide. As this network fault is impacting multiple
>> clients, the event has increased visibility with CenturyLink leadership. As
>> such, client trouble tickets associated to this fault have been
>> automatically escalated to higher priority.
>>
>>
>>
>> The NOC is engaged and investigating in order to isolate the cause.
>> Please be advised that updates for this event will be relayed at a minimum
>> of hourly unless otherwise noted. The information conveyed hereafter is
>> associated to live troubleshooting effort and as the discovery process
>> evolves through to service resolution, ticket closure, or post incident
>> review, details may evolve.
>>
>>
>>
>>
>>
>> *From:* Michael McDonald <mmcdonald at liquidweb.com>
>> *Sent:* Sunday, August 30, 2020 9:56 AM
>> *To:* Dylan Ebner <dylan.ebner at crlmed.com>
>> *Cc:* Jay Ashworth <jra at baylink.com>; outages at outages.org
>> *Subject:* Re: [outages] 3556 problems
>>
>>
>>
>> Not on that ticket, so thanks a ton for the update! Execs are asking
>> questions about what's going on and it's nice to give them feedback!
>>
>>
>>
>> On Sun, Aug 30, 2020 at 10:52 AM Dylan Ebner via Outages <
>> outages at outages.org> wrote:
>>
>> For those who aren't on the master CL ticket, here is the latest from CL.
>> Ticket number: 19545403
>>
>>
>>
>>
>>
>>
>> 08/30/2020 13:19:02 GMT - The IP NOC has isolated multiple Border Gateway
>> Protocol (BGP) issues causing service impacts across multiple markets.
>> Cooperative escalated investigation efforts with CenturyLink leadership are
>> underway to isolate and troubleshoot the issue and expedite service
>> restoral.
>>
>>
>>
>>
>>
>>
>>
>> -------- Original message --------
>>
>> From: Jay Ashworth via Outages <outages at outages.org>
>>
>> Date: 8/30/20 9:36 AM (GMT-06:00)
>>
>> To: outages at outages.org
>>
>> Subject: [outages] 3556 problems
>>
>>
>>
>> Okay, it's pretty clear that they're having trouble. :-) Let's limit it
>> now to actual tickets and reports of clearance.
>> --
>> Sent from my Android device with K-9 Mail. Please excuse my brevity.
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
--
Sincerely,
Jason W Kuehl
Cell 920-419-8983
jason.w.kuehl at gmail.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20200830/7b671db2/attachment.htm>
More information about the Outages
mailing list