From jay at west.net  Wed Dec  5 14:30:33 2018
From: jay at west.net (Jay Hennigan)
Date: Wed, 5 Dec 2018 11:30:33 -0800
Subject: [Outages-discussion] Fwd: [outages] facebook slow
In-Reply-To: <401296156.165685.1542728911540.JavaMail.zimbra@baylink.com>
References: <7978vdpbeb65fva8pbprt1798vklg9s1n2@4ax.com>
 <6D7A9384-FD4D-4B12-9EEE-C8C4EFABBE16@gmail.com>
 <907507812.7556.1542725738773.JavaMail.mhammett@ThunderFuck>
 <CACoNLrzT_idoZcYxcL_Pm-UnQ6uyzcpOwmrsD8LEkVrA8QeG-Q@mail.gmail.com>
 <028c01d480e4$e7484900$b5d8db00$@simplyip.net>
 <6667065C51295744A4C8587193D2DED31AE85750@PRI-MB1.has.com>
 <1992817339.387602.1542727731852@emg-ca-2-1>
 <BYAPR01MB363987FDA9D49807D8652E9DCFD90@BYAPR01MB3639.prod.exchangelabs.com>
 <401296156.165685.1542728911540.JavaMail.zimbra@baylink.com>
Message-ID: <f3e3b536-149d-7059-cc5f-2bd1fef8ffdf@west.net>

On 11/20/18 7:48 AM, Jay R. Ashworth wrote:
> First, Silo Phenomena Arrogance would be a great name for a rock band.
> 
> :-)

Go for it, but act quickly before someone snatches it up!

jay$ whois silophenomenaarrogance.com
No match for "SILOPHENOMENAARROGANCE.COM".
 >>> Last update of whois database: 2018-12-05T19:29:03Z <<<

-- 
Jay Hennigan - jay at west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

From virendra.rode at outages.org  Wed Dec 19 20:49:59 2018
From: virendra.rode at outages.org (outages)
Date: Wed, 19 Dec 2018 17:49:59 -0800
Subject: [Outages-discussion] [outages] Level 3 fiber cut San Bernardino, CA
In-Reply-To: <45341DB7-B726-4F01-A158-0B693040D7F9@outages.org>
References: <d6451cff5c9a971d88e2565e85024887@mail.knight-networks.com>
 <920dd160c6caff2657facbf8173a1795@mail.knight-networks.com>
 <CA+LTh5XQcH5X5idbFjkV3ZnWYvqgX0GAHei+8+irFkd2-R9N+Q@mail.gmail.com>
 <2CB0A145-7E67-484B-8213-BC1787BF5E19@outages.org>
 <CA+LTh5XdZi2AF+3XYGo4u4uZPEOk5aABEtqy1MEfr+-w7iPHsA@mail.gmail.com>
 <CA+LTh5W8vvpS-R0kGb5QN3koz7xTXBDNoiGzFYFUdEUeQRhe1Q@mail.gmail.com>
 <45341DB7-B726-4F01-A158-0B693040D7F9@outages.org>
Message-ID: <753D4901-6F19-475A-95C5-ED54F6E4FFB5@outages.org>

Hi Mehmet,

I apologize for a very slow response. I?ve been neck deep with network integration. I hope to catch with you before year end if not early part of January 2019.

?
regards,
/vrode

> On Oct 31, 2018, at 09:52, outages <virendra.rode at outages.org> wrote:
> 
> Thanks Mehmet. Will ping you offline.
> 
> ?
> regards,
> /vrode
> 
>> On Oct 31, 2018, at 09:01, Mehmet Akcin <mehmet at akcin.net> wrote:
>> 
>> by the way wanted to share some screenshots of what we are developing.
>> 
>> https://drive.google.com/drive/folders/1JsV7QRaWkzj9W3oEwJe7Qm-vqwXjOdu3?usp=sharing
>> 
>> if you are interested in helping, please contact me offlist.
>> 
>> 
>>> On Wed, Oct 24, 2018 at 10:44 PM Mehmet Akcin <mehmet at akcin.net> wrote:
>>> Lets collaborate 
>>> 
>>>> On Wed, Oct 24, 2018 at 10:44 PM outages <virendra.rode at outages.org> wrote:
>>>> I was doing something similar. I started to plot outages & heatmaps on tracker.outages.org. I even got some google API hooks to report on weather storms similar to weather.com. I tried to scrape twitter & outages email feeds which was partially successfully as I had to manually vett these outages in the backend. Had to shelf it due to day job. Have some thoughts to resurrect tracker back to life. 
>>>> 
>>>> ?
>>>> regards,
>>>> /vrode
>>>> 
>>>>> On Oct 24, 2018, at 12:36, Mehmet Akcin via Outages <outages at outages.org> wrote:
>>>>> 
>>>>> I am trying to get these type of outages reported on http://www.networkatlas.org for easy to visaual display
>>>>> 
>>>>> Will you be able to tell me if centurylink has their kmzs available somewhere publicly? I have looked at but didn?t find it.(they have pdfs)
>>>>> 
>>>>> Mehmet
>>>>> 
>>>>> 
>>>>>> On Wed, Oct 24, 2018 at 11:06 AM Brian Knight via Outages <outages at outages.org> wrote:
>>>>>> We've heard that splicing has been going on for the past several hours 
>>>>>> on the 264-count cable.  Estimated time to finish that splice is 20:00 
>>>>>> GMT.
>>>>>> 
>>>>>> -Brian
>>>>>> 
>>>>>> On 2018-10-24 07:43, Brian Knight via Outages wrote:
>>>>>> > Got a Level 3 fiber cut in San Bernardino, CA going on for about 17
>>>>>> > hours now.  Our first link alarm was at 3:47pm CT yesterday.  $DAYJOB
>>>>>> > has two 1G circuits affected.
>>>>>> > 
>>>>>> > Looks like they had a hell of a time getting the replacement cable and
>>>>>> > splice conduit installed.  Splicing is estimated to start in another
>>>>>> > 2.5-3.5 hours on the east side of the cut.
>>>>>> > 
>>>>>> > Here's the latest we received from Level 3.
>>>>>> > 
>>>>>> > 
>>>>>> > 10/24/2018 12:29:56 GMT -
>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 12:22:53 GMT From CASE: 
>>>>>> > 15371425 - Event
>>>>>> > The Outside Plant Engineer has reported that cable preparations on
>>>>>> > both the west side and the east side of the cable is ongoing. It is
>>>>>> > estimated splicing on the west side will begin at 14:00 GMT and
>>>>>> > splicing on the east side will begin between 15:00 GMT and 16:00 GMT.
>>>>>> > 
>>>>>> > 
>>>>>> > 10/24/2018 11:10:34 GMT -
>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 11:04:06 GMT From CASE: 
>>>>>> > 15371425 - Event
>>>>>> > The Outside Plant Engineer has confirmed the trench is dug and the
>>>>>> > replacement cable is being ran between manholes at the west side of
>>>>>> > the damage. Preparations are complete at the east side and splicing
>>>>>> > expected to begin at 11:30 GMT with the 264 count cable, then the 84
>>>>>> > count cable followed by a 60 count lateral cable. Restoration efforts
>>>>>> > are ongoing with an ETTR that remains unchanged, 17:30 GMT.
>>>>>> > 
>>>>>> > 
>>>>>> > 10/24/2018 10:05:30 GMT -
>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 09:58:52 GMT From CASE: 
>>>>>> > 15371425 - Event
>>>>>> > The Outside Plant Engineer reported the replacement section of cable
>>>>>> > got stuck when pulling it into position. A second splice enclosure
>>>>>> > will be installed in the next manhole. Field Operations will dig a
>>>>>> > trench between the two enclosures to complete the installation of the
>>>>>> > 2,400 foot section of 264 count replacement cable. The trench is
>>>>>> > expected to be complete at approximately 10:30 GMT with an estimated
>>>>>> > time splicing will begin of 12:30 GMT.
>>>>>> > 
>>>>>> > 
>>>>>> > 10/24/2018 08:08:37 GMT -
>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 08:01:50 GMT From CASE: 
>>>>>> > 15371425 - Event
>>>>>> > The Outside Plant Engineer reported the replacement section of fiber
>>>>>> > cable has arrived and is currently being pulled into position. Along
>>>>>> > with the section of fiber cable, a new splice enclosure is being
>>>>>> > prepped and will be replaced to complete repairs of the 264 count
>>>>>> > cable. The 84 count cable has been prepped and ready for splicing.
>>>>>> > Field Operations have moved to an adjacent manhole looking for
>>>>>> > available slack to repair to the 168 count cable. The Outside Plant
>>>>>> > Engineer reported splicing is expected to commence at 10:00 GMT. The
>>>>>> > next update will be sent at that time or if additional information is
>>>>>> > received.
>>>>>> > 
>>>>>> > 
>>>>>> > -Brian
>>>>>> > _______________________________________________
>>>>>> > Outages mailing list
>>>>>> > Outages at outages.org
>>>>>> > https://puck.nether.net/mailman/listinfo/outages
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>> -- 
>>>>> Mehmet
>>>>> +1-424-298-1903
>>>>> _______________________________________________
>>>>> Outages mailing list
>>>>> Outages at outages.org
>>>>> https://puck.nether.net/mailman/listinfo/outages
>>> -- 
>>> Mehmet
>>> +1-424-298-1903
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181219/e499a4c7/attachment.html>

From virendra.rode at outages.org  Wed Dec 19 20:59:34 2018
From: virendra.rode at outages.org (outages)
Date: Wed, 19 Dec 2018 17:59:34 -0800
Subject: [Outages-discussion] Fwd: [outages] Level 3 fiber cut San
 Bernardino, CA
References: <753D4901-6F19-475A-95C5-ED54F6E4FFB5@outages.org>
Message-ID: <AE7EEAB9-F988-43AF-9098-0380CB112C2D@outages.org>

Apologize for the noise, wasn?t meant for the list. 

?
regards,
/vrode

Begin forwarded message:

> From: outages <virendra.rode at outages.org>
> Date: December 19, 2018 at 17:49:59 PST
> To: Mehmet Akcin <mehmet at akcin.net>
> Cc: Brian Knight <ml at knight-networks.com>, outages-discussion at outages.org
> Subject: Re: [Outages-discussion] [outages] Level 3 fiber cut San Bernardino, CA
> 
> Hi Mehmet,
> 
> I apologize for a very slow response. I?ve been neck deep with network integration. I hope to catch with you before year end if not early part of January 2019.
> 
> ?
> regards,
> /vrode
> 
>> On Oct 31, 2018, at 09:52, outages <virendra.rode at outages.org> wrote:
>> 
>> Thanks Mehmet. Will ping you offline.
>> 
>> ?
>> regards,
>> /vrode
>> 
>>> On Oct 31, 2018, at 09:01, Mehmet Akcin <mehmet at akcin.net> wrote:
>>> 
>>> by the way wanted to share some screenshots of what we are developing.
>>> 
>>> https://drive.google.com/drive/folders/1JsV7QRaWkzj9W3oEwJe7Qm-vqwXjOdu3?usp=sharing
>>> 
>>> if you are interested in helping, please contact me offlist.
>>> 
>>> 
>>>> On Wed, Oct 24, 2018 at 10:44 PM Mehmet Akcin <mehmet at akcin.net> wrote:
>>>> Lets collaborate 
>>>> 
>>>>> On Wed, Oct 24, 2018 at 10:44 PM outages <virendra.rode at outages.org> wrote:
>>>>> I was doing something similar. I started to plot outages & heatmaps on tracker.outages.org. I even got some google API hooks to report on weather storms similar to weather.com. I tried to scrape twitter & outages email feeds which was partially successfully as I had to manually vett these outages in the backend. Had to shelf it due to day job. Have some thoughts to resurrect tracker back to life. 
>>>>> 
>>>>> ?
>>>>> regards,
>>>>> /vrode
>>>>> 
>>>>>> On Oct 24, 2018, at 12:36, Mehmet Akcin via Outages <outages at outages.org> wrote:
>>>>>> 
>>>>>> I am trying to get these type of outages reported on http://www.networkatlas.org for easy to visaual display
>>>>>> 
>>>>>> Will you be able to tell me if centurylink has their kmzs available somewhere publicly? I have looked at but didn?t find it.(they have pdfs)
>>>>>> 
>>>>>> Mehmet
>>>>>> 
>>>>>> 
>>>>>>> On Wed, Oct 24, 2018 at 11:06 AM Brian Knight via Outages <outages at outages.org> wrote:
>>>>>>> We've heard that splicing has been going on for the past several hours 
>>>>>>> on the 264-count cable.  Estimated time to finish that splice is 20:00 
>>>>>>> GMT.
>>>>>>> 
>>>>>>> -Brian
>>>>>>> 
>>>>>>> On 2018-10-24 07:43, Brian Knight via Outages wrote:
>>>>>>> > Got a Level 3 fiber cut in San Bernardino, CA going on for about 17
>>>>>>> > hours now.  Our first link alarm was at 3:47pm CT yesterday.  $DAYJOB
>>>>>>> > has two 1G circuits affected.
>>>>>>> > 
>>>>>>> > Looks like they had a hell of a time getting the replacement cable and
>>>>>>> > splice conduit installed.  Splicing is estimated to start in another
>>>>>>> > 2.5-3.5 hours on the east side of the cut.
>>>>>>> > 
>>>>>>> > Here's the latest we received from Level 3.
>>>>>>> > 
>>>>>>> > 
>>>>>>> > 10/24/2018 12:29:56 GMT -
>>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 12:22:53 GMT From CASE: 
>>>>>>> > 15371425 - Event
>>>>>>> > The Outside Plant Engineer has reported that cable preparations on
>>>>>>> > both the west side and the east side of the cable is ongoing. It is
>>>>>>> > estimated splicing on the west side will begin at 14:00 GMT and
>>>>>>> > splicing on the east side will begin between 15:00 GMT and 16:00 GMT.
>>>>>>> > 
>>>>>>> > 
>>>>>>> > 10/24/2018 11:10:34 GMT -
>>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 11:04:06 GMT From CASE: 
>>>>>>> > 15371425 - Event
>>>>>>> > The Outside Plant Engineer has confirmed the trench is dug and the
>>>>>>> > replacement cable is being ran between manholes at the west side of
>>>>>>> > the damage. Preparations are complete at the east side and splicing
>>>>>>> > expected to begin at 11:30 GMT with the 264 count cable, then the 84
>>>>>>> > count cable followed by a 60 count lateral cable. Restoration efforts
>>>>>>> > are ongoing with an ETTR that remains unchanged, 17:30 GMT.
>>>>>>> > 
>>>>>>> > 
>>>>>>> > 10/24/2018 10:05:30 GMT -
>>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 09:58:52 GMT From CASE: 
>>>>>>> > 15371425 - Event
>>>>>>> > The Outside Plant Engineer reported the replacement section of cable
>>>>>>> > got stuck when pulling it into position. A second splice enclosure
>>>>>>> > will be installed in the next manhole. Field Operations will dig a
>>>>>>> > trench between the two enclosures to complete the installation of the
>>>>>>> > 2,400 foot section of 264 count replacement cable. The trench is
>>>>>>> > expected to be complete at approximately 10:30 GMT with an estimated
>>>>>>> > time splicing will begin of 12:30 GMT.
>>>>>>> > 
>>>>>>> > 
>>>>>>> > 10/24/2018 08:08:37 GMT -
>>>>>>> > *** CASCADED EXTERNAL NOTES 24-Oct-2018 08:01:50 GMT From CASE: 
>>>>>>> > 15371425 - Event
>>>>>>> > The Outside Plant Engineer reported the replacement section of fiber
>>>>>>> > cable has arrived and is currently being pulled into position. Along
>>>>>>> > with the section of fiber cable, a new splice enclosure is being
>>>>>>> > prepped and will be replaced to complete repairs of the 264 count
>>>>>>> > cable. The 84 count cable has been prepped and ready for splicing.
>>>>>>> > Field Operations have moved to an adjacent manhole looking for
>>>>>>> > available slack to repair to the 168 count cable. The Outside Plant
>>>>>>> > Engineer reported splicing is expected to commence at 10:00 GMT. The
>>>>>>> > next update will be sent at that time or if additional information is
>>>>>>> > received.
>>>>>>> > 
>>>>>>> > 
>>>>>>> > -Brian
>>>>>>> > _______________________________________________
>>>>>>> > Outages mailing list
>>>>>>> > Outages at outages.org
>>>>>>> > https://puck.nether.net/mailman/listinfo/outages
>>>>>>> _______________________________________________
>>>>>>> Outages mailing list
>>>>>>> Outages at outages.org
>>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>>> -- 
>>>>>> Mehmet
>>>>>> +1-424-298-1903
>>>>>> _______________________________________________
>>>>>> Outages mailing list
>>>>>> Outages at outages.org
>>>>>> https://puck.nether.net/mailman/listinfo/outages
>>>> -- 
>>>> Mehmet
>>>> +1-424-298-1903
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181219/fccc7405/attachment-0001.html>

From frnkblk at iname.com  Thu Dec 27 10:01:34 2018
From: frnkblk at iname.com (Frank Bulk)
Date: Thu, 27 Dec 2018 09:01:34 -0600
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
Message-ID: <002501d49df5$0f564f50$2e02edf0$@iname.com>

Thanks for sharing.  IPv6 access to www.qwest.com <http://www.qwest.com>
and www.centurylink.com <http://www.centurylink.com>  has been intermittent
since 5:00 am U.S. Central.

 

Frank

 

From: Outages <outages-bounces at outages.org> On Behalf Of Erik Sundberg via
Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org
Subject: [outages] CenturyLink Outages this morning

 

We are seeing a lot of centurylink circuits and services down this morning.

 

Centurylink Control Center portal comes up but auth failing unable to login

 

Waves down or bouncing 

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)


ENNI Down in Denver (Down)

 

Still tying to open tickets with them there portal is not working and stuck
on hold right now.

 

 

  _____  


CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
or previous e-mail messages attached to it may contain confidential
information that is legally privileged. If you are not the intended
recipient, or a person responsible for delivering it to the intended
recipient, you are hereby notified that any disclosure, copying,
distribution or use of any of the information contained in or attached to
this transmission is STRICTLY PROHIBITED. If you have received this
transmission in error please notify the sender immediately by replying to
this e-mail. You must destroy the original transmission and its attachments
without reading or saving in any manner. Thank you.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/87b17397/attachment.html>

From tdensmore at tarpit.cybermesa.com  Thu Dec 27 10:26:11 2018
From: tdensmore at tarpit.cybermesa.com (Tim Densmore)
Date: Thu, 27 Dec 2018 08:26:11 -0700
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <002501d49df5$0f564f50$2e02edf0$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
Message-ID: <1b157eaa-133a-650a-f65c-bddf32be874e@tarpit.cybermesa.com>

An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/32af654f/attachment.html>

From russellzen at outlook.com  Thu Dec 27 11:57:25 2018
From: russellzen at outlook.com (Russell Zen)
Date: Thu, 27 Dec 2018 16:57:25 +0000
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <1b157eaa-133a-650a-f65c-bddf32be874e@tarpit.cybermesa.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>,
 <1b157eaa-133a-650a-f65c-bddf32be874e@tarpit.cybermesa.com>
Message-ID: <BN8PR10MB373170EB89251BDDB8520AC3CFB60@BN8PR10MB3731.namprd10.prod.outlook.com>

FYI, I am on the OG-AWS slackspace and I am getting reports that users in the KS/MO region are impacted by this outage as well. I know we established earlier that this is "probably" a nationwide outage, but we don't know exactly which territories are impacted yet, hence the email.

Since an hour has passed, has anything heard from CenturyLink on current status or next steps?
________________________________
From: Outages-discussion <outages-discussion-bounces at outages.org> on behalf of Tim Densmore <tdensmore at tarpit.cybermesa.com>
Sent: Thursday, December 27, 2018 8:26 AM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

Just a data point, probably not of a hell of a lot of use.  We have a DIA link with them, which was up, getting full routes via BGP, and advertising our routes to the world.  Nothing from us made it past 2 hops into their network, and nothing from outside made it to us.  Assuming routes were being handled via RR through a different path than our data takes or something along those lines.


On 12/27/2018 08:01 AM, Frank Bulk wrote:

Thanks for sharing.  IPv6 access to www.qwest.com<http://www.qwest.com> and www.centurylink.com<http://www.centurylink.com> has been intermittent since 5:00 am U.S. Central.



Frank



From: Outages <outages-bounces at outages.org><mailto:outages-bounces at outages.org> On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] CenturyLink Outages this morning



We are seeing a lot of centurylink circuits and services down this morning.



Centurylink Control Center portal comes up but auth failing unable to login



Waves down or bouncing

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)

ENNI Down in Denver (Down)



Still tying to open tickets with them there portal is not working and stuck on hold right now.





________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.



_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org<mailto:Outages-discussion at outages.org>
https://puck.nether.net/mailman/listinfo/outages-discussion


-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/7dcb7a16/attachment-0001.html>

From mprice at tqhosting.com  Thu Dec 27 12:09:46 2018
From: mprice at tqhosting.com (Mark Price)
Date: Thu, 27 Dec 2018 12:09:46 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <BN8PR10MB373170EB89251BDDB8520AC3CFB60@BN8PR10MB3731.namprd10.prod.outlook.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <1b157eaa-133a-650a-f65c-bddf32be874e@tarpit.cybermesa.com>
 <BN8PR10MB373170EB89251BDDB8520AC3CFB60@BN8PR10MB3731.namprd10.prod.outlook.com>
Message-ID: <CACBmxzd7pevGfEruYxhtqggkCpXfqFa755PEFCOFqxp3Bsh+2Q@mail.gmail.com>

We are also seeing high packet loss and latency on Centurylink (as209) DIA
from west coast to east coast.



On Thu, Dec 27, 2018 at 12:00 PM Russell Zen <russellzen at outlook.com> wrote:

> FYI, I am on the OG-AWS slackspace and I am getting reports that users in
> the KS/MO region are impacted by this outage as well. I know we established
> earlier that this is "probably" a nationwide outage, but we don't know
> exactly which territories are impacted yet, hence the email.
>
> Since an hour has passed, has anything heard from CenturyLink on current
> status or next steps?
> ------------------------------
> *From:* Outages-discussion <outages-discussion-bounces at outages.org> on
> behalf of Tim Densmore <tdensmore at tarpit.cybermesa.com>
> *Sent:* Thursday, December 27, 2018 8:26 AM
> *To:* outages-discussion at outages.org
> *Subject:* Re: [Outages-discussion] CenturyLink Outages this morning
>
> Just a data point, probably not of a hell of a lot of use.  We have a DIA
> link with them, which was up, getting full routes via BGP, and advertising
> our routes to the world.  Nothing from us made it past 2 hops into their
> network, and nothing from outside made it to us.  Assuming routes were
> being handled via RR through a different path than our data takes or
> something along those lines.
>
>
> On 12/27/2018 08:01 AM, Frank Bulk wrote:
>
> Thanks for sharing.  IPv6 access to www.qwest.com and www.centurylink.com
> has been intermittent since 5:00 am U.S. Central.
>
>
>
> Frank
>
>
>
> *From:* Outages <outages-bounces at outages.org>
> <outages-bounces at outages.org> *On Behalf Of *Erik Sundberg via Outages
> *Sent:* Thursday, December 27, 2018 8:45 AM
> *To:* outages at outages.org
> *Subject:* [outages] CenturyLink Outages this morning
>
>
>
> We are seeing a lot of centurylink circuits and services down this morning.
>
>
>
> Centurylink Control Center portal comes up but auth failing unable to login
>
>
>
> Waves down or bouncing
>
> Denver - Seattle (Bouncing)
>
> Denver - Chicago (Bouncing)
>
> New York - Los Angeles (Down)
>
> Chicago - Atlanta (Down)
>
>
> ENNI Down in Denver (Down)
>
>
>
> Still tying to open tickets with them there portal is not working and
> stuck on hold right now.
>
>
>
>
> ------------------------------
>
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>
>
> _______________________________________________
> Outages-discussion mailing listOutages-discussion at outages.orghttps://puck.nether.net/mailman/listinfo/outages-discussion
>
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/c5da0275/attachment.html>

From frnkblk at iname.com  Thu Dec 27 12:26:08 2018
From: frnkblk at iname.com (Frank Bulk)
Date: Thu, 27 Dec 2018 11:26:08 -0600
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <BN8PR14MB313778E20123EF06346A0211C9B60@BN8PR14MB3137.namprd14.prod.outlook.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <BN8PR14MB313778E20123EF06346A0211C9B60@BN8PR14MB3137.namprd14.prod.outlook.com>
Message-ID: <000001d49e09$40f1cdf0$c2d569d0$@iname.com>

I received a report from a business customer in Sioux City, Iowa that they
were have some issues calling out long distance . their local service is
with CTL and their long distance is PIC'd to CTL as well.

 

Frank 

 

From: Outages <outages-bounces at outages.org> On Behalf Of Michael Bolton via
Outages
Sent: Thursday, December 27, 2018 8:55 AM
To: Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
Subject: Re: [outages] CenturyLink Outages this morning

 

Seeing the same in NC. Circuits are up but the routing between POPs has
changed and we are seeing higher than normal latency. 

 

VoIP portal is down also. Allows you to login but you cannot make any
changes or view any details.

 

 

 

From: Outages <outages-bounces at outages.org
<mailto:outages-bounces at outages.org> > On Behalf Of Erik Sundberg via
Outages
Sent: Thursday, December 27, 2018 9:45 AM
To: outages at outages.org <mailto:outages at outages.org> 
Subject: [outages] CenturyLink Outages this morning

 

 

CAUTION: This email originated from outside of the organization. Do not
click links or open attachments unless you recognize the sender and know the
content is safe.

We are seeing a lot of centurylink circuits and services down this morning.

 

Centurylink Control Center portal comes up but auth failing unable to login

 

Waves down or bouncing 

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)


ENNI Down in Denver (Down)

 

Still tying to open tickets with them there portal is not working and stuck
on hold right now.

 

 

  _____  


CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
or previous e-mail messages attached to it may contain confidential
information that is legally privileged. If you are not the intended
recipient, or a person responsible for delivering it to the intended
recipient, you are hereby notified that any disclosure, copying,
distribution or use of any of the information contained in or attached to
this transmission is STRICTLY PROHIBITED. If you have received this
transmission in error please notify the sender immediately by replying to
this e-mail. You must destroy the original transmission and its attachments
without reading or saving in any manner. Thank you.

IMPORTANT NOTICE: This e-mail message is intended to be received only by
persons entitled to receive the confidential information it may contain.
E-mail messages to clients of Holmes Security Systems may contain
information that is confidential and legally privileged. Please do not read,
copy, forward, or store this message unless you are an intended recipient of
it. If you have received this message in error, please forward it to the
sender and delete it completely from your computer system. 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/a0f73279/attachment.html>

From mike.lyon at gmail.com  Thu Dec 27 12:30:17 2018
From: mike.lyon at gmail.com (mike.lyon at gmail.com)
Date: Thu, 27 Dec 2018 09:30:17 -0800
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <000001d49e09$40f1cdf0$c2d569d0$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <BN8PR14MB313778E20123EF06346A0211C9B60@BN8PR14MB3137.namprd14.prod.outlook.com>
 <000001d49e09$40f1cdf0$c2d569d0$@iname.com>
Message-ID: <404164C9-0349-4898-8199-216D27640000@gmail.com>

Our call center in Northern Washington state also cant dial or recieve local or LD calls through their CL link.

Fun times...

-Mike

> On Dec 27, 2018, at 09:26, Frank Bulk <frnkblk at iname.com> wrote:
> 
> I received a report from a business customer in Sioux City, Iowa that they were have some issues calling out long distance ? their local service is with CTL and their long distance is PIC?d to CTL as well.
>  
> Frank
>  
> From: Outages <outages-bounces at outages.org> On Behalf Of Michael Bolton via Outages
> Sent: Thursday, December 27, 2018 8:55 AM
> To: Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
> Subject: Re: [outages] CenturyLink Outages this morning
>  
> Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency.
>  
> VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.
>  
>  
>  
> From: Outages <outages-bounces at outages.org> On Behalf Of Erik Sundberg via Outages
> Sent: Thursday, December 27, 2018 9:45 AM
> To: outages at outages.org
> Subject: [outages] CenturyLink Outages this morning
>  
>  
> CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.
> We are seeing a lot of centurylink circuits and services down this morning.
> 
>  
> 
> Centurylink Control Center portal comes up but auth failing unable to login
> 
>  
> 
> Waves down or bouncing 
> 
> Denver - Seattle (Bouncing)
> 
> Denver - Chicago (Bouncing)
> 
> New York - Los Angeles (Down)
> 
> Chicago - Atlanta (Down)
> 
> 
> ENNI Down in Denver (Down)
> 
>  
> 
> Still tying to open tickets with them there portal is not working and stuck on hold right now.
> 
>  
> 
>  
> 
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
> IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system.
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/d5d3dc5b/attachment-0001.html>

From frnkblk at iname.com  Thu Dec 27 13:18:08 2018
From: frnkblk at iname.com (Frank Bulk)
Date: Thu, 27 Dec 2018 12:18:08 -0600
Subject: [Outages-discussion] [outages] [External]Re: CenturyLink
 Outages this morning
In-Reply-To: <OF5BBF4292.49912E47-ON85258370.006252AD-85258370.00628238@navyfederal.org>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>,
 <BN8PR14MB313778E20123EF06346A0211C9B60@BN8PR14MB3137.namprd14.prod.outlook.com>
 <CY4PR17MB12074A772D9729E2CD37946ED3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <CY4PR14MB139895B989F97269235DA5DCA5B60@CY4PR14MB1398.namprd14.prod.outlook.com>
 <OF5BBF4292.49912E47-ON85258370.006252AD-85258370.00628238@navyfederal.org>
Message-ID: <000a01d49e10$8557dbe0$900793a0$@iname.com>

Interesting, that may also explains why HTTPv6 checks of signup.us-west-2.netflix.com and www.us-west-2.netflix.com <http://www.us-west-2.netflix.com>  have been intermittent from 10:41 to 11:10 am (U.S. Central).

 

Frank 

 

From: Outages <outages-bounces at outages.org> On Behalf Of Rob via Outages
Sent: Thursday, December 27, 2018 11:56 AM
To: outages at outages.org
Subject: Re: [outages] [External]Re: CenturyLink Outages this morning

 

 <https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/> https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ 
 <https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/?utm_source=reddit&utm_medium=usertext&utm_name=sysadmin&utm_content=t1_ecnu814> https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/?utm_source=reddit&utm_medium=usertext&utm_name=sysadmin&utm_content=t1_ecnu814 

from the 2nd thread:

"Spoke to CenturyLink this morning about a downed 10G LAN PHY in WA, the guy I talked to said they had a major DWDM outage causing problems all over the west side of the country. Their own internal ticketing system is currently down, ControlCenter is unreachable from various locations on the west side, all kinds of problems. 
They were working with their DWDM vendor, weren't able to give me an ETR." 

Take it for what it's worth & from where I found it.... on the internet nobody knows you're a dog... 



From:        Jason Baugher via Outages <outages at outages.org <mailto:outages at outages.org> > 
To:        "outages at outages.org <mailto:outages at outages.org> " <outages at outages.org <mailto:outages at outages.org> > 
Date:        12/27/2018 12:53 PM 
Subject:        [External]Re: [outages] CenturyLink Outages this morning 
Sent by:        Outages <outages-bounces at outages.org <mailto:outages-bounces at outages.org> > 

  _____  

 

Anyone seeing indications of this affecting the legacy Qwest LD network? We?re seeing an issue with an SS7 trunk group used for LD, haven?t been able to reach anyone at CenturyLink yet.

 

-Jason

 

From: Outages <outages-bounces at outages.org <mailto:outages-bounces at outages.org> > On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 9:02 AM
To: Michael Bolton <mbolton at holmeselectricsecurity.com <mailto:mbolton at holmeselectricsecurity.com> >; outages at outages.org <mailto:outages at outages.org> 
Subject: Re: [outages] CenturyLink Outages this morning

 


So it's a bad day for centurylink. 
  
The lady who answered the phone for there NOC said that we are having a nation wide outage and we are unable to open tickets at this time. Please call be in 60-90 mins. 
  
She had no info on what was going on 
  

 

  _____  

From: Michael Bolton < <mailto:mbolton at holmeselectricsecurity.com> mbolton at holmeselectricsecurity.com>
Sent: Thursday, December 27, 2018 8:55:03 AM
To: Erik Sundberg;  <mailto:outages at outages.org> outages at outages.org
Subject: RE: CenturyLink Outages this morning 

 

Seeing the same in NC. Circuits are up but the routing between POPs has changed and we are seeing higher than normal latency. 

 

VoIP portal is down also. Allows you to login but you cannot make any changes or view any details.

 

 

 

From: Outages < <mailto:outages-bounces at outages.org> outages-bounces at outages.org> On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 9:45 AM
To:  <mailto:outages at outages.org> outages at outages.org
Subject: [outages] CenturyLink Outages this morning

 

 

CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you recognize the sender and know the content is safe.


We are seeing a lot of centurylink circuits and services down this morning. 
  
Centurylink Control Center portal comes up but auth failing unable to login 
  
Waves down or bouncing 
Denver - Seattle (Bouncing) 
Denver - Chicago (Bouncing) 
New York - Los Angeles (Down) 
Chicago - Atlanta (Down) 

ENNI Down in Denver (Down) 
  
Still tying to open tickets with them there portal is not working and stuck on hold right now. 
  

 

  _____  


CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

IMPORTANT NOTICE: This e-mail message is intended to be received only by persons entitled to receive the confidential information it may contain. E-mail messages to clients of Holmes Security Systems may contain information that is confidential and legally privileged. Please do not read, copy, forward, or store this message unless you are an intended recipient of it. If you have received this message in error, please forward it to the sender and delete it completely from your computer system. 

 

  _____  


CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.



Jason Baugher, Network Operations Manager
405 Emminga Road | PO Box 217 | Golden, IL 62339-0217
P:(217) 696-4411 | F:(217) 696-4811 |  <http://www.adams.net/> www.adams.net

  _____  

The information contained in this email message is PRIVILEGED AND CONFIDENTIAL, and is intended for the use of the addressee and no one else. If you are not the intended recipient, please do not read, distribute, reproduce or use this email message (or the attachments) and notify the sender of the mistaken transmission. Thank you._______________________________________________
Outages mailing list
Outages at outages.org <mailto:Outages at outages.org> 
 <https://puck.nether.net/mailman/listinfo/outages> https://puck.nether.net/mailman/listinfo/outages




-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/01c3c096/attachment.html>

From frnkblk at iname.com  Thu Dec 27 18:42:11 2018
From: frnkblk at iname.com (Frank Bulk)
Date: Thu, 27 Dec 2018 17:42:11 -0600
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <002501d49df5$0f564f50$2e02edf0$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
Message-ID: <001a01d49e3d$c9fc3610$5df4a230$@iname.com>

My IPv6 access to www.qwest.com and www.centurylink.com has been stable
since 3:29 pm U.S. Central.

 

downdetector.com has flatlined for a while, but still not close to zero, and
based on what I see in the reddit thread, the issue is not resolved.

 

Frank 

 

From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf
Of Frank Bulk
Sent: Thursday, December 27, 2018 9:02 AM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

 

Thanks for sharing.  IPv6 access to www.qwest.com <http://www.qwest.com>
and www.centurylink.com <http://www.centurylink.com>  has been intermittent
since 5:00 am U.S. Central.

 

Frank

 

From: Outages <outages-bounces at outages.org
<mailto:outages-bounces at outages.org> > On Behalf Of Erik Sundberg via
Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org <mailto:outages at outages.org> 
Subject: [outages] CenturyLink Outages this morning

 

We are seeing a lot of centurylink circuits and services down this morning.

 

Centurylink Control Center portal comes up but auth failing unable to login

 

Waves down or bouncing 

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)


ENNI Down in Denver (Down)

 

Still tying to open tickets with them there portal is not working and stuck
on hold right now.

 

 

  _____  


CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
or previous e-mail messages attached to it may contain confidential
information that is legally privileged. If you are not the intended
recipient, or a person responsible for delivering it to the intended
recipient, you are hereby notified that any disclosure, copying,
distribution or use of any of the information contained in or attached to
this transmission is STRICTLY PROHIBITED. If you have received this
transmission in error please notify the sender immediately by replying to
this e-mail. You must destroy the original transmission and its attachments
without reading or saving in any manner. Thank you.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/a33a391e/attachment-0001.html>

From JBiddle at ntst.com  Thu Dec 27 19:34:08 2018
From: JBiddle at ntst.com (Biddle, Josh)
Date: Fri, 28 Dec 2018 00:34:08 +0000
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
Message-ID: <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>

Anyone have root cause of Century Link outage and ETTR?

From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 6:42 PM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

My IPv6 access to www.qwest.com<http://www.qwest.com> and www.centurylink.com<http://www.centurylink.com> has been stable since 3:29 pm U.S. Central.

downdetector.com has flatlined for a while, but still not close to zero, and based on what I see in the reddit thread, the issue is not resolved.

Frank

From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 9:02 AM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

Thanks for sharing.  IPv6 access to www.qwest.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=> and www.centurylink.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=> has been intermittent since 5:00 am U.S. Central.

Frank

From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] CenturyLink Outages this morning


We are seeing a lot of centurylink circuits and services down this morning.



Centurylink Control Center portal comes up but auth failing unable to login



Waves down or bouncing

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)

ENNI Down in Denver (Down)



Still tying to open tickets with them there portal is not working and stuck on hold right now.



________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com immediately and permanently delete this email and any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181228/a6ebcd78/attachment.html>

From JBiddle at ntst.com  Thu Dec 27 21:57:42 2018
From: JBiddle at ntst.com (Biddle, Josh)
Date: Fri, 28 Dec 2018 02:57:42 +0000
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <000f01d49e52$797ed890$6c7c89b0$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
Message-ID: <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>

Thanks.  This is one for the books I guess.  I just got in to work a few hours ago so I am playing catch up.  It sounds to me like they still don't know what happened.  In that thread I see multiple reports of circuits going into loopback.  I'm guessing this is CL's way of trying to segregate parts of their core infrastructure in attempt to kill whatever is broadcasting and reconverge their network.  It sounds like they still have not figured out the root cause.

Anyone have any thoughts or updates?


From: frnkblk at iname.com <frnkblk at iname.com>
Sent: Thursday, December 27, 2018 9:10 PM
To: Biddle, Josh <JBiddle at ntst.com>
Subject: RE: [Outages-discussion] CenturyLink Outages this morning

This Reddit post is probably the best/closes to what you're asking for: https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecnsbct/<https://urldefense.proofpoint.com/v2/url?u=https-3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=>

Frank

From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Biddle, Josh
Sent: Thursday, December 27, 2018 6:34 PM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

Anyone have root cause of Century Link outage and ETTR?


From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 6:42 PM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

My IPv6 access to www.qwest.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-DS_zFzzr3sACarc7Az1k7o8&e=> and www.centurylink.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=> has been stable since 3:29 pm U.S. Central.

downdetector.com has flatlined for a while, but still not close to zero, and based on what I see in the reddit thread, the issue is not resolved.

Frank

From: Outages-discussion <outages-discussion-bounces at outages.org<mailto:outages-discussion-bounces at outages.org>> On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 9:02 AM
To: outages-discussion at outages.org<mailto:outages-discussion at outages.org>
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

Thanks for sharing.  IPv6 access to www.qwest.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=> and www.centurylink.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=> has been intermittent since 5:00 am U.S. Central.

Frank

From: Outages <outages-bounces at outages.org<mailto:outages-bounces at outages.org>> On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org<mailto:outages at outages.org>
Subject: [outages] CenturyLink Outages this morning


We are seeing a lot of centurylink circuits and services down this morning.



Centurylink Control Center portal comes up but auth failing unable to login



Waves down or bouncing

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)

ENNI Down in Denver (Down)



Still tying to open tickets with them there portal is not working and stuck on hold right now.



________________________________

CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com<mailto:compliance at NTST.com> immediately and permanently delete this email and any attachments.
This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com immediately and permanently delete this email and any attachments.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181228/82eabc35/attachment-0001.html>

From erik.wooding at wework.com  Thu Dec 27 22:05:48 2018
From: erik.wooding at wework.com (Erik Wooding)
Date: Thu, 27 Dec 2018 19:05:48 -0800
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
 <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
Message-ID: <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>

They're definitely lost. Last troubleshoot was a card issue in Denver which
didn't change anything.

These are the updates from the ticket we have with them if anyone isn't
getting these.

------------------------------------------------------------------------------------------------------------------------

2018-12-28 02:02:14 GMT - Once the card was removed in Denver, CO it was
confirmed that there was no significant improvement. Additional packet
captures, and logs will be pulled from the device with the card removed to
further isolate the root cause. The Equipment vendor continues to work with
CenturyLink Field Operations at multiple sites to remove the secondary
communication channel tunnel across the network until full visibility can
be restored. The equipment vendor has identified a number of additional
nodes that visibility has been restored to, and their engineers are
currently working to apply the necessary filter to each of the reachable
nodes.

2018-12-28 01:03:11 GMT - Following the review of the logs and packet
captures, the Equipment Vendor?s Tier IV Support team has identified a
suspected card issue in Denver, CO.??Field Operations has arrived on site
and are working in cooperation with the Equipment Vendor to remove the
card.

2018-12-28 00:01:53 GMT - The Equipment Vendor is currently reviewing the
logs and packet captures from devices that have been completed, while logs
and packet captures continue to be pulled from additional devices. The
necessary teams continue to remove a secondary communication channel tunnel
across the network until visibility can be restored. All technical teams
continue to diligently work to review the information obtained in an effort
to isolate the root cause.

2018-12-27 22:58:43 GMT - Multiple teams continue work to pull additional
logs and packet captures on devices that have had visibility restored,
which will be scrutinized during root cause analysis. The Tier IV Equipment
Vendor Technical Support team in conjunction with Field Operations are
working to remove a secondary communication channel tunnel across the
network until visibility can be restored. The Equipment Vendor Support team
has dispatched their Field Operations team to the site in Chicago, IL and
has been obtaining data directly from the equipment.

2018-12-27 21:36:58 GMT - It has been advised that visibility has been
restored to both the Chicago, IL and Atlanta, GA sites. Engineering and
Tier IV Equipment Vendor Technical Support are currently working to obtain
additional logs from devices across multiple sites including Chicago and
Atlanta to further isolate the root cause.

2018-12-27 20:19:36 GMT - Tier IV Equipment Vendor Technical Support
continues to work with CenturyLink Field Operations and Engineering to
restore visibility and apply the filter to devices in Atlanta, GA and
Chicago, IL. While those efforts are ongoing additional logs have been
pulled from the devices in Kansas City, MO and New Orleans, LA following
the restoral of visibility and the necessary filter application to obtain
additional pertinent information now that the device is remotely
accessible.

2018-12-27 19:30:01 GMT - Spoke with Sean advised we were seeing system
reestored he verified and advised his other circuit is showing restored at
the same time as this circuit Related to nationwide outage. adding to
parent ticket Sean request we leave this ticket open until they are
handsoff please when you are available could you send us your traceroutes
thank you

2018-12-27 19:17:01 GMT - Efforts to regain visibility to sites in Atlanta,
GA and Chicago, IL remain ongoing. Once visibility has been restored the
filter will be applied to limit communication traffic between sites which
was causing CPU spikes that in turn prevented the devices from functioning
properly.

2018-12-27 18:13:33 GMT - The equipment vendor?s Tier IV Technical Support
team continued to investigate the equipment logs to further assist with
isolation. Field Operations were dispatched to multiple sites to
investigate equipment in Kansas City, MO, New Orleans, LA, as well as
Atlanta, GA. It has been advised that a controller card has been stabilized
in New Orleans, LA restoring visibility to the equipment to allow
additional investigations to continue. A filter was then applied to
equipment in Kansas City, MO that further alleviated the impact observed.
Investigations remain ongoing to further restore network services.

2018-12-27 15:46:56 GMT - Following the isolation of a node in San Antonio,
TX that alleviated some of the impact the necessary teams have shifted
troubleshooting focus to additional nodes experiencing issues. A node in
Atlanta, GA as well as a site in New Orleans, LA are currently being
investigated.

2018-12-27 14:44:42 GMT - Field Operations dispatched to various locations
to troubleshoot cooperatively with the equipment vendor. During cooperative
troubleshooting, a device in San Antonio, TX was seeming to broadcast
traffic consuming capacity and impacting other nodes in the network. The
node was isolated from the network, which appears to have alleviated some
impact; however, troubleshooting efforts continue to restore all impacted
services. We understand how important these services are to our clients and
the issue has been escalated to the highest levels within CenturyLink
Service Assurance Leadership.

2018-12-27 13:50:05 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink
identified a service impact in various locations. The NOC engaged to begin
investigations. The NOC engaged Tier IV Vendor Support to assist in
troubleshooting and fault isolation efforts. The NOC engaged Field
Operations to cooperatively troubleshoot. Field Operations arrived on site
in Denver, CO. Field Operations dispatched additional technicians to Kansas
City, MO and Omaha, NE. The NOC is continuing to cooperatively troubleshoot
with Tier IV Vendor Support for a site in San Antonio, TX.

2018-12-27 13:14:12 GMT - Field Operations has arrived on site at the
Denver location.

2018-12-27 12:49:55 GMT - The NOC has engaged Tier IV Vendor Support to
assist in troubleshooting efforts.

2018-12-27 12:31:34 GMT - Field Operations has dispatched a second
technician to another site to assist with troubleshooting and fault
isolation efforts. An ETA of 13:30 GMT has been provided.

2018-12-27 12:21:46 GMT - The NOC has engaged Field Operations to assist in
troubleshooting and fault isolation efforts. An ETA of 13:15 GMT has been
provided.

 2018-12-27 11:57:40 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink
identified a service impact in New Orleans, LA. The NOC is engaged and
investigating in order to isolate the cause. Please be advised that updates
for this event will be relayed at a minimum of hourly unless otherwise
noted. The information conveyed hereafter is associated to live
troubleshooting effort and as the discovery process evolves through to
service resolution, ticket closure, or post incident review, details may
evolve.

On Thu, Dec 27, 2018 at 6:59 PM Biddle, Josh <JBiddle at ntst.com> wrote:

> Thanks.  This is one for the books I guess.  I just got in to work a few
> hours ago so I am playing catch up.  It sounds to me like they still don?t
> know what happened.  In that thread I see multiple reports of circuits
> going into loopback.  I?m guessing this is CL?s way of trying to segregate
> parts of their core infrastructure in attempt to kill whatever is
> broadcasting and reconverge their network.  It sounds like they still have
> not figured out the root cause.
>
>
>
> Anyone have any thoughts or updates?
>
>
>
>
>
> *From:* frnkblk at iname.com <frnkblk at iname.com>
> *Sent:* Thursday, December 27, 2018 9:10 PM
> *To:* Biddle, Josh <JBiddle at ntst.com>
> *Subject:* RE: [Outages-discussion] CenturyLink Outages this morning
>
>
>
> This Reddit post is probably the best/closes to what you?re asking for:
> https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecnsbct/
> <https://urldefense.proofpoint.com/v2/url?u=https-3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=>
>
>
>
> Frank
>
>
>
> *From:* Outages-discussion <outages-discussion-bounces at outages.org> *On
> Behalf Of *Biddle, Josh
> *Sent:* Thursday, December 27, 2018 6:34 PM
> *To:* outages-discussion at outages.org
> *Subject:* Re: [Outages-discussion] CenturyLink Outages this morning
>
>
>
> Anyone have root cause of Century Link outage and ETTR?
>
>
>
>
>
> *From:* Outages-discussion <outages-discussion-bounces at outages.org> *On
> Behalf Of *Frank Bulk
> *Sent:* Thursday, December 27, 2018 6:42 PM
> *To:* outages-discussion at outages.org
> *Subject:* Re: [Outages-discussion] CenturyLink Outages this morning
>
>
>
> My IPv6 access to www.qwest.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-DS_zFzzr3sACarc7Az1k7o8&e=>
> and www.centurylink.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=>
> has been stable since 3:29 pm U.S. Central.
>
>
>
> downdetector.com has flatlined for a while, but still not close to zero,
> and based on what I see in the reddit thread, the issue is not resolved.
>
>
>
> Frank
>
>
>
> *From:* Outages-discussion <outages-discussion-bounces at outages.org> *On
> Behalf Of *Frank Bulk
> *Sent:* Thursday, December 27, 2018 9:02 AM
> *To:* outages-discussion at outages.org
> *Subject:* Re: [Outages-discussion] CenturyLink Outages this morning
>
>
>
> Thanks for sharing.  IPv6 access to www.qwest.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=>
> and www.centurylink.com
> <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=>
> has been intermittent since 5:00 am U.S. Central.
>
>
>
> Frank
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> Sundberg via Outages
> *Sent:* Thursday, December 27, 2018 8:45 AM
> *To:* outages at outages.org
> *Subject:* [outages] CenturyLink Outages this morning
>
>
>
> We are seeing a lot of centurylink circuits and services down this morning.
>
>
>
> Centurylink Control Center portal comes up but auth failing unable to login
>
>
>
> Waves down or bouncing
>
> Denver - Seattle (Bouncing)
>
> Denver - Chicago (Bouncing)
>
> New York - Los Angeles (Down)
>
> Chicago - Atlanta (Down)
>
>
> ENNI Down in Denver (Down)
>
>
>
> Still tying to open tickets with them there portal is not working and
> stuck on hold right now.
>
>
>
>
> ------------------------------
>
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>
> This email and its attachments may contain privileged and confidential
> information and/or protected health information (PHI) intended solely for
> the use of Netsmart Technologies and the recipient(s) named above. If you
> are not the recipient, or the employee or agent responsible for delivering
> this message to the intended recipient, you are hereby notified that any
> review, dissemination, distribution, printing or copying of this email
> message and/or any attachments is strictly prohibited. If you have received
> this transmission in error, please email compliance at NTST.com immediately
> and permanently delete this email and any attachments.
> This email and its attachments may contain privileged and confidential
> information and/or protected health information (PHI) intended solely for
> the use of Netsmart Technologies and the recipient(s) named above. If you
> are not the recipient, or the employee or agent responsible for delivering
> this message to the intended recipient, you are hereby notified that any
> review, dissemination, distribution, printing or copying of this email
> message and/or any attachments is strictly prohibited. If you have received
> this transmission in error, please email compliance at NTST.com immediately
> and permanently delete this email and any attachments.
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>


-- 
*WeWork | Erik Wooding*
Manager of Network Engineering, US & Canada West, Latin America
O: 917-810-9345
wework.com <http://www.wework.com>

Create Your Life's Work

Get rewarded for good ideas and good people!
Apply for funding at creatorawards.wework.com or
help grow the community at refer.wework.com
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/41d94349/attachment-0001.html>

From frnkblk at iname.com  Thu Dec 27 22:48:20 2018
From: frnkblk at iname.com (frnkblk at iname.com)
Date: Thu, 27 Dec 2018 21:48:20 -0600
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
 <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
Message-ID: <001901d49e60$2e142c30$8a3c8490$@iname.com>

I wonder why they want to remove the secondary communication channel ? is it because they believe it?s helping facilitate the broadcast traffic? Or is to simplify the application of filters? Or simplify troubleshooting of the issue?

 

I don?t know what Infinera platforms are involved with this issue, but Figure 3-24 of this (very old) document shows a Control Path B/secondary control path:

https://fccid.io/ANATEL/01163-09-05381/Manual-02/072CE5D9-762C-49CE-B9F0-FEC86BF9077B/PDF

 

Frank 

 

From: Erik Wooding <erik.wooding at wework.com> 
Sent: Thursday, December 27, 2018 9:06 PM
To: Biddle, Josh <JBiddle at ntst.com>
Cc: frnkblk at iname.com; outages-discussion at outages.org
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

 

They're definitely lost. Last troubleshoot was a card issue in Denver which didn't change anything. 

 

These are the updates from the ticket we have with them if anyone isn't getting these.  

 

------------------------------------------------------------------------------------------------------------------------

 

2018-12-28 02:02:14 GMT - Once the card was removed in Denver, CO it was confirmed that there was no significant improvement. Additional packet captures, and logs will be pulled from the device with the card removed to further isolate the root cause. The Equipment vendor continues to work with CenturyLink Field Operations at multiple sites to remove the secondary communication channel tunnel across the network until full visibility can be restored. The equipment vendor has identified a number of additional nodes that visibility has been restored to, and their engineers are currently working to apply the necessary filter to each of the reachable nodes. 

 

2018-12-28 01:03:11 GMT - Following the review of the logs and packet captures, the Equipment Vendor?s Tier IV Support team has identified a suspected card issue in Denver, CO.??Field Operations has arrived on site and are working in cooperation with the Equipment Vendor to remove the card. 

 

2018-12-28 00:01:53 GMT - The Equipment Vendor is currently reviewing the logs and packet captures from devices that have been completed, while logs and packet captures continue to be pulled from additional devices. The necessary teams continue to remove a secondary communication channel tunnel across the network until visibility can be restored. All technical teams continue to diligently work to review the information obtained in an effort to isolate the root cause. 

 

2018-12-27 22:58:43 GMT - Multiple teams continue work to pull additional logs and packet captures on devices that have had visibility restored, which will be scrutinized during root cause analysis. The Tier IV Equipment Vendor Technical Support team in conjunction with Field Operations are working to remove a secondary communication channel tunnel across the network until visibility can be restored. The Equipment Vendor Support team has dispatched their Field Operations team to the site in Chicago, IL and has been obtaining data directly from the equipment. 

 

2018-12-27 21:36:58 GMT - It has been advised that visibility has been restored to both the Chicago, IL and Atlanta, GA sites. Engineering and Tier IV Equipment Vendor Technical Support are currently working to obtain additional logs from devices across multiple sites including Chicago and Atlanta to further isolate the root cause. 

 

2018-12-27 20:19:36 GMT - Tier IV Equipment Vendor Technical Support continues to work with CenturyLink Field Operations and Engineering to restore visibility and apply the filter to devices in Atlanta, GA and Chicago, IL. While those efforts are ongoing additional logs have been pulled from the devices in Kansas City, MO and New Orleans, LA following the restoral of visibility and the necessary filter application to obtain additional pertinent information now that the device is remotely accessible. 

 

2018-12-27 19:30:01 GMT - Spoke with Sean advised we were seeing system reestored he verified and advised his other circuit is showing restored at the same time as this circuit Related to nationwide outage. adding to parent ticket Sean request we leave this ticket open until they are handsoff please when you are available could you send us your traceroutes thank you 

 

2018-12-27 19:17:01 GMT - Efforts to regain visibility to sites in Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been restored the filter will be applied to limit communication traffic between sites which was causing CPU spikes that in turn prevented the devices from functioning properly. 

 

2018-12-27 18:13:33 GMT - The equipment vendor?s Tier IV Technical Support team continued to investigate the equipment logs to further assist with isolation. Field Operations were dispatched to multiple sites to investigate equipment in Kansas City, MO, New Orleans, LA, as well as Atlanta, GA. It has been advised that a controller card has been stabilized in New Orleans, LA restoring visibility to the equipment to allow additional investigations to continue. A filter was then applied to equipment in Kansas City, MO that further alleviated the impact observed. Investigations remain ongoing to further restore network services. 

 

2018-12-27 15:46:56 GMT - Following the isolation of a node in San Antonio, TX that alleviated some of the impact the necessary teams have shifted troubleshooting focus to additional nodes experiencing issues. A node in Atlanta, GA as well as a site in New Orleans, LA are currently being investigated. 

 

2018-12-27 14:44:42 GMT - Field Operations dispatched to various locations to troubleshoot cooperatively with the equipment vendor. During cooperative troubleshooting, a device in San Antonio, TX was seeming to broadcast traffic consuming capacity and impacting other nodes in the network. The node was isolated from the network, which appears to have alleviated some impact; however, troubleshooting efforts continue to restore all impacted services. We understand how important these services are to our clients and the issue has been escalated to the highest levels within CenturyLink Service Assurance Leadership. 

 

2018-12-27 13:50:05 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in various locations. The NOC engaged to begin investigations. The NOC engaged Tier IV Vendor Support to assist in troubleshooting and fault isolation efforts. The NOC engaged Field Operations to cooperatively troubleshoot. Field Operations arrived on site in Denver, CO. Field Operations dispatched additional technicians to Kansas City, MO and Omaha, NE. The NOC is continuing to cooperatively troubleshoot with Tier IV Vendor Support for a site in San Antonio, TX. 

 

2018-12-27 13:14:12 GMT - Field Operations has arrived on site at the Denver location. 

 

2018-12-27 12:49:55 GMT - The NOC has engaged Tier IV Vendor Support to assist in troubleshooting efforts. 

 

2018-12-27 12:31:34 GMT - Field Operations has dispatched a second technician to another site to assist with troubleshooting and fault isolation efforts. An ETA of 13:30 GMT has been provided. 

 

2018-12-27 12:21:46 GMT - The NOC has engaged Field Operations to assist in troubleshooting and fault isolation efforts. An ETA of 13:15 GMT has been provided.

 

 2018-12-27 11:57:40 GMT - On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact in New Orleans, LA. The NOC is engaged and investigating in order to isolate the cause. Please be advised that updates for this event will be relayed at a minimum of hourly unless otherwise noted. The information conveyed hereafter is associated to live troubleshooting effort and as the discovery process evolves through to service resolution, ticket closure, or post incident review, details may evolve.

 

On Thu, Dec 27, 2018 at 6:59 PM Biddle, Josh <JBiddle at ntst.com <mailto:JBiddle at ntst.com> > wrote:

///Thanks.  This is one for the books I guess.  I just got in to work a few hours ago so I am playing catch up.  It sounds to me like they still don?t know what happened.  In that thread I see multiple reports of circuits going into loopback.  I?m guessing this is CL?s way of trying to segregate parts of their core infrastructure in attempt to kill whatever is broadcasting and reconverge their network.  It sounds like they still have not figured out the root cause. 

 

Anyone have any thoughts or updates?

 

 

From: frnkblk at iname.com <mailto:frnkblk at iname.com>  <frnkblk at iname.com <mailto:frnkblk at iname.com> > 
Sent: Thursday, December 27, 2018 9:10 PM
To: Biddle, Josh <JBiddle at ntst.com <mailto:JBiddle at ntst.com> >
Subject: RE: [Outages-discussion] CenturyLink Outages this morning

 

This Reddit post is probably the best/closes to what you?re asking for: https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecnsbct/ <https://urldefense.proofpoint.com/v2/url?u=https-3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=> 

 

Frank 

 

From: Outages-discussion <outages-discussion-bounces at outages.org <mailto:outages-discussion-bounces at outages.org> > On Behalf Of Biddle, Josh
Sent: Thursday, December 27, 2018 6:34 PM
To: outages-discussion at outages.org <mailto:outages-discussion at outages.org> 
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

 

Anyone have root cause of Century Link outage and ETTR?

 

 

From: Outages-discussion <outages-discussion-bounces at outages.org <mailto:outages-discussion-bounces at outages.org> > On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 6:42 PM
To: outages-discussion at outages.org <mailto:outages-discussion at outages.org> 
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

 

My IPv6 access to www.qwest.com <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-DS_zFzzr3sACarc7Az1k7o8&e=>  and www.centurylink.com <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=>  has been stable since 3:29 pm U.S. Central.

 

downdetector.com <http://downdetector.com>  has flatlined for a while, but still not close to zero, and based on what I see in the reddit thread, the issue is not resolved.

 

Frank 

 

From: Outages-discussion <outages-discussion-bounces at outages.org <mailto:outages-discussion-bounces at outages.org> > On Behalf Of Frank Bulk
Sent: Thursday, December 27, 2018 9:02 AM
To: outages-discussion at outages.org <mailto:outages-discussion at outages.org> 
Subject: Re: [Outages-discussion] CenturyLink Outages this morning

 

Thanks for sharing.  IPv6 access to www.qwest.com <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=>  and www.centurylink.com <https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=>  has been intermittent since 5:00 am U.S. Central.

 

Frank

 

From: Outages <outages-bounces at outages.org <mailto:outages-bounces at outages.org> > On Behalf Of Erik Sundberg via Outages
Sent: Thursday, December 27, 2018 8:45 AM
To: outages at outages.org <mailto:outages at outages.org> 
Subject: [outages] CenturyLink Outages this morning

 

We are seeing a lot of centurylink circuits and services down this morning.

 

Centurylink Control Center portal comes up but auth failing unable to login

 

Waves down or bouncing 

Denver - Seattle (Bouncing)

Denver - Chicago (Bouncing)

New York - Los Angeles (Down)

Chicago - Atlanta (Down)


ENNI Down in Denver (Down)

 

Still tying to open tickets with them there portal is not working and stuck on hold right now.

 

 

  _____  


CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files or previous e-mail messages attached to it may contain confidential information that is legally privileged. If you are not the intended recipient, or a person responsible for delivering it to the intended recipient, you are hereby notified that any disclosure, copying, distribution or use of any of the information contained in or attached to this transmission is STRICTLY PROHIBITED. If you have received this transmission in error please notify the sender immediately by replying to this e-mail. You must destroy the original transmission and its attachments without reading or saving in any manner. Thank you.

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com <mailto:compliance at NTST.com>  immediately and permanently delete this email and any attachments. 

This email and its attachments may contain privileged and confidential information and/or protected health information (PHI) intended solely for the use of Netsmart Technologies and the recipient(s) named above. If you are not the recipient, or the employee or agent responsible for delivering this message to the intended recipient, you are hereby notified that any review, dissemination, distribution, printing or copying of this email message and/or any attachments is strictly prohibited. If you have received this transmission in error, please email compliance at NTST.com <mailto:compliance at NTST.com>  immediately and permanently delete this email and any attachments. 

_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org <mailto:Outages-discussion at outages.org> 
https://puck.nether.net/mailman/listinfo/outages-discussion




 

-- 

	

WeWork | Erik Wooding 
Manager of Network Engineering, US & Canada West, Latin America 
O: 917-810-9345 
 <http://www.wework.com> wework.com 

Create Your Life's Work 

Get rewarded for good ideas and good people! 
Apply for funding at  <http://creatorawards.wework.com/> creatorawards.wework.com or 
help grow the community at  <http://refer.wework.com> refer.wework.com

 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/022e90c4/attachment-0001.html>

From kmedcalf at dessus.com  Thu Dec 27 22:58:27 2018
From: kmedcalf at dessus.com (Keith Medcalf)
Date: Thu, 27 Dec 2018 20:58:27 -0700
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
Message-ID: <0d28195bf0a3e84ca398ae96b8042a48@mail.dessus.com>


One wonders why they do not simply "undo" whatever change they made between when things were "working" and when they "broke" ...


---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.


>-----Original Message-----
>From: Outages-discussion [mailto:outages-discussion-
>bounces at outages.org] On Behalf Of Erik Wooding
>Sent: Thursday, 27 December, 2018 20:06
>To: Biddle, Josh
>Cc: outages-discussion at outages.org
>Subject: Re: [Outages-discussion] CenturyLink Outages this morning
>
>They're definitely lost. Last troubleshoot was a card issue in Denver
>which didn't change anything.
>
>These are the updates from the ticket we have with them if anyone
>isn't getting these.
>
>---------------------------------------------------------------------
>---------------------------------------------------
>
>2018-12-28 02:02:14 GMT - Once the card was removed in Denver, CO it
>was confirmed that there was no significant improvement. Additional
>packet captures, and logs will be pulled from the device with the
>card removed to further isolate the root cause. The Equipment vendor
>continues to work with CenturyLink Field Operations at multiple sites
>to remove the secondary communication channel tunnel across the
>network until full visibility can be restored. The equipment vendor
>has identified a number of additional nodes that visibility has been
>restored to, and their engineers are currently working to apply the
>necessary filter to each of the reachable nodes.
>
>
>2018-12-28 01:03:11 GMT - Following the review of the logs and packet
>captures, the Equipment Vendor?s Tier IV Support team has identified
>a suspected card issue in Denver, CO.??Field Operations has arrived
>on site and are working in cooperation with the Equipment Vendor to
>remove the card.
>
>
>2018-12-28 00:01:53 GMT - The Equipment Vendor is currently reviewing
>the logs and packet captures from devices that have been completed,
>while logs and packet captures continue to be pulled from additional
>devices. The necessary teams continue to remove a secondary
>communication channel tunnel across the network until visibility can
>be restored. All technical teams continue to diligently work to
>review the information obtained in an effort to isolate the root
>cause.
>
>
>2018-12-27 22:58:43 GMT - Multiple teams continue work to pull
>additional logs and packet captures on devices that have had
>visibility restored, which will be scrutinized during root cause
>analysis. The Tier IV Equipment Vendor Technical Support team in
>conjunction with Field Operations are working to remove a secondary
>communication channel tunnel across the network until visibility can
>be restored. The Equipment Vendor Support team has dispatched their
>Field Operations team to the site in Chicago, IL and has been
>obtaining data directly from the equipment.
>
>
>2018-12-27 21:36:58 GMT - It has been advised that visibility has
>been restored to both the Chicago, IL and Atlanta, GA sites.
>Engineering and Tier IV Equipment Vendor Technical Support are
>currently working to obtain additional logs from devices across
>multiple sites including Chicago and Atlanta to further isolate the
>root cause.
>
>
>2018-12-27 20:19:36 GMT - Tier IV Equipment Vendor Technical Support
>continues to work with CenturyLink Field Operations and Engineering
>to restore visibility and apply the filter to devices in Atlanta, GA
>and Chicago, IL. While those efforts are ongoing additional logs have
>been pulled from the devices in Kansas City, MO and New Orleans, LA
>following the restoral of visibility and the necessary filter
>application to obtain additional pertinent information now that the
>device is remotely accessible.
>
>
>2018-12-27 19:30:01 GMT - Spoke with Sean advised we were seeing
>system reestored he verified and advised his other circuit is showing
>restored at the same time as this circuit Related to nationwide
>outage. adding to parent ticket Sean request we leave this ticket
>open until they are handsoff please when you are available could you
>send us your traceroutes thank you
>
>
>2018-12-27 19:17:01 GMT - Efforts to regain visibility to sites in
>Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been
>restored the filter will be applied to limit communication traffic
>between sites which was causing CPU spikes that in turn prevented the
>devices from functioning properly.
>
>
>2018-12-27 18:13:33 GMT - The equipment vendor?s Tier IV Technical
>Support team continued to investigate the equipment logs to further
>assist with isolation. Field Operations were dispatched to multiple
>sites to investigate equipment in Kansas City, MO, New Orleans, LA,
>as well as Atlanta, GA. It has been advised that a controller card
>has been stabilized in New Orleans, LA restoring visibility to the
>equipment to allow additional investigations to continue. A filter
>was then applied to equipment in Kansas City, MO that further
>alleviated the impact observed. Investigations remain ongoing to
>further restore network services.
>
>
>2018-12-27 15:46:56 GMT - Following the isolation of a node in San
>Antonio, TX that alleviated some of the impact the necessary teams
>have shifted troubleshooting focus to additional nodes experiencing
>issues. A node in Atlanta, GA as well as a site in New Orleans, LA
>are currently being investigated.
>
>
>2018-12-27 14:44:42 GMT - Field Operations dispatched to various
>locations to troubleshoot cooperatively with the equipment vendor.
>During cooperative troubleshooting, a device in San Antonio, TX was
>seeming to broadcast traffic consuming capacity and impacting other
>nodes in the network. The node was isolated from the network, which
>appears to have alleviated some impact; however, troubleshooting
>efforts continue to restore all impacted services. We understand how
>important these services are to our clients and the issue has been
>escalated to the highest levels within CenturyLink Service Assurance
>Leadership.
>
>
>2018-12-27 13:50:05 GMT - On December 27, 2018 at 02:40 GMT,
>CenturyLink identified a service impact in various locations. The NOC
>engaged to begin investigations. The NOC engaged Tier IV Vendor
>Support to assist in troubleshooting and fault isolation efforts. The
>NOC engaged Field Operations to cooperatively troubleshoot. Field
>Operations arrived on site in Denver, CO. Field Operations dispatched
>additional technicians to Kansas City, MO and Omaha, NE. The NOC is
>continuing to cooperatively troubleshoot with Tier IV Vendor Support
>for a site in San Antonio, TX.
>
>
>2018-12-27 13:14:12 GMT - Field Operations has arrived on site at the
>Denver location.
>
>
>2018-12-27 12:49:55 GMT - The NOC has engaged Tier IV Vendor Support
>to assist in troubleshooting efforts.
>
>
>2018-12-27 12:31:34 GMT - Field Operations has dispatched a second
>technician to another site to assist with troubleshooting and fault
>isolation efforts. An ETA of 13:30 GMT has been provided.
>
>
>2018-12-27 12:21:46 GMT - The NOC has engaged Field Operations to
>assist in troubleshooting and fault isolation efforts. An ETA of
>13:15 GMT has been provided.
>
>
> 2018-12-27 11:57:40 GMT - On December 27, 2018 at 02:40 GMT,
>CenturyLink identified a service impact in New Orleans, LA. The NOC
>is engaged and investigating in order to isolate the cause. Please be
>advised that updates for this event will be relayed at a minimum of
>hourly unless otherwise noted. The information conveyed hereafter is
>associated to live troubleshooting effort and as the discovery
>process evolves through to service resolution, ticket closure, or
>post incident review, details may evolve.
>
>
>On Thu, Dec 27, 2018 at 6:59 PM Biddle, Josh <JBiddle at ntst.com>
>wrote:
>
>
>	Thanks.  This is one for the books I guess.  I just got in to
>work a few hours ago so I am playing catch up.  It sounds to me like
>they still don?t know what happened.  In that thread I see multiple
>reports of circuits going into loopback.  I?m guessing this is CL?s
>way of trying to segregate parts of their core infrastructure in
>attempt to kill whatever is broadcasting and reconverge their
>network.  It sounds like they still have not figured out the root
>cause.
>
>
>
>	Anyone have any thoughts or updates?
>
>
>
>
>
>	From: frnkblk at iname.com <frnkblk at iname.com>
>	Sent: Thursday, December 27, 2018 9:10 PM
>	To: Biddle, Josh <JBiddle at ntst.com>
>	Subject: RE: [Outages-discussion] CenturyLink Outages this
>morning
>
>
>
>	This Reddit post is probably the best/closes to what you?re
>asking for:
>https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outag
>e_west_coast/ecnsbct/
><https://urldefense.proofpoint.com/v2/url?u=https-
>3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-
>5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-
>7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-
>BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-
>jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=>
>
>
>
>	Frank
>
>
>
>	From: Outages-discussion <outages-discussion-
>bounces at outages.org> On Behalf Of Biddle, Josh
>	Sent: Thursday, December 27, 2018 6:34 PM
>	To: outages-discussion at outages.org
>	Subject: Re: [Outages-discussion] CenturyLink Outages this
>morning
>
>
>
>	Anyone have root cause of Century Link outage and ETTR?
>
>
>
>
>
>	From: Outages-discussion <outages-discussion-
>bounces at outages.org> On Behalf Of Frank Bulk
>	Sent: Thursday, December 27, 2018 6:42 PM
>	To: outages-discussion at outages.org
>	Subject: Re: [Outages-discussion] CenturyLink Outages this
>morning
>
>
>
>	My IPv6 access to www.qwest.com
><https://urldefense.proofpoint.com/v2/url?u=http-
>3A__www.qwest.com&d=DwMFAg&c=-
>7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-
>BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-
>DS_zFzzr3sACarc7Az1k7o8&e=>  and www.centurylink.com
><https://urldefense.proofpoint.com/v2/url?u=http-
>3A__www.centurylink.com&d=DwMFAg&c=-
>7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-
>BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-
>_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=>  has been stable since 3:29
>pm U.S. Central.
>
>
>
>	downdetector.com has flatlined for a while, but still not close
>to zero, and based on what I see in the reddit thread, the issue is
>not resolved.
>
>
>
>	Frank
>
>
>
>	From: Outages-discussion <outages-discussion-
>bounces at outages.org> On Behalf Of Frank Bulk
>	Sent: Thursday, December 27, 2018 9:02 AM
>	To: outages-discussion at outages.org
>	Subject: Re: [Outages-discussion] CenturyLink Outages this
>morning
>
>
>
>	Thanks for sharing.  IPv6 access to www.qwest.com
><https://urldefense.proofpoint.com/v2/url?u=http-
>3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-
>p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gV
>uUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=>  and
>www.centurylink.com <https://urldefense.proofpoint.com/v2/url?u=http-
>3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-
>p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gV
>uUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=>  has
>been intermittent since 5:00 am U.S. Central.
>
>
>
>	Frank
>
>
>
>	From: Outages <outages-bounces at outages.org> On Behalf Of Erik
>Sundberg via Outages
>	Sent: Thursday, December 27, 2018 8:45 AM
>	To: outages at outages.org
>	Subject: [outages] CenturyLink Outages this morning
>
>
>
>	We are seeing a lot of centurylink circuits and services down
>this morning.
>
>
>
>	Centurylink Control Center portal comes up but auth failing
>unable to login
>
>
>
>	Waves down or bouncing
>
>	Denver - Seattle (Bouncing)
>
>	Denver - Chicago (Bouncing)
>
>	New York - Los Angeles (Down)
>
>	Chicago - Atlanta (Down)
>
>
>	ENNI Down in Denver (Down)
>
>
>
>	Still tying to open tickets with them there portal is not
>working and stuck on hold right now.
>
>
>
>
>
>________________________________
>
>
>	CONFIDENTIALITY NOTICE: This e-mail transmission, and any
>documents, files or previous e-mail messages attached to it may
>contain confidential information that is legally privileged. If you
>are not the intended recipient, or a person responsible for
>delivering it to the intended recipient, you are hereby notified that
>any disclosure, copying, distribution or use of any of the
>information contained in or attached to this transmission is STRICTLY
>PROHIBITED. If you have received this transmission in error please
>notify the sender immediately by replying to this e-mail. You must
>destroy the original transmission and its attachments without reading
>or saving in any manner. Thank you.
>
>	This email and its attachments may contain privileged and
>confidential information and/or protected health information (PHI)
>intended solely for the use of Netsmart Technologies and the
>recipient(s) named above. If you are not the recipient, or the
>employee or agent responsible for delivering this message to the
>intended recipient, you are hereby notified that any review,
>dissemination, distribution, printing or copying of this email
>message and/or any attachments is strictly prohibited. If you have
>received this transmission in error, please email compliance at NTST.com
><mailto:compliance at NTST.com>  immediately and permanently delete this
>email and any attachments.
>
>	This email and its attachments may contain privileged and
>confidential information and/or protected health information (PHI)
>intended solely for the use of Netsmart Technologies and the
>recipient(s) named above. If you are not the recipient, or the
>employee or agent responsible for delivering this message to the
>intended recipient, you are hereby notified that any review,
>dissemination, distribution, printing or copying of this email
>message and/or any attachments is strictly prohibited. If you have
>received this transmission in error, please email compliance at NTST.com
>immediately and permanently delete this email and any attachments.
>	_______________________________________________
>	Outages-discussion mailing list
>	Outages-discussion at outages.org
>	https://puck.nether.net/mailman/listinfo/outages-discussion
>
>
>
>
>--
>
>
>WeWork | Erik Wooding
>Manager of Network Engineering, US & Canada West, Latin America
>O: 917-810-9345
>wework.com <http://www.wework.com>
>
>Create Your Life's Work
>
>Get rewarded for good ideas and good people!
>Apply for funding at creatorawards.wework.com
><http://creatorawards.wework.com/>  or
>help grow the community at refer.wework.com





From aaron at heyaaron.com  Thu Dec 27 23:08:49 2018
From: aaron at heyaaron.com (Aaron C. de Bruyn)
Date: Thu, 27 Dec 2018 20:08:49 -0800
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <0d28195bf0a3e84ca398ae96b8042a48@mail.dessus.com>
References: <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
 <0d28195bf0a3e84ca398ae96b8042a48@mail.dessus.com>
Message-ID: <CAEE+rGptE2-QS8aGR11cbaFwzFWGZLsoBAEtVzJLZnZT_PLcvw@mail.gmail.com>

On Thu, Dec 27, 2018 at 7:59 PM Keith Medcalf <kmedcalf at dessus.com> wrote:

> One wonders why they do not simply "undo" whatever change they made
> between when things were "working" and when they "broke" ...
>

That's assuming this was change-related and not some triggered bug in
equipment...


-A
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181227/b581812a/attachment.html>

From sethm at rollernet.us  Thu Dec 27 23:58:21 2018
From: sethm at rollernet.us (Seth Mattinen)
Date: Thu, 27 Dec 2018 20:58:21 -0800
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <001901d49e60$2e142c30$8a3c8490$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
 <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
 <001901d49e60$2e142c30$8a3c8490$@iname.com>
Message-ID: <c6e99cb9-4207-535f-c441-428e61a88da3@rollernet.us>

On 12/27/18 7:48 PM, frnkblk at iname.com wrote:
> I wonder why they want to remove the secondary communication channel ? 
> is it because they believe it?s helping facilitate the broadcast 
> traffic? Or is to simplify the application of filters? Or simplify 
> troubleshooting of the issue?

To me that just sounds like it's reached the point of unplugging things 
and seeing if the problem stops.

From jra at baylink.com  Fri Dec 28 00:11:36 2018
From: jra at baylink.com (Jay Ashworth)
Date: Fri, 28 Dec 2018 00:11:36 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <001901d49e60$2e142c30$8a3c8490$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
 <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
 <001901d49e60$2e142c30$8a3c8490$@iname.com>
Message-ID: <0211B1CB-E31B-4E68-A1E0-FBF3B85C12DE@baylink.com>

I got to tell you, reading back through that ticket, it certainly feels like the big ss7 storm from about 10 years ago. The question is, is it a bug or an attack. 

Or a test.

On December 27, 2018 10:48:20 PM EST, frnkblk at iname.com wrote:
>I wonder why they want to remove the secondary communication channel ?
>is it because they believe it?s helping facilitate the broadcast
>traffic? Or is to simplify the application of filters? Or simplify
>troubleshooting of the issue?
>
> 
>
>I don?t know what Infinera platforms are involved with this issue, but
>Figure 3-24 of this (very old) document shows a Control Path
>B/secondary control path:
>
>https://fccid.io/ANATEL/01163-09-05381/Manual-02/072CE5D9-762C-49CE-B9F0-FEC86BF9077B/PDF
>
> 
>
>Frank 
>
> 
>
>From: Erik Wooding <erik.wooding at wework.com> 
>Sent: Thursday, December 27, 2018 9:06 PM
>To: Biddle, Josh <JBiddle at ntst.com>
>Cc: frnkblk at iname.com; outages-discussion at outages.org
>Subject: Re: [Outages-discussion] CenturyLink Outages this morning
>
> 
>
>They're definitely lost. Last troubleshoot was a card issue in Denver
>which didn't change anything. 
>
> 
>
>These are the updates from the ticket we have with them if anyone isn't
>getting these.  
>
> 
>
>------------------------------------------------------------------------------------------------------------------------
>
> 
>
>2018-12-28 02:02:14 GMT - Once the card was removed in Denver, CO it
>was confirmed that there was no significant improvement. Additional
>packet captures, and logs will be pulled from the device with the card
>removed to further isolate the root cause. The Equipment vendor
>continues to work with CenturyLink Field Operations at multiple sites
>to remove the secondary communication channel tunnel across the network
>until full visibility can be restored. The equipment vendor has
>identified a number of additional nodes that visibility has been
>restored to, and their engineers are currently working to apply the
>necessary filter to each of the reachable nodes. 
>
> 
>
>2018-12-28 01:03:11 GMT - Following the review of the logs and packet
>captures, the Equipment Vendor?s Tier IV Support team has identified a
>suspected card issue in Denver, CO.??Field Operations has arrived on
>site and are working in cooperation with the Equipment Vendor to remove
>the card. 
>
> 
>
>2018-12-28 00:01:53 GMT - The Equipment Vendor is currently reviewing
>the logs and packet captures from devices that have been completed,
>while logs and packet captures continue to be pulled from additional
>devices. The necessary teams continue to remove a secondary
>communication channel tunnel across the network until visibility can be
>restored. All technical teams continue to diligently work to review the
>information obtained in an effort to isolate the root cause. 
>
> 
>
>2018-12-27 22:58:43 GMT - Multiple teams continue work to pull
>additional logs and packet captures on devices that have had visibility
>restored, which will be scrutinized during root cause analysis. The
>Tier IV Equipment Vendor Technical Support team in conjunction with
>Field Operations are working to remove a secondary communication
>channel tunnel across the network until visibility can be restored. The
>Equipment Vendor Support team has dispatched their Field Operations
>team to the site in Chicago, IL and has been obtaining data directly
>from the equipment. 
>
> 
>
>2018-12-27 21:36:58 GMT - It has been advised that visibility has been
>restored to both the Chicago, IL and Atlanta, GA sites. Engineering and
>Tier IV Equipment Vendor Technical Support are currently working to
>obtain additional logs from devices across multiple sites including
>Chicago and Atlanta to further isolate the root cause. 
>
> 
>
>2018-12-27 20:19:36 GMT - Tier IV Equipment Vendor Technical Support
>continues to work with CenturyLink Field Operations and Engineering to
>restore visibility and apply the filter to devices in Atlanta, GA and
>Chicago, IL. While those efforts are ongoing additional logs have been
>pulled from the devices in Kansas City, MO and New Orleans, LA
>following the restoral of visibility and the necessary filter
>application to obtain additional pertinent information now that the
>device is remotely accessible. 
>
> 
>
>2018-12-27 19:30:01 GMT - Spoke with Sean advised we were seeing system
>reestored he verified and advised his other circuit is showing restored
>at the same time as this circuit Related to nationwide outage. adding
>to parent ticket Sean request we leave this ticket open until they are
>handsoff please when you are available could you send us your
>traceroutes thank you 
>
> 
>
>2018-12-27 19:17:01 GMT - Efforts to regain visibility to sites in
>Atlanta, GA and Chicago, IL remain ongoing. Once visibility has been
>restored the filter will be applied to limit communication traffic
>between sites which was causing CPU spikes that in turn prevented the
>devices from functioning properly. 
>
> 
>
>2018-12-27 18:13:33 GMT - The equipment vendor?s Tier IV Technical
>Support team continued to investigate the equipment logs to further
>assist with isolation. Field Operations were dispatched to multiple
>sites to investigate equipment in Kansas City, MO, New Orleans, LA, as
>well as Atlanta, GA. It has been advised that a controller card has
>been stabilized in New Orleans, LA restoring visibility to the
>equipment to allow additional investigations to continue. A filter was
>then applied to equipment in Kansas City, MO that further alleviated
>the impact observed. Investigations remain ongoing to further restore
>network services. 
>
> 
>
>2018-12-27 15:46:56 GMT - Following the isolation of a node in San
>Antonio, TX that alleviated some of the impact the necessary teams have
>shifted troubleshooting focus to additional nodes experiencing issues.
>A node in Atlanta, GA as well as a site in New Orleans, LA are
>currently being investigated. 
>
> 
>
>2018-12-27 14:44:42 GMT - Field Operations dispatched to various
>locations to troubleshoot cooperatively with the equipment vendor.
>During cooperative troubleshooting, a device in San Antonio, TX was
>seeming to broadcast traffic consuming capacity and impacting other
>nodes in the network. The node was isolated from the network, which
>appears to have alleviated some impact; however, troubleshooting
>efforts continue to restore all impacted services. We understand how
>important these services are to our clients and the issue has been
>escalated to the highest levels within CenturyLink Service Assurance
>Leadership. 
>
> 
>
>2018-12-27 13:50:05 GMT - On December 27, 2018 at 02:40 GMT,
>CenturyLink identified a service impact in various locations. The NOC
>engaged to begin investigations. The NOC engaged Tier IV Vendor Support
>to assist in troubleshooting and fault isolation efforts. The NOC
>engaged Field Operations to cooperatively troubleshoot. Field
>Operations arrived on site in Denver, CO. Field Operations dispatched
>additional technicians to Kansas City, MO and Omaha, NE. The NOC is
>continuing to cooperatively troubleshoot with Tier IV Vendor Support
>for a site in San Antonio, TX. 
>
> 
>
>2018-12-27 13:14:12 GMT - Field Operations has arrived on site at the
>Denver location. 
>
> 
>
>2018-12-27 12:49:55 GMT - The NOC has engaged Tier IV Vendor Support to
>assist in troubleshooting efforts. 
>
> 
>
>2018-12-27 12:31:34 GMT - Field Operations has dispatched a second
>technician to another site to assist with troubleshooting and fault
>isolation efforts. An ETA of 13:30 GMT has been provided. 
>
> 
>
>2018-12-27 12:21:46 GMT - The NOC has engaged Field Operations to
>assist in troubleshooting and fault isolation efforts. An ETA of 13:15
>GMT has been provided.
>
> 
>
>2018-12-27 11:57:40 GMT - On December 27, 2018 at 02:40 GMT,
>CenturyLink identified a service impact in New Orleans, LA. The NOC is
>engaged and investigating in order to isolate the cause. Please be
>advised that updates for this event will be relayed at a minimum of
>hourly unless otherwise noted. The information conveyed hereafter is
>associated to live troubleshooting effort and as the discovery process
>evolves through to service resolution, ticket closure, or post incident
>review, details may evolve.
>
> 
>
>On Thu, Dec 27, 2018 at 6:59 PM Biddle, Josh <JBiddle at ntst.com
><mailto:JBiddle at ntst.com> > wrote:
>
>///Thanks.  This is one for the books I guess.  I just got in to work a
>few hours ago so I am playing catch up.  It sounds to me like they
>still don?t know what happened.  In that thread I see multiple reports
>of circuits going into loopback.  I?m guessing this is CL?s way of
>trying to segregate parts of their core infrastructure in attempt to
>kill whatever is broadcasting and reconverge their network.  It sounds
>like they still have not figured out the root cause. 
>
> 
>
>Anyone have any thoughts or updates?
>
> 
>
> 
>
>From: frnkblk at iname.com <mailto:frnkblk at iname.com>  <frnkblk at iname.com
><mailto:frnkblk at iname.com> > 
>Sent: Thursday, December 27, 2018 9:10 PM
>To: Biddle, Josh <JBiddle at ntst.com <mailto:JBiddle at ntst.com> >
>Subject: RE: [Outages-discussion] CenturyLink Outages this morning
>
> 
>
>This Reddit post is probably the best/closes to what you?re asking for:
>https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecnsbct/
><https://urldefense.proofpoint.com/v2/url?u=https-3A__old.reddit.com_r_networking_comments_a9z6tb_centurylink-5Foutage-5Fwest-5Fcoast_ecnsbct_&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=aOKuybu-jzAMU3kMtXlyoHlJowOoZk7j8l5nWAnrqoo&e=>
>
>
> 
>
>Frank 
>
> 
>
>From: Outages-discussion <outages-discussion-bounces at outages.org
><mailto:outages-discussion-bounces at outages.org> > On Behalf Of Biddle,
>Josh
>Sent: Thursday, December 27, 2018 6:34 PM
>To: outages-discussion at outages.org
><mailto:outages-discussion at outages.org> 
>Subject: Re: [Outages-discussion] CenturyLink Outages this morning
>
> 
>
>Anyone have root cause of Century Link outage and ETTR?
>
> 
>
> 
>
>From: Outages-discussion <outages-discussion-bounces at outages.org
><mailto:outages-discussion-bounces at outages.org> > On Behalf Of Frank
>Bulk
>Sent: Thursday, December 27, 2018 6:42 PM
>To: outages-discussion at outages.org
><mailto:outages-discussion at outages.org> 
>Subject: Re: [Outages-discussion] CenturyLink Outages this morning
>
> 
>
>My IPv6 access to www.qwest.com
><https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=Dw0K5qO9ViQuAnU-lsg-DS_zFzzr3sACarc7Az1k7o8&e=>
>and www.centurylink.com
><https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMFAg&c=-7HNwxqfpkdcRXCW8HB54Q&r=svX1Si7sopSBMitBL3bFwQ&m=W4lc_zzFTBgLx4j-BCYt1OS6aM_k2UNM7ApSaARhJ08&s=QlmOIb-_7Xxxz5CMZT6NxzOo9JvY2bcQsR61xQ91dK0&e=>
> has been stable since 3:29 pm U.S. Central.
>
> 
>
>downdetector.com <http://downdetector.com>  has flatlined for a while,
>but still not close to zero, and based on what I see in the reddit
>thread, the issue is not resolved.
>
> 
>
>Frank 
>
> 
>
>From: Outages-discussion <outages-discussion-bounces at outages.org
><mailto:outages-discussion-bounces at outages.org> > On Behalf Of Frank
>Bulk
>Sent: Thursday, December 27, 2018 9:02 AM
>To: outages-discussion at outages.org
><mailto:outages-discussion at outages.org> 
>Subject: Re: [Outages-discussion] CenturyLink Outages this morning
>
> 
>
>Thanks for sharing.  IPv6 access to www.qwest.com
><https://urldefense.proofpoint.com/v2/url?u=http-3A__www.qwest.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=An5q7Z3-RdwFPPQ1wVimppYrcLzYIv3sRw2vDYhT4mw&e=>
>and www.centurylink.com
><https://urldefense.proofpoint.com/v2/url?u=http-3A__www.centurylink.com&d=DwMF-g&c=-7HNwxqfpkdcRXCW8HB54Q&r=54dbn-p1oLZUKMLcLIhi8XEoEqm1EAqlQUibNY0yxyg&m=MZI5frpDmYAKCJ3LImVo75vY0h4gVuUnYEgmLhjxBPc&s=Nu02qgI6ZbcvF86z7lerio9zVCJHCW42q2uf_GMqY7k&e=>
> has been intermittent since 5:00 am U.S. Central.
>
> 
>
>Frank
>
> 
>
>From: Outages <outages-bounces at outages.org
><mailto:outages-bounces at outages.org> > On Behalf Of Erik Sundberg via
>Outages
>Sent: Thursday, December 27, 2018 8:45 AM
>To: outages at outages.org <mailto:outages at outages.org> 
>Subject: [outages] CenturyLink Outages this morning
>
> 
>
>We are seeing a lot of centurylink circuits and services down this
>morning.
>
> 
>
>Centurylink Control Center portal comes up but auth failing unable to
>login
>
> 
>
>Waves down or bouncing 
>
>Denver - Seattle (Bouncing)
>
>Denver - Chicago (Bouncing)
>
>New York - Los Angeles (Down)
>
>Chicago - Atlanta (Down)
>
>
>ENNI Down in Denver (Down)
>
> 
>
>Still tying to open tickets with them there portal is not working and
>stuck on hold right now.
>
> 
>
> 
>
>  _____  
>
>
>CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
>files or previous e-mail messages attached to it may contain
>confidential information that is legally privileged. If you are not the
>intended recipient, or a person responsible for delivering it to the
>intended recipient, you are hereby notified that any disclosure,
>copying, distribution or use of any of the information contained in or
>attached to this transmission is STRICTLY PROHIBITED. If you have
>received this transmission in error please notify the sender
>immediately by replying to this e-mail. You must destroy the original
>transmission and its attachments without reading or saving in any
>manner. Thank you.
>
>This email and its attachments may contain privileged and confidential
>information and/or protected health information (PHI) intended solely
>for the use of Netsmart Technologies and the recipient(s) named above.
>If you are not the recipient, or the employee or agent responsible for
>delivering this message to the intended recipient, you are hereby
>notified that any review, dissemination, distribution, printing or
>copying of this email message and/or any attachments is strictly
>prohibited. If you have received this transmission in error, please
>email compliance at NTST.com <mailto:compliance at NTST.com>  immediately and
>permanently delete this email and any attachments. 
>
>This email and its attachments may contain privileged and confidential
>information and/or protected health information (PHI) intended solely
>for the use of Netsmart Technologies and the recipient(s) named above.
>If you are not the recipient, or the employee or agent responsible for
>delivering this message to the intended recipient, you are hereby
>notified that any review, dissemination, distribution, printing or
>copying of this email message and/or any attachments is strictly
>prohibited. If you have received this transmission in error, please
>email compliance at NTST.com <mailto:compliance at NTST.com>  immediately and
>permanently delete this email and any attachments. 
>
>_______________________________________________
>Outages-discussion mailing list
>Outages-discussion at outages.org <mailto:Outages-discussion at outages.org> 
>https://puck.nether.net/mailman/listinfo/outages-discussion
>
>
>
>
> 
>
>-- 
>
>	
>
>WeWork | Erik Wooding 
>Manager of Network Engineering, US & Canada West, Latin America 
>O: 917-810-9345 
> <http://www.wework.com> wework.com 
>
>Create Your Life's Work 
>
>Get rewarded for good ideas and good people! 
>Apply for funding at  <http://creatorawards.wework.com/>
>creatorawards.wework.com or 
>help grow the community at  <http://refer.wework.com> refer.wework.com
>
> 

-- 
Sent from my Android device with K-9 Mail. Please excuse my brevity.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20181228/adb99a8d/attachment-0001.html>

From jared at puck.nether.net  Fri Dec 28 00:21:04 2018
From: jared at puck.nether.net (Jared Mauch)
Date: Fri, 28 Dec 2018 00:21:04 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <0211B1CB-E31B-4E68-A1E0-FBF3B85C12DE@baylink.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
 <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
 <001901d49e60$2e142c30$8a3c8490$@iname.com>
 <0211B1CB-E31B-4E68-A1E0-FBF3B85C12DE@baylink.com>
Message-ID: <0B024869-7C9A-4F9C-A859-08F2B2123125@puck.nether.net>



> On Dec 28, 2018, at 12:11 AM, Jay Ashworth <jra at baylink.com> wrote:
> 
> I got to tell you, reading back through that ticket, it certainly feels like the big ss7 storm from about 10 years ago. The question is, is it a bug or an attack. 
> 
> Or a test.

<tinfoil hat=on>

Given the DNS, BGP and other issues this year that have resulted in the theft of $$$, if I were a bank or an insurance company of a bank in the impacted QWEST territory I would have each ACH/WIRE/SWIFT transaction monitored and reviewed.

</tinfoil hat>

This smells like software upgrade gone wrong, or someone plugged a cable back in to something else and made a network topology loop via their switch or something similar.

I?m not involved other than as a spectator for $dayJob.  I?ve been in the midst of some crazy stuff before, I don?t envy the people looking into this problem right now.  It?s a bad time to be finding people, and most places effectively shut down between Dec25 and Jan2.

I do feel sorry for the consumers who are stuck behind 209 without an alternative.  I feel bad for those 911 centers that had to give out their direct numbers today.

I also hope the involved vendor(s) and their customer solve the problem soon.  It seems to be intermittent as they troubleshoot more things based on the systems at $dayJob.  It?s certainly driving above average call volume.

- Jared


From jayfar at jayfar.com  Fri Dec 28 00:36:26 2018
From: jayfar at jayfar.com (Jay Farrell)
Date: Fri, 28 Dec 2018 00:36:26 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <0B024869-7C9A-4F9C-A859-08F2B2123125@puck.nether.net>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <002501d49df5$0f564f50$2e02edf0$@iname.com>
 <001a01d49e3d$c9fc3610$5df4a230$@iname.com>
 <CY1PR0701MB2011B3E7B58AE79DD73EDCB6C0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <000f01d49e52$797ed890$6c7c89b0$@iname.com>
 <CY1PR0701MB2011471AA79B4B4FF37062FCC0B70@CY1PR0701MB2011.namprd07.prod.outlook.com>
 <CAGJsP4w4n9mv29uAwBRVP7WoR9za-R8qBzecuOE8FjKwYxmqxA@mail.gmail.com>
 <001901d49e60$2e142c30$8a3c8490$@iname.com>
 <0211B1CB-E31B-4E68-A1E0-FBF3B85C12DE@baylink.com>
 <0B024869-7C9A-4F9C-A859-08F2B2123125@puck.nether.net>
Message-ID: <CAN414Uc5EKj2FC4L4SDAOmwP2A08k=iVjstOXu6qVA46HE3RyQ@mail.gmail.com>

Looks like it'll be all good in less than 4 more hours, according to
this recent tweet:

--quote--

CenturyLink engineers have identified a network element that was
impacting customer services and are addressing the issue in order to
fully restore services. We estimate services will be fully restored
within 4 hours. We apologize for any inconvenience this caused our
customers.
11:47 PM - 27 Dec 2018

https://twitter.com/CenturyLink/status/1078512853664059392


On Fri, Dec 28, 2018 at 12:22 AM Jared Mauch <jared at puck.nether.net> wrote:
>
>
>
> > On Dec 28, 2018, at 12:11 AM, Jay Ashworth <jra at baylink.com> wrote:
> >
> > I got to tell you, reading back through that ticket, it certainly feels like the big ss7 storm from about 10 years ago. The question is, is it a bug or an attack.
> >
> > Or a test.
>
> <tinfoil hat=on>
>
> Given the DNS, BGP and other issues this year that have resulted in the theft of $$$, if I were a bank or an insurance company of a bank in the impacted QWEST territory I would have each ACH/WIRE/SWIFT transaction monitored and reviewed.
>
> </tinfoil hat>
>
> This smells like software upgrade gone wrong, or someone plugged a cable back in to something else and made a network topology loop via their switch or something similar.
>
> I?m not involved other than as a spectator for $dayJob.  I?ve been in the midst of some crazy stuff before, I don?t envy the people looking into this problem right now.  It?s a bad time to be finding people, and most places effectively shut down between Dec25 and Jan2.
>
> I do feel sorry for the consumers who are stuck behind 209 without an alternative.  I feel bad for those 911 centers that had to give out their direct numbers today.
>
> I also hope the involved vendor(s) and their customer solve the problem soon.  It seems to be intermittent as they troubleshoot more things based on the systems at $dayJob.  It?s certainly driving above average call volume.
>
> - Jared
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion

From valdis.kletnieks at vt.edu  Fri Dec 28 01:08:25 2018
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Fri, 28 Dec 2018 01:08:25 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <0d28195bf0a3e84ca398ae96b8042a48@mail.dessus.com>
References: <0d28195bf0a3e84ca398ae96b8042a48@mail.dessus.com>
Message-ID: <16218.1545977305@turing-police.cc.vt.edu>

On Thu, 27 Dec 2018 20:58:27 -0700, "Keith Medcalf" said:
> One wonders why they do not simply "undo" whatever change they made between
> when things were "working" and when they "broke" ...

Sometimes, it's not so simple.  One Boston medical school found out the hard way
in 2002:

"All of this has earned Halamka a considerable measure of renown. For two years
running, InformationWeek named Halamka's IT organization number one among
hospitals in its yearly ranking of innovative IT groups. In September 2002,
CareGroup was ranked 16th on InformationWeek's list of 500.

Two months later, Beth Israel Deaconess experienced one of the worst
health-care IT disasters ever. Over four days, Halamka's network crashed
repeatedly, forcing the hospital to revert to the paper patient-records system
that it had abandoned years ago. Lab reports that doctors normally had in hand
within 45 minutes took as long as five hours to process. The emergency
department diverted traffic for hours during the course of two days.
Ultimately, the hospital's network would have to be completely overhauled."

https://www.computerworld.com/article/2581420/disaster-recovery/all-systems-down.html

Some Cisco marketing stuff, but explains the gory details of how spanning tree failed.

https://community.cisco.com/kxiwq67737/attachments/kxiwq67737/6016-discussions-lan-switching-routing/239903/2/140889-CS401%20-%20All%20systems%20down%20-%20Edwin%20Hoffman%20CSA.pdf


From kmedcalf at dessus.com  Fri Dec 28 02:39:14 2018
From: kmedcalf at dessus.com (Keith Medcalf)
Date: Fri, 28 Dec 2018 00:39:14 -0700
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <16218.1545977305@turing-police.cc.vt.edu>
Message-ID: <2ff1ce51b0f22c428954742cf357c817@mail.dessus.com>


On Thursday, 27 December, 2018 23:08, valdis.kletnieks at vt.edu wrote:
>On Thu, 27 Dec 2018 20:58:27 -0700, "Keith Medcalf" said:

>> One wonders why they do not simply "undo" whatever change they made
>> between when things were "working" and when they "broke" ...

>Sometimes, it's not so simple.  One Boston medical school found out
>the hard way in 2002:

Well, yes.  Some people run (or inherit) shoddy ill-considered and ill-maintained networks that manage to teeter on by luck when they should have fallen to bits long past.  Eventually they do fall apart -- this is the nature of shoddy construction.  Sometimes your structural engineers will tell you how to shore up the bridge before it comes tumbling down.  Sometimes not.  Sometimes the sending in of Engineers to take a look is not desired because of what they may find and being able to claim ignorance is bliss.  That reminds of the one about the hooker and the lawyers.  Never mind.  Off topic.

Are you suggesting that CenturyLink is operating such an ill-conceived network and that it was just working by happenstance and luck and that it was bound to fall apart sooner or later due to its well-known built-in flaws that were never remediated?  

How did you come across this information?  Is it in their Risk Disclosure with the SEC?

---
The fact that there's a Highway to Hell but only a Stairway to Heaven says a lot about anticipated traffic volume.




From jayfar at jayfar.com  Fri Dec 28 06:02:46 2018
From: jayfar at jayfar.com (Jay Farrell)
Date: Fri, 28 Dec 2018 06:02:46 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <2ff1ce51b0f22c428954742cf357c817@mail.dessus.com>
References: <16218.1545977305@turing-police.cc.vt.edu>
 <2ff1ce51b0f22c428954742cf357c817@mail.dessus.com>
Message-ID: <CAN414UdGo8vwOUmRbGoOOt7Yf_8SL-LBs7MdXPLNHmwBxThWeg@mail.gmail.com>

Progress is being made?

Fri Dec 28 2018 04:27 (Eastern Standard Time)
Fri Dec 28 2018 09:27 (UTC)
We're seeing positive progress with our service restoration efforts
and our recovery plan is being accelerated as much as possible. We are
working tirelessly around the clock until restoration is complete. We
apologize for any inconvenience this disruption is causing our
customers.

From valdis.kletnieks at vt.edu  Fri Dec 28 06:58:19 2018
From: valdis.kletnieks at vt.edu (valdis.kletnieks at vt.edu)
Date: Fri, 28 Dec 2018 06:58:19 -0500
Subject: [Outages-discussion] CenturyLink Outages this morning
In-Reply-To: <2ff1ce51b0f22c428954742cf357c817@mail.dessus.com>
References: <2ff1ce51b0f22c428954742cf357c817@mail.dessus.com>
Message-ID: <5510.1545998299@turing-police.cc.vt.edu>

On Fri, 28 Dec 2018 00:39:14 -0700, "Keith Medcalf" said:
> 
> On Thursday, 27 December, 2018 23:08, valdis.kletnieks at vt.edu wrote:
> >On Thu, 27 Dec 2018 20:58:27 -0700, "Keith Medcalf" said:
>
> >> One wonders why they do not simply "undo" whatever change they made
> >> between when things were "working" and when they "broke" ...

> Are you suggesting that CenturyLink is operating such an ill-conceived network and

tl;dr: <insert "One does not simply..." meme here>

No, just pointing out that "simply undo whatever change" may not be as "simply"
as you are trying to make it sound.

I spent a good chunk of 2 days earlier this week wondering why my Fedora box
got an update for a package called glib2, and the login manager started
crashing inside the glib2 shared library.  Slam dunk, right?  Nope. Even after
rolling back that change, and even rebooting, it was still crashing.  Turned
out to be something else entirely. (The morbidly interested can look at bug
numbers 1661952, 1662168, and 1662080 in RedHat's bugzilla)

And I have a good bar story where me, several other co-workers, IBM, SGI,
Brocade, and DDN spent 18 months tracking down an intermittent data corruption
in a data storage system issue that started showing up after a system software
upgrade on a system that that had been in production for a year. Symptoms were
the system reading data from the wrong LUN in the storage array.  Finally
tracked it down to buggy firmware on a 10G ethernet card in several servers - the
same vendor, card, EC, and firmware level that was working like a champ in
several other servers in the same rack.

So forgive me if I look askance at people who say "Simply undo what you changed".



From jdc at koitsu.org  Fri Dec 28 16:38:07 2018
From: jdc at koitsu.org (Jeremy Chadwick)
Date: Fri, 28 Dec 2018 13:38:07 -0800
Subject: [Outages-discussion] [outages] Outages Digest, Vol 127, Issue 31
In-Reply-To: <CAN-OaZHG=QFJqMsRG2U4C+qLt6brAqrv_xnB7ieK_aDjnCCxcw@mail.gmail.com>
References: <mailman.322.1546017142.14206.outages@outages.org>
 <CAGPX-LMJyRi0xJPRSwa8u64Y8Cr42a4Vz4yvLiZ+r7WVnVPH9g@mail.gmail.com>
 <CAN-OaZHG=QFJqMsRG2U4C+qLt6brAqrv_xnB7ieK_aDjnCCxcw@mail.gmail.com>
Message-ID: <20181228213807.GA67793@icarus.home.lan>

Likewise there was this, which has a timeline.  Source is from someone
on HackerNews.  Note there is a hyphen at the end of the URL (don't ask;
DSLR/BBR does that):

https://www.dslreports.com/forum/r32235431-

-- 
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |

On Fri, Dec 28, 2018 at 01:39:32PM -0600, lamrya benrob via Outages wrote:
> This Reddit thread has been pretty informative (you'll have to spend some
> time weeding through the noise but some good insight on things):
> 
> https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/
> 
> Sort the thread by new or old.
> 
> 
> 
> On Fri, Dec 28, 2018 at 12:17 PM Aaron Henderson via Outages <
> outages at outages.org> wrote:
> 
> > My apologies if this message breaks the mailing list, I have never replied
> > to one of these before.
> >
> > I work for a rural ISP, we do not use CenturyLink as an upstream; however,
> > some of our customers are reporting issues since the outage. Management has
> > tasked my office with putting together a timeline/details about the outage
> > to brief upper management. Is there a detailed timeline of events
> > documenting this outage?
> >
> > Thanks,
> >
> > A
> >
> > On Fri, Dec 28, 2018 at 12:13 PM <outages-request at outages.org> wrote:
> >
> >> Send Outages mailing list submissions to
> >>         outages at outages.org
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >>         https://puck.nether.net/mailman/listinfo/outages
> >> or, via email, send a message with subject or body 'help' to
> >>         outages-request at outages.org
> >>
> >> You can reach the person managing the list at
> >>         outages-owner at outages.org
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of Outages digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >>    1. Re: CenturyLink Outages this morning (Aaron C. de Bruyn)
> >>
> >>
> >> ----------------------------------------------------------------------
> >>
> >> Message: 1
> >> Date: Fri, 28 Dec 2018 09:12:03 -0800
> >> From: "Aaron C. de Bruyn" <aaron at heyaaron.com>
> >> To: Dom Colangelo <dom.colangelo at simmons.edu>
> >> Cc: Bob Antia <antia at ba14.com>, outages at outages.org
> >> Subject: Re: [outages] CenturyLink Outages this morning
> >> Message-ID:
> >>         <CAEE+rGqaysPos=
> >> VB_QtS++CBs1bNuRzXcLig0iBmntLQcVVk4w at mail.gmail.com>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Several times late last night we had our weather alert radio activate and
> >> broadcast an unintelligible message about 911 being down.  It wasn't
> >> reception problems. The cadence of the dumb computer voice made it nearly
> >> impossible to understand.
> >>
> >> https://photos.app.goo.gl/M4spwKAkKPnoSrJw9
> >>
> >> The alert came through ~5 hours after 911 went offline everywhere.
> >>
> >> -A
> >>
> >>
> >> On Fri, Dec 28, 2018, 08:44 Dom Colangelo via Outages <
> >> outages at outages.org
> >> wrote:
> >>
> >> > Received a WEA from MEMA in Boston, MA noting that 911 is down for
> >> > cellular users.
> >> >
> >> > Dom Colangelo
> >> >
> >> > Senior Windows Systems Administrator
> >> >
> >> > Infrastructure and Online Services
> >> >
> >> > Simmons University
> >> >
> >> > 617.521.2669
> >> >
> >> >
> >> > On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages <
> >> outages at outages.org>
> >> > wrote:
> >> >
> >> >> Massachusetts emergency management agency (MEMA) sent this out this
> >> >> morning and followed up at 0736 that the issues had been resolved.
> >> >>
> >> >>
> >> >> *Massachusetts Emergency Management Agency*
> >> >> ------------------------------
> >> >>
> >> >> *Massachusetts Emergency Management Agency*
> >> >>
> >> >> *Situational Awareness Statement*
> >> >>
> >> >> *December 28, 2018  7:00 AM*
> >> >>
> >> >>
> >> >>
> >> >> *Re: Cellular 911 Experiencing Technical Difficulties and Sporadic
> >> >> Outages*
> >> >>
> >> >>
> >> >>
> >> >> The Massachusetts Emergency Management Agency is sharing the following
> >> >> information from the Commonwealth Fusion Center (CFC) with our local
> >> >> partners and stakeholders.  MEMA also is in touch with the State 911
> >> >> Department.
> >> >>
> >> >>
> >> >>
> >> >> *From the Commonwealth Fusion Center*
> >> >>
> >> >>
> >> >>
> >> >> *Situation:*
> >> >>
> >> >> A nationwide outage affecting CenturyLink, an Internet communications
> >> >> company, is impacting wireless 911 call capability in various parts of
> >> the
> >> >> country. We believe Massachusetts has been impacted by the outage and
> >> urge
> >> >> residents using cellular phones, if necessary, to contact emergency
> >> >> first-responders by using the full 10-digit number of a public safety
> >> >> answering point emergency call center (PSAP) or their local police
> >> >> department.
> >> >>
> >> >>
> >> >>
> >> >> The Massachusetts State Police operate three regional PSAPs and a
> >> fourth
> >> >> is staffed by State 911 at MSP headquarters in Framingham.
> >> >> Residents using a cell phone to call for emergency services should
> >> first
> >> >> try to dial 911, and if it does not work, instead call one of these
> >> four
> >> >> 10-digit numbers:
> >> >> Shelburne Control PSAP 413-625-8200
> >> >> New Braintree Control PSAP 508-867-1170
> >> >> Northampton Control PSAP 413-586-1508
> >> >> Framingham PSAP 508-820-2121.
> >> >>
> >> >>
> >> >> Residents using cell phones may also, if dialing 911 does not work, may
> >> >> call the 10-digit general number of their local police department.
> >> They may
> >> >> also call the 10-digit number of a State Police Barracks in their area
> >> >> (search for barracks location and phone numbers on the MSP web site at
> >> >> www.mass.gov and search for State Police locations).
> >> >> Other agencies operate additional PSAPs in Massachusetts beyond those
> >> >> listed above.
> >> >>
> >> >>
> >> >>
> >> >> The outage seems to only be affecting wireless 911 calls; landline
> >> calls
> >> >> to 911 appear to be working.
> >> >>
> >> >>
> >> >> CenturyLink has indicted it is working on repairing what the company
> >> >> called a ?network element? that is causing the problem.
> >> >>
> >> >>
> >> >>
> >> >>  *MEMA Operations*
> >> >>
> >> >> The State Emergency Operations Center (SEOC) is currently operating at
> >> >> Level 1 (Steady State Monitoring). MEMA will continue to monitor the
> >> >> situation and will disseminate additional Situational Awareness
> >> Statements
> >> >> as necessary. Cities and towns wishing to share information or request
> >> >> assistance should contact their MEMA Regional office or MEMA?s 24/7
> >> >> communications center at 508-820-2000.
> >> >>
> >> >>  *Stay Informed:*
> >> >>
> >> >> Utilize *Massachusetts Alerts* to receive emergency notifications and
> >> >> information from the Massachusetts Emergency Management Agency and the
> >> >> National Weather Service.  *Massachusetts Alerts* is a free app that is
> >> >> available for Android and iPhones. To learn more about *Massachusetts
> >> >> Alerts*, and for information on how to download the free app onto your
> >> >> smartphone, visit:
> >> >>
> >> http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app.
> >> >>
> >> >> Utilize MEMA?s real-time power outage viewer to stay informed about
> >> >> current power outages in your community and region, and across the
> >> state,
> >> >> including information from utility companies about restoration times:
> >> >> http://mema.mapsonline.net/public.html
> >> >>
> >> >> Utilize MEMA?s live weather radar and forecasting tools:
> >> >> http://www.mass.gov/map-resources
> >> >>
> >> >>
> >> >> *Online Resources:*For additional information and resources, visit:
> >> >> Massachusetts Emergency Management Agency at www.mass.gov/mema
> >> >> MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA
> >> >> MEMA Twitter: @MassEMA <http://twitter.com/massema>
> >> >> Federal Emergency Management Agency at www.fema.gov
> >> >> National Weather Service Boston/Norton at www.weather.gov/boston
> >> >> National Weather Service Albany, NY at www.weather.gov/albany
> >> >> National Weather Service Weather Prediction Center:
> >> >> http://www.wpc.ncep.noaa.gov
> >> >> National Weather Service Storm Prediction Center:
> >> >> http://www.spc.noaa.gov/
> >> >> Mass211 at www.mass211.org
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ------------------------------
> >> >>
> >> >> This message has been sent from MEMA's ESF Team Listserv.
> >> >> To unsubscribe from this listserv, call MEMA Operations at (508)
> >> 820-2000.
> >> >>
> >> >> +1.781.300.0000
> >> >>
> >> >>
> >> >> On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages <
> >> >> outages at outages.org> wrote:
> >> >>
> >> >>> Most public safety agencies in Massachusetts posted on Facebook and
> >> >>> Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by
> >> >>> CenturyLink. It was affecting wireless routing, so some calls were
> >> being
> >> >>> misrouted to other towns, or even other states; some callers were
> >> getting
> >> >>> reorder tones or other error messages. The problem was reported to be
> >> >>> resolved prior to 8 AM EST.
> >> >>>
> >> >>> Seeing reports from PSAP's that the issue started again in the last 15
> >> >>> minutes.
> >> >>>
> >> >>> This is wireless only. Landline 911 calls were / are not affected.
> >> >>>
> >> >>>
> >> >>> On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <
> >> >>> outages at outages.org> wrote:
> >> >>>
> >> >>>> Don't be a fear monger. I searched Facebook and found *three* posts
> >> >>>> about 911 outages, all of them simply reposts of local news stories.
> >> >>>>
> >> >>>> Facebook is not "blowing up".
> >> >>>>
> >> >>>> Sheesh!
> >> >>>>
> >> >>>> [image: image1.PNG]
> >> >>>>
> >> >>>> -mel via cell
> >> >>>>
> >> >>>> -mel via cell
> >> >>>>
> >> >>>> On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <
> >> >>>> outages at outages.org> wrote:
> >> >>>>
> >> >>>> Facebook is blowing up with 911 outages and claiming it's CenturyLink
> >> >>>> to blame.
> >> >>>>
> >> >>>> Josh Luthman
> >> >>>> Office: 937-552-2340
> >> >>>> Direct: 937-552-2343
> >> >>>> 1100 Wayne St
> >> >>>> Suite 1337
> >> >>>> Troy, OH 45373
> >> >>>>
> >> >>>> On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <
> >> >>>> outages at outages.org wrote:
> >> >>>>
> >> >>>>> Maybe light at the end of the tunnel...
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> latest we got from Centurylink was at 10:45pm CST.
> >> >>>>>
> >> >>>>> Our engineers and technicians have identified the network element
> >> that
> >> >>>>> has affected our customer services. Services are restoring, and the
> >> current
> >> >>>>> estimated time for full recovery is four hours.
> >> >>>>>
> >> >>>>> CenturyLink will be conducting an extensive post-incident
> >> >>>>> investigation and root cause analysis to provide follow-up
> >> information to
> >> >>>>> our customers impacted by this event. We will provide one more
> >> Sales Alert
> >> >>>>> from this address once services are fully restored.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Rich Breton <richdbreton at gmail.com>
> >> >>>>> *Sent:* Thursday, December 27, 2018 2:59 PM
> >> >>>>> *To:* Christopher Conley <CConley at forsmarshgroup.com>
> >> >>>>> *Cc:* Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> I understand that the transport waves went out first, and then they
> >> >>>>> try to reroute to something in Arizona that begin introducing all
> >> of the
> >> >>>>> broadcast traffic. How legit that is I?m not sure
> >> >>>>>
> >> >>>>> Sent from my iPhone
> >> >>>>>
> >> >>>>>
> >> >>>>> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <
> >> >>>>> outages at outages.org> wrote:
> >> >>>>>
> >> >>>>> > Finally received something useful from centurylink? I have more
> >> >>>>> questions now like how broadcast traffic effect 4x Transport waves.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Ineptitude.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Christopher Conley
> >> >>>>>
> >> >>>>> Systems Administrator |  Fors Marsh Group
> >> >>>>>
> >> >>>>> Arlington, VA
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >> >>>>> Sundberg via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 2:46 PM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Finally received something useful from centurylink? I have more
> >> >>>>> questions now like how broadcast traffic effect 4x Transport waves.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Update:
> >> >>>>>
> >> >>>>> On December 27, 2018 at 02:40 GMT, CenturyLink identified a service
> >> >>>>> impact in New Orleans, LA. The NOC is engaged and investigating in
> >> order to
> >> >>>>> isolate the cause. Field Operations were engaged and dispatched for
> >> >>>>> additional investigations. Tier IV Equipment Vendor Support was
> >> later
> >> >>>>> engaged. During cooperative troubleshooting a device in San
> >> Antonio, TX was
> >> >>>>> isolated from the network as it was seeming to broadcast traffic
> >> consuming
> >> >>>>> capacity, which seemed to alleviate some impact. Investigations
> >> remained
> >> >>>>> ongoing. Following the isolation of the San Antonio, TX device
> >> >>>>> troubleshooting efforts focused on additional sites that teams were
> >> >>>>> remotely unable to troubleshoot. Field Operations were dispatched
> >> to sites
> >> >>>>> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL.
> >> Tier IV
> >> >>>>> Equipment Vendor Support continued to investigate the equipment
> >> logs to
> >> >>>>> further assist with isolation. Once visibility was restored to the
> >> site in
> >> >>>>> Kansas City, MO a filter was applied to the equipment to further
> >> alleviate
> >> >>>>> the impact observed. All of the necessary troubleshooting teams in
> >> >>>>> cooperation with Tier IV Equipment Vendor Support are working to
> >> restore
> >> >>>>> remote visibility to the remaining sites at this time. We
> >> understand how
> >> >>>>> important these services are to our clients and the issue has been
> >> >>>>> escalated to the highest levels within CenturyLink Service Assurance
> >> >>>>> Leadership.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Bruce
> >> >>>>> Freshwater via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 1:24 PM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> CenturyLink finally acknowledged:
> >> >>>>>
> >> >>>>> Great Lakes Region CenturyLink Partners/Agents:
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> FYI. Our network is experiencing a disruption affecting customer
> >> >>>>> services. Customers are currently unable to generate tickets
> >> through the
> >> >>>>> Control Center portal.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> We know how important these services are to our customers, and we
> >> are
> >> >>>>> working to restore services as quickly as possible.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Our technical teams are working to resolve the issue, and we will
> >> >>>>> provide updates as the situation progresses.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> -bruce
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages"
> >> <
> >> >>>>> outages at outages.org> wrote:
> >> >>>>>
> >> >>>>> So it's a bad day for centurylink.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> The lady who answered the phone for there NOC said that we are
> >> having
> >> >>>>> a nation wide outage and we are unable to open tickets at this
> >> time. Please
> >> >>>>> call be in 60-90 mins.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> She had no info on what was going on
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>> *From:* Michael Bolton <mbolton at holmeselectricsecurity.com>
> >> >>>>> *Sent:* Thursday, December 27, 2018 8:55:03 AM
> >> >>>>> *To:* Erik Sundberg; outages at outages.org
> >> >>>>> *Subject:* RE: CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Seeing the same in NC. Circuits are up but the routing between POPs
> >> >>>>> has changed and we are seeing higher than normal latency.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> VoIP portal is down also. Allows you to login but you cannot make
> >> any
> >> >>>>> changes or view any details.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >> >>>>> Sundberg via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 9:45 AM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *CAUTION:* This email originated from outside of the organization.
> >> Do
> >> >>>>> not click links or open attachments unless you recognize the sender
> >> and
> >> >>>>> know the content is safe.
> >> >>>>>
> >> >>>>> We are seeing a lot of centurylink circuits and services down this
> >> >>>>> morning.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Centurylink Control Center portal comes up but auth failing unable
> >> to
> >> >>>>> login
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Waves down or bouncing
> >> >>>>>
> >> >>>>> Denver - Seattle (Bouncing)
> >> >>>>>
> >> >>>>> Denver - Chicago (Bouncing)
> >> >>>>>
> >> >>>>> New York - Los Angeles (Down)
> >> >>>>>
> >> >>>>> Chicago - Atlanta (Down)
> >> >>>>>
> >> >>>>>
> >> >>>>> ENNI Down in Denver (Down)
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Still tying to open tickets with them there portal is not working
> >> and
> >> >>>>> stuck on hold right now.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the intended
> >> >>>>> recipient, or a person responsible for delivering it to the intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>> IMPORTANT NOTICE: This e-mail message is intended to be received
> >> only
> >> >>>>> by persons entitled to receive the confidential information it may
> >> contain.
> >> >>>>> E-mail messages to clients of Holmes Security Systems may contain
> >> >>>>> information that is confidential and legally privileged. Please do
> >> not
> >> >>>>> read, copy, forward, or store this message unless you are an
> >> intended
> >> >>>>> recipient of it. If you have received this message in error, please
> >> forward
> >> >>>>> it to the sender and delete it completely from your computer system.
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the intended
> >> >>>>> recipient, or a person responsible for delivering it to the intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the intended
> >> >>>>> recipient, or a person responsible for delivering it to the intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>> _______________________________________________
> >> >>>>> Outages mailing list
> >> >>>>> Outages at outages.org
> >> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the intended
> >> >>>>> recipient, or a person responsible for delivering it to the intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>> _______________________________________________
> >> >>>>> Outages mailing list
> >> >>>>> Outages at outages.org
> >> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>>
> >> >>>> _______________________________________________
> >> >>>> Outages mailing list
> >> >>>> Outages at outages.org
> >> >>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> Outages mailing list
> >> >>>> Outages at outages.org
> >> >>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>
> >> >>> _______________________________________________
> >> >>> Outages mailing list
> >> >>> Outages at outages.org
> >> >>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>
> >> >> _______________________________________________
> >> >> Outages mailing list
> >> >> Outages at outages.org
> >> >> https://puck.nether.net/mailman/listinfo/outages
> >> >>
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org
> >> > https://puck.nether.net/mailman/listinfo/outages
> >> >
> >> -------------- next part --------------
> >> An HTML attachment was scrubbed...
> >> URL: <
> >> https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/attachment.html
> >> >
> >> -------------- next part --------------
> >> A non-text attachment was scrubbed...
> >> Name: image1.PNG
> >> Type: image/png
> >> Size: 55125 bytes
> >> Desc: not available
> >> URL: <
> >> https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/attachment.png
> >> >
> >>
> >> ------------------------------
> >>
> >> Subject: Digest Footer
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >> ------------------------------
> >>
> >> End of Outages Digest, Vol 127, Issue 31
> >> ****************************************
> >>
> >
> >
> > --
> > *Aaron K. Henderson*
> >
> > Agri-Valley Communications, Inc.
> > Network Technician
> > *O:* (989) 453-4400 <9894534400>
> > *C:* (989) 551-0060 <9895510060>
> > *netadmin at avci.net <netadmin at avci.net> *
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >

> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages


From jay at west.net  Fri Dec 28 19:16:25 2018
From: jay at west.net (Jay Hennigan)
Date: Fri, 28 Dec 2018 16:16:25 -0800
Subject: [Outages-discussion] [outages] Centurylink outage and list best
 practice
In-Reply-To: <0dc77a48-a4c4-73af-9427-12f5b35fc9b0@west.net>
References: <0dc77a48-a4c4-73af-9427-12f5b35fc9b0@west.net>
Message-ID: <918c4abd-6cb0-f07f-826b-fc52a61eb4c8@west.net>

On 12/28/18 10:11 AM, Aaron Henderson via Outages wrote:
> My apologies if this message breaks the mailing list, I have never 
> replied to one of these before.

A few basics:

* Don't reply to an existing topic to start a new topic. Just compose a 
new email with an appropriate subject line and address it to the list.

* If you're subscribed in digest form and are replying to a specific 
topic in the digest, change your subject line to that topic. Don't leave 
it as, "Outages digest..."

* Don't quote more of the original message than is pertinent to your 
reply. In particular, don't ever quote an entire digest to comment about 
one topic within it.

* The outages list is for active reporting and timely updates of new, 
existing and ongoing significant outages only. Many network operators 
have the outages list tied to an alerting system such as a pager or 
smartphone alert. This list is intended as an early warning system of 
problems with major infrastructure. It's also worldwide. Something at 4 
PM in your timezone will be at 4 AM to others monitoring the list.

* There is a parallel list, outages-discussion at outages.org, for 
discussion, follow-up, post-mortem, links to RFO and the like. 
Discussion regarding previously reported outages should take place on 
outages-discussion. I've moved my reply to you there as well as replying 
to you directly. You probably want to subscribe to both to get the full 
picture because questions and followups regarding a previously reported 
issue should always go to outages-discussion. This reduces noise on the 
"outages" list and keeps it useful as an early warning tool while 
allowing commentary on the discussion list.

> I work for a rural ISP, we do not use CenturyLink as an upstream; 
> however, some of our customers are reporting issues since the outage. 
> Management has tasked my office with putting together a timeline/details 
> about the outage to brief upper management. Is there a detailed timeline 
> of events documenting this outage?

There will probably be an official statement from Centurylink within the 
next few days. Keep in mind that official company statements and RFOs, 
especially from a company as large as Centurylink and an outage as 
significant as this one, start with the technical people who know what 
really happened. They are then often filtered, massaged, and wordsmithed 
through an array of lawyers, marketing people, and assorted spin doctors 
before being released to the public. Take these with several grains of salt.

The reason that your customers were impacted is that this was a large 
outage affecting a major transport provider over an extended period of 
time. Any sites or resources that your customers were trying to reach 
that required passing through Centurylink facilities would have been 
down or impaired.

I suggest that you subscribe to outages-discussion and review the 
archives for information there as well as the links therein pointing to 
Reddit and other operators' experience.

-- 
Jay Hennigan - jay at west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV

From frnkblk at iname.com  Sat Dec 29 11:59:15 2018
From: frnkblk at iname.com (Frank Bulk)
Date: Sat, 29 Dec 2018 10:59:15 -0600
Subject: [Outages-discussion] [outages] Outages Digest, Vol 127, Issue 31
In-Reply-To: <20181228213807.GA67793@icarus.home.lan>
References: <mailman.322.1546017142.14206.outages@outages.org>
 <CAGPX-LMJyRi0xJPRSwa8u64Y8Cr42a4Vz4yvLiZ+r7WVnVPH9g@mail.gmail.com>
 <CAN-OaZHG=QFJqMsRG2U4C+qLt6brAqrv_xnB7ieK_aDjnCCxcw@mail.gmail.com>
 <20181228213807.GA67793@icarus.home.lan>
Message-ID: <003301d49f97$d57045d0$8050d170$@iname.com>

Initial RCA here:
https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_
coast/ecszjax/

Frank 

-----Original Message-----
From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf
Of Jeremy Chadwick
Sent: Friday, December 28, 2018 3:38 PM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] [outages] Outages Digest, Vol 127, Issue
31

Likewise there was this, which has a timeline.  Source is from someone
on HackerNews.  Note there is a hyphen at the end of the URL (don't ask;
DSLR/BBR does that):

https://www.dslreports.com/forum/r32235431-

-- 
| Jeremy Chadwick                                 jdc at koitsu.org |
| UNIX Systems Administrator                      PGP 0x2A389531 |
| Making life hard for others since 1977.                        |

On Fri, Dec 28, 2018 at 01:39:32PM -0600, lamrya benrob via Outages wrote:
> This Reddit thread has been pretty informative (you'll have to spend some
> time weeding through the noise but some good insight on things):
> 
>
https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_
coast/
> 
> Sort the thread by new or old.
> 
> 
> 
> On Fri, Dec 28, 2018 at 12:17 PM Aaron Henderson via Outages <
> outages at outages.org> wrote:
> 
> > My apologies if this message breaks the mailing list, I have never
replied
> > to one of these before.
> >
> > I work for a rural ISP, we do not use CenturyLink as an upstream;
however,
> > some of our customers are reporting issues since the outage. Management
has
> > tasked my office with putting together a timeline/details about the
outage
> > to brief upper management. Is there a detailed timeline of events
> > documenting this outage?
> >
> > Thanks,
> >
> > A
> >
> > On Fri, Dec 28, 2018 at 12:13 PM <outages-request at outages.org> wrote:
> >
> >> Send Outages mailing list submissions to
> >>         outages at outages.org
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >>         https://puck.nether.net/mailman/listinfo/outages
> >> or, via email, send a message with subject or body 'help' to
> >>         outages-request at outages.org
> >>
> >> You can reach the person managing the list at
> >>         outages-owner at outages.org
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of Outages digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >>    1. Re: CenturyLink Outages this morning (Aaron C. de Bruyn)
> >>
> >>
> >> ----------------------------------------------------------------------
> >>
> >> Message: 1
> >> Date: Fri, 28 Dec 2018 09:12:03 -0800
> >> From: "Aaron C. de Bruyn" <aaron at heyaaron.com>
> >> To: Dom Colangelo <dom.colangelo at simmons.edu>
> >> Cc: Bob Antia <antia at ba14.com>, outages at outages.org
> >> Subject: Re: [outages] CenturyLink Outages this morning
> >> Message-ID:
> >>         <CAEE+rGqaysPos=
> >> VB_QtS++CBs1bNuRzXcLig0iBmntLQcVVk4w at mail.gmail.com>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Several times late last night we had our weather alert radio activate
and
> >> broadcast an unintelligible message about 911 being down.  It wasn't
> >> reception problems. The cadence of the dumb computer voice made it
nearly
> >> impossible to understand.
> >>
> >> https://photos.app.goo.gl/M4spwKAkKPnoSrJw9
> >>
> >> The alert came through ~5 hours after 911 went offline everywhere.
> >>
> >> -A
> >>
> >>
> >> On Fri, Dec 28, 2018, 08:44 Dom Colangelo via Outages <
> >> outages at outages.org
> >> wrote:
> >>
> >> > Received a WEA from MEMA in Boston, MA noting that 911 is down for
> >> > cellular users.
> >> >
> >> > Dom Colangelo
> >> >
> >> > Senior Windows Systems Administrator
> >> >
> >> > Infrastructure and Online Services
> >> >
> >> > Simmons University
> >> >
> >> > 617.521.2669
> >> >
> >> >
> >> > On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages <
> >> outages at outages.org>
> >> > wrote:
> >> >
> >> >> Massachusetts emergency management agency (MEMA) sent this out this
> >> >> morning and followed up at 0736 that the issues had been resolved.
> >> >>
> >> >>
> >> >> *Massachusetts Emergency Management Agency*
> >> >> ------------------------------
> >> >>
> >> >> *Massachusetts Emergency Management Agency*
> >> >>
> >> >> *Situational Awareness Statement*
> >> >>
> >> >> *December 28, 2018  7:00 AM*
> >> >>
> >> >>
> >> >>
> >> >> *Re: Cellular 911 Experiencing Technical Difficulties and Sporadic
> >> >> Outages*
> >> >>
> >> >>
> >> >>
> >> >> The Massachusetts Emergency Management Agency is sharing the
following
> >> >> information from the Commonwealth Fusion Center (CFC) with our local
> >> >> partners and stakeholders.  MEMA also is in touch with the State 911
> >> >> Department.
> >> >>
> >> >>
> >> >>
> >> >> *From the Commonwealth Fusion Center*
> >> >>
> >> >>
> >> >>
> >> >> *Situation:*
> >> >>
> >> >> A nationwide outage affecting CenturyLink, an Internet
communications
> >> >> company, is impacting wireless 911 call capability in various parts
of
> >> the
> >> >> country. We believe Massachusetts has been impacted by the outage
and
> >> urge
> >> >> residents using cellular phones, if necessary, to contact emergency
> >> >> first-responders by using the full 10-digit number of a public
safety
> >> >> answering point emergency call center (PSAP) or their local police
> >> >> department.
> >> >>
> >> >>
> >> >>
> >> >> The Massachusetts State Police operate three regional PSAPs and a
> >> fourth
> >> >> is staffed by State 911 at MSP headquarters in Framingham.
> >> >> Residents using a cell phone to call for emergency services should
> >> first
> >> >> try to dial 911, and if it does not work, instead call one of these
> >> four
> >> >> 10-digit numbers:
> >> >> Shelburne Control PSAP 413-625-8200
> >> >> New Braintree Control PSAP 508-867-1170
> >> >> Northampton Control PSAP 413-586-1508
> >> >> Framingham PSAP 508-820-2121.
> >> >>
> >> >>
> >> >> Residents using cell phones may also, if dialing 911 does not work,
may
> >> >> call the 10-digit general number of their local police department.
> >> They may
> >> >> also call the 10-digit number of a State Police Barracks in their
area
> >> >> (search for barracks location and phone numbers on the MSP web site
at
> >> >> www.mass.gov and search for State Police locations).
> >> >> Other agencies operate additional PSAPs in Massachusetts beyond
those
> >> >> listed above.
> >> >>
> >> >>
> >> >>
> >> >> The outage seems to only be affecting wireless 911 calls; landline
> >> calls
> >> >> to 911 appear to be working.
> >> >>
> >> >>
> >> >> CenturyLink has indicted it is working on repairing what the company
> >> >> called a ?network element? that is causing the problem.
> >> >>
> >> >>
> >> >>
> >> >>  *MEMA Operations*
> >> >>
> >> >> The State Emergency Operations Center (SEOC) is currently operating
at
> >> >> Level 1 (Steady State Monitoring). MEMA will continue to monitor the
> >> >> situation and will disseminate additional Situational Awareness
> >> Statements
> >> >> as necessary. Cities and towns wishing to share information or
request
> >> >> assistance should contact their MEMA Regional office or MEMA?s 24/7
> >> >> communications center at 508-820-2000.
> >> >>
> >> >>  *Stay Informed:*
> >> >>
> >> >> Utilize *Massachusetts Alerts* to receive emergency notifications
and
> >> >> information from the Massachusetts Emergency Management Agency and
the
> >> >> National Weather Service.  *Massachusetts Alerts* is a free app that
is
> >> >> available for Android and iPhones. To learn more about
*Massachusetts
> >> >> Alerts*, and for information on how to download the free app onto
your
> >> >> smartphone, visit:
> >> >>
> >>
http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app.
> >> >>
> >> >> Utilize MEMA?s real-time power outage viewer to stay informed about
> >> >> current power outages in your community and region, and across the
> >> state,
> >> >> including information from utility companies about restoration
times:
> >> >> http://mema.mapsonline.net/public.html
> >> >>
> >> >> Utilize MEMA?s live weather radar and forecasting tools:
> >> >> http://www.mass.gov/map-resources
> >> >>
> >> >>
> >> >> *Online Resources:*For additional information and resources, visit:
> >> >> Massachusetts Emergency Management Agency at www.mass.gov/mema
> >> >> MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA
> >> >> MEMA Twitter: @MassEMA <http://twitter.com/massema>
> >> >> Federal Emergency Management Agency at www.fema.gov
> >> >> National Weather Service Boston/Norton at www.weather.gov/boston
> >> >> National Weather Service Albany, NY at www.weather.gov/albany
> >> >> National Weather Service Weather Prediction Center:
> >> >> http://www.wpc.ncep.noaa.gov
> >> >> National Weather Service Storm Prediction Center:
> >> >> http://www.spc.noaa.gov/
> >> >> Mass211 at www.mass211.org
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ------------------------------
> >> >>
> >> >> This message has been sent from MEMA's ESF Team Listserv.
> >> >> To unsubscribe from this listserv, call MEMA Operations at (508)
> >> 820-2000.
> >> >>
> >> >> +1.781.300.0000
> >> >>
> >> >>
> >> >> On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages <
> >> >> outages at outages.org> wrote:
> >> >>
> >> >>> Most public safety agencies in Massachusetts posted on Facebook and
> >> >>> Twitter between 6 AM and 8 AM EST about wireless 911 outages caused
by
> >> >>> CenturyLink. It was affecting wireless routing, so some calls were
> >> being
> >> >>> misrouted to other towns, or even other states; some callers were
> >> getting
> >> >>> reorder tones or other error messages. The problem was reported to
be
> >> >>> resolved prior to 8 AM EST.
> >> >>>
> >> >>> Seeing reports from PSAP's that the issue started again in the last
15
> >> >>> minutes.
> >> >>>
> >> >>> This is wireless only. Landline 911 calls were / are not affected.
> >> >>>
> >> >>>
> >> >>> On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <
> >> >>> outages at outages.org> wrote:
> >> >>>
> >> >>>> Don't be a fear monger. I searched Facebook and found *three*
posts
> >> >>>> about 911 outages, all of them simply reposts of local news
stories.
> >> >>>>
> >> >>>> Facebook is not "blowing up".
> >> >>>>
> >> >>>> Sheesh!
> >> >>>>
> >> >>>> [image: image1.PNG]
> >> >>>>
> >> >>>> -mel via cell
> >> >>>>
> >> >>>> -mel via cell
> >> >>>>
> >> >>>> On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <
> >> >>>> outages at outages.org> wrote:
> >> >>>>
> >> >>>> Facebook is blowing up with 911 outages and claiming it's
CenturyLink
> >> >>>> to blame.
> >> >>>>
> >> >>>> Josh Luthman
> >> >>>> Office: 937-552-2340
> >> >>>> Direct: 937-552-2343
> >> >>>> 1100 Wayne St
> >> >>>> Suite 1337
> >> >>>> Troy, OH 45373
> >> >>>>
> >> >>>> On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <
> >> >>>> outages at outages.org wrote:
> >> >>>>
> >> >>>>> Maybe light at the end of the tunnel...
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> latest we got from Centurylink was at 10:45pm CST.
> >> >>>>>
> >> >>>>> Our engineers and technicians have identified the network element
> >> that
> >> >>>>> has affected our customer services. Services are restoring, and
the
> >> current
> >> >>>>> estimated time for full recovery is four hours.
> >> >>>>>
> >> >>>>> CenturyLink will be conducting an extensive post-incident
> >> >>>>> investigation and root cause analysis to provide follow-up
> >> information to
> >> >>>>> our customers impacted by this event. We will provide one more
> >> Sales Alert
> >> >>>>> from this address once services are fully restored.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Rich Breton <richdbreton at gmail.com>
> >> >>>>> *Sent:* Thursday, December 27, 2018 2:59 PM
> >> >>>>> *To:* Christopher Conley <CConley at forsmarshgroup.com>
> >> >>>>> *Cc:* Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> I understand that the transport waves went out first, and then
they
> >> >>>>> try to reroute to something in Arizona that begin introducing all
> >> of the
> >> >>>>> broadcast traffic. How legit that is I?m not sure
> >> >>>>>
> >> >>>>> Sent from my iPhone
> >> >>>>>
> >> >>>>>
> >> >>>>> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <
> >> >>>>> outages at outages.org> wrote:
> >> >>>>>
> >> >>>>> > Finally received something useful from centurylink? I have more
> >> >>>>> questions now like how broadcast traffic effect 4x Transport
waves.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Ineptitude.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Christopher Conley
> >> >>>>>
> >> >>>>> Systems Administrator |  Fors Marsh Group
> >> >>>>>
> >> >>>>> Arlington, VA
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >> >>>>> Sundberg via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 2:46 PM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Finally received something useful from centurylink? I have more
> >> >>>>> questions now like how broadcast traffic effect 4x Transport
waves.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Update:
> >> >>>>>
> >> >>>>> On December 27, 2018 at 02:40 GMT, CenturyLink identified a
service
> >> >>>>> impact in New Orleans, LA. The NOC is engaged and investigating
in
> >> order to
> >> >>>>> isolate the cause. Field Operations were engaged and dispatched
for
> >> >>>>> additional investigations. Tier IV Equipment Vendor Support was
> >> later
> >> >>>>> engaged. During cooperative troubleshooting a device in San
> >> Antonio, TX was
> >> >>>>> isolated from the network as it was seeming to broadcast traffic
> >> consuming
> >> >>>>> capacity, which seemed to alleviate some impact. Investigations
> >> remained
> >> >>>>> ongoing. Following the isolation of the San Antonio, TX device
> >> >>>>> troubleshooting efforts focused on additional sites that teams
were
> >> >>>>> remotely unable to troubleshoot. Field Operations were dispatched
> >> to sites
> >> >>>>> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL.
> >> Tier IV
> >> >>>>> Equipment Vendor Support continued to investigate the equipment
> >> logs to
> >> >>>>> further assist with isolation. Once visibility was restored to
the
> >> site in
> >> >>>>> Kansas City, MO a filter was applied to the equipment to further
> >> alleviate
> >> >>>>> the impact observed. All of the necessary troubleshooting teams
in
> >> >>>>> cooperation with Tier IV Equipment Vendor Support are working to
> >> restore
> >> >>>>> remote visibility to the remaining sites at this time. We
> >> understand how
> >> >>>>> important these services are to our clients and the issue has
been
> >> >>>>> escalated to the highest levels within CenturyLink Service
Assurance
> >> >>>>> Leadership.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of
*Bruce
> >> >>>>> Freshwater via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 1:24 PM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> CenturyLink finally acknowledged:
> >> >>>>>
> >> >>>>> Great Lakes Region CenturyLink Partners/Agents:
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> FYI. Our network is experiencing a disruption affecting customer
> >> >>>>> services. Customers are currently unable to generate tickets
> >> through the
> >> >>>>> Control Center portal.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> We know how important these services are to our customers, and we
> >> are
> >> >>>>> working to restore services as quickly as possible.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Our technical teams are working to resolve the issue, and we will
> >> >>>>> provide updates as the situation progresses.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> -bruce
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via
Outages"
> >> <
> >> >>>>> outages at outages.org> wrote:
> >> >>>>>
> >> >>>>> So it's a bad day for centurylink.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> The lady who answered the phone for there NOC said that we are
> >> having
> >> >>>>> a nation wide outage and we are unable to open tickets at this
> >> time. Please
> >> >>>>> call be in 60-90 mins.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> She had no info on what was going on
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>> *From:* Michael Bolton <mbolton at holmeselectricsecurity.com>
> >> >>>>> *Sent:* Thursday, December 27, 2018 8:55:03 AM
> >> >>>>> *To:* Erik Sundberg; outages at outages.org
> >> >>>>> *Subject:* RE: CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Seeing the same in NC. Circuits are up but the routing between
POPs
> >> >>>>> has changed and we are seeing higher than normal latency.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> VoIP portal is down also. Allows you to login but you cannot make
> >> any
> >> >>>>> changes or view any details.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >> >>>>> Sundberg via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 9:45 AM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *CAUTION:* This email originated from outside of the
organization.
> >> Do
> >> >>>>> not click links or open attachments unless you recognize the
sender
> >> and
> >> >>>>> know the content is safe.
> >> >>>>>
> >> >>>>> We are seeing a lot of centurylink circuits and services down
this
> >> >>>>> morning.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Centurylink Control Center portal comes up but auth failing
unable
> >> to
> >> >>>>> login
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Waves down or bouncing
> >> >>>>>
> >> >>>>> Denver - Seattle (Bouncing)
> >> >>>>>
> >> >>>>> Denver - Chicago (Bouncing)
> >> >>>>>
> >> >>>>> New York - Los Angeles (Down)
> >> >>>>>
> >> >>>>> Chicago - Atlanta (Down)
> >> >>>>>
> >> >>>>>
> >> >>>>> ENNI Down in Denver (Down)
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Still tying to open tickets with them there portal is not working
> >> and
> >> >>>>> stuck on hold right now.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>> IMPORTANT NOTICE: This e-mail message is intended to be received
> >> only
> >> >>>>> by persons entitled to receive the confidential information it
may
> >> contain.
> >> >>>>> E-mail messages to clients of Holmes Security Systems may contain
> >> >>>>> information that is confidential and legally privileged. Please
do
> >> not
> >> >>>>> read, copy, forward, or store this message unless you are an
> >> intended
> >> >>>>> recipient of it. If you have received this message in error,
please
> >> forward
> >> >>>>> it to the sender and delete it completely from your computer
system.
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>> _______________________________________________
> >> >>>>> Outages mailing list
> >> >>>>> Outages at outages.org
> >> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>> _______________________________________________
> >> >>>>> Outages mailing list
> >> >>>>> Outages at outages.org
> >> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>>
> >> >>>> _______________________________________________
> >> >>>> Outages mailing list
> >> >>>> Outages at outages.org
> >> >>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> Outages mailing list
> >> >>>> Outages at outages.org
> >> >>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>
> >> >>> _______________________________________________
> >> >>> Outages mailing list
> >> >>> Outages at outages.org
> >> >>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>
> >> >> _______________________________________________
> >> >> Outages mailing list
> >> >> Outages at outages.org
> >> >> https://puck.nether.net/mailman/listinfo/outages
> >> >>
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org
> >> > https://puck.nether.net/mailman/listinfo/outages
> >> >
> >> -------------- next part --------------
> >> An HTML attachment was scrubbed...
> >> URL: <
> >>
https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/atta
chment.html
> >> >
> >> -------------- next part --------------
> >> A non-text attachment was scrubbed...
> >> Name: image1.PNG
> >> Type: image/png
> >> Size: 55125 bytes
> >> Desc: not available
> >> URL: <
> >>
https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/atta
chment.png
> >> >
> >>
> >> ------------------------------
> >>
> >> Subject: Digest Footer
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >> ------------------------------
> >>
> >> End of Outages Digest, Vol 127, Issue 31
> >> ****************************************
> >>
> >
> >
> > --
> > *Aaron K. Henderson*
> >
> > Agri-Valley Communications, Inc.
> > Network Technician
> > *O:* (989) 453-4400 <9894534400>
> > *C:* (989) 551-0060 <9895510060>
> > *netadmin at avci.net <netadmin at avci.net> *
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >

> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion



From jayfar at jayfar.com  Sat Dec 29 12:06:24 2018
From: jayfar at jayfar.com (Jay Farrell)
Date: Sat, 29 Dec 2018 12:06:24 -0500
Subject: [Outages-discussion] [outages] [External]Re: CenturyLink
 Outages this morning
In-Reply-To: <000a01d49e10$8557dbe0$900793a0$@iname.com>
References: <CY4PR17MB1207360322140445B6956A2DD3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <BN8PR14MB313778E20123EF06346A0211C9B60@BN8PR14MB3137.namprd14.prod.outlook.com>
 <CY4PR17MB12074A772D9729E2CD37946ED3B60@CY4PR17MB1207.namprd17.prod.outlook.com>
 <CY4PR14MB139895B989F97269235DA5DCA5B60@CY4PR14MB1398.namprd14.prod.outlook.com>
 <OF5BBF4292.49912E47-ON85258370.006252AD-85258370.00628238@navyfederal.org>
 <000a01d49e10$8557dbe0$900793a0$@iname.com>
Message-ID: <CAN414Ud+89r0PYQozC6Rt0VekQtevcxRC9hYf8RRoya3XHSBMw@mail.gmail.com>

>From the reddit thread:

Please note we have updated your ticket:

*Event Conclusion Summary*

Outage Start: December 27, 2018 08:40 GMT
Outage Stop: December 29, 2018 10:12 GMT

Root Cause: A CenturyLink network management card in Denver, CO was
propagating invalid frame packets across devices.

Fix Action: To restore services the card in Denver was removed from
the equipment, secondary communication channel tunnels between
specific devices were removed across the network, and a polling filter
was applied to adjust the way the packets were received in the
equipment. As repair actions were underway, it became apparent that
additional restoration steps were required for certain nodes, which
included either line card resets or Field Operations dispatches for
local equipment login. Once completed, all services restored.

RFO Summary: On December 27, 2018 at 08:40 GMT, CenturyLink identified
an initial service impact in New Orleans, LA. The NOC was engaged to
investigate the cause, and Field Operations were dispatched for
assistance onsite. Tier IV Equipment Vendor Support was engaged as it
was determined that the issue was larger than a single site. During
cooperative troubleshooting between the Equipment Vendor and
CenturyLink, a decision was made to isolate a device in San Antonio,
TX from the network as it seemed to be broadcasting traffic and
consuming capacity. This action did alleviate impact; however,
investigations remained ongoing. Focus shifted to additional sites
where network teams were unable to remotely troubleshoot equipment.
Field Operations were dispatched to sites in Kansas City, MO, Atlanta,
GA, New Orleans, LA and Chicago, IL for onsite support. As visibility
to equipment was regained, Tier IV Equipment Vendor Support evaluated
the logs to further assist with isolation. Additionally, a polling
filter was applied to the equipment in Kansas City, MO and New
Orleans, LA to prevent any additional effects. All necessary
troubleshooting teams, in cooperation with Tier IV Equipment Vendor
Support, were working to restore remote visibility to the remaining
sites. The issue had CenturyLink Executive level awareness for the
duration. A plan was formed to remove secondary communication channels
between select network devices until visibility could be restored,
which was undertaken by the Tier IV Equipment Vendor Technical Support
team in conjunction with CenturyLink Field Operations and NOC
engineers. While that effort continued, investigations into the logs,
including packet captures, was occurring in tandem, which ultimately
identified a suspected card issue in Denver, CO. Field Operations were
dispatched to remove the card. Once removed, it did not appear there
had been significant improvement; however, the logs were further
scrutinized by the Vendor's Advanced Support team and CenturyLink
Network Operations to identify that the source packet did originate
from this card. CenturyLink Tier III Technical Support shifted focus
to the application of strategic polling filters along with the
continued efforts to remove the secondary communication channels
between select nodes. Services began incrementally restoring. An
estimated restoral time of 09:00 GMT was provided; however, as repair
efforts steadily progressed, additional steps were identified for
certain nodes that impeded the restoration process. This included
either line card resets or Field Operations dispatches for local
equipment login. Various repair teams worked in tandem on these
actions to ensure that services were restored in the most expeditious
method available. By 2:30 GMT on December 29, it was confirmed that
the impacted IP, Voice, and Ethernet Access services were once again
operational. Point-to-point Transport Waves as well as Ethernet
Private Lines were still experiencing issues as multiple Optical
Carrier Groups (OCG) were still out of service. The Transport NOC
continued to work with the Tier IV Equipment Vendor Support and
CenturyLink Field Operations to replace additional line cards to
resolve the OCG issues. Several cards had to be ordered from the
nearest sparing depot. Once the remaining cards were replaced it was
confirmed that all services except a very small set of circuits had
restored, and the Transport NOC will continue to troubleshoot the
remaining impacted services under a separate Network Event. Services
were confirmed restored at 10:12 GMT. Please contact the Repair center
to address any lingering service issues.

Additional Information:
Please note that as formal post incident investigations and analysis
occur the details relayed here may evolve. Locating the management
card in Denver, CO that was sending invalid frame packets across the
network took significant analysis and packet captures to be identified
as a source as it was not in an alarm status. The CenturyLink network
continued to rebroadcast the invalid packets through the redundant
(secondary) communication routes. CenturyLink will review
troubleshooting steps to ensure that any areas of opportunity
regarding potential for restoral acceleration are addressed. These
invalid frame packets did not have a source, destination, or
expiration and were cleared out of the network via the application of
the polling filters and removal of the secondary communication paths
between specific nodes. The management card has been sent to the
equipment vendor where extensive forensic analysis will occur
regarding the underlying cause, how the packets were introduced in
this particular manner. The card has not been replaced and will not be
until the vendor review is supplied. There is no increased network
risk with leaving it unseated. At this time, there is no indication
that there was maintenance work on the card, software, or adjacent
equipment. The CenturyLink network is not at risk of reoccurrence due
to the placement of the poling filters and the removal of the
secondary communication routes between select nodes.

https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_coast/ecszjax/