[outages] CenturyLink Outages this morning

[Matt Brennan]

Most public safety agencies in Massachusetts posted on Facebook and Twitter
between 6 AM and 8 AM EST about wireless 911 outages caused by CenturyLink.
It was affecting wireless routing, so some calls were being misrouted to
other towns, or even other states; some callers were getting reorder tones
or other error messages. The problem was reported to be resolved prior to 8
AM EST.

Seeing reports from PSAP's that the issue started again in the last 15
minutes.

This is wireless only. Landline 911 calls were / are not affected.


On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <outages at outages.org>
wrote:

> Don't be a fear monger. I searched Facebook and found *three* posts about
> 911 outages, all of them simply reposts of local news stories.
>
> Facebook is not "blowing up".
>
> Sheesh!
>
> [image: image1.PNG]
>
> -mel via cell
>
> -mel via cell
>
> On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <outages at outages.org>
> wrote:
>
> Facebook is blowing up with 911 outages and claiming it's CenturyLink to
> blame.
>
> Josh Luthman
> Office: 937-552-2340
> Direct: 937-552-2343
> 1100 Wayne St
> Suite 1337
> Troy, OH 45373
>
> On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <
> outages at outages.org wrote:
>
>> Maybe light at the end of the tunnel...
>>
>>
>>
>> latest we got from Centurylink was at 10:45pm CST.
>>
>> Our engineers and technicians have identified the network element that
>> has affected our customer services. Services are restoring, and the current
>> estimated time for full recovery is four hours.
>>
>> CenturyLink will be conducting an extensive post-incident investigation
>> and root cause analysis to provide follow-up information to our customers
>> impacted by this event. We will provide one more Sales Alert from this
>> address once services are fully restored.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Rich Breton <richdbreton at gmail.com>
>> *Sent:* Thursday, December 27, 2018 2:59 PM
>> *To:* Christopher Conley <CConley at forsmarshgroup.com>
>> *Cc:* Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
>> *Subject:* Re: [outages] CenturyLink Outages this morning
>>
>>
>>
>> I understand that the transport waves went out first, and then they try
>> to reroute to something in Arizona that begin introducing all of the
>> broadcast traffic. How legit that is I’m not sure
>>
>> Sent from my iPhone
>>
>>
>> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <
>> outages at outages.org> wrote:
>>
>> > Finally received something useful from centurylink… I have more
>> questions now like how broadcast traffic effect 4x Transport waves.
>>
>>
>>
>> Ineptitude.
>>
>>
>>
>>
>>
>> Christopher Conley
>>
>> Systems Administrator |  Fors Marsh Group
>>
>> Arlington, VA
>>
>>
>>
>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
>> Sundberg via Outages
>> *Sent:* Thursday, December 27, 2018 2:46 PM
>> *To:* outages at outages.org
>> *Subject:* Re: [outages] CenturyLink Outages this morning
>>
>>
>>
>> Finally received something useful from centurylink… I have more questions
>> now like how broadcast traffic effect 4x Transport waves.
>>
>>
>>
>>
>>
>> Update:
>>
>> On December 27, 2018 at 02:40 GMT, CenturyLink identified a service
>> impact in New Orleans, LA. The NOC is engaged and investigating in order to
>> isolate the cause. Field Operations were engaged and dispatched for
>> additional investigations. Tier IV Equipment Vendor Support was later
>> engaged. During cooperative troubleshooting a device in San Antonio, TX was
>> isolated from the network as it was seeming to broadcast traffic consuming
>> capacity, which seemed to alleviate some impact. Investigations remained
>> ongoing. Following the isolation of the San Antonio, TX device
>> troubleshooting efforts focused on additional sites that teams were
>> remotely unable to troubleshoot. Field Operations were dispatched to sites
>> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV
>> Equipment Vendor Support continued to investigate the equipment logs to
>> further assist with isolation. Once visibility was restored to the site in
>> Kansas City, MO a filter was applied to the equipment to further alleviate
>> the impact observed. All of the necessary troubleshooting teams in
>> cooperation with Tier IV Equipment Vendor Support are working to restore
>> remote visibility to the remaining sites at this time. We understand how
>> important these services are to our clients and the issue has been
>> escalated to the highest levels within CenturyLink Service Assurance
>> Leadership.
>>
>>
>>
>>
>>
>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Bruce
>> Freshwater via Outages
>> *Sent:* Thursday, December 27, 2018 1:24 PM
>> *To:* outages at outages.org
>> *Subject:* Re: [outages] CenturyLink Outages this morning
>>
>>
>>
>> CenturyLink finally acknowledged:
>>
>> Great Lakes Region CenturyLink Partners/Agents:
>>
>>
>>
>> FYI. Our network is experiencing a disruption affecting customer
>> services. Customers are currently unable to generate tickets through the
>> Control Center portal.
>>
>>
>>
>> We know how important these services are to our customers, and we are
>> working to restore services as quickly as possible.
>>
>>
>>
>> Our technical teams are working to resolve the issue, and we will provide
>> updates as the situation progresses.
>>
>>
>>
>>
>>
>> -bruce
>>
>>
>>
>>
>>
>> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <
>> outages at outages.org> wrote:
>>
>> So it's a bad day for centurylink.
>>
>>
>>
>> The lady who answered the phone for there NOC said that we are having a
>> nation wide outage and we are unable to open tickets at this time. Please
>> call be in 60-90 mins.
>>
>>
>>
>> She had no info on what was going on
>>
>>
>>
>>
>> ------------------------------
>>
>> *From:* Michael Bolton <mbolton at holmeselectricsecurity.com>
>> *Sent:* Thursday, December 27, 2018 8:55:03 AM
>> *To:* Erik Sundberg; outages at outages.org
>> *Subject:* RE: CenturyLink Outages this morning
>>
>>
>>
>> Seeing the same in NC. Circuits are up but the routing between POPs has
>> changed and we are seeing higher than normal latency.
>>
>>
>>
>> VoIP portal is down also. Allows you to login but you cannot make any
>> changes or view any details.
>>
>>
>>
>>
>>
>>
>>
>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
>> Sundberg via Outages
>> *Sent:* Thursday, December 27, 2018 9:45 AM
>> *To:* outages at outages.org
>> *Subject:* [outages] CenturyLink Outages this morning
>>
>>
>>
>>
>>
>> *CAUTION:* This email originated from outside of the organization. Do
>> not click links or open attachments unless you recognize the sender and
>> know the content is safe.
>>
>> We are seeing a lot of centurylink circuits and services down this
>> morning.
>>
>>
>>
>> Centurylink Control Center portal comes up but auth failing unable to
>> login
>>
>>
>>
>> Waves down or bouncing
>>
>> Denver - Seattle (Bouncing)
>>
>> Denver - Chicago (Bouncing)
>>
>> New York - Los Angeles (Down)
>>
>> Chicago - Atlanta (Down)
>>
>>
>> ENNI Down in Denver (Down)
>>
>>
>>
>> Still tying to open tickets with them there portal is not working and
>> stuck on hold right now.
>>
>>
>>
>>
>> ------------------------------
>>
>>
>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
>> files or previous e-mail messages attached to it may contain confidential
>> information that is legally privileged. If you are not the intended
>> recipient, or a person responsible for delivering it to the intended
>> recipient, you are hereby notified that any disclosure, copying,
>> distribution or use of any of the information contained in or attached to
>> this transmission is STRICTLY PROHIBITED. If you have received this
>> transmission in error please notify the sender immediately by replying to
>> this e-mail. You must destroy the original transmission and its attachments
>> without reading or saving in any manner. Thank you.
>>
>> IMPORTANT NOTICE: This e-mail message is intended to be received only by
>> persons entitled to receive the confidential information it may contain.
>> E-mail messages to clients of Holmes Security Systems may contain
>> information that is confidential and legally privileged. Please do not
>> read, copy, forward, or store this message unless you are an intended
>> recipient of it. If you have received this message in error, please forward
>> it to the sender and delete it completely from your computer system.
>>
>>
>> ------------------------------
>>
>>
>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
>> files or previous e-mail messages attached to it may contain confidential
>> information that is legally privileged. If you are not the intended
>> recipient, or a person responsible for delivering it to the intended
>> recipient, you are hereby notified that any disclosure, copying,
>> distribution or use of any of the information contained in or attached to
>> this transmission is STRICTLY PROHIBITED. If you have received this
>> transmission in error please notify the sender immediately by replying to
>> this e-mail. You must destroy the original transmission and its attachments
>> without reading or saving in any manner. Thank you.
>>
>>
>> ------------------------------
>>
>>
>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
>> files or previous e-mail messages attached to it may contain confidential
>> information that is legally privileged. If you are not the intended
>> recipient, or a person responsible for delivering it to the intended
>> recipient, you are hereby notified that any disclosure, copying,
>> distribution or use of any of the information contained in or attached to
>> this transmission is STRICTLY PROHIBITED. If you have received this
>> transmission in error please notify the sender immediately by replying to
>> this e-mail. You must destroy the original transmission and its attachments
>> without reading or saving in any manner. Thank you.
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
>>
>> ------------------------------
>>
>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
>> files or previous e-mail messages attached to it may contain confidential
>> information that is legally privileged. If you are not the intended
>> recipient, or a person responsible for delivering it to the intended
>> recipient, you are hereby notified that any disclosure, copying,
>> distribution or use of any of the information contained in or attached to
>> this transmission is STRICTLY PROHIBITED. If you have received this
>> transmission in error please notify the sender immediately by replying to
>> this e-mail. You must destroy the original transmission and its attachments
>> without reading or saving in any manner. Thank you.
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20181228/b90c2831/attachment.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image1.PNG
Type: image/png
Size: 55125 bytes
Desc: not available
URL: <https://puck.nether.net/pipermail/outages/attachments/20181228/b90c2831/attachment.png>