[outages] CenturyLink Outages this morning

[Josh Luthman]

Facebook is blowing up with 911 outages and claiming it's CenturyLink to
blame.

Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373

On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <outages at outages.org
wrote:

> Maybe light at the end of the tunnel...
>
>
>
> latest we got from Centurylink was at 10:45pm CST.
>
> Our engineers and technicians have identified the network element that has
> affected our customer services. Services are restoring, and the current
> estimated time for full recovery is four hours.
>
> CenturyLink will be conducting an extensive post-incident investigation
> and root cause analysis to provide follow-up information to our customers
> impacted by this event. We will provide one more Sales Alert from this
> address once services are fully restored.
>
>
>
>
>
>
>
> *From:* Rich Breton <richdbreton at gmail.com>
> *Sent:* Thursday, December 27, 2018 2:59 PM
> *To:* Christopher Conley <CConley at forsmarshgroup.com>
> *Cc:* Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
> *Subject:* Re: [outages] CenturyLink Outages this morning
>
>
>
> I understand that the transport waves went out first, and then they try to
> reroute to something in Arizona that begin introducing all of the broadcast
> traffic. How legit that is I’m not sure
>
> Sent from my iPhone
>
>
> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <
> outages at outages.org> wrote:
>
> > Finally received something useful from centurylink… I have more
> questions now like how broadcast traffic effect 4x Transport waves.
>
>
>
> Ineptitude.
>
>
>
>
>
> Christopher Conley
>
> Systems Administrator |  Fors Marsh Group
>
> Arlington, VA
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> Sundberg via Outages
> *Sent:* Thursday, December 27, 2018 2:46 PM
> *To:* outages at outages.org
> *Subject:* Re: [outages] CenturyLink Outages this morning
>
>
>
> Finally received something useful from centurylink… I have more questions
> now like how broadcast traffic effect 4x Transport waves.
>
>
>
>
>
> Update:
>
> On December 27, 2018 at 02:40 GMT, CenturyLink identified a service impact
> in New Orleans, LA. The NOC is engaged and investigating in order to
> isolate the cause. Field Operations were engaged and dispatched for
> additional investigations. Tier IV Equipment Vendor Support was later
> engaged. During cooperative troubleshooting a device in San Antonio, TX was
> isolated from the network as it was seeming to broadcast traffic consuming
> capacity, which seemed to alleviate some impact. Investigations remained
> ongoing. Following the isolation of the San Antonio, TX device
> troubleshooting efforts focused on additional sites that teams were
> remotely unable to troubleshoot. Field Operations were dispatched to sites
> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL. Tier IV
> Equipment Vendor Support continued to investigate the equipment logs to
> further assist with isolation. Once visibility was restored to the site in
> Kansas City, MO a filter was applied to the equipment to further alleviate
> the impact observed. All of the necessary troubleshooting teams in
> cooperation with Tier IV Equipment Vendor Support are working to restore
> remote visibility to the remaining sites at this time. We understand how
> important these services are to our clients and the issue has been
> escalated to the highest levels within CenturyLink Service Assurance
> Leadership.
>
>
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Bruce
> Freshwater via Outages
> *Sent:* Thursday, December 27, 2018 1:24 PM
> *To:* outages at outages.org
> *Subject:* Re: [outages] CenturyLink Outages this morning
>
>
>
> CenturyLink finally acknowledged:
>
> Great Lakes Region CenturyLink Partners/Agents:
>
>
>
> FYI. Our network is experiencing a disruption affecting customer services.
> Customers are currently unable to generate tickets through the Control
> Center portal.
>
>
>
> We know how important these services are to our customers, and we are
> working to restore services as quickly as possible.
>
>
>
> Our technical teams are working to resolve the issue, and we will provide
> updates as the situation progresses.
>
>
>
>
>
> -bruce
>
>
>
>
>
> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <
> outages at outages.org> wrote:
>
> So it's a bad day for centurylink.
>
>
>
> The lady who answered the phone for there NOC said that we are having a
> nation wide outage and we are unable to open tickets at this time. Please
> call be in 60-90 mins.
>
>
>
> She had no info on what was going on
>
>
>
>
> ------------------------------
>
> *From:* Michael Bolton <mbolton at holmeselectricsecurity.com>
> *Sent:* Thursday, December 27, 2018 8:55:03 AM
> *To:* Erik Sundberg; outages at outages.org
> *Subject:* RE: CenturyLink Outages this morning
>
>
>
> Seeing the same in NC. Circuits are up but the routing between POPs has
> changed and we are seeing higher than normal latency.
>
>
>
> VoIP portal is down also. Allows you to login but you cannot make any
> changes or view any details.
>
>
>
>
>
>
>
> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> Sundberg via Outages
> *Sent:* Thursday, December 27, 2018 9:45 AM
> *To:* outages at outages.org
> *Subject:* [outages] CenturyLink Outages this morning
>
>
>
>
>
> *CAUTION:* This email originated from outside of the organization. Do not
> click links or open attachments unless you recognize the sender and know
> the content is safe.
>
> We are seeing a lot of centurylink circuits and services down this morning.
>
>
>
> Centurylink Control Center portal comes up but auth failing unable to login
>
>
>
> Waves down or bouncing
>
> Denver - Seattle (Bouncing)
>
> Denver - Chicago (Bouncing)
>
> New York - Los Angeles (Down)
>
> Chicago - Atlanta (Down)
>
>
> ENNI Down in Denver (Down)
>
>
>
> Still tying to open tickets with them there portal is not working and
> stuck on hold right now.
>
>
>
>
> ------------------------------
>
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>
> IMPORTANT NOTICE: This e-mail message is intended to be received only by
> persons entitled to receive the confidential information it may contain.
> E-mail messages to clients of Holmes Security Systems may contain
> information that is confidential and legally privileged. Please do not
> read, copy, forward, or store this message unless you are an intended
> recipient of it. If you have received this message in error, please forward
> it to the sender and delete it completely from your computer system.
>
>
> ------------------------------
>
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>
>
> ------------------------------
>
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ------------------------------
>
> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents, files
> or previous e-mail messages attached to it may contain confidential
> information that is legally privileged. If you are not the intended
> recipient, or a person responsible for delivering it to the intended
> recipient, you are hereby notified that any disclosure, copying,
> distribution or use of any of the information contained in or attached to
> this transmission is STRICTLY PROHIBITED. If you have received this
> transmission in error please notify the sender immediately by replying to
> this e-mail. You must destroy the original transmission and its attachments
> without reading or saving in any manner. Thank you.
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20181228/01918dd0/attachment.htm>