[outages] Outages Digest, Vol 127, Issue 31

Fri Dec 28 13:11:39 EST 2018

My apologies if this message breaks the mailing list, I have never replied
to one of these before.

I work for a rural ISP, we do not use CenturyLink as an upstream; however,
some of our customers are reporting issues since the outage. Management has
tasked my office with putting together a timeline/details about the outage
to brief upper management. Is there a detailed timeline of events
documenting this outage?

Thanks,

A

On Fri, Dec 28, 2018 at 12:13 PM <outages-request at outages.org> wrote:

> Send Outages mailing list submissions to
>         outages at outages.org
>
> To subscribe or unsubscribe via the World Wide Web, visit
>         https://puck.nether.net/mailman/listinfo/outages
> or, via email, send a message with subject or body 'help' to
>         outages-request at outages.org
>
> You can reach the person managing the list at
>         outages-owner at outages.org
>
> When replying, please edit your Subject line so it is more specific
> than "Re: Contents of Outages digest..."
>
>
> Today's Topics:
>
>    1. Re: CenturyLink Outages this morning (Aaron C. de Bruyn)
>
>
> ----------------------------------------------------------------------
>
> Message: 1
> Date: Fri, 28 Dec 2018 09:12:03 -0800
> From: "Aaron C. de Bruyn" <aaron at heyaaron.com>
> To: Dom Colangelo <dom.colangelo at simmons.edu>
> Cc: Bob Antia <antia at ba14.com>, outages at outages.org
> Subject: Re: [outages] CenturyLink Outages this morning
> Message-ID:
>         <CAEE+rGqaysPos=
> VB_QtS++CBs1bNuRzXcLig0iBmntLQcVVk4w at mail.gmail.com>
> Content-Type: text/plain; charset="utf-8"
>
> Several times late last night we had our weather alert radio activate and
> broadcast an unintelligible message about 911 being down.  It wasn't
> reception problems. The cadence of the dumb computer voice made it nearly
> impossible to understand.
>
> https://photos.app.goo.gl/M4spwKAkKPnoSrJw9
>
> The alert came through ~5 hours after 911 went offline everywhere.
>
> -A
>
>
> On Fri, Dec 28, 2018, 08:44 Dom Colangelo via Outages <outages at outages.org
> wrote:
>
> > Received a WEA from MEMA in Boston, MA noting that 911 is down for
> > cellular users.
> >
> > Dom Colangelo
> >
> > Senior Windows Systems Administrator
> >
> > Infrastructure and Online Services
> >
> > Simmons University
> >
> > 617.521.2669
> >
> >
> > On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages <
> outages at outages.org>
> > wrote:
> >
> >> Massachusetts emergency management agency (MEMA) sent this out this
> >> morning and followed up at 0736 that the issues had been resolved.
> >>
> >>
> >> *Massachusetts Emergency Management Agency*
> >> ------------------------------
> >>
> >> *Massachusetts Emergency Management Agency*
> >>
> >> *Situational Awareness Statement*
> >>
> >> *December 28, 2018  7:00 AM*
> >>
> >>
> >>
> >> *Re: Cellular 911 Experiencing Technical Difficulties and Sporadic
> >> Outages*
> >>
> >>
> >>
> >> The Massachusetts Emergency Management Agency is sharing the following
> >> information from the Commonwealth Fusion Center (CFC) with our local
> >> partners and stakeholders.  MEMA also is in touch with the State 911
> >> Department.
> >>
> >>
> >>
> >> *From the Commonwealth Fusion Center*
> >>
> >>
> >>
> >> *Situation:*
> >>
> >> A nationwide outage affecting CenturyLink, an Internet communications
> >> company, is impacting wireless 911 call capability in various parts of
> the
> >> country. We believe Massachusetts has been impacted by the outage and
> urge
> >> residents using cellular phones, if necessary, to contact emergency
> >> first-responders by using the full 10-digit number of a public safety
> >> answering point emergency call center (PSAP) or their local police
> >> department.
> >>
> >>
> >>
> >> The Massachusetts State Police operate three regional PSAPs and a fourth
> >> is staffed by State 911 at MSP headquarters in Framingham.
> >> Residents using a cell phone to call for emergency services should first
> >> try to dial 911, and if it does not work, instead call one of these four
> >> 10-digit numbers:
> >> Shelburne Control PSAP 413-625-8200
> >> New Braintree Control PSAP 508-867-1170
> >> Northampton Control PSAP 413-586-1508
> >> Framingham PSAP 508-820-2121.
> >>
> >>
> >> Residents using cell phones may also, if dialing 911 does not work, may
> >> call the 10-digit general number of their local police department. They
> may
> >> also call the 10-digit number of a State Police Barracks in their area
> >> (search for barracks location and phone numbers on the MSP web site at
> >> www.mass.gov and search for State Police locations).
> >> Other agencies operate additional PSAPs in Massachusetts beyond those
> >> listed above.
> >>
> >>
> >>
> >> The outage seems to only be affecting wireless 911 calls; landline calls
> >> to 911 appear to be working.
> >>
> >>
> >> CenturyLink has indicted it is working on repairing what the company
> >> called a ?network element? that is causing the problem.
> >>
> >>
> >>
> >>  *MEMA Operations*
> >>
> >> The State Emergency Operations Center (SEOC) is currently operating at
> >> Level 1 (Steady State Monitoring). MEMA will continue to monitor the
> >> situation and will disseminate additional Situational Awareness
> Statements
> >> as necessary. Cities and towns wishing to share information or request
> >> assistance should contact their MEMA Regional office or MEMA?s 24/7
> >> communications center at 508-820-2000.
> >>
> >>  *Stay Informed:*
> >>
> >> Utilize *Massachusetts Alerts* to receive emergency notifications and
> >> information from the Massachusetts Emergency Management Agency and the
> >> National Weather Service.  *Massachusetts Alerts* is a free app that is
> >> available for Android and iPhones. To learn more about *Massachusetts
> >> Alerts*, and for information on how to download the free app onto your
> >> smartphone, visit:
> >> http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app
> .
> >>
> >> Utilize MEMA?s real-time power outage viewer to stay informed about
> >> current power outages in your community and region, and across the
> state,
> >> including information from utility companies about restoration times:
> >> http://mema.mapsonline.net/public.html
> >>
> >> Utilize MEMA?s live weather radar and forecasting tools:
> >> http://www.mass.gov/map-resources
> >>
> >>
> >> *Online Resources:*For additional information and resources, visit:
> >> Massachusetts Emergency Management Agency at www.mass.gov/mema
> >> MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA
> >> MEMA Twitter: @MassEMA <http://twitter.com/massema>
> >> Federal Emergency Management Agency at www.fema.gov
> >> National Weather Service Boston/Norton at www.weather.gov/boston
> >> National Weather Service Albany, NY at www.weather.gov/albany
> >> National Weather Service Weather Prediction Center:
> >> http://www.wpc.ncep.noaa.gov
> >> National Weather Service Storm Prediction Center:
> >> http://www.spc.noaa.gov/
> >> Mass211 at www.mass211.org
> >>
> >>
> >>
> >>
> >> ------------------------------
> >>
> >> This message has been sent from MEMA's ESF Team Listserv.
> >> To unsubscribe from this listserv, call MEMA Operations at (508)
> 820-2000.
> >>
> >> +1.781.300.0000
> >>
> >>
> >> On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages <
> >> outages at outages.org> wrote:
> >>
> >>> Most public safety agencies in Massachusetts posted on Facebook and
> >>> Twitter between 6 AM and 8 AM EST about wireless 911 outages caused by
> >>> CenturyLink. It was affecting wireless routing, so some calls were
> being
> >>> misrouted to other towns, or even other states; some callers were
> getting
> >>> reorder tones or other error messages. The problem was reported to be
> >>> resolved prior to 8 AM EST.
> >>>
> >>> Seeing reports from PSAP's that the issue started again in the last 15
> >>> minutes.
> >>>
> >>> This is wireless only. Landline 911 calls were / are not affected.
> >>>
> >>>
> >>> On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <
> >>> outages at outages.org> wrote:
> >>>
> >>>> Don't be a fear monger. I searched Facebook and found *three* posts
> >>>> about 911 outages, all of them simply reposts of local news stories.
> >>>>
> >>>> Facebook is not "blowing up".
> >>>>
> >>>> Sheesh!
> >>>>
> >>>> [image: image1.PNG]
> >>>>
> >>>> -mel via cell
> >>>>
> >>>> -mel via cell
> >>>>
> >>>> On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <
> >>>> outages at outages.org> wrote:
> >>>>
> >>>> Facebook is blowing up with 911 outages and claiming it's CenturyLink
> >>>> to blame.
> >>>>
> >>>> Josh Luthman
> >>>> Office: 937-552-2340
> >>>> Direct: 937-552-2343
> >>>> 1100 Wayne St
> >>>> Suite 1337
> >>>> Troy, OH 45373
> >>>>
> >>>> On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <
> >>>> outages at outages.org wrote:
> >>>>
> >>>>> Maybe light at the end of the tunnel...
> >>>>>
> >>>>>
> >>>>>
> >>>>> latest we got from Centurylink was at 10:45pm CST.
> >>>>>
> >>>>> Our engineers and technicians have identified the network element
> that
> >>>>> has affected our customer services. Services are restoring, and the
> current
> >>>>> estimated time for full recovery is four hours.
> >>>>>
> >>>>> CenturyLink will be conducting an extensive post-incident
> >>>>> investigation and root cause analysis to provide follow-up
> information to
> >>>>> our customers impacted by this event. We will provide one more Sales
> Alert
> >>>>> from this address once services are fully restored.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> *From:* Rich Breton <richdbreton at gmail.com>
> >>>>> *Sent:* Thursday, December 27, 2018 2:59 PM
> >>>>> *To:* Christopher Conley <CConley at forsmarshgroup.com>
> >>>>> *Cc:* Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >>>>>
> >>>>>
> >>>>>
> >>>>> I understand that the transport waves went out first, and then they
> >>>>> try to reroute to something in Arizona that begin introducing all of
> the
> >>>>> broadcast traffic. How legit that is I?m not sure
> >>>>>
> >>>>> Sent from my iPhone
> >>>>>
> >>>>>
> >>>>> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <
> >>>>> outages at outages.org> wrote:
> >>>>>
> >>>>> > Finally received something useful from centurylink? I have more
> >>>>> questions now like how broadcast traffic effect 4x Transport waves.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Ineptitude.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Christopher Conley
> >>>>>
> >>>>> Systems Administrator |  Fors Marsh Group
> >>>>>
> >>>>> Arlington, VA
> >>>>>
> >>>>>
> >>>>>
> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >>>>> Sundberg via Outages
> >>>>> *Sent:* Thursday, December 27, 2018 2:46 PM
> >>>>> *To:* outages at outages.org
> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >>>>>
> >>>>>
> >>>>>
> >>>>> Finally received something useful from centurylink? I have more
> >>>>> questions now like how broadcast traffic effect 4x Transport waves.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> Update:
> >>>>>
> >>>>> On December 27, 2018 at 02:40 GMT, CenturyLink identified a service
> >>>>> impact in New Orleans, LA. The NOC is engaged and investigating in
> order to
> >>>>> isolate the cause. Field Operations were engaged and dispatched for
> >>>>> additional investigations. Tier IV Equipment Vendor Support was later
> >>>>> engaged. During cooperative troubleshooting a device in San Antonio,
> TX was
> >>>>> isolated from the network as it was seeming to broadcast traffic
> consuming
> >>>>> capacity, which seemed to alleviate some impact. Investigations
> remained
> >>>>> ongoing. Following the isolation of the San Antonio, TX device
> >>>>> troubleshooting efforts focused on additional sites that teams were
> >>>>> remotely unable to troubleshoot. Field Operations were dispatched to
> sites
> >>>>> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL.
> Tier IV
> >>>>> Equipment Vendor Support continued to investigate the equipment logs
> to
> >>>>> further assist with isolation. Once visibility was restored to the
> site in
> >>>>> Kansas City, MO a filter was applied to the equipment to further
> alleviate
> >>>>> the impact observed. All of the necessary troubleshooting teams in
> >>>>> cooperation with Tier IV Equipment Vendor Support are working to
> restore
> >>>>> remote visibility to the remaining sites at this time. We understand
> how
> >>>>> important these services are to our clients and the issue has been
> >>>>> escalated to the highest levels within CenturyLink Service Assurance
> >>>>> Leadership.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Bruce
> >>>>> Freshwater via Outages
> >>>>> *Sent:* Thursday, December 27, 2018 1:24 PM
> >>>>> *To:* outages at outages.org
> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >>>>>
> >>>>>
> >>>>>
> >>>>> CenturyLink finally acknowledged:
> >>>>>
> >>>>> Great Lakes Region CenturyLink Partners/Agents:
> >>>>>
> >>>>>
> >>>>>
> >>>>> FYI. Our network is experiencing a disruption affecting customer
> >>>>> services. Customers are currently unable to generate tickets through
> the
> >>>>> Control Center portal.
> >>>>>
> >>>>>
> >>>>>
> >>>>> We know how important these services are to our customers, and we are
> >>>>> working to restore services as quickly as possible.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Our technical teams are working to resolve the issue, and we will
> >>>>> provide updates as the situation progresses.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> -bruce
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via Outages" <
> >>>>> outages at outages.org> wrote:
> >>>>>
> >>>>> So it's a bad day for centurylink.
> >>>>>
> >>>>>
> >>>>>
> >>>>> The lady who answered the phone for there NOC said that we are having
> >>>>> a nation wide outage and we are unable to open tickets at this time.
> Please
> >>>>> call be in 60-90 mins.
> >>>>>
> >>>>>
> >>>>>
> >>>>> She had no info on what was going on
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ------------------------------
> >>>>>
> >>>>> *From:* Michael Bolton <mbolton at holmeselectricsecurity.com>
> >>>>> *Sent:* Thursday, December 27, 2018 8:55:03 AM
> >>>>> *To:* Erik Sundberg; outages at outages.org
> >>>>> *Subject:* RE: CenturyLink Outages this morning
> >>>>>
> >>>>>
> >>>>>
> >>>>> Seeing the same in NC. Circuits are up but the routing between POPs
> >>>>> has changed and we are seeing higher than normal latency.
> >>>>>
> >>>>>
> >>>>>
> >>>>> VoIP portal is down also. Allows you to login but you cannot make any
> >>>>> changes or view any details.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >>>>> Sundberg via Outages
> >>>>> *Sent:* Thursday, December 27, 2018 9:45 AM
> >>>>> *To:* outages at outages.org
> >>>>> *Subject:* [outages] CenturyLink Outages this morning
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> *CAUTION:* This email originated from outside of the organization. Do
> >>>>> not click links or open attachments unless you recognize the sender
> and
> >>>>> know the content is safe.
> >>>>>
> >>>>> We are seeing a lot of centurylink circuits and services down this
> >>>>> morning.
> >>>>>
> >>>>>
> >>>>>
> >>>>> Centurylink Control Center portal comes up but auth failing unable to
> >>>>> login
> >>>>>
> >>>>>
> >>>>>
> >>>>> Waves down or bouncing
> >>>>>
> >>>>> Denver - Seattle (Bouncing)
> >>>>>
> >>>>> Denver - Chicago (Bouncing)
> >>>>>
> >>>>> New York - Los Angeles (Down)
> >>>>>
> >>>>> Chicago - Atlanta (Down)
> >>>>>
> >>>>>
> >>>>> ENNI Down in Denver (Down)
> >>>>>
> >>>>>
> >>>>>
> >>>>> Still tying to open tickets with them there portal is not working and
> >>>>> stuck on hold right now.
> >>>>>
> >>>>>
> >>>>>
> >>>>>
> >>>>> ------------------------------
> >>>>>
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >>>>> files or previous e-mail messages attached to it may contain
> confidential
> >>>>> information that is legally privileged. If you are not the intended
> >>>>> recipient, or a person responsible for delivering it to the intended
> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >>>>> distribution or use of any of the information contained in or
> attached to
> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >>>>> transmission in error please notify the sender immediately by
> replying to
> >>>>> this e-mail. You must destroy the original transmission and its
> attachments
> >>>>> without reading or saving in any manner. Thank you.
> >>>>>
> >>>>> IMPORTANT NOTICE: This e-mail message is intended to be received only
> >>>>> by persons entitled to receive the confidential information it may
> contain.
> >>>>> E-mail messages to clients of Holmes Security Systems may contain
> >>>>> information that is confidential and legally privileged. Please do
> not
> >>>>> read, copy, forward, or store this message unless you are an intended
> >>>>> recipient of it. If you have received this message in error, please
> forward
> >>>>> it to the sender and delete it completely from your computer system.
> >>>>>
> >>>>>
> >>>>> ------------------------------
> >>>>>
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >>>>> files or previous e-mail messages attached to it may contain
> confidential
> >>>>> information that is legally privileged. If you are not the intended
> >>>>> recipient, or a person responsible for delivering it to the intended
> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >>>>> distribution or use of any of the information contained in or
> attached to
> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >>>>> transmission in error please notify the sender immediately by
> replying to
> >>>>> this e-mail. You must destroy the original transmission and its
> attachments
> >>>>> without reading or saving in any manner. Thank you.
> >>>>>
> >>>>>
> >>>>> ------------------------------
> >>>>>
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >>>>> files or previous e-mail messages attached to it may contain
> confidential
> >>>>> information that is legally privileged. If you are not the intended
> >>>>> recipient, or a person responsible for delivering it to the intended
> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >>>>> distribution or use of any of the information contained in or
> attached to
> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >>>>> transmission in error please notify the sender immediately by
> replying to
> >>>>> this e-mail. You must destroy the original transmission and its
> attachments
> >>>>> without reading or saving in any manner. Thank you.
> >>>>>
> >>>>> _______________________________________________
> >>>>> Outages mailing list
> >>>>> Outages at outages.org
> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >>>>>
> >>>>>
> >>>>> ------------------------------
> >>>>>
> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any documents,
> >>>>> files or previous e-mail messages attached to it may contain
> confidential
> >>>>> information that is legally privileged. If you are not the intended
> >>>>> recipient, or a person responsible for delivering it to the intended
> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >>>>> distribution or use of any of the information contained in or
> attached to
> >>>>> this transmission is STRICTLY PROHIBITED. If you have received this
> >>>>> transmission in error please notify the sender immediately by
> replying to
> >>>>> this e-mail. You must destroy the original transmission and its
> attachments
> >>>>> without reading or saving in any manner. Thank you.
> >>>>> _______________________________________________
> >>>>> Outages mailing list
> >>>>> Outages at outages.org
> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >>>>>
> >>>> _______________________________________________
> >>>> Outages mailing list
> >>>> Outages at outages.org
> >>>> https://puck.nether.net/mailman/listinfo/outages
> >>>>
> >>>> _______________________________________________
> >>>> Outages mailing list
> >>>> Outages at outages.org
> >>>> https://puck.nether.net/mailman/listinfo/outages
> >>>>
> >>> _______________________________________________
> >>> Outages mailing list
> >>> Outages at outages.org
> >>> https://puck.nether.net/mailman/listinfo/outages
> >>>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
> -------------- next part --------------
> An HTML attachment was scrubbed...
> URL: <
> https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/attachment.html
> >
> -------------- next part --------------
> A non-text attachment was scrubbed...
> Name: image1.PNG
> Type: image/png
> Size: 55125 bytes
> Desc: not available
> URL: <
> https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/attachment.png
> >
>
> ------------------------------
>
> Subject: Digest Footer
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
>
> ------------------------------
>
> End of Outages Digest, Vol 127, Issue 31
> ****************************************
>

-- 
*Aaron K. Henderson*

Agri-Valley Communications, Inc.
Network Technician
*O:* (989) 453-4400 <9894534400>
*C:* (989) 551-0060 <9895510060>
*netadmin at avci.net <netadmin at avci.net> *
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20181228/6f49c635/attachment.htm>