[Outages-discussion] [outages] Outages Digest, Vol 127, Issue 31
Frank Bulk
frnkblk at iname.com
Sat Dec 29 11:59:15 EST 2018
Initial RCA here:
https://old.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_
coast/ecszjax/
Frank
-----Original Message-----
From: Outages-discussion <outages-discussion-bounces at outages.org> On Behalf
Of Jeremy Chadwick
Sent: Friday, December 28, 2018 3:38 PM
To: outages-discussion at outages.org
Subject: Re: [Outages-discussion] [outages] Outages Digest, Vol 127, Issue
31
Likewise there was this, which has a timeline. Source is from someone
on HackerNews. Note there is a hyphen at the end of the URL (don't ask;
DSLR/BBR does that):
https://www.dslreports.com/forum/r32235431-
--
| Jeremy Chadwick jdc at koitsu.org |
| UNIX Systems Administrator PGP 0x2A389531 |
| Making life hard for others since 1977. |
On Fri, Dec 28, 2018 at 01:39:32PM -0600, lamrya benrob via Outages wrote:
> This Reddit thread has been pretty informative (you'll have to spend some
> time weeding through the noise but some good insight on things):
>
>
https://www.reddit.com/r/networking/comments/a9z6tb/centurylink_outage_west_
coast/
>
> Sort the thread by new or old.
>
>
>
> On Fri, Dec 28, 2018 at 12:17 PM Aaron Henderson via Outages <
> outages at outages.org> wrote:
>
> > My apologies if this message breaks the mailing list, I have never
replied
> > to one of these before.
> >
> > I work for a rural ISP, we do not use CenturyLink as an upstream;
however,
> > some of our customers are reporting issues since the outage. Management
has
> > tasked my office with putting together a timeline/details about the
outage
> > to brief upper management. Is there a detailed timeline of events
> > documenting this outage?
> >
> > Thanks,
> >
> > A
> >
> > On Fri, Dec 28, 2018 at 12:13 PM <outages-request at outages.org> wrote:
> >
> >> Send Outages mailing list submissions to
> >> outages at outages.org
> >>
> >> To subscribe or unsubscribe via the World Wide Web, visit
> >> https://puck.nether.net/mailman/listinfo/outages
> >> or, via email, send a message with subject or body 'help' to
> >> outages-request at outages.org
> >>
> >> You can reach the person managing the list at
> >> outages-owner at outages.org
> >>
> >> When replying, please edit your Subject line so it is more specific
> >> than "Re: Contents of Outages digest..."
> >>
> >>
> >> Today's Topics:
> >>
> >> 1. Re: CenturyLink Outages this morning (Aaron C. de Bruyn)
> >>
> >>
> >> ----------------------------------------------------------------------
> >>
> >> Message: 1
> >> Date: Fri, 28 Dec 2018 09:12:03 -0800
> >> From: "Aaron C. de Bruyn" <aaron at heyaaron.com>
> >> To: Dom Colangelo <dom.colangelo at simmons.edu>
> >> Cc: Bob Antia <antia at ba14.com>, outages at outages.org
> >> Subject: Re: [outages] CenturyLink Outages this morning
> >> Message-ID:
> >> <CAEE+rGqaysPos=
> >> VB_QtS++CBs1bNuRzXcLig0iBmntLQcVVk4w at mail.gmail.com>
> >> Content-Type: text/plain; charset="utf-8"
> >>
> >> Several times late last night we had our weather alert radio activate
and
> >> broadcast an unintelligible message about 911 being down. It wasn't
> >> reception problems. The cadence of the dumb computer voice made it
nearly
> >> impossible to understand.
> >>
> >> https://photos.app.goo.gl/M4spwKAkKPnoSrJw9
> >>
> >> The alert came through ~5 hours after 911 went offline everywhere.
> >>
> >> -A
> >>
> >>
> >> On Fri, Dec 28, 2018, 08:44 Dom Colangelo via Outages <
> >> outages at outages.org
> >> wrote:
> >>
> >> > Received a WEA from MEMA in Boston, MA noting that 911 is down for
> >> > cellular users.
> >> >
> >> > Dom Colangelo
> >> >
> >> > Senior Windows Systems Administrator
> >> >
> >> > Infrastructure and Online Services
> >> >
> >> > Simmons University
> >> >
> >> > 617.521.2669
> >> >
> >> >
> >> > On Fri, Dec 28, 2018 at 9:56 AM Bob Antia via Outages <
> >> outages at outages.org>
> >> > wrote:
> >> >
> >> >> Massachusetts emergency management agency (MEMA) sent this out this
> >> >> morning and followed up at 0736 that the issues had been resolved.
> >> >>
> >> >>
> >> >> *Massachusetts Emergency Management Agency*
> >> >> ------------------------------
> >> >>
> >> >> *Massachusetts Emergency Management Agency*
> >> >>
> >> >> *Situational Awareness Statement*
> >> >>
> >> >> *December 28, 2018 7:00 AM*
> >> >>
> >> >>
> >> >>
> >> >> *Re: Cellular 911 Experiencing Technical Difficulties and Sporadic
> >> >> Outages*
> >> >>
> >> >>
> >> >>
> >> >> The Massachusetts Emergency Management Agency is sharing the
following
> >> >> information from the Commonwealth Fusion Center (CFC) with our local
> >> >> partners and stakeholders. MEMA also is in touch with the State 911
> >> >> Department.
> >> >>
> >> >>
> >> >>
> >> >> *From the Commonwealth Fusion Center*
> >> >>
> >> >>
> >> >>
> >> >> *Situation:*
> >> >>
> >> >> A nationwide outage affecting CenturyLink, an Internet
communications
> >> >> company, is impacting wireless 911 call capability in various parts
of
> >> the
> >> >> country. We believe Massachusetts has been impacted by the outage
and
> >> urge
> >> >> residents using cellular phones, if necessary, to contact emergency
> >> >> first-responders by using the full 10-digit number of a public
safety
> >> >> answering point emergency call center (PSAP) or their local police
> >> >> department.
> >> >>
> >> >>
> >> >>
> >> >> The Massachusetts State Police operate three regional PSAPs and a
> >> fourth
> >> >> is staffed by State 911 at MSP headquarters in Framingham.
> >> >> Residents using a cell phone to call for emergency services should
> >> first
> >> >> try to dial 911, and if it does not work, instead call one of these
> >> four
> >> >> 10-digit numbers:
> >> >> Shelburne Control PSAP 413-625-8200
> >> >> New Braintree Control PSAP 508-867-1170
> >> >> Northampton Control PSAP 413-586-1508
> >> >> Framingham PSAP 508-820-2121.
> >> >>
> >> >>
> >> >> Residents using cell phones may also, if dialing 911 does not work,
may
> >> >> call the 10-digit general number of their local police department.
> >> They may
> >> >> also call the 10-digit number of a State Police Barracks in their
area
> >> >> (search for barracks location and phone numbers on the MSP web site
at
> >> >> www.mass.gov and search for State Police locations).
> >> >> Other agencies operate additional PSAPs in Massachusetts beyond
those
> >> >> listed above.
> >> >>
> >> >>
> >> >>
> >> >> The outage seems to only be affecting wireless 911 calls; landline
> >> calls
> >> >> to 911 appear to be working.
> >> >>
> >> >>
> >> >> CenturyLink has indicted it is working on repairing what the company
> >> >> called a ?network element? that is causing the problem.
> >> >>
> >> >>
> >> >>
> >> >> *MEMA Operations*
> >> >>
> >> >> The State Emergency Operations Center (SEOC) is currently operating
at
> >> >> Level 1 (Steady State Monitoring). MEMA will continue to monitor the
> >> >> situation and will disseminate additional Situational Awareness
> >> Statements
> >> >> as necessary. Cities and towns wishing to share information or
request
> >> >> assistance should contact their MEMA Regional office or MEMA?s 24/7
> >> >> communications center at 508-820-2000.
> >> >>
> >> >> *Stay Informed:*
> >> >>
> >> >> Utilize *Massachusetts Alerts* to receive emergency notifications
and
> >> >> information from the Massachusetts Emergency Management Agency and
the
> >> >> National Weather Service. *Massachusetts Alerts* is a free app that
is
> >> >> available for Android and iPhones. To learn more about
*Massachusetts
> >> >> Alerts*, and for information on how to download the free app onto
your
> >> >> smartphone, visit:
> >> >>
> >>
http://www.mass.gov/service-details/massachusetts-alerts-smartphone-app.
> >> >>
> >> >> Utilize MEMA?s real-time power outage viewer to stay informed about
> >> >> current power outages in your community and region, and across the
> >> state,
> >> >> including information from utility companies about restoration
times:
> >> >> http://mema.mapsonline.net/public.html
> >> >>
> >> >> Utilize MEMA?s live weather radar and forecasting tools:
> >> >> http://www.mass.gov/map-resources
> >> >>
> >> >>
> >> >> *Online Resources:*For additional information and resources, visit:
> >> >> Massachusetts Emergency Management Agency at www.mass.gov/mema
> >> >> MEMA's Facebook page: http://www.facebook.com/MassachusettsEMA
> >> >> MEMA Twitter: @MassEMA <http://twitter.com/massema>
> >> >> Federal Emergency Management Agency at www.fema.gov
> >> >> National Weather Service Boston/Norton at www.weather.gov/boston
> >> >> National Weather Service Albany, NY at www.weather.gov/albany
> >> >> National Weather Service Weather Prediction Center:
> >> >> http://www.wpc.ncep.noaa.gov
> >> >> National Weather Service Storm Prediction Center:
> >> >> http://www.spc.noaa.gov/
> >> >> Mass211 at www.mass211.org
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> ------------------------------
> >> >>
> >> >> This message has been sent from MEMA's ESF Team Listserv.
> >> >> To unsubscribe from this listserv, call MEMA Operations at (508)
> >> 820-2000.
> >> >>
> >> >> +1.781.300.0000
> >> >>
> >> >>
> >> >> On Fri, Dec 28, 2018 at 9:52 AM Matt Brennan via Outages <
> >> >> outages at outages.org> wrote:
> >> >>
> >> >>> Most public safety agencies in Massachusetts posted on Facebook and
> >> >>> Twitter between 6 AM and 8 AM EST about wireless 911 outages caused
by
> >> >>> CenturyLink. It was affecting wireless routing, so some calls were
> >> being
> >> >>> misrouted to other towns, or even other states; some callers were
> >> getting
> >> >>> reorder tones or other error messages. The problem was reported to
be
> >> >>> resolved prior to 8 AM EST.
> >> >>>
> >> >>> Seeing reports from PSAP's that the issue started again in the last
15
> >> >>> minutes.
> >> >>>
> >> >>> This is wireless only. Landline 911 calls were / are not affected.
> >> >>>
> >> >>>
> >> >>> On Fri, Dec 28, 2018 at 9:32 AM Mel Beckman via Outages <
> >> >>> outages at outages.org> wrote:
> >> >>>
> >> >>>> Don't be a fear monger. I searched Facebook and found *three*
posts
> >> >>>> about 911 outages, all of them simply reposts of local news
stories.
> >> >>>>
> >> >>>> Facebook is not "blowing up".
> >> >>>>
> >> >>>> Sheesh!
> >> >>>>
> >> >>>> [image: image1.PNG]
> >> >>>>
> >> >>>> -mel via cell
> >> >>>>
> >> >>>> -mel via cell
> >> >>>>
> >> >>>> On Dec 28, 2018, at 5:23 AM, Josh Luthman via Outages <
> >> >>>> outages at outages.org> wrote:
> >> >>>>
> >> >>>> Facebook is blowing up with 911 outages and claiming it's
CenturyLink
> >> >>>> to blame.
> >> >>>>
> >> >>>> Josh Luthman
> >> >>>> Office: 937-552-2340
> >> >>>> Direct: 937-552-2343
> >> >>>> 1100 Wayne St
> >> >>>> Suite 1337
> >> >>>> Troy, OH 45373
> >> >>>>
> >> >>>> On Fri, Dec 28, 2018, 2:09 AM Erik Sundberg via Outages <
> >> >>>> outages at outages.org wrote:
> >> >>>>
> >> >>>>> Maybe light at the end of the tunnel...
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> latest we got from Centurylink was at 10:45pm CST.
> >> >>>>>
> >> >>>>> Our engineers and technicians have identified the network element
> >> that
> >> >>>>> has affected our customer services. Services are restoring, and
the
> >> current
> >> >>>>> estimated time for full recovery is four hours.
> >> >>>>>
> >> >>>>> CenturyLink will be conducting an extensive post-incident
> >> >>>>> investigation and root cause analysis to provide follow-up
> >> information to
> >> >>>>> our customers impacted by this event. We will provide one more
> >> Sales Alert
> >> >>>>> from this address once services are fully restored.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Rich Breton <richdbreton at gmail.com>
> >> >>>>> *Sent:* Thursday, December 27, 2018 2:59 PM
> >> >>>>> *To:* Christopher Conley <CConley at forsmarshgroup.com>
> >> >>>>> *Cc:* Erik Sundberg <ESundberg at nitelusa.com>; outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> I understand that the transport waves went out first, and then
they
> >> >>>>> try to reroute to something in Arizona that begin introducing all
> >> of the
> >> >>>>> broadcast traffic. How legit that is I?m not sure
> >> >>>>>
> >> >>>>> Sent from my iPhone
> >> >>>>>
> >> >>>>>
> >> >>>>> On Dec 27, 2018, at 12:58 PM, Christopher Conley via Outages <
> >> >>>>> outages at outages.org> wrote:
> >> >>>>>
> >> >>>>> > Finally received something useful from centurylink? I have more
> >> >>>>> questions now like how broadcast traffic effect 4x Transport
waves.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Ineptitude.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Christopher Conley
> >> >>>>>
> >> >>>>> Systems Administrator | Fors Marsh Group
> >> >>>>>
> >> >>>>> Arlington, VA
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >> >>>>> Sundberg via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 2:46 PM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Finally received something useful from centurylink? I have more
> >> >>>>> questions now like how broadcast traffic effect 4x Transport
waves.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Update:
> >> >>>>>
> >> >>>>> On December 27, 2018 at 02:40 GMT, CenturyLink identified a
service
> >> >>>>> impact in New Orleans, LA. The NOC is engaged and investigating
in
> >> order to
> >> >>>>> isolate the cause. Field Operations were engaged and dispatched
for
> >> >>>>> additional investigations. Tier IV Equipment Vendor Support was
> >> later
> >> >>>>> engaged. During cooperative troubleshooting a device in San
> >> Antonio, TX was
> >> >>>>> isolated from the network as it was seeming to broadcast traffic
> >> consuming
> >> >>>>> capacity, which seemed to alleviate some impact. Investigations
> >> remained
> >> >>>>> ongoing. Following the isolation of the San Antonio, TX device
> >> >>>>> troubleshooting efforts focused on additional sites that teams
were
> >> >>>>> remotely unable to troubleshoot. Field Operations were dispatched
> >> to sites
> >> >>>>> in Kansas City, MO, Atlanta, GA, New Orleans, LA and Chicago, IL.
> >> Tier IV
> >> >>>>> Equipment Vendor Support continued to investigate the equipment
> >> logs to
> >> >>>>> further assist with isolation. Once visibility was restored to
the
> >> site in
> >> >>>>> Kansas City, MO a filter was applied to the equipment to further
> >> alleviate
> >> >>>>> the impact observed. All of the necessary troubleshooting teams
in
> >> >>>>> cooperation with Tier IV Equipment Vendor Support are working to
> >> restore
> >> >>>>> remote visibility to the remaining sites at this time. We
> >> understand how
> >> >>>>> important these services are to our clients and the issue has
been
> >> >>>>> escalated to the highest levels within CenturyLink Service
Assurance
> >> >>>>> Leadership.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of
*Bruce
> >> >>>>> Freshwater via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 1:24 PM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* Re: [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> CenturyLink finally acknowledged:
> >> >>>>>
> >> >>>>> Great Lakes Region CenturyLink Partners/Agents:
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> FYI. Our network is experiencing a disruption affecting customer
> >> >>>>> services. Customers are currently unable to generate tickets
> >> through the
> >> >>>>> Control Center portal.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> We know how important these services are to our customers, and we
> >> are
> >> >>>>> working to restore services as quickly as possible.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Our technical teams are working to resolve the issue, and we will
> >> >>>>> provide updates as the situation progresses.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> -bruce
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> On Thu, Dec 27, 2018 at 10:16 AM -0500, "Erik Sundberg via
Outages"
> >> <
> >> >>>>> outages at outages.org> wrote:
> >> >>>>>
> >> >>>>> So it's a bad day for centurylink.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> The lady who answered the phone for there NOC said that we are
> >> having
> >> >>>>> a nation wide outage and we are unable to open tickets at this
> >> time. Please
> >> >>>>> call be in 60-90 mins.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> She had no info on what was going on
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>> *From:* Michael Bolton <mbolton at holmeselectricsecurity.com>
> >> >>>>> *Sent:* Thursday, December 27, 2018 8:55:03 AM
> >> >>>>> *To:* Erik Sundberg; outages at outages.org
> >> >>>>> *Subject:* RE: CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Seeing the same in NC. Circuits are up but the routing between
POPs
> >> >>>>> has changed and we are seeing higher than normal latency.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> VoIP portal is down also. Allows you to login but you cannot make
> >> any
> >> >>>>> changes or view any details.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *From:* Outages <outages-bounces at outages.org> *On Behalf Of *Erik
> >> >>>>> Sundberg via Outages
> >> >>>>> *Sent:* Thursday, December 27, 2018 9:45 AM
> >> >>>>> *To:* outages at outages.org
> >> >>>>> *Subject:* [outages] CenturyLink Outages this morning
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> *CAUTION:* This email originated from outside of the
organization.
> >> Do
> >> >>>>> not click links or open attachments unless you recognize the
sender
> >> and
> >> >>>>> know the content is safe.
> >> >>>>>
> >> >>>>> We are seeing a lot of centurylink circuits and services down
this
> >> >>>>> morning.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Centurylink Control Center portal comes up but auth failing
unable
> >> to
> >> >>>>> login
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Waves down or bouncing
> >> >>>>>
> >> >>>>> Denver - Seattle (Bouncing)
> >> >>>>>
> >> >>>>> Denver - Chicago (Bouncing)
> >> >>>>>
> >> >>>>> New York - Los Angeles (Down)
> >> >>>>>
> >> >>>>> Chicago - Atlanta (Down)
> >> >>>>>
> >> >>>>>
> >> >>>>> ENNI Down in Denver (Down)
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> Still tying to open tickets with them there portal is not working
> >> and
> >> >>>>> stuck on hold right now.
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>> IMPORTANT NOTICE: This e-mail message is intended to be received
> >> only
> >> >>>>> by persons entitled to receive the confidential information it
may
> >> contain.
> >> >>>>> E-mail messages to clients of Holmes Security Systems may contain
> >> >>>>> information that is confidential and legally privileged. Please
do
> >> not
> >> >>>>> read, copy, forward, or store this message unless you are an
> >> intended
> >> >>>>> recipient of it. If you have received this message in error,
please
> >> forward
> >> >>>>> it to the sender and delete it completely from your computer
system.
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>>
> >> >>>>> _______________________________________________
> >> >>>>> Outages mailing list
> >> >>>>> Outages at outages.org
> >> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>>
> >> >>>>>
> >> >>>>> ------------------------------
> >> >>>>>
> >> >>>>> CONFIDENTIALITY NOTICE: This e-mail transmission, and any
documents,
> >> >>>>> files or previous e-mail messages attached to it may contain
> >> confidential
> >> >>>>> information that is legally privileged. If you are not the
intended
> >> >>>>> recipient, or a person responsible for delivering it to the
intended
> >> >>>>> recipient, you are hereby notified that any disclosure, copying,
> >> >>>>> distribution or use of any of the information contained in or
> >> attached to
> >> >>>>> this transmission is STRICTLY PROHIBITED. If you have received
this
> >> >>>>> transmission in error please notify the sender immediately by
> >> replying to
> >> >>>>> this e-mail. You must destroy the original transmission and its
> >> attachments
> >> >>>>> without reading or saving in any manner. Thank you.
> >> >>>>> _______________________________________________
> >> >>>>> Outages mailing list
> >> >>>>> Outages at outages.org
> >> >>>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>>
> >> >>>> _______________________________________________
> >> >>>> Outages mailing list
> >> >>>> Outages at outages.org
> >> >>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>
> >> >>>> _______________________________________________
> >> >>>> Outages mailing list
> >> >>>> Outages at outages.org
> >> >>>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>>
> >> >>> _______________________________________________
> >> >>> Outages mailing list
> >> >>> Outages at outages.org
> >> >>> https://puck.nether.net/mailman/listinfo/outages
> >> >>>
> >> >> _______________________________________________
> >> >> Outages mailing list
> >> >> Outages at outages.org
> >> >> https://puck.nether.net/mailman/listinfo/outages
> >> >>
> >> > _______________________________________________
> >> > Outages mailing list
> >> > Outages at outages.org
> >> > https://puck.nether.net/mailman/listinfo/outages
> >> >
> >> -------------- next part --------------
> >> An HTML attachment was scrubbed...
> >> URL: <
> >>
https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/atta
chment.html
> >> >
> >> -------------- next part --------------
> >> A non-text attachment was scrubbed...
> >> Name: image1.PNG
> >> Type: image/png
> >> Size: 55125 bytes
> >> Desc: not available
> >> URL: <
> >>
https://puck.nether.net/pipermail/outages/attachments/20181228/e8c66add/atta
chment.png
> >> >
> >>
> >> ------------------------------
> >>
> >> Subject: Digest Footer
> >>
> >> _______________________________________________
> >> Outages mailing list
> >> Outages at outages.org
> >> https://puck.nether.net/mailman/listinfo/outages
> >>
> >>
> >> ------------------------------
> >>
> >> End of Outages Digest, Vol 127, Issue 31
> >> ****************************************
> >>
> >
> >
> > --
> > *Aaron K. Henderson*
> >
> > Agri-Valley Communications, Inc.
> > Network Technician
> > *O:* (989) 453-4400 <9894534400>
> > *C:* (989) 551-0060 <9895510060>
> > *netadmin at avci.net <netadmin at avci.net> *
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages-discussion mailing list
Outages-discussion at outages.org
https://puck.nether.net/mailman/listinfo/outages-discussion
More information about the Outages-discussion
mailing list