[Outages-discussion] [outages] Centurylink outage and list best practice

[Jay Hennigan]

On 12/28/18 10:11 AM, Aaron Henderson via Outages wrote:
> My apologies if this message breaks the mailing list, I have never 
> replied to one of these before.

A few basics:

* Don't reply to an existing topic to start a new topic. Just compose a 
new email with an appropriate subject line and address it to the list.

* If you're subscribed in digest form and are replying to a specific 
topic in the digest, change your subject line to that topic. Don't leave 
it as, "Outages digest..."

* Don't quote more of the original message than is pertinent to your 
reply. In particular, don't ever quote an entire digest to comment about 
one topic within it.

* The outages list is for active reporting and timely updates of new, 
existing and ongoing significant outages only. Many network operators 
have the outages list tied to an alerting system such as a pager or 
smartphone alert. This list is intended as an early warning system of 
problems with major infrastructure. It's also worldwide. Something at 4 
PM in your timezone will be at 4 AM to others monitoring the list.

* There is a parallel list, outages-discussion at outages.org, for 
discussion, follow-up, post-mortem, links to RFO and the like. 
Discussion regarding previously reported outages should take place on 
outages-discussion. I've moved my reply to you there as well as replying 
to you directly. You probably want to subscribe to both to get the full 
picture because questions and followups regarding a previously reported 
issue should always go to outages-discussion. This reduces noise on the 
"outages" list and keeps it useful as an early warning tool while 
allowing commentary on the discussion list.

> I work for a rural ISP, we do not use CenturyLink as an upstream; 
> however, some of our customers are reporting issues since the outage. 
> Management has tasked my office with putting together a timeline/details 
> about the outage to brief upper management. Is there a detailed timeline 
> of events documenting this outage?

There will probably be an official statement from Centurylink within the 
next few days. Keep in mind that official company statements and RFOs, 
especially from a company as large as Centurylink and an outage as 
significant as this one, start with the technical people who know what 
really happened. They are then often filtered, massaged, and wordsmithed 
through an array of lawyers, marketing people, and assorted spin doctors 
before being released to the public. Take these with several grains of salt.

The reason that your customers were impacted is that this was a large 
outage affecting a major transport provider over an extended period of 
time. Any sites or resources that your customers were trying to reach 
that required passing through Centurylink facilities would have been 
down or impaired.

I suggest that you subscribe to outages-discussion and review the 
archives for information there as well as the links therein pointing to 
Reddit and other operators' experience.

-- 
Jay Hennigan - jay at west.net
Network Engineering - CCIE #7880
503 897-8550 - WB6RDV