[outages] Linode Fremont outage

[DTNX Postmaster]

On 31 May 2015, at 18:47, ryanL via Outages <outages at outages.org> wrote:

> for all we know the generator was proactively tested on a regular basis. testing doesn't preclude failure on a subsequent run.

And until they release a thorough post-incident report, there's no more value in your answer than in Jim's. Like, 'for all we know', their systems are oversubscribed, they're behind on maintenance and planned upgrades, and their tests aren't reflective of the actual load during an outage.

Eight generators total, and the failure of a single one takes out an entire suite. If I were Linode, I'd want a damn good explanation for that one.

--

> On Sun, May 31, 2015 at 9:23 AM Jim Popovitch via Outages <outages at outages.org> wrote:
> On Sun, May 31, 2015 at 12:16 PM, Chris Vervais <cvervais at me.com> wrote:
> >
> >> On May 31, 2015, at 08:06, Jim Popovitch via Outages <outages at outages.org> wrote:
> >>
> >> On Sun, May 31, 2015 at 11:00 AM, Mark Keymer via Outages
> >> <outages at outages.org> wrote:
> >>> But honestly I am guessing it was a failure with no foul play happening.
> >>
> >> Apparently no solid proactive testing plan either.
> >
> > For all we know the generator was checked last week and was fine.
> >
> > In the absence of all the facts we should probably restrain from assuming one thing or another and give them the benefit of the doubt. And, that they're taking this is a learning experience and delving into where gaps exist and how to close them.
> 
> Of course, for a new DC or relatively new DC operator.  But come on,
> HE @ FMT2 are where I would expect perfection.
> 
> -Jim P.