[outages] Linode outage in Newark, NJ starting at 3:30 pm

[Blake Pfankuch]

Sounds like to me they possibly switched over to battery successfully, however battery power was not sufficient while they got the generator functioning.  Most large datacenters I have experience with have 30-45 minutes of battery.  In an instance where you have generator issues, sometimes it could take a half an hour or more to get something functioning.  

We had an issue in our small datacenter about 2 years ago where the ATS didn't properly detect the power failure (long story, bad electrician).  As it occurred at about 2am (we are not 24/7 in building) it took me nearly 30 minutes to get to the office thanks to snow.  I barely beat the battery as everything was under  10% remaining.

I have had customers in the same situation where their generator didn't start properly for one reason or another.  Even with staff in house, it still has taken 20-25 minutes to get generators started.  

This to me stresses the importance of planned generator testing.  Testing the generator, battery systems and ATS systems is very critical.  This is why about once a month, I flip the mains to the datacenter (planned of course) to make sure everything is still fully functional.  A planned 30 second test alerts us of any possible physical issues.  Also by planning these we find out about possible issues sooner rather than later.  If an ATS fails during a test, we ride on battery a few minutes and I go back to utility with no harm done.

Blake 
Cisco, Microsoft, Adtran and VMware Certification Information available upon request.

-----Original Message-----
From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Lonnie Bozeman
Sent: Friday, August 31, 2012 4:53 PM
To: reza a; outages at outages.org
Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm

It is common to have not only strings of batteries but an ATS, or automatic transfer switch. In this case, it sounds like the fault was on the ATS side of the plant based on the IR below.

- LB

-----Original Message-----
From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of reza a
Sent: Friday, August 31, 2012 3:29 PM
To: outages at outages.org
Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm

Is it common practice to have an array of batteries in between the colo and the generators to give you time to start the generators?

----- Original Message -----
From: "Sadiq Saif" <sadiq at asininetech.com>
To: "Jack Carrozzo" <jack at crepinc.com>
Cc: outages at outages.org
Sent: Friday, August 31, 2012 2:59:16 PM
Subject: Re: [outages] Linode outage in Newark, NJ starting at 3:30 pm

This is the incident report from NAC:
Subsequent to the utility power failure at our Cedar Knolls facility the generator that powers systems A and D did not start automatically.
Manual intervention was required to get generator power running.
Unfortunately before the generator was manually engaged customers on these systems experienced a complete power loss. We are still investigating why the generator did not engage automatically.Utility power has been restored, and transfer back to utility was successful.

On Fri, Aug 31, 2012 at 3:59 PM, Jack Carrozzo <jack at crepinc.com> wrote:
> 3:52pm (EDT): NAC has informed us of a power issue affecting at least 
> some portion of the datacenter. As soon as power is restored we are 
> poised to execute our recovery procedures to all affected systems.
>
> On Fri, Aug 31, 2012 at 3:46 PM, Smith, Kyle <ksmith at litle.com> wrote:
>> FYI,
>>
>>
>>
>> http://status.linode.com/
>>
>>
>>
>> We started getting alerts at roughly ~3:30
>>
>>
>>
>> - Kyle
>>
>>
>>
>> The information in this message is for the intended recipient(s) only 
>> and may be the proprietary and/or confidential property of Litle & 
>> Co., LLC, and thus protected from disclosure. If you are not the 
>> intended recipient(s), or an employee or agent responsible for 
>> delivering this message to the intended recipient, you are hereby 
>> notified that any use, dissemination, distribution or copying of this 
>> communication is prohibited. If you have received this communication 
>> in error, please notify Litle & Co. immediately by replying to this 
>> message and then promptly deleting it and your reply permanently from your computer.
>>
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
>>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages



--
Sadiq S
O< ascii ribbon campaign - stop html mail - www.asciiribbon.org _______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages
_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages

_______________________________________________
Outages mailing list
Outages at outages.org
https://puck.nether.net/mailman/listinfo/outages