[Outages-discussion] [outages] Evocative EWR1 (formerly INAP NYJ004)

[Phil K.]

The outage is also very dependant on what tripped power in the first place.
We've been on the receiving end of multiple outages owing to a wonky &
poorly documented Emergency Power Off circuit in a datacenter. If the fire
suppression system engages, power to some / most / all systems (depending
on configuration) gets cut to protect firefighters. In our case, there was
*supposed* to be a bypass for the EPO, e.g. "turn this key to prevent the
EPO from engaging when you do your annual fire alarm testing & maintenance."

Well, that testing procedure hadn't been fully followed for a number of
years. Someone finally got a wild hair and decided to do it by the book.
Turn the key...engage the fire alarm test...(whump) dark datacenter...turns
out that system didn't *quite* follow the schematics the old vendor
provided. But at least the EPO functioned exactly as it was supposed to.
Good times...good times...

New Jersey also lost a pair of firefighters this week while responding to a
ship board fire. They're going to be a might touchy about firefighter
safety issues for a while.
~Phil

On Tue, Jul 11, 2023, 8:24 PM Matthew Petach via Outages-discussion <
outages-discussion at outages.org> wrote:

>
> Many, many years ago when I was in a similar position, the fire marshal
> told me in no uncertain terms that if I fired up our genset before he gave
> the all-clear, he had the authority revoke our occupancy permit for the
> building.  As fire marshals are often sworn members of law enforcement,
> authorized to carry a weapon and enforce the law, I felt it would be
> prudent to follow their directives and leave the building dark until they
> gave the all-clear.
>
> I don't think the folks there want to jeopardize their business by jumping
> the gun and disobeying the fire marshal.  ^_^;
>
>
> On Tue, Jul 11, 2023 at 5:09 PM Charles Sprickman via Outages-discussion <
> outages-discussion at outages.org> wrote:
>
>> This is New Jersey, sounds like some palms have not been properly
>> greased. :)
>>
>> "Nice datacenter you got here, shame if it didn't have some
>> electricity..."
>>
>> Curious what would happen if they just opted to power from the gensets.
>>
>> Charles
>>
>> > On Jul 11, 2023, at 7:57 PM, Christopher J. Pilkington via Outages <
>> outages at outages.org> wrote:
>> >
>> > On Tue, Jul 11, 2023, at 08:52, Alex via Outages wrote:
>> >> Looks like EWR1 will be down at least until the afternoon.
>> >
>> > Latest update, 9am meet with fire marshal. Best case, customer loads
>> energized 4-5 hours after.
>> >
>> > -cjp
>> > _______________________________________________
>> > Outages mailing list
>> > Outages at outages.org
>> > https://puck.nether.net/mailman/listinfo/outages
>>
>> _______________________________________________
>> Outages-discussion mailing list
>> Outages-discussion at outages.org
>> https://puck.nether.net/mailman/listinfo/outages-discussion
>>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages-discussion/attachments/20230711/9c6aaf18/attachment.htm>