[outages] NYC Outages

Tue Oct 30 11:56:31 EDT 2012

I had to summarize for an internal resource in the last few minutes, figure I would share:
--

Of our three datacenters, this is what we saw:

Parsippany 1 (OCT) - The worst we saw here was several sub-second power hits. UPS's held without problem, and we did not transfer to generator at all yet.

Parsippany 2 (WBR) - Transferred to generator at about 7:55 PM EST as a precautionary measure due to ongoing utility power hits. However, shortly after transfer, utility voltage went to 0 on all phases; around 10p power returned, but abnormally high (seeing about 550 volts on 480 volt bus). We remain on generator with at least 60 hours of fuel left on site.

Cedar Knolls 1 (MMU) - Briefly transferred to generator around 7:10, then back to utility. We then force transferred to generator around 8pm and stayed until this morning. Returned to utility and all systems are normal.

Fuel trucks are stationed at our facilities as of noon yesterday, and remain. Across all centers, consumption would be - in worst case - around 250 gallons per hour. We have 6,880 on site in generator tanks (27 hours), and additional 12,500 gallons (50 hours) on site available for transfer as needed, for a total of run time of 77 hours. However, we are only consuming fuel at one site right now (WBR, rate of 43 gal/hr).

We have accommodated several customers by allowing them to use our conference rooms and some offices, mainly because they lost power at their buildings, lost access to their buildings, or generators ran out of fuel.

There has been no service interruption of any sort to any of our datacenter customers, as of this writing.

We lost one span of fiber optics between MMU and 165 Halsey, Newark (NWR), network healed in less than 1 second. It was imperceptible. That was early though, yesterday afternoon.

We have sales people onsite who are actively taking sales calls, and there is a measurable amount of customers moving equipment into our facilities as I write this. Many are moving equipment from offices with no power and no ETR in sight. We have electricians onsite to help accommodate.

The facilities department has been manning all locations around the clock.

All sites remain accessible by road. There was never a situation where you could not over the last 24 hours.

We have no equipment in 111 8th, so I cannot offer any telemetry there. However, our gear in 60 Hudson St (Tel-X 9th floor) is still online.

We saw no interruption at 165 Halsey, except for some very brief utility hits. UPS's held.

We are seeing hundreds, if not thousands, of DSL / T1 / T3 / Ethernet customers down - presumably mostly power loss on the CPE. We lost our connection to Covad, presumably they have gear in a building which lost power.

We have seen substantial interruption to the cellular networks; in many cases, ATT Wireless (who NAC uses) is unable to complete a call. In certain areas, there is no signal at all.

Where I live, in 07874 (Byram), many roads are impassable and most everyone does not have power. Trees are down everywhere. I heard that a Byram squad car had a tree land on it while responding to a call. Schools are closed, Halloween has been "rescheduled" according to our governor. My cable is down, cable modem is down, POTS service is out - yet I still have xDSL (loop by Verizon, IP by NAC) and I run on that successfully. I am unable to get any ATT cell service at my house (usually at least three bars). I am on generator at my house, consuming about 2 gal/hour (propane), with a little over 500 gallons left.

That is all I have.

This is the road leading to the Byram Police Dept - I took this earlier this morning:

https://sphotos-a.xx.fbcdn.net/hphotos-prn1/12696_4311181250657_900557870_n.jpg

This is typically what Northern NJ looks like:

https://sphotos-a.xx.fbcdn.net/hphotos-ash3/548544_4311185050752_1011136048_n.jpg

From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On Behalf Of Alex Rubenstein
Sent: Tuesday, October 30, 2012 10:07 AM
To: outages at outages.org
Subject: [outages] NYC Outages

Hello,

Nothing to report outage-wise. However, if there is anything NAC can do to help anyone affected by the hurricane, please feel free to email me.

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20121030/9dc5ecf5/attachment.htm>