[outages] NYC Outages

[Mitch]

Ouch, and RE: Covad/Megapath lines, in NY they have been lighting up
our email with outage reports, they all end up with a status of:

Description:
We are currently experiencing a service interruption due to equipment
failure at the NYCMNYBS central office in New York, NY.

UPDATE: Due to hurricane storm Sandy, the city is under full state of
emergency as roads and bridges are closed. Services will remain down
during duration of storm as no dispatches can be made. NOC will send
further update whenever there is change in condition.

On Tue, Oct 30, 2012 at 11:56 AM, Alex Rubenstein <alex at corp.nac.net> wrote:
> I had to summarize for an internal resource in the last few minutes, figure
> I would share:
>
> --
>
>
>
> Of our three datacenters, this is what we saw:
>
>
>
> Parsippany 1 (OCT) – The worst we saw here was several sub-second power
> hits. UPS’s held without problem, and we did not transfer to generator at
> all yet.
>
>
>
> Parsippany 2 (WBR) – Transferred to generator at about 7:55 PM EST as a
> precautionary measure due to ongoing utility power hits. However, shortly
> after transfer, utility voltage went to 0 on all phases; around 10p power
> returned, but abnormally high (seeing about 550 volts on 480 volt bus). We
> remain on generator with at least 60 hours of fuel left on site.
>
>
>
> Cedar Knolls 1 (MMU) – Briefly transferred to generator around 7:10, then
> back to utility. We then force transferred to generator around 8pm and
> stayed until this morning. Returned to utility and all systems are normal.
>
>
>
> Fuel trucks are stationed at our facilities as of noon yesterday, and
> remain. Across all centers, consumption would be - in worst case - around
> 250 gallons per hour. We have 6,880 on site in generator tanks (27 hours),
> and additional 12,500 gallons (50 hours) on site available for transfer as
> needed, for a total of run time of 77 hours. However, we are only consuming
> fuel at one site right now (WBR, rate of 43 gal/hr).
>
>
>
> We have accommodated several customers by allowing them to use our
> conference rooms and some offices, mainly because they lost power at their
> buildings, lost access to their buildings, or generators ran out of fuel.
>
>
>
> There has been no service interruption of any sort to any of our datacenter
> customers, as of this writing.
>
>
>
> We lost one span of fiber optics between MMU and 165 Halsey, Newark (NWR),
> network healed in less than 1 second. It was imperceptible. That was early
> though, yesterday afternoon.
>
>
>
> We have sales people onsite who are actively taking sales calls, and there
> is a measurable amount of customers moving equipment into our facilities as
> I write this. Many are moving equipment from offices with no power and no
> ETR in sight. We have electricians onsite to help accommodate.
>
>
>
> The facilities department has been manning all locations around the clock.
>
>
>
> All sites remain accessible by road. There was never a situation where you
> could not over the last 24 hours.
>
>
>
> We have no equipment in 111 8th, so I cannot offer any telemetry there.
> However, our gear in 60 Hudson St (Tel-X 9th floor) is still online.
>
>
>
> We saw no interruption at 165 Halsey, except for some very brief utility
> hits. UPS’s held.
>
>
>
> We are seeing hundreds, if not thousands, of DSL / T1 / T3 / Ethernet
> customers down – presumably mostly power loss on the CPE. We lost our
> connection to Covad, presumably they have gear in a building which lost
> power.
>
>
>
> We have seen substantial interruption to the cellular networks; in many
> cases, ATT Wireless (who NAC uses) is unable to complete a call. In certain
> areas, there is no signal at all.
>
>
>
> Where I live, in 07874 (Byram), many roads are impassable and most everyone
> does not have power. Trees are down everywhere. I heard that a Byram squad
> car had a tree land on it while responding to a call. Schools are closed,
> Halloween has been “rescheduled” according to our governor. My cable is
> down, cable modem is down, POTS service is out – yet I still have xDSL (loop
> by Verizon, IP by NAC) and I run on that successfully. I am unable to get
> any ATT cell service at my house (usually at least three bars). I am on
> generator at my house, consuming about 2 gal/hour (propane), with a little
> over 500 gallons left.
>
>
>
> That is all I have.
>
>
>
> This is the road leading to the Byram Police Dept – I took this earlier this
> morning:
>
> https://sphotos-a.xx.fbcdn.net/hphotos-prn1/12696_4311181250657_900557870_n.jpg
>
>
>
> This is typically what Northern NJ looks like:
>
>
>
> https://sphotos-a.xx.fbcdn.net/hphotos-ash3/548544_4311185050752_1011136048_n.jpg
>
>
>
>
>
>
>
>
>
>
>
>
>
> From: outages-bounces at outages.org [mailto:outages-bounces at outages.org] On
> Behalf Of Alex Rubenstein
> Sent: Tuesday, October 30, 2012 10:07 AM
> To: outages at outages.org
> Subject: [outages] NYC Outages
>
>
>
> Hello,
>
>
>
> Nothing to report outage-wise. However, if there is anything NAC can do to
> help anyone affected by the hurricane, please feel free to email me.
>
>
>
>
>
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>