[VoiceOps] Sandy Watch

Wed Oct 31 20:43:37 EDT 2012

Major issues out of 75 broad with Windstream paetec. We have had no did's inbound from the for 24 hours. They are completely dark in that location. Mind you they are not just ny did's. they're nationwide did's 

First of two updates from Windstream all day

NY2 has been OOS since approx 8:30 last night. As a result of the storm surge Monday night, basement of the building that NY2 & NY3 are in, flooded. Late yesterday afternoon, the pump that moves diesel fuel from the basement to where our generators are located on the 18th floor, failed. Shortly thereafter, our generators ran out of fuel and shut down. Batteries to NY2 lasted 2+ hours while we unsuccessfully brainstormed alternative solutions to getting fuel to the 18th floor. Keep in mind, there is no power in this building and most if not all of the emergency lighting is no out, so flashlights are primary lighting. NY2 is also where majority of connectivity feeds through, to include some connectivity to Newark, so when the batteries ran down, we lost not only NY2, but effectively lost NY3 and some connectivity to Newark.

NY2 5E switch 16th floor
NY3 Soft switch 4th floor
Generators. 18th floor
Basement. Fuel tanks

We are currently working on 2 solutions in parallel:
1. Procuring a fuel lift pump and 200' of high pressure hose to pump our own fuel from ground level to 18th floor generators. Prime and restart generators, charge batteries for 30 min and begin rebooting equipment.
2. Contracted with local vendor to do something similar for us

Since they are both sub-optimal solutions, we are working both simultaneously to improve of chances of success. Best case is early afternoon restoration.

Building management, who is responsible for providing fuel to us continues to pump out the flooded basement in order to expose pumps, assess situation with intent of permanent repair/replacement of the main pump

Sent from my mobile device.

On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com> wrote:

> How are things going out there?  We're seeing pretty broad failures in both termination and origination with Verizon.  If we route calls out Level3, we can at least call out.
> 
> Thanks,
> 
> David
> 
> ________________________________________
> From: voiceops-bounces at voiceops.org [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger [jared at compuwizz.net]
> Sent: Tuesday, October 30, 2012 2:35 PM
> To: voiceops at voiceops.org
> Subject: Re: [VoiceOps] Sandy Watch
> 
> Looks like TATA NYC1 and 2 came back up. Telia went down. Belgacom is
> back up but taking the Washington -> Paris route instead of the NYC ->
> London route.
> 
> Novatel in NYC went completely down. IPs aren't even being advertised in BGP.
> 
> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger <jared at compuwizz.net> wrote:
>> We're connected to 2 others. The one in NJ seems up and we're
>> connected to London as well. Looks like the path across the Atlantic
>> is fine.
>> 
>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown <zavoid at gmail.com> wrote:
>>> If your connected with Tata hopefully your connected to at least one of the 13 other gsxes they have.
>>> 
>>> On Oct 30, 2012, at 9:34 AM, Jared Geiger <jared at compuwizz.net> wrote:
>>> 
>>>> 1) Looks like TATA NYC1 and NYC2 GSXes are down.
>>>> 2) I got an email saying Telia only had 6 hours of power left at 6 AM
>>>> Eastern, so they may go dark as well at noon.
>>>> 3) Our IP routes to Belgacom are broken and die in NYC on nlayer,
>>>> Cogent, and Abovenet.
>>>> 4) Looks like Hypercube in NYC is down.
>>>> 5) Reliance is going down in NYC
>>>> 
>>>> It seems all of our other trunks are up.
>>>> 
>>>> On Tue, Oct 30, 2012 at 2:37 AM, Hiers, David (DS) <David.Hiers at adp.com> wrote:
>>>>> Nothing too surprising so far...
>>>>> 
>>>>> There seems to be pretty strong correlation between flooded areas with no electricity and unregistered devices.  I probably don't need to drop a 650 into a fish tank to see if the issue is reproducible, but it is oddly tempting.
>>>>> 
>>>>> 
>>>>> David
>>>>> 
>>>>> 
>>>>> 
>>>>> ________________________________________
>>>>> From: voiceops-bounces at voiceops.org [voiceops-bounces at voiceops.org] On Behalf Of Hiers, David (DS)
>>>>> Sent: Monday, October 29, 2012 11:58 AM
>>>>> To: voiceops at voiceops.org
>>>>> Subject: [VoiceOps] Sandy Watch
>>>>> 
>>>>> Hi,
>>>>> 
>>>>> Is anyone seeing any storm related issues yet?  I've got the expected drop in calls as the NE shuts down, not much more on the radar.
>>>>> 
>>>>> I'll be watching/updating this list (and outages) tonight for any significant events.
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> David
>>>>> 
>>>>> PS:  Having your machines prepped for disaster is good, but what about your people?  All that expensive gear might not be not worth much if you've got no one to operate it when the big one hits.  After all, even first responders don't respond if doing so would imperil their families... are your people wired in to ready.gov?
>>>>> 
>>>>> 
>>>>> 
>>>>> ________________________________
>>>>> 
>>>>> ________________________________
>>>>> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>>>>> 
>>>>> 
>>>>> 
>>>>> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
>>>>> 
>>>>> _______________________________________________
>>>>> VoiceOps mailing list
>>>>> VoiceOps at voiceops.org
>>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>> 
>>>> _______________________________________________
>>>> VoiceOps mailing list
>>>> VoiceOps at voiceops.org
>>>> https://puck.nether.net/mailman/listinfo/voiceops
>>> 
>>> _______________________________________________
>>> VoiceOps mailing list
>>> VoiceOps at voiceops.org
>>> https://puck.nether.net/mailman/listinfo/voiceops
> 
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
> 
> 
> This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by e-mail and delete the message and any attachments from your system.
> 
> _______________________________________________
> VoiceOps mailing list
> VoiceOps at voiceops.org
> https://puck.nether.net/mailman/listinfo/voiceops
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/voiceops/attachments/20121031/4657783e/attachment-0001.html>