[VoiceOps] Sandy Watch
Matt Yaklin
myaklin at g4.net
Wed Oct 31 22:11:28 EDT 2012
Here is some info I was reading earlier today and thought
people on this list might enjoy reading it if they have not
already.
http://www.webhostingtalk.com/showthread.php?t=1205042
http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover-after-sandy/
matt
On Wed, 31 Oct 2012, Shripal Daphtary wrote:
> Major issues out of 75 broad with Windstream paetec. We have had no did's
> inbound from the for 24 hours. They are completely dark in that location.
> Mind you they are not just ny did's. they're nationwide did's
>
> First of two updates from Windstream all day
>
>
>
> NY2 has been OOS since approx 8:30 last night. As a result of the storm
> surge Monday night, basement of the building that NY2 & NY3 are in, flooded.
> Late yesterday afternoon, the pump that moves diesel fuel from the basement
> to where our generators are located on the 18th floor, failed. Shortly
> thereafter, our generators ran out of fuel and shut down. Batteries to NY2
> lasted 2+ hours while we unsuccessfully brainstormed alternative solutions
> to getting fuel to the 18th floor. Keep in mind, there is no power in this
> building and most if not all of the emergency lighting is no out, so
> flashlights are primary lighting. NY2 is also where majority of connectivity
> feeds through, to include some connectivity to Newark, so when the batteries
> ran down, we lost not only NY2, but effectively lost NY3 and some
> connectivity to Newark.
>
> NY2 5E switch 16th floor
> NY3 Soft switch 4th floor
> Generators. 18th floor
> Basement. Fuel tanks
>
> We are currently working on 2 solutions in parallel:
> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump our own
> fuel from ground level to 18th floor generators. Prime and restart
> generators, charge batteries for 30 min and begin rebooting equipment.
> 2. Contracted with local vendor to do something similar for us
>
> Since they are both sub-optimal solutions, we are working both
> simultaneously to improve of chances of success. Best case is early
> afternoon restoration.
>
> Building management, who is responsible for providing fuel to us continues
> to pump out the flooded basement in order to expose pumps, assess situation
> with intent of permanent repair/replacement of the main pump
>
>
>
>
> Sent from my mobile device.
>
> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com>
> wrote:
>
> How are things going out there? We're seeing pretty broad
> failures in both termination and origination with Verizon. If
> we route calls out Level3, we can at least call out.
>
>
> Thanks,
>
>
> David
>
>
> ________________________________________
>
> From: voiceops-bounces at voiceops.org
> [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger
> [jared at compuwizz.net]
>
> Sent: Tuesday, October 30, 2012 2:35 PM
>
> To: voiceops at voiceops.org
>
> Subject: Re: [VoiceOps] Sandy Watch
>
>
> Looks like TATA NYC1 and 2 came back up. Telia went down.
> Belgacom is
>
> back up but taking the Washington -> Paris route instead of the
> NYC ->
>
> London route.
>
>
> Novatel in NYC went completely down. IPs aren't even being
> advertised in BGP.
>
>
> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
> <jared at compuwizz.net> wrote:
>
> We're connected to 2 others. The one in NJ seems up
> and we're
>
> connected to London as well. Looks like the path
> across the Atlantic
>
> is fine.
>
>
> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
> <zavoid at gmail.com> wrote:
>
> If your connected with Tata hopefully
> your connected to at least one of the 13
> other gsxes they have.
>
>
> On Oct 30, 2012, at 9:34 AM, Jared
> Geiger <jared at compuwizz.net> wrote:
>
>
> 1) Looks like TATA NYC1 and
> NYC2 GSXes are down.
>
> 2) I got an email saying
> Telia only had 6 hours of
> power left at 6 AM
>
> Eastern, so they may go dark
> as well at noon.
>
> 3) Our IP routes to Belgacom
> are broken and die in NYC on
> nlayer,
>
> Cogent, and Abovenet.
>
> 4) Looks like Hypercube in
> NYC is down.
>
> 5) Reliance is going down in
> NYC
>
>
> It seems all of our other
> trunks are up.
>
>
> On Tue, Oct 30, 2012 at 2:37
> AM, Hiers, David (DS)
> <David.Hiers at adp.com> wrote:
>
> Nothing too
> surprising so
> far...
>
>
> There seems to
> be pretty strong
> correlation
> between flooded
> areas with no
> electricity and
> unregistered
> devices. I
> probably don't
> need to drop a
> 650 into a fish
> tank to see if
> the issue is
> reproducible,
> but it is oddly
> tempting.
>
>
>
> David
>
>
>
>
> ________________________________________
>
> From:
> voiceops-bounces at voiceops.org
> [voiceops-bounces at voiceops.org]
> On Behalf Of
> Hiers, David
> (DS)
>
> Sent: Monday,
> October 29, 2012
> 11:58 AM
>
> To:
> voiceops at voiceops.org
>
> Subject:
> [VoiceOps] Sandy
> Watch
>
>
> Hi,
>
>
> Is anyone seeing
> any storm
> related issues
> yet? I've got
> the expected
> drop in calls as
> the NE shuts
> down, not much
> more on the
> radar.
>
>
> I'll be
> watching/updating
> this list (and
> outages) tonight
> for any
> significant
> events.
>
>
> Thanks,
>
>
> David
>
>
> PS: Having your
> machines prepped
> for disaster is
> good, but what
> about your
> people? All
> that expensive
> gear might not
> be not worth
> much if you've
> got no one to
> operate it when
> the big one
> hits. After
> all, even first
> responders don't
> respond if doing
> so would imperil
> their
> families... are
> your people
> wired in to
> ready.gov?
>
>
>
>
> ________________________________
>
>
> ________________________________
>
> This message and
> any attachments
> are intended
> only for the use
> of the addressee
> and may contain
> information that
> is privileged
> and
> confidential. If
> the reader of
> the message is
> not the intended
> recipient or an
> authorized
> representative
> of the intended
> recipient, you
> are hereby
> notified that
> any
> dissemination of
> this
> communication is
> strictly
> prohibited. If
> you have
> received this
> communication in
> error, please
> notify us
> immediately by
> e-mail and
> delete the
> message and any
> attachments from
> your system.
>
>
>
>
> This message and
> any attachments
> are intended
> only for the use
> of the addressee
> and may contain
> information that
> is privileged
> and
> confidential. If
> the reader of
> the message is
> not the intended
> recipient or an
> authorized
> representative
> of the intended
> recipient, you
> are hereby
> notified that
> any
> dissemination of
> this
> communication is
> strictly
> prohibited. If
> you have
> received this
> communication in
> error, please
> notify us
> immediately by
> e-mail and
> delete the
> message and any
> attachments from
> your system.
>
>
> _______________________________________________
>
> VoiceOps mailing
> list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
> This message and any attachments are intended only for the use
> of the addressee and may contain information that is privileged
> and confidential. If the reader of the message is not the
> intended recipient or an authorized representative of the
> intended recipient, you are hereby notified that any
> dissemination of this communication is strictly prohibited. If
> you have received this communication in error, please notify us
> immediately by e-mail and delete the message and any attachments
> from your system.
>
>
> _______________________________________________
>
> VoiceOps mailing list
>
> VoiceOps at voiceops.org
>
> https://puck.nether.net/mailman/listinfo/voiceops
>
>
>
More information about the VoiceOps
mailing list