[VoiceOps] Sandy Watch
Darren Schreiber
darren at 2600hz.com
Wed Oct 31 22:56:40 EDT 2012
I also saw this interesting article on 911 emergency service availability.
It was an interesting read.
http://www.nytimes.com/2012/11/01/technology/fcc-details-cellphone-problems
.html
And then of course there's this:
http://www.cnn.com/2012/10/31/tech/mobile/att-tmobile-networks-sandy/
As for outages...
We ourselves have been seeing fluctuations in network traffic to NAC for
very short durations of time (like a few seconds) but have still kept our
traffic away from the east coast for now, until we hear nobody is on
generators anymore.
And finally, huge kudos to these guys. Sometimes people forget there are
real humans who operate data centers, and they do amazing things to
protect our silly little computers in the event of emergency. Amazing
pictures.
http://www.huffingtonpost.com/2012/10/31/sandy-data-center_n_2051659.html
--
Darren Schreiber
CEO / Co-Founder
2600hz
On 10/31/12 7:11 PM, "Matt Yaklin" <myaklin at g4.net> wrote:
>
>Here is some info I was reading earlier today and thought
>people on this list might enjoy reading it if they have not
>already.
>
>http://www.webhostingtalk.com/showthread.php?t=1205042
>
>http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover-
>after-sandy/
>
>matt
>
>On Wed, 31 Oct 2012, Shripal Daphtary wrote:
>
>> Major issues out of 75 broad with Windstream paetec. We have had no
>>did's
>> inbound from the for 24 hours. They are completely dark in that
>>location.
>> Mind you they are not just ny did's. they're nationwide did's
>>
>> First of two updates from Windstream all day
>>
>>
>>
>> NY2 has been OOS since approx 8:30 last night. As a result of the storm
>> surge Monday night, basement of the building that NY2 & NY3 are in,
>>flooded.
>> Late yesterday afternoon, the pump that moves diesel fuel from the
>>basement
>> to where our generators are located on the 18th floor, failed. Shortly
>> thereafter, our generators ran out of fuel and shut down. Batteries to
>>NY2
>> lasted 2+ hours while we unsuccessfully brainstormed alternative
>>solutions
>> to getting fuel to the 18th floor. Keep in mind, there is no power in
>>this
>> building and most if not all of the emergency lighting is no out, so
>> flashlights are primary lighting. NY2 is also where majority of
>>connectivity
>> feeds through, to include some connectivity to Newark, so when the
>>batteries
>> ran down, we lost not only NY2, but effectively lost NY3 and some
>> connectivity to Newark.
>>
>> NY2 5E switch 16th floor
>> NY3 Soft switch 4th floor
>> Generators. 18th floor
>> Basement. Fuel tanks
>>
>> We are currently working on 2 solutions in parallel:
>> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump
>>our own
>> fuel from ground level to 18th floor generators. Prime and restart
>> generators, charge batteries for 30 min and begin rebooting equipment.
>> 2. Contracted with local vendor to do something similar for us
>>
>> Since they are both sub-optimal solutions, we are working both
>> simultaneously to improve of chances of success. Best case is early
>> afternoon restoration.
>>
>> Building management, who is responsible for providing fuel to us
>>continues
>> to pump out the flooded basement in order to expose pumps, assess
>>situation
>> with intent of permanent repair/replacement of the main pump
>>
>>
>>
>>
>> Sent from my mobile device.
>>
>> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com>
>> wrote:
>>
>> How are things going out there? We're seeing pretty broad
>> failures in both termination and origination with Verizon. If
>> we route calls out Level3, we can at least call out.
>>
>>
>> Thanks,
>>
>>
>> David
>>
>>
>> ________________________________________
>>
>> From: voiceops-bounces at voiceops.org
>> [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger
>> [jared at compuwizz.net]
>>
>> Sent: Tuesday, October 30, 2012 2:35 PM
>>
>> To: voiceops at voiceops.org
>>
>> Subject: Re: [VoiceOps] Sandy Watch
>>
>>
>> Looks like TATA NYC1 and 2 came back up. Telia went down.
>> Belgacom is
>>
>> back up but taking the Washington -> Paris route instead of the
>> NYC ->
>>
>> London route.
>>
>>
>> Novatel in NYC went completely down. IPs aren't even being
>> advertised in BGP.
>>
>>
>> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
>> <jared at compuwizz.net> wrote:
>>
>> We're connected to 2 others. The one in NJ seems up
>> and we're
>>
>> connected to London as well. Looks like the path
>> across the Atlantic
>>
>> is fine.
>>
>>
>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
>> <zavoid at gmail.com> wrote:
>>
>> If your connected with Tata hopefully
>> your connected to at least one of the 13
>> other gsxes they have.
>>
>>
>> On Oct 30, 2012, at 9:34 AM, Jared
>> Geiger <jared at compuwizz.net> wrote:
>>
>>
>> 1) Looks like TATA NYC1 and
>> NYC2 GSXes are down.
>>
>> 2) I got an email saying
>> Telia only had 6 hours of
>> power left at 6 AM
>>
>> Eastern, so they may go dark
>> as well at noon.
>>
>> 3) Our IP routes to Belgacom
>> are broken and die in NYC on
>> nlayer,
>>
>> Cogent, and Abovenet.
>>
>> 4) Looks like Hypercube in
>> NYC is down.
>>
>> 5) Reliance is going down in
>> NYC
>>
>>
>> It seems all of our other
>> trunks are up.
>>
>>
>> On Tue, Oct 30, 2012 at 2:37
>> AM, Hiers, David (DS)
>> <David.Hiers at adp.com> wrote:
>>
>> Nothing too
>> surprising so
>> far...
>>
>>
>> There seems to
>> be pretty strong
>> correlation
>> between flooded
>> areas with no
>> electricity and
>> unregistered
>> devices. I
>> probably don't
>> need to drop a
>> 650 into a fish
>> tank to see if
>> the issue is
>> reproducible,
>> but it is oddly
>> tempting.
>>
>>
>>
>> David
>>
>>
>>
>>
>> ________________________________________
>>
>> From:
>> voiceops-bounces at voiceops.org
>> [voiceops-bounces at voiceops.org]
>> On Behalf Of
>> Hiers, David
>> (DS)
>>
>> Sent: Monday,
>> October 29, 2012
>> 11:58 AM
>>
>> To:
>> voiceops at voiceops.org
>>
>> Subject:
>> [VoiceOps] Sandy
>> Watch
>>
>>
>> Hi,
>>
>>
>> Is anyone seeing
>> any storm
>> related issues
>> yet? I've got
>> the expected
>> drop in calls as
>> the NE shuts
>> down, not much
>> more on the
>> radar.
>>
>>
>> I'll be
>> watching/updating
>> this list (and
>> outages) tonight
>> for any
>> significant
>> events.
>>
>>
>> Thanks,
>>
>>
>> David
>>
>>
>> PS: Having your
>> machines prepped
>> for disaster is
>> good, but what
>> about your
>> people? All
>> that expensive
>> gear might not
>> be not worth
>> much if you've
>> got no one to
>> operate it when
>> the big one
>> hits. After
>> all, even first
>> responders don't
>> respond if doing
>> so would imperil
>> their
>> families... are
>> your people
>> wired in to
>> ready.gov?
>>
>>
>>
>>
>> ________________________________
>>
>>
>> ________________________________
>>
>> This message and
>> any attachments
>> are intended
>> only for the use
>> of the addressee
>> and may contain
>> information that
>> is privileged
>> and
>> confidential. If
>> the reader of
>> the message is
>> not the intended
>> recipient or an
>> authorized
>> representative
>> of the intended
>> recipient, you
>> are hereby
>> notified that
>> any
>> dissemination of
>> this
>> communication is
>> strictly
>> prohibited. If
>> you have
>> received this
>> communication in
>> error, please
>> notify us
>> immediately by
>> e-mail and
>> delete the
>> message and any
>> attachments from
>> your system.
>>
>>
>>
>>
>> This message and
>> any attachments
>> are intended
>> only for the use
>> of the addressee
>> and may contain
>> information that
>> is privileged
>> and
>> confidential. If
>> the reader of
>> the message is
>> not the intended
>> recipient or an
>> authorized
>> representative
>> of the intended
>> recipient, you
>> are hereby
>> notified that
>> any
>> dissemination of
>> this
>> communication is
>> strictly
>> prohibited. If
>> you have
>> received this
>> communication in
>> error, please
>> notify us
>> immediately by
>> e-mail and
>> delete the
>> message and any
>> attachments from
>> your system.
>>
>>
>>
>>_______________________________________________
>>
>> VoiceOps mailing
>> list
>>
>> VoiceOps at voiceops.org
>>
>>
>>https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>>
>>https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
>>
>> This message and any attachments are intended only for the use
>> of the addressee and may contain information that is privileged
>> and confidential. If the reader of the message is not the
>> intended recipient or an authorized representative of the
>> intended recipient, you are hereby notified that any
>> dissemination of this communication is strictly prohibited. If
>> you have received this communication in error, please notify us
>> immediately by e-mail and delete the message and any attachments
>> from your system.
>>
>>
>> _______________________________________________
>>
>> VoiceOps mailing list
>>
>> VoiceOps at voiceops.org
>>
>> https://puck.nether.net/mailman/listinfo/voiceops
>>
>>
More information about the VoiceOps
mailing list