[VoiceOps] Sandy Watch
Darren Schreiber
d at d-man.org
Wed Oct 31 23:00:22 EDT 2012
Oops, I included the wrong picture link for Peer1. And no, I'm not getting
paid by them for ads or anything (we don't even host there - it's just a
cool story).
http://status.squarespace.com/ . The pictures are just too crazy.
Also, to be clear on NAC, they've been rock solid for us, the internet
blips are likely transport/fiber between us in California and NAC in New
Jersey.
Hope everyone weathered the storm OK. Good luck out there.
On 10/31/12 7:56 PM, "Darren Schreiber" <darren at 2600hz.com> wrote:
>I also saw this interesting article on 911 emergency service availability.
>It was an interesting read.
>http://www.nytimes.com/2012/11/01/technology/fcc-details-cellphone-problem
>s
>.html
>
>
>And then of course there's this:
>http://www.cnn.com/2012/10/31/tech/mobile/att-tmobile-networks-sandy/
>
>
>
>As for outages...
>We ourselves have been seeing fluctuations in network traffic to NAC for
>very short durations of time (like a few seconds) but have still kept our
>traffic away from the east coast for now, until we hear nobody is on
>generators anymore.
>
>
>And finally, huge kudos to these guys. Sometimes people forget there are
>real humans who operate data centers, and they do amazing things to
>protect our silly little computers in the event of emergency. Amazing
>pictures.
>http://www.huffingtonpost.com/2012/10/31/sandy-data-center_n_2051659.html
>
>
>--
>Darren Schreiber
>CEO / Co-Founder
>2600hz
>
>
>
>
>
>On 10/31/12 7:11 PM, "Matt Yaklin" <myaklin at g4.net> wrote:
>
>>
>>Here is some info I was reading earlier today and thought
>>people on this list might enjoy reading it if they have not
>>already.
>>
>>http://www.webhostingtalk.com/showthread.php?t=1205042
>>
>>http://slashdot.org/topic/datacenter/nyc-data-centers-struggle-to-recover
>>-
>>after-sandy/
>>
>>matt
>>
>>On Wed, 31 Oct 2012, Shripal Daphtary wrote:
>>
>>> Major issues out of 75 broad with Windstream paetec. We have had no
>>>did's
>>> inbound from the for 24 hours. They are completely dark in that
>>>location.
>>> Mind you they are not just ny did's. they're nationwide did's
>>>
>>> First of two updates from Windstream all day
>>>
>>>
>>>
>>> NY2 has been OOS since approx 8:30 last night. As a result of the storm
>>> surge Monday night, basement of the building that NY2 & NY3 are in,
>>>flooded.
>>> Late yesterday afternoon, the pump that moves diesel fuel from the
>>>basement
>>> to where our generators are located on the 18th floor, failed. Shortly
>>> thereafter, our generators ran out of fuel and shut down. Batteries to
>>>NY2
>>> lasted 2+ hours while we unsuccessfully brainstormed alternative
>>>solutions
>>> to getting fuel to the 18th floor. Keep in mind, there is no power in
>>>this
>>> building and most if not all of the emergency lighting is no out, so
>>> flashlights are primary lighting. NY2 is also where majority of
>>>connectivity
>>> feeds through, to include some connectivity to Newark, so when the
>>>batteries
>>> ran down, we lost not only NY2, but effectively lost NY3 and some
>>> connectivity to Newark.
>>>
>>> NY2 5E switch 16th floor
>>> NY3 Soft switch 4th floor
>>> Generators. 18th floor
>>> Basement. Fuel tanks
>>>
>>> We are currently working on 2 solutions in parallel:
>>> 1. Procuring a fuel lift pump and 200' of high pressure hose to pump
>>>our own
>>> fuel from ground level to 18th floor generators. Prime and restart
>>> generators, charge batteries for 30 min and begin rebooting equipment.
>>> 2. Contracted with local vendor to do something similar for us
>>>
>>> Since they are both sub-optimal solutions, we are working both
>>> simultaneously to improve of chances of success. Best case is early
>>> afternoon restoration.
>>>
>>> Building management, who is responsible for providing fuel to us
>>>continues
>>> to pump out the flooded basement in order to expose pumps, assess
>>>situation
>>> with intent of permanent repair/replacement of the main pump
>>>
>>>
>>>
>>>
>>> Sent from my mobile device.
>>>
>>> On Oct 31, 2012, at 8:34 PM, "Hiers, David (DS)" <David.Hiers at adp.com>
>>> wrote:
>>>
>>> How are things going out there? We're seeing pretty broad
>>> failures in both termination and origination with Verizon. If
>>> we route calls out Level3, we can at least call out.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> David
>>>
>>>
>>> ________________________________________
>>>
>>> From: voiceops-bounces at voiceops.org
>>> [voiceops-bounces at voiceops.org] On Behalf Of Jared Geiger
>>> [jared at compuwizz.net]
>>>
>>> Sent: Tuesday, October 30, 2012 2:35 PM
>>>
>>> To: voiceops at voiceops.org
>>>
>>> Subject: Re: [VoiceOps] Sandy Watch
>>>
>>>
>>> Looks like TATA NYC1 and 2 came back up. Telia went down.
>>> Belgacom is
>>>
>>> back up but taking the Washington -> Paris route instead of the
>>> NYC ->
>>>
>>> London route.
>>>
>>>
>>> Novatel in NYC went completely down. IPs aren't even being
>>> advertised in BGP.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 11:51 AM, Jared Geiger
>>> <jared at compuwizz.net> wrote:
>>>
>>> We're connected to 2 others. The one in NJ seems up
>>> and we're
>>>
>>> connected to London as well. Looks like the path
>>> across the Atlantic
>>>
>>> is fine.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 9:41 AM, Colin Brown
>>> <zavoid at gmail.com> wrote:
>>>
>>> If your connected with Tata hopefully
>>> your connected to at least one of the 13
>>> other gsxes they have.
>>>
>>>
>>> On Oct 30, 2012, at 9:34 AM, Jared
>>> Geiger <jared at compuwizz.net> wrote:
>>>
>>>
>>> 1) Looks like TATA NYC1 and
>>> NYC2 GSXes are down.
>>>
>>> 2) I got an email saying
>>> Telia only had 6 hours of
>>> power left at 6 AM
>>>
>>> Eastern, so they may go dark
>>> as well at noon.
>>>
>>> 3) Our IP routes to Belgacom
>>> are broken and die in NYC on
>>> nlayer,
>>>
>>> Cogent, and Abovenet.
>>>
>>> 4) Looks like Hypercube in
>>> NYC is down.
>>>
>>> 5) Reliance is going down in
>>> NYC
>>>
>>>
>>> It seems all of our other
>>> trunks are up.
>>>
>>>
>>> On Tue, Oct 30, 2012 at 2:37
>>> AM, Hiers, David (DS)
>>> <David.Hiers at adp.com> wrote:
>>>
>>> Nothing too
>>> surprising so
>>> far...
>>>
>>>
>>> There seems to
>>> be pretty strong
>>> correlation
>>> between flooded
>>> areas with no
>>> electricity and
>>> unregistered
>>> devices. I
>>> probably don't
>>> need to drop a
>>> 650 into a fish
>>> tank to see if
>>> the issue is
>>> reproducible,
>>> but it is oddly
>>> tempting.
>>>
>>>
>>>
>>> David
>>>
>>>
>>>
>>>
>>> ________________________________________
>>>
>>> From:
>>> voiceops-bounces at voiceops.org
>>> [voiceops-bounces at voiceops.org]
>>> On Behalf Of
>>> Hiers, David
>>> (DS)
>>>
>>> Sent: Monday,
>>> October 29, 2012
>>> 11:58 AM
>>>
>>> To:
>>> voiceops at voiceops.org
>>>
>>> Subject:
>>> [VoiceOps] Sandy
>>> Watch
>>>
>>>
>>> Hi,
>>>
>>>
>>> Is anyone seeing
>>> any storm
>>> related issues
>>> yet? I've got
>>> the expected
>>> drop in calls as
>>> the NE shuts
>>> down, not much
>>> more on the
>>> radar.
>>>
>>>
>>> I'll be
>>> watching/updating
>>> this list (and
>>> outages) tonight
>>> for any
>>> significant
>>> events.
>>>
>>>
>>> Thanks,
>>>
>>>
>>> David
>>>
>>>
>>> PS: Having your
>>> machines prepped
>>> for disaster is
>>> good, but what
>>> about your
>>> people? All
>>> that expensive
>>> gear might not
>>> be not worth
>>> much if you've
>>> got no one to
>>> operate it when
>>> the big one
>>> hits. After
>>> all, even first
>>> responders don't
>>> respond if doing
>>> so would imperil
>>> their
>>> families... are
>>> your people
>>> wired in to
>>> ready.gov?
>>>
>>>
>>>
>>>
>>> ________________________________
>>>
>>>
>>> ________________________________
>>>
>>> This message and
>>> any attachments
>>> are intended
>>> only for the use
>>> of the addressee
>>> and may contain
>>> information that
>>> is privileged
>>> and
>>> confidential. If
>>> the reader of
>>> the message is
>>> not the intended
>>> recipient or an
>>> authorized
>>> representative
>>> of the intended
>>> recipient, you
>>> are hereby
>>> notified that
>>> any
>>> dissemination of
>>> this
>>> communication is
>>> strictly
>>> prohibited. If
>>> you have
>>> received this
>>> communication in
>>> error, please
>>> notify us
>>> immediately by
>>> e-mail and
>>> delete the
>>> message and any
>>> attachments from
>>> your system.
>>>
>>>
>>>
>>>
>>> This message and
>>> any attachments
>>> are intended
>>> only for the use
>>> of the addressee
>>> and may contain
>>> information that
>>> is privileged
>>> and
>>> confidential. If
>>> the reader of
>>> the message is
>>> not the intended
>>> recipient or an
>>> authorized
>>> representative
>>> of the intended
>>> recipient, you
>>> are hereby
>>> notified that
>>> any
>>> dissemination of
>>> this
>>> communication is
>>> strictly
>>> prohibited. If
>>> you have
>>> received this
>>> communication in
>>> error, please
>>> notify us
>>> immediately by
>>> e-mail and
>>> delete the
>>> message and any
>>> attachments from
>>> your system.
>>>
>>>
>>>
>>>_______________________________________________
>>>
>>> VoiceOps mailing
>>> list
>>>
>>> VoiceOps at voiceops.org
>>>
>>>
>>>https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>>
>>>https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>>>
>>> This message and any attachments are intended only for the use
>>> of the addressee and may contain information that is privileged
>>> and confidential. If the reader of the message is not the
>>> intended recipient or an authorized representative of the
>>> intended recipient, you are hereby notified that any
>>> dissemination of this communication is strictly prohibited. If
>>> you have received this communication in error, please notify us
>>> immediately by e-mail and delete the message and any attachments
>>> from your system.
>>>
>>>
>>> _______________________________________________
>>>
>>> VoiceOps mailing list
>>>
>>> VoiceOps at voiceops.org
>>>
>>> https://puck.nether.net/mailman/listinfo/voiceops
>>>
>>>
>
>
>_______________________________________________
>VoiceOps mailing list
>VoiceOps at voiceops.org
>https://puck.nether.net/mailman/listinfo/voiceops
More information about the VoiceOps
mailing list