[Outages-discussion] Securus Responds to Unprecedented Weather Related Service Outage at Data Center in Dallas

[Warren Kumari]

On Tue, Jun 2, 2015 at 6:31 PM, Frank Bulk <frnkblk at iname.com> wrote:
> http://www.prnewswire.com/news-releases/securus-responds-to-unprecedented-weather-related-service-outage-at-data-center-in-dallas-texas-300092570.html
>
> Normally, this cooling would have been restored within minutes – but the
> flooding caused a longer response time – Securus' equipment overheated (>
> 120 degrees F), and we had to take the extraordinary measure of shutting
> down the data centers to avoid a catastrophic situation that would extend
> the service outage.
>
> Any comments on this?  Doesn’t most network equipment shutdown by itself
> before it overheats?

Some does, much doesn't. Most gear will send alarms at over
temperature conditions, but a large amount of it won't shut down.
Actually, there is a lot of gear that you cannot really "shut down"
remotely - you can reboot it, but not necessarily power off. There is
also lots of gear where "shut down" still leaves the power supplies on
and running - in which case the power supplies still overheat
eventually.  Even if you have stuff on remote control PDUs, turning
devices off and on again often makes them die, especially if they are
older. There are all sorts of reasons for this, including inrush
current, and thermal cycling (if a line-card has been running for
years at 40C, quickly heats because the AC failed and then quickly
cools down  because you yanked power you stress all sorts of
components, including solder and connectors).

W

>
> Frank
>
>
> _______________________________________________
> Outages-discussion mailing list
> Outages-discussion at outages.org
> https://puck.nether.net/mailman/listinfo/outages-discussion
>



-- 
I don't think the execution is relevant when it was obviously a bad
idea in the first place.
This is like putting rabid weasels in your pants, and later expressing
regret at having chosen those particular rabid weasels and that pair
of pants.
   ---maf