[c-nsp] Multiple power supply failures. Advise needed

[Randy McAnally]

The double sided sticky tape method mentioned should work, assuming there are
still particles in the air.  There is a possibility it came in a short burst
while they were doing work in or near the air plenum.

I would definitely look for particles inside the servers, if you have any you
can take offline temporarily.

--
Randy
www.FastServ.com

---------- Original Message -----------
From: Michael Ulitskiy <mulitskiy at acedsl.com>
To: "Randy McAnally" <rsm at fast-serv.com>
Cc: cisco-nsp at puck.nether.net
Sent: Wed, 2 Sep 2009 12:47:57 -0400
Subject: Re: [c-nsp] Multiple power supply failures. Advise needed

> You made me doubt myself and I went ahead and checked the air flow 
> direction in a couple of servers I have in the office handy. I was 
> under impression that all servers power supplies are sucking air,
>  but I was wrong. Apparently we have servers working both ways. And 
> here I find another clue. All server PSUs that failed were sucking 
> air. Does it mean that I'm now to expect system board problems in 
> those servers with PSUs sucking air from inside the chassis? Oh, 
> come on... Also I'm not sure about air flow direction in cisco power 
> supplies. I'll check it out on my next visit to colo.
> 
> Michael
> 
> On Wednesday 02 September 2009 11:49:59 am Randy McAnally wrote:
> > You mentioned a couple servers failed -- most if not all servers power
> > supplies blow outward, sucking air from inside the chassis.  Many Cisco
> > devices work this way also.  I have rarely seen a power supply that sucks air
> > in directly from the outside of the chassis.
> > 
> > Given the above, much of the dust would settle on the system board.  The
> > extremely high tolerances of a typical system board far outweigh those of a
> > power supply, thus I would expect system instability before the power supply
> > failed.
> > 
> > My bet is still on power spikes/dips.
> > 
> > --
> > Randy
> > www.FastServ.com
> > 
> > ---------- Original Message -----------
> > From: Michael Ulitskiy <mulitskiy at acedsl.com>
> > To: "Randy McAnally" <rsm at fast-serv.com>
> > Cc: cisco-nsp at puck.nether.net
> > Sent: Wed, 2 Sep 2009 11:08:24 -0400
> > Subject: Re: [c-nsp] Multiple power supply failures. Advise needed
> > 
> > > What about the fact that most (if not all) power supplies have 
> > > independent sucking fan and that power supply air flow is separate 
> > > from the system board flow.
> > > 
> > > Plus all system board I saw are covered with some insulating 
> > > coating. I've never pulled apart a modern power supply. I'd expect 
> > > them to have something like that too, but who knows? Plus since PSU 
> > > is the only part that's dealing with high voltages I expect it to be 
> > > more sensitive to momentary shorts. Am I wrong?
> > > 
> > > I'm expecting report for provider ordered unintrusive power 
> > > monitoring. I'm almost positive they won't find anything, though. 
> > > I'm still looking for advice on independent power analysis source in 
> > > New York, NY if anyone has this kind of experience. Thanks,
> > > 
> > > Michael
> > > 
> > > On Wednesday 02 September 2009 10:29:07 am Randy McAnally wrote:
> > > > Plain old dust wouldn't be so picky...it has to be ingested past the
system
> > > > board before it hits the power supply in most cases.  System boards
are WAY
> > > > more sensitive to this kind of thing.
> > > > 
> > > > The fact you have ONLY PSU's failing still makes me think you have power
> > issues.
> > > > 
> > > > --
> > > > Randy
> > > > www.FastServ.com
> > > > 
> > > > ---------- Original Message -----------
> > > > From: Michael Ulitskiy <mulitskiy at acedsl.com>
> > > > To: "Randy McAnally" <rsm at fast-serv.com>
> > > > Cc: "Scott Granados" <gsgranados at comcast.net>, "Seth Mattinen"
> > > > <sethm at rollernet.us>, cisco-nsp at puck.nether.net
> > > > Sent: Tue, 1 Sep 2009 23:21:23 -0400
> > > > Subject: Re: [c-nsp] Multiple power supply failures. Advise needed
> > > > 
> > > > > This is my main suspect now. They are doing work in the facility. 
> > > > > Not heavy construction, but they do install cages and cabinets for 
> > > > > new tenants and they're definitely using tools that  produce metal dust.
> > > > > My theory is that because of we've been the 1st customer who moved 
> > > > > into that facility we've been collecting that metal dust for longest 
> > > > > and so we're having a lot of problems with our equipment. To my 
> > > > > knowledge none of our neighbors are having the same problem, but 
> > > > > none of them have been in the place long enough. So the question 
> > > > > remains: is there any way to fight it/protect from it except from 
> > > > > going through the huge-huge-huge headache of undertaking another move?
> > > > > 
> > > > > Michael
> > > > > 
> > > > > On Tuesday 01 September 2009 08:48:38 pm Randy McAnally wrote:
> > > > > > He mentioned he was one of the first customers in the colo so
> > > > > > this might be a possibility
> > > > > > 
> > > > > > --
> > > > > > Randy
> > > > > > 
> > > > > > ---------- Original Message -----------
> > > > > > From: "Scott Granados" <gsgranados at comcast.net>
> > > > > > To: "Seth Mattinen" <sethm at rollernet.us>, "Michael Ulitskiy"
> > > > > > <mulitskiy at acedsl.com>
> > > > > > Cc: cisco-nsp at puck.nether.net
> > > > > > Sent: Tue, 1 Sep 2009 17:35:34 -0700
> > > > > > Subject: Re: [c-nsp] Multiple power supply failures. Advise needed
> > > > > > 
> > > > > > > Also make sure that the provider isn't doing work in the facility. 
> > > > > > >  I'll never forget going to an L3 datacenter and arriving to find 
> > > > > > > workmen in the overhead grinding away and dropping dust and who 
> > > > > > > knows what else in to all the racks below including a rack of Netra 
> > > > > > > T1's that promptly sucked in the dust and kicked out power 
> > > > > > > supplies.;)  It was definitely metal shavings because they were 
> > > > > > > using a grinding type tool up in  the over head frames.
> > > > > > > 
> > > > > > 
> > > > > >
> > > > ------- End of Original Message -------
> > > > 
> > > >
> > ------- End of Original Message -------
> > 
> >
------- End of Original Message -------