[c-nsp] Data Center switch replacement

[chris at lavin-llc.com]

Sometimes the simplest things can grow into the biggest projects.

For those of you that provide or support large data centers, I have a
situation that I'd like to pick your brains about.

We have some older switch gear that needs replaced because line cards are
beyond MTBF and several have failed recently. Because of these outages the
need to upgrade isn't a hard sell. We already have the new gear on hand.

The challenge is in the migration to these new switches. The four switches
are deployed as pairs. One switch of a pair is the primary link for
servers and the other switch is the secondary link for servers.

I have two options I'm kicking around. The first and most disruptive is to
power down a switch, unplug the cables, remove the switch from the rack,
install the new switch and plug the server connections in. My clients are
pushing hard against this option because of the downtime involved. The
second option is to stand up the new switches in other places within the
data center and run patch panel to patch panel connections. This would
provide for much less down time (est. 30 seconds per server) but would
incur a pretty hefty cabling cost. Of course, money is an issue.

In the end I realize someone will have to evaluate the trade off and tell
me to either execute and take the customers down or pay the piper, buy the
additional cabling and minimize the impact to the customers.

I'm curious to know how other folks have approached this situation before?
How do you move 600+ server connections from an old switching environment
to a new switching environment? Since we're experiencing outages, time is
a piece of the equation.

Thanks,
-chris