[c-nsp] Data Center switch replacement

[chris at lavin-llc.com]

> Hi,
>
> On Thu, Dec 17, 2009 at 08:42:49PM -0500, chris at lavin-llc.com wrote:
>> adherence to our standard has led to two problems. First, several
>> servers
>> didn't get cabled with two connections. Second, the folks who manage the
>> servers have challenges with the NIC configurations. So while we expect
>> many of the servers can sustain the loss of one NIC, we have several
>> that
>> we know and many that we may not know, will lose network connectivity as
>> we flip the connection to the new switch.
>
> Now that's a good opportunity to clean up broken server configurations and
> connections.
>
> "If it's meant to be redundant, and it isn't, then it's not the network's
> fault if it breaks.  Go and fix it!"
>
> gert
> --

Thanks to everyone that responded. I appreciate learning how much several
of us have in common. I especially liked those that shared stories with me
about similar challenges of server and NIC settings for what should be a
redundant design with Primary/Secondary configurations.

I'll update my recommended options to include a third scenario.
1. Complete blackout to power down each switch and replace it with the new
one.
2. Eat the cabling/rack/etc. cost and stand up the new switches and
migrate the connections in one night (performing some due diligence ahead
of time) and hoping all servers are properly configured for a
Primary/Secondary network connection.
3. Eat the cabling/rack/etc. cost and stand up the new switches and
migrate slowly over a period of several maintenance windows while hoping
we don't have any more line card failures during the extended migration
period.

Much appreciated,
-chris