[c-nsp] 8 Racks of Servers and Growing; switch/layout recommendations

[Buhrmaster, Gary]

> What's a good method for growing here?  Do people like top-of-rack for
> situations where we have a cage (all the racks are side-by-side), or do you
> prefer end-of-row?  

Top-of-rack vs. end-of-row approaches a religious debate.  There
are arguments on both sides, and the believers will attempt to
convince you of the value of their faith.  The answer, unfortunately,
will always be "it depends".  An important question will be
where are you heading (quo vadis?) with your deployment.

*Some* considerations include:

   Cabling costs (tends to favor tor)
   Virtualization (layer-2 requirements, tends to favor eor)
   bisection bandwidth (tends to favor eor)
   Racks (tends to favor tor since 1U is "free")
   Redundancy (arguments on both sides, power, supervisor, links...)
   10Gb/sec server plans (and how that effects the other issues)
   Flexibility (tends to favor eor - any service at any port)
   Unified fabric plans (currently in the Nexus line)
   Cost (will depend on the eor/tor switch chosen, of course)
   
It should be noted, as you mention, that a "data center" top
of rack switch is often different than a wiring closet switch.
The 4948, for example, is really a lot different than a 2960G.
Note that the Nexus line may offer some options for both tor
and eor.

"We" tend(*) to use tor for the lower bandwidth management ports on
our servers, and eor for the higher bandwidth data centric ports
(note that our environment is data intensive science, which is
going to be unlike many other deployments).

Gary

(*) There are exceptions to every rule.