[c-nsp] Top of Rack Switching

[Scott Altman]

Looking for some input from the collective on building out a data
center using a "Top of Rack Switching" model compared to the more
traditional centralized switching design.

The ToRS model would consist of a pair of 4948's in every server rack
providing 5-8 VLANs to the servers in the rack (Productionx2, Backup,
Management, etc).  The 4948's would have a 2x1Gb EthC between them and
they would each have a 2x1Gb EthC up to a 6509 for aggregation;
basically creating a U-shaped topology from a spanning-tree
perspective.  Due to other design requirements, my L3 boundary can't
come any closer.

For quantities we are talking about 60 pair of 4948's per IDF, so net
is 120 4948's, 2 6509's.  I chose to use the SFP version of the 4948's
vs. the 10Gb based on the availability of upstream 10Gb ports in the
6509 (4 10Gb, vs 48SFP) and my bandwidth requirements don't warrant
greater than 4Gb out of each server rack.

What I would like to hear about are:
- Issues with managing the complexity of the environment (managing 122
switches in an IDF is not common)
- General design concerns
- Suggestions for improvement
- Experience with similar configurations

Replies welcome on or off list...

Thanks!

- Scott