[c-nsp] Large networks

[Steve Bertrand]

Shaun R. wrote:

Your message is intoxicable ;)

> I worked for a company in the past that had a very large flat network.  
> The network consisted of two /20's (255.255.240.0) that were configured
> on a 7206 npe-300 router that connected to a bunch of catalyst 2924
> switches (the old school ones).

I still use them, and they work great!

> Everything was on vlan1.  The company
> was a small hosting company that provided mainly dedicated servers. 

> This company was constantly having problems with what i called broadcast
> attacks.  The network graphs would show traffic on all interfaces spike
> and normally the 100mbit uplink between the switches would saturate and
> the network would die.  From that experience i took my time to design
> and deploy my network to be as correct as possible. 

Out of curiosity, did your experience find that the issues were related
to actual broadcast problems?

> I put each customer
> on there own vlan with there own subnet carved out.  My 3750 stack is my
> access/core and i have 7206-VXR-npe-g2's for borders (bgp/ospf).  Every
> edge switch is uplinked twice with gigE (2gbit of bw) and customers are
> uplinked normally at 100mbit.  For years this was fine and worked great
> but when deploying our own servers i always found myself kicking out a
> new vlan and subnet.  I wasnt sure if it was needed being that it was
> our own servers (our own servers meaning that we managed them, customer
> do not have admin/root access).

Again, out of pure curiosity, why did you do it this way? Do you manage
ACLs per server? We're a small op that provides access, hosting and
colo, and I'm wondering why you'd adopt this strategy.

> Then came virtual server hosting.  With VPS Hosting we have one physical
> server (a host) that we carve out a /26 for and assign it to it's own
> vlan. We've done this for a few years now and it's worked fine but it's
> also kind of caused problems.  One problem is that some hosts needs more
> ips than other hosts.  We end up with some hosts having 20 ips free in
> there subnet while other hosts have none and need another allocation
> assigned to them. Also, we cannot move a customer from one host to
> another with out making the customer change ip address's.  

I'm but a rookie, but it appears as though some design research
regarding IP assignment strategies may be beneficial.

> For a while
> now i've been wanting to just combine all the VPS hosts into one vlan
> and carve them out /24's as needed. Then each host could just get a ip
> from that pool and when that pool started to become depleated i could
> assign another /24.  

Let the big boys criticize me here... what I've done is push our
'hosting' arm to the outside of the edge of our network. The
'hosting/colo' acts as a client premise. Even though it resides within
our primary PoP, it connects to the network the same way that a client
aggregation router does.

I use iBGP from the 'hosting' routers to the edge routers in order to
provide 'in-house' redundancy.

Then, I allow the 'hosting' routers to advertise whatever IP blocks that
they need. I have certain (relatively) large prefixes dedicated to
hosting that are reserved, but can be (and sometimes are) re-purposed in
a heartbeat because of the dynamic setup.

In your statement above, to "assign another /24" would allow you to
re-purpose prefixes for the pool, and use them on a slice-by-slice basis.

Either way, the comment that I'm currently quoting appears more sound
than anything else so far.

> Another problem that company i worked for had was that they where
> calculating bandwidth usage off the 2924 network interfaces.  The
> problem with this we later found was that ARP/Broadcast traffic ended up
> being a huge amount added to there bill at the end of the month.  I want
> to say that each customer had around 4-6GB of transfer tacked onto there
> bandwidth usage.

...relative. I'd like to know what you used to do your billing, and who
authorized the billing. With that kind of potential over-billing, a job
at your employer could mean a massive pay raise :)

> So what i'm really asking is...

>    1. When should i really cut out a new vlan for a server or group of
> servers for my own use (meaning the customer doesnt have admin
> privileges to the machine)?

Whenever you deem it necessary.

>    2. Was the problem with the large network that they didnt cut the /20
> into smaller subnets or was the problem that they didnt cut them into
> smaller subnets and put them into there own vlans?

It's a matter of implementation. Whether you look at it as a subnet or a
vlan, it doesn't matter.

When I first got our /21 from ARIN, I had to do some serious reading. My
primary objective was how to allow the migration of clients without
having to make them renumber. (During which time, I renumbered our
entire network twice, once from MCI IP's, and from our then 'new'
upstream IP's).

After that, I slowly started to learn (via experience and MUCH reading)
that renumbering IP's wasn't the only 'renumbering'. We also want to
have no client impact with prefix lengths (subnet mask) and default
gateway addresses.

Only experience will tell you what went wrong, and where, given much
feedback from diagnostics and other troubleshooting. (I've found that
seeing this on numerous 'small' networks, it comes naturally. When
dealing with a network larger than what I'm used to, the experience in
troubleshooting/documenting paves the way).

>    3. Say i combine all the VPS Hosts, 1000 Virtual servers in 1 vlan,
> with say 15 /24's... Is this ok? 

...you stated above that you suspected broadcast issues, but now you
want to put... nevermind.

No, it's not ok.

> how is this compared to say having 25
> vlans/subnets with each pysical host in one of them?

Because technically and logically, each VLAN is it's own broadcast
domain. It means that my server can scream to everyone in it's VLAN
(broadcast domain) all it wants, but it can't affect any other host in
any other VLAN.

It can only craft a well-designed broadcast attack on the network that
isn't protecting against it, but only affect a limited number of
connected nodes.

Otherwise, put all hosts in one VLAN, a single problem on a single host
will surely take down your entire network. (with cascading and
escalating detriment).

> Anything else i should be worried about here?

Your sanity.

I'd advise that you hire someone who has had the experience of designing
an infrastructure the size of yours, at least as a consultant.

Steve

ps. feedback welcome on anything that I wrote...
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 3233 bytes
Desc: S/MIME Cryptographic Signature
URL: <https://puck.nether.net/pipermail/cisco-nsp/attachments/20090825/62ead822/attachment-0001.bin>