[c-nsp] Campus - Best Practices

[Phil Mayers]

Paul Stewart wrote:
> We are currently bidding on a campus deployment for a local educational
> facility.  The requirement involves approximately 1000 ethernet drops to
> student residences.

We do exactly the same in-house for our own users (~3000 bed spaces). 
The network is currently cisco 1900s (long story) with 3550s as site 
routers running EMI and VRF-Lite. VMPS is used to put mac-based VLANs in 
place, with 4 VLANS per site. Each VLAN is bound to a VRF and they 
cannot talk to each other or the world except via the resnet firewall:

SWITCHMAN - obvious, doesn't go to user-facing ports, accessible only 
via NOC

PROD - normal default route with public IPs

PREREG - unknown MACs go here and get a web-based registration page via 
HTTP intercept on the default route

BANNED - virus-infected ones go here and get a web-based explanation 
page via HTTP intercept on the default route

I cannot recommend the VRF-Lite and VMPS (or any other vlan assignment) 
solution highly enough. It literally transformed they way we do things 
on that network, so much so that we are implementing it campus-wide.

Things to watch out for - NICs with dodgy drivers that emit weird MAC 
addresses (e.g. 0c00.0000.0000) on bootup confusing VMPS, minihubs 
(they're forbidden), internet connection sharing and wireless. Airports 
are a particular offender with their NATing behaviour.

> 
> Cisco Clean Access (or NAC) is a requirement and we are considering

I'm surprised - many/most student residences have a diverse enough 
client base that putting the NAC client on all of them is tricky, which 
is why we stuck with MAC-based auth.

> deploying Cisco 3560-48TS switches throughout the campus linked on GigE
> fiber between them.  Our original plan was for something along the lines
> of 6509's but because of the way the ethernet drops are located, we need
> to put smaller switches in more locations than a centralized deployment.

Same here

> 
> A Cisco 7206VXR would then provide DHCP services with public IP
> addresses (a requirement) to each of the desktops via the switches.

It has been our (and others) experience that a large amount of traffic 
coming out of residence networks is p2p. So something with good 
bandwidth management capabilities will be useful (ideally an aggregate 
plus microflow policers). Don't personally know enough about the 7200 
but I believe it does that well.

> 
> What is best practice in a setup like this and/or should we look at areasonably 
> completely different setup?  I presume NAC can communicate via SNMP with
> almost any switch that supports VLAN's?

As I understand it, NAC has 2 operational modes:

  1. to the edge - each switchport is an 802.1x port, the NAC client 
lives on the PC and appends NAC info to the 802.1x EAP exchange (which 
the switch does not care about) and the radius server replies with 
accept/reject and/or vlan info

  2. to the router - each layer2 is unprotected, and the NAC client 
lives on the PC and communicates via EAP-over-UDP with the router, which 
passes the EAP to the radius server.

So the former will work on any switch with 802.1x (which is all of them, 
these days). The latter will work on any switch at all but offers much 
looser protection. Not sure where SNMP comes into it?

I could be wrong - not a NAC expert.

> 
> Because this is really one large LAN, what kind of security can be

Do not under any circumstances deploy a large flat network. Seriously.

As well as giving you a huge broadcast domain, it compromises the 
utility of uRPF or ACL-based source spoof prevention (which is a MUST) 
since the wider the netmask, the more IPs a hacked client can spoof and 
many many DDoS clients will spoof within their enclosing /16 these days.

If you can keep each layer2 vlan to a single switch / stack / cluster 
you can also leave spanning tree behind (which though you may not need 
it now, designing it in as early as possible will help you/the customer 
implement resilience later). And it greatly improves your ability to 
resolve faults if the fault domain is smaller of course.

> provided to stop "snooping" of other traffic, man-in-middle attacks etc.

Within a layer2, "private ports" (ports communicate only with the uplink 
and not each other) and on switches that have the grunt per-port ACLs 
are ways to do that. We can't currently since the 1900s are ancient, but 
we *do* have the network sliced down into /24s or /23s with ACLs on the 
layer3 interfaces.

DHCP snooping and arp inspection are neat but did not fly for us since 
the number and cost of special cases (CCTV & swipe access controllers, 
EFT machines, etc.) outweighed the benefits.

> etc?  Any pointers from people who have done lots of these would be very
> appreciated.  Also, traffic will be approximately 200 Mb/s throughout
> the entire network at peak time....

Without bandwidth management it will climb as high as the network will 
support, and the most aggressively-transmitting machines will contend 
the links the bulk of the time. Plan to implement bandwidth management 
from day one - we went for absolute quotas, set at a pretty generous 
level - 5Gb in a rolling 24 hour period. "I have no interest in whether 
you'd downloading movies or linux DVD ISOs - you're still contending the 
network for the other students". The levels also compare quite 
favourably to those that residential broadband providers here in the UK 
are starting to implement, so are a familiar concept to students. Maybe 
not so where you are. Other unis (in the USA for example) have used 
various BW management appliances to reasonable effect. I'm not a fan 
personally.

Some more things to think about:

  1. When a machine is banned, it is helpful to leave them the ability 
to reach patching services like windows update, security.symantec.com 
(online ActiveX based AV scanner - v. useful) and so on. Doing this by 
netblock is painful - some sort of HTTP/HTTPS proxy or such can be handy 
(don't know if NAC has all this built in)

  2. Bandwidth quotas are a very very effective method of keeping 
utilisation sensible. So getting something that can give you utilisation 
info per-IP can be useful (that could be switch port stats, netflow, a 
port mirror and "ipfm" running on Linux, or whatever). The port mirror 
is probably the most useful in terms of diagnostics.

  3. For pitys sake, block outbound 135-139, 445 and inbound default 
deny. Please, for the sake of the internet! Student residence networks 
are festering pools of botnets. Persuading the institute to buy a site 
license for a good AV scanner (if they don't have one) and mandating the 
students install it would not go amiss...

  4. Getting offtopic - plan for cabling faults and socket breakages if 
you or they haven't already.

That about covers the basics! :o)