[j-nsp] EX 8200 deployment

[Richard A Steenbergen]

On Sat, Mar 20, 2010 at 06:50:58PM +0500, Fahad Khan wrote:
> Hi Folks,
> 
> Please share your experiences regarding the deployment of EX 8200, I need to
> deploy two chassis in VRRP. Please let shed some light on the following
> point
> 
> - Any trick in power/power requirements??
> - stability
> - best design( like Virtual routers are needed or not)
> - possible caveats
> - Best junos version
> 
> Add any trick or issue which you have found out?

We just deployed our first EX8208 a few days ago, running 10.1R1. 
Gotchas so far:

* Obviously this is a very different architecture from Juniper's normal 
boxes, so be prepared for vlan space being shared across the entire box, 
not a per-interface basis.

* In a move straight out of Foundry's playbook of how to fail at making
a useable product, EX has no packet counters (cli or snmp) available for
L3 vlan interfaces. It DOES have working counters if you do traditional 
Juniper subinterface style vlans (interface blah, vlan-tagging, unit 123 
vlan-id 123), but it does NOT work if you have to do RVI style (vlan 
blah l3-interface vlan.123 and then put vlan blah in an ethernet 
switching interface). Subinterface style is my preference anyways, so as 
long as you only ever use vlans on point-to-point links this isn't a 
problem, but the instant you need to put a VLAN on more than one port 
you no longer get packet counters.

* Related to the issue above, you can't mix "subinterface style" and 
"RVI style" vlans on the same trunk port. The instant you need to do 
anything more than classic subinterface style vlans, you have to convert 
everything on the trunk to vlan/rvi style. For example, where I might 
otherwise be able to get away with doing interface xe-1/0/0 unit 123 
vlan-id 123 family inet blah, if I want to trunk a layer 2 vlan on that 
same interface I now have to convert unit 123 to RVI style. One possible 
workaround I have yet to test is doing a CCC instead of a vlan, to keep 
the subinterface style. This would only work with 2-port member vlans 
though, and I have yet to test the implications for mixing tagged and 
untagged ports on EX, so this may not actually work for anyone at all. 

* There is a hardcoded default maximum data memory per process of 512MB
on the EX8200 series, which isn't enough memory for rpd to run any kind
of complex BGP configuration. Juniper says they tested it to 3.2 million
paths and it only used ~400MB, but I guess they found some artificially
low test scenario (I still wonder how they did this, even with no
communities and no as-path attributes it's only a ~10% memory
difference), and they didn't bother to ask the rpd developers how much
memory usage to expect, because real world usage is obviously FAR
higher. In our testing rpd coredumped at about 900k BGP paths, which is
probably a bad thing if you actually want to route on these boxes.
Juniper is working on this issue, but as a temporary workaround, edit
your /boot/loader.conf file as root, replace the kern.maxdsiz=512M with
something more sensible like 1536M, and reboot. This will of course be
blown away every time you upgrade JUNOS, so it helps to have dual REs so
you can pre-stage things. :)

* Firewall filters are still a bit of a mess. You can't count or log
anything, you can't use policers on either control plane or egress
filters (heck you can't even commit a firewall filter with a policer in
it if applied as an output filter), you can't match frags, etc, etc.
Basically if you're coming from another Juniper router, be prepared to
completely redesign all of your firewall filters across the board. For
example, you can't currently do a match like "from port 179" on EX, you
have to do "from source-port 179" and "from destination-port 179", which
means splitting what was previously a single term into two terms. This
is expected to get better with future sw, but my feeling is probably in
small increments over the next year+ or so.

* I don't know who thought 2GB of storage on an RE was sufficient, but 
it isn't. The best idea I've come up with so far is to grab some small 
USB flash devices like http://www.geckoandfly.com/tag/small-usb/ and 
deploy them on every RE so you have a little bit of working space.

Other than that we haven't found any fundamental flaws in the box yet
(though that may change by the time MPLS features get implemented :P). 
Plenty of bugs to be sure, DOM isn't working right on any of our
interfaces, pfe statistics don't work right, monitor interface on vlans
isn't displaying correctly, prior to 10.1 the FPCs crashed if you tried
to speak BGP flowspec to the box, etc, but we have cases open on all of
the above. IMHO it definitely has the potential to be a very good box in
the long term, but whoever didn't think to put vlan counters into the
hardware really screwed the pooch something fierce. :)

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)