[j-nsp] EX 8200 deployment

[Chris Evans]

Richard,

I agree with you with the EX platform.. It's really buggy. I personally
think that Juniper is moving too slow bringing new features and bug fixes to
the platform.. We're deploying the new Cisco Nexus platforms in our data
centers at this point. Cisco is moving at light speed with these platforms,
while Juniper is crawling to bring their aging boxes into the lime light
left by the Cisco Cat6500 days.

On another note. I know you're upset about the limit on the routing that the
EX series can do, but a better question is why are you using this box for
that high end routing solution? In my opinion, the EX's 8200's are a switch
built for the data center, they shouldn't require much more than a default
route and/or a few hundred routes to your core.

Discussion?

On Sat, Mar 20, 2010 at 11:03 AM, Richard A Steenbergen <ras at e-gerbil.net>wrote:

> On Sat, Mar 20, 2010 at 06:50:58PM +0500, Fahad Khan wrote:
> > Hi Folks,
> >
> > Please share your experiences regarding the deployment of EX 8200, I need
> to
> > deploy two chassis in VRRP. Please let shed some light on the following
> > point
> >
> > - Any trick in power/power requirements??
> > - stability
> > - best design( like Virtual routers are needed or not)
> > - possible caveats
> > - Best junos version
> >
> > Add any trick or issue which you have found out?
>
> We just deployed our first EX8208 a few days ago, running 10.1R1.
> Gotchas so far:
>
> * Obviously this is a very different architecture from Juniper's normal
> boxes, so be prepared for vlan space being shared across the entire box,
> not a per-interface basis.
>
> * In a move straight out of Foundry's playbook of how to fail at making
> a useable product, EX has no packet counters (cli or snmp) available for
> L3 vlan interfaces. It DOES have working counters if you do traditional
> Juniper subinterface style vlans (interface blah, vlan-tagging, unit 123
> vlan-id 123), but it does NOT work if you have to do RVI style (vlan
> blah l3-interface vlan.123 and then put vlan blah in an ethernet
> switching interface). Subinterface style is my preference anyways, so as
> long as you only ever use vlans on point-to-point links this isn't a
> problem, but the instant you need to put a VLAN on more than one port
> you no longer get packet counters.
>
> * Related to the issue above, you can't mix "subinterface style" and
> "RVI style" vlans on the same trunk port. The instant you need to do
> anything more than classic subinterface style vlans, you have to convert
> everything on the trunk to vlan/rvi style. For example, where I might
> otherwise be able to get away with doing interface xe-1/0/0 unit 123
> vlan-id 123 family inet blah, if I want to trunk a layer 2 vlan on that
> same interface I now have to convert unit 123 to RVI style. One possible
> workaround I have yet to test is doing a CCC instead of a vlan, to keep
> the subinterface style. This would only work with 2-port member vlans
> though, and I have yet to test the implications for mixing tagged and
> untagged ports on EX, so this may not actually work for anyone at all.
>
> * There is a hardcoded default maximum data memory per process of 512MB
> on the EX8200 series, which isn't enough memory for rpd to run any kind
> of complex BGP configuration. Juniper says they tested it to 3.2 million
> paths and it only used ~400MB, but I guess they found some artificially
> low test scenario (I still wonder how they did this, even with no
> communities and no as-path attributes it's only a ~10% memory
> difference), and they didn't bother to ask the rpd developers how much
> memory usage to expect, because real world usage is obviously FAR
> higher. In our testing rpd coredumped at about 900k BGP paths, which is
> probably a bad thing if you actually want to route on these boxes.
> Juniper is working on this issue, but as a temporary workaround, edit
> your /boot/loader.conf file as root, replace the kern.maxdsiz=512M with
> something more sensible like 1536M, and reboot. This will of course be
> blown away every time you upgrade JUNOS, so it helps to have dual REs so
> you can pre-stage things. :)
>
> * Firewall filters are still a bit of a mess. You can't count or log
> anything, you can't use policers on either control plane or egress
> filters (heck you can't even commit a firewall filter with a policer in
> it if applied as an output filter), you can't match frags, etc, etc.
> Basically if you're coming from another Juniper router, be prepared to
> completely redesign all of your firewall filters across the board. For
> example, you can't currently do a match like "from port 179" on EX, you
> have to do "from source-port 179" and "from destination-port 179", which
> means splitting what was previously a single term into two terms. This
> is expected to get better with future sw, but my feeling is probably in
> small increments over the next year+ or so.
>
> * I don't know who thought 2GB of storage on an RE was sufficient, but
> it isn't. The best idea I've come up with so far is to grab some small
> USB flash devices like http://www.geckoandfly.com/tag/small-usb/ and
> deploy them on every RE so you have a little bit of working space.
>
> Other than that we haven't found any fundamental flaws in the box yet
> (though that may change by the time MPLS features get implemented :P).
> Plenty of bugs to be sure, DOM isn't working right on any of our
> interfaces, pfe statistics don't work right, monitor interface on vlans
> isn't displaying correctly, prior to 10.1 the FPCs crashed if you tried
> to speak BGP flowspec to the box, etc, but we have cases open on all of
> the above. IMHO it definitely has the potential to be a very good box in
> the long term, but whoever didn't think to put vlan counters into the
> hardware really screwed the pooch something fierce. :)
>
> --
> Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
> GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
> _______________________________________________
> juniper-nsp mailing list juniper-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp
>