[j-nsp] EX 8200 deployment

[Cord MacLeod]

On Mar 20, 2010, at 5:31 PM, Chris Evans wrote:

> Richard,
> 
> I agree with you with the EX platform.. It's really buggy. I personally
> think that Juniper is moving too slow bringing new features and bug fixes to
> the platform.. We're deploying the new Cisco Nexus platforms in our data
> centers at this point. Cisco is moving at light speed with these platforms,
> while Juniper is crawling to bring their aging boxes into the lime light
> left by the Cisco Cat6500 days.
> 
> On another note. I know you're upset about the limit on the routing that the
> EX series can do, but a better question is why are you using this box for
> that high end routing solution? In my opinion, the EX's 8200's are a switch
> built for the data center, they shouldn't require much more than a default
> route and/or a few hundred routes to your core.
> 
> Discussion?

Agreed.  High end routing with dense port configurations is called an MX.

> 
> On Sat, Mar 20, 2010 at 11:03 AM, Richard A Steenbergen <ras at e-gerbil.net>wrote:
> 
>> On Sat, Mar 20, 2010 at 06:50:58PM +0500, Fahad Khan wrote:
>>> Hi Folks,
>>> 
>>> Please share your experiences regarding the deployment of EX 8200, I need
>> to
>>> deploy two chassis in VRRP. Please let shed some light on the following
>>> point
>>> 
>>> - Any trick in power/power requirements??
>>> - stability
>>> - best design( like Virtual routers are needed or not)
>>> - possible caveats
>>> - Best junos version
>>> 
>>> Add any trick or issue which you have found out?
>> 
>> We just deployed our first EX8208 a few days ago, running 10.1R1.
>> Gotchas so far:
>> 
>> * Obviously this is a very different architecture from Juniper's normal
>> boxes, so be prepared for vlan space being shared across the entire box,
>> not a per-interface basis.
>> 
>> * In a move straight out of Foundry's playbook of how to fail at making
>> a useable product, EX has no packet counters (cli or snmp) available for
>> L3 vlan interfaces. It DOES have working counters if you do traditional
>> Juniper subinterface style vlans (interface blah, vlan-tagging, unit 123
>> vlan-id 123), but it does NOT work if you have to do RVI style (vlan
>> blah l3-interface vlan.123 and then put vlan blah in an ethernet
>> switching interface). Subinterface style is my preference anyways, so as
>> long as you only ever use vlans on point-to-point links this isn't a
>> problem, but the instant you need to put a VLAN on more than one port
>> you no longer get packet counters.
>> 
>> * Related to the issue above, you can't mix "subinterface style" and
>> "RVI style" vlans on the same trunk port. The instant you need to do
>> anything more than classic subinterface style vlans, you have to convert
>> everything on the trunk to vlan/rvi style. For example, where I might
>> otherwise be able to get away with doing interface xe-1/0/0 unit 123
>> vlan-id 123 family inet blah, if I want to trunk a layer 2 vlan on that
>> same interface I now have to convert unit 123 to RVI style. One possible
>> workaround I have yet to test is doing a CCC instead of a vlan, to keep
>> the subinterface style. This would only work with 2-port member vlans
>> though, and I have yet to test the implications for mixing tagged and
>> untagged ports on EX, so this may not actually work for anyone at all.
>> 
>> * There is a hardcoded default maximum data memory per process of 512MB
>> on the EX8200 series, which isn't enough memory for rpd to run any kind
>> of complex BGP configuration. Juniper says they tested it to 3.2 million
>> paths and it only used ~400MB, but I guess they found some artificially
>> low test scenario (I still wonder how they did this, even with no
>> communities and no as-path attributes it's only a ~10% memory
>> difference), and they didn't bother to ask the rpd developers how much
>> memory usage to expect, because real world usage is obviously FAR
>> higher. In our testing rpd coredumped at about 900k BGP paths, which is
>> probably a bad thing if you actually want to route on these boxes.
>> Juniper is working on this issue, but as a temporary workaround, edit
>> your /boot/loader.conf file as root, replace the kern.maxdsiz=512M with
>> something more sensible like 1536M, and reboot. This will of course be
>> blown away every time you upgrade JUNOS, so it helps to have dual REs so
>> you can pre-stage things. :)
>> 
>> * Firewall filters are still a bit of a mess. You can't count or log
>> anything, you can't use policers on either control plane or egress
>> filters (heck you can't even commit a firewall filter with a policer in
>> it if applied as an output filter), you can't match frags, etc, etc.
>> Basically if you're coming from another Juniper router, be prepared to
>> completely redesign all of your firewall filters across the board. For
>> example, you can't currently do a match like "from port 179" on EX, you
>> have to do "from source-port 179" and "from destination-port 179", which
>> means splitting what was previously a single term into two terms. This
>> is expected to get better with future sw, but my feeling is probably in
>> small increments over the next year+ or so.
>> 
>> * I don't know who thought 2GB of storage on an RE was sufficient, but
>> it isn't. The best idea I've come up with so far is to grab some small
>> USB flash devices like http://www.geckoandfly.com/tag/small-usb/ and
>> deploy them on every RE so you have a little bit of working space.
>> 
>> Other than that we haven't found any fundamental flaws in the box yet
>> (though that may change by the time MPLS features get implemented :P).
>> Plenty of bugs to be sure, DOM isn't working right on any of our
>> interfaces, pfe statistics don't work right, monitor interface on vlans
>> isn't displaying correctly, prior to 10.1 the FPCs crashed if you tried
>> to speak BGP flowspec to the box, etc, but we have cases open on all of
>> the above. IMHO it definitely has the potential to be a very good box in
>> the long term, but whoever didn't think to put vlan counters into the
>> hardware really screwed the pooch something fierce. :)
>> 
>> --
>> Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
>> GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)
>> _______________________________________________
>> juniper-nsp mailing list juniper-nsp at puck.nether.net
>> https://puck.nether.net/mailman/listinfo/juniper-nsp
>> 
> _______________________________________________
> juniper-nsp mailing list juniper-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/juniper-nsp