[j-nsp] EX 8200 deployment

[Richard A Steenbergen]

On Sun, Mar 21, 2010 at 06:08:33PM +1100, Julien Goodwin wrote:
> > * Obviously this is a very different architecture from Juniper's normal 
> > boxes, so be prepared for vlan space being shared across the entire box, 
> > not a per-interface basis.
> 
> So far, apart from the MX I'm not aware of any Juniper gear that does
> switching with multiple VLAN spaces.

That's not exactly correct. For all intents and purposes MX is a much
cheaper ethernet-optimized T640, with an extra layer of "stuff" added to
let it do ethernet switching between some interfaces. Under that layer
of stuff though, it is still the same basic architecture as a T-series,
in which every interface has its own totally unique vlan space. On a M/T
series you didn't have the "ethernet switching" component, but that has 
nothing to do with the vlan id space.

This is completely and totally different from the architecture of a
"layer 3 switch" (like a 6509, Nexus, Foundry, Force10, etc) which is at
its core a layer 2 ethernet switch, with some "stuff" glued on to make
it do routing. In a "layer 3 switch" the vlan space is shared globally
across the box just like in a layer 2 switch, and when you want to
"route" on an interface what you're actually doing is secretly stealing
one of those 4096 box-wide vlans, using some hacks to do routing on it, 
and applying it as an access mode vlan to a single interface.

The ironic part of the whole thing is that, as far as I understand at
any rate, EX isn't actually a layer 3 switch architecture like the rest
of those boxes. In Juniper's rush to get into the enterprise switch
market, it seems like they decided to copy the other enterprise-marketed
switches out there, bad designs and all. At the end of the day it
doesn't "really" matter, most of us are comparing this to similar boxes
in the market segment which have the same limitations (Nexus 7k, etc),
it's just something people coming from other Juniper products need to
know.

> Thank you for doing the testing on this, I was assuming this was a bug
> as I'd thought they couldn't be *that* stupid.
> 
> To make things worse counters for vlan.XXX traffic are also only the
> traffic destined *to* the interface, not counting traffic routed
> *through*.

Correct. I actually found some old gripes about this when I searched
j-nsp after noticing the problem, but it is a big enough issue that I 
think it needs to be repeated again (and again and again, until it gets 
fixed :P).

At one point I work working on an sflow to snmp emulator for some of my 
poor Foundry-using friends who can't bill customers landing on vlans, 
may end up having to dust that off as a workaround for the EX design 
flaw.

> Lack of outbound policers also makes it fairly useless in many roles
> where enforcing max bandwidth on a WAN link is required (At least here
> in Australia carriers complain if you actually dump 100Mbit of traffic
> on a 100Mbit point-to-point link).

I believe this is just a current limitation of the software, not a flaw 
in the design of the hardware, so it should be "coming soon". Again, 
just something people need to be aware of, since as you say it can be a 
big problem if you need those features. :)

> I've only just upgraded a bunch of stuff *to* 2GB, and don't have any
> real space issues. I would very much appreciate if Juniper would just
> give us two, externally accessible CF slots for storage and have that
> be it.

I'm talking about the EX8200 here, which has 2GB of DRAM, not a
stackable box. When the thing crashes, where do you plan to write the
kernel panic file? I wasn't even able to work with my 512mb rpd
coredump, not enough free space to uncompress the tarball. Maybe you
aren't a power user and you don't notice the issue right now, but trust
me this is a setup for a lot of problems in the future.

> So the EX (4200) bits from my personal list:
> * EX4200 - bootp relay doesn't work when configured inside a
> routing-instance, works when configured at top to use an instance
> * The commands in
> http://kb.juniper.net/index?page=content&id=KB13206&cat=JUNOS_EX&actp=LIST
> don't exist in 9.5
> 
> I'm mostly on 10.0R2.10, but all our EX's are still 9.5.

I'm more interested in the 8200 than the 4200, so we haven't done that
much interesting testing on the 4200, but what I can tell (for the sake
of the mailing list archives) you is that the 3200/4200 and 8200 are two
different revisions of the same family of ASICs, with different feature
supported by the hardware, and different levels of support in software
for those two different revisions of hardware. What 3200/4200 does is
not necessarily the same as what 8200 does.

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)