[j-nsp] Going Juniper

[mike+jnsp at willitsonline.com]

On 04/09/2018 08:07 PM, Chris via juniper-nsp wrote:
> Hi,
>
> On 10/04/2018 9:45 AM, mike+jnsp at willitsonline.com wrote:
>>      I see there is a terrific amount of used mx104 and mx240 out there
>> and the specs all seem great. What I'm looking to do is have 2x 10g
>> feeds, route bgp, do flow exporting, and do a certain amount of ingress
>> filtering to protect the network from ddos.Id even like to do cgnat for
>> up to 5000 users but not sure if a single box setup would be wise.
>
> I can't speak for the MX240, but we have some deployments of the
> MX104, MX80 and the vMX.
>
> For the MX104 (and the MX80) the main limitation they have is that the
> CPU on the routing engine is terribly slow. This can be a problem for
> you if you are taking multiple full tables with BGP. Even without
> taking full tables, the RE CPU on the MX104's I have is basically
> always at 100%. Commits are pretty slow as well. This shouldn't be
> such an issue with the MX240 as it has a wider range of routing
> engines available with much better specs.
>
> The MX104's (and MX80's) have the MS-MIC-16G installed. We use the
> MS-MIC-16G for IPSEC, NAT and stateful firewalling (service filters
> are used to only send certain traffic to the stateful firewall). So
> far there has only been 1 issue that I have personally encountered
> with the MS-MIC-16 - the card has crashed on a previous release of
> JunOS when adding a large number of IPSEC peers. Since upgraded I have
> not experienced the same issue though.
>
> I also have some vMX's deployed (they are running on top of Dell
> R740's with 3 x Intel X710 cards to give 12 x 10G interfaces). The
> painful part on getting the vMX to work was the host setup with KVM -
> the documents are severly lacking on Junipers side (but I have written
> up the exact instructions to get the most recent 18.1R1 release
> working on CentOS with no issues).
>
> So far after getting the issues with the KVM host ironed out I have
> been very happy with the performance of the vMX. Since 17.4R1 you can
> deploy a virtual MS-MPC (which requires extra CPU resources) which
> will give you NAT support as well as stateful firewalling support.
> Since its virtualised and the RE runs as a seperate VM you can assign
> more or less resources to it as needed - I have 16G of RAM allocated
> with 6 cores and the time to process/install a full table is only a
> few seconds. They have survived some DDoS attacks that were large
> enough to fill up the transit links with no issues as well. The
> biggest thing is to make sure you get NIC's that support SR-IOV and
> make sure the CPU is fast enough/has enough cores for your
> requirements (you cannot over-allocate the cores!). For my use case, I
> don't think I will be buying any more physical MX's unless I have an
> actual reason to need their hardware, the vMX suites my needs just
> fine. Juniper does provide a (limited) demo of the vMX, happy to send
> you the install guide I wrote up for getting it working on KVM with
> CentOS 7.4 (Ubuntu is also supported for KVM but the install process
> is basically terrible). 

I have a lot of experience and confidence running embedded linux as a
router doing bgp/ospf and so forth. Also running pfsense in virtual
machines for various features. Never knew juniper had a software only
option, that is pretty interesting to me.

I know it can be set up and run like a champ and do some (undefined)
number of gigabits without issue. What concerns me is that there are
performance limitations in these software only platforms based on your
processor/bus/card choices, and of course the performance of a software
hash vs a hardware cam for forwarding table lookups. And usually
(imho),  you hit the platform limits like a truck running into a brick
wall. However, if I knew I was only going to have just a few gbps (3?),
I likely would be very interested doing a live deployment. However, with
that said, it certainly is interesting enough to investigate and I'd
love to see your writeup. At a minimum it sounds very useful and I may
use vMX for pre-deployment testing purposes. 

On your mx104 you said cpu was pegged at %100 - operationally does this
cause you any grief? How long does it take for your routes to recover
after a peer flaps? (eg: your sending traffic to a dead peer before
forwarding is updated to remove those). If you are logged in doing
normal network stuff like looking up routes or making minor config
updates, is the cli slogwash or can you move around and work?

Thank you so much for the feedback!

Mike-