[j-nsp] Miercom Competitive Performance Testing Results: Cisco ASR9000 vs Juniper MX960

[Stefan Fouant]

On Thu, Sep 24, 2009 at 9:30 AM, Richard A Steenbergen <ras at e-gerbil.net>wrote:

> Well, we saw something similar (that I've described on this list several
> times) but not exactly what you described, starting somewhere in 7.x and
> continuing until 8.5 where it was fixed on most platforms (including
> MX). Basically what happens is during a path change when it adds the new
> path and removes the old, something will cause the update process to
> stall, resulting in many minutes taken to install the new path into the
> hardware. If you did a show route on it while it is trying to make the
> update, you'll see a - on the old path and a + on the new path as it
> tries to update the FIB. It seemed like "something" would cause one
> particular update to stall, which would block that and all further
> updates behind that from taking effect for some number of minutes. Back
> on the M160 this could be 30 minutes to in some cases hours, but when we
> saw it on MX 7 minutes was a good average value. Eventually whatever
> route update was stalling things would go through, and then all the
> other routes that were stuck behind it would come flooding through. If
> you did a "show bgp summary" during this event, the "stuck" routes would
> show up in the "Pending" state counter up top.
>
> This normally didn't cause any packet loss for us, as it would keep
> forwarding via the old path until the new one became active. If the
> next-hop became invalid it would pull the routes immediately (or at
> least, we never got a complaint or saw an example where it didn't, in
> the many hundreds of times we observed this behavior over the years).
> The only time we would see packet loss was when the old path would drop
> BGP and not be able to continue to forward traffic, but not drop link.
> We complained about this for years and years, it went from "we don't see
> this" to "we can't reproduce this" to "we have no idea whats causing
> this". Eventually at 8.5 it become "we changed a whole bunch of things
> to try and fix this", but they never did tell us exactly what. After 8.5
> we have not seen this problem since, at least on the MX platform (and we
> tossed all our M160s into the nearest landfill). Ironically enough the
> problem still exists on J-series running JUNOS (real JUNOS, we stopped
> testing after it was unfixed in the last JUNOS version to be released
> before the switch to -ES), despite there not actually being a hardware
> fib that needs updating. This pretty much put a kibosh on plans to use
> J-series as a BGP route-reflector, since BGP will not propagate the
> route change to other neighbors until it successfully installs the new
> path in its own FIB. If anyone from Juniper wants to step forward and
> describe what the problem or fix was I'd be interested in hearing it,
> our original issue was spread out over too many years and too many PRs
> to ever get anyone to even understand the question let alone give us a
> straight answer.
>

I thought this was the whole idea behind the introduction of the indirect
next-hop.  Basically abstracting a level of recursion so that when
underlying next-hop paths changed, all they needed to do was to do a KRT
change for the indirect-next-hop pointer to the new path.  Any routes that
referenced the indirect-next-hop then automatically utilized the new path
via this mechanism, without having to do a KRT update for thousands of
routes...

Am I missing something or did this behavior change at some point?

-- 
Stefan Fouant