[j-nsp] Miercom Competitive Performance Testing Results: Cisco ASR9000 vs Juniper MX960

[Richard A Steenbergen]

On Thu, Sep 24, 2009 at 12:13:41PM +0300, Pekka Savola wrote:
> Have you tested MX also during RIB or FIB changes?  I'd like to use 
> this as a soapbox.  On Juniper platforms, when a next-hop changes, 
> this results in first DELETE and then (maybe much later) ADD 
> operations in FIB.  We have noticed that when BGP next-hop changes for 
> 300K prefixes, this has led to up to 7 minutes of packet loss due to a 
> lack of FIB entry for a destination. This was seen on T-series, but MX 
> has the same architectural limitations.  This slowness was introduced 
> sometime in JunOS 8.x. And this is "working as designed" and "no way 
> to speed it up".
> 
> Hopefully at some point someone will do performance testing in these 
> kind of scenarios as well; I don't recall seeing such a test myself.

Well, we saw something similar (that I've described on this list several
times) but not exactly what you described, starting somewhere in 7.x and
continuing until 8.5 where it was fixed on most platforms (including
MX). Basically what happens is during a path change when it adds the new
path and removes the old, something will cause the update process to
stall, resulting in many minutes taken to install the new path into the
hardware. If you did a show route on it while it is trying to make the
update, you'll see a - on the old path and a + on the new path as it
tries to update the FIB. It seemed like "something" would cause one 
particular update to stall, which would block that and all further 
updates behind that from taking effect for some number of minutes. Back 
on the M160 this could be 30 minutes to in some cases hours, but when we 
saw it on MX 7 minutes was a good average value. Eventually whatever 
route update was stalling things would go through, and then all the 
other routes that were stuck behind it would come flooding through. If 
you did a "show bgp summary" during this event, the "stuck" routes would 
show up in the "Pending" state counter up top.

This normally didn't cause any packet loss for us, as it would keep
forwarding via the old path until the new one became active. If the
next-hop became invalid it would pull the routes immediately (or at
least, we never got a complaint or saw an example where it didn't, in
the many hundreds of times we observed this behavior over the years).
The only time we would see packet loss was when the old path would drop
BGP and not be able to continue to forward traffic, but not drop link. 
We complained about this for years and years, it went from "we don't see
this" to "we can't reproduce this" to "we have no idea whats causing
this". Eventually at 8.5 it become "we changed a whole bunch of things
to try and fix this", but they never did tell us exactly what. After 8.5
we have not seen this problem since, at least on the MX platform (and we
tossed all our M160s into the nearest landfill). Ironically enough the
problem still exists on J-series running JUNOS (real JUNOS, we stopped
testing after it was unfixed in the last JUNOS version to be released
before the switch to -ES), despite there not actually being a hardware
fib that needs updating. This pretty much put a kibosh on plans to use
J-series as a BGP route-reflector, since BGP will not propagate the
route change to other neighbors until it successfully installs the new
path in its own FIB. If anyone from Juniper wants to step forward and
describe what the problem or fix was I'd be interested in hearing it,
our original issue was spread out over too many years and too many PRs
to ever get anyone to even understand the question let alone give us a
straight answer.

-- 
Richard A Steenbergen <ras at e-gerbil.net>       http://www.e-gerbil.net/ras
GPG Key ID: 0xF8B12CBC (7535 7F59 8204 ED1F CC1C 53AF 4C41 5ECA F8B1 2CBC)