[nsp] ip load-sharing per-packet - cef accelerated ?

[Matt Buford]

On Monday, 10 Mar 2003, Dan Hollis wrote:
> On Mon, 10 Mar 2003, Alex Rubenstein wrote:
> > Because of packet-out-of-order issues?
>
> if you have a non sucky tcp stack, this isnt an issue. at least,
> linux/w2k and even w98 dont seem to have problems...
>
> maybe its a problem with trumpet winsock on windows 3.11 ...

This has more to do with how much arrives out-of-order than it does with TCP
implementation.  On two identical relatively high speed links, loss of order
tends to not happen very often so you don't generally notice the ill
effects.

I once had an ISDN and an IDSL line at home.  ISDN got roughly 15kB/sec and
IDSL got roughly 20kB/sec.  I set up per-packet load balancing. The low link
speeds, the difference in link speeds, and the differences in latency
resulted in significant packet reordering.  TCP transfers then got about
25kB/sec and pegged both links.  This doesn't add up you say?  How can 15kB
+ 20kB = 25kB pegging both links?  More bandwidth was consumed to acheive
the same TCP transfer rate.  The answer became clear with tcpdump.  Running
tcpdump on the receiver, it was obvious that there were significant
retransmissions of packets going on.  The interesting thing is the
retransmissions were of packets that were not lost, but simply arrived late
(out of order).  The receiver then eventually saw both the original and the
retransmitted packet.  Thus bandwidth was wasted while still resulting in a
faster transfer rate than any one link.

I also found PPTP and RealPlayer were completely unusable on this setup, as
they both appear to just drop (without retransmission) any packet with a
sequence number lower than the previous highest received.

The end result is that if your links are identical, then as link speed grows
packet reordering becomes less and less of an issue.  I would highly
recommend against per-packet load balancing on anything less than T1 links,
and certainly on anything where circuits are differ in speed and/or latency.
If you are on T1 or faster identical links, then use it at your own risk.
It is clearly not best-practice but ends out working just fine for most
people.  If you feel you must do it, go ahead.  Just understand the
potential ramifications of your actions.

Also, it is rude to reorder other people's microflows.  Don't do per-packet
unless the only packets going across the link are your own (and not
downstream customer networks).