[c-nsp] carrier load balancing packets across unequal path lengths

[Arie Vayner (avayner)]

Matt,

In general using unequal paths for load sharing is a valid solution, BUT, this should not introduce packet reordering.
This means that the routers on SP environments should not be operating in per-packet load balancing, but should be using some hash function to make sure that the same session always takes the same path.
Actually, most SP-grade devices (Cisco 6500/7600/GSR etc) don't even support the per-packet option...

I suggest you have a chat with your SP again. 
You could use this document for reference (1st best match on Google...):
http://networks.cs.ucdavis.edu/~yslee/reference/a-2-laor-network-2002.pdf


Arie

-----Original Message-----
From: cisco-nsp-bounces at puck.nether.net [mailto:cisco-nsp-bounces at puck.nether.net] On Behalf Of Matt Carter
Sent: Friday, January 16, 2009 04:04
To: Cisco Mailing list
Subject: [c-nsp] carrier load balancing packets across unequal path lengths

hi all , 

i have an issue where a carrier that provides transit has decided to begin load balancing traffic across unequal path lengths.

ie, instead of 

     --R3--
   /        \
R1           R2
   \        /
     --R4--


we are seeing something a lot more like this

R1-----R3------R2
  \            /
    --R4      /
         \   /
          R5

as a result packets going via R4 & R5 are arriving with a different TTL and out of order to the tune of 30 ms or so. its throwing our monitoring tools out of wack because one minute a host is at hop 11, the next it's at 10. (watching path changes to bgp beacons) . with data payloads consider a voice stream of 10 packets egressing at 20ms interval where by every other packet is being sent via the longer path 

1 2 3 4 5 6 7 8 9 10

aside from the TTL issue we end up with packets arriving like this
1 x = x
2 x+20+30 = +50
3 x+40 = +40
4 x+60+30 = +90 
5 x+80 = +80
6 x+100+30 = +130
7 x+120 = +120
8 x+140+30 = +170
9 x+160 = +160
10 x+180+30 = +210

so for the original 1 through 10 packets that were egressed sequentially ends up arriving as

1 , 3 , 2 , 5 , 4 , 7 , 6 , 9 , 8 , 10

the carrier i'm dealing with doesn't seem to even comprehend that this is a problem .. can't say i've ever had that kind of reponse before and i'm left a little bewildered.. i'm used to hiding all this stuff away in the MPLS core and preserving the customer TTL's.. anyone else interacting with carriers who seem to think this is perfectly ok network design?? thoughts/comments/suggestions??

kind regards,

--matt

_______________________________________________
cisco-nsp mailing list  cisco-nsp at puck.nether.net
https://puck.nether.net/mailman/listinfo/cisco-nsp
archive at http://puck.nether.net/pipermail/cisco-nsp/