[c-nsp] carrier load balancing packets across unequal path lengths

[Matt Carter]

hi all , 

i have an issue where a carrier that provides transit has decided to begin load balancing traffic across unequal path lengths.

ie, instead of 

     --R3--
   /        \
R1           R2
   \        /
     --R4--


we are seeing something a lot more like this

R1-----R3------R2
  \            /
    --R4      /
         \   /
          R5

as a result packets going via R4 & R5 are arriving with a different TTL and out of order to the tune of 30 ms or so. its throwing our monitoring tools out of wack because one minute a host is at hop 11, the next it's at 10. (watching path changes to bgp beacons) . with data payloads consider a voice stream of 10 packets egressing at 20ms interval where by every other packet is being sent via the longer path 

1 2 3 4 5 6 7 8 9 10

aside from the TTL issue we end up with packets arriving like this
1 x = x
2 x+20+30 = +50
3 x+40 = +40
4 x+60+30 = +90 
5 x+80 = +80
6 x+100+30 = +130
7 x+120 = +120
8 x+140+30 = +170
9 x+160 = +160
10 x+180+30 = +210

so for the original 1 through 10 packets that were egressed sequentially ends up arriving as

1 , 3 , 2 , 5 , 4 , 7 , 6 , 9 , 8 , 10

the carrier i'm dealing with doesn't seem to even comprehend that this is a problem .. can't say i've ever had that kind of reponse before and i'm left a little bewildered.. i'm used to hiding all this stuff away in the MPLS core and preserving the customer TTL's.. anyone else interacting with carriers who seem to think this is perfectly ok network design?? thoughts/comments/suggestions??

kind regards,

--matt