[c-nsp] Troubleshoot UDP out-of-sequence

[Lamar Owen]

On Monday, September 12, 2011 07:47:14 PM Persio Pucci wrote:
> The bad part is that I believe my customers (and the boss) won't take
> "that's how UDP works" for an answer. Although there's TCP Replay for the
> multicast streams, it is somewhat "delayed" from the realtime data and that
> puts them behind other competitors.

First of all, you have my sympathy, really you do.  I have been in the position of being between customer/employer demands and the hard limitations of the technology being used, and it can be difficult to bring the customer/employer to the realization that the technology chosen just simply is not suited to the required task.

Having said that, the technology that comes to my mind first to try to get a stream of UDP packets across the link in an ordered way is layer 2 tunneling of some sort, like L2TPv3.

That will introduce its own latency, of course.

But the speed of light is introducing its own latency there as well due to the path length, and it may be that any resiliency in the SDH network you're using may require path diversity, and if those paths aren't the same length, it might be that a packet could start on a shorter path after another packet starts on the longer path, and the first packet arrives first, out of order.  And it doesn't matter what the customer wants; the speed of light is the speed of light.  UDP is what UDP is; there is no concept of sequence built into the protocol.

Tunneling, of course, introduces possibly unacceptable latency that could be greater than the latency of the TCP Replay.  Tunneling in the way I'm talking about also puts all those stations on the same layer 2 broadcast domain, with all the problems that sort of network implies. But as soon as that UDP stream hits a router, that router will assume that it can forward those packets out of order if it needs to do so (where need is determined dynamically by the router based on other traffic traversing the router, resource allocation on the router, etc).  And to top it all off, Cisco is not likely to treat UDP out-of-order delivery as a bug.

But it will be interesting to see what you find, especially if you find a 100% reliable way of forcing UDP to have ordered delivery.   That could prove useful for other applications. 

Yes, you need your ducks in a row to determine where the re-ordering is occuring, as well as a way to determine that the packet was indeed re-ordered.  This is not going to be easy, nor is it going to be inexpensive to find, in my opinion.  

But I of course reserve the right to be wrong.  Good luck.