[c-nsp] Troubleshoot UDP out-of-sequence

[Christopher.Marget at usc-bt.com]

> There is no part of this path with parallel, load-balanced connections, which could be a
> obvious cause. What else could I check? The packets do arrive, so they are not being
> dropped on the way, but they arrive out of sequence, being useless to the application.

I've seen UDP packets (also multicast pricing data) get re-ordered as they transited a transparent bridge.  It was lots of fun explaining the ordered delivery LAN invariant to Tipping Point support folks :-)

I mention this case to illustrate that eliminating parallel paths doesn't guarantee anything.

QoS somewhere perhaps?  Do the QoS markings on these late messages match the others?  Even if the late message wasn't marked down, it may have been queued differently by a policer somewhere along the path.

Are you sure that the packets in question left the originating system in the correct order?  I've seen sending systems at the exchanges do bad things in this regard also.  One way to check the sender is to watch the ID field in the IP packets.  Some OSes increment this number with every packet sent.  If your sender is one of these OSes, then the "late" UDP message should be stamped with a "late" (too low for its position in the packet stream) IP Identifier.

OTOH, if the "late" message is stamped with a IP Identifier value that falls between the IDs of the preceding and following packets, then you can blame the sending system.

/chris