[c-nsp] Troubleshoot UDP out-of-sequence

[Persio Pucci]

Chris,

there's no QoS anywhere on the path to avoid this kind of problems. As for
the sequence being received right int the first place, we do have servers
monitoring the channels in Sao Paulo and NY and we see gaps (out-of-sequence
packets) only in NY.

Giving a quick look at live capture I see that the packets DO have IDs on
the IP header. I'll check on later tonight, after the daily gaps summary,
and see if the sequence and the IDs match. I think this is unlikely, but
worth give it a try.

Persio

On Mon, Sep 12, 2011 at 10:57 AM, <Christopher.Marget at usc-bt.com> wrote:

> > There is no part of this path with parallel, load-balanced connections,
> which could be a
> > obvious cause. What else could I check? The packets do arrive, so they
> are not being
> > dropped on the way, but they arrive out of sequence, being useless to the
> application.
>
> I've seen UDP packets (also multicast pricing data) get re-ordered as they
> transited a transparent bridge.  It was lots of fun explaining the ordered
> delivery LAN invariant to Tipping Point support folks :-)
>
> I mention this case to illustrate that eliminating parallel paths doesn't
> guarantee anything.
>
> QoS somewhere perhaps?  Do the QoS markings on these late messages match
> the others?  Even if the late message wasn't marked down, it may have been
> queued differently by a policer somewhere along the path.
>
> Are you sure that the packets in question left the originating system in
> the correct order?  I've seen sending systems at the exchanges do bad things
> in this regard also.  One way to check the sender is to watch the ID field
> in the IP packets.  Some OSes increment this number with every packet sent.
>  If your sender is one of these OSes, then the "late" UDP message should be
> stamped with a "late" (too low for its position in the packet stream) IP
> Identifier.
>
> OTOH, if the "late" message is stamped with a IP Identifier value that
> falls between the IDs of the preceding and following packets, then you can
> blame the sending system.
>
> /chris
>