[c-nsp] Troubleshoot UDP out-of-sequence

[Aaron]

Might be something funny going on with APS, and if you are only seeing it at
the NY end, then I would start at the NY landing station sonet equipment.

Reason is, the NY Sonet equipment may be passing both data streams (working
+ protect) on occasion.
I am making assumptions here that the local router is not doing aps, the
stm-4 is protected between the landing station and the router site, and the
submarine system is protected between Sao Paulo and NY (which would be
expected).

Aaron



On Mon, Sep 12, 2011 at 10:19, Persio Pucci <persio at gmail.com> wrote:

> Chris,
>
> there's no QoS anywhere on the path to avoid this kind of problems. As for
> the sequence being received right int the first place, we do have servers
> monitoring the channels in Sao Paulo and NY and we see gaps
> (out-of-sequence
> packets) only in NY.
>
> Giving a quick look at live capture I see that the packets DO have IDs on
> the IP header. I'll check on later tonight, after the daily gaps summary,
> and see if the sequence and the IDs match. I think this is unlikely, but
> worth give it a try.
>
> Persio
>
> On Mon, Sep 12, 2011 at 10:57 AM, <Christopher.Marget at usc-bt.com> wrote:
>
> > > There is no part of this path with parallel, load-balanced connections,
> > which could be a
> > > obvious cause. What else could I check? The packets do arrive, so they
> > are not being
> > > dropped on the way, but they arrive out of sequence, being useless to
> the
> > application.
> >
> > I've seen UDP packets (also multicast pricing data) get re-ordered as
> they
> > transited a transparent bridge.  It was lots of fun explaining the
> ordered
> > delivery LAN invariant to Tipping Point support folks :-)
> >
> > I mention this case to illustrate that eliminating parallel paths doesn't
> > guarantee anything.
> >
> > QoS somewhere perhaps?  Do the QoS markings on these late messages match
> > the others?  Even if the late message wasn't marked down, it may have
> been
> > queued differently by a policer somewhere along the path.
> >
> > Are you sure that the packets in question left the originating system in
> > the correct order?  I've seen sending systems at the exchanges do bad
> things
> > in this regard also.  One way to check the sender is to watch the ID
> field
> > in the IP packets.  Some OSes increment this number with every packet
> sent.
> >  If your sender is one of these OSes, then the "late" UDP message should
> be
> > stamped with a "late" (too low for its position in the packet stream) IP
> > Identifier.
> >
> > OTOH, if the "late" message is stamped with a IP Identifier value that
> > falls between the IDs of the preceding and following packets, then you
> can
> > blame the sending system.
> >
> > /chris
> >
> _______________________________________________
> cisco-nsp mailing list  cisco-nsp at puck.nether.net
> https://puck.nether.net/mailman/listinfo/cisco-nsp
> archive at http://puck.nether.net/pipermail/cisco-nsp/
>