[c-nsp] Troubleshoot UDP out-of-sequence

[Persio Pucci]

Hi Nick,

except for two minor corrections (multicast source is SP, users in NY; and
Sao Paulo to Rio does not use RADs, they are PA-OC12s) that's pretty much
what is going on here.

QoS is off on the 3560, but really cannot give up on him for now. Would you
consider it a suspect more than the others for any specific reason?

Out of your possibilities, we can rule out out-of-sequence from the source
(we have feedhandlers in SP and NY, SP says no gaps, NY says gaps), and QoS
(no QoS the whole path).

Well, time to get started then... tks all!

On Tue, Sep 13, 2011 at 6:57 AM, Nick Hilliard <nick at foobar.org> wrote:

> On 13/09/2011 04:22, Persio Pucci wrote:
> > all Cisco gear, in the following order:
> >
> > 7200 receiving the multicast on a NPE-G1, sending on the same NPE-G1 to a
> > 7600, arriving on a ws6748 with DFC3, going to a OC12 to Rio, arriving on
> a
> > 7600 on a OC12, uplinking to a 3560 via a Sup32, 3560 going to NY
> arriving
> > on a 6500 on a ws6748 with DFC3. There are RADs converting GigE to SDH on
> > both Rio and NY.
>
> so it looks like this:
>
> Sao Paolo:
> user <-> c7200/npe-g1 <-> 7600/6748 <-> (ethernet)RAD(sdh) <-> SDH to Rio
>
> Rio:
> (sdh)RAD(ethernet) <-> 7600/sup32 <-> c3560 <-> (ethernet)RAD(sdh) -> SDH
> to NYC
>
> NYC:
> (sdh)RAD(ethernet) <-> 6500/6748 <-> multicast source
>
> Obviously this excludes all the SDH infrastructure in the middle.  And it's
> all Cisco gear, except for all the gear which isn't Cisco. :-)
>
> Summarising peoples' theories about what could be causing the out-of-order
> packet delivery:
>
> - out-of-order packet delivery from multicast source
> - QoS on any L2 / L3 device
> - bugs on any L1/L2/L3 device
> - maybe FEC on the SDH links fixing up packets broken in transit?
> - activation of protection mechanisms on the SDH links
>
> Looking at this, the only way to diagnose what's going on is to get down on
> your hands and knees and run a packet-by-packet diagnosis to try to find
> out where the problem is occurring.  It would probably be a good idea to
> implement packet sniffing using line splitters rather than using SPAN /
> RSPAN / ERSPAN - if there's a problem happening somewhere, you need to
> debug what's on the wire rather than what the L2 equipment says is on the
> wire.
>
> Incidentally, have you disabled qos completely on the c3560?  That may be a
> good place to start.  Even better, remove the c3560 from the network path
> completely.
>
> It would probably also be a good idea to file a bug report with the
> application developers to request that they support out-of-order packet
> delivery - UDP packet delivery is not guaranteed to implement a coherent
> data stream, and expecting ordered packet delivery for multicast is a naive
> assumption on their part.
>
> Nick
>