[c-nsp] Troubleshoot UDP out-of-sequence

[Nick Hilliard]

On 13/09/2011 04:22, Persio Pucci wrote:
> all Cisco gear, in the following order:
> 
> 7200 receiving the multicast on a NPE-G1, sending on the same NPE-G1 to a
> 7600, arriving on a ws6748 with DFC3, going to a OC12 to Rio, arriving on a
> 7600 on a OC12, uplinking to a 3560 via a Sup32, 3560 going to NY arriving
> on a 6500 on a ws6748 with DFC3. There are RADs converting GigE to SDH on
> both Rio and NY.

so it looks like this:

Sao Paolo:
user <-> c7200/npe-g1 <-> 7600/6748 <-> (ethernet)RAD(sdh) <-> SDH to Rio

Rio:
(sdh)RAD(ethernet) <-> 7600/sup32 <-> c3560 <-> (ethernet)RAD(sdh) -> SDH
to NYC

NYC:
(sdh)RAD(ethernet) <-> 6500/6748 <-> multicast source

Obviously this excludes all the SDH infrastructure in the middle.  And it's
all Cisco gear, except for all the gear which isn't Cisco. :-)

Summarising peoples' theories about what could be causing the out-of-order
packet delivery:

- out-of-order packet delivery from multicast source
- QoS on any L2 / L3 device
- bugs on any L1/L2/L3 device
- maybe FEC on the SDH links fixing up packets broken in transit?
- activation of protection mechanisms on the SDH links

Looking at this, the only way to diagnose what's going on is to get down on
your hands and knees and run a packet-by-packet diagnosis to try to find
out where the problem is occurring.  It would probably be a good idea to
implement packet sniffing using line splitters rather than using SPAN /
RSPAN / ERSPAN - if there's a problem happening somewhere, you need to
debug what's on the wire rather than what the L2 equipment says is on the
wire.

Incidentally, have you disabled qos completely on the c3560?  That may be a
good place to start.  Even better, remove the c3560 from the network path
completely.

It would probably also be a good idea to file a bug report with the
application developers to request that they support out-of-order packet
delivery - UDP packet delivery is not guaranteed to implement a coherent
data stream, and expecting ordered packet delivery for multicast is a naive
assumption on their part.

Nick