[c-nsp] IP packet out of order.

[Olivier Gosselet]

Andre, Adel,

I answer both of you in the same mail ... Thanks for your valuable feedback
on this one.
I could have put more details like the following in the first mail. Let's
take back the setup details :

PC_A - Ethernet LAN - Router A - Ethernet LAN - Bridge Eth/ATM - ATM Network
- PA-A3 Router B - Ethernet LAN - PC_B

I made some more precise tests with various ping packet size :

* 1472 Bytes Data - only one packet no problem

20 Bytes IP + 8 Bytes ICMP + 1472 Bytes Data = 1500 Bytes

* 1473 Bytes Data - an IP fragment need to be created ... 

20 Bytes IP + 8 Bytes ICMP + 1472 Bytes Data = 1500 Bytes
20 Bytes IP + 1 Byte Data = 21 Bytes

Start of the problem :

echo request from PC_B -> PC_A all packet are ordered
echo reply from PC_A -> PC_B small fragment comes first

* 1498 Bytes Data - the fragment still fits into 1 ATM cell ?

20 Bytes IP + 8 Bytes ICMP + 1472 Bytes Data = 1500 Bytes
20 Bytes IP + 26 Bytes Data = 46 Bytes

* 1499 Bytes Data - the problem disappear

20 Bytes IP + 8 Bytes ICMP + 1472 Bytes Data = 1500 Bytes
20 Bytes IP + 27 Bytes Data = 47 Bytes

> I assume the Bridge is a DSL CPE.

It is a broadband wireless access device from Alcatel.

> Any chance that these packets have differing TOS/DSCP? Not 
> that they would influence a bridge, though...

No

> So there are two locations where reordering might happen:
> 
> a) The bridge which takes Ethernet frames and serializes them 
> into AAL5 SNAP cell sequences
> b) The router which reassembles these sequences into SNAP frames and
> then terminates them to l3 in some kind of hidden BVI (known as
> route-bridged encapsulation)
> 
> I'm getting some feeling that what happens, happens during reassembly.

I have the same feeling ... As you can see from what I have added, the
problem only happens in the direction where the Cisco device re-assemble.

> In your setup, it's not actually IP, it's Ethernet and SNAP. 
> While you could fit that 38byte packet into a single AAL5 
> cell using "aal5mux ip"
> easily, I actually doubt you can fit the SNAP variant of 
> that. But I'm not having the data handy to verify, and 
> actually it makes no difference whether it goes in one cell or two.

I agree with you. I don't know exactly how many cells are required to
transport the fragment but from my tests, I saw that as soon as the fragment
becomes 47 Bytes (limit of 1 to 2 cell ?) the problem disappear. But OK ...
I agree this 1 or 2 cell theory is still a guess for now. I need to go more
deeply in aal5snap encapsulation to understand this more clearly.

> > I know there is only one physical path in the ATM network from the 
> > bridge to the router. I have no control on the ATM Bridge and/or 
> > Network but I would assume they send and transport the 
> resulting cells 
> > on a FIFO basis meaning all the cell of packet 1 followed 
> by the cell of packet 2.
> 
> But there is still a chance that the bridge (the DSL CPE) 
> intersplices cells from packet 2 into the cell stream 
> generated from packet 1 that is still going out. Notice that 
> such CPE is usually a bandwidth bottle- neck, especially in 
> the upstream direction of ADSL. 
> As it is usually also doing 
> policing, it will send out the cell stream slow enough for a 
> second frame beeing able to enter the bridge while the first 
> is still beeing sent out in a cell stream. What happens all 
> depends on how the box is built, but honestly, I assume the 
> second packet is queued up and not sliced into cells until 
> the first cell stream is out. But you cannot be sure of that. 
> With some parallelity in a hardware implementation of 
> serialization, it can indeed splice the cell(s) of packet 2 
> into the still not completed ones of packet 1.

This is something I need to check with Alcatel. In this case
upstream/downstream is symetric and we go up to 8 Mbit/s in both directions
with this kind of device.

> > Would there be any chances that the Router B dequeues the second 
> > packet first while the ATM cells were all received in the 
> correct order.
> 
> I think this can happen. The reason for this beeing that 
> there is not just a dequeueing of a packet, but a reassembly 
> of cells into a packet. If this is done in parallel with 
> hardware support, it could well happen that packet 2 is 
> reassembled earlier or at the same time that packet 1 is, 
> allowing it to come out earlier.

I don't think I could find a magic command to avoid this behaviour then ...
It would mean a router awaiting to process packet 2 which is complete while
a cell of packet 1 may never arrive. 
 
> Do you actually have problems with out of order packets? I 
> was under the impression that todays implementations of IP 
> (during fragment reassembly) and TCP (segment reassembly) 
> deal quite well with out of order packets, it's just some 
> firewalls that choke on it...

Well I tought like you but one of my customer is doing a VPN PPTP connection
between an MS-XP and a Linux ...
As a way to avoid fragmentation and MTU issues, Microsoft has implemented a
"Negotiate multi-link for single link connections" options ... -> you ends
up with a normal size data packet beeing fragmented in 2 pieces and carried
into PPP multilink packet in turns encapsulated in GRE ! And of course there
is a big and a small packet. And of course Microsoft Point-to-Point
Tunneling Protocol (PPTP) discards all packets that are received out of
sequence.

I am facing a customer saying "it is working for all our remote sites except
the one connected to your network" and not willing to change its setup -> I
want to be 100% sure that I can't do anything myself to avoid this before
asking and helping him to find another solution to avoid a "problem" of my
backbone.   

Regards,

Olivier