[f-nsp] Serveriron Session Timeout (delayed FIN ACK)

[Nils Domrose]

FYI in case someone is facing the same problem:

I opened a case at foundry as this was replicate able on all apache  
mod_proxy or rewrite setups on Solaris as well as on Linux I tested.  
If you have some slower apps it looks like apache as a client is  
handling proxy requests slower in terms of TCP closing - the last  
(apache) client FIN is delayed. I saw this with some java clients, too.

I got great support from the new TAC in Amsterdam and we jointly setup  
a test system where the TAC was able to replicate the issue.

Looks like there are two age timers: one for the forwarding session  
and one for the reverse session  which are not synced by default.
Because of that the delayed client FIN was forwarded to the correct  
real server but no nat was done for the delayed packet by the SI as  
the session has timed out already.


I got the hint by the TAC engineer to configure:

server no-one-way-session-age

That fixed all our issues.


Nils
On Sep 7, 2008, at 2:43 PM, Nils Domrose wrote:

> Nils Domrose wrote:
>> Hi,
>>
>> while we introduced a new frontend firewall between out Frontend  
>> Apache server and out Backends, I experienced a strange behavior:
>>
>> in front of out Backend Servers we have a pair of SI4G running  
>> 10.2.0 Software and source-nat.
>> Frontend Server=Client
>> Backend Server=Real Server
>>
>> On the Firewall I can see frequent blocks of FIN ACK packets  
>> (Client to VIP) and RST Packets (Real! Server to Client).
>>
> Of cause I see the FIN ACK blocks only after the tcp.closing timer  
> on the firewall has expired due to the retransmits. I do see the  
> blocked RST packets all the time since there is no state and the  
> firewall does not know anything about the real server the RST is  
> coming from.
>
>> After running multiple snoops the picture looks like this:
>>
>> 1. Client establishes a connection to the VIP (HTTP 1.1).
>> 2. Multiple requests are served via the connection.
>> 3. After a while the connection is idle (and all Packets are ACK'ed
>> 4. After 15 seconds idle as configured on apache side, the backend  
>> server sends a FIN (ACK) to close the connection
>> 5. The Frontend server sends immediately an ACK
>>
>> now the interesting part starts:
>>
>> 6. after 80+ seconds the Frontend Server send the FIN ACK
>> 7. the Packet is not NAT'ed any more when reaching the real server  
>> (Source: Frontend Server -> Dest: real server).
>> also due to the missing NAT the port number does no longer match  
>> the port number of the previous connection from real server point  
>> of view.
>>
>> (the previous connection was source-nat IP:nated port -> real  
>> server:9080
>>
>> 8. The real server sends a RST (due to the unknown TCP connection)  
>> to the Frontend Server
>>
>> Of cause 6 to 8 happen several times afterwards since the frontend  
>> server never receives an answer for the FIN ACK and retransmits the  
>> FIN ACK.
>>
>>
>> I wonder why apache as a client (sometimes) needs more than 80  
>> seconds to send a final FIN ACK but i have not found any  
>> information that this is wrong/ not allowed. We run Apache 2.0  
>> using mod rewrite (we also tested mod proxy) in frontend and backend.
>>
>> Does anyone have an idea what timeout value is responsible for the  
>> deletion of the Serveriron session and is causing the source NAT to  
>> fail?
>> When does this timeout start (i guess once the Serveriron recieves  
>> the first FIN)
>> And last but not least is this configurable ?
>>
>>
>> Any hint is welcome!
>>
>> Nils
>>
>> _______________________________________________
>> foundry-nsp mailing list
>> foundry-nsp at puck.nether.net
>> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp