[f-nsp] Outgoing FTP dies mysteriously after upgrading to 10.2.01z

[David Miller]

We have two sets of servers that move data about with FTP.  No, I can't get the developers to fix it with a serious protocol:)  We protect/load balance everything with a pair of ServerIrons.

The two sets are NAT'd out to different IP addresses.  We're nowhere near any NAT limit; we might have a few thousand incoming connections to our web servers and less than a hundred outgoing connections.  

The problem is that maybe 5% of the time part of the connection 'goes away' from the client perspective.  Oddly enough, the application gets a 421 response code from the remote FTP server, indicating that the control channel is still open.  The 421 response always comes right at the 5 minute mark, the timeout limit of the FTP server.  I'm guessing serverIron is closing the data connection a bit before the remote server is ready for that.

It worked under 10.2.01o; a recent upgrade to rev Z broke it.  We upgraded to Z because outgoing connections to remote web servers were failing. 

It's sporadic enough I can't come up with a good test case for it.  It's positively the serverIrons; changing the outgoing route to a <shudder> sonicwall fixes the problem.

Machine particulars follow.  Hints/suggestion on how to solve or better debug the problem most welcome.

--- David


SSH at Foundry1#sho ver
  SW: Version 10.2.01zTI4 Copyright (c) 1996-2007 Foundry Networks, Inc.
      Compiled on Feb 13 2012 at 16:56:21 labeled as WJR10201z
  HW: Stackable Router, SYSIF version 21, Serial #: Non-exist
==========================================================================
    ServerIron 4G SSL, SYSIF 2 (Mini GBIC)
      Serial #:   CH10080936
    0 MB SHM, 1 Application Processors
 4096 KB BRAM, JetCore ASIC IGC version 49
  SW: (1)10.2.01zTJ3
32768 KB PRAM and 2M-Bit*1 CAM for IGC  0, version 0449
==========================================================================
  1.0 GHz Power PC processor 750GX (version 7002/0102) 66 MHz bus
  512 KB boot flash memory
16384 KB code flash memory
  512 KB SRAM
  512 MB DRAM