[f-nsp] very strange NAT/PAT issue on ServerIron XL

[Eric Hankins]

Folks,

Have encountered an odd issue related to translating a group of
servers on a private IP subnet, overloaded behind a single public IP.

It works fine...until a server is rebooted for whatever reason. The
server comes back but is unable to get out to the public net *for
awhile* and then starts working again. The time period for which
connectivity is broken is not constant...could be 2 minutes, could be
an hour. Can ping around the private subnet just fine, including
to/from the ServerIron.

Sniffing the interface on the server shows it is definitely firing
packets onto the wire. Turning on 'monitor both' for the server port
in question, setting a mirror-port for another server on the switch,
and sniffing on that server shows that the switch is definitely
receiving the packets. However, turning on 'debug ip nat trans' on the
switch pleads ignorance, showing absolutely nothing for traffic
sourced from the server in question. But, I leave my ping running and
it eventually "just starts working" and everything looks normal after
awhile. Note that it "just starts working" on its own regardless of me
pinging or otherwise attempting to connect out to the public net -
based on logs of services that regularly connect to other hosts on the
Internet.

I've already tried blaming Solaris for this, and after trying
different interface types, drivers, and addresses, the issue persists.
I've proven that traffic arrives at the ServerIron, but it simply
refuses to translate it for awhile. It happens with every machine
connected to the switch, and I'm really at a loss to come up with an
explanation. If anyone has any theories, or can tell me I'm stupid and
doing something wrong, I'd appreciate it.

Config is below, with public addresses removed to protect the innocent.

Note that the single public address used in the NAT pool is unique and
does not match any of the virtuals or the ServerIron itself.

telnet at slb01#sh conf
Startup configuration:
!
ver 07.3.06aT12
global-protocol-vlan
!
!
server tcp-age reset both

server icmp-message
!
!
!
!
!
!
!
!
!
!
!
!
!
server real web01 10.10.100.51
 port 81
 port 81 url "HEAD /"
 port http
 port http url "HEAD /"
!
server real web02 10.10.100.52
 port 82
 port 82 url "HEAD /"
 port http
 port http url "HEAD /"
!
server real app01 10.10.100.21
 port 8009
 port 8080
!
server real app02 10.10.100.22
 port 8009
 port 8080
!
server real db01 10.10.100.101
 port 3306
!
!
server virtual web xxx.xxx.xxx.xxx
 port 82 concurrent
 port 81 concurrent
 port http concurrent
 bind 82 web02 82
 bind 81 web01 81
 bind http web01 http web02 http
!
server virtual db xxx.xxx.xxx.xxx
 port 3306 concurrent
 bind 3306 db01 3306
!
server virtual app xxx.xxx.xxx.xxx
 port 8009 concurrent
 port 8080 concurrent
 bind 8009 app01 8009 app02 8009
 bind 8080 app01 8080 app02 8080
!

!
vlan 1 name DEFAULT-VLAN by port
!
vlan 10 name PUBLIC by port
 untagged ethe 1 to 2
  router-interface ve 10
!
vlan 100 name PRIVATE by port
 untagged ethe 3 to 22
  router-interface ve 20
!
enable telnet password .....
enable super-user-password .....
hostname slb01
ip route 192.168.101.0/24 10.10.100.254
ip forward
ip address 10.10.100.1/24
ip nat inside
ip nat inside source list 10 pool SVCEXTIP overload
ip nat pool SVCEXTIP xxx.xxx.xxx.xxx xxx.xxx.xxx.xxx prefix-len 32
ip nat trans dns maximum
ip nat trans finrst 30
ip nat trans max-entries 3600
ip nat trans timeout maximum
ip nat trans udp maximum
ip show-subnet-length
ip default-gateway xxx.xxx.xxx.xxx
ip policy 1 cache tcp 0 global
ip policy 2 cache udp 0 global
server connection-log all
clock summer-time
clock timezone us Pacific
interface e 1
 port-name CORE01 UPLINK (public)
 speed-duplex 100-full
!
interface e 2
 speed-duplex 100-full
!
interface e 3
 port-name DB01 SVC
 speed-duplex 100-full
!
<snipped remainder of interface section, nothing special here>

interface ve 10
 ip address xxx.xxx.xxx.xxx 255.255.255.128
!
!
interface ve 20
 ip address 10.10.100.1 255.255.255.0
!
!
!
!
!
access-list 10 permit 10.10.100.0/24
!
!
end