[j-nsp] ISG Dropping TCP packets

[Nicholas Oas]

I believe we may have discovered a bug in ScreenOS 6.3 for ISG-1000
platforms running in Transparent/Layer-2 mode.

The symptom is TCP traffic not being delivered to single IP Addresses
behind the firewall. This happens on a frequency of about once a
month, one IP address behind the firewall will simply stop receiving
inbound TCP traffic. ICMP and UDP packets are unaffected and get
delivered without issue.

The device is under support and I am working with JTAC. Our next step
is to wait until the condition happens again, and collect a packet
traces of both the Untrust and Trust interface of the ISG.

However, the bug is difficult to reproduce so I am reaching out to the
JNSP community to see if anyone else has experienced this issue.

The only way to stop the error condition and restore service to the
affected IP is to reset the ISG. I tried clearing the session table,
ARP table, MAC-learn table, and I tried disconnect/reconnecting the
affected node’s Ethernet cable from the network to no avail. I also
tried disabling TCP Syn Check, TCP Seq Check, and the HTTP ALG. HTTP
session cache is also disabled. Only a reset of the ISG restores
service, which wipes any useful debugging info about the problem.

Packet traces collected from within the Trust zone confirm that the
packets are not being delivered by the firewall.

There is one curious output from debug flow basic.
----------------------------------------------------------------------------
----------------------------------------------------------------------------
When the traffic works correctly, it looks like this:
----------------------------------------------------------------------------
isg-1000-fw-> set ff src-ip 10.0.2.44 dst-ip 10.0.10.36
isg-1000-fw-> set ff src-ip 10.0.10.36 dst-ip 10.0.2.44
isg-1000-fw-> debug flow basic
isg-1000-fw-> get db str
**st: <V1-Untrust|ethernet1/1|Root|4f1> 483dd40:
35b4:10.0.2.44/e7b4->10.0.10.36/50,6,60
****** 00338.0: <V1-Untrust/ethernet1/1> packet received [60]******
  ipid = 13748(35b4), @0483dd40
  packet passed sanity check.
  packet with vlan 1, vlan-group vlan1, vsd 0
   v1-untrust:10.0.2.44/59316->10.0.10.36/80,6<Root>
found mac 00e0deadbeef on ethernet1/4
  flow_decap_vector IPv4 process
  no session found
  flow_first_sanity_check: in <v1-untrust>, out <v1-trust>
  policy search from zone 11-> zone 12
 policy_flow_search  policy search nat_crt from zone 11-> zone 12
  RPC Mapping Table search returned 0 matched service(s) for (vsys
Root, ip 10.0.10.36, port 80, proto 6)
  No SW RPC rule match, search HW rule
swrs_search_ip: policy matched id/idx/action = 104/1/0x1
  Permitted by policy 104
  No src xlate   choose interface v1-trust as outgoing phy if
  session application type 6, name HTTP, nas_id 0, timeout 1800sec
  service lookup identified service 0.
  flow_first_final_check: in <v1-untrust>, out <v1-trust>
  existing vector list 2-4b0f7ac.
  Session (id:129990) created for first pak
  flow_first_install_session======>
xpt: cache src mac in session
xpt: cache dst mac in session
Success installing work and forward sessions
  flow got session.
  flow session id 129990
  skip ttl adjust for packet.
  transfer packet to hardware.
----------------------------------------------------------------------------
----------------------------------------------------------------------------
When the traffic fails, it looks like this:
----------------------------------------------------------------------------
----------------------------------------------------------------------------
**st: <V1-Untrust|ethernet1/1|Root|4f1> 4811b40:
df23:10.0.2.44/b471->10.0.10.36/50,6,60
****** 3026650.0: <V1-Untrust/ethernet1/1> packet received [60]******
  ipid = 57123(df23), @04811b40
  packet passed sanity check.
  packet with vlan 1, vlan-group vlan1, vsd 0
  v1-untrust:10.0.2.44/46193->10.0.10.36/80,6<Root>
found mac 00e0deadbeef on ethernet1/4
  flow_decap_vector IPv4 process
  no session found
  flow_first_sanity_check: in <v1-untrust>, out <v1-trust>
  policy search from zone 11-> zone 12
 policy_flow_search  policy search nat_crt from zone 11-> zone 12
  RPC Mapping Table search returned 0 matched service(s) for (vsys
Root, ip 10.0.10.36, port 80, proto 6)
  No SW RPC rule match, search HW rule
swrs_search_ip: policy matched id/idx/action = 104/0/0x1
  Permitted by policy 104
  No src xlate   choose interface v1-trust as outgoing phy if
  session application type 6, name HTTP, nas_id 0, timeout 1800sec
  service lookup identified service 0.
  flow_first_final_check: in <v1-untrust>, out <v1-trust>
  existing vector list 12-1aa2b174.
  Session (id:127841) created for first pak
  flow_first_install_session======>
xpt: cache src mac in session
xpt: cache dst mac in session
Success installing work and forward sessions
  flow got session.
  flow session id 127841
  skip ttl adjust for packet.
  tcp proxy processing...
----------------------------------------------------------------------------
----------------------------------------------------------------------------
(The IP and MAC addresses in this output have been sanitized)

Notice the only difference is the last line, packets that fail are
"tcp proxy processing" and packets that succeed are transferred to
hardware. TCP Syn Proxy is disabled, and 'get counter flow' shows 0
hits for 'tcp proxy'.

Has anyone else out there with an ISG running 6.3 in transparent mode
experienced this issue??

-Nicholas