[f-nsp] strange 7.4.00 behaviour on serveriron XL

[Matt Stockdale]

Greetings folks-

  Recently I've brought up an 8 port ServerIron XL running 7.4.00T12,
and run into some odd behavior.

Specifically, this new SI is running behind 2 redundant Linux firewalls.
each linux firewall has an address on their internal firewalled network
segment, and in addition, the active firewall will alias the internal
gateway address, and arp-spoof it out when it takes over. We have at
least 4 identical instances of this configuration here, and everything
"just works".

The difference here is that this XL is running 7.4.00T12, and the others
the last of the 7.3 train.

I'm seeing a few different things.

1) when I fail over from the primary to the secondary firewall, I lose
connectivity to the serveriron. On the console, it becomes clear that it
just can't pick up the MAC of the gateway.

> si#sh arp
> IP              Mac            Type Port Age  Vlan
> xxx.yyy.zzz.129 0000.0000.0000 Inv  6    0    3524

Port 6? there is nothing on port 6. only port 1.  

2) failing back, it will sometimes but not always come back to life.
When it did not, it required a reload. Attempting to ping it from the
secondary (then live) firewall (and receiving no responses), the arp
tables looked thusly

> si#sh arp
> IP              Mac            Type Port Age  Vlan
> xxx.yyy.zzz.129 0000.0000.0000 Inv  6    0    3524
> xxx.yyy.zzz.131 0000.0000.0000 Inv  6    0    3524

Clearly, it was receiving the ping packet, as .131 is the address of the
secondary firewall.

3) As an attempted workaround, I tried adding static arp's for the real
internal interfaces of the firewalls. This seemed to remove it from a
broken state, however, a packet dump on the live firewall showed that it
began spewing ICMP, about once a second. (I should have left the
timestamps on, sorry)

> [root at fw-sec root]# tcpdump -tni eth1 icmp
> tcpdump: listening on eth1
> xxx.73.52.155 > 0.0.0.0: icmp: echo request
> xxx.73.52.155 > 0.0.0.0: icmp: echo request
> xxx.73.52.155 > 0.0.0.0: icmp: echo request
> xxx.73.52.155 > 0.0.0.0: icmp: echo request

once I reloaded, it now behaves as I would expect it to. However, I'm
concerned that I had to define static arp entries. It doesn't feel
right, and I've never had to do it before.

> si#show arp
> IP              Mac            Type Port Age  Vlan
> xxx.yyy.zzz.130 0030.4841.e58a Sta  1    0    1
> xxx.yyy.zzz.131 0030.4841.7ac8 Sta  1    0    1
> xxx.yyy.zzz.129 0030.4841.e58a Dyn  1    0    1
> Total Arp Entries : 3

I've begun the process of opening a ticket w/ foundry support, but my
past experience with them leads me to believe that nothing (useful) will
come of it. So, I turn to you, my esteemed colleagues.

Thoughts?

  Matt