[j-nsp] MX960 ARP issues

[John Neiberger]

I'll preface this question by saying that I don't think this is a
problem on the router, but I'm stumped and I'm curious if anyone else
has run into this. We have a Cisco 4948 with two uplinks to different
MX960s we'll call RouterA and Router B. There are a few linux servers
connected to the switch. We have good layer two connectivity between
the routers through this vlan, evidenced by good ARP tables,
responsive pings, and since VRRP is working correctly.

The problem is that the linux servers only respond to ARP requests
from RouterA. When RouterB sends an ARP request, the servers never see
it. Packet captures done on the servers don't even show the packets
arriving. I know they are because ARP is working between the routers
and we also have an SVI on the switch in the same VLAN. We have no
problems with ARP and those other devices. It is only these linux
servers that don't see these particular requests.

I've used "monitor traffic" to verify that the ARP requests are
leaving the router. I also tried setting a static ARP for one of the
servers and I was able to ping it, so we know the path is good. I
don't know much about linux system administration, but I did ask them
to check if iptables or arptables were running and they said no.

The reason I'm nearly certain this has to be their problem is this: if
they reboot their servers, they will respond to ARP requests for a
short time and then they stop. That tells me that something running on
the server must be blocking ARP requests, but why only from one
router? It's very unusual. We've been working on this off and on for a
few weeks and haven't been able to nail down the root cause.

Any ideas? Have any of you seen anything like this before?