[j-nsp] Strange ARP issue on M7i

[Markus]

Hi all,

last night I encountered something weird (in my opinion). Not sure if 
Juniper related but maybe someone here has seen something like this?

I was experiencing a strange effect that several websites hosted on a 
Linux KVM VM didn't load properly. They would load but 90% of the time 
hang in some strange way, the browser displaying "Waiting for 
www.sitename.com..." after all the page has loaded, or even before 
anything of the page was displayed. A minute later it would work 
sometimes, but only for a short period of time. After eliminating all 
MySQL, Apache, KVM etc. as the source of the problem I logged into the 
M7i in front of that host and saw:

admin at ffm01.rt> show arp no-resolve |grep 195.100.100.7
00:25:90:38:66:c6 195.100.100.7    ge-0/0/0.0    none
00:25:90:38:66:c6 195.100.101.34   ge-0/0/0.0    none

With 195.100.100.7 being the KVM host. So I thought: why is 101.34 up? 
It's an IP that wasn't in use for years. And in the Juniper config a 
whole /24 was still getting routed to it. I thought, OK, the KVM host 
got hax0red or something and the intruder assigned 101.34, but couldnt 
find anything. 101.34 wasn't reachable from any machine in the same LAN 
and the MAC could not be seen either. No traffic to/from it on the 
Switch monitoring port either. All I saw was traffic (port scans I 
think) to the /24 which ended up on the KVM host (195.100.100.7). That 
was an indicator that the KVM host was really also saying "I have 
195.100.101.34". Or the Juniper insisted that the IP is at that MAC. I 
suspect the latter. I shutdown the KVM host physically and cleared the 
ARP cache on the Juniper, 195.100.100.7 was gone, but 195.100.101.34 was 
still there with the identical MAC, as before.
I then removed the static route entry for the /24 which was pointing to 
195.100.101.34 and only then the arp entry for 195.100.101.34 disappeared!

Isn't that weird? Where did that arp entry come from and why was it 
saved on the Juniper for so long, and only got removed after I removed 
the static routing of that /24?

I'm running JUNOS 8.0R2.8. :)

This didn't eliminate the problem with the websites reachability, I 
think it is something local with my dialup connection as I see a lot of 
TCP retransmission errors when accessing all sites on any of the VMs 
hosted on that KVM host. Through an alternative dialup provider 
everything is fine. Other sites on other boxes in the same LAN work just 
fine though via the first provider. The problem comes and goes now. 
Really puzzled!

Anyway, can't stop thinking about the ARP thing so I thought I would ask 
here! Thank you very much!

Regards
Markus