[f-nsp] Fun with DSR and servers-as-routers

[Chris Woodfield]

Had a fun ADX troubleshooting session over the weekend that I think deserves mention here.

The reported symptom was a real server on a DSR VIP that was being marked as active even when the VIP loopback was removed from the real. When the healthcheck URL was removed, the server was marked down, but if the lo:0 loopback (which is the IP of the VIP), the real would be marked as active despite this. Even looking at healthcheck counters suggested that sucessful responses were being received, verifying that nothing was "stuck" on the ADX.

After some time poking and pcapping, what we found out was that the ip_forward sysctl was set to 1 on the real in question, which is Linux-speak for "this server can act as a router". Apparently this includes one-armed routing, as the following is what I found:

- ADX sends healthcheck packet to real server's MAC address, with VIP IP as destination (and another IP local to the ADX as the source IP).
- real server looks up its table, and lacking a /32 for that IP, it forwards the packet along - to the VIP itself.
- The VIP then handles the incoming packet like any other, despite the fact that the source IP is local to the ADX and the packet is obviously coming in from the outside. The connection then gets forwarded to an active real, and returned back to the ADX. At this time the real is marked active.

Connections from the outside that get forwarded to the "broken" real are looped back to the VIP in the same fashion, so X% of your incoming connections go through the ADX twice (and X% of *those* get sent through a third time, and so on...) and if you're doing sticky, those connections will loop endlessly between the real and the VIP until the TTL expires.

So, other than pointing out the obvious brokenness of the lack of anti-spoofing filters on the incoming VIPs, I thought of some questions on how to prevent this:

1. Is there an easy way under Linux/BSD to prevent this type of one-armed routing, where a packet is forwarded out the same interface it is received on? I can't think of many cases where this is desirable behavior.

2. On the ADX, is there a knob I'm missing to prevent packets with source IPs local to the box arriving at VIPs from external hosts?*

Thanks,

-Chris

*I realize that symmetric setups could complicate this - the source IP could be on a backup LB, not the LB receiving the incoming connection. A way to negotiate these for filtering would be nice as well.