[f-nsp] Multiple VIPs with Different Subnets

[Lazuardi Nasution]

Hi Jack,

Is this route-map can be applied to Mangement IPs ?

Currently, I have put two default gateway with different distance for
both subnet. For avoiding the access to non default gateway VIP, I
have put different set of DNS for each VIP. This way can push the
clients to always use VIP A if the DNS come to VIP A (lower distance
subnet) by using VIP A IP for all DNS records bound with VIP A and
vice versa. If DNS query is coming to VIP B (higher distance subnet),
the reply will never acknowledged by the clients since it will be
routed through the VIP A. This way can make only VIP A will do the
good reply as long the ISP A link is good.

Best regards,

On Sun, Dec 13, 2009 at 8:18 AM, Jack Stewart <jstewart at caltech.edu> wrote:
>
> Hi Lazuardi,
>
> You are correct in saying that you need VIP's on both subnets. Only a VIP
> subnet A can use ISP A and only a VIP on Subnet B can use ISP B.
>
> With ISP A as your default route, VIPs on Subnet A work just fine. The load
> balancer sees the destination address of the client and sends it to the
> default route. But when the real server sends return traffic for a traffic
> that came in from VIP B, the routing table tells it to go to the  gateway
> for ISP A. Somewhere along the path the return traffic will get dropped into
> the bit bucket. A route-map can fix this.
>
> A route-map with the "ip policy route-map" command applies a set of ACL's
> and route related actions to traffic before the load balance looks at the
> routing table. Route-maps have precedence. In an earlier example below, I
> took advantage of using an extended ACL in order to force the return traffic
> for a VIP on subnet B to go out through the Gateway B. The load balancer
> changes the source address of the real server to the VIP, my route map saw
> that the source address of the VIP was on subnet B, and so it set the next
> hop to Gateway B. The destination address and default gateway did not matter
> for this traffic.
>
> This is a pretty terse description, let me know if it is along the right
> track and we can get into details off line if needed.
>
> The web is a can of worms. Is the content dynamic or static? Is the traffic
> encrypted or is it not encrypted? There are a lot of different tricks. I
> would start with the basics - don't worry about it unless it is a problem or
> you know will be a problem. For the most part the client will cache the
> destination IP for a period of time and most web content is static. Also,
> are we talking encrypted or unencrypted traffic? If it is encrypted you can
> go off of the ssl session id. If this is really a problem, I'll put on my
> sysadmin thinking cap but I'll need to know O/S, http. For example, Apache
> 2.2 has a mod_header module so that you can set the particular server. This
> only applies to unencrypted traffic as the load balancer can't decrypt it.
>
> Lazuardi Nasution wrote:
>>
>> Hi Jack,
>>
>> I'm interested with route-maps. Can you tell me more about that ?
>> Currently I just using static route for two default gateway with
>> different distance.
>>
>> My requirements is simple, just to do server load balancing no matter
>> where the traffic come in. Since I have two ISPs which give me two
>> subnets, so I think I must have two VIPs on different subnets too.  I
>> need Cookie Switching since some of clients are behind the proxy or
>> NAT. I cannot use Source NAT since I have to preserve clients source
>> IP for logging purpose on the real server. Since I don't have to do
>> the link load balancing, I do static routing for default gateway, so
>> only single link will be bidirectional on the same time. But I don't
>> have any idea how to make those VIPs share the session table or at
>> least threat the request not as first request if there is related
>> cookie inside (ex. ServerID). If I can do that I'm sure I can switch
>> the request to the same real server no matter which VIP the request
>> come in.
>>
>> Any idea ?
>>
>> Best regards,
>>
>> On Fri, Dec 11, 2009 at 4:19 AM, Jack Stewart <jstewart at caltech.edu>
>> wrote:
>>>
>>> Hi Lazuardi,
>>>
>>> I'm running the routing code. To the best of my knowledge, route-maps
>>> apply
>>> only to the routing code.
>>>
>>> route-maps allow you to set the gateway by source address, destination
>>> address, or port #. Static routes allow you to set gateway by
>>> destination.
>>> The VIP is the source address of the outgoing traffic. So in the example
>>> below, gateway for a VIP is based on its address (and not the destination
>>> client). This was a really hard concept for me to wrap my head around.
>>>
>>> It isn't clear to me that your case is the same. My setup is very
>>> atypical.
>>> I had a lot of trouble debugging it (traffic would leave the client but
>>> never come back to it).  You might be able to get at it by looking at the
>>> interface traffic of the gateways.
>>>
>>> Let me know the solution you come up with, I'm curious.
>>>
>>> ---Jack
>>>
>>> Lazuardi Nasution wrote:
>>>>
>>>> Hi Jack,
>>>>
>>>> I think this solution is for Switch Code since with Router Code I can
>>>> have many Management IP even with different subnets. The default
>>>> gateway can be specified statically on the routing table or by using
>>>> routing protocol from the routers.
>>>>
>>>> Best regards,
>>>>
>>>> On Fri, Dec 11, 2009 at 2:25 AM, Jack Stewart <jstewart at caltech.edu>
>>>> wrote:
>>>>>
>>>>> Hi Lazuardi,
>>>>>
>>>>> I ran into similar issues - this is what ultimately work in my
>>>>> environment.
>>>>> It may not be the same but hopefully there are some takeaways. Please
>>>>> let
>>>>> me
>>>>> know how much of this makes sense - that feedback will be helpful with
>>>>> my
>>>>> documentation.
>>>>>
>>>>> First, DNS is special and the following is generic.
>>>>>
>>>>> You can only have one management IP and one default route. The
>>>>> management
>>>>> IP
>>>>> should live on the same subnet that has the default route. The first
>>>>> public
>>>>> subnet with the management IP & default route (Pub_Subnet_1) was not a
>>>>> problem.
>>>>>
>>>>> All of my real servers are on a different private subnet that the two
>>>>> public
>>>>> subnets and they all have the Load Balancer defined as their default
>>>>> gateway.
>>>>>
>>>>> To get subnet 2 (pub_subnet_2), I needed to define a router interface
>>>>> for
>>>>> that subnetwork (ve2) and policy routing/route-maps. The route-maps are
>>>>> for
>>>>> making sure that the return traffic goes out via the same gateway that
>>>>> it
>>>>> came in for non directly attached subnets. The way the mapping works
>>>>> for
>>>>> me
>>>>> in the configuration is:
>>>>>
>>>>> !
>>>>> ip access-list extended match_pub_subnet_2
>>>>>  permit ip match_pubsub2/24 any
>>>>> !
>>>>> route-map more_default_routes permit 10
>>>>>  match ip address match_subnet_2
>>>>>  set ip next-hop subnet_2_gateway
>>>>> !
>>>>> ip policy prefer-direct-route
>>>>> ip policy route-map more_default_routes
>>>>>
>>>>> In route-maps, the 'permit #' is just the precedence order. You can add
>>>>> additional entries to a route-map. Route-maps are processed before
>>>>> static
>>>>> routes.
>>>>>
>>>>> Lastly, I defined an outside NAT policy on Public_Subnet_1 for traffic
>>>>> originating private subnet traffic (i.e. directly attached servers).
>>>>> I'm
>>>>> not
>>>>> 100% sure this is a requirement but it helps with traceroute, etc.
>>>>>
>>>>> In my case it was necessary to add VRRP but that is because I've more
>>>>> than
>>>>> one box and it isn't clear you need that.
>>>>>
>>>>> Once this was done, everything worked nicely from outside to inside.
>>>>>
>>>>> This is a global static approach. Most people seem to route-maps to
>>>>> filter
>>>>> routing protocols, but I'm not allowed to exchange LB routing protocols
>>>>> with
>>>>> our routers by policy.
>>>>>
>>>>> For VIPs and real servers on the same private subnet, I found that
>>>>> either
>>>>> DSR or source-nat with ACL's works well. If you are using DSR with
>>>>> Linux
>>>>> (it
>>>>> seems to apply to other 2.6 kernels), you'll probably want to look at
>>>>> the
>>>>> brocade wiki).
>>>>>
>>>>> With DNS, source-nat with ACL's is probably the simplest and easier way
>>>>> to
>>>>> go.
>>>>>
>>>>> ---Jack
>>>>>
>>>>> Lazuardi Nasution wrote:
>>>>>>
>>>>>> Hi Jack,
>>>>>>
>>>>>> Yes, there is different router per subnet and I have done the static
>>>>>> routing for that. VIP1 is in the same subnet with Management IP and
>>>>>> the Router1 is connected to eth1, so I just simply put Management IP
>>>>>> on eth1. Since Router2 is connected to eth2, should I do something on
>>>>>> eth2, ex. put another management IP on the eth2 which is in the same
>>>>>> subnet with VIP2 ? The other ethernet ports are for Real Server so I
>>>>>> have give ve1 for those ports.
>>>>>>
>>>>>> There is another weird problem. I have made DNS binding from VIP1 and
>>>>>> RE1 and I have put ve1 IP in the same subnet with RE1. RE1 default
>>>>>> gateway is ve1 IP. I can query the DNS through VIP1 but RE1 cannot do
>>>>>> traceroute to the Internet, stuck on the ServerIron. What's happen
>>>>>> here ?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> On Thu, Dec 10, 2009 at 3:41 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Lazuardi,
>>>>>>>
>>>>>>> Yeah! A question that might be up my alley. I've done this however I
>>>>>>> need
>>>>>>> some more details.
>>>>>>>
>>>>>>> Do these VIPs need different "static" default gateways on a per
>>>>>>> subnet
>>>>>>> basis? It's possible with the routing code and I can send out the
>>>>>>> details
>>>>>>> if
>>>>>>> you are interested.
>>>>>>>
>>>>>>> Otherwise the main trick with subnet A to subnet B traffic is to make
>>>>>>> sure
>>>>>>> that the return traffic goes though the load balancer. The client &
>>>>>>> server
>>>>>>> need to see the Load Balancer as the gateway between subnet A &
>>>>>>> subnet
>>>>>>> B.
>>>>>>> DSR and source NAT are also options.
>>>>>>>
>>>>>>> So more details, please. Depending on what you need to do it might
>>>>>>> help
>>>>>>> knock out some of my documentation.
>>>>>>>
>>>>>>> ---Jack
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Is it possible to have multiple VIPs with different Subnets on
>>>>>>>> ServerIron 4G or ServerIron ADX1000 ? How can I do that ? I'm using
>>>>>>>> router code of firmware.
>>>>>>>>
>