[f-nsp] Multiple VIPs with Different Subnets

[Lazuardi Nasution]

Hi Jack,

I have implemented route-maps. It is OK until I do cookie switching.
When I put cookie-switching, the second VIP never response the HTTP
request. The connection is OK but no response. If I remove the
cookie-switching, the traffic is normal again. I need the
cookie-switching for handling the clients behind big NAT/Proxy and
make sure that the request switched to the same server on the event of
link state changes.

Beside that, it seem that route-map cannot work for ICMP. I still
cannot Ping to second VIP.

Best regards,

On Mon, Dec 14, 2009 at 6:57 AM, Jack Stewart <jstewart at caltech.edu> wrote:
>
> Hi Lazuardi,
>
> Setting up a DNS response strategy seems like a good option - the idea is
> sound and it is the foundation for GSLB for many vendors. I've read about
> GSLB but never implemented it.
>
> I applied route-maps to the management IP subnet for giggles and removed the
> default route. It works but I suspect it isn't a best practice. A route-map
> is more like pre-routing table. If you get a match in the route-map, then
> the specific action is applied. Otherwise, it goes to the routing table.
> Another way of thinking about it is that route-maps (or policy routing) have
> the better metric but you still have your default and static routes.
>
> Your setup isn't something that I've done but as long as it works, stick
> with it. In my situation I have two class C subnets that are part of a
> larger class B and default routes couldn't do the job.
>
> Route maps are pretty cool and now that I understand them, I'm currently
> exploring PBR as a mechanism for handling Email flows differently based on
> source. Load balancing using DNS is way down the road for me.
>
> Cheers,
>
> ---Jack
>
>
> Lazuardi Nasution wrote:
>>
>> Hi Jack,
>>
>> Is this route-map can be applied to Mangement IPs ?
>>
>> Currently, I have put two default gateway with different distance for
>> both subnet. For avoiding the access to non default gateway VIP, I
>> have put different set of DNS for each VIP. This way can push the
>> clients to always use VIP A if the DNS come to VIP A (lower distance
>> subnet) by using VIP A IP for all DNS records bound with VIP A and
>> vice versa. If DNS query is coming to VIP B (higher distance subnet),
>> the reply will never acknowledged by the clients since it will be
>> routed through the VIP A. This way can make only VIP A will do the
>> good reply as long the ISP A link is good.
>>
>> Best regards,
>>
>> On Sun, Dec 13, 2009 at 8:18 AM, Jack Stewart <jstewart at caltech.edu>
>> wrote:
>>>
>>> Hi Lazuardi,
>>>
>>> You are correct in saying that you need VIP's on both subnets. Only a VIP
>>> subnet A can use ISP A and only a VIP on Subnet B can use ISP B.
>>>
>>> With ISP A as your default route, VIPs on Subnet A work just fine. The
>>> load
>>> balancer sees the destination address of the client and sends it to the
>>> default route. But when the real server sends return traffic for a
>>> traffic
>>> that came in from VIP B, the routing table tells it to go to the  gateway
>>> for ISP A. Somewhere along the path the return traffic will get dropped
>>> into
>>> the bit bucket. A route-map can fix this.
>>>
>>> A route-map with the "ip policy route-map" command applies a set of ACL's
>>> and route related actions to traffic before the load balance looks at the
>>> routing table. Route-maps have precedence. In an earlier example below, I
>>> took advantage of using an extended ACL in order to force the return
>>> traffic
>>> for a VIP on subnet B to go out through the Gateway B. The load balancer
>>> changes the source address of the real server to the VIP, my route map
>>> saw
>>> that the source address of the VIP was on subnet B, and so it set the
>>> next
>>> hop to Gateway B. The destination address and default gateway did not
>>> matter
>>> for this traffic.
>>>
>>> This is a pretty terse description, let me know if it is along the right
>>> track and we can get into details off line if needed.
>>>
>>> The web is a can of worms. Is the content dynamic or static? Is the
>>> traffic
>>> encrypted or is it not encrypted? There are a lot of different tricks. I
>>> would start with the basics - don't worry about it unless it is a problem
>>> or
>>> you know will be a problem. For the most part the client will cache the
>>> destination IP for a period of time and most web content is static. Also,
>>> are we talking encrypted or unencrypted traffic? If it is encrypted you
>>> can
>>> go off of the ssl session id. If this is really a problem, I'll put on my
>>> sysadmin thinking cap but I'll need to know O/S, http. For example,
>>> Apache
>>> 2.2 has a mod_header module so that you can set the particular server.
>>> This
>>> only applies to unencrypted traffic as the load balancer can't decrypt
>>> it.
>>>
>>> Lazuardi Nasution wrote:
>>>>
>>>> Hi Jack,
>>>>
>>>> I'm interested with route-maps. Can you tell me more about that ?
>>>> Currently I just using static route for two default gateway with
>>>> different distance.
>>>>
>>>> My requirements is simple, just to do server load balancing no matter
>>>> where the traffic come in. Since I have two ISPs which give me two
>>>> subnets, so I think I must have two VIPs on different subnets too.  I
>>>> need Cookie Switching since some of clients are behind the proxy or
>>>> NAT. I cannot use Source NAT since I have to preserve clients source
>>>> IP for logging purpose on the real server. Since I don't have to do
>>>> the link load balancing, I do static routing for default gateway, so
>>>> only single link will be bidirectional on the same time. But I don't
>>>> have any idea how to make those VIPs share the session table or at
>>>> least threat the request not as first request if there is related
>>>> cookie inside (ex. ServerID). If I can do that I'm sure I can switch
>>>> the request to the same real server no matter which VIP the request
>>>> come in.
>>>>
>>>> Any idea ?
>>>>
>>>> Best regards,
>>>>
>>>> On Fri, Dec 11, 2009 at 4:19 AM, Jack Stewart <jstewart at caltech.edu>
>>>> wrote:
>>>>>
>>>>> Hi Lazuardi,
>>>>>
>>>>> I'm running the routing code. To the best of my knowledge, route-maps
>>>>> apply
>>>>> only to the routing code.
>>>>>
>>>>> route-maps allow you to set the gateway by source address, destination
>>>>> address, or port #. Static routes allow you to set gateway by
>>>>> destination.
>>>>> The VIP is the source address of the outgoing traffic. So in the
>>>>> example
>>>>> below, gateway for a VIP is based on its address (and not the
>>>>> destination
>>>>> client). This was a really hard concept for me to wrap my head around.
>>>>>
>>>>> It isn't clear to me that your case is the same. My setup is very
>>>>> atypical.
>>>>> I had a lot of trouble debugging it (traffic would leave the client but
>>>>> never come back to it).  You might be able to get at it by looking at
>>>>> the
>>>>> interface traffic of the gateways.
>>>>>
>>>>> Let me know the solution you come up with, I'm curious.
>>>>>
>>>>> ---Jack
>>>>>
>>>>> Lazuardi Nasution wrote:
>>>>>>
>>>>>> Hi Jack,
>>>>>>
>>>>>> I think this solution is for Switch Code since with Router Code I can
>>>>>> have many Management IP even with different subnets. The default
>>>>>> gateway can be specified statically on the routing table or by using
>>>>>> routing protocol from the routers.
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> On Fri, Dec 11, 2009 at 2:25 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Lazuardi,
>>>>>>>
>>>>>>> I ran into similar issues - this is what ultimately work in my
>>>>>>> environment.
>>>>>>> It may not be the same but hopefully there are some takeaways. Please
>>>>>>> let
>>>>>>> me
>>>>>>> know how much of this makes sense - that feedback will be helpful
>>>>>>> with
>>>>>>> my
>>>>>>> documentation.
>>>>>>>
>>>>>>> First, DNS is special and the following is generic.
>>>>>>>
>>>>>>> You can only have one management IP and one default route. The
>>>>>>> management
>>>>>>> IP
>>>>>>> should live on the same subnet that has the default route. The first
>>>>>>> public
>>>>>>> subnet with the management IP & default route (Pub_Subnet_1) was not
>>>>>>> a
>>>>>>> problem.
>>>>>>>
>>>>>>> All of my real servers are on a different private subnet that the two
>>>>>>> public
>>>>>>> subnets and they all have the Load Balancer defined as their default
>>>>>>> gateway.
>>>>>>>
>>>>>>> To get subnet 2 (pub_subnet_2), I needed to define a router interface
>>>>>>> for
>>>>>>> that subnetwork (ve2) and policy routing/route-maps. The route-maps
>>>>>>> are
>>>>>>> for
>>>>>>> making sure that the return traffic goes out via the same gateway
>>>>>>> that
>>>>>>> it
>>>>>>> came in for non directly attached subnets. The way the mapping works
>>>>>>> for
>>>>>>> me
>>>>>>> in the configuration is:
>>>>>>>
>>>>>>> !
>>>>>>> ip access-list extended match_pub_subnet_2
>>>>>>>  permit ip match_pubsub2/24 any
>>>>>>> !
>>>>>>> route-map more_default_routes permit 10
>>>>>>>  match ip address match_subnet_2
>>>>>>>  set ip next-hop subnet_2_gateway
>>>>>>> !
>>>>>>> ip policy prefer-direct-route
>>>>>>> ip policy route-map more_default_routes
>>>>>>>
>>>>>>> In route-maps, the 'permit #' is just the precedence order. You can
>>>>>>> add
>>>>>>> additional entries to a route-map. Route-maps are processed before
>>>>>>> static
>>>>>>> routes.
>>>>>>>
>>>>>>> Lastly, I defined an outside NAT policy on Public_Subnet_1 for
>>>>>>> traffic
>>>>>>> originating private subnet traffic (i.e. directly attached servers).
>>>>>>> I'm
>>>>>>> not
>>>>>>> 100% sure this is a requirement but it helps with traceroute, etc.
>>>>>>>
>>>>>>> In my case it was necessary to add VRRP but that is because I've more
>>>>>>> than
>>>>>>> one box and it isn't clear you need that.
>>>>>>>
>>>>>>> Once this was done, everything worked nicely from outside to inside.
>>>>>>>
>>>>>>> This is a global static approach. Most people seem to route-maps to
>>>>>>> filter
>>>>>>> routing protocols, but I'm not allowed to exchange LB routing
>>>>>>> protocols
>>>>>>> with
>>>>>>> our routers by policy.
>>>>>>>
>>>>>>> For VIPs and real servers on the same private subnet, I found that
>>>>>>> either
>>>>>>> DSR or source-nat with ACL's works well. If you are using DSR with
>>>>>>> Linux
>>>>>>> (it
>>>>>>> seems to apply to other 2.6 kernels), you'll probably want to look at
>>>>>>> the
>>>>>>> brocade wiki).
>>>>>>>
>>>>>>> With DNS, source-nat with ACL's is probably the simplest and easier
>>>>>>> way
>>>>>>> to
>>>>>>> go.
>>>>>>>
>>>>>>> ---Jack
>>>>>>>
>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>
>>>>>>>> Hi Jack,
>>>>>>>>
>>>>>>>> Yes, there is different router per subnet and I have done the static
>>>>>>>> routing for that. VIP1 is in the same subnet with Management IP and
>>>>>>>> the Router1 is connected to eth1, so I just simply put Management IP
>>>>>>>> on eth1. Since Router2 is connected to eth2, should I do something
>>>>>>>> on
>>>>>>>> eth2, ex. put another management IP on the eth2 which is in the same
>>>>>>>> subnet with VIP2 ? The other ethernet ports are for Real Server so I
>>>>>>>> have give ve1 for those ports.
>>>>>>>>
>>>>>>>> There is another weird problem. I have made DNS binding from VIP1
>>>>>>>> and
>>>>>>>> RE1 and I have put ve1 IP in the same subnet with RE1. RE1 default
>>>>>>>> gateway is ve1 IP. I can query the DNS through VIP1 but RE1 cannot
>>>>>>>> do
>>>>>>>> traceroute to the Internet, stuck on the ServerIron. What's happen
>>>>>>>> here ?
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> On Thu, Dec 10, 2009 at 3:41 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Lazuardi,
>>>>>>>>>
>>>>>>>>> Yeah! A question that might be up my alley. I've done this however
>>>>>>>>> I
>>>>>>>>> need
>>>>>>>>> some more details.
>>>>>>>>>
>>>>>>>>> Do these VIPs need different "static" default gateways on a per
>>>>>>>>> subnet
>>>>>>>>> basis? It's possible with the routing code and I can send out the
>>>>>>>>> details
>>>>>>>>> if
>>>>>>>>> you are interested.
>>>>>>>>>
>>>>>>>>> Otherwise the main trick with subnet A to subnet B traffic is to
>>>>>>>>> make
>>>>>>>>> sure
>>>>>>>>> that the return traffic goes though the load balancer. The client &
>>>>>>>>> server
>>>>>>>>> need to see the Load Balancer as the gateway between subnet A &
>>>>>>>>> subnet
>>>>>>>>> B.
>>>>>>>>> DSR and source NAT are also options.
>>>>>>>>>
>>>>>>>>> So more details, please. Depending on what you need to do it might
>>>>>>>>> help
>>>>>>>>> knock out some of my documentation.
>>>>>>>>>
>>>>>>>>> ---Jack
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Is it possible to have multiple VIPs with different Subnets on
>>>>>>>>>> ServerIron 4G or ServerIron ADX1000 ? How can I do that ? I'm
>>>>>>>>>> using
>>>>>>>>>> router code of firmware.
>>>>>>>>>>
>
> --
> Jack Stewart
> Academic Computing Services, IMSS,
> California Institute of Technology
> jstewart at caltech.edu
> 626-395-4690 office / 626-437-6035 cell