[f-nsp] Multiple VIPs with Different Subnets

[Lazuardi Nasution]

Hi Jack,

Route-map cannot work with Management IP too. It seem that Management
IP still use route table than PBR.

With Wireshark, I can see that VIP2 never reply the SYN request if the
cookie-switching is enabled, but this SYN is not rejected. DNS request
to VIP2 is normal, so do the HTTP request if the cookie-switching is
disabled. On the rconsole I can see on session table that the
destination of  HTTP session on VIP2 is always "n/a" when the
cookie-switching is enabled. It seem that VIP2 is "confused" to
redirect the HTTP request on that condition. On the test, I have made
sure that all bound real servers are online.

Regarding too SSL, we have plan to use it but not for all traffic. We
still need non-SSL traffic for faster page loading, especially for
image files because they can be cached on client side.

On Tue, Dec 15, 2009 at 2:55 AM, Jack Stewart <jstewart at caltech.edu> wrote:
>
> Congratulations! In sounds like you are beginning to get route-maps. This is
> no small think. Every book that I've read so far says "route-maps are beyond
> the scope of this book but here are some neat tricks"
>
> Hmm. I'm going to have to lab both of these tomorrow to be 100% sure.
>
> ICMP should be pretty straight forward, you should just need to add an ACL
> to your route map to premit ICMP (all you are letting through is IP).
>
> Cookie Switching looks interesting and it is on my to do list to look at -
> I'll have to consult with my web guru. My guess is that the cookie is
> getting used by the switch to tell it which which to go to. It looks like
> cookie switching is trumping route maps on a first match basis.
>
> You've probably already thought of this but what about forcing this data to
> be ssl and then using session-id-switch for future connections? It is a
> unique client/server keypair. If the customer you are worried about comes
> from the big NAT/Proxy box, you can force their connections to be written
> into SSL (mod_rewrite on the server)
>
> Just a thought.
>
> Ciao,
>
> ---Jack
>
>
> Lazuardi Nasution wrote:
>>
>> Hi Jack,
>>
>> I have implemented route-maps. It is OK until I do cookie switching.
>> When I put cookie-switching, the second VIP never response the HTTP
>> request. The connection is OK but no response. If I remove the
>> cookie-switching, the traffic is normal again. I need the
>> cookie-switching for handling the clients behind big NAT/Proxy and
>> make sure that the request switched to the same server on the event of
>> link state changes.
>>
>> Beside that, it seem that route-map cannot work for ICMP. I still
>> cannot Ping to second VIP.
>>
>> Best regards,
>>
>> On Mon, Dec 14, 2009 at 6:57 AM, Jack Stewart <jstewart at caltech.edu>
>> wrote:
>>>
>>> Hi Lazuardi,
>>>
>>> Setting up a DNS response strategy seems like a good option - the idea is
>>> sound and it is the foundation for GSLB for many vendors. I've read about
>>> GSLB but never implemented it.
>>>
>>> I applied route-maps to the management IP subnet for giggles and removed
>>> the
>>> default route. It works but I suspect it isn't a best practice. A
>>> route-map
>>> is more like pre-routing table. If you get a match in the route-map, then
>>> the specific action is applied. Otherwise, it goes to the routing table.
>>> Another way of thinking about it is that route-maps (or policy routing)
>>> have
>>> the better metric but you still have your default and static routes.
>>>
>>> Your setup isn't something that I've done but as long as it works, stick
>>> with it. In my situation I have two class C subnets that are part of a
>>> larger class B and default routes couldn't do the job.
>>>
>>> Route maps are pretty cool and now that I understand them, I'm currently
>>> exploring PBR as a mechanism for handling Email flows differently based
>>> on
>>> source. Load balancing using DNS is way down the road for me.
>>>
>>> Cheers,
>>>
>>> ---Jack
>>>
>>>
>>> Lazuardi Nasution wrote:
>>>>
>>>> Hi Jack,
>>>>
>>>> Is this route-map can be applied to Mangement IPs ?
>>>>
>>>> Currently, I have put two default gateway with different distance for
>>>> both subnet. For avoiding the access to non default gateway VIP, I
>>>> have put different set of DNS for each VIP. This way can push the
>>>> clients to always use VIP A if the DNS come to VIP A (lower distance
>>>> subnet) by using VIP A IP for all DNS records bound with VIP A and
>>>> vice versa. If DNS query is coming to VIP B (higher distance subnet),
>>>> the reply will never acknowledged by the clients since it will be
>>>> routed through the VIP A. This way can make only VIP A will do the
>>>> good reply as long the ISP A link is good.
>>>>
>>>> Best regards,
>>>>
>>>> On Sun, Dec 13, 2009 at 8:18 AM, Jack Stewart <jstewart at caltech.edu>
>>>> wrote:
>>>>>
>>>>> Hi Lazuardi,
>>>>>
>>>>> You are correct in saying that you need VIP's on both subnets. Only a
>>>>> VIP
>>>>> subnet A can use ISP A and only a VIP on Subnet B can use ISP B.
>>>>>
>>>>> With ISP A as your default route, VIPs on Subnet A work just fine. The
>>>>> load
>>>>> balancer sees the destination address of the client and sends it to the
>>>>> default route. But when the real server sends return traffic for a
>>>>> traffic
>>>>> that came in from VIP B, the routing table tells it to go to the
>>>>>  gateway
>>>>> for ISP A. Somewhere along the path the return traffic will get dropped
>>>>> into
>>>>> the bit bucket. A route-map can fix this.
>>>>>
>>>>> A route-map with the "ip policy route-map" command applies a set of
>>>>> ACL's
>>>>> and route related actions to traffic before the load balance looks at
>>>>> the
>>>>> routing table. Route-maps have precedence. In an earlier example below,
>>>>> I
>>>>> took advantage of using an extended ACL in order to force the return
>>>>> traffic
>>>>> for a VIP on subnet B to go out through the Gateway B. The load
>>>>> balancer
>>>>> changes the source address of the real server to the VIP, my route map
>>>>> saw
>>>>> that the source address of the VIP was on subnet B, and so it set the
>>>>> next
>>>>> hop to Gateway B. The destination address and default gateway did not
>>>>> matter
>>>>> for this traffic.
>>>>>
>>>>> This is a pretty terse description, let me know if it is along the
>>>>> right
>>>>> track and we can get into details off line if needed.
>>>>>
>>>>> The web is a can of worms. Is the content dynamic or static? Is the
>>>>> traffic
>>>>> encrypted or is it not encrypted? There are a lot of different tricks.
>>>>> I
>>>>> would start with the basics - don't worry about it unless it is a
>>>>> problem
>>>>> or
>>>>> you know will be a problem. For the most part the client will cache the
>>>>> destination IP for a period of time and most web content is static.
>>>>> Also,
>>>>> are we talking encrypted or unencrypted traffic? If it is encrypted you
>>>>> can
>>>>> go off of the ssl session id. If this is really a problem, I'll put on
>>>>> my
>>>>> sysadmin thinking cap but I'll need to know O/S, http. For example,
>>>>> Apache
>>>>> 2.2 has a mod_header module so that you can set the particular server.
>>>>> This
>>>>> only applies to unencrypted traffic as the load balancer can't decrypt
>>>>> it.
>>>>>
>>>>> Lazuardi Nasution wrote:
>>>>>>
>>>>>> Hi Jack,
>>>>>>
>>>>>> I'm interested with route-maps. Can you tell me more about that ?
>>>>>> Currently I just using static route for two default gateway with
>>>>>> different distance.
>>>>>>
>>>>>> My requirements is simple, just to do server load balancing no matter
>>>>>> where the traffic come in. Since I have two ISPs which give me two
>>>>>> subnets, so I think I must have two VIPs on different subnets too.  I
>>>>>> need Cookie Switching since some of clients are behind the proxy or
>>>>>> NAT. I cannot use Source NAT since I have to preserve clients source
>>>>>> IP for logging purpose on the real server. Since I don't have to do
>>>>>> the link load balancing, I do static routing for default gateway, so
>>>>>> only single link will be bidirectional on the same time. But I don't
>>>>>> have any idea how to make those VIPs share the session table or at
>>>>>> least threat the request not as first request if there is related
>>>>>> cookie inside (ex. ServerID). If I can do that I'm sure I can switch
>>>>>> the request to the same real server no matter which VIP the request
>>>>>> come in.
>>>>>>
>>>>>> Any idea ?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> On Fri, Dec 11, 2009 at 4:19 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>> wrote:
>>>>>>>
>>>>>>> Hi Lazuardi,
>>>>>>>
>>>>>>> I'm running the routing code. To the best of my knowledge, route-maps
>>>>>>> apply
>>>>>>> only to the routing code.
>>>>>>>
>>>>>>> route-maps allow you to set the gateway by source address,
>>>>>>> destination
>>>>>>> address, or port #. Static routes allow you to set gateway by
>>>>>>> destination.
>>>>>>> The VIP is the source address of the outgoing traffic. So in the
>>>>>>> example
>>>>>>> below, gateway for a VIP is based on its address (and not the
>>>>>>> destination
>>>>>>> client). This was a really hard concept for me to wrap my head
>>>>>>> around.
>>>>>>>
>>>>>>> It isn't clear to me that your case is the same. My setup is very
>>>>>>> atypical.
>>>>>>> I had a lot of trouble debugging it (traffic would leave the client
>>>>>>> but
>>>>>>> never come back to it).  You might be able to get at it by looking at
>>>>>>> the
>>>>>>> interface traffic of the gateways.
>>>>>>>
>>>>>>> Let me know the solution you come up with, I'm curious.
>>>>>>>
>>>>>>> ---Jack
>>>>>>>
>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>
>>>>>>>> Hi Jack,
>>>>>>>>
>>>>>>>> I think this solution is for Switch Code since with Router Code I
>>>>>>>> can
>>>>>>>> have many Management IP even with different subnets. The default
>>>>>>>> gateway can be specified statically on the routing table or by using
>>>>>>>> routing protocol from the routers.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2009 at 2:25 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>> Hi Lazuardi,
>>>>>>>>>
>>>>>>>>> I ran into similar issues - this is what ultimately work in my
>>>>>>>>> environment.
>>>>>>>>> It may not be the same but hopefully there are some takeaways.
>>>>>>>>> Please
>>>>>>>>> let
>>>>>>>>> me
>>>>>>>>> know how much of this makes sense - that feedback will be helpful
>>>>>>>>> with
>>>>>>>>> my
>>>>>>>>> documentation.
>>>>>>>>>
>>>>>>>>> First, DNS is special and the following is generic.
>>>>>>>>>
>>>>>>>>> You can only have one management IP and one default route. The
>>>>>>>>> management
>>>>>>>>> IP
>>>>>>>>> should live on the same subnet that has the default route. The
>>>>>>>>> first
>>>>>>>>> public
>>>>>>>>> subnet with the management IP & default route (Pub_Subnet_1) was
>>>>>>>>> not
>>>>>>>>> a
>>>>>>>>> problem.
>>>>>>>>>
>>>>>>>>> All of my real servers are on a different private subnet that the
>>>>>>>>> two
>>>>>>>>> public
>>>>>>>>> subnets and they all have the Load Balancer defined as their
>>>>>>>>> default
>>>>>>>>> gateway.
>>>>>>>>>
>>>>>>>>> To get subnet 2 (pub_subnet_2), I needed to define a router
>>>>>>>>> interface
>>>>>>>>> for
>>>>>>>>> that subnetwork (ve2) and policy routing/route-maps. The route-maps
>>>>>>>>> are
>>>>>>>>> for
>>>>>>>>> making sure that the return traffic goes out via the same gateway
>>>>>>>>> that
>>>>>>>>> it
>>>>>>>>> came in for non directly attached subnets. The way the mapping
>>>>>>>>> works
>>>>>>>>> for
>>>>>>>>> me
>>>>>>>>> in the configuration is:
>>>>>>>>>
>>>>>>>>> !
>>>>>>>>> ip access-list extended match_pub_subnet_2
>>>>>>>>>  permit ip match_pubsub2/24 any
>>>>>>>>> !
>>>>>>>>> route-map more_default_routes permit 10
>>>>>>>>>  match ip address match_subnet_2
>>>>>>>>>  set ip next-hop subnet_2_gateway
>>>>>>>>> !
>>>>>>>>> ip policy prefer-direct-route
>>>>>>>>> ip policy route-map more_default_routes
>>>>>>>>>
>>>>>>>>> In route-maps, the 'permit #' is just the precedence order. You can
>>>>>>>>> add
>>>>>>>>> additional entries to a route-map. Route-maps are processed before
>>>>>>>>> static
>>>>>>>>> routes.
>>>>>>>>>
>>>>>>>>> Lastly, I defined an outside NAT policy on Public_Subnet_1 for
>>>>>>>>> traffic
>>>>>>>>> originating private subnet traffic (i.e. directly attached
>>>>>>>>> servers).
>>>>>>>>> I'm
>>>>>>>>> not
>>>>>>>>> 100% sure this is a requirement but it helps with traceroute, etc.
>>>>>>>>>
>>>>>>>>> In my case it was necessary to add VRRP but that is because I've
>>>>>>>>> more
>>>>>>>>> than
>>>>>>>>> one box and it isn't clear you need that.
>>>>>>>>>
>>>>>>>>> Once this was done, everything worked nicely from outside to
>>>>>>>>> inside.
>>>>>>>>>
>>>>>>>>> This is a global static approach. Most people seem to route-maps to
>>>>>>>>> filter
>>>>>>>>> routing protocols, but I'm not allowed to exchange LB routing
>>>>>>>>> protocols
>>>>>>>>> with
>>>>>>>>> our routers by policy.
>>>>>>>>>
>>>>>>>>> For VIPs and real servers on the same private subnet, I found that
>>>>>>>>> either
>>>>>>>>> DSR or source-nat with ACL's works well. If you are using DSR with
>>>>>>>>> Linux
>>>>>>>>> (it
>>>>>>>>> seems to apply to other 2.6 kernels), you'll probably want to look
>>>>>>>>> at
>>>>>>>>> the
>>>>>>>>> brocade wiki).
>>>>>>>>>
>>>>>>>>> With DNS, source-nat with ACL's is probably the simplest and easier
>>>>>>>>> way
>>>>>>>>> to
>>>>>>>>> go.
>>>>>>>>>
>>>>>>>>> ---Jack
>>>>>>>>>
>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>
>>>>>>>>>> Hi Jack,
>>>>>>>>>>
>>>>>>>>>> Yes, there is different router per subnet and I have done the
>>>>>>>>>> static
>>>>>>>>>> routing for that. VIP1 is in the same subnet with Management IP
>>>>>>>>>> and
>>>>>>>>>> the Router1 is connected to eth1, so I just simply put Management
>>>>>>>>>> IP
>>>>>>>>>> on eth1. Since Router2 is connected to eth2, should I do something
>>>>>>>>>> on
>>>>>>>>>> eth2, ex. put another management IP on the eth2 which is in the
>>>>>>>>>> same
>>>>>>>>>> subnet with VIP2 ? The other ethernet ports are for Real Server so
>>>>>>>>>> I
>>>>>>>>>> have give ve1 for those ports.
>>>>>>>>>>
>>>>>>>>>> There is another weird problem. I have made DNS binding from VIP1
>>>>>>>>>> and
>>>>>>>>>> RE1 and I have put ve1 IP in the same subnet with RE1. RE1 default
>>>>>>>>>> gateway is ve1 IP. I can query the DNS through VIP1 but RE1 cannot
>>>>>>>>>> do
>>>>>>>>>> traceroute to the Internet, stuck on the ServerIron. What's happen
>>>>>>>>>> here ?
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> On Thu, Dec 10, 2009 at 3:41 AM, Jack Stewart
>>>>>>>>>> <jstewart at caltech.edu>
>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>> Hi Lazuardi,
>>>>>>>>>>>
>>>>>>>>>>> Yeah! A question that might be up my alley. I've done this
>>>>>>>>>>> however
>>>>>>>>>>> I
>>>>>>>>>>> need
>>>>>>>>>>> some more details.
>>>>>>>>>>>
>>>>>>>>>>> Do these VIPs need different "static" default gateways on a per
>>>>>>>>>>> subnet
>>>>>>>>>>> basis? It's possible with the routing code and I can send out the
>>>>>>>>>>> details
>>>>>>>>>>> if
>>>>>>>>>>> you are interested.
>>>>>>>>>>>
>>>>>>>>>>> Otherwise the main trick with subnet A to subnet B traffic is to
>>>>>>>>>>> make
>>>>>>>>>>> sure
>>>>>>>>>>> that the return traffic goes though the load balancer. The client
>>>>>>>>>>> &
>>>>>>>>>>> server
>>>>>>>>>>> need to see the Load Balancer as the gateway between subnet A &
>>>>>>>>>>> subnet
>>>>>>>>>>> B.
>>>>>>>>>>> DSR and source NAT are also options.
>>>>>>>>>>>
>>>>>>>>>>> So more details, please. Depending on what you need to do it
>>>>>>>>>>> might
>>>>>>>>>>> help
>>>>>>>>>>> knock out some of my documentation.
>>>>>>>>>>>
>>>>>>>>>>> ---Jack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>>>
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Is it possible to have multiple VIPs with different Subnets on
>>>>>>>>>>>> ServerIron 4G or ServerIron ADX1000 ? How can I do that ? I'm
>>>>>>>>>>>> using
>>>>>>>>>>>> router code of firmware.
>>>>>>>>>>>>
>>> --
>>> Jack Stewart
>>> Academic Computing Services, IMSS,
>>> California Institute of Technology
>>> jstewart at caltech.edu
>>> 626-395-4690 office / 626-437-6035 cell
>
> --
> Jack Stewart
> Academic Computing Services, IMSS,
> California Institute of Technology
> jstewart at caltech.edu
> 626-395-4690 office / 626-437-6035 cell
>