[f-nsp] Multiple VIPs with Different Subnets

Mon Dec 14 15:18:30 EST 2009

ICMP to the VIP won't work if the VIP uses a route-map, even if it's
included in the ACL.  It's just a limitation of the ServerIron.  It should
continue to work for VIPs that aren't affected by the route-map and are
following the default and/or dynamic routes.  

Jared

-----Original Message-----
From: foundry-nsp-bounces at puck.nether.net
[mailto:foundry-nsp-bounces at puck.nether.net] On Behalf Of Jack Stewart
Sent: Monday, December 14, 2009 12:55 PM
To: Lazuardi Nasution
Cc: foundry-nsp at puck.nether.net
Subject: Re: [f-nsp] Multiple VIPs with Different Subnets

Congratulations! In sounds like you are beginning to get route-maps. 
This is no small think. Every book that I've read so far says 
"route-maps are beyond the scope of this book but here are some neat tricks"

Hmm. I'm going to have to lab both of these tomorrow to be 100% sure.

ICMP should be pretty straight forward, you should just need to add an 
ACL to your route map to premit ICMP (all you are letting through is IP).

Cookie Switching looks interesting and it is on my to do list to look at 
- I'll have to consult with my web guru. My guess is that the cookie is 
getting used by the switch to tell it which which to go to. It looks 
like cookie switching is trumping route maps on a first match basis.

You've probably already thought of this but what about forcing this data 
to be ssl and then using session-id-switch for future connections? It is 
a unique client/server keypair. If the customer you are worried about 
comes from the big NAT/Proxy box, you can force their connections to be 
written into SSL (mod_rewrite on the server)

Just a thought.

Ciao,

---Jack

Lazuardi Nasution wrote:
> Hi Jack,
> 
> I have implemented route-maps. It is OK until I do cookie switching.
> When I put cookie-switching, the second VIP never response the HTTP
> request. The connection is OK but no response. If I remove the
> cookie-switching, the traffic is normal again. I need the
> cookie-switching for handling the clients behind big NAT/Proxy and
> make sure that the request switched to the same server on the event of
> link state changes.
> 
> Beside that, it seem that route-map cannot work for ICMP. I still
> cannot Ping to second VIP.
> 
> Best regards,
> 
> On Mon, Dec 14, 2009 at 6:57 AM, Jack Stewart <jstewart at caltech.edu>
wrote:
>> Hi Lazuardi,
>>
>> Setting up a DNS response strategy seems like a good option - the idea is
>> sound and it is the foundation for GSLB for many vendors. I've read about
>> GSLB but never implemented it.
>>
>> I applied route-maps to the management IP subnet for giggles and removed
the
>> default route. It works but I suspect it isn't a best practice. A
route-map
>> is more like pre-routing table. If you get a match in the route-map, then
>> the specific action is applied. Otherwise, it goes to the routing table.
>> Another way of thinking about it is that route-maps (or policy routing)
have
>> the better metric but you still have your default and static routes.
>>
>> Your setup isn't something that I've done but as long as it works, stick
>> with it. In my situation I have two class C subnets that are part of a
>> larger class B and default routes couldn't do the job.
>>
>> Route maps are pretty cool and now that I understand them, I'm currently
>> exploring PBR as a mechanism for handling Email flows differently based
on
>> source. Load balancing using DNS is way down the road for me.
>>
>> Cheers,
>>
>> ---Jack
>>
>>
>> Lazuardi Nasution wrote:
>>> Hi Jack,
>>>
>>> Is this route-map can be applied to Mangement IPs ?
>>>
>>> Currently, I have put two default gateway with different distance for
>>> both subnet. For avoiding the access to non default gateway VIP, I
>>> have put different set of DNS for each VIP. This way can push the
>>> clients to always use VIP A if the DNS come to VIP A (lower distance
>>> subnet) by using VIP A IP for all DNS records bound with VIP A and
>>> vice versa. If DNS query is coming to VIP B (higher distance subnet),
>>> the reply will never acknowledged by the clients since it will be
>>> routed through the VIP A. This way can make only VIP A will do the
>>> good reply as long the ISP A link is good.
>>>
>>> Best regards,
>>>
>>> On Sun, Dec 13, 2009 at 8:18 AM, Jack Stewart <jstewart at caltech.edu>
>>> wrote:
>>>> Hi Lazuardi,
>>>>
>>>> You are correct in saying that you need VIP's on both subnets. Only a
VIP
>>>> subnet A can use ISP A and only a VIP on Subnet B can use ISP B.
>>>>
>>>> With ISP A as your default route, VIPs on Subnet A work just fine. The
>>>> load
>>>> balancer sees the destination address of the client and sends it to the
>>>> default route. But when the real server sends return traffic for a
>>>> traffic
>>>> that came in from VIP B, the routing table tells it to go to the
gateway
>>>> for ISP A. Somewhere along the path the return traffic will get dropped
>>>> into
>>>> the bit bucket. A route-map can fix this.
>>>>
>>>> A route-map with the "ip policy route-map" command applies a set of
ACL's
>>>> and route related actions to traffic before the load balance looks at
the
>>>> routing table. Route-maps have precedence. In an earlier example below,
I
>>>> took advantage of using an extended ACL in order to force the return
>>>> traffic
>>>> for a VIP on subnet B to go out through the Gateway B. The load
balancer
>>>> changes the source address of the real server to the VIP, my route map
>>>> saw
>>>> that the source address of the VIP was on subnet B, and so it set the
>>>> next
>>>> hop to Gateway B. The destination address and default gateway did not
>>>> matter
>>>> for this traffic.
>>>>
>>>> This is a pretty terse description, let me know if it is along the
right
>>>> track and we can get into details off line if needed.
>>>>
>>>> The web is a can of worms. Is the content dynamic or static? Is the
>>>> traffic
>>>> encrypted or is it not encrypted? There are a lot of different tricks.
I
>>>> would start with the basics - don't worry about it unless it is a
problem
>>>> or
>>>> you know will be a problem. For the most part the client will cache the
>>>> destination IP for a period of time and most web content is static.
Also,
>>>> are we talking encrypted or unencrypted traffic? If it is encrypted you
>>>> can
>>>> go off of the ssl session id. If this is really a problem, I'll put on
my
>>>> sysadmin thinking cap but I'll need to know O/S, http. For example,
>>>> Apache
>>>> 2.2 has a mod_header module so that you can set the particular server.
>>>> This
>>>> only applies to unencrypted traffic as the load balancer can't decrypt
>>>> it.
>>>>
>>>> Lazuardi Nasution wrote:
>>>>> Hi Jack,
>>>>>
>>>>> I'm interested with route-maps. Can you tell me more about that ?
>>>>> Currently I just using static route for two default gateway with
>>>>> different distance.
>>>>>
>>>>> My requirements is simple, just to do server load balancing no matter
>>>>> where the traffic come in. Since I have two ISPs which give me two
>>>>> subnets, so I think I must have two VIPs on different subnets too.  I
>>>>> need Cookie Switching since some of clients are behind the proxy or
>>>>> NAT. I cannot use Source NAT since I have to preserve clients source
>>>>> IP for logging purpose on the real server. Since I don't have to do
>>>>> the link load balancing, I do static routing for default gateway, so
>>>>> only single link will be bidirectional on the same time. But I don't
>>>>> have any idea how to make those VIPs share the session table or at
>>>>> least threat the request not as first request if there is related
>>>>> cookie inside (ex. ServerID). If I can do that I'm sure I can switch
>>>>> the request to the same real server no matter which VIP the request
>>>>> come in.
>>>>>
>>>>> Any idea ?
>>>>>
>>>>> Best regards,
>>>>>
>>>>> On Fri, Dec 11, 2009 at 4:19 AM, Jack Stewart <jstewart at caltech.edu>
>>>>> wrote:
>>>>>> Hi Lazuardi,
>>>>>>
>>>>>> I'm running the routing code. To the best of my knowledge, route-maps
>>>>>> apply
>>>>>> only to the routing code.
>>>>>>
>>>>>> route-maps allow you to set the gateway by source address,
destination
>>>>>> address, or port #. Static routes allow you to set gateway by
>>>>>> destination.
>>>>>> The VIP is the source address of the outgoing traffic. So in the
>>>>>> example
>>>>>> below, gateway for a VIP is based on its address (and not the
>>>>>> destination
>>>>>> client). This was a really hard concept for me to wrap my head
around.
>>>>>>
>>>>>> It isn't clear to me that your case is the same. My setup is very
>>>>>> atypical.
>>>>>> I had a lot of trouble debugging it (traffic would leave the client
but
>>>>>> never come back to it).  You might be able to get at it by looking at
>>>>>> the
>>>>>> interface traffic of the gateways.
>>>>>>
>>>>>> Let me know the solution you come up with, I'm curious.
>>>>>>
>>>>>> ---Jack
>>>>>>
>>>>>> Lazuardi Nasution wrote:
>>>>>>> Hi Jack,
>>>>>>>
>>>>>>> I think this solution is for Switch Code since with Router Code I
can
>>>>>>> have many Management IP even with different subnets. The default
>>>>>>> gateway can be specified statically on the routing table or by using
>>>>>>> routing protocol from the routers.
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> On Fri, Dec 11, 2009 at 2:25 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>>> wrote:
>>>>>>>> Hi Lazuardi,
>>>>>>>>
>>>>>>>> I ran into similar issues - this is what ultimately work in my
>>>>>>>> environment.
>>>>>>>> It may not be the same but hopefully there are some takeaways.
Please
>>>>>>>> let
>>>>>>>> me
>>>>>>>> know how much of this makes sense - that feedback will be helpful
>>>>>>>> with
>>>>>>>> my
>>>>>>>> documentation.
>>>>>>>>
>>>>>>>> First, DNS is special and the following is generic.
>>>>>>>>
>>>>>>>> You can only have one management IP and one default route. The
>>>>>>>> management
>>>>>>>> IP
>>>>>>>> should live on the same subnet that has the default route. The
first
>>>>>>>> public
>>>>>>>> subnet with the management IP & default route (Pub_Subnet_1) was
not
>>>>>>>> a
>>>>>>>> problem.
>>>>>>>>
>>>>>>>> All of my real servers are on a different private subnet that the
two
>>>>>>>> public
>>>>>>>> subnets and they all have the Load Balancer defined as their
default
>>>>>>>> gateway.
>>>>>>>>
>>>>>>>> To get subnet 2 (pub_subnet_2), I needed to define a router
interface
>>>>>>>> for
>>>>>>>> that subnetwork (ve2) and policy routing/route-maps. The route-maps
>>>>>>>> are
>>>>>>>> for
>>>>>>>> making sure that the return traffic goes out via the same gateway
>>>>>>>> that
>>>>>>>> it
>>>>>>>> came in for non directly attached subnets. The way the mapping
works
>>>>>>>> for
>>>>>>>> me
>>>>>>>> in the configuration is:
>>>>>>>>
>>>>>>>> !
>>>>>>>> ip access-list extended match_pub_subnet_2
>>>>>>>>  permit ip match_pubsub2/24 any
>>>>>>>> !
>>>>>>>> route-map more_default_routes permit 10
>>>>>>>>  match ip address match_subnet_2
>>>>>>>>  set ip next-hop subnet_2_gateway
>>>>>>>> !
>>>>>>>> ip policy prefer-direct-route
>>>>>>>> ip policy route-map more_default_routes
>>>>>>>>
>>>>>>>> In route-maps, the 'permit #' is just the precedence order. You can
>>>>>>>> add
>>>>>>>> additional entries to a route-map. Route-maps are processed before
>>>>>>>> static
>>>>>>>> routes.
>>>>>>>>
>>>>>>>> Lastly, I defined an outside NAT policy on Public_Subnet_1 for
>>>>>>>> traffic
>>>>>>>> originating private subnet traffic (i.e. directly attached
servers).
>>>>>>>> I'm
>>>>>>>> not
>>>>>>>> 100% sure this is a requirement but it helps with traceroute, etc.
>>>>>>>>
>>>>>>>> In my case it was necessary to add VRRP but that is because I've
more
>>>>>>>> than
>>>>>>>> one box and it isn't clear you need that.
>>>>>>>>
>>>>>>>> Once this was done, everything worked nicely from outside to
inside.
>>>>>>>>
>>>>>>>> This is a global static approach. Most people seem to route-maps to
>>>>>>>> filter
>>>>>>>> routing protocols, but I'm not allowed to exchange LB routing
>>>>>>>> protocols
>>>>>>>> with
>>>>>>>> our routers by policy.
>>>>>>>>
>>>>>>>> For VIPs and real servers on the same private subnet, I found that
>>>>>>>> either
>>>>>>>> DSR or source-nat with ACL's works well. If you are using DSR with
>>>>>>>> Linux
>>>>>>>> (it
>>>>>>>> seems to apply to other 2.6 kernels), you'll probably want to look
at
>>>>>>>> the
>>>>>>>> brocade wiki).
>>>>>>>>
>>>>>>>> With DNS, source-nat with ACL's is probably the simplest and easier
>>>>>>>> way
>>>>>>>> to
>>>>>>>> go.
>>>>>>>>
>>>>>>>> ---Jack
>>>>>>>>
>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>> Hi Jack,
>>>>>>>>>
>>>>>>>>> Yes, there is different router per subnet and I have done the
static
>>>>>>>>> routing for that. VIP1 is in the same subnet with Management IP
and
>>>>>>>>> the Router1 is connected to eth1, so I just simply put Management
IP
>>>>>>>>> on eth1. Since Router2 is connected to eth2, should I do something
>>>>>>>>> on
>>>>>>>>> eth2, ex. put another management IP on the eth2 which is in the
same
>>>>>>>>> subnet with VIP2 ? The other ethernet ports are for Real Server so
I
>>>>>>>>> have give ve1 for those ports.
>>>>>>>>>
>>>>>>>>> There is another weird problem. I have made DNS binding from VIP1
>>>>>>>>> and
>>>>>>>>> RE1 and I have put ve1 IP in the same subnet with RE1. RE1 default
>>>>>>>>> gateway is ve1 IP. I can query the DNS through VIP1 but RE1 cannot
>>>>>>>>> do
>>>>>>>>> traceroute to the Internet, stuck on the ServerIron. What's happen
>>>>>>>>> here ?
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> On Thu, Dec 10, 2009 at 3:41 AM, Jack Stewart
<jstewart at caltech.edu>
>>>>>>>>> wrote:
>>>>>>>>>> Hi Lazuardi,
>>>>>>>>>>
>>>>>>>>>> Yeah! A question that might be up my alley. I've done this
however
>>>>>>>>>> I
>>>>>>>>>> need
>>>>>>>>>> some more details.
>>>>>>>>>>
>>>>>>>>>> Do these VIPs need different "static" default gateways on a per
>>>>>>>>>> subnet
>>>>>>>>>> basis? It's possible with the routing code and I can send out the
>>>>>>>>>> details
>>>>>>>>>> if
>>>>>>>>>> you are interested.
>>>>>>>>>>
>>>>>>>>>> Otherwise the main trick with subnet A to subnet B traffic is to
>>>>>>>>>> make
>>>>>>>>>> sure
>>>>>>>>>> that the return traffic goes though the load balancer. The client
&
>>>>>>>>>> server
>>>>>>>>>> need to see the Load Balancer as the gateway between subnet A &
>>>>>>>>>> subnet
>>>>>>>>>> B.
>>>>>>>>>> DSR and source NAT are also options.
>>>>>>>>>>
>>>>>>>>>> So more details, please. Depending on what you need to do it
might
>>>>>>>>>> help
>>>>>>>>>> knock out some of my documentation.
>>>>>>>>>>
>>>>>>>>>> ---Jack
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Is it possible to have multiple VIPs with different Subnets on
>>>>>>>>>>> ServerIron 4G or ServerIron ADX1000 ? How can I do that ? I'm
>>>>>>>>>>> using
>>>>>>>>>>> router code of firmware.
>>>>>>>>>>>
>> --
>> Jack Stewart
>> Academic Computing Services, IMSS,
>> California Institute of Technology
>> jstewart at caltech.edu
>> 626-395-4690 office / 626-437-6035 cell

-- 
Jack Stewart
Academic Computing Services, IMSS,
California Institute of Technology
jstewart at caltech.edu
626-395-4690 office / 626-437-6035 cell
_______________________________________________
foundry-nsp mailing list
foundry-nsp at puck.nether.net
http://puck.nether.net/mailman/listinfo/foundry-nsp