[f-nsp] Multiple VIPs with Different Subnets

Mon Dec 14 15:26:27 EST 2009

Hi Jared,

I think you are right regarding to ICMP. I have tried but failed. What
about Management IP. It seem to be failed too.

Best regards,

On Tue, Dec 15, 2009 at 3:18 AM, Jared Valentine <hidden at xmission.com> wrote:
> ICMP to the VIP won't work if the VIP uses a route-map, even if it's
> included in the ACL.  It's just a limitation of the ServerIron.  It should
> continue to work for VIPs that aren't affected by the route-map and are
> following the default and/or dynamic routes.
>
> Jared
>
>
> -----Original Message-----
> From: foundry-nsp-bounces at puck.nether.net
> [mailto:foundry-nsp-bounces at puck.nether.net] On Behalf Of Jack Stewart
> Sent: Monday, December 14, 2009 12:55 PM
> To: Lazuardi Nasution
> Cc: foundry-nsp at puck.nether.net
> Subject: Re: [f-nsp] Multiple VIPs with Different Subnets
>
>
> Congratulations! In sounds like you are beginning to get route-maps.
> This is no small think. Every book that I've read so far says
> "route-maps are beyond the scope of this book but here are some neat tricks"
>
> Hmm. I'm going to have to lab both of these tomorrow to be 100% sure.
>
> ICMP should be pretty straight forward, you should just need to add an
> ACL to your route map to premit ICMP (all you are letting through is IP).
>
> Cookie Switching looks interesting and it is on my to do list to look at
> - I'll have to consult with my web guru. My guess is that the cookie is
> getting used by the switch to tell it which which to go to. It looks
> like cookie switching is trumping route maps on a first match basis.
>
> You've probably already thought of this but what about forcing this data
> to be ssl and then using session-id-switch for future connections? It is
> a unique client/server keypair. If the customer you are worried about
> comes from the big NAT/Proxy box, you can force their connections to be
> written into SSL (mod_rewrite on the server)
>
> Just a thought.
>
> Ciao,
>
> ---Jack
>
>
> Lazuardi Nasution wrote:
>> Hi Jack,
>>
>> I have implemented route-maps. It is OK until I do cookie switching.
>> When I put cookie-switching, the second VIP never response the HTTP
>> request. The connection is OK but no response. If I remove the
>> cookie-switching, the traffic is normal again. I need the
>> cookie-switching for handling the clients behind big NAT/Proxy and
>> make sure that the request switched to the same server on the event of
>> link state changes.
>>
>> Beside that, it seem that route-map cannot work for ICMP. I still
>> cannot Ping to second VIP.
>>
>> Best regards,
>>
>> On Mon, Dec 14, 2009 at 6:57 AM, Jack Stewart <jstewart at caltech.edu>
> wrote:
>>> Hi Lazuardi,
>>>
>>> Setting up a DNS response strategy seems like a good option - the idea is
>>> sound and it is the foundation for GSLB for many vendors. I've read about
>>> GSLB but never implemented it.
>>>
>>> I applied route-maps to the management IP subnet for giggles and removed
> the
>>> default route. It works but I suspect it isn't a best practice. A
> route-map
>>> is more like pre-routing table. If you get a match in the route-map, then
>>> the specific action is applied. Otherwise, it goes to the routing table.
>>> Another way of thinking about it is that route-maps (or policy routing)
> have
>>> the better metric but you still have your default and static routes.
>>>
>>> Your setup isn't something that I've done but as long as it works, stick
>>> with it. In my situation I have two class C subnets that are part of a
>>> larger class B and default routes couldn't do the job.
>>>
>>> Route maps are pretty cool and now that I understand them, I'm currently
>>> exploring PBR as a mechanism for handling Email flows differently based
> on
>>> source. Load balancing using DNS is way down the road for me.
>>>
>>> Cheers,
>>>
>>> ---Jack
>>>
>>>
>>> Lazuardi Nasution wrote:
>>>> Hi Jack,
>>>>
>>>> Is this route-map can be applied to Mangement IPs ?
>>>>
>>>> Currently, I have put two default gateway with different distance for
>>>> both subnet. For avoiding the access to non default gateway VIP, I
>>>> have put different set of DNS for each VIP. This way can push the
>>>> clients to always use VIP A if the DNS come to VIP A (lower distance
>>>> subnet) by using VIP A IP for all DNS records bound with VIP A and
>>>> vice versa. If DNS query is coming to VIP B (higher distance subnet),
>>>> the reply will never acknowledged by the clients since it will be
>>>> routed through the VIP A. This way can make only VIP A will do the
>>>> good reply as long the ISP A link is good.
>>>>
>>>> Best regards,
>>>>
>>>> On Sun, Dec 13, 2009 at 8:18 AM, Jack Stewart <jstewart at caltech.edu>
>>>> wrote:
>>>>> Hi Lazuardi,
>>>>>
>>>>> You are correct in saying that you need VIP's on both subnets. Only a
> VIP
>>>>> subnet A can use ISP A and only a VIP on Subnet B can use ISP B.
>>>>>
>>>>> With ISP A as your default route, VIPs on Subnet A work just fine. The
>>>>> load
>>>>> balancer sees the destination address of the client and sends it to the
>>>>> default route. But when the real server sends return traffic for a
>>>>> traffic
>>>>> that came in from VIP B, the routing table tells it to go to the
> gateway
>>>>> for ISP A. Somewhere along the path the return traffic will get dropped
>>>>> into
>>>>> the bit bucket. A route-map can fix this.
>>>>>
>>>>> A route-map with the "ip policy route-map" command applies a set of
> ACL's
>>>>> and route related actions to traffic before the load balance looks at
> the
>>>>> routing table. Route-maps have precedence. In an earlier example below,
> I
>>>>> took advantage of using an extended ACL in order to force the return
>>>>> traffic
>>>>> for a VIP on subnet B to go out through the Gateway B. The load
> balancer
>>>>> changes the source address of the real server to the VIP, my route map
>>>>> saw
>>>>> that the source address of the VIP was on subnet B, and so it set the
>>>>> next
>>>>> hop to Gateway B. The destination address and default gateway did not
>>>>> matter
>>>>> for this traffic.
>>>>>
>>>>> This is a pretty terse description, let me know if it is along the
> right
>>>>> track and we can get into details off line if needed.
>>>>>
>>>>> The web is a can of worms. Is the content dynamic or static? Is the
>>>>> traffic
>>>>> encrypted or is it not encrypted? There are a lot of different tricks.
> I
>>>>> would start with the basics - don't worry about it unless it is a
> problem
>>>>> or
>>>>> you know will be a problem. For the most part the client will cache the
>>>>> destination IP for a period of time and most web content is static.
> Also,
>>>>> are we talking encrypted or unencrypted traffic? If it is encrypted you
>>>>> can
>>>>> go off of the ssl session id. If this is really a problem, I'll put on
> my
>>>>> sysadmin thinking cap but I'll need to know O/S, http. For example,
>>>>> Apache
>>>>> 2.2 has a mod_header module so that you can set the particular server.
>>>>> This
>>>>> only applies to unencrypted traffic as the load balancer can't decrypt
>>>>> it.
>>>>>
>>>>> Lazuardi Nasution wrote:
>>>>>> Hi Jack,
>>>>>>
>>>>>> I'm interested with route-maps. Can you tell me more about that ?
>>>>>> Currently I just using static route for two default gateway with
>>>>>> different distance.
>>>>>>
>>>>>> My requirements is simple, just to do server load balancing no matter
>>>>>> where the traffic come in. Since I have two ISPs which give me two
>>>>>> subnets, so I think I must have two VIPs on different subnets too.  I
>>>>>> need Cookie Switching since some of clients are behind the proxy or
>>>>>> NAT. I cannot use Source NAT since I have to preserve clients source
>>>>>> IP for logging purpose on the real server. Since I don't have to do
>>>>>> the link load balancing, I do static routing for default gateway, so
>>>>>> only single link will be bidirectional on the same time. But I don't
>>>>>> have any idea how to make those VIPs share the session table or at
>>>>>> least threat the request not as first request if there is related
>>>>>> cookie inside (ex. ServerID). If I can do that I'm sure I can switch
>>>>>> the request to the same real server no matter which VIP the request
>>>>>> come in.
>>>>>>
>>>>>> Any idea ?
>>>>>>
>>>>>> Best regards,
>>>>>>
>>>>>> On Fri, Dec 11, 2009 at 4:19 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>> wrote:
>>>>>>> Hi Lazuardi,
>>>>>>>
>>>>>>> I'm running the routing code. To the best of my knowledge, route-maps
>>>>>>> apply
>>>>>>> only to the routing code.
>>>>>>>
>>>>>>> route-maps allow you to set the gateway by source address,
> destination
>>>>>>> address, or port #. Static routes allow you to set gateway by
>>>>>>> destination.
>>>>>>> The VIP is the source address of the outgoing traffic. So in the
>>>>>>> example
>>>>>>> below, gateway for a VIP is based on its address (and not the
>>>>>>> destination
>>>>>>> client). This was a really hard concept for me to wrap my head
> around.
>>>>>>>
>>>>>>> It isn't clear to me that your case is the same. My setup is very
>>>>>>> atypical.
>>>>>>> I had a lot of trouble debugging it (traffic would leave the client
> but
>>>>>>> never come back to it).  You might be able to get at it by looking at
>>>>>>> the
>>>>>>> interface traffic of the gateways.
>>>>>>>
>>>>>>> Let me know the solution you come up with, I'm curious.
>>>>>>>
>>>>>>> ---Jack
>>>>>>>
>>>>>>> Lazuardi Nasution wrote:
>>>>>>>> Hi Jack,
>>>>>>>>
>>>>>>>> I think this solution is for Switch Code since with Router Code I
> can
>>>>>>>> have many Management IP even with different subnets. The default
>>>>>>>> gateway can be specified statically on the routing table or by using
>>>>>>>> routing protocol from the routers.
>>>>>>>>
>>>>>>>> Best regards,
>>>>>>>>
>>>>>>>> On Fri, Dec 11, 2009 at 2:25 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>>>> wrote:
>>>>>>>>> Hi Lazuardi,
>>>>>>>>>
>>>>>>>>> I ran into similar issues - this is what ultimately work in my
>>>>>>>>> environment.
>>>>>>>>> It may not be the same but hopefully there are some takeaways.
> Please
>>>>>>>>> let
>>>>>>>>> me
>>>>>>>>> know how much of this makes sense - that feedback will be helpful
>>>>>>>>> with
>>>>>>>>> my
>>>>>>>>> documentation.
>>>>>>>>>
>>>>>>>>> First, DNS is special and the following is generic.
>>>>>>>>>
>>>>>>>>> You can only have one management IP and one default route. The
>>>>>>>>> management
>>>>>>>>> IP
>>>>>>>>> should live on the same subnet that has the default route. The
> first
>>>>>>>>> public
>>>>>>>>> subnet with the management IP & default route (Pub_Subnet_1) was
> not
>>>>>>>>> a
>>>>>>>>> problem.
>>>>>>>>>
>>>>>>>>> All of my real servers are on a different private subnet that the
> two
>>>>>>>>> public
>>>>>>>>> subnets and they all have the Load Balancer defined as their
> default
>>>>>>>>> gateway.
>>>>>>>>>
>>>>>>>>> To get subnet 2 (pub_subnet_2), I needed to define a router
> interface
>>>>>>>>> for
>>>>>>>>> that subnetwork (ve2) and policy routing/route-maps. The route-maps
>>>>>>>>> are
>>>>>>>>> for
>>>>>>>>> making sure that the return traffic goes out via the same gateway
>>>>>>>>> that
>>>>>>>>> it
>>>>>>>>> came in for non directly attached subnets. The way the mapping
> works
>>>>>>>>> for
>>>>>>>>> me
>>>>>>>>> in the configuration is:
>>>>>>>>>
>>>>>>>>> !
>>>>>>>>> ip access-list extended match_pub_subnet_2
>>>>>>>>>  permit ip match_pubsub2/24 any
>>>>>>>>> !
>>>>>>>>> route-map more_default_routes permit 10
>>>>>>>>>  match ip address match_subnet_2
>>>>>>>>>  set ip next-hop subnet_2_gateway
>>>>>>>>> !
>>>>>>>>> ip policy prefer-direct-route
>>>>>>>>> ip policy route-map more_default_routes
>>>>>>>>>
>>>>>>>>> In route-maps, the 'permit #' is just the precedence order. You can
>>>>>>>>> add
>>>>>>>>> additional entries to a route-map. Route-maps are processed before
>>>>>>>>> static
>>>>>>>>> routes.
>>>>>>>>>
>>>>>>>>> Lastly, I defined an outside NAT policy on Public_Subnet_1 for
>>>>>>>>> traffic
>>>>>>>>> originating private subnet traffic (i.e. directly attached
> servers).
>>>>>>>>> I'm
>>>>>>>>> not
>>>>>>>>> 100% sure this is a requirement but it helps with traceroute, etc.
>>>>>>>>>
>>>>>>>>> In my case it was necessary to add VRRP but that is because I've
> more
>>>>>>>>> than
>>>>>>>>> one box and it isn't clear you need that.
>>>>>>>>>
>>>>>>>>> Once this was done, everything worked nicely from outside to
> inside.
>>>>>>>>>
>>>>>>>>> This is a global static approach. Most people seem to route-maps to
>>>>>>>>> filter
>>>>>>>>> routing protocols, but I'm not allowed to exchange LB routing
>>>>>>>>> protocols
>>>>>>>>> with
>>>>>>>>> our routers by policy.
>>>>>>>>>
>>>>>>>>> For VIPs and real servers on the same private subnet, I found that
>>>>>>>>> either
>>>>>>>>> DSR or source-nat with ACL's works well. If you are using DSR with
>>>>>>>>> Linux
>>>>>>>>> (it
>>>>>>>>> seems to apply to other 2.6 kernels), you'll probably want to look
> at
>>>>>>>>> the
>>>>>>>>> brocade wiki).
>>>>>>>>>
>>>>>>>>> With DNS, source-nat with ACL's is probably the simplest and easier
>>>>>>>>> way
>>>>>>>>> to
>>>>>>>>> go.
>>>>>>>>>
>>>>>>>>> ---Jack
>>>>>>>>>
>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>> Hi Jack,
>>>>>>>>>>
>>>>>>>>>> Yes, there is different router per subnet and I have done the
> static
>>>>>>>>>> routing for that. VIP1 is in the same subnet with Management IP
> and
>>>>>>>>>> the Router1 is connected to eth1, so I just simply put Management
> IP
>>>>>>>>>> on eth1. Since Router2 is connected to eth2, should I do something
>>>>>>>>>> on
>>>>>>>>>> eth2, ex. put another management IP on the eth2 which is in the
> same
>>>>>>>>>> subnet with VIP2 ? The other ethernet ports are for Real Server so
> I
>>>>>>>>>> have give ve1 for those ports.
>>>>>>>>>>
>>>>>>>>>> There is another weird problem. I have made DNS binding from VIP1
>>>>>>>>>> and
>>>>>>>>>> RE1 and I have put ve1 IP in the same subnet with RE1. RE1 default
>>>>>>>>>> gateway is ve1 IP. I can query the DNS through VIP1 but RE1 cannot
>>>>>>>>>> do
>>>>>>>>>> traceroute to the Internet, stuck on the ServerIron. What's happen
>>>>>>>>>> here ?
>>>>>>>>>>
>>>>>>>>>> Best regards,
>>>>>>>>>>
>>>>>>>>>> On Thu, Dec 10, 2009 at 3:41 AM, Jack Stewart
> <jstewart at caltech.edu>
>>>>>>>>>> wrote:
>>>>>>>>>>> Hi Lazuardi,
>>>>>>>>>>>
>>>>>>>>>>> Yeah! A question that might be up my alley. I've done this
> however
>>>>>>>>>>> I
>>>>>>>>>>> need
>>>>>>>>>>> some more details.
>>>>>>>>>>>
>>>>>>>>>>> Do these VIPs need different "static" default gateways on a per
>>>>>>>>>>> subnet
>>>>>>>>>>> basis? It's possible with the routing code and I can send out the
>>>>>>>>>>> details
>>>>>>>>>>> if
>>>>>>>>>>> you are interested.
>>>>>>>>>>>
>>>>>>>>>>> Otherwise the main trick with subnet A to subnet B traffic is to
>>>>>>>>>>> make
>>>>>>>>>>> sure
>>>>>>>>>>> that the return traffic goes though the load balancer. The client
> &
>>>>>>>>>>> server
>>>>>>>>>>> need to see the Load Balancer as the gateway between subnet A &
>>>>>>>>>>> subnet
>>>>>>>>>>> B.
>>>>>>>>>>> DSR and source NAT are also options.
>>>>>>>>>>>
>>>>>>>>>>> So more details, please. Depending on what you need to do it
> might
>>>>>>>>>>> help
>>>>>>>>>>> knock out some of my documentation.
>>>>>>>>>>>
>>>>>>>>>>> ---Jack
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>>> Hi,
>>>>>>>>>>>>
>>>>>>>>>>>> Is it possible to have multiple VIPs with different Subnets on
>>>>>>>>>>>> ServerIron 4G or ServerIron ADX1000 ? How can I do that ? I'm
>>>>>>>>>>>> using
>>>>>>>>>>>> router code of firmware.
>>>>>>>>>>>>
>>> --
>>> Jack Stewart
>>> Academic Computing Services, IMSS,
>>> California Institute of Technology
>>> jstewart at caltech.edu
>>> 626-395-4690 office / 626-437-6035 cell
>
> --
> Jack Stewart
> Academic Computing Services, IMSS,
> California Institute of Technology
> jstewart at caltech.edu
> 626-395-4690 office / 626-437-6035 cell
> _______________________________________________
> foundry-nsp mailing list
> foundry-nsp at puck.nether.net
> http://puck.nether.net/mailman/listinfo/foundry-nsp
>
>