[f-nsp] Multiple VIPs with Different Subnets

Mon Dec 14 22:42:03 EST 2009

Hi Jack,

Have you find solution of "confused" VIP2 on cookie-switching ?

Best regards,

On Tue, Dec 15, 2009 at 7:56 AM, Jared Valentine <hidden at xmission.com> wrote:
> Management IP also follows learned/default routes and is not affected by
> route-maps.  Actually, the name of the feature here isn't "route-maps"
> per-se, it's "policy-based routing for reverse slb traffic".
>
> From my experience, management and ICMP to a VIP both follow the
> learned/default routes.  Only reverse SLB traffic actually follows the
> route-map.
>
> Thanks,
>
> Jared
>
> -----Original Message-----
> From: Lazuardi Nasution [mailto:mrxlazuardin at gmail.com]
> Sent: Monday, December 14, 2009 1:26 PM
> To: Jared Valentine
> Cc: Jack Stewart; foundry-nsp at puck.nether.net
> Subject: Re: [f-nsp] Multiple VIPs with Different Subnets
>
> Hi Jared,
>
> I think you are right regarding to ICMP. I have tried but failed. What
> about Management IP. It seem to be failed too.
>
> Best regards,
>
> On Tue, Dec 15, 2009 at 3:18 AM, Jared Valentine <hidden at xmission.com>
> wrote:
>> ICMP to the VIP won't work if the VIP uses a route-map, even if it's
>> included in the ACL.  It's just a limitation of the ServerIron.  It should
>> continue to work for VIPs that aren't affected by the route-map and are
>> following the default and/or dynamic routes.
>>
>> Jared
>>
>>
>> -----Original Message-----
>> From: foundry-nsp-bounces at puck.nether.net
>> [mailto:foundry-nsp-bounces at puck.nether.net] On Behalf Of Jack Stewart
>> Sent: Monday, December 14, 2009 12:55 PM
>> To: Lazuardi Nasution
>> Cc: foundry-nsp at puck.nether.net
>> Subject: Re: [f-nsp] Multiple VIPs with Different Subnets
>>
>>
>> Congratulations! In sounds like you are beginning to get route-maps.
>> This is no small think. Every book that I've read so far says
>> "route-maps are beyond the scope of this book but here are some neat
> tricks"
>>
>> Hmm. I'm going to have to lab both of these tomorrow to be 100% sure.
>>
>> ICMP should be pretty straight forward, you should just need to add an
>> ACL to your route map to premit ICMP (all you are letting through is IP).
>>
>> Cookie Switching looks interesting and it is on my to do list to look at
>> - I'll have to consult with my web guru. My guess is that the cookie is
>> getting used by the switch to tell it which which to go to. It looks
>> like cookie switching is trumping route maps on a first match basis.
>>
>> You've probably already thought of this but what about forcing this data
>> to be ssl and then using session-id-switch for future connections? It is
>> a unique client/server keypair. If the customer you are worried about
>> comes from the big NAT/Proxy box, you can force their connections to be
>> written into SSL (mod_rewrite on the server)
>>
>> Just a thought.
>>
>> Ciao,
>>
>> ---Jack
>>
>>
>> Lazuardi Nasution wrote:
>>> Hi Jack,
>>>
>>> I have implemented route-maps. It is OK until I do cookie switching.
>>> When I put cookie-switching, the second VIP never response the HTTP
>>> request. The connection is OK but no response. If I remove the
>>> cookie-switching, the traffic is normal again. I need the
>>> cookie-switching for handling the clients behind big NAT/Proxy and
>>> make sure that the request switched to the same server on the event of
>>> link state changes.
>>>
>>> Beside that, it seem that route-map cannot work for ICMP. I still
>>> cannot Ping to second VIP.
>>>
>>> Best regards,
>>>
>>> On Mon, Dec 14, 2009 at 6:57 AM, Jack Stewart <jstewart at caltech.edu>
>> wrote:
>>>> Hi Lazuardi,
>>>>
>>>> Setting up a DNS response strategy seems like a good option - the idea
> is
>>>> sound and it is the foundation for GSLB for many vendors. I've read
> about
>>>> GSLB but never implemented it.
>>>>
>>>> I applied route-maps to the management IP subnet for giggles and removed
>> the
>>>> default route. It works but I suspect it isn't a best practice. A
>> route-map
>>>> is more like pre-routing table. If you get a match in the route-map,
> then
>>>> the specific action is applied. Otherwise, it goes to the routing table.
>>>> Another way of thinking about it is that route-maps (or policy routing)
>> have
>>>> the better metric but you still have your default and static routes.
>>>>
>>>> Your setup isn't something that I've done but as long as it works, stick
>>>> with it. In my situation I have two class C subnets that are part of a
>>>> larger class B and default routes couldn't do the job.
>>>>
>>>> Route maps are pretty cool and now that I understand them, I'm currently
>>>> exploring PBR as a mechanism for handling Email flows differently based
>> on
>>>> source. Load balancing using DNS is way down the road for me.
>>>>
>>>> Cheers,
>>>>
>>>> ---Jack
>>>>
>>>>
>>>> Lazuardi Nasution wrote:
>>>>> Hi Jack,
>>>>>
>>>>> Is this route-map can be applied to Mangement IPs ?
>>>>>
>>>>> Currently, I have put two default gateway with different distance for
>>>>> both subnet. For avoiding the access to non default gateway VIP, I
>>>>> have put different set of DNS for each VIP. This way can push the
>>>>> clients to always use VIP A if the DNS come to VIP A (lower distance
>>>>> subnet) by using VIP A IP for all DNS records bound with VIP A and
>>>>> vice versa. If DNS query is coming to VIP B (higher distance subnet),
>>>>> the reply will never acknowledged by the clients since it will be
>>>>> routed through the VIP A. This way can make only VIP A will do the
>>>>> good reply as long the ISP A link is good.
>>>>>
>>>>> Best regards,
>>>>>
>>>>> On Sun, Dec 13, 2009 at 8:18 AM, Jack Stewart <jstewart at caltech.edu>
>>>>> wrote:
>>>>>> Hi Lazuardi,
>>>>>>
>>>>>> You are correct in saying that you need VIP's on both subnets. Only a
>> VIP
>>>>>> subnet A can use ISP A and only a VIP on Subnet B can use ISP B.
>>>>>>
>>>>>> With ISP A as your default route, VIPs on Subnet A work just fine. The
>>>>>> load
>>>>>> balancer sees the destination address of the client and sends it to
> the
>>>>>> default route. But when the real server sends return traffic for a
>>>>>> traffic
>>>>>> that came in from VIP B, the routing table tells it to go to the
>> gateway
>>>>>> for ISP A. Somewhere along the path the return traffic will get
> dropped
>>>>>> into
>>>>>> the bit bucket. A route-map can fix this.
>>>>>>
>>>>>> A route-map with the "ip policy route-map" command applies a set of
>> ACL's
>>>>>> and route related actions to traffic before the load balance looks at
>> the
>>>>>> routing table. Route-maps have precedence. In an earlier example
> below,
>> I
>>>>>> took advantage of using an extended ACL in order to force the return
>>>>>> traffic
>>>>>> for a VIP on subnet B to go out through the Gateway B. The load
>> balancer
>>>>>> changes the source address of the real server to the VIP, my route map
>>>>>> saw
>>>>>> that the source address of the VIP was on subnet B, and so it set the
>>>>>> next
>>>>>> hop to Gateway B. The destination address and default gateway did not
>>>>>> matter
>>>>>> for this traffic.
>>>>>>
>>>>>> This is a pretty terse description, let me know if it is along the
>> right
>>>>>> track and we can get into details off line if needed.
>>>>>>
>>>>>> The web is a can of worms. Is the content dynamic or static? Is the
>>>>>> traffic
>>>>>> encrypted or is it not encrypted? There are a lot of different tricks.
>> I
>>>>>> would start with the basics - don't worry about it unless it is a
>> problem
>>>>>> or
>>>>>> you know will be a problem. For the most part the client will cache
> the
>>>>>> destination IP for a period of time and most web content is static.
>> Also,
>>>>>> are we talking encrypted or unencrypted traffic? If it is encrypted
> you
>>>>>> can
>>>>>> go off of the ssl session id. If this is really a problem, I'll put on
>> my
>>>>>> sysadmin thinking cap but I'll need to know O/S, http. For example,
>>>>>> Apache
>>>>>> 2.2 has a mod_header module so that you can set the particular server.
>>>>>> This
>>>>>> only applies to unencrypted traffic as the load balancer can't decrypt
>>>>>> it.
>>>>>>
>>>>>> Lazuardi Nasution wrote:
>>>>>>> Hi Jack,
>>>>>>>
>>>>>>> I'm interested with route-maps. Can you tell me more about that ?
>>>>>>> Currently I just using static route for two default gateway with
>>>>>>> different distance.
>>>>>>>
>>>>>>> My requirements is simple, just to do server load balancing no matter
>>>>>>> where the traffic come in. Since I have two ISPs which give me two
>>>>>>> subnets, so I think I must have two VIPs on different subnets too.  I
>>>>>>> need Cookie Switching since some of clients are behind the proxy or
>>>>>>> NAT. I cannot use Source NAT since I have to preserve clients source
>>>>>>> IP for logging purpose on the real server. Since I don't have to do
>>>>>>> the link load balancing, I do static routing for default gateway, so
>>>>>>> only single link will be bidirectional on the same time. But I don't
>>>>>>> have any idea how to make those VIPs share the session table or at
>>>>>>> least threat the request not as first request if there is related
>>>>>>> cookie inside (ex. ServerID). If I can do that I'm sure I can switch
>>>>>>> the request to the same real server no matter which VIP the request
>>>>>>> come in.
>>>>>>>
>>>>>>> Any idea ?
>>>>>>>
>>>>>>> Best regards,
>>>>>>>
>>>>>>> On Fri, Dec 11, 2009 at 4:19 AM, Jack Stewart <jstewart at caltech.edu>
>>>>>>> wrote:
>>>>>>>> Hi Lazuardi,
>>>>>>>>
>>>>>>>> I'm running the routing code. To the best of my knowledge,
> route-maps
>>>>>>>> apply
>>>>>>>> only to the routing code.
>>>>>>>>
>>>>>>>> route-maps allow you to set the gateway by source address,
>> destination
>>>>>>>> address, or port #. Static routes allow you to set gateway by
>>>>>>>> destination.
>>>>>>>> The VIP is the source address of the outgoing traffic. So in the
>>>>>>>> example
>>>>>>>> below, gateway for a VIP is based on its address (and not the
>>>>>>>> destination
>>>>>>>> client). This was a really hard concept for me to wrap my head
>> around.
>>>>>>>>
>>>>>>>> It isn't clear to me that your case is the same. My setup is very
>>>>>>>> atypical.
>>>>>>>> I had a lot of trouble debugging it (traffic would leave the client
>> but
>>>>>>>> never come back to it).  You might be able to get at it by looking
> at
>>>>>>>> the
>>>>>>>> interface traffic of the gateways.
>>>>>>>>
>>>>>>>> Let me know the solution you come up with, I'm curious.
>>>>>>>>
>>>>>>>> ---Jack
>>>>>>>>
>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>> Hi Jack,
>>>>>>>>>
>>>>>>>>> I think this solution is for Switch Code since with Router Code I
>> can
>>>>>>>>> have many Management IP even with different subnets. The default
>>>>>>>>> gateway can be specified statically on the routing table or by
> using
>>>>>>>>> routing protocol from the routers.
>>>>>>>>>
>>>>>>>>> Best regards,
>>>>>>>>>
>>>>>>>>> On Fri, Dec 11, 2009 at 2:25 AM, Jack Stewart
> <jstewart at caltech.edu>
>>>>>>>>> wrote:
>>>>>>>>>> Hi Lazuardi,
>>>>>>>>>>
>>>>>>>>>> I ran into similar issues - this is what ultimately work in my
>>>>>>>>>> environment.
>>>>>>>>>> It may not be the same but hopefully there are some takeaways.
>> Please
>>>>>>>>>> let
>>>>>>>>>> me
>>>>>>>>>> know how much of this makes sense - that feedback will be helpful
>>>>>>>>>> with
>>>>>>>>>> my
>>>>>>>>>> documentation.
>>>>>>>>>>
>>>>>>>>>> First, DNS is special and the following is generic.
>>>>>>>>>>
>>>>>>>>>> You can only have one management IP and one default route. The
>>>>>>>>>> management
>>>>>>>>>> IP
>>>>>>>>>> should live on the same subnet that has the default route. The
>> first
>>>>>>>>>> public
>>>>>>>>>> subnet with the management IP & default route (Pub_Subnet_1) was
>> not
>>>>>>>>>> a
>>>>>>>>>> problem.
>>>>>>>>>>
>>>>>>>>>> All of my real servers are on a different private subnet that the
>> two
>>>>>>>>>> public
>>>>>>>>>> subnets and they all have the Load Balancer defined as their
>> default
>>>>>>>>>> gateway.
>>>>>>>>>>
>>>>>>>>>> To get subnet 2 (pub_subnet_2), I needed to define a router
>> interface
>>>>>>>>>> for
>>>>>>>>>> that subnetwork (ve2) and policy routing/route-maps. The
> route-maps
>>>>>>>>>> are
>>>>>>>>>> for
>>>>>>>>>> making sure that the return traffic goes out via the same gateway
>>>>>>>>>> that
>>>>>>>>>> it
>>>>>>>>>> came in for non directly attached subnets. The way the mapping
>> works
>>>>>>>>>> for
>>>>>>>>>> me
>>>>>>>>>> in the configuration is:
>>>>>>>>>>
>>>>>>>>>> !
>>>>>>>>>> ip access-list extended match_pub_subnet_2
>>>>>>>>>>  permit ip match_pubsub2/24 any
>>>>>>>>>> !
>>>>>>>>>> route-map more_default_routes permit 10
>>>>>>>>>>  match ip address match_subnet_2
>>>>>>>>>>  set ip next-hop subnet_2_gateway
>>>>>>>>>> !
>>>>>>>>>> ip policy prefer-direct-route
>>>>>>>>>> ip policy route-map more_default_routes
>>>>>>>>>>
>>>>>>>>>> In route-maps, the 'permit #' is just the precedence order. You
> can
>>>>>>>>>> add
>>>>>>>>>> additional entries to a route-map. Route-maps are processed before
>>>>>>>>>> static
>>>>>>>>>> routes.
>>>>>>>>>>
>>>>>>>>>> Lastly, I defined an outside NAT policy on Public_Subnet_1 for
>>>>>>>>>> traffic
>>>>>>>>>> originating private subnet traffic (i.e. directly attached
>> servers).
>>>>>>>>>> I'm
>>>>>>>>>> not
>>>>>>>>>> 100% sure this is a requirement but it helps with traceroute, etc.
>>>>>>>>>>
>>>>>>>>>> In my case it was necessary to add VRRP but that is because I've
>> more
>>>>>>>>>> than
>>>>>>>>>> one box and it isn't clear you need that.
>>>>>>>>>>
>>>>>>>>>> Once this was done, everything worked nicely from outside to
>> inside.
>>>>>>>>>>
>>>>>>>>>> This is a global static approach. Most people seem to route-maps
> to
>>>>>>>>>> filter
>>>>>>>>>> routing protocols, but I'm not allowed to exchange LB routing
>>>>>>>>>> protocols
>>>>>>>>>> with
>>>>>>>>>> our routers by policy.
>>>>>>>>>>
>>>>>>>>>> For VIPs and real servers on the same private subnet, I found that
>>>>>>>>>> either
>>>>>>>>>> DSR or source-nat with ACL's works well. If you are using DSR with
>>>>>>>>>> Linux
>>>>>>>>>> (it
>>>>>>>>>> seems to apply to other 2.6 kernels), you'll probably want to look
>> at
>>>>>>>>>> the
>>>>>>>>>> brocade wiki).
>>>>>>>>>>
>>>>>>>>>> With DNS, source-nat with ACL's is probably the simplest and
> easier
>>>>>>>>>> way
>>>>>>>>>> to
>>>>>>>>>> go.
>>>>>>>>>>
>>>>>>>>>> ---Jack
>>>>>>>>>>
>>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>> Hi Jack,
>>>>>>>>>>>
>>>>>>>>>>> Yes, there is different router per subnet and I have done the
>> static
>>>>>>>>>>> routing for that. VIP1 is in the same subnet with Management IP
>> and
>>>>>>>>>>> the Router1 is connected to eth1, so I just simply put Management
>> IP
>>>>>>>>>>> on eth1. Since Router2 is connected to eth2, should I do
> something
>>>>>>>>>>> on
>>>>>>>>>>> eth2, ex. put another management IP on the eth2 which is in the
>> same
>>>>>>>>>>> subnet with VIP2 ? The other ethernet ports are for Real Server
> so
>> I
>>>>>>>>>>> have give ve1 for those ports.
>>>>>>>>>>>
>>>>>>>>>>> There is another weird problem. I have made DNS binding from VIP1
>>>>>>>>>>> and
>>>>>>>>>>> RE1 and I have put ve1 IP in the same subnet with RE1. RE1
> default
>>>>>>>>>>> gateway is ve1 IP. I can query the DNS through VIP1 but RE1
> cannot
>>>>>>>>>>> do
>>>>>>>>>>> traceroute to the Internet, stuck on the ServerIron. What's
> happen
>>>>>>>>>>> here ?
>>>>>>>>>>>
>>>>>>>>>>> Best regards,
>>>>>>>>>>>
>>>>>>>>>>> On Thu, Dec 10, 2009 at 3:41 AM, Jack Stewart
>> <jstewart at caltech.edu>
>>>>>>>>>>> wrote:
>>>>>>>>>>>> Hi Lazuardi,
>>>>>>>>>>>>
>>>>>>>>>>>> Yeah! A question that might be up my alley. I've done this
>> however
>>>>>>>>>>>> I
>>>>>>>>>>>> need
>>>>>>>>>>>> some more details.
>>>>>>>>>>>>
>>>>>>>>>>>> Do these VIPs need different "static" default gateways on a per
>>>>>>>>>>>> subnet
>>>>>>>>>>>> basis? It's possible with the routing code and I can send out
> the
>>>>>>>>>>>> details
>>>>>>>>>>>> if
>>>>>>>>>>>> you are interested.
>>>>>>>>>>>>
>>>>>>>>>>>> Otherwise the main trick with subnet A to subnet B traffic is to
>>>>>>>>>>>> make
>>>>>>>>>>>> sure
>>>>>>>>>>>> that the return traffic goes though the load balancer. The
> client
>> &
>>>>>>>>>>>> server
>>>>>>>>>>>> need to see the Load Balancer as the gateway between subnet A &
>>>>>>>>>>>> subnet
>>>>>>>>>>>> B.
>>>>>>>>>>>> DSR and source NAT are also options.
>>>>>>>>>>>>
>>>>>>>>>>>> So more details, please. Depending on what you need to do it
>> might
>>>>>>>>>>>> help
>>>>>>>>>>>> knock out some of my documentation.
>>>>>>>>>>>>
>>>>>>>>>>>> ---Jack
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Lazuardi Nasution wrote:
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Is it possible to have multiple VIPs with different Subnets on
>>>>>>>>>>>>> ServerIron 4G or ServerIron ADX1000 ? How can I do that ? I'm
>>>>>>>>>>>>> using
>>>>>>>>>>>>> router code of firmware.
>>>>>>>>>>>>>
>>>> --
>>>> Jack Stewart
>>>> Academic Computing Services, IMSS,
>>>> California Institute of Technology
>>>> jstewart at caltech.edu
>>>> 626-395-4690 office / 626-437-6035 cell
>>
>> --
>> Jack Stewart
>> Academic Computing Services, IMSS,
>> California Institute of Technology
>> jstewart at caltech.edu
>> 626-395-4690 office / 626-437-6035 cell
>> _______________________________________________
>> foundry-nsp mailing list
>> foundry-nsp at puck.nether.net
>> http://puck.nether.net/mailman/listinfo/foundry-nsp
>>
>>
>
>