[c-nsp] ACE & load-balancing of DNS / ALG / inspection

[Phil Mayers]

Phil Mayers wrote:
> All,
> 
> We've recently deployed config on our ACE (blades in 6500s) to provide 
> resilient DNS.
> 
> However, the ACE seems to be doing some kind of DNS inspection, and is 
> (incorrectly I think) closing the SLB session the instant a DNS answer 
> comes back. This causes problems with clients that make 2 lookups very 
> quickly, from the same source port.
> 
> i.e. I am seeing:
> 
> client sport=5000 dport=53 query id=2346 hostname A
> client sport=5000 dport=53 query id=4646 hostname AAAA
> server dport=5000 sport=53 reply id=2346 A=192.168.x.y
> 
> ...and that's it. The 2nd reply is dropped. If the client makes the 
> queries "slowly" they work fine:

Just a follow-on.

The specific issue seems to be that the ACE *requires* at least one UDP 
reply packet from the server before fully "opening" the UDP session. 
Monitoring at the "rserver" end shows for the above:

client sport=5000 dport=53 query id=2346 hostname A
server dport=5000 sport=53 reply id=2346 A=192.168.x.y

i.e. the 2nd *request* is dropped.

Once that 1st reply is sent, you can send as many queries as you want:

client sport=5000 dport=53 query id=2346 hostname A
server dport=5000 sport=53 reply id=2346 A=192.168.x.y
client sport=5000 dport=53 query id=2346 hostname1 A
client sport=5000 dport=53 query id=2346 hostname1 A
client sport=5000 dport=53 query id=2346 hostname1 A
client sport=5000 dport=53 query id=2346 hostname1 A
server dport=5000 sport=53 reply id=2346 A=192.168.x.y



> 
> client sport=5000 dport=53 query id=2346 hostname A
> server dport=5000 sport=53 reply id=2346 A=192.168.x.y
> client sport=5000 dport=53 query id=4646 hostname AAAA
> server dport=5000 sport=53 reply id=4646 AAAA=...
> 
> Our old DNS servers (via static anycast routes) and a different service 
> (via eBGP multipath anycast) don't exhibit the problem, so I'm certain 
> it's the ACE.
> 
> FYI, this causes problems with the glibc changes present in 2.10 & 
> Fedora 11 - the glibc always tries two queries in quick succession for A 
> and AAAA records, and the timeouts can destroy kerberos/ldap logins...
> 
> I'm aware of the "inspect" commands, but they're off by default and I 
> can't "no inspect"; it tells me it's already turned off.
> 
> Does anyone know if and how I can persuade the ACE to stop being so 
> "clever" and just treat the DNS as "plain old UDP"?
> 
> version info is:
> 
> Software
>    loader:    Version 12.2[120]
>    system:    Version A2(1.1) [build 3.0(0)A2(1.1) 
> adbuild_00:25:02-2008/06/05_/auto/adbu-rel3/rel_a2_1_1_throttle/REL_3_0_0_A2_1_1]
>    system image file: [LCP] disk0:c6ace-t1k9-mz.A2_1_1.bin
>    installed license: ACE-08G-LIC ACE-SEC-LIC-K9
> 
> ...and the config we're using is:
> 
> 
> serverfarm host RECURSIVE-DNS
>    transparent
>    predictor leastconns
>    probe TCP_53
>    rserver xxx 53
>      inservice
>    rserver yyy 53
>      inservice
>    rserver www 53
>      inservice
>    rserver zzz 53
>      inservice
> 
> class-map match-any VIP_SPONCON-DNS
>    2 match virtual-address 192.168.a.b udp eq domain
>    3 match virtual-address 192.168.a.b tcp eq domain
> 
> policy-map type loadbalance first-match SLB_RECURSIVE-DNS
>    class class-default
>      serverfarm RECURSIVE-DNS
> 
> policy-map multi-match VIPS_VLANxx
>    !.. various config, then
>    class VIP_SPONCON-DNS
>      loadbalance vip inservice
>      loadbalance policy SLB_RECURSIVE-DNS
>      loadbalance vip icmp-reply
>      loadbalance vip advertise
>