[j-nsp] SRX 3600 dropped packets - how to debug?

[Phil Mayers]

On 28/05/13 14:40, Julien Goodwin wrote:
> On 28/05/13 19:40, ashish verma wrote:
>>> That said, I can't believe the firewall was *actually* dropping 1500pps of
>>> DNS traffic; we'd have widespread problems reported, surely. So, it seems
>>> that maybe ALG-processed traffic is being counted under "packets dropped"
>>> for "show security flow statistics"?
>
> eDNS fallback perhaps?

IME eDNS is pretty much exclusively confined to server-server DNS 
traffic (modulo newer stuff like in-app DNSSEC resolvers) and these are 
ordinary clients, which don't tend to make eDNS requests. A SPAN of the 
link(s) into the SRX tends to support this - no eDNS traffic in a 32Gb 
capture.

I have my suspicions about what exactly the ALG is (mis)counting as a 
drop, and will be trying to reproduce it on the bench now it's been 
taken out of service.

> I never understood the use of DNS ALG's, unless it's to perform a NAT
> translation on addresses (which is a really bad idea) they just seem
> like a waste of valuable resources. Far better to ACL down so that DNS
> queries can only go to trusted DNS servers which can run something that
> doesn't break on a malformed query.

I'm not a fan of ALGs, and in principle I agree with you.

However, if you disable the DNS ALG, you might see a sharp increase in 
session utilisation, as the UDP session stays open for the UDP timeout, 
as opposed to the fraction of a second that the DNS request/response 
takes. With even moderate load, this can be tens or hundreds of 
thousands of sessions "wasted", particularly because clients tend to use 
different UDP source ports - I know this because I tried it, both on our 
"real" Netscreen 5400s and on this SRX, and sure enough - big jumps.


The best solution here is to avoid the DNS traffic hitting the firewall 
altogether. If you use L3VPN (as we do) this involves the well-known 
"services VRF" solution using appropriate route-targets, or simply 
multi-homing your DNS servers into each security zone.

But this only gets *your* DNS servers. If you have people using Google 
DNS, Open DNS or whatever, you are faced with either blocking access to 
those services and eating the support costs ("I have a roomful of VIPs 
and 3 of them can't access the internet") or worse, intercepting traffic 
to those services and pretending to be them.

I'm not wild about the upswing in public DNS resolvers and their 
apparent popularity amongst customers, but they're a fact of life now, 
particularly in more open networks (such as universities). The idea of a 
malfunctioning client with 8.8.8.8 as a DNS server consuming 250,000 
sessions on our firewall is not attractive :o(