[outages] FAA.gov nameserver outage

[T.Suzuki]

On Sun, 26 Mar 2023 08:35:29 -0700
Hugo Slabbert <hugo at slabnet.com> wrote:

> What would be the symptoms here of a "water torture attack" rather than
> what John had indicated as a firewall failure in their infrastructure:
> 
> > Initial looks from the firewall team point to an automatic failover event
> and the secondary failed.
> 
> And the symptoms of which lined up with network level info from Paul
> earlier:
> 
> > They only seem to have two auth nameservers for faa, both within the
> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
> servers are in all die just within each block run by the FAA.
> >
> > Seems like an internal routing meltdown making the only 2 nameservers
> unreachable reliably.
> 
> Are you saying that your open resolvers have a per client rate limit
> applied, that rate limit got tripped, and shortly thereafter the resolvers
> became unavailable, suggesting query floods for the domain(s) that knocked
> the resolvers offline (or from the other discussion, possibly was the thing
> that overwhelmed that firewall layer, causing the initial failover and
> possibly also causing the firewall secondary to fail to come online)?

Yes. (limitting per client, and per second for all)
Perhaps, large numbers open resolvers including no ratelimit are used.
Then massive random subdomain queries caused the firewall symptoms.
(It's only my guess.)

> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages at outages.org>
> wrote:
> 
> > Hi, I'm a researcher of DNS vulnerabilities.
> >
> > It loos like random subdomain attacks (water tourtue attack).
> >
> > This is the data of my rate-limitted openresolver as a honeypot.
> > http://www.e-ontap.com/dns/todaydowngov.txt
> > http://www.e-ontap.com/dns/todaydown.txt
> > (You can not view these page if you are using 8.8.8.8, sorry.)
> >
> > Raw logs of my Unbound (Time is JST)
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> > -5
> > Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> > unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> > Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> > unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> > Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> > exceeded ratelimit for zone faa.gov.
> > Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> > SERVFAIL 15.112813 0 30
> > Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> > exceeded ratelimit for zone faa.gov.
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> > head -5
> > Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> > all servers for this domain failed, at zone faa.gov. from
> > 2620:74:27::2:30 no server to query nameserver addresses not usable
> > Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> > A IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> > servers for this domain failed, at zone faa.gov. no server to query
> > nameserver addresses not usable
> > Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> > all servers for this domain failed, at zone faa.gov. upstream server
> > timeout
> > Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> > A IN>: all servers for this domain failed, at zone faa.gov. upstream
> > server timeout
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> > tail -5
> > Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> > servers for this domain failed, at zone faa.gov. no server to query
> > nameserver addresses not usable
> > Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> > A IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> > eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> > zone faa.gov. no server to query nameserver addresses not usable
> > Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> > faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> > at zone faa.gov. no server to query nameserver addresses not usable
> > Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> > IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> > -5
> > Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> > eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> > Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> > faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> > at zone faa.gov. no server to query nameserver addresses not usable
> > Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> > faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> > Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> > IN>: all servers for this domain failed, at zone faa.gov. no server to
> > query nameserver addresses not usable
> > Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> > IN SERVFAIL 0.000000 0 34
> > local/etc/unbound%
> > local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> > -l
> >     1408
> >
> > --
> > T.Suzuki
> > --
> > T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >


-- 
T.Suzuki / E.F.シューマッハーとI.イリイチを読もう