[outages] FAA.gov nameserver outage

[T.Suzuki]

On Sun, 26 Mar 2023 17:17:25 -0700
Mike Lyon <mike.lyon at gmail.com> wrote:

> Can’t believe it’s still dead…
> 
> -Mike

The attack appears to be over, at Mar 26 13:41:28 JST (GMT +0900)
(This may be specific to my server).
Maybe the cause is something else.
 Or the person in charge of manual recovery is on holiday.

Mar 26 13:41:08 unbound[48103:0] reply: 24.199.82.210 asm.faa.gov. A IN SERVFAIL 0.000000 0 29
Mar 26 13:41:15 unbound[48103:0] query: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN
Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:15 unbound[48103:0] reply: 24.199.82.210 sas-uss.edc.nas.faa.gov. A IN SERVFAIL 0.000000 0 41
Mar 26 13:41:22 unbound[48103:0] query: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN
Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210 eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
Mar 26 13:41:23 unbound[48103:0] query: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN
Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210 faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
Mar 26 13:41:28 unbound[48103:0] query: 24.199.82.210 chronos3.faa.gov. A IN
Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A IN>: all servers for this domain failed, at zone faa.gov. no server to query nameserver addresses not usable
Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A IN SERVFAIL 0.000000 0 34

> > On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages at outages.org> wrote:
> > 
> > 〓On Sun, 26 Mar 2023 08:35:29 -0700
> > Hugo Slabbert <hugo at slabnet.com> wrote:
> > 
> >> What would be the symptoms here of a "water torture attack" rather than
> >> what John had indicated as a firewall failure in their infrastructure:
> >> 
> >>> Initial looks from the firewall team point to an automatic failover event
> >> and the secondary failed.
> >> 
> >> And the symptoms of which lined up with network level info from Paul
> >> earlier:
> >> 
> >>> They only seem to have two auth nameservers for faa, both within the
> >> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
> >> servers are in all die just within each block run by the FAA.
> >>> 
> >>> Seems like an internal routing meltdown making the only 2 nameservers
> >> unreachable reliably.
> >> 
> >> Are you saying that your open resolvers have a per client rate limit
> >> applied, that rate limit got tripped, and shortly thereafter the resolvers
> >> became unavailable, suggesting query floods for the domain(s) that knocked
> >> the resolvers offline (or from the other discussion, possibly was the thing
> >> that overwhelmed that firewall layer, causing the initial failover and
> >> possibly also causing the firewall secondary to fail to come online)?
> > 
> > Yes. (limitting per client, and per second for all)
> > Perhaps, large numbers open resolvers including no ratelimit are used.
> > Then massive random subdomain queries caused the firewall symptoms.
> > (It's only my guess.)
> > 
> >>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages at outages.org>
> >>> wrote:
> >>> 
> >>> Hi, I'm a researcher of DNS vulnerabilities.
> >>> 
> >>> It loos like random subdomain attacks (water tourtue attack).
> >>> 
> >>> This is the data of my rate-limitted openresolver as a honeypot.
> >>> http://www.e-ontap.com/dns/todaydowngov.txt
> >>> http://www.e-ontap.com/dns/todaydown.txt
> >>> (You can not view these page if you are using 8.8.8.8, sorry.)
> >>> 
> >>> Raw logs of my Unbound (Time is JST)
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> >>> -5
> >>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> >>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> >>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> >>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> >>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> >>> exceeded ratelimit for zone faa.gov.
> >>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> >>> SERVFAIL 15.112813 0 30
> >>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> >>> exceeded ratelimit for zone faa.gov.
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> >>> head -5
> >>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> >>> all servers for this domain failed, at zone faa.gov. from
> >>> 2620:74:27::2:30 no server to query nameserver addresses not usable
> >>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> >>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> >>> servers for this domain failed, at zone faa.gov. no server to query
> >>> nameserver addresses not usable
> >>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> >>> all servers for this domain failed, at zone faa.gov. upstream server
> >>> timeout
> >>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> >>> A IN>: all servers for this domain failed, at zone faa.gov. upstream
> >>> server timeout
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> >>> tail -5
> >>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> >>> servers for this domain failed, at zone faa.gov. no server to query
> >>> nameserver addresses not usable
> >>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> >>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> >>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> >>> zone faa.gov. no server to query nameserver addresses not usable
> >>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> >>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> >>> at zone faa.gov. no server to query nameserver addresses not usable
> >>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> >>> IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> >>> -5
> >>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> >>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> >>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> >>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> >>> at zone faa.gov. no server to query nameserver addresses not usable
> >>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> >>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> >>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> >>> IN>: all servers for this domain failed, at zone faa.gov. no server to
> >>> query nameserver addresses not usable
> >>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> >>> IN SERVFAIL 0.000000 0 34
> >>> local/etc/unbound%
> >>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> >>> -l
> >>>    1408
> >>> 
> >>> --
> >>> T.Suzuki
> >>> --
> >>> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
> >>> _______________________________________________
> >>> Outages mailing list
> >>> Outages at outages.org
> >>> https://puck.nether.net/mailman/listinfo/outages
> >>> 
> > 
> > 
> > -- 
> > T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> 


-- 
T.Suzuki / E.F.シューマッハーとI.イリイチを読もう