[outages] FAA.gov nameserver outage

[Hugo Slabbert]

What would be the symptoms here of a "water torture attack" rather than
what John had indicated as a firewall failure in their infrastructure:

> Initial looks from the firewall team point to an automatic failover event
and the secondary failed.

And the symptoms of which lined up with network level info from Paul
earlier:

> They only seem to have two auth nameservers for faa, both within the
faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
servers are in all die just within each block run by the FAA.
>
> Seems like an internal routing meltdown making the only 2 nameservers
unreachable reliably.

Are you saying that your open resolvers have a per client rate limit
applied, that rate limit got tripped, and shortly thereafter the resolvers
became unavailable, suggesting query floods for the domain(s) that knocked
the resolvers offline (or from the other discussion, possibly was the thing
that overwhelmed that firewall layer, causing the initial failover and
possibly also causing the firewall secondary to fail to come online)?

On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages at outages.org>
wrote:

> Hi, I'm a researcher of DNS vulnerabilities.
>
> It loos like random subdomain attacks (water tourtue attack).
>
> This is the data of my rate-limitted openresolver as a honeypot.
> http://www.e-ontap.com/dns/todaydowngov.txt
> http://www.e-ontap.com/dns/todaydown.txt
> (You can not view these page if you are using 8.8.8.8, sorry.)
>
> Raw logs of my Unbound (Time is JST)
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
> -5
> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
> exceeded ratelimit for zone faa.gov.
> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
> SERVFAIL 15.112813 0 30
> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
> exceeded ratelimit for zone faa.gov.
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> head -5
> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
> all servers for this domain failed, at zone faa.gov. from
> 2620:74:27::2:30 no server to query nameserver addresses not usable
> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
> servers for this domain failed, at zone faa.gov. no server to query
> nameserver addresses not usable
> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
> all servers for this domain failed, at zone faa.gov. upstream server
> timeout
> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
> A IN>: all servers for this domain failed, at zone faa.gov. upstream
> server timeout
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
> tail -5
> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
> servers for this domain failed, at zone faa.gov. no server to query
> nameserver addresses not usable
> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
> A IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
> zone faa.gov. no server to query nameserver addresses not usable
> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> at zone faa.gov. no server to query nameserver addresses not usable
> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
> -5
> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
> at zone faa.gov. no server to query nameserver addresses not usable
> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
> IN>: all servers for this domain failed, at zone faa.gov. no server to
> query nameserver addresses not usable
> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
> IN SERVFAIL 0.000000 0 34
> local/etc/unbound%
> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
> -l
>     1408
>
> --
> T.Suzuki
> --
> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20230326/255eb4da/attachment.htm>