[outages] FAA.gov nameserver outage
Jay R. Ashworth
jra at baylink.com
Fri Mar 31 13:13:43 EDT 2023
As is Generalissimo Francisco Franco.
Indeed, folks; please move these meta conversations to the -discuss list; they
are off topic for the main notification list.
Cheers,
-- jr '<admin/>' a
----- Original Message -----
> From: "Mike Lyon via Outages" <outages at outages.org>
> To: "T.Suzuki" <tss-outage at e-ontap.com>
> Cc: "Michael Loftis via Outages" <outages at outages.org>
> Sent: Sunday, March 26, 2023 8:17:25 PM
> Subject: Re: [outages] FAA.gov nameserver outage
> Can’t believe it’s still dead…
>
> -Mike
>
>> On Mar 26, 2023, at 17:13, T.Suzuki via Outages <outages at outages.org> wrote:
>>
>> On Sun, 26 Mar 2023 08:35:29 -0700
>> Hugo Slabbert <hugo at slabnet.com> wrote:
>>
>>> What would be the symptoms here of a "water torture attack" rather than
>>> what John had indicated as a firewall failure in their infrastructure:
>>>
>>>> Initial looks from the firewall team point to an automatic failover event
>>> and the secondary failed.
>>>
>>> And the symptoms of which lined up with network level info from Paul
>>> earlier:
>>>
>>>> They only seem to have two auth nameservers for faa, both within the
>>> faa.gov domain. Don't seem to be anycasted and the 2 v4 and 2 v6 blocks the
>>> servers are in all die just within each block run by the FAA.
>>>>
>>>> Seems like an internal routing meltdown making the only 2 nameservers
>>> unreachable reliably.
>>>
>>> Are you saying that your open resolvers have a per client rate limit
>>> applied, that rate limit got tripped, and shortly thereafter the resolvers
>>> became unavailable, suggesting query floods for the domain(s) that knocked
>>> the resolvers offline (or from the other discussion, possibly was the thing
>>> that overwhelmed that firewall layer, causing the initial failover and
>>> possibly also causing the firewall secondary to fail to come online)?
>>
>> Yes. (limitting per client, and per second for all)
>> Perhaps, large numbers open resolvers including no ratelimit are used.
>> Then massive random subdomain queries caused the firewall symptoms.
>> (It's only my guess.)
>>
>>>> On Sun, Mar 26, 2023, 01:13 T.Suzuki via Outages <outages at outages.org>
>>>> wrote:
>>>>
>>>> Hi, I'm a researcher of DNS vulnerabilities.
>>>>
>>>> It loos like random subdomain attacks (water tourtue attack).
>>>>
>>>> This is the data of my rate-limitted openresolver as a honeypot.
>>>> http://www.e-ontap.com/dns/todaydowngov.txt
>>>> http://www.e-ontap.com/dns/todaydown.txt
>>>> (You can not view these page if you are using 8.8.8.8, sorry.)
>>>>
>>>> Raw logs of my Unbound (Time is JST)
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | head
>>>> -5
>>>> Mar 26 12:00:35 unbound[48103:0] error: SERVFAIL <
>>>> unnamed568.orphaned.faa.gov. A IN>: exceeded ratelimit for zone faa.gov.
>>>> Mar 26 12:00:35 unbound[48103:0] reply: 24.199.82.210
>>>> unnamed568.orphaned.faa.gov. A IN SERVFAIL 9.226781 0 45
>>>> Mar 26 12:04:31 unbound[48103:0] error: SERVFAIL <amax.faa.gov. A IN>:
>>>> exceeded ratelimit for zone faa.gov.
>>>> Mar 26 12:04:31 unbound[48103:0] reply: 24.199.82.210 amax.faa.gov. A IN
>>>> SERVFAIL 15.112813 0 30
>>>> Mar 26 12:04:37 unbound[48103:0] error: SERVFAIL <dallatx.faa.gov. A IN>:
>>>> exceeded ratelimit for zone faa.gov.
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
>>>> head -5
>>>> Mar 26 12:05:26 unbound[48103:0] error: SERVFAIL <epoxy.faa.gov. A IN>:
>>>> all servers for this domain failed, at zone faa.gov. from
>>>> 2620:74:27::2:30 no server to query nameserver addresses not usable
>>>> Mar 26 12:05:27 unbound[48103:0] error: SERVFAIL <lyndas365project.faa.gov.
>>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> Mar 26 12:05:28 unbound[48103:0] error: SERVFAIL <lmn.faa.gov. A IN>: all
>>>> servers for this domain failed, at zone faa.gov. no server to query
>>>> nameserver addresses not usable
>>>> Mar 26 12:05:30 unbound[48103:0] error: SERVFAIL <host244.faa.gov. A IN>:
>>>> all servers for this domain failed, at zone faa.gov. upstream server
>>>> timeout
>>>> Mar 26 12:05:33 unbound[48103:0] error: SERVFAIL <leased-line188.faa.gov.
>>>> A IN>: all servers for this domain failed, at zone faa.gov. upstream
>>>> server timeout
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all servers" |
>>>> tail -5
>>>> Mar 26 13:41:08 unbound[48103:0] error: SERVFAIL <asm.faa.gov. A IN>: all
>>>> servers for this domain failed, at zone faa.gov. no server to query
>>>> nameserver addresses not usable
>>>> Mar 26 13:41:15 unbound[48103:0] error: SERVFAIL <sas-uss.edc.nas.faa.gov.
>>>> A IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> Mar 26 13:41:22 unbound[48103:0] error: SERVFAIL <
>>>> eforms-stagedev.hq.faa.gov. A IN>: all servers for this domain failed, at
>>>> zone faa.gov. no server to query nameserver addresses not usable
>>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
>>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
>>>> at zone faa.gov. no server to query nameserver addresses not usable
>>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
>>>> IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "SERVFAIL" | tail
>>>> -5
>>>> Mar 26 13:41:22 unbound[48103:0] reply: 24.199.82.210
>>>> eforms-stagedev.hq.faa.gov. A IN SERVFAIL 0.000000 0 44
>>>> Mar 26 13:41:23 unbound[48103:0] error: SERVFAIL <
>>>> faardm-mceast2.idrac.faa.gov. A IN>: all servers for this domain failed,
>>>> at zone faa.gov. no server to query nameserver addresses not usable
>>>> Mar 26 13:41:23 unbound[48103:0] reply: 24.199.82.210
>>>> faardm-mceast2.idrac.faa.gov. A IN SERVFAIL 0.000000 0 46
>>>> Mar 26 13:41:28 unbound[48103:0] error: SERVFAIL <chronos3.faa.gov. A
>>>> IN>: all servers for this domain failed, at zone faa.gov. no server to
>>>> query nameserver addresses not usable
>>>> Mar 26 13:41:28 unbound[48103:0] reply: 24.199.82.210 chronos3.faa.gov. A
>>>> IN SERVFAIL 0.000000 0 34
>>>> local/etc/unbound%
>>>> local/etc/unbound% grep "\.faa\.gov" unbound.log | grep "all server" | wc
>>>> -l
>>>> 1408
>>>>
>>>> --
>>>> T.Suzuki
>>>> --
>>>> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
>>>> _______________________________________________
>>>> Outages mailing list
>>>> Outages at outages.org
>>>> https://puck.nether.net/mailman/listinfo/outages
>>>>
>>
>>
>> --
>> T.Suzuki / E.F.シューマッハーとI.イリイチを読もう
>> _______________________________________________
>> Outages mailing list
>> Outages at outages.org
>> https://puck.nether.net/mailman/listinfo/outages
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
--
Jay R. Ashworth Baylink jra at baylink.com
Designer The Things I Think RFC 2100
Ashworth & Associates http://www.bcp38.info 2000 Land Rover DII
St Petersburg FL USA BCP38: Ask For It By Name! +1 727 647 1274
More information about the Outages
mailing list