[outages] DNS SERVFAIL for nist.gov

Mon Jun 14 12:19:30 EDT 2021

Matt,

I support many financial networks, and comply with the same FINRA rule 6820 you do. This rule doesn’t state that the time must be synchronized over the Internet using public IP addresses, but only “to within a fifty (50) millisecond tolerance of the time maintained by the atomic clock” at NIST. Because GPS clock times are legally traceable to NIST, and deviation logs are available in real-time, there is no reason to depend on IP-based NTP over the Internet, and good reason not to as today’s event demonstrates. The 1ms accuracy is well within the 50ms limit.

 -mel

On Jun 14, 2021, at 7:06 AM, Matthew Huff <mhuff at ox.com> wrote:

Of course.

Like I stated in the original email, we don’t use NIST for time sync. We actually have a GPS and a PTP feed.

WE MUST, HOWEVER, VALIDATE OUR SYSTEM TIMES VERSUS THE INTERNET NIST SERVERS VIA FINRA REGULATIONS

Yes, it is stupid
Yes, it isn’t a good idea
But if we don’t want to get fined and prevented from trading, we have to follow FINRA regulations.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mhuff at ox.com<mailto:mhuff at ox.com> | www.ox.com<http://www.ox.com>
...........................................................................................................................................

From: Mel Beckman <mel at beckman.org>
Sent: Monday, June 14, 2021 9:40 AM
To: Matthew Huff <mhuff at ox.com>
Cc: outages at outages.org
Subject: Re: [outages] DNS SERVFAIL for nist.gov

You should consider eliminating dependence on Internet-delivered NIST time and switch to GPS-based time servers. The GPS network has its own airborne atomic clocks that use a well-disciplined protocol to synchronize to NIST reference atomic time without transiting the Internet.

According to NIST’s documentation:

“Currently, the GPS system provides time to the general public with uncertainties measured in nanoseconds. With a well-designed receiver system the user can obtain the time to better than 100 ns in a few minutes, and to about +/- 10 ns with a 24 hour average (and a good local clock).”

All sources of error in GPS time propagation total less than one millisecond, well within your 50ms tolerance.

https://www.nist.gov/pml/time-and-frequency-division/time-services/one-way-gps-time-transfer

NIST maintains publicly-accessible logs of al clock differences to provide documented compliance under FINRA clock synchronization rules.

<image001.jpg>

The log has a one-hour resolution, satisfying the FINRA requirement to verify synchronization “throughout the day”.

IP-based GPS clocks are widely available with low-drift oven-controlled crystal oscillators (OXCO), or even internal cesium-based atomic clocks, for as little as a few thousand dollars. This lets you ride out time signal outages of days or even weeks.

The US DHS recommends discontinuation of unauthenticated Internet-based reference clocks, owing to their vulnerability to IP address spoofing:

https://www.dhs.gov/sites/default/files/publications/GPS-PNT-Best-Practices-Time-Frequency-Sources-Fixed-Locations-508.pdf

-mel via cell

On Jun 14, 2021, at 3:51 AM, Matthew Huff via Outages <outages at outages.org<mailto:outages at outages.org>> wrote:
We have to query and compare against NIST time servers for FINRA compliance This morning I noticed our systems are unable to DNS query the NIST time servers. Neither our local resolvers or google (8.8.8.8) work.

[root at bacall log]# dig @8.8.8.8 time-a-g.nist.gov

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 time-a-g.nist.gov
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 36018
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;time-a-g.nist.gov.             IN      A

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:27:45 EDT 2021
;; MSG SIZE  rcvd: 46

[root at bacall log]# dig @8.8.8.8 nist.gov in soa

; <<>> DiG 9.11.4-P2-RedHat-9.11.4-26.P2.el7_9.5 <<>> @8.8.8.8 nist.gov in soa
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 17779
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;nist.gov.                      IN      SOA

;; Query time: 5 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Mon Jun 14 06:31:59 EDT 2021
;; MSG SIZE  rcvd: 37

The time servers are documented here: https://tf.nist.gov/tf-cgi/servers.cgi

Using the IP addresses work, it look like the nist.gov domain is offline.

Matthew Huff | Director of Technical Operations | OTA Management LLC

Office: 914-460-4039
mhuff at ox.com<mailto:mhuff at ox.com> | www.ox.com<http://www.ox.com>
.........................................................................................................................................

_______________________________________________
Outages mailing list
Outages at outages.org<mailto:Outages at outages.org>
https://puck.nether.net/mailman/listinfo/outages
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20210614/116d409b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.jpg
Type: image/jpeg
Size: 631916 bytes
Desc: image001.jpg
URL: <https://puck.nether.net/pipermail/outages/attachments/20210614/116d409b/attachment-0001.jpg>