[outages] ServiceNow DNSSEC issues?

[Michael Sinatra]

Looks like ServiceNow may have recently botched a KSK roll.

8.8.8.8 is intermittently giving out SERVFAIL responses, and my own 
resolvers had SERVFAILs (we are a SN shop), starting around 1545 US/Pacific.

Examining the cache (and DNSVIZ) shows that at some point within the 
last 3 hours, the DS record for service-now.com switched from having 
keytag 30126 to keytag 31893.  The DS record has a TTL of 24 hours.

It appears that at the same time, or a *short* (under 3 hours) time 
later, the DNSKEY record for KSK with tag 30126 was removed from 
service-now.com.  The DNSKEY records have a TTL of 2 hours.

This would have caused systems to continue to reference the cached DS 
record to fail validation after the DNSKEY cache TTL expired.  (The 
DNSKEY would have needed to be retained for at least a full TTL, and 
probably 2xTTL to be on the safe side.)  Again, this is causing 
intermittent breakage of name resolution in service-now.com.

Appendix: repeated queries of the DS record for service-now.com to 8.8.8.8:

;; ANSWER SECTION:
service-now.com.        86395   IN      DS      31893 7 1 
18035BB11DFF4E2B75D677724833E8FDB9FAEC13

;; Query time: 6 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Jan 28 00:33:58 UTC 2017

;; ANSWER SECTION:
service-now.com.        60788   IN      DS      30126 7 1 
8EB362449D1500DFFD359200F958471D915F83A8

;; Query time: 24 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Jan 28 00:33:59 UTC 2017
;; MSG SIZE  rcvd: 80

I'll spare you the dig output, but suffice it to say that the auth 
servers for service-now.com don't have KSK 30126 in their DNSKEY RRSET.

michael