[outages] NTP Issues Today

[Jeremy Chadwick]

I'm still waiting for someone who was affected by this to provide
coherent logs from ntpd showing exactly when the time change happened.
Getting these, at least on an *IX system, is far from difficult folks.

Please don't omit anything from the logs either; for example if you know
*exactly* what NTP servers were in use (not "ones you had configured"
but which one was primarily chosen by ntpd ('*' mark) and which were
secondary comparisons/fallbacks ('+' mark)), that would also be greatly
helpful.  This would be output from "ntpq -c peers" when run on your NTP
server *at or around the time* the incident happened and recovered.

What's been provided so far is that "something happened", with reports
of clocks going back to year 2000, and other reports of clocks going
back to (presumably) epoch time; those reporting it were using either
usno.navy.mil, NIST, or Microsoft NTP servers.  usno.navy.mil uses
dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS.  No idea what
Microsoft uses.

I asked on a public *IX forum if anyone saw anything NTP-wise that was
out of the ordinary and not a single admin saw anything.  I also saw
nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE,
running base system ntpd 4.2.4p8), but I sync with very specific stratum
1 and stratum 2 servers across the United States.

As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd
will not allow large clock jumps -- the largest it'll allow out of the
box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless
you're running with the -g flag (and shame on if you're you doing that).
So I'm very surprised by this problem altogether.  Can't deny what
happened did, but figuring out *why* is important.

Also, for Mike Lyon -- I looked at NIST's GPS graphs.  Did you notice
they have no data for 11/18, 11/19, or 11/20?  I find that unnerving,
do you not?

-- 
| Jeremy Chadwick                                   jdc at koitsu.org |
| UNIX Systems Administrator                http://jdc.koitsu.org/ |
| Mountain View, CA, US                                            |
| Making life hard for others since 1977.             PGP 4BD6C0CB |

On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:
> Same thing happened to us yesterday.  ended up having to reboot everything
> after we got time fixed.  Major outage.
> 
> Scott
> 
> 
> On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao at ctigroup.com> wrote:
> 
> > We had multiple servers synchronized with Windows/MS time change their
> > clock to the year 2000 today.  It broke many things, including AD
> > authentication.
> >
> > These servers had been properly synchronized for years.
> >
> > They were synchronized with Microsoft and NIST NTP servers.
> >
> > This may not be isolated.
> >
> > Sid Rao | CTI Group | +1 (317) 262-4677
> >
> > On Nov 19, 2012, at 10:29 PM, "George Herbert" <george.herbert at gmail.com>
> > wrote:
> >
> > > crossreplying to outages list.
> > >
> > > Is anyone ELSE seeing GPS issues?  This could well have been an
> > > unrelated issue on that particular PBX.
> > >
> > > If this was real, then the mother of all infrastructure attacks might
> > > be underway...
> > >
> > > One glitch on tick and tock and one malfunctioning PBX is not
> > > sufficient evidence of pattern - much less hostile activity - to
> > > induce panic, but it would perhaps be a wise time to check
> > > time-related logs?
> > >
> > >
> > > -george
> > >
> > > On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith
> > > <kwallace at pcconnection.com> wrote:
> > >> Just got paged with a pbx alarm that had 1970 as the year. By the time
> > I logged in , it was showing 2012.  Using GPS for time and date.
> > >>
> > >> -----Original Message-----
> > >> From: Mark Andrews [mailto:marka at isc.org]
> > >> Sent: Monday, November 19, 2012 8:42 PM
> > >> To: Van Wolfe
> > >> Cc: nanog at nanog.org
> > >> Subject: Re: NTP Issues Today
> > >>
> > >>
> > >> In message <
> > CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ at mail.gmail.com>
> > >> , Van Wolfe writes:
> > >>> Hello,
> > >>>
> > >>> Did anyone else experience issues with NTP today?  We had our server
> > >>> times update to the year 2000 at around 3:30 MT, then revert back to
> > 2012.
> > >>>
> > >>> Thanks,
> > >>> Van
> > >>
> > >> NTP should be immune from this sort of behaviour unless you did a
> > ntpdate at the wrong moment.  The clocks should have been marked as insane.
> > >>
> > >> Mark
> > >> --
> > >> Mark Andrews, ISC
> > >> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> > >> PHONE: +61 2 9871 4742                 INTERNET: marka at isc.org
> > >>
> > >>
> > >
> > >
> > >
> > > --
> > > -george william herbert
> > > george.herbert at gmail.com
> > >
> > >
> >
> >
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
> >

> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages