<div>Nov 19 16:04:38 marty ntpd[2214]: synchronized to LOCAL(0), stratum 10</div><div>Nov 19 16:18:23 marty named[2160]: zone my.slave.internal.zone/IN/internal: refresh: non-authoritative answer from master 127.0.0.1#53 (source 0.0.0.0#0)</div>
<div>Nov 19 16:21:51 marty ntpd[2214]: synchronized to 192.5.41.41, stratum 1</div><div>#this is where it crashed</div><div>Nov 19 16:38:41 marty ntpd[2214]: time correction of -378691201 seconds exceeds sanity limit (1000); set clock manually to the correct UTC time.</div>
<div>#this is where i restarted ntpd by hand</div><div>Nov 19 16:45:53 marty ntpd[5043]: ntpd 4.2.2p1@1.1570-o Fri Nov 18 13:21:16 UTC 2011 (1)</div><div>Nov 19 16:45:54 marty ntpd[5044]: precision = 1.000 usec</div><div>
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface wildcard, 0.0.0.0#123 Disabled</div><div>Nov 19 16:45:54 marty ntpd[5044]: Listening on interface wildcard, ::#123 Disabled</div><div>Nov 19 16:45:54 marty ntpd[5044]: Listening on interface lo, ::1#123 Enabled</div>
<div>Nov 19 16:45:54 marty ntpd[5044]: Listening on interface eth0, fe80::20c:29ff:fe07:8dc7#123 Enabled</div><div>Nov 19 16:45:54 marty ntpd[5044]: Listening on interface lo, 127.0.0.1#123 Enabled</div><div>Nov 19 16:45:54 marty ntpd[5044]: Listening on interface eth0, 96.11.78.2#123 Enabled</div>
<div>Nov 19 16:45:54 marty ntpd[5044]: Listening on interface pntun1, 10.255.255.254#123 Enabled</div><div>Nov 19 16:45:54 marty ntpd[5044]: kernel time sync status 0040</div><div>Nov 19 16:45:54 marty ntpd[5044]: frequency initialized 1.864 PPM from /var/lib/ntp/drift</div>
<div>Nov 19 16:49:10 marty ntpd[5044]: synchronized to LOCAL(0), stratum 10</div><div>Nov 19 16:49:10 marty ntpd[5044]: kernel time sync disabled 0001</div><div>Nov 19 17:38:20 marty ntpd[5044]: synchronized to 192.5.41.41, stratum 1</div>
<div class="gmail_extra"><br clear="all">Josh Luthman<br>Office: 937-552-2340<br>Direct: 937-552-2343<br>1100 Wayne St<br>Suite 1337<br>Troy, OH 45373<br>
<br><br><div class="gmail_quote">On Tue, Nov 20, 2012 at 10:38 AM, Jeremy Chadwick <span dir="ltr"><<a href="mailto:jdc@koitsu.org" target="_blank">jdc@koitsu.org</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm still waiting for someone who was affected by this to provide<br>
coherent logs from ntpd showing exactly when the time change happened.<br>
Getting these, at least on an *IX system, is far from difficult folks.<br>
<br>
Please don't omit anything from the logs either; for example if you know<br>
*exactly* what NTP servers were in use (not "ones you had configured"<br>
but which one was primarily chosen by ntpd ('*' mark) and which were<br>
secondary comparisons/fallbacks ('+' mark)), that would also be greatly<br>
helpful. This would be output from "ntpq -c peers" when run on your NTP<br>
server *at or around the time* the incident happened and recovered.<br>
<br>
What's been provided so far is that "something happened", with reports<br>
of clocks going back to year 2000, and other reports of clocks going<br>
back to (presumably) epoch time; those reporting it were using either<br>
<a href="http://usno.navy.mil" target="_blank">usno.navy.mil</a>, NIST, or Microsoft NTP servers. <a href="http://usno.navy.mil" target="_blank">usno.navy.mil</a> uses<br>
dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS. No idea what<br>
Microsoft uses.<br>
<br>
I asked on a public *IX forum if anyone saw anything NTP-wise that was<br>
out of the ordinary and not a single admin saw anything. I also saw<br>
nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE,<br>
running base system ntpd 4.2.4p8), but I sync with very specific stratum<br>
1 and stratum 2 servers across the United States.<br>
<br>
As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd<br>
will not allow large clock jumps -- the largest it'll allow out of the<br>
box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless<br>
you're running with the -g flag (and shame on if you're you doing that).<br>
So I'm very surprised by this problem altogether. Can't deny what<br>
happened did, but figuring out *why* is important.<br>
<br>
Also, for Mike Lyon -- I looked at NIST's GPS graphs. Did you notice<br>
they have no data for 11/18, 11/19, or 11/20? I find that unnerving,<br>
do you not?<br>
<div class="im HOEnZb"><br>
--<br>
| Jeremy Chadwick <a href="mailto:jdc@koitsu.org">jdc@koitsu.org</a> |<br>
| UNIX Systems Administrator <a href="http://jdc.koitsu.org/" target="_blank">http://jdc.koitsu.org/</a> |<br>
| Mountain View, CA, US |<br>
| Making life hard for others since 1977. PGP 4BD6C0CB |<br>
<br>
</div><div class="HOEnZb"><div class="h5">On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:<br>
> Same thing happened to us yesterday. ended up having to reboot everything<br>
> after we got time fixed. Major outage.<br>
><br>
> Scott<br>
><br>
><br>
> On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <<a href="mailto:srao@ctigroup.com">srao@ctigroup.com</a>> wrote:<br>
><br>
> > We had multiple servers synchronized with Windows/MS time change their<br>
> > clock to the year 2000 today. It broke many things, including AD<br>
> > authentication.<br>
> ><br>
> > These servers had been properly synchronized for years.<br>
> ><br>
> > They were synchronized with Microsoft and NIST NTP servers.<br>
> ><br>
> > This may not be isolated.<br>
> ><br>
> > Sid Rao | CTI Group | <a href="tel:%2B1%20%28317%29%20262-4677" value="+13172624677">+1 (317) 262-4677</a><br>
> ><br>
> > On Nov 19, 2012, at 10:29 PM, "George Herbert" <<a href="mailto:george.herbert@gmail.com">george.herbert@gmail.com</a>><br>
> > wrote:<br>
> ><br>
> > > crossreplying to outages list.<br>
> > ><br>
> > > Is anyone ELSE seeing GPS issues? This could well have been an<br>
> > > unrelated issue on that particular PBX.<br>
> > ><br>
> > > If this was real, then the mother of all infrastructure attacks might<br>
> > > be underway...<br>
> > ><br>
> > > One glitch on tick and tock and one malfunctioning PBX is not<br>
> > > sufficient evidence of pattern - much less hostile activity - to<br>
> > > induce panic, but it would perhaps be a wise time to check<br>
> > > time-related logs?<br>
> > ><br>
> > ><br>
> > > -george<br>
> > ><br>
> > > On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith<br>
> > > <<a href="mailto:kwallace@pcconnection.com">kwallace@pcconnection.com</a>> wrote:<br>
> > >> Just got paged with a pbx alarm that had 1970 as the year. By the time<br>
> > I logged in , it was showing 2012. Using GPS for time and date.<br>
> > >><br>
> > >> -----Original Message-----<br>
> > >> From: Mark Andrews [mailto:<a href="mailto:marka@isc.org">marka@isc.org</a>]<br>
> > >> Sent: Monday, November 19, 2012 8:42 PM<br>
> > >> To: Van Wolfe<br>
> > >> Cc: <a href="mailto:nanog@nanog.org">nanog@nanog.org</a><br>
> > >> Subject: Re: NTP Issues Today<br>
> > >><br>
> > >><br>
> > >> In message <<br>
> > <a href="mailto:CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA%2BKzJ97anHFonjwZhdQ@mail.gmail.com">CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ@mail.gmail.com</a>><br>
> > >> , Van Wolfe writes:<br>
> > >>> Hello,<br>
> > >>><br>
> > >>> Did anyone else experience issues with NTP today? We had our server<br>
> > >>> times update to the year 2000 at around 3:30 MT, then revert back to<br>
> > 2012.<br>
> > >>><br>
> > >>> Thanks,<br>
> > >>> Van<br>
> > >><br>
> > >> NTP should be immune from this sort of behaviour unless you did a<br>
> > ntpdate at the wrong moment. The clocks should have been marked as insane.<br>
> > >><br>
> > >> Mark<br>
> > >> --<br>
> > >> Mark Andrews, ISC<br>
> > >> 1 Seymour St., Dundas Valley, NSW 2117, Australia<br>
> > >> PHONE: <a href="tel:%2B61%202%209871%204742" value="+61298714742">+61 2 9871 4742</a> INTERNET: <a href="mailto:marka@isc.org">marka@isc.org</a><br>
> > >><br>
> > >><br>
> > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > -george william herbert<br>
> > > <a href="mailto:george.herbert@gmail.com">george.herbert@gmail.com</a><br>
> > ><br>
> > ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Outages mailing list<br>
> > <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> > <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
> ><br>
<br>
> _______________________________________________<br>
> Outages mailing list<br>
> <a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
> <a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
<br>
_______________________________________________<br>
Outages mailing list<br>
<a href="mailto:Outages@outages.org">Outages@outages.org</a><br>
<a href="https://puck.nether.net/mailman/listinfo/outages" target="_blank">https://puck.nether.net/mailman/listinfo/outages</a><br>
</div></div></blockquote></div><br></div>