[outages] NTP Issues Today
Josh Luthman
josh at imaginenetworksllc.com
Tue Nov 20 10:44:11 EST 2012
Nov 19 16:04:38 marty ntpd[2214]: synchronized to LOCAL(0), stratum 10
Nov 19 16:18:23 marty named[2160]: zone my.slave.internal.zone/IN/internal:
refresh: non-authoritative answer from master 127.0.0.1#53 (source
0.0.0.0#0)
Nov 19 16:21:51 marty ntpd[2214]: synchronized to 192.5.41.41, stratum 1
#this is where it crashed
Nov 19 16:38:41 marty ntpd[2214]: time correction of -378691201 seconds
exceeds sanity limit (1000); set clock manually to the correct UTC time.
#this is where i restarted ntpd by hand
Nov 19 16:45:53 marty ntpd[5043]: ntpd 4.2.2p1 at 1.1570-o Fri Nov 18 13:21:16
UTC 2011 (1)
Nov 19 16:45:54 marty ntpd[5044]: precision = 1.000 usec
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface wildcard,
0.0.0.0#123 Disabled
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface wildcard, ::#123
Disabled
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface lo, ::1#123 Enabled
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface eth0,
fe80::20c:29ff:fe07:8dc7#123 Enabled
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface lo, 127.0.0.1#123
Enabled
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface eth0,
96.11.78.2#123 Enabled
Nov 19 16:45:54 marty ntpd[5044]: Listening on interface pntun1,
10.255.255.254#123 Enabled
Nov 19 16:45:54 marty ntpd[5044]: kernel time sync status 0040
Nov 19 16:45:54 marty ntpd[5044]: frequency initialized 1.864 PPM from
/var/lib/ntp/drift
Nov 19 16:49:10 marty ntpd[5044]: synchronized to LOCAL(0), stratum 10
Nov 19 16:49:10 marty ntpd[5044]: kernel time sync disabled 0001
Nov 19 17:38:20 marty ntpd[5044]: synchronized to 192.5.41.41, stratum 1
Josh Luthman
Office: 937-552-2340
Direct: 937-552-2343
1100 Wayne St
Suite 1337
Troy, OH 45373
On Tue, Nov 20, 2012 at 10:38 AM, Jeremy Chadwick <jdc at koitsu.org> wrote:
> I'm still waiting for someone who was affected by this to provide
> coherent logs from ntpd showing exactly when the time change happened.
> Getting these, at least on an *IX system, is far from difficult folks.
>
> Please don't omit anything from the logs either; for example if you know
> *exactly* what NTP servers were in use (not "ones you had configured"
> but which one was primarily chosen by ntpd ('*' mark) and which were
> secondary comparisons/fallbacks ('+' mark)), that would also be greatly
> helpful. This would be output from "ntpq -c peers" when run on your NTP
> server *at or around the time* the incident happened and recovered.
>
> What's been provided so far is that "something happened", with reports
> of clocks going back to year 2000, and other reports of clocks going
> back to (presumably) epoch time; those reporting it were using either
> usno.navy.mil, NIST, or Microsoft NTP servers. usno.navy.mil uses
> dedicated IRIG/AFNOR TCRs boxes, while NIST uses GPS. No idea what
> Microsoft uses.
>
> I asked on a public *IX forum if anyone saw anything NTP-wise that was
> out of the ordinary and not a single admin saw anything. I also saw
> nothing anomalous on either of my FreeBSD machines (9.1-PRERELEASE,
> running base system ntpd 4.2.4p8), but I sync with very specific stratum
> 1 and stratum 2 servers across the United States.
>
> As Mark Andrews from the ISC stated below (read slowly/carefully), ntpd
> will not allow large clock jumps -- the largest it'll allow out of the
> box is 1000s (and on some systems like Solaris ntpd, 500s) -- unless
> you're running with the -g flag (and shame on if you're you doing that).
> So I'm very surprised by this problem altogether. Can't deny what
> happened did, but figuring out *why* is important.
>
> Also, for Mike Lyon -- I looked at NIST's GPS graphs. Did you notice
> they have no data for 11/18, 11/19, or 11/20? I find that unnerving,
> do you not?
>
> --
> | Jeremy Chadwick jdc at koitsu.org |
> | UNIX Systems Administrator http://jdc.koitsu.org/ |
> | Mountain View, CA, US |
> | Making life hard for others since 1977. PGP 4BD6C0CB |
>
> On Tue, Nov 20, 2012 at 07:18:45AM -0800, Scott Voll wrote:
> > Same thing happened to us yesterday. ended up having to reboot
> everything
> > after we got time fixed. Major outage.
> >
> > Scott
> >
> >
> > On Mon, Nov 19, 2012 at 7:58 PM, Sid Rao <srao at ctigroup.com> wrote:
> >
> > > We had multiple servers synchronized with Windows/MS time change their
> > > clock to the year 2000 today. It broke many things, including AD
> > > authentication.
> > >
> > > These servers had been properly synchronized for years.
> > >
> > > They were synchronized with Microsoft and NIST NTP servers.
> > >
> > > This may not be isolated.
> > >
> > > Sid Rao | CTI Group | +1 (317) 262-4677
> > >
> > > On Nov 19, 2012, at 10:29 PM, "George Herbert" <
> george.herbert at gmail.com>
> > > wrote:
> > >
> > > > crossreplying to outages list.
> > > >
> > > > Is anyone ELSE seeing GPS issues? This could well have been an
> > > > unrelated issue on that particular PBX.
> > > >
> > > > If this was real, then the mother of all infrastructure attacks might
> > > > be underway...
> > > >
> > > > One glitch on tick and tock and one malfunctioning PBX is not
> > > > sufficient evidence of pattern - much less hostile activity - to
> > > > induce panic, but it would perhaps be a wise time to check
> > > > time-related logs?
> > > >
> > > >
> > > > -george
> > > >
> > > > On Mon, Nov 19, 2012 at 6:08 PM, Wallace Keith
> > > > <kwallace at pcconnection.com> wrote:
> > > >> Just got paged with a pbx alarm that had 1970 as the year. By the
> time
> > > I logged in , it was showing 2012. Using GPS for time and date.
> > > >>
> > > >> -----Original Message-----
> > > >> From: Mark Andrews [mailto:marka at isc.org]
> > > >> Sent: Monday, November 19, 2012 8:42 PM
> > > >> To: Van Wolfe
> > > >> Cc: nanog at nanog.org
> > > >> Subject: Re: NTP Issues Today
> > > >>
> > > >>
> > > >> In message <
> > > CAMeggd4cDQwhxQE_JbvpNR-PKKe9LXqA+KzJ97anHFonjwZhdQ at mail.gmail.com>
> > > >> , Van Wolfe writes:
> > > >>> Hello,
> > > >>>
> > > >>> Did anyone else experience issues with NTP today? We had our
> server
> > > >>> times update to the year 2000 at around 3:30 MT, then revert back
> to
> > > 2012.
> > > >>>
> > > >>> Thanks,
> > > >>> Van
> > > >>
> > > >> NTP should be immune from this sort of behaviour unless you did a
> > > ntpdate at the wrong moment. The clocks should have been marked as
> insane.
> > > >>
> > > >> Mark
> > > >> --
> > > >> Mark Andrews, ISC
> > > >> 1 Seymour St., Dundas Valley, NSW 2117, Australia
> > > >> PHONE: +61 2 9871 4742 INTERNET: marka at isc.org
> > > >>
> > > >>
> > > >
> > > >
> > > >
> > > > --
> > > > -george william herbert
> > > > george.herbert at gmail.com
> > > >
> > > >
> > >
> > >
> > > _______________________________________________
> > > Outages mailing list
> > > Outages at outages.org
> > > https://puck.nether.net/mailman/listinfo/outages
> > >
>
> > _______________________________________________
> > Outages mailing list
> > Outages at outages.org
> > https://puck.nether.net/mailman/listinfo/outages
>
> _______________________________________________
> Outages mailing list
> Outages at outages.org
> https://puck.nether.net/mailman/listinfo/outages
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://puck.nether.net/pipermail/outages/attachments/20121120/90f55d0d/attachment.htm>
More information about the Outages
mailing list