a. ensures ntpd is running and starts it if its not.
b. checks `nptq -rv` and looks for the self-reported state of ntp and bounces if it is not sync'd.
c. 'manually' checks ntpdate for the drift against a known stratum 1 and bounces the daemon if the drift is out of bounds (something like 50ms is a pretty good cutoff since ntpd normally does much better than that).
You can run this out of chef/puppet/whatever but it needs to run with a frequency of about an hour (faster than that and ntpd often won't settle well, and slower than that and you can be needlessly waiting too long to fix problems). Running it as a cronjob uncouples it from how often your config management runs.
This will catch all kinds of issues -- crashing ntpds, ntdps that randomly lose sync, kernel problems that keep them from keeping sync, ntpds that lie about their sync status, etc.
I set it up to e-mail me as a form of monitoring (with heavy-duty procmail filters).
Had 30,000 servers with this script running against it. I think 6 of them had shitty clocks that were getting reset every single hour, but were still keeping time correctly to under a second. Every couple days there would be a server that'd flip out and send a few e-mails until it quieted down and synch'd. At one point we had a shipment of several hundred new servers that all had issues with kernel drivers and the cronjob was bouncing ntpd constantly on them until a kernel upgrade fixed the problem.
I also used to bounce ntpd once a night, but had to take that out because it would cause non-monotonic slew in the clock which timers around service calls would turn up as negative seconds that due to unsigned int conversion would turn into 4 billion second p100 times.
You do want to be careful about things like network outages, if you can't ping your upstream stratum1/2 its probably better to leave things as it is. And again, you really want to limit the frequency that you bounce it at, and you want to be aware of servers where it gets into a state where you're bouncing ntpd constantly.
This is a terrible idea when dealing with an application other than "users checking the time on their system clock".
Performing hard resets of the clock continually is a catastrophic infrastructure failure. A high performance 24/7 application that relies on sub millisecond let alone sub microsecond accuracy across multiple physical systems will be utterly destroyed by this approach.
To elaborate, monotonicity means that a value is either always increasing or always decreasing, and never changes direction. If a program is built on the assumption that time always goes forward, it may not like you setting the clock back 0.000000001sec every hour.
Very true, a lot of software with time limited trials or features like "check out a network license for 30 days of offline use" used to be easy to bypass by turning your clock back so that it never expired.
This is pretty widely fixed now, but often overzealously. Clock changed unexpectedly? User is a dirty pirate and the license should deactivate.
I had 30,000 servers, all but 6 of them were not getting hard resets. 6 of them were getting hard resets every hour because they were sick.
Occasionally a ntpd on a server would go a little crazy and it would get a few hard resets. This script got it back into spec so that it didn't need hard resets.
I mentioned specifically that you can't send hard resets to ntpd more than about once an hour or it won't have time to settle and you'll wind up in a situation where you are constantly sending hard resets to it (which is bad).
I took out the nightly forced hard resets of ntpd specifically because it affected monotonic time.
If you look at the algorithm I wound up with ntpd will only be reset if:
a. it is already crashed and not running
b. it is self-reporting that it is out of synch
c. is is more than 50ms out of synch with an upstream time server
In all of those cases, you have already lost the battle to get sub-microsecond accuracy.
a. ensures ntpd is running and starts it if its not.
b. checks `nptq -rv` and looks for the self-reported state of ntp and bounces if it is not sync'd.
c. 'manually' checks ntpdate for the drift against a known stratum 1 and bounces the daemon if the drift is out of bounds (something like 50ms is a pretty good cutoff since ntpd normally does much better than that).
You can run this out of chef/puppet/whatever but it needs to run with a frequency of about an hour (faster than that and ntpd often won't settle well, and slower than that and you can be needlessly waiting too long to fix problems). Running it as a cronjob uncouples it from how often your config management runs.
This will catch all kinds of issues -- crashing ntpds, ntdps that randomly lose sync, kernel problems that keep them from keeping sync, ntpds that lie about their sync status, etc.
I set it up to e-mail me as a form of monitoring (with heavy-duty procmail filters).
Had 30,000 servers with this script running against it. I think 6 of them had shitty clocks that were getting reset every single hour, but were still keeping time correctly to under a second. Every couple days there would be a server that'd flip out and send a few e-mails until it quieted down and synch'd. At one point we had a shipment of several hundred new servers that all had issues with kernel drivers and the cronjob was bouncing ntpd constantly on them until a kernel upgrade fixed the problem.
I also used to bounce ntpd once a night, but had to take that out because it would cause non-monotonic slew in the clock which timers around service calls would turn up as negative seconds that due to unsigned int conversion would turn into 4 billion second p100 times.
You do want to be careful about things like network outages, if you can't ping your upstream stratum1/2 its probably better to leave things as it is. And again, you really want to limit the frequency that you bounce it at, and you want to be aware of servers where it gets into a state where you're bouncing ntpd constantly.