System administrators generally like things to be predictable and to function in a regular, organized fashion. This attitude undoubtedly stems from the fact that we never have enough time to do everything that we need to get done. Indeed, we often feel that we have no control over our time. Ironically, in this column, we will look at a facility for controlling time. If we can't control our own time, at least we can control time on the computers that we manage.
|Figure One: The atomic time PC desktop clock.|
System administrators generally like things to be predictable and to function in a regular, organized fashion. This attitude undoubtedly stems from the fact that we never have enough time to do everything that we need to get done. Indeed, we often feel that we have no control over our time. Ironically, in this column, we will look at a facility for controlling time. If we can’t control our own time, at least we can control time on the computers that we manage.
One of the important system administration meta-tasks is keeping all of the computers running smoothly and communicating efficiently. However, sometimes the systems themselves seem to be trying to thwart efforts to achieve this goal. One of the ways that communication can be hindered is by different computers having different ideas about what time it is. The Network Time Protocol (NTP) was designed to remedy this situation.
You may wonder how computer clocks get out of sync in the first place. Computers contain a quartz oscillator and associated hardware to interface to the CPU, and these components interrupt the CPU every few milliseconds. In this way, the CPU receives the clock ticks upon which it depends. Instability in the quartz oscillator (due to temperature changes for example), as well as latencies in computer hardware and software, cause errors in the system clock (known as wander and jitter, respectively). Thus, over time, the clock settings of different computers initially set to the same time will diverge, since the errors introduced by their respective hardware will be different.
NTP is designed to deal with these realities in a very sophisticated manner. Originating in 1980, it was designed and written by Professor David L. Mills of the University of Delaware and his students. This protocol provides time synchronization for all of the computers within a network; it is constructed to be both fault-tolerant and scalable to very large networks. It also includes features for authentication between clients and servers, and for collecting and displaying various statistics about its operations. The protocol has a target precision of about 232 picoseconds.
How NTP Works
NTP operates in a hierarchical client/ server fashion, with authoritative time values moving down from the top-level servers through lower-level servers and then to clients. The entire scheme is based upon the availability of what it calls “stratum 1 servers” — servers that receive current time updates from a known-to-be-reliable source, such as an attached reference clock. Servers that receive time values from these servers are known as stratum 2 servers (and so on down the server hierarchy).
There are several options for obtaining authoritative time. An external atomic clock may be used to generate these time values. Or, you can connect to NIST by modem and receive them. And, there are Global Positioning System (GPS) based devices that can receive time values, as well as positioning information, from satellite sources. Alternatively, you can obtain the authoritative time values for your network from external stratum 1 NTP servers on the Internet. The latter option is the most common practice for organizations that do not require the extreme precision needed for a few real-time applications (e.g. air traffic control). The Web page http://www.eecis.udel.edu/~mills/ntp/servers.htm contains links to lists of Internet-accessible stratum 1 and 2 servers; for most sites, a stratum 2 server is sufficient. Note that some servers require advance permission before you may connect to them, so check the requirements carefully before setting up a connection to an Internet NTP server. As recently as a couple of years ago, there were more than 300 stratum 1 servers and more than 20,000 stratum 2 servers.
In client mode, NTP makes periodic adjustments to the system clock based upon the authoritative time data that it receives from the relevant servers. If the current time on the system differs from the correct time by more than 128 milliseconds, NTP will reset the system clock. However, in its normal mode of operation, NTP makes adjustments to the system clock gradually by adjusting the parameters of the software clock to achieve the needed correction. For example, NTP may change the virtual frequency of the clock to make it go slightly faster or slightly slower until the time is correct.
In addition, over time, the NTP daemon on the system will record and analyze successive time errors (known as clock drift) and will continue to correct the time automatically based upon this data even when it cannot reach its timeserver system. This entire process is known as disciplining the system clock.
In actual practice, NTP requires multiple sources of authoritative time. This strategy is used to protect against both single points of failure and the possible unreliability of any single server (due to hardware failure, malicious tampering, and so on). In other words, NTP views all time data with a certain amount of distrust; its algorithms prefer at least three different time sources. Each distinct server is sampled multiple times; the NTP algorithms determine the best value to use for the current time from all of this data (naturally taking into account the network latency, which is the amount of time required for the time value to be transmitted from the remote server to the local system). This value is then used to adjust the time on the local system as described above. In a similar manner, client systems may also be configured to request time data from multiple NTP servers.
All in all, NTP functions extremely well, and all of the systems within a network can achieve synchronization to within a few milliseconds of one another. This level of accuracy is more than sufficient for most organizations.
Setting UP NTP
The first step for implementing NTP within your network is, as usual, planning. You’ll need to decide several things, including how and where to obtain authoritative time values, the placement of NTP servers within the network, and which clients will connect to which servers. To begin with, you might connect one or two local servers to three external stratum 2 servers; the local servers will become the top-level NTP servers within your organization. Then, you can connect clients to the servers, and time synchronization will begin. If it becomes necessary, you can always add additional top-level servers later, or even another layer of servers within your organization, that will use the externally connected ones as their authoritative time source.
Within an individual system, the NTP facility consists of a daemon process, a boot script, a configuration file, several log files, and a few utilities. Installing it is very easy. You can either download and build the package from source code or install it from your Linux distribution’s CD set (many distributions include NTP among their packages but most do not install it by default).
Once the software is installed, the next step is to configure the facility. NTP’s configuration file is conventionally /etc/ntp.conf. Here is a very simple sample file for a client system:
The first line specifies a server to use when obtaining time data. Remaining lines specify locations of NTP’s log file and drift file, respectively (the latter stores data about local clock errors for future time corrections).
The configuration files for servers will also include a server entry for their sources of time data. In addition, they may include lines like this one:
This entry indicates that the indicated server is a peer — a computer with which the local system will exchange (send and receive) time data. Generally, the top-level servers within the organization can be configured as peers — functioning as both clients and servers with respect to one another, and as servers with respect to general client systems. The key keyword is used to specify an authentication key for this connection (discussed below).
If a server has a reference clock connected to it, the server entry within its configuration file is somewhat different. It should look something like the following:
server 127.127.8.0 mode 5
Reference clocks are usually connected via a serial line. They are specified with an IP address beginning with 127.127. The final two components of the IP address indicate the type of device and unit number, respectively.
NTP also includes an authentication facility that enables clients and servers to verify that they are communicating with known and trusted computers. The facility is based upon a private key scheme; keys are conventionally stored in the file /etc/ntp.keys. This file can contain up to 65,536 32-bit keys. When in use, the facility adds several lines to the configuration file:
trustedkey 1 2 3 4 5 6 7 15
request key 15
control key 15
The first line identifies the NTP key file. The second line activates the indicated keys within the file. The last two lines specify which key to use for NTP queries and configuration changes where the indicated key functions as a password in those contexts (corresponding to the ntpdc and ntpq utilities respectively). Once specified and enabled, these keys may be used with the key keyword in server and peer entries.
The most recent versions of NTP also include an additional authentication option referred to as the autokey mechanism. This scheme is designed for NTP’s multicast mode, in which time data is simply broadcasted rather than being explicitly exchanged between clients and servers. Using it, clients can generate session keys that can be used to confirm the authenticity of received data.
Once configured, the NTP daemon must be started at boot time. This is accomplished via a boot script within the usual /etc/rc.d script hierarchy (included as part of the NTP package). In addition, for client machines, the system time may also be explicitly synchronized to that of its server by running the ntpdate utility included with the package. This command takes a form similar to the following:
ntpdate -bs 192.168.15.56
where the -b option says to set the system time explicitly (rather than adjusting it in the normal manner), and the -s option says to send the command output to the syslog facility (rather than standard output). The remaining item on the simple command line is the IP address of a server from which to request the current time. You can always specify multiple servers if you want to. Be aware that running ntpdate must take place before the NTP daemon is started.
In addition, many application programs and their associated server processes react rather badly to substantial clock changes after they have started. It is a good idea to perform time synchronization activities early enough in the boot process that they precede the starting of other servers that might depend on them.
A Simple Authentic Time Option
For many sites, all of the available authentic time options have significant inconvenience associated with them. Reference clocks and GPS devices can be expensive, and using an Internet-based timeserver can be an inconvenience if your connectivity to the Web is intermittent. At my site, we’ve found a low-cost and simple solution suitable for our network. It involves using an inexpensive clock that automatically synchronizes to NIST’s WWVB time code by receiving its radio transmission. In our case, the specific clock is an “Atomic Time PC Desktop Clock” (see http://www.arctime.com under desktop clocks for more details), which retails for about $100. The device appears in Figure One at left.
Devices of this type can be used as reference clocks utilizing the usual NTP facilities. However, this model is not supported. Additionally, in our case, this complexity is unnecessary. We use a simple Expect script to communicate with the device (which is attached to the computer via a serial port) and retrieve the current time. The script for doing this is contained in Listing One.
Listing One: Expect Script Used to Communicate with Atomic Clock
set clock /dev/ttyS0
spawn -open [open $clock r+]
# set the serial line characteristics
stty ispeed 300 ospeed 300 parenb -parodd \
cs7 hupcl -cstopb cread clocal -icrnl \
-opost -isig -icanon -iexten -echo -noflsh < $clock
expect -re “(.)”
send “\r” a
expect -re “(.)”
# expect 16 or more characters
expect -re “(…………….*)”
The script defines a variable pointing to the appropriate serial line, sets the line characteristics using the stty command, and then communicates with the device via a series of send and expect commands. These tell the clock to transmit the current time, and the script displays the resulting data on standard output:
Mon Oct 09 13:32:22 2000 -0.975372 seconds
We then use a Perl script to parse and reconstruct the data into a form required by the date command. For example:
(Keep in mind that the format for the date argument is mmddhhmmyyyy.ss.) We can then use configuration file entries like the following to set up NTP on that computer:
server 127.127.1.1 # LCL (the local clock)
fudge 127.127.1.1 stratum 12
These lines specify the local system clock as the NTP time source. This server then becomes the authoritative source of time information for all the other systems within the network. The other systems use the standard NTP facility for synchronizing to this time source. This level of time accuracy is perfectly adequate for our needs.
As always, there is more to say about NTP than we have room to cover in this column. The next place to look is always the man page. If you’ve already checked the man page, and you still hunger for more information on NTP, there is plenty more available on the Web. See the Additional NTP Information Sources, page 74, for more resources. Well, that’s it for this month — at least now if you don’t know what time it is, your system likely will.
Does Anybody Really Know What Time It Is?
Here, we look at time strictly from a standards point of view:
* A second was defined in 1967 as “the duration of 9,192,631,770 periods of the radiation corresponding to the transition between the two hyperfine levels of the ground state of the Cesium-133 atom.”
* Universal Time Coordinated (UTC) is the official standard for the current time used by NTP (utilizing the aforementioned definition of the second). UTC evolved from the previous standard, Greenwich Mean Time (GMT).
* Unfortunately, this definition of time does not exactly mesh with how long it really takes the earth to rotate on its axis. As a result, “leap seconds” are added or subtracted from UTC about every 18 months in order to maintain synchronization with the planet’s rotation (which is itself slightly irregular).
Æleen Frisch is the author of Essential System Administration, published by O’Reilly & Associates. She can be reached at firstname.lastname@example.org.