Keeping Track of What Goes On, Part I

Although I understand the importance of keeping careful records of various sorts -- financial records for the IRS, hardware configuration records for my key computer systems, clothing size information about close relatives, and so on -- I am the type of person who continually forgets to note down such information when it is easy to do so or file away associated paperwork so I can find it again later. I do much better when I set up a system which collects and stores the relevant information automatically. That way, I set things up once and then forget about them until I actually need to use the information for some reason.

Although I understand the importance of keeping careful records of various sorts — financial records for the IRS, hardware configuration records for my key computer systems, clothing size information about close relatives, and so on — I am the type of person who continually forgets to note down such information when it is easy to do so or file away associated paperwork so I can find it again later. I do much better when I set up a system which collects and stores the relevant information automatically. That way, I set things up once and then forget about them until I actually need to use the information for some reason.

Focusing on system administration, people like me are fortunate that Linux comes with lots of great tools for recording and storing information about system operations on an ongoing basis. This is accomplished via the syslog facility, a central system message logging facility standard on all modern Unix systems.

Under Linux, this subsystem consists of a number of components: two daemons, a configuration file, and a number of log files that will record the actual data. The daemons are called syslogd and klogd, and they are usually started automatically when the system boots. (The file that actually does the work is typically /etc/rc.d/init. d/syslog, executed via a link to a file in one of the rcn.d subdirectories.)

The syslog facility collects all of the system messages produced by all of the various subsystems running on Linux and either records them or forwards them to some responsible party as specified by its configuration file. Most messages tend to be written to a log file, and the conventional location for these log files is /var/log.

The log files that are produced by syslog are just simple text files. Figure One contains a few sample lines from the /var/log/messages file, which is the standard default system message file.

Figure One: Sample Lines from the /var/log/messages file

Jun  4 00:00:00 bella /USR/SBIN/CRON[12866]: (root) CMD ( … )
Jun 7 17:25:57 bella pam_rhosts_auth[22185]: allowed to
aefrisch@ada as aefrisch
Jun 7 17:26:36 bella su: (to root) aefrisch on /dev/pts/0
Jun 12 14:09:15 bella kernel: eth0: Transmit error, Tx
status register 82.
Jun 12 14:09:21 bella last message repeated 4 times

The excerpt in Figure One contains a message from the cron subsystem indicating that user root ran the command cron on June 4th. User aefrisch used rlogin to access the system on June 7th (and su‘ed to root shortly thereafter), and the system had some minor Ethernet transmission errors on June 12th. The various messages all begin with a date and time stamp, followed by the system hostname, the name of the system facility which is generating the message (optionally followed by a process ID enclosed square brackets), and ending with the message text (separated from the facility by a colon).

You may wonder at this point why you should care about this information at all. System messages are important for two reasons. First, they provide you with a good picture of the ongoing activity on the system. Second, they provide data for instant and retrospective analysis of system problems of all kinds once they are discovered.

Which messages are collected and where they are stored is controlled by the syslog facility’s configuration file, conventionally /etc/syslog.conf. Before we look at some entries from it, we need to define two central concepts used by the subsystem. The first is “facility,” which is what syslog names the various subsystems which generate messages. There are quite a few defined facilities. The chart in Figure Two lists the most important.

Figure Two: Some Important syslog Facilities

Syslog Facility    Associated Subsystem

authpriv Login authentication
cron cron subsystem
daemon System server processes
kern Linux kernel
lpr Spooling subsystem
mail Mail subsystem
news News subsystem
localN Locally-defined syslog facilities
N runs from 0 to 7)

Severity Levels

The syslog subsystem also uses a construct known as the severity level in order to classify system messages by urgency and importance. The severity levels are (running from least to most serious): debug, info, notice, warning, err, crit, alert, and emerg. Not all subsystems use every available security level. In addition, the special severity level none may be used to indicate that no severity level should be matched in order to discard/exclude all messages from a facility.

Entries in the syslog configuration file specify how messages from each facility at the various severity levels should be handled. The general syntax of a configuration file entry is:

facility.level     destination

where the first field specifies the facilities and/or severity levels to which this entry applies, and the destination field indicates where the message should be sent or stored. Files, devices, computers and user names may all be used as destinations.

The most straightforward syntax for the first field is to use a facility name and severity level separated by a period, for example: cron.err. Using this format, the entry applies to all messages from the named facility (cron) having a severity level equal to or greater than the one specified (err).

Multiple facility and severity level designations may be used within a single configuration file entry by separating them with semicolons. The same severity level may be defined for multiple facilities by separating the desired facility names with commas. Figure Three contains some examples.

Figure Three: Defining Multiple Facilities and Severity Levels in a syslog Configuration File

authpriv.info;kern.err    login authentication messages at level info or higher and
kernel messages at level err or higher

authpriv,kern.info login authentication and kernel messages at level info or higher
authpriv.info,kern.err login authentication and kernel messages at level err or higher

The last example in Figure Three is a bit tricky. The comma may be a typo for a semicolon, but as it is written, everything up until the final severity level is interpreted as part of the list of facility names, and any severity levels that happen to be within this list are simply ignored.

Asterisks may be used as wildcards for either facility names or severity levels. The Linux syslog facility also includes syntax enhancements to the standard formats we’ve considered so far. A severity level may be appended to a facility name with “.=” to specify a single, exact severity level, rather than the usual range of the named level and higher levels. In addition, the “.!” characters may precede a severity level to indicate all levels not matching the specified level and higher levels, and “.!=” may be similarly used to specify all levels not matching the indicated level. Figure Four contains some examples of how to use these characters properly.

Figure Four: Using .= and .! to Specify Security Levels

mail.warn     mail subsystem messages at severity level warn or higher
mail.=warn only mail warning messages
mail.!warn mail messages less serious than level warn
mail.!=warn mail messages at any severity level other than warn

Send It Where?

The destination may be specified in a variety of formats, some of which are including in the following:

* A filename: messages are written to the specified file.

* A device (often a terminal): messages are written to the specified device. A previously-created named pipe may also be used by preceding its name with a “|” character.

* A list of one or more user names (comma-separated): write is used to send the message immediately to the specified user(s); if a single asterisk is used for this field, then the wall (“write all”) command is used to send the message immediately to all logged-in users.

* A hostname (preceded by an @): messages are sent to the syslog facility on the designated host for proper processing.

Figure Five contains a couple of examples of complete syslog configuration file entries.

Figure Five: Some Complete syslog Configuration File Entries

kern.=warn;*.err;authpriv.none            /dev/console
lpr.debug -/var/log/lpd-errs
*.emerg root
*.err;daemon,authpriv.notice;mail.none /var/log/messages

These entries specify that kernel warning messages and all .err and higher messages from subsystems other than login authentication will be written to the system console terminal, all messages from the spooling subsystem will be recorded in the file /var/log/lpd-errs, all emergency-level messages will be sent to user root via the write command, and all error-level and higher messages from subsystems other than mail as well as notice- and warning-level messages from system daemons and login authentication will be sent to the file /var/ log/messages. The “-” preceding the filename in the second entry indicates that writes to this file should be done asynchronously (i.e., buffered). There is a slight performance advantage to this approach, but it creates the potential for losing a few messages in the event of a system crash.

The last destination format we mentioned above (the format with the @ symbol) is used when you want to collect and record most syslog messages on a central server system rather than doing so individually for every system in the network. In this case, the syslog configuration file on the outlying systems would be fairly simple, sending most messages on to the designated syslog server system, and retaining only a few of the most critical locally.

In contrast, the configuration file on the server would be much more elaborate — collating and storing messages in a variety of log files according to the side’s specific needs. Figures Six and Seven illustrate simple syslog configuration files for the two system types.

Figure Six: Sample syslog Configuration File for a Client System

# Display/record the most important messages locally:
# console, xconsole (if any, via pipe) and log file.
kern.warn /dev/console
kern.warn |/dev/xconsole
*.err;kern.warn /var/log/messages

# Tell whoever’s logged in about real emergencies
*.emerg;user.none *

# Send everything except mail to the syslog server
*.info;mail.none @syslog_srv

Figure Seven: Sample syslog Configuration File for the syslog Server

# Display critical messages
*.emerg;kern.warn;user.none *

# Separate messages by subsystem
kern.* /var/log/kernel.log
mail.* /var/log/maillog
local7.* /var/log/boot.log
# lpd errors are very rare, but we’ll set them up anyway
lpr.* /var/log/lpd.errs

# Save authentication-related messages to protected files
# Maintain a separate su log file
auth,authpriv.warn;user.* /var/log/secure
auth,authpriv.=notice /var/log/sulog

# Main system messages log file
*.info;mail,news,lpr,authpriv,auth.none /var/log/messages

As you can see from these examples, setting up an effective syslog configuration requires a fairly detailed knowledge of the facility and severity level which produce various types of system messages. You can gain the relevant knowledge and experience by becoming familiar with the typical contents of the main system messages log file on your system.

The Logger Command

User applications and administrative scripts can also create entries within the syslog facility, using the logger command. It has the following syntax:

logger -p facility.level -t
tag message | -f file

where the -p option specifies the syslog facility and severity level, and the -t option specifies the identifying tag within the syslog entry. The final command parameter specifies the text of the message, which may be specified literally on the command line or redirected from a file with the -f option. In the latter case, each line in the file will become a syslog message. Another useful option is -i, which adds the process ID to the syslog message following the tag.

For example, the following command would send a message to the syslog local1 facility at severity level err:

logger -i -p local1.err -t
FFT “Job failed –
scratch disk full”

Here is the resulting message file entry:

Jun 12 14:09:15 bella
FFT[19232]: Job failed
— scratch disk full

This month we got a good general overview of what syslog is and how it works. Next month we’ll take a look at managing and processing all the information that syslog can collect (and believe me, it can get to be quite a lot!)

Syslog: The Next Generation

The syslog-ng package is an enhanced syslog facility designed to be both more powerful and more flexible than the standard version. It is widely available in various Linux archives, and its home page is located at http://www.balabit.hu/products/syslog-ng.html. Some of its specific goals are to make message filtering more fine-grained and content-based and to make forwarding messages beyond the local area network easier.

The facility explicitly defines message sources, destinations, and filters as separate entities and then creates actual logs by associating them together. Here are a few example lines from its configuration file that will give you a sense of how this facility works.

source src { unix-stream /dev/log; internal; };
destination cron { file /var/log/cron.log; };
filter f_cron { facility(cron); };
filter f_bad { level(warn..emerg); };

log { source src;
filter f_cron;
filter f_bad;
destination cron; };

These lines define the standard Unix message stream, a destination file named cron. log, and two filters which select only messages generated by the cron subsystem and messages above the warn severity level, respectively. The second section creates a log using the specified source, filtered by the two filters, and written to the destination log file.

Æleen Frisch is the author O’Reilly & Associates’ Essential System Administration. She can be reached at aefrisch@lorentzian.com.

Comments are closed.