Can't bring yourself to love logs? You should take a second look.
The lowly and lonely log files sit there day after day gathering dust and events as your system purrs along without issue. That is, until something bad happens. Then you scramble to find out why the system rebooted or had a memory problem. Maybe it was a network denial of service attack. Or was it a runaway process? Or worse still, a hacker after your MP3 collection. How will you know? If you said, “Look at the logs”, then you’re halfway to a resolution.
In most cases, those lowly log files are your best friends. Disasters, system anomalies, user error and careless hackers all leaves tracks in the logs. If you know where to look and what to look for, you’re that much better off.
Log files (logs) are text files, owned by root or the application’s daemon process user account, that receive entries whenever some significant event occurs. And, significant can mean anything from an HTML page view to an attempt to crack your root user account. Events can range from informational, in the example of an HTML page view, to critical for sudo failures, often indicative of root account hack attempts.
Processes associated with specific applications, services or daemons often have their own log files. The logging level or log detail is a configurable parameter. To save space and avoid log “madness,” system administrators will alter logging levels to capture only higher priority events and filter out informational notices. When you only log negative events, your logfiles stay smaller and reduce complexity when searching for a root cause of a system or application failure.
Perhaps the only exception to that statement is web logging. Webmasters and statistics programs collect every event, informational or otherwise, and use them for producing reports on site usage.
Each log file has a corresponding configuration file that governs its location, log level and other configurable parameters. Some log files capture events for multiple processes, as is the case for system events. The Apache web server has two log files, by default, the access log and the error log. Most programs and applications send all of their events to a single log file.
Location, Location, Location
You’ll find most of your log files located in the /var/log directory. Some administrators and webmasters change the default locations for their logfiles. Apache administrators, on shared systems, will create log directives so that each hosted web site has its own access and error log.
Figure 1 shows you a Debian 5 system’s /var/log directory listing. This is a standard (default) Debian 5 x86_64 installation with no additional software.
Figure 1: Debian System’s /var/log directory
In Figure 2, you can examine a CentOS 5.x system, that has a standard installation plus several additional applications including the X Window Desktops GNOME and KDM.
Figure 2: CentOS 5 System’s /var/log Directory
The /etc directory is home for most configuration files on your Linux system but certain applications (Apache, for example) either keep their configuration file(s) in an /etc subdirectory or in an application-specific configuration directory.
On the CentOS system, the Apache configuration installed under /etc/httpd. The logs directory is a symbolic link that points to /var/log/httpd where you saw it in Figure 2.
Rotation, Rotation, Rotation
Log rotation is one of those automated system processes that we, as system administrators, take for granted. We shouldn’t. Log rotation is highly configurable but most of us lazy administrators just leave everything as is, where is. As for the “where” part, the logrotate.conf, like other configuration files, resides in the /etc directory. The following shows you the default logrotate.conf from the example CentOS system.
$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
# keep 4 weeks worth of backlogs
# create new (empty) log files after rotating old ones
# uncomment this if you want your log files compressed
# RPM packages drop log rotation information into this directory
# no packages own wtmp -- we'll rotate them here
create 0664 root utmp
# system-specific logs may be also be configured here.
From this listing, you can see that your logs rotate on a weekly basis and the system retains four previous copies of your rotated logs. That’s why, when you look at Figure 2, you see messages, messages.1, messages.2, messages.3 and messages.4 files. The messages file is the current syslog. The other messages.X files are the rotated logs from previous weeks.
$ ls -la /var/log/messages*
-rw------- 1 root root 3798 May 23 18:56 /var/log/messages
-rw------- 1 root root 186528 May 23 04:02 /var/log/messages.1
-rw------- 1 root root 130014 May 16 04:02 /var/log/messages.2
-rw------- 1 root root 340531 May 9 04:02 /var/log/messages.3
-rw------- 1 root root 60222 May 2 04:02 /var/log/messages.4
On May 30, you’ll see a new log file appear as messages.1, the current messages.1 renames to messages.2, messages.2 becomes messages.3. The messages.4 file drops off and the oldmessages.3 replaces it. The logs rotate according to the rules in /etc/logrotate.conf.
If you need to force a log rotation, you can by entering the logrotate command with the -f (force) switch and the configuration file for the logs you want to rotate. To force a rotation for the system logs on the CentOS system, issue the command:
$ sudo logrotate -f /etc/logrotate.conf
$ ls -la messages*
-rw------- 1 root root 563 May 23 20:09 messages
-rw------- 1 root root 5013 May 23 20:09 messages.1
-rw------- 1 root root 186528 May 23 04:02 messages.2
-rw------- 1 root root 130014 May 16 04:02 messages.3
-rw------- 1 root root 340531 May 9 04:02 messages.4
Note that the system log (messages) rotated as predicted.
For those of you who must live in a graphical environment, here’s a log viewer tool for you: The GNOME System Log Viewer. From the GNOME menu, find this log viewer under System->Administration->System Log. See the GNOME System Log Viewer in Figure 3.
Figure 3: GNOME System Log Viewer Application
The entries in bold, shown in Figure 3, are bold because new entries have been added since you opened the viewer and viewed those logs.
Logs. Aren’t you glad you have them. Exciting? Maybe not. Helpful? Always. It’s true that logs aren’t the celebrities of a Linux system but they’re the “behind the scenes” heroes and you’d do yourself a favor by familiarizing yourself with them.
Next week, you’ll continue to log some time with log files, when you take a closer look at log contents.
Kenneth Hess is a Linux evangelist and freelance technical writer on a variety of open source topics including Linux, SQL, databases, and web services. Ken can be reached via his website at http://www.kenhess.com
. Practical Virtualization Solutions by Kenneth Hess and Amy Newman is available now.