dcsimg

Logs: Your Linux System’s Lovable Worker Bees

Can't bring yourself to love logs? You should take a second look.

The lowly and lonely log files sit there day after day gathering dust and events as your system purrs along without issue. That is, until something bad happens. Then you scramble to find out why the system rebooted or had a memory problem. Maybe it was a network denial of service attack. Or was it a runaway process? Or worse still, a hacker after your MP3 collection. How will you know? If you said, “Look at the logs”, then you’re halfway to a resolution.

In most cases, those lowly log files are your best friends. Disasters, system anomalies, user error and careless hackers all leaves tracks in the logs. If you know where to look and what to look for, you’re that much better off.

The Basics

Log files (logs) are text files, owned by root or the application’s daemon process user account, that receive entries whenever some significant event occurs. And, significant can mean anything from an HTML page view to an attempt to crack your root user account. Events can range from informational, in the example of an HTML page view, to critical for sudo failures, often indicative of root account hack attempts.

Processes associated with specific applications, services or daemons often have their own log files. The logging level or log detail is a configurable parameter. To save space and avoid log “madness,” system administrators will alter logging levels to capture only higher priority events and filter out informational notices. When you only log negative events, your logfiles stay smaller and reduce complexity when searching for a root cause of a system or application failure.

Perhaps the only exception to that statement is web logging. Webmasters and statistics programs collect every event, informational or otherwise, and use them for producing reports on site usage.

Each log file has a corresponding configuration file that governs its location, log level and other configurable parameters. Some log files capture events for multiple processes, as is the case for system events. The Apache web server has two log files, by default, the access log and the error log. Most programs and applications send all of their events to a single log file.

Location, Location, Location

You’ll find most of your log files located in the /var/log directory. Some administrators and webmasters change the default locations for their logfiles. Apache administrators, on shared systems, will create log directives so that each hosted web site has its own access and error log.

Figure 1 shows you a Debian 5 system’s /var/log directory listing. This is a standard (default) Debian 5 x86_64 installation with no additional software.

Debian System's /var/log directory
Figure 1: Debian System’s /var/log directory

In Figure 2, you can examine a CentOS 5.x system, that has a standard installation plus several additional applications including the X Window Desktops GNOME and KDM.

CentOS 5 System's /var/log Directory
Figure 2: CentOS 5 System’s /var/log Directory

The /etc directory is home for most configuration files on your Linux system but certain applications (Apache, for example) either keep their configuration file(s) in an /etc subdirectory or in an application-specific configuration directory.

On the CentOS system, the Apache configuration installed under /etc/httpd. The logs directory is a symbolic link that points to /var/log/httpd where you saw it in Figure 2.

Rotation, Rotation, Rotation

Log rotation is one of those automated system processes that we, as system administrators, take for granted. We shouldn’t. Log rotation is highly configurable but most of us lazy administrators just leave everything as is, where is. As for the “where” part, the logrotate.conf, like other configuration files, resides in the /etc directory. The following shows you the default logrotate.conf from the example CentOS system.

$ cat /etc/logrotate.conf
# see "man logrotate" for details
# rotate log files weekly
weekly

# keep 4 weeks worth of backlogs
rotate 4

# create new (empty) log files after rotating old ones
create

# uncomment this if you want your log files compressed
#compress

# RPM packages drop log rotation information into this directory
include /etc/logrotate.d

# no packages own wtmp -- we'll rotate them here
/var/log/wtmp {
    monthly
    create 0664 root utmp
    rotate 1
}

# system-specific logs may be also be configured here.

From this listing, you can see that your logs rotate on a weekly basis and the system retains four previous copies of your rotated logs. That’s why, when you look at Figure 2, you see messages, messages.1, messages.2, messages.3 and messages.4 files. The messages file is the current syslog. The other messages.X files are the rotated logs from previous weeks.

$ ls -la /var/log/messages*
-rw------- 1 root root   3798 May 23 18:56 /var/log/messages
-rw------- 1 root root 186528 May 23 04:02 /var/log/messages.1
-rw------- 1 root root 130014 May 16 04:02 /var/log/messages.2
-rw------- 1 root root 340531 May  9 04:02 /var/log/messages.3
-rw------- 1 root root  60222 May  2 04:02 /var/log/messages.4

On May 30, you’ll see a new log file appear as messages.1, the current messages.1 renames to messages.2, messages.2 becomes messages.3. The messages.4 file drops off and the oldmessages.3 replaces it. The logs rotate according to the rules in /etc/logrotate.conf.

If you need to force a log rotation, you can by entering the logrotate command with the -f (force) switch and the configuration file for the logs you want to rotate. To force a rotation for the system logs on the CentOS system, issue the command:

$ sudo logrotate -f /etc/logrotate.conf
$  ls -la messages*
-rw------- 1 root root    563 May 23 20:09 messages
-rw------- 1 root root   5013 May 23 20:09 messages.1
-rw------- 1 root root 186528 May 23 04:02 messages.2
-rw------- 1 root root 130014 May 16 04:02 messages.3
-rw------- 1 root root 340531 May  9 04:02 messages.4

Note that the system log (messages) rotated as predicted.

Graphical, Shmaphical

For those of you who must live in a graphical environment, here’s a log viewer tool for you: The GNOME System Log Viewer. From the GNOME menu, find this log viewer under System->Administration->System Log. See the GNOME System Log Viewer in Figure 3.

GNOME System Log Viewer Application
Figure 3: GNOME System Log Viewer Application

The entries in bold, shown in Figure 3, are bold because new entries have been added since you opened the viewer and viewed those logs.

Logs. Aren’t you glad you have them. Exciting? Maybe not. Helpful? Always. It’s true that logs aren’t the celebrities of a Linux system but they’re the “behind the scenes” heroes and you’d do yourself a favor by familiarizing yourself with them.

Next week, you’ll continue to log some time with log files, when you take a closer look at log contents.

Comments on "Logs: Your Linux System’s Lovable Worker Bees"

martin.marcher

Honestly this article lacks any sort of detail.

I can imagine that a person who just started with system administration _and_ never saw linux before will find the single bit of information usefull that tells us to look in /var/log/.

Apart from that…I\’ll be waiting for next week. Hoping for an in-depth writeup about (somewhere in this series) about log servers, event correlation, different products on the market to analyze logs (splunk,…. – not working for them but so far the best I found. I\’m open to alternatives)

Reply
bpier99

What a \’fluff\’ piece!!
I was hoping to at least revisit log purpose, processing, management, etc., but reached the end of the article and wondered where the missing pages were! This article is absolutely ridiculous. If the intention was to merely expose a newbie to Linux logs, then it would be appropriate to say so, at least as a disclaimer.
Where was the discussion of myriad other log viewing / searching tools?
Where was any discussion about CONTENT type in various logs?
What about the directory structure often present in modern log directories?
What about log file aggregation?
What about remote logging?
What about log digesting and triggering based on log contents?
What about log monitoring?
Heck, this \’article\’ could have even been a short series based on many \’log\’ related topics!

… and I suppose you want me to continue reading future articles too?

Bill Pier

Reply
khess

It\’s an introductory piece. Next week\’s goes more in depth but I had to establish a baseline so that everyone would have an even starting place. I didn\’t want to jump in to an advanced logs discussion and alienate new people from the article. It\’s an important topic and newbies need to learn it too.

Reply

    they don’t want to turn in logs for pircavy or other reasons and the first comments out accuse the person of cheating. Or hiding something that must be bad. Questioning their character. Wow.

    Reply
srosen

Thanks for the article. I hope your going to make a series out of it. I need to have knowledge of logs. Hes right. You have to start low for some of us and work your way up.

Reply
hybrazil

There sure are some rude people about. If the article is too basic for you, perhaps you should go and have a look at the docs in the OS – that\’s the place that was intended for an expert audience to go. Then the rest of us can enjoy the intro article in peace.

I thought that the article was a good start to a series aimed at helping people learn how to use logs. Thanks for a great article.

Reply
mdiehn

I\’d never bothered to look at the Gnome System Log Viewer and now I have. So thanks for that!

Reply
adarkling

I echo mdiehn\’s comment.
We\’re not all Linux gurus yet. I\’ve been on Ubuntu for around 2 years now & never looked at the logs unless I absolutely had to — which isn\’t that often (thanks Linux!). Even then I\’m looking for that one file that solves my specific problem. I\’m so focused on finding that one tree that I haven\’t yet fully grokked how the file structure of the whole forest works.

I did not know that I could rotate the logs that easily! I usually employ the Windows method of emptying the file out by hand to track changes. This is much better.

And that\’s why the intro article was a good idea.

Reply

This is often a cause of problems in the database world where multiple clients may be executing queries on desparent data.
menonlinefashions.com

Reply

I think that what you said was very logical.

However, think about this, suppose you composed a catchier title?
I am not suggesting your content isn’t solid., however what if you added something that grabbed people’s attention?
I mean Logs: Your Linux System?s Lovable Worker Bees | Linux Magazine
is kinda boring. You might look at Yahoo’s home page and see how they create article headlines to grab people interested. You might try adding a video or a picture or two to grab people excited about what you’ve
got to say. Just my opinion, it might make your website a little livelier.

Reply

Hello to all, the contents present at this
web site are truly awesome for people knowledge, well, keep
up the good work fellows.

Reply

Looking forward to reading more. Great article post. Really Great.

Reply

Hi there! Do you use Twitter? I’d like to follow you if that would be ok. I’m definitely enjoying your blog and look forward to new posts.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>