From the Sys Admin Toolbox: Logwatch

Last week we provided an introduction to the Linux system logs. (See Logs: Your Linux System’s Lovable Worker Bees.) Now, what will you, as a system administrator, use to watch logs? Logwatch, of course. As so many others have succinctly put it, messing with log files is a royal pain in the backside. Logwatch makes the experience of keeping track of system activity almost painless for you.

Logwatch is a log file parser program (Perl script) that provides a report to you on any “interesting” activity on your system. It is not, I repeat not, a pre-emptive tool or a tool that’s used to catch anyone “in the act” of breaking into your system. It is an after-the-fact tool that provides you with a daily report of service activity. It reports on yesterday’s log information.

We’ll explore active monitoring tools in the coming weeks to catch a would-be system hacker. Logwatch’s value isn’t in its ability to catch a criminal with his hands on your system, its value is instead, designed to save you the effort of manually scraping logs.

The Basics

Before we begin, if you need a bit more background on Linux logs and system information, check out any of the following articles:

• Linux System Information Decoded
• The Linux System Logging Daemon
• Using Logwatch and Watching DVDs
• Writing a Trace System
• Logging with syslog-ng, Part One
• Logging with syslog-ng, Part Two

Now, let’s dive in.

Install logwatch in the usual way for your particular distribution using a package manager or download the source from the Logwatch Project page.

Primary setup is easy. If you installed via package, an automatic cron entry in cron.daily runs logwatch every day for you. Default setup includes all services, default log location /var/log and mail to the local root account. Logwatch installs to the /usr/share/logwatch directory for Debian-based and Red Hat-based systems. The main configuration file is under /usr/share/logwatch/default.conf. Some packagers create a symbolic link from /usr/sbin/logwatch to the perl script under /usr/share/logwatch/scripts/logwatch.pl.

Using Logwatch

Logwatch, by default, runs daily on yesterday’s logs, sends an email to the local root account with a low level of detail. For most of you, this is enough information in a daily summary to satisfy your needs and curiosity about what’s going on with your system. For others with systems that are a bit more security sensitive, you’ll need to slightly tweak the parameters of this “near perfect by default” tool.

You can run logwatch with a specific date range. For example, if you want to see information about today’s SSHD activity, you can run this command:

# logwatch --service sshd --range=Today

Check root’s mail to see the details of the report. Shown is the SSHD section of the report.

 --------------------- SSHD Begin ------------------------

 Illegal users from:
    192.168.1.83: 12 times

 **Unmatched Entries**
 pam_succeed_if(sshd:auth): error retrieving information about user raphael : 3 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user bob : 6 time(s)
 PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.83  : 4 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user george : 3 time(s)

 ---------------------- SSHD End -------------------------

This is your default “Low” detail logwatch output report. For a more detailed report, try using –detail=Medium. Check root’s mail for the report.

# logwatch --service sshd --range=Today --detail=Medium

 --------------------- SSHD Begin ------------------------

 Illegal users from:
    192.168.1.83: 12 times
       bob/password: 6 times
       george/password: 3 times
       raphael/password: 3 times

 **Unmatched Entries**
 pam_succeed_if(sshd:auth): error retrieving information about user raphael : 3time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user bob : 6 time(s)
 PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.83  : 4 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user george : 3 time(s)

 ---------------------- SSHD End -------------------------

You can see that there’s more detail in this report. You can try using the High parameter to squeeze more detail from logs.

# logwatch --service sshd --range=Today --detail=High

 --------------------- SSHD Begin ------------------------

 Illegal users from:
    192.168.1.83: 12 times
       bob/password: 6 times
       george/password: 3 times
       raphael/password: 3 times

 **Unmatched Entries**
 pam_succeed_if(sshd:auth): error retrieving information about user raphael : 3time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user bob : 6 time(s)
 PAM 2 more authentication failures; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.83  : 4 time(s)
 pam_succeed_if(sshd:auth): error retrieving information about user george : 3 time(s)

 ---------------------- SSHD End -------------------------

As you can see, the detail didn’t change from Medium to High for SSHD. Alternatively, you can specify the detail level using numeric values using 0, 5 or 10, where 10 is the highest level of detail. Please note that you won’t receive more detail than what’s supplied in the logs.

For those of you, like me, who don’t love command line mail, you can tell logwatch to save its report to a file.

# logwatch --service sshd --range=Today --detail=High --save=logwatch.today

If you need other command line tweaks, a quick man logwatch (Yes, the developers wrote a manual page for Logwatch) will give you what you need.

Logwatch Paranoid Configuration

As shown in the previous section, you can use logwatch at the command line when needed. Those command line options supercede any configured parameters hard-coded in the configuration file located in /usr/share/logwatch/default.conf.

For those of you who are paranoid, or need to be, there are some settings in the configuration file worth noting.

The defaults settings are

Range = yesterday

Detail = Low

Archives = No.
You should change these to:

Range = All

Detail = High

Archives = Yes.

The Archives setting grabs data from your recent logs as well. If you remember from the first post in this series, they’re located under the /var/log directory and have a .X filename to designate their rotation number.

These settings will process all logs for all services and provide the highest level of reporting. The command line equivalent is shown below.

# logwatch --detail=High --range=All --archives

Customizing Output

Here’s a little bonus for those of you who enjoy reading HTML pages more than catting a text file or reading command line email. You can publish your logwatch reports in HTML. It only takes a minor bit of tweaking to create professional-looking automated HTML reports complete with formatting.

The command line version is shown below.

# logwatch --service sshd --detail=High --range=Today --output=html --save=/var/www/html/logwatch/logwatch.html

The equivalent settings in the configuration file are:

Save = /var/www/html/logwatch.html

Output = html

The SSHD section from the HTML report is shown in Figure 1.

Figure 1: SSHD Section from the Logwatch HTML Report

Logwatch is a useful script for system administrators who don’t have the time or the patience to grep and page through logfiles. The work has been done for you by Kirk Bauer who develops and maintains logwatch. Logwatch should be one of your standard system administrator tools that’s installed on every system you manage.

Next week, you’ll take a look at some real-time log monitoring with swatch.