Big Brother Gives you Solid System Monitoring…for Free

Winston sat in the darkened room. All around him lights blinked and flashed. Regardless of what he did, Big Brother knew. There was nothing he could do without Big Brother being aware. When Winston did something that Big Brother did not particularly like, alarms would sound, and people would come storming into the room.

Guru/Big Brother/
Figure One: The Initial Big Brother Display.

Winston sat in the darkened room. All around him lights blinked and flashed. Regardless of what he did, Big Brother knew. There was nothing he could do without Big Brother being aware. When Winston did something that Big Brother did not particularly like, alarms would sound, and people would come storming into the room.

Although this sounds like it comes right out of George Orwell’s 1984, I’m talking about a different Winston and different Big Brother. In fact, Winston can go by many names. However, regardless of what name is used, it is under the ever-present watch of a real-life Big Brother, who watches and waits for one false move.

Fortunately, this Big Brother is truly your friend. However, like its Orwellian namesake, it is constantly on the lookout for things it doesn’t like, waiting to sound an alarm. I am talking about a systems-monitoring tool developed by Sean MacGuire of The MacLawran Group (http://www.maclawran.ca/bb-dnld/) called Big Brother.

Big Brother does most things that you’ll find in commercial monitoring tools; it can let you know when a machine on your network is down or becoming overloaded or when a filesystem is getting too full; it can tell you when specific processes are or are not running on clients; it can even page you when a specific event occurs. It can be used to monitor Unix, Linux, Windows NT, and NetWare clients.

One of the main reasons you’ll want to try out Big Brother is because of its simplicity. It is composed of just a handful of scripts and programs, which collect information and report it to a central server, which displays everything in an accessible HTML format. Big Brother’s scripts are easy to change and reconfigure, allowing you to customize the software to suit your network.

Although it is not covered directly by the GNU General Public License, you can download Big Brother for free from the MacLawran Group’s Web site. It is covered by a “fair use” license, which requires written permission from the MacLawran Group to redistribute it.

The Components of Big Brother

Big Brother consists of five key components. There is a central monitoring station or display server, which receives incoming messages from the clients, processes them, and makes them available in the form of Web pages. This means that although the display server currently runs only on Unix or Linux machines, you can monitor the server anywhere. In shell scripts, the BBDISPLAY variable refers to this server.

Network connectivity is monitored from the server using the shell script bb-network.sh, which tries to contact the various clients using a simple ping. You can configure Big Brother to check for connectivity to specific services on the client, such as HTTP, FTP, and SMTP.

The machine that checks connectivity does not necessarily need to be the display server and is referred to in the scripts as the variable BBNET. In some cases, it might make sense to have the BBNET machine sit on a machine acting as a router and direct the information it gathers to a server inside a firewall. This arrangement would then allow you to check connectivity to the Internet as well.

If checking connectivity to the machine is sufficient for your systems, you do not need to do any further configuration as all the work is done by the server. However, if the machine needs to report information to the display server such as disk space, running processes, etc., you need to configure the client. This is accomplished by the “Local System Monitor” component, which is embodied in the shell script bb-local.sh.

Big Brother can also be configured to send pager messages to the system administrator, based on any monitored event. When one of those events occurs, the machine where the event occurred uses the bb-page.sh shell script to send a message to the BBPAGER server, which then uses Kermit to contact the pager via modem. (Note that Kermit is not included as part of the Big Brother package.) Like the BBNET server, the BBPAGER server does not need to be the same machine as the display server.

Finally, there are a handful of support programs, such as the Big Brother daemon (bbd), which sits on the various servers, and there is the client program (bb), which sends the appropriate messages to the display and pager servers.

The same event on two different machines can have different meanings depending on your environment. For example, a database server that stores all of the data in a single file may take up an entire filesystem. So although the filesystem might be constantly at 99 percent full, you probably won’t want report it. However, a fileserver that reaches 80 percent full might generate a warning and then a panic message when the filesystem reaches 95 percent full.

A Slick Interface

Big Brother’s primary configuration file is etc/bb-hosts, which sits in the directory where you install Big Brother. By default, this is /usr/local/bb, but you can change this when you compile the source for your specific system. The bb-hosts file has a similar structure to /etc/hosts, but also includes information about whether or not certain specific networking services should be checked.

Since all of the pages are prepared in HTML, your BBDISPLAY server needs to have a an HTTP server running on it in order to serve the pages to the clients. Big Brother updates the page index.html in regular intervals so the information is always current. However, this does not mean you have to keep pressing the refresh or update button to see the current information, as each page is automatically refreshed every 60 seconds.

At the top of the page in Figure One is a legend, which describes what each of the different colors means, the time the page was last updated, and links to various sources of information. Clicking on the picture of Big Brother (which is actually a picture of Sean MacGuire) brings you to the Big Brother home page.

Big Brother also has the ability to group machines so that they are displayed in a separate table. Each column represents one specific aspect being monitored, with the rows representing the particular system. If something is not monitored on a particular system, there is a dash in that cell of the table.

The column headings (what is being monitored) are linked to a help page. Each help page gives you details about what is being monitored, which particular script is doing the monitoring, and in some cases, specific information about how the monitoring is being done.

Configuring the Display Server

The first step for installing Big Brother is to retrieve the package and extract the files. Configure it for your system by changing into the doc directory and running bbconfig OS, where OS is the name of the your operating system. If you do not specify an OS, bbconfig provides a list of supported systems to choose from.

Next, change to the src directory under the Big Brother root. Run make to compile all of the components, then make install to install them in their default location. If you need to change the location, you can change the location in the Makefile.

After the binaries are installed, edit runbb.sh in the Big Brother root directory and set the BBHOME variable to the directory where you installed Big Brother. Next, change to the etc directory and edit the bb-hosts file, which determines what aspects of the system will be monitored. It has a structure similar to a traditional hosts file, but is broken into three parts:


If you have turned on fully qualified domain names (the FQDN variable), then the HOSTNAME must also contain the domain name. DIRECTIVES is essentially a list of what needs to be monitored on the remote site. For example, if the host is one of the servers, then DIRECTIVES contains BBDISPLAY, BBPAGER, or BBNET, depending on what kind of server it is. Table One is a list of the more common directives. For some examples, check the default bb-hosts file. Finally, run bbchkcfg.sh and bbchkhosts.sh to check for errors in the configuration files.

Table One: Common bb-host Directives

BBDISPLAY Central server where information is displayed.
BBPAGER Server that sends pages to administrators.
BBNET Host that checks connectivity to the network services.
http://www_path/ Host and path to check for http connections. Multiple paths can be specified by separating using a pipe (“|”).
ftp Check the ftp service
smtp Check the smtp server
pop3 Check the pop3 server
telnet Check the telnet service
ssh Check the ssh server
nntp Check the nntp server
dns Checks for name resolution server
noping Don’t do ping test for this host
dialup If host is down then display clear button
NOTE: The service name must be written exactly the way it appears in /etc/services. On some systems, the POP3 server is written “pop-3″.

You must tell your Web server where to get the Big Brother pages. An easy way to do this is by making a symbolic link somewhere in the DocumentRoot directory of your Web server to the Big Brother root directory. However, you must make sure that the Web server is configured to follow symbolic links.

When you think you have this configured correctly, move back to the Big Brother root directory and start Big Brother by running the script runbb.sh. Like many scripts that start services, you can use the arguments start, stop, and restart.

By default, Big Brother is not configured to run when the system boots, so you probably need to add something to your /etc/rc.d directory or wherever the start-up scripts are on your machine. If Big Brother fails to start, check the file BBOUT for any errors. At this point, the server display server should be ready to go.

Configuring the Clients

The first step of configuring the Unix clients is to configure your bb-hosts file for each client. Once this is done, you create a tarball for the client using the script doc/bbclient. This tarball contains the necessary programs and files. Copy the tarball into the BBHOME directory on the client and unpack it.

If you have different Unix platforms, you will need to install a client on one machine for each different platform, then create a tarball, which is then copied to the other clients of that type. Make sure to copy your master bb-hosts file (it has a common format across platforms) and check your configuration using both bbchkcfg.sh and bbchkhosts.cfg.

Configuring Windows NT clients is completely different from configuring Unix clients. As of this writing, the NT Big Brother client is only available in binary form as a zip archive, which you need to unpack on a local drive. Both Alpha (axp) and Intel (x86) versions are available, and you need to rename the appropriate binary for your system. For example, on an Intel machine you would run:

 ren bbnt-x86.exe bbnt.exe

You next run bbnt.exe to install the program.

 bbnt [-y] -install BBDISPLAY FQDN IPPORT

BBDISPLAY is the IP address of the Big Brother display server. FQDN is either Y or N depending on whether or not Big Brother should return the fully qualified domain name of the machine. IPPORT is the port used for communication between the local Big Brother client and the Big Brother server. The -y option simply tells Big Brother to install without prompting for confirmation.

Big Brother runs as a system service under Windows NT. Once installed, it can be managed from either the Control Panel->Services, Server Manager->Services, or the command line using the net command (e.g. net stop, net start, and so on).

Because it is running as a service, there is a potential security problem if the bbnt.exe program gets replaced. Therefore, you need to make absolutely sure that the bbnt.exe binary is readable, executable, and also writable only by an administrative account.

The NT client has two additional options to either upgrade (-upgrade) or remove Big Brother completely
(-remove). It is a good idea to stop the service before you try to remove or upgrade it.

Customizing Big Brother for Your Site

The NT clients have a very comfortable GUI configuration interface (bbcfg. exe). In general, the available configuration options are self-explanatory. However, detailed configuration instructions are available in the included README file.

I have managed several machines where the default configuration on Unix machines is sufficient. However, Big Brother has a number of different parameters that you can customize. The primary configuration file is etc/bbdef.sh (the BB definitions file), which bbrun.sh reads when it starts up. Here, you define not only basic parameters, such as whether or not to display the fully-qualified names, but also specific behavior, such as how full the disk needs to be before it is reported by Big Brother.

By default, Big Brother monitors your filesystems and reports when they get too full (a warning at 90 percent, a panic message at 95 percent). These levels can be changed globally by setting the DFWARN and DFPANIC variables to different values, or you can define these levels on a per filesystem basis by changing the bbdef.sh file.

Similarly, both CPUWARN and
CPUPANIC are used to report on CPU activity. These are based on the load average as reported by the uptime command (and then multiplied by 100). By default, these are set to 150 and 300 respectively and can also be changed in the bbdef.sh file.

You can also monitor specific processes to see if they are running on the client. The PROCS variable defines which processes to monitor and report as a warning in the event that they are not running, whereas the PAGEPROC defines which are defined as a panic. Should the specified process not be running, Big Brother will page the administrator.

MSGS and PAGEMSGS are used to monitor log files. By default, MSGS is set to NOTICE WARNING. This means that only messages containing the words NOTICE or WARNING are examined. PAGEMSG is set to NOTICE, which means that should such a message appear, it generates a red condition (panic) and pages an administrator if the system is so configured. There is also the IGNMSGS variable, which tells Big Brother which messages to specifically ignore.

Keep in mind that the runbb.sh script reads the configuration information from bbdef.sh file when it is started. Therefore, after each change to bbdef. sh, you will have to restart Big Brother.

Although you probably will not need to change it, another important file is etc/bbsys.sh. This contains information about your specific operating system, such as the location of programs on your system, and which arguments to use. My suggestion is that you do not mess with this file unless you know what you are doing. It is generated for you when you install Big Brother, and there is normally no need to change it.

Another important variable is BBTMP, which defines the location of the directory that Big Brother users to store temporary files. By default, this is BBHOME/tmp and there is generally no reason to change it. However, if you do want it somewhere else, you need to make sure that normal users do not have write access as it represents a potential security hole.

Configuration of Big Brother goes beyond just setting variables. Big Brother provides a mechanism that allows you to define your own tests. These tests are located in the BBHOME/ext directory and a template is provided to show you the general syntax.

Once you have Big Brother up and running, you should investigate the scripts and configuration files.

You should also subscribe to the Big Brother mailing list (send email to majordomo@taex001.tamu.edu with the body of the message “subscribe bb”). This provides a great forum for asking questions, and is a source for some neat tips and tricks from more advanced users.

James Mohr is the author of books on Linux, SCO Unix, Web site administration, and Unix-Windows integration. He works in Coburg, Germany, for a systems integrator, and can be reached at jimmo@blitz.de.

Comments are closed.