Monitor and trend the machines in your network the quick and easy way.
In many “Tech Support” columns I cover performance and optimization. While it’s important to consider performance and scalability in your original design plans, it’s easy to focus too much on optimization once you have a running infrastructure in place. As Donald Knuth said, “premature optimization is the root of all evil.”
So how do you know where to focus your optimization efforts? Far too many people assume they know where the bottlenecks are, and just work off their gut feelings. Unfortunately, you’d be amazed how consistently those gut feelings are wrong. When considering a particular application, you’ll want to setup profiling and unit testing and then use reliable benchmarking methodologies.
There are also tools such as dtrace and systemtap that allow you to get real time information on a production system. If you’re in charge of system infrastructure, however, you want a broader high level overview. There are many tools available to do this. In the September 2004 “Tech Support,” I covered cacti.
Today I’ll cover another tool that takes advantage of RRDTool: Munin, which regularly monitors various aspects of a machine and produces nice graphs which make spotting trends and aberrations quick and easy.
Munin is written in Perl and available under the GNU General Public License (GPL). There are two components to a Munin installation: the node component, which needs to be installed on every machine you’d like to monitor, and the master component, which needs to be installed on only one machine which will collect and display the data.
Note that if you want to monitor the master machine, it does need to have the node component installed. By default Munin monitors a fairly comprehensive amount of data, but it’s flexible plugin architecture allows you to monitor almost anything you desire. Many of the default plugins, such as load average, memory usage, CPU usage and Ethernet traffic will work out of the box with no configuration necessary.
Other plugins, such as Apache and MySQL, may require some setup: such as configuring /server-status or creating a user with the proper credentials. Additional plugins can be downloaded here.
Munin packages are available for most distributions. Visit http://munin.projects.linpro.no/wiki/LinuxInstallation to locate the appropriate packages and dependencies. For this article we’ll be using Red Hat Enterprise Linux on two servers, foo.example.com (10.0.0.1, munin-node and master) and bar.example.com (10.0.0.2, munin-node).
First, install the munin-node package on all machines you’d like monitored. Next, add the following to /etc/munin/munin-node.conf on each machine.
host 10.0.0.2 #IP address of machine
allow ^10\.0\.0\.1$ # IP address of Munin master>
Now, restart the Munin node with /etc/init.d/munin-node restart and ensure you’ve set munin-node to automatically start on boot. Once you’ve done this for every machine, it’s time to move on to the Munin master. First, install the munin package. Next, edit /etc/munin/munin.conf.
# a simple host tree
Munin will automatically be run on the master via cron every 5 minutes. The last step to configuring Munin is to add a virtual host to your web server with /var/www/munin as the docroot. Due to the amount of information Munin makes available, if the machine is publicly accessible, you will want to password protect it.
Pointing your browser to the new virtual host will give you a tree with example.com as the base, with a branch for each node. Next to each node name you’ll see a list of checks that correspond to the plugins that are currently enabled.
Figure 1: Munin Graph
Plugins are where the actual checks for Munin are done. All available plugins are installed in /usr/share/munin/plugins, with a symlink in /etc/munin/plugins for all enabled plugins. Running
munin-node-configure with no parameters will give you a list of available plugins, along with which ones are enabled. To enable a plugin, simply add the appropriate symlink and restart munin-node. Running the plugin with the autoconf directive should tell you if it needs any configuration. For example:
no (no apache server-status on ports 80)
If you can’t find a plugin to monitor a desired program at MuninExchange, you should be able to use one of the existing plugins as a base to write your own. If you think it would be of general use, please do consider adding it to MuninExchange. In addition to the base functionality I’ve covered so far, Munin also has some more advanced features that you may want to look into. The internal alerting system can be used to automatically contact you if certain thresholds are broken for a particular monitored value. Additionally, Munin will plug into Nagios either via a plugin or NSCA. You can also aggregate graphs in a variety of ways.
Using Munin to monitor and trend the machines in your network is quick and easy, but will pay huge dividends in spotting aberrant behavior, allocating resources and monitoring the health of your infrastructure.