dcsimg

Gandalf! Monitor Your Machines with Hobbit

Not quite a ring of power, but Hobbit can take some of the pain out of machine, network and services management.

Hobbit is a tool for monitoring servers, applications, and networks, inspired by and compatible with Big Brother, but with the advantage of being open source and under active development. You install Hobbit on a central server, and a client for Hobbit on all the machines you want to monitor.

The central server collects information about the status of your computers and the applications on them, and displays it via an automatically updating Web page. You can also set up email (or text message) alerts for particular situations.

It can monitor various services (SSH, FTP, HTTP…), connectivity, disk space, CPU usage, Web sites, and a variety of other aspects of your machines and network. It’s very configurable, extendable, easy to use, and can handle large numbers of machines.

In this column, I’ll provide a quick overview of how to set it up, configure it, and use alerts.

Setup

First, identify the machine you wish to use as a central server — here I use hobbitserver.example.com. You’ll need to install a basic Apache setup (or have one already available). Hobbit is available as a Debian package for Debian or Ubuntu, or as an RPM Fedora Core 5, so should be very straightforward to install. (Or you can of course install from source if you prefer.)

All packages are available from the project’s SourceForge page, or possibly via your distribution’s repositories.

Basic configuration

For the most basic setup (monitoring only itself), you only need to edit a single file, /etc/hobbit/bb-hosts, and update the Web server.

First, check that bb-hosts includes this line:

127.0.0.1	localhost	# bbd http://localhost/ conn

Secondly, you’ll find instructions on how to set up Apache in /etc/hobbit/hobbit-apache.conf. Add the configuration as directed in this file, and reload your Web server.

Now restart the server (/etc/init.d/hobbit restart). Wait for a few seconds — maybe as much as a minute — and then go to the Webpage at http://localhost/hobbit/ or http://hobbitserver.example.com/hobbit/ and you should see the single line entry for localhost. It should show the Hobbit server running OK, and will also have information about disk usage, CPU usage, and connection, among other things.

There are several possible statuses. Green means that status is OK; yellow is a warning; red means critical. Clear means that there is no data, purple that there is no report. Blue means that the monitoring aspect is disabled.

Next, try a test client. Install the Hobbit client software (see the sourceforge page for packages) on the test client. Next add your test client machine to the configuration, by editing the file /etc/hobbit/bb-hosts on the hobbit server again, and adding a similar line to the localhost one, but with the appropriate IP address and machine name.

You shouldn’t need to restart hobbit on the server, although obviously you will need to make sure that it is running on the client. Again, wait for a minute or so, then refresh the Web page, and the client will appear.

More Advanced Configuration

Using Hobbit, you can choose what services you check. For example, if your client webserver.example.com is running a Web site at http://www.example.com, you can add that to the services to check by editing the relevant line in the bb-hosts configuration file as follows:

192.0.2.25  webserver         # conn http://www.example.com

You can list as many sites as you like, separated by a space. You can also monitor SSH and NFS:

192.0.2.30  nfsserver	# conn ssh rpc=mount,nlockmgr,nfs

And various other services, including FTP, Telnet, SMTP, POP3, IMAP, spamd, and SSL-enabled services. You’ll find a full list on the Hobbit Web site. You can monitor any of these services by simply adding them to the relevant client line in bb-hosts on the server running Hobbit.

Specifying danger levels

You can specify the boundaries for alerts — what is defined as “warn” (yellow) or “panic” (red) — for individual machines, or change the defaults. You do this by editing /etc/hobbit/hobbit-clients.cfg. Here’s a sample:

HOST=client1
    DISK    /data   IGNORE

DEFAULT
    UP      1h
    LOAD    5.0 10.0
    DISK    / 90 95
    DISK    %^/local.* 99 99.5
    MEMPHYS 100 101
    MEMSWAP 50 80
    MEMACT  90 97
	LOG		%.* %WARNING %COLOR=yellow

Note that the default section must be the last one in the config file. The first section tells hobbit to ignore a particular disk — /data1 on client1. (Perhaps this data disk is known to be full and we don’t care about it.)

The next section sets various defaults:

  • UP: sets “warn” status if the system has been up for less than 1 hour. If you add a second number, it will set “warn” status if the system has been up for longer than that.
  • LOAD: warn if CPU usage is above 5.0, panic if it is above 10.0.
  • DISK: specify a regular expression for the disk name, noting that any such regexp must begin with %. Again, panic and warn levels are set (for percentage disk use).
  • The various memory options are taken from proc.
  • LOG: The indicator will go yellow if a line in any log file contains WARNING.

You can alter these as you prefer. You’ll see a handful of other options, documentation of which is at the top of the hobbit-clients.cfg file.

Logfile Monitoring

To monitor logfiles, you need both the section in hobbit-clients.cfg, as above, and a section in /etc/hobbit/client-local.cfg (on the Hobbit server). The clients will pick up the configuration corresponding to their hostname or operating system from client-local.cfg when they connect to the server.

The default for Linux clients is just to check /var/log/messages, and to ignore lines beginning with MARK. You may also want to add /var/log/syslog, and to ignore cronjob lines in that (cronjobs should email you if they go wrong, and there are often a great many cron lines in syslog!). The following section of client-local.cfg should do this:

[linux]
log:/var/log/messages:10240
ignore MARK
log:/var/log/syslog:10240
ignore MARK
ignore /USR/SBIN/CRON

The number at the end of the line is a maximum number of bytes that the client will send — it also only sends the last 30 minutes’ worth of data, so this is a suitable upper bound.

Email and Other Alerts

One of the most useful things about Hobbit is that you can configure it to send you email alerts in certain circumstances. To do this, you edit the /etc/hobbit/hobbit-alerts.cfg configuration file.

There are numerous available options to configure your alerts. For example, the following setup would email you if any machine (again, the regular expression must start with %) had been unavailable for 30 minutes (DURATION), would then repeat every 24 hours (1400 minutes) thereafter, and would email you on recovery (RECOVERED):

HOST=%.* SERVICE=conn
	MAIL admin@example.com DURATION>30 REPEAT=1440 RECOVERED

You can also set more than one alert. For example, you might want to email a different address (your own home address, or the on-call address), if the service went down out of hours:

HOST=%.* SERVICE=ssh
	MAIL admin@example.com DURATION>30 REPEAT=1440 RECOVERED
	MAIL admin@home.com DURATION>30 REPEAT=1440 RECOVERED TIME=*:1800:0800

Or for a particular host, there may be different people to mail in different situations, in which case the SERVICE can be specified in the MAIL line:

HOST=webserver.example.com
	MAIL admin@example.com SERVICE=disk,ssh DURATION>30 REPEAT=1h RECOVERED
	MAIL webmaster@example.com SERVICE=http DURATION>30 REPEAT=1h RECOVERED

Here the repeat would be every hour. You can also use the SCRIPT keyword instead of MAIL, to run a particular script if there is a problem. For example, if monitoring an email service, sending an email in case of problems might not make a lot of sense! Hobbit provides various environment variables which will be passed to the script — see the documentation for further information.

This file will also utilize variables. So you can set up a default alert setting, and then specify that using that variable thereafter. To do this using the first example above:

$MAILADMIN=MAIL admin@example.com REPEAT=1440 RECOVERED

HOST=%.* SERVICE=conn
    $MAILADMIN

Finally, you can exclude particular machines if you want. For example if you have a Windows machine that isn’t running ssh, you can avoid being warned that this is the case:

HOST=%.* SERVICE=ssh
    IGNORE HOST=windows1
    $MAILADMIN

The IGNORE keyword stops processing that section altogether at the point where it appears — so it must occur before any alerts to work properly.

Conclusion

Hobbit makes monitoring your services and network much more straightforward, and is a more manageable solution than the clutch of homebrew scripts that many sysadmins employ! In particular, the configurability of the alert system is excellent and makes it much easier to react in a timely fashion when problems occur. Given how easy it is to get running, it’s well worth taking the time to experiment with it.

Comments on "Gandalf! Monitor Your Machines with Hobbit"

jlargent

Looks like 1/4 finished copy of nagios to me.

Reply
lagosm

I agree with that. It looks like nagios.

Reply
richzendy

What about zabbix ?

Reply
mbw

Does anyone know if nagios or hobbit have a fully-baked VMWare ESX server plugin that runs in the service console and tells the status of the Disk and CPU of the whole machine?

I’ve rooted around for a Hobbit plugin, but there is nothing that works without a lot of hacking.

Reply
krowton

Hobbit is a great place to start monitoring. Opensource tools like Nagios (Much more robust and harder to configure) and Groundworks (A nagios front end and more) will do you much better in the long run.

Hobbit is by far the easiest to install and configure of the three I have mentioned.

Reply
silversphere

I still have to try, I used Nagios for a long time, I find it complex, it was error prone for some times, Has any one used BigBrother !

Reply
evert

You may also want to look at the Dude. Originally made for Windows, but it runs fine under Wine. It’s got some good auto-discovery/scanning/probing features.

Reply
andyg54321

Dude? Dude.

Reply
gchetrick

The Dude? not only does the name suck but, if they can’t run a native agent in linux, probably not going to get installed on my machines, seems a little bonky to run monitoring in WINE.

Reply
gchetrick

I have used both Nagios and BB, I prefer Nagios. But they both have advantages, BB is much eaiser to configure. Hobbit is basically BB, if I remember correctly the original developer of BB is the current developer of Hobbit, just after he sold BB to Quest. In early versions the alert screens looked the same.

Reply
kevinw1

We previously used BB but have moved to hobbit and have been greatly impressed. It is very easy to produce your own alerts or modify existingones. There are lots of standard ones to choose from at deadcat.org. One of the most useful things with hobbit is it is easy to fire off a script to fix common problems.

Reply
jrichard@sciquest.co

We used bb for years, and have recently migrated to hobbit… Transition was painless, all of our custom bb scripts came over with little or no modification. We have > 30 monitoring everything from db2 to tomcat.

Hobbit is much more stable then bb was. We used to have problems with bb server side network checks getting hung up and driving subsequent checks purple. But this has been fixed with the C based hobbit server side checks. Also the escalation of checks from 5 minutes to every minute on red has improved our SLA numbers since the minimum outage is now a minute instead of 5.

We liked BB we Love Hobbit. :)

We’ve also recently played with ZenOss… It’s a different animal as it’s an snnp console. I think we may wind up with both eventually. But there are snnp hooks for bb/hobbit we will probably play with those before we decide one way or another.

Reply
weismanm

this looks very interesting, and I appreciate everyone’s comments. I am always looking for easy to install and use systems and will try this. nagois, zenoss and BB are a bear to install and configure (at least for me)

Has anyone ever used Big Sister? It’s another BB clone, but quite easy to install and use, not so easy to configure past basic stuff, though.

Reply
majeska

Hobbit smokes Nagios. I used Nagios for years and recently switched to Hobbit. Hobbit comes ready to graph the data it collects and data collected by your own scripts so you can see trends in server and service health over time for capacity planning, diagnosing problems, etc. It can be done in Nagios but its alot more painful to setup. I’ve found that Hobbit is capable of doing so much more then Nagios.

Reply
beerse

The nice thing about all these monitoring tools (hobbit, bb, nagios, zabbix) is that they have an agent for _most_ systems. How about the other systems? Do they come with a snmp interface? Is this a self-discovery interface? does it accept mib-files? Because in the end, you like to monitor _all_ systems, even the ones for which there is no agent and the ones that cannot handle an agent, like printers, switches, routers and other attached stuff.

Look at HP with their server-agents: they add entries to snmp which is already available in the os. Then your own snmp-based monitoring can also monitor the HP-special parameters. That’s the way it should be: Use an available standard, then you can get much higher.

And for the microsoft operating systems: wmi is not the standard.

Reply
codemunkee

I’ll stick with Nagios and NRPE. Splunk is also awesome if you’re wanting to monitor syslogs.

And for Gentoo Admins, like myself, it becomes more clear what to use when you run:

emerge -s hobbit

and

emerge -s nagios

Reply
jjabusch

I don’t know about Zabbix, but I can speak to Nagios and Zenoss. Nagios is okay for doing simple testing and monitoring / alerting. However, when establishing SNMP tests, it’s more challenging.

Zenoss, however, was a little easier to setup than Nagios, and its autodiscovery works just fine. Plus, for devices it doesn’t already know (we use a lot of Nortel equipment) – I just had to acquire the Nortel MIBS, and import them in through a menu option.

Nagios has some basic charting abilities plus the ability to dump stats to other tools, although you may have to get creative. Zenoss will chart just about any stat it captures for you, with little effort.

With that all said, I run both. Nagios is primarily just an alerting tool, because the stats and the ability to delve deeper into equipment is built into Zenoss, and I’d have to do a lot of scripting to make it work in Nagios.

Plus, Nagios’ dashboard-like functionality is still a little ahead of Zenoss’ dashboard, IMO.

Reply
ashusethi

I find Nagios is a better one though, had extensively used it, specially there is alot of development of supporting applications going on.. as one i remeber was a desktop alert client for windows, it used to show a pop up on my windows host that something is wrong with some server.

Zabbix may be another good bet with a filtered GUI.

Cheers
ash

Reply
mitsuto

All your comments about all NMS being fittest than Nagios for your environment do not consider monitoring about 1000 servers. Any hosting provider employee here uses zabbix/zenoss/groundwork/hobbit/bb on thousand-fold servers data-center? I don’t think so. When scaling is what matter, nothing scale more than Nagios + Cacti(with boost plugin) as for NMS+trends_graphing solution.

Reply
andreaferraris

What about Monit (http://mmonit.com/monit/)
Fo me it worked great.

Reply
renders

I have used Nagios since 2001 and more recently, the Opsview version. Opsview has improved the SNMP integration of Nagios. I have compared Nagios with Zenoss, BB and Zabbix, I believe Nagios is more robust and scalable. Custom scripts are a cinch with Nagios plugins.
I run Opsview within a vm, monitor aprox 200 services and 75 hosts so not a huge system..

Home of Opsview

Reply

Hi Guys,

just read all the information. We extensively use Hobbit / Xymon at http://www.mrmail.com to monitor our whole info structure and so far we are very happy about it.

Reply

Great info. Lucky me I ran across your site by accident (stumbleupon).
I’ve saved as a favorite for later!

Reply

wonderful put up, very informative. I wonder why the opposite specialists of this sector do not understand this. You must continue your writing. I’m confident, you’ve a great readers’ base already!

Reply

Hi, i think that i saw you visited my blog thus i got here to “go back the want”.I am trying to to find things to enhance my website!I assume its good enough to use some of your ideas!!

Reply

Virtually all of the things you articulate is supprisingly precise and that makes me ponder the reason why I had not looked at this with this light before. Your article really did switch the light on for me as far as this specific subject goes. But there is one position I am not necessarily too cozy with so whilst I make an effort to reconcile that with the actual central theme of your point, permit me observe exactly what all the rest of the visitors have to say.Nicely done.

Reply

This really answered my problem, thank you!

Reply

An impressive share, I simply given this onto a colleague who was doing somewhat evaluation on this. And he actually purchased me breakfast as a result of I discovered it for him.. smile. So let me reword that: Thnx for the deal with! However yeah Thnkx for spending the time to debate this, I feel strongly about it and love reading more on this topic. If potential, as you grow to be expertise, would you mind updating your blog with extra details? It is extremely useful for me. Big thumb up for this blog post!

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>