One of the results of which was a ‘‘ that sits on the desk top. The lamp showed multi-colored blobs that moved and distorted in the window bottle. The color was a hash of the process name so was always the same color for some particular process. the size of the blob was its memory foot-print relative to the other blobs, and the speed it moved represented how much CPU that process was using.

basically it remains unobtrusive until needed. A quick glance shows that all is well! Better that having to live in a ‘doom’ world all the time, just to keep the monsters from fighting each other!

THE COMING AGE OF CALM TECHNOLOGY
http://www.ubiq.com/hypertext/weiser/acmfuture2endnote.htm

Ubiquitous Computing
http://www.ubiq.com/hypertext/weiser/UbiHome.html

and LavaPS a Lava Lamp Monitor
http://www.isi.edu/~johnh/SOFTWARE/LAVAPS/index.html

Monitoring in this case is not just monitoring the nodes from a sysadmin perspective, but also monitoring user’s behavior and tendencies as well as the progress of computations, etc. However, having been a sysadmin, I have a tendency to look at that aspect first.

One of the first things we as a community could do, is to develop a set of monitoring tools that are really more “push” than “pull”. The idea is that the compute nodes monitor themselves and when there is a substantive change in whatever their are monitoring, they push only that value to the master node or to a monitoring node. This cuts down on traffic, etc. It might increase OS jitter a bit since the nodes have to be self-monitoring to some extent, which probably means daemons waking up and doing something peridocially. But I think overall it makes life a bit easier.

For example, why not put a simple USB stick in the front USB port of each node. Then the monitoring data is written to the stick fairly quickly. You could even stash /var/log/messages on the stick. If the node fails or if a job hangs, you could either pull the data from the stick over the network, or just go grab the stick, swap in a fresh stick, and then read the data on the stick for any clues (but all sysadmin’s know that if a node fails, it usually doesn’t leave a trail of breadcrumbs that are easy to follow). You could even dedicated a single core to do nothing but monitor the node (this might be more palatable if we increase the core count yet again – but don’t blink that’s coming later this year from both AMD and Intel).

There are other aspects to monitoring as well. Doug’s comments about visualizing the results of a computation while it’s still running has been done in the aerospace world for several years. There is a package called PV3 that allows you to send back solutions from a running job and examine them. This way you can tell if a job seems to be running well or not. You can also have the running job dump the current solution periodically and then you can post-process it to examine the current solution (yep – this requires a pretty decent IO subsystem as well as a good post-processing computational capabilities and network). In the aerospace world they’ve adopted the phrase, “computation steering”.

Lots of cool things to be done in this area. It’s just not sexy enough to get funding from a gov. agency and most companies won’t fund it either. It’s not going to improve the speed of your applications to any noticeable degree. But it can improve the productivity and reduce some of the administration challenges associated with clusters. I most definite worthwhile effort in my opinion but unlikely to be done by anyone.

Regardless – thanks Doug for the reminder that we need to think more creatively about how to approach clusters.

Jeff