one real problem I see with tools for storage monitoring is lack of them for finding the real “abuser” in your system when you have a IO bottleneck. For example, with iostat you can see the whole performance goes to hell on a busy system but you can not see which exactly process for example is doing some nasty IO. Of course there are some other tools like atop that can pin down the real “abuser” but again there are some limitations like you need a recent kernel version with some patches to be able to use this utility.

I wonder, as the author does, how come there are no good and proven tools for IO monitoring like ps, top, etc. for process monitoring …

Sounds like we belong to the same tribe – the tribe of “we need better tools!”.

While I’m not a “coder” in the literal sense of the word (all my coding was around my research area with is aeronautics and astronautics), I have started down the path of trying to put something together like you mentioned – finding the “abuser”. I think with a combination of SystemTap and some other things, a simple tool can be created to monitor the system and when certain conditions happen, such as a high load, the tool can find the process that is driving the problems. I’m just beginning some research into what the pieces that go into the tool and how they get glued together.

I’m not sure what to do once you identify the rogue process. I thought about just killing the rogue process but that’s way to draconian. I also thought about trying to throttle the rogue process so that the load (or whatever the measurement is), doesn’t go crazy. But I think for a while the default will be just to inform the admin of the problem and let them determine the course of action.

But this tool will be slow in coming since my day job interferes with real work time.

Thanks for the post and comments!

Jeff

http://guichaz.free.fr/iotop/
=====
iotop does for I/O usage what top(1) does for CPU usage. It watches I/O usage information output by the Linux kernel and displays a table of current I/O usage by processes on the system. It is handy for answering the question “Why is the disk churning so much?”.
=====

Great comment and I agree with you. Determining what is “normal” is one thing but we must also determine what is “acceptable”. In addition, I think we can also include what is “maximum”. So is normal some percentage of the maximum? Can that normal fluctuate?

I personally think we also need to focus on what is acceptable. For example, is it OK for an NFS server to have a very high peak load where the system apparently freezes for a few seconds but then returns to “normal”. This behavior might be acceptable. That is, going beyond the normal levels for a small period of time. But then again, if it happens too often it may not be acceptable.

Sounds like a machine learning or AI kind of problem to me :) Maybe this is a way for AI to make a come-back! :)

Again – thanks for the insightful comments.

Jeff