dcsimg

Tools for Storage Monitoring: iostat

The world of Linux storage tools for both monitoring and management just stinks. But that doesn't mean that there is absolutely nothing to help with monitoring. One such tool, iostat, can be used to watch what your storage devices are up to.

In general you get two type of reports with iostat (you can use options to get one report or the other but the default is both types of reports). The first report has CPU usage and the second report is the device utilization report. The CPU report contains the following information:

  • %user: Shows the percentage of CPU utilization that occurred while executing at the user level (this is the application usage).
  • %nice: Shows the percentage of CPU utilization that occurred while executing at the user level with “nice” priority.
  • %system: Shows the percentage of CPU utilization that occurred while executing at the system level (kernel).
  • %iowait: Shows the percentage of time that the CPU or CPU’s were idle during which the system had an outstanding disk I/O request.
  • %steal: Shows the percentage of time spent in involuntary wait by the virtual CPU or CPU’s while the hypervisor was servicing another virtual processor.
  • %idle: Shows the percentage of time that the CPU or CPU’s were idle and the systems did not have an outstanding disk I/O request.

Some of these values should be fairly familiar to you. The values are computed as system-wide averages for all processors when your system has more than 1 core (which is pretty much everything today).

The second report is all about the device utilization. It prints out all kinds of details about the device utilization (can be a physical device or a partition). If you don’t use a device on the command line, then iostat will print out values for all devices (alternatively you can use “ALL” as the device). Typically the report output includes the following:

  • Device: Device name
  • rrqm/s: The number of read requests merged per second that were queued to the device
  • wrqm/s: The number of write requests merged per second that were queued to the device
  • r/s: The number of read requests that were issued to the device per second.
  • w/s: The number of write requests that were issued to the device per second.
  • rMB/s: The number of megabytes read from the device per second.
  • wMB/s: The number of megabytes written to the device per second.
  • avgrq-sz: The average size (in sectors) of the requests that were issued to the device.
  • avgqu-sz: The average queue length of the requests that were issued to the device.
  • await: The average time (milliseconds) for I/O requests issued to the device to be served. This includes the time spent by the requests in queue and the time spent servicing them.
  • svctm: The average service time (milliseconds) for I/O requests that were issued to the device.
  • %util: Percentage of CPU time during which I/O requests were issued to the device (bandwidth utilization for the device). Device saturation occurs when this values is close to 100%.

These fields are specific to the set of options used as well as the device.

It is also important to remember that the first report generated by iostat provides values (statistics) using the time interval since the system was booted. All subsequent reports use the time interval that you specify.

If you go back to the example you will see that the first report (the combination of the first CPU report and first device report) have non-zero values. Remember that this first report computes values based on the system since it first booted. The reports after the first one show little if any activity on /dev/md0 but you do notice some differences in the CPU reporting.

Let’s take a look at more interesting example – running iostat while the system is running iozone.

IOstat Example with IOzone

The first example was pretty boring with nothing happening. Let’s try using iostat when running iozone to show something more interesting (helps with understanding iostat).

Before starting iozone I started iostat with the following command:

[laytonjb@test64 IOSTAT]$ iostat -m -x /dev/sdd 1

The options I used are:

  • I’m using extended output (“-x”) to get more output
  • I want the output to be in megabytes (“-m”) rather than blocks
  • I’m examining the device, /dev/sdd which is an Intel X25-E SSD (it rocks)
  • Finally the last input “1″ tells iostat that I want a report every second and I want it to go on indefinitely until I interrupt it (the indefinitely comes from the fact that I didn’t give it a second input after the “1″).

At first the output is pretty boring since I haven’t started iozone.

Linux 2.6.35-rc4+ (test64) 	11/13/201

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.63    0.13    0.39    9.20    0.00   88.65

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.44     0.06  0.14  0.09     0.00     0.00    25.98     0.00    0.26   0.21   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.25    0.00    0.25    0.00    0.00   98.50

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.25    0.00    0.00   99.50

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

Notice that the read throughput (“rMB/s”) is zero as is the write performance (“wMB/s”). There are also no read requests (“r/s”) nor write requests (“w/s”). Finally the CPU load is very load (less than 2%) and there are iowaits happening.

The output below shows what happens when iozone is started.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.25    0.00    0.00   99.50

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.00    0.00    0.75   16.46    0.00   81.80

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00  4.00  1.00     0.27     0.00   110.40     0.01    1.20   1.00   0.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    0.50   24.56    0.00   74.94

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00  0.00  0.00     0.00     0.00     0.00     0.00    0.00   0.00   0.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00   13.47   11.72    0.00   74.31

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00    12.00  0.00 13.00     0.00     0.10    15.38     0.01    0.69   0.23   0.30

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.75    0.00   26.25   18.50    0.00   54.50

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 55628.00  0.00 301.00     0.00   144.06   980.17    92.88  234.15   2.31  69.50

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.50    0.00   10.25   40.25    0.00   49.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 50282.00  0.00 392.00     0.00   195.96  1023.80   135.48  345.05   2.55 100.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.00    0.00    6.73   43.14    0.00   49.13

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 47117.00  0.00 390.00     0.00   195.00  1024.00   141.50  350.19   2.57 100.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           1.25    0.00   10.00   40.25    0.00   48.50

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 54991.00  0.00 418.00     0.00   209.00  1024.00   135.03  339.14   2.39 100.00

At first the CPU usage is very low and there are no iowaits. Then the second report shows some iowaits (16.46%) which keeps increasing. You can also see the system CPU utilization increasing steadily (this is a 4-core AMD Opteron system). Then part way through the output you see the write throughput (“wMB/s”) start at 0.10 MB/s and then kick up to 144.06 MB/s and it continues increasing. During this time, the read throughput is zero. This shows the beginning of the first iozone test which is a write throughput test. You also see the number of write requests (“w/s”) increasing greatly when the write throughput increases. The same is true for the average size (in sectors) of the requests to the device (“avgrq-sz”) and the average queue length for /dev/sdd (“avgqu-sz”).

One interesting statistic that I like to examine is “%util”. This will tell me if the underlying device is approaching saturation or 100% utilization. In the case of the write throughput test the %util stays low at first then quickly reaches 100%, indicating, you guessed it, that we have saturated the device.

We can go forward through the iostat output to find the same kind of output for reading.

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    3.74   46.38    0.00   49.88

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 52308.91  0.00 416.83     0.00   208.42  1024.00   141.10  338.68   2.38  99.11

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    2.25   41.00    0.00   56.75

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00 27560.00  0.00 341.00     0.00   170.01  1021.04   122.06  407.75   2.93 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    0.50   24.25    0.00   75.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00 230.00 18.00    28.23     7.50   295.10     2.22   38.24   4.02  99.80

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.00    0.00    2.00   23.25    0.00   74.75

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00 766.00  0.00    95.75     0.00   256.00     1.95    2.55   1.31 100.10

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    5.75   20.00    0.00   74.00

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00 1967.00  0.00   245.75     0.00   255.87     1.86    0.95   0.51 100.00

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           0.25    0.00    4.98   20.90    0.00   73.88

Device:         rrqm/s   wrqm/s   r/s   w/s    rMB/s    wMB/s avgrq-sz avgqu-sz   await  svctm  %util
sdd               0.00     0.00 1976.00  0.00   246.88     0.00   255.87     1.88    0.95   0.51 100.20

Notice the transition from write throughput (“wMB/s”) to read throughput (“rMB/s”). What is also interesting is the percentage of iowaits goes down once the read throughput test has started. I believe this is because SSD’s have amazing read performance. Also notice that the average queue length for /dev/sdd (“avgqu-sz”) goes down when switching over to read testing. Again, I believe this is because of the amazing read performance of SSD’s. However, the %util for the device is 100% during the read testing.

There are many ways you can use iostat for monitoring your storage server. For example, a simple way is that if your storage server load starts going up, you can use iostat to look for the offending device. This can be done by using the keyword “ALL” in place of the specific device so that you can find the offending device or devices. You can have iostat generate a report every second for a few minutes and see which device (or devices) has the problem. During this time you should examine the throughput as well as the queue information. Plus I like to examine the “%iowait” output to see if the system has a large back log of I/O operations waiting. I can couple this with the information in %util to determine if the device is saturated and how much data is in the queue.

Summary

I think it is fairly obvious that there is a severe lack of good good, integrated tools for managing and monitoring storage in Linux. As misery loves company, this is true of storage in general. Given that data is growing extremely fast, one would think that it would be logical for there to be a good set of tools for storage, but that obviously isn’t the case.

While I am ranting about the horrible state of storage tools, I also recognize that there are some tools out there that are pretty much the “fingers in the dike” for storage management/monitoring and we must at least understand how to use them.

IOstat is one of the tools that comes in the sysstat package. It can provide some useful information about the state of storage servers both from a CPU perspective and an underlying storage device perspective. It can give you a reasonable amount of data including the number of I/O operations and throughput. It also can give you information on the state of the I/O queues which helps you understand if the there is a great deal of data remaining to be serviced or if the queues are lightly loaded indicating that the storage devices are keeping up with the workload.

So when your storage servers are greatly loaded and/or storage performance is suffering, iostat can help you begin to uncover what is happening with your system.

Comments on "Tools for Storage Monitoring: iostat"

I really enjoy the article.Thanks Again. Will read on…

Here are some hyperlinks to web-sites that we link to because we feel they are really worth visiting.

Very few sites that occur to be comprehensive below, from our point of view are undoubtedly nicely really worth checking out.

Check below, are some entirely unrelated websites to ours, even so, they may be most trustworthy sources that we use.

Here are several of the web-sites we suggest for our visitors.

That could be the end of this post. Right here you will locate some web pages that we feel you?ll enjoy, just click the links.

Here are some links to sites that we link to simply because we feel they are really worth visiting.

Here are several of the websites we suggest for our visitors.

Although internet sites we backlink to below are considerably not associated to ours, we feel they’re really really worth a go by, so possess a look.

I simply want to tell you that I am just beginner to blogging and site-building and truly liked this web blog. Almost certainly I’m going to bookmark your website . You surely have superb writings. Many thanks for sharing with us your webpage.

That could be the end of this post. Right here you?ll discover some sites that we think you?ll appreciate, just click the links.

Here are some links to web-sites that we link to due to the fact we assume they’re worth visiting.

Please go to the websites we comply with, like this 1, as it represents our picks in the web.

We came across a cool site which you might enjoy. Take a look in the event you want.

Below you?ll obtain the link to some web pages that we consider it is best to visit.

Check below, are some completely unrelated internet sites to ours, on the other hand, they may be most trustworthy sources that we use.

Wonderful story, reckoned we could combine a couple of unrelated data, nonetheless really worth taking a look, whoa did one understand about Mid East has got additional problerms as well.

One of our visitors just lately encouraged the following website.

Check beneath, are some absolutely unrelated internet websites to ours, nonetheless, they are most trustworthy sources that we use.

Every once in a while we select blogs that we study. Listed below would be the most up-to-date sites that we decide on.

Below you?ll discover the link to some sites that we assume you’ll want to visit.

Leave a Reply