Comments on: Intro to IO Profiling of Applications

By: Kern Sibbald

Kern Sibbald — Fri, 18 May 2012 07:45:48 +0000

Hmm. This article is a bit old, but I stumbled across it. Having written my own fwrite routine in 1982, I can make a good guess why glibc’s fwrite splits the write into two parts. First fwrite() is buffering writes, most likely in blocks of 4096 bytes. When you write enough data to fwrite, at some point (probably when it is more than 4096 bytes), it will pass that data directly to the system via a write() call, but it will up to the largest multiple of 4096 bytes. The rest of the data, in your case 2688 bytes, it keeps in its buffer waiting for more data. In your case, the fclose() flushes those remaining 2688 bytes to disk in a second call to the kernel.

By: dougalb

dougalb — Wed, 30 Nov -0001 00:00:00 +0000

would you not consider \’oprofile\’ or \’systemtap\’ also effective tools for gathering IO inforamtion?

By: laytonjb

laytonjb — Wed, 30 Nov -0001 00:00:00 +0000

@dougalb,

Those are system profilers, not application profiles. I\'m after the ability to look at what the application wants/needs. I don\'t think either can do that.

I haven\'t used oprofile before on systems though. Looks interesting.

I saw some things on lwn.net that indicate systemtap is not a favorite of many kernel hackers. I think there is something new being developed.

Jeff

By: jsoda

jsoda — Wed, 30 Nov -0001 00:00:00 +0000

the write is broken up into two calls because fwrite is a library call that executes its own logic before calling write(2). if you replace fopen/fwrite/fclose with open/write/close and update your code accordingly, i suspect you\'ll see one write.
page size writing alignments do not come into play at this level.

By: laytonjb

laytonjb — Wed, 30 Nov -0001 00:00:00 +0000

@jsoda,

Thanks for the insight! Any idea why fwrite splits the write?

Thanks!

Jeff

By: john.fusco

john.fusco — Wed, 30 Nov -0001 00:00:00 +0000

The -c option is useful for profiling as well. It doesn\'t give you elapsed time with the -T option, unfortunately. However, it does give you counts which is helpful.

I use it on running processes when I want to know what an unresponsive process is doing (if anything).

By: gordonmessmer

gordonmessmer — Wed, 30 Nov -0001 00:00:00 +0000

Your understanding of strace seems to be slightly flawed. It does not trace \"function calls\" in libraries. Instead it traces system calls to the kernel. If you wanted to trace library function calls you might use ltrace instead. That\'s an important distinction since there is a difference between the write() function call provided by glibc and the write() system call provided by the kernel.

The dual-write call is not a function of your compiler. Your request is passed as a single call to glibc\'s fwrite(), which then decides how to pass it to the kernel. It appears to pass one block of data which is composed of as many full filesystem blocks (gathered by the fstat() call) as possible, followed by any remaining data. Beyond that, I don\'t know why it doesn\'t pass the whole request to the kernel and let Linux figure it out.

You contend that there are no other tools for gathering this information, but that\'s not quite true. :)

As others have mentioned, Oprofile can give you a good idea of both what your application and your system are doing during its execution. You can also use tools like gprof and valgrind (with callgrind) to get \"real\" profiling information and use tools that are already available for graphing and visualizing the resulting data. Hopefully that will save you some time when writing the upcoming articles you mentioned at the end of your article.

I\'m very glad to see this written, and look forward to additional articles. I fully agree that profiling is important, but widely neglected.

By: laytonjb

laytonjb — Wed, 30 Nov -0001 00:00:00 +0000

@gordonmessmer

You are correct that my writing about strace was flawed. Believe it or not I actually understand strace but I didn\'t explain it well :) Thanks for pointing that out. I\'ve been so close to profiling for so long I just forgot how to explain it.

I didn\'t know about Oprofile until it was mentioned here. I will definitely give it a whirl to see what it can do for profiling applications.

Once again - thanks for pointing how my flawed explanation of strace. I will fix that ASAP (but it takes a while to propagate).

Thanks!

Jeff

By: jirihorky

jirihorky — Wed, 30 Nov -0001 00:00:00 +0000

Hi Jeff,
Nice article. I have written something similar with a GUI interface and ability to trace child processes too (-f switch). It also plots the access pattern diagram for each file. If you are interested, please check it out on http://code.google.com/p/ioapps/