IO Profiling of Applications: MPI Apps

In the last article we looked at using strace to examine the IO patterns of simple serial applications. In the High Performance Computing (HPC) world, applications use MPI (Message Passing Interface) to create parallel applications. This time around we discuss how to attack parallel applications using strace.

A Simple Example

Let’s start with a simple example from the MPI-2 book by Bill Gropp, et al. In Chapter 2 the authors present a simple example of an MPI code where each process of a total of N processes writes data to an individual file (this is usually referred to as N-N IO). I modified the code to write more data than originally presented. Here is the modified C code:

/* example of parallel Unix write into separate files */
#include "mpi.h"
#define BUFSIZE 100000

int main(int argc, char *argv[])
   int i, myrank, buf[BUFSIZE];
   char filename[128];
   FILE *myfile;

   MPI_Init(&argc, &argv);
   MPI_Comm_rank(MPI_COMM_WORLD, &myrank);
   for (i=0; i < BUFSIZE; i++)
      buf[i] = myrank * BUFSIZE + i;
   sprintf(filename, "testfile.%d", myrank);
   myfile = fopen(filename, "w");
   fwrite(buf, sizeof(int), BUFSIZE, myfile);
   return 0;

I don't want to cover the MPI functions used in the code in this article but you can see that the basic code is almost the same as the serial code in the previous article. This program has each of the MPI processes create it's own output file ("testfile.#" where "#" is the number of the MPI process) and write some data to it.

The two scripts, main.sh and code1.sh, that are used to run the application are fairly simple. The main script, main.sh, looks like the following:

mpirun -machinefile ./MACHINEFILE -np 4 /home/laytonj/TESTING/code1.sh

I didn't hard-code the path to mpirun in the script but I could easily do that to make sure I get the correct mpirun (this can sometimes be a problem in the HPC world). However, I did hard-code the path to the code script to make sure I executed the script I wanted.

The "code1.sh" script that actually contains the command line we want to execute the MPI executable but also using strace.

/usr/bin/strace -T -ttt -o /tmp/strace.out.$$ /home/laytonj/TESTING/test1_mpi $@

where "test1_mpi" is the name of the MPI executable I created from the above code.

Next, you run the "main.sh" script. When the job is finished you have to go to each node used in the run, and copy the files from /tmp back to whatever file system is more permanent than /tmp. You could write all of the strace output files to a central file system, but you run the risk that you could get two PIDs that are the same. The chances of this are fairly small, but I don't like to take this chance.

Analyzing the Strace Output

When I ran the code using the "main.sh" script on my simple quad-core desktop, four output files were created in the /tmp directory:

  • strace.out.4301
  • strace.out.4302
  • strace.out.4303
  • strace.out.4304

In the serial equivalent of the MPI code presented here, the strace file contained only 33 lines (the number of lines can vary depending upon your distribution, compiler, etc.). For the MPI example each strace file contained 1,240 lines! A great deal of the output is related to MPI - loading MPI .so libraries, running MPI functions, network connections, etc.

Let's pull out some highlights from the first strace output file, strace.out.4301. Note that line numbers are included to illustrate relative locations of interesting data in the file.

      1 1267931716.455334 execve("/home/laytonjb/ARTICLES/STORAGE039/test1_mpi", ["/home/laytonjb/ARTICLES/STORAGE0"...], [/* 78 vars */]) = 0 <0.000105>
      2 1267931716.455687 brk(0)                = 0x602000 <0.000021>
      3 1267931716.455748 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f09bd9b5000 <0.000009>
      4 1267931716.455817 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f09bd9b4000 <0.000022>
      5 1267931716.455875 access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory) <0.000016>
      6 1267931716.455952 open("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/tls/x86_64/libmpi.so.0", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000023>
      7 1267931716.456020 stat("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/tls/x86_64", 0x7fffe1b72300) = -1 ENOENT (No such file or directory) <0.000011>
      8 1267931716.456070 open("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/tls/libmpi.so.0", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000013>
      9 1267931716.456124 stat("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/tls", 0x7fffe1b72300) = -1 ENOENT (No such file or directory) <0.000011>
     10 1267931716.456180 open("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/x86_64/libmpi.so.0", O_RDONLY) = -1 ENOENT (No such file or directory) <0.000024>
     11 1267931716.456242 stat("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/x86_64", 0x7fffe1b72300) = -1 ENOENT (No such file or directory) <0.000021>
     12 1267931716.456308 open("/home/laytonjb/bin/openmpi-1.4-gcc-4.1.2/lib/libmpi.so.0", O_RDONLY) = 3 <0.000012>
     13 1267931716.456505 read(3, "\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\200\243\1\0\0\0\0\0"..., 832) = 832 <0.000015>
     14 1267931716.456565 fstat(3, {st_mode=S_IFREG|0755, st_size=818907, ...}) = 0 <0.000011>


   1166 1267931718.542514 open("testfile.0", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 10 <0.000026>
   1167 1267931718.542567 fstat(10, {st_mode=S_IFREG|0664, st_size=0, ...}) = 0 <0.000004>
   1168 1267931718.542603 mmap(NULL, 4096, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_ANONYMOUS, -1, 0) = 0x7f09bd253000 <0.000007>
   1169 1267931718.542629 write(10, "\0\0\0\0\1\0\0\0\2\0\0\0\3\0\0\0\4\0\0\0\5\0\0\0\6\0\0\0\7\0\0\0"..., 397312) = 397312 <0.000792>
   1170 1267931718.543460 write(10, "\0\204\1\0\1\204\1\0\2\204\1\0\3\204\1\0\4\204\1\0\5\204\1\0\6\204\1\0\7\204\1\0"..., 2688) = 2688 <0.000019>
   1171 1267931718.543501 close(10)             = 0 <0.000006>


   1231 1267931718.546397 open("/openmpi-sessions-laytonjb@test64_0/3400466433/0", O_RDONLY|O_NONBLOCK|O_DIRECTORY) = -1 ENOENT (No such file or directory) <0.000007>
   1232 1267931718.546443 munmap(0x7f09bcc2b000, 2106776) = 0 <0.000014>
   1233 1267931718.546499 munmap(0x7f09bce2e000, 2101136) = 0 <0.000010>
   1234 1267931718.546547 munmap(0x7f09bd02f000, 2115040) = 0 <0.000011>
   1235 1267931718.546582 munmap(0x7f09bbfe7000, 2103408) = 0 <0.000010>
   1236 1267931718.546619 munmap(0x7f09bbde5000, 2102824) = 0 <0.000009>
   1237 1267931718.546651 munmap(0x7f09ba9a2000, 2111512) = 0 <0.000010>
   1238 1267931718.546682 munmap(0x7f09b92d3000, 2138064) = 0 <0.000010>
   1239 1267931718.546712 munmap(0x7f09b90c7000, 2146256) = 0 <0.000011>
   1240 1267931718.546766 exit_group(0)         = ?

So you can see that the first 1,165 lines do all kinds of things, mostly related to MPI but also starting the application itself. Then at line 1,166 the local file testfile.# is opened. Lines 1,169 and 1,170 actually perform the write() syscalls to the output file.

While not pertinent to the discussion of using strace, notice that the file name listed on line 1,166 has the file name "testfile.0". The code uses the "number" of the particular MPI process - in this case, this is the 0-th (zero-th) process. The other processes will have file names such as "testfile.1", "testfile.2", and "testfile.3".

As with the serial example in the previous article, let's walk through the strace output and examine some of the statistics for the IO functions, starting with the write() syscalls. The elapsed time for the first write() syscall is 0.000792 seconds (792 microseconds). The amount of data written was 397,312 bytes (the same as the serial case). So the throughput was,

397312 Bytes / 0.000792 secs. = 501,656,565.7 Bytes/s = 501.66 MB/s

We can do the same computations to the second write():

2688 Bytes / 0.000019 secs = 141,473,684.2 Bytes/s = 141.4736 MB/s

The total elapsed time for the application is also easy to compute.

1267931718.546766 - 1267931716.455334 = 2.091432 seconds


While getting strace output is generally easy, in this article we found that not to be the case for general MPI codes. We had to create two scripts so we could get the strace output from each MPI process (which is what we really want) rather than the strace from the MPI command used to start the application. After those scripts are created getting the strace output for any number of MPI processes is quite easy (Note: you can always add some lines to the scripts to copy the strace files back to your home directory or some centralized location).

One thing that is fairly obvious is that even simple codes can produce a great deal of output. Going from a simple serial code to an MPI code increased the number of lines by a factor of 40. Most of that addition is the startup and shutdown code for the application which is somewhat the same for any application but which can increase with the number of processes used. Just imagine having a real application that could produce several hundred thousand lines of strace output.

What happens if you have a much more complicated application? Would you examine the IO syscalls by hand? Actually the whole examination process cry's out for some sort of automation.

Comments on "IO Profiling of Applications: MPI Apps"

That might have been a tribute to Steve Mc – Queen and the movie Bullitt.

You can access all recorded information on computer from
any part of the world. Shop 4 Books is an online bookstore for everything books.

Write more, thats all I have to say. Literally,
itt seems as though you relied on the video to make your point.
Yoou clearly know what youre talking about, why throw away
your intelligence on just posting videos to yoour
site when you could be givinmg us something informative to read?

My webpage Cheap Car Insurance

Hi there just wanted to give you a brief heads upp aand lett you know a
feww of the images aren’t loadinjg properly. I’m nott sure why
but I think its a linking issue. I’ve tried it in two different web browsers and both show the
same results.

Stop by my web-site; cheap car insurance

Howdy excellent blog! Does running a blog such ass this
take a largge amount of work? I have absolutely no knowledge of programming however I
had been hoping to start my own blog in the near future.
Anyways, hould you have any suggestions or techniques for new blog owners please share.

I know this is off subject nevertheless I simply had to ask.
Many thanks!

Here is my homepage cheap car insurance

Hello, i read your blog occasionally and i own a siumilar one and i was just curious if you get
a lot of spoam responses? If so hhow do you reduce it,
any plugin oor anything you can advise? I geet so much lately it’s driving mee mad so any assistance is very much appreciated.

My blog post cheap car insurance in ma

Hello there! Do you use Twitter? I’d liike to follow
you if that would bbe okay. I’m undoubtedly enjoying your blog and
look forward to new updates.

Feel free to surf to my webpage :: cheap car insurance

I blog frequently andd I really appreciate your information. This great article has really peaked my interest.
I wilkl bookmark your site and keep checking for new details about once perr week.
I subscribed to your Feed as well.

my blog: cheap full coverage car insurance

I absolutely love your blog and find many oof your post’s to be precisely what
I’m looking for. Would you offer guestt writers to write content for
yourself? I wouldn’t mind composing a polst or elaborating on most of the subjects you write related to here.
Again, awesome website!

my web page cheap car insurance

Everything is very open with a really clear
explanation of the challenges. It was truly informative.
Your site is very helpful. Thank you for sharing!

hey there and thank you for your info – I have certainly picked up something new from right here.
I did however expertise some technical points using this
weeb site, sibce I experienced to reload the web site a lot of times prdevious to I could get it to load properly.

I had been wondering if your hosting is
OK? Not that I’m complaining, but slow loading instances times will sometimes aaffect your
placement in google and can damage your high-quality score if advertising and marketing with Adwords.
Anyway I’m adding this RSS to my email and could look out for
a lot more of your respective intriguing content. Ensure that you update this again very soon.

My web site :: cheap car insurance in md

If some oone wishes to be updted with most up-to-date techynologies then he must
be goo to see this website and be up to date everyday.

Look into mmy web site cheap car insurance

You’re so cool! I do not believe I have read anything like
this before. So wonderful to find another person with unique thoughts on this subject.
Really.. thanks ffor starting this up. This website
is something that is needed on the internet,
someone wiith a little originality!

Here is my homepage cheap full coverage car insurance

Hmm it seems like your site ate my first comment (it was extremely long) so I guess I’ll just sum it up
what I submitted and say, I’m thoroughly enjoying your
blog. I too am an aspiring blog writer but I’m still new tto the whole thing.
Do you hve any tips and hints for rookie blog writers?
I’d definitely appreciate it.

My web-site cheap car insurance in pa

Admiring the timme and effort you put into your blog and in depth
information you provide. It’s awesome to come
across a blog every oncfe in a whiloe that
isn’t the same out of date rehashed information. Excellent read!
I’ve saved your site and I’m including your RSS feeds to my Google account.

My website – cheap car insurance in florida

Hello myy loved one! I wish too say that this article is awesome, great written and come with almost
all vital infos. I’d like to see more posts like this .

Alsoo visit my blog cheap car insurance

heloo there and thank you forr your info – I have definitely picked up something new
from right here. I did however expertise ssome technical points using this web site, since
I experienced to reload the web site many times previous
to I could get it tto load correctly. I had been wondering if your hosting is
OK? Not that I’m complaining, but sluggish loading
instances times will often affect youur placement in google andd could damage your quality score if
ads and marketing with Adwords. Anyway I am adding this RSS
to mmy e-mail and can look out for much more of your respective intrioguing content.
Make sure you update this agasin very soon.

Here is my blog post – cheap car insurance

Hello! I simply would like to give you a huge thumbs up for the great info you have got right here on this post.
I am coming back to your website for more

My web blog :: top car insurance companies

At this time I am going to do my breakfast, once having my breakfast coming again to read other

Feel free to surf to my web blog … cheap car insurance in ga

Do you have a spam problem on this website; I also am a blogger, and I was wondering your situation; we
have created some nice methods and we are looking to exchange solutions with others, poease shoot me an e-mail if interested.

Feel free to visiut my web-site … cheap full coverage car insurance

Thanks for sharing your thoughts on cheap classic car insurance.

My web-site: Cheap Car Insurance

I’ll immediately take hold of your rss as I can not in finding your email subscription hyperlink or newsletter service.
Do you have any? Please allow me realize in order that I may just
subscribe. Thanks.

Have a look at my blog cheap car insurance in pa

I am so grateful for your article.Really looking forward to read more. Want more.

Looking forward to reading more. Great article post.Thanks Again. Will read on…

That is a very good tip especially to those fresh to the blogosphere.
Simple but very precise info… Appreciate your sharing this one.
A must read article!

I’ll right away clutch your rss as I can not in finding your email subscription hyperlink or newsletter service.
Do you’ve any? Kindly permit me recognize so that I could subscribe.

Feel free to visit my weblog: canadian pharmacy

Because you are serious about your domain flipping business,
you should treat it as such because you are looking for long term growth
and not just quick short term money. It is important to
set a time schedule for working. How to Become Successful: The Four Blueprint Success Questions.

Here is an excellent Blog You might Discover Interesting that we encourage you to visit.

Hi there to all, how is the whole thing, I think every one is getting more from this site, and your views are pleasant
for new people.

my web blog :: discount drugs canada

Thanks for sjaring your thoughts on Best Free Bets ?C UK Betting
Offers & Bonuses. Regards

my weblog :: Cricket Bookies Free bonus

9 million dollars in 4 months (Last year Carbon Copy Pro paid 6 millions – just to give you an idea
of how fast EN is going) David Wood’s goal is to pay $50 Million this year which is totally doable specially now that EN is all over the world and has the empower network ewallet in place.

Simply press your index finger just beneath the knot as it forms.

Thus, the expected growth and demand for CSR providers are expected to intensify.

Here is my page giuseppe zanotti mens

Hi there! Do you know if they make any plugins to help with Search
Engine Optimization? I’m trying to get my blog too rnk for some ttargeted keywords but I’m
noot seeing very good success. If you know of any please share.
Thank you!

Feel free to surf to my site dyson robot vacuum

Hello There. I found your blog using msn. This is a really well written article.

I will make sure to bookmark it and return to read more of your useful information. Thanks for the post.
I’ll definitely return.

Also visit my page – dyson bladeless fan

I visit each day some web sites and blogs to read articles, but this website presents
quality based content.

Founded in 1984, this brand is completely dedicated to the modern girl who is independent and romantic and has a flair for vintage designs.
Thus, if you want to look for the best in Italian designer shoes, Dior fur shoes
is of great your choice and offers stylish beauty for the tasteful ladies.
It is meant to provide comfort and proper support so you can compete at the top

my webpage … giuseppe zanotti outlet

The adjoining Highland Park Golf Studying Heart offers 50
grass and mat lighted hitting stations with efficiency membership
becoming, PGA skilled instruction, brief recreation area and
miniature golf.

My site golf n stuff

Frank Breimling is a recogniz?d expert in Affiliate Marketing.
For example, it can give you advice on the maintrnance of inventories
of cor?orate mergers annd acquisitions, and expnsion of thhe business.

?ou ?aan also find Bisk textbooks t?at ?ave
tips, ideas and strateg?es ?egarding how to plan for the
CPA exam, along with 1000s of practice questions supoported bby
in-depth expl?nations of correct and incor?ect answers.

Also visit my web-site :: boca Raton cpa’s

CBT ?s the only format in which the tte?t is offereed but if reque?tged at
the time of application, certain accommo?ations in accordance with thee Am?ricans with Disabilit?es
Act (ADA) can be m?de f?r specific candidates.
For most small non-farm business?s, thee dsductions are taken oon a Sc?edule C (Profit or Loss? From B?siness),
or a Schedule C-EZ (for thosee w?th deductions of ?es thaan $5000).
The skills yoou acquire in y?ur training to become a
CPA will g?ve you confidence in yo?rself and your newfound abilities.

Also visit my site; ft lauderdale Accountant

Jim Tr?ppon CPA and his team offer comprehensive personal and business C?A ser?ices for clients near the B?nker Hill
Village area. Those are some of t?e important
characteri?tics that you w?nt to look for as you’re sear?hing for a ?PA.

A CPA must complete a certain amount off CPE (Continuing Profe?sional Education) units each year.

Alsso visit my blog :: Boca Raton Cpas

One of our visitors lately proposed the following website.

Yes! Finally something about Watch Warcraft Free.

Please visit the web pages we follow, including this one, as it represents our picks in the web.

Here are several of the web-sites we advise for our visitors.

I simply want to tell you that I am newbie to weblog and absolutely liked your web-site. Most likely I’m planning to bookmark your website . You amazingly come with wonderful writings. Many thanks for sharing with us your web site.

The info talked about inside the article are a few of the ideal readily available.

I am truly grateful to the owner of this website who has
shared this impressive post at here.

Leave a Reply