Avoiding Common MPI Programming Errors

Convenience and ease-of-use were NOT design goals of MPI. Learn how to avoid some common pitfalls.

Pitfalls of Non-Blocking Point-to-Point Communication

Non-blocking point-to-point communication allows a program to initiate a message (send or receive) then proceed without waiting for the message to complete. This enables the overlap of communication with computation in order to hide communication latency. In other words, the program can do useful work while messages are in transit. (At least this is the theory. Most MPI implementations, even modern ones, are weak in this regard. The state-of-the- art hasn’t changed much since White and Bova published “Where’s the Overlap? An Analysis of Popular MPI Implementations” almost ten years ago.) The only restriction is that the program must not access the send or receive buffers while the communication is in progress. Listing 3A illustrates an obvious data race on a send-buffer. The program must verify that the point-to-point communication is finished before it is safe to reuse the message buffers. Strictly speaking, overwriting the send-buffer with the same data or even reading the send-buffer while the message is in transit violates the MPI standard so the code in Listing 3B and 3C is also incorrect.

It’s easy to understand why modifying the send-buffer during message transfer is unsafe but it’s not immediately apparent why reading the send-buffer is unsafe. A real-world example helps to explain. Let’s say an MPI program is running on a heterogeneous cluster containing big-endian and little-endian systems. An MPI library that supports this environment may convert the send- buffer from big-endian to little-endian and vice versa. This would make the value of the variable “var” indeterminate in Listing 3C. The MPI-2.2 standard may relax this restriction, but for now, it is unsafe to even read the send- buffer while the message is in transit.

Listing 3A: This code contains a data race because the send-buffer is modified while a message is still in transit.

   buf = 1;
   var = 2;
   MPI_Isend (&buf, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &req);
   buf = var;
   MPI_Wait (&req, &stat);

Listing 3B: Overwriting the send-buffer, even with the same data, is not allowed while a message is in transit.

   buf = var = 1;
   MPI_Isend (&buf, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &req);
   buf = 1;
   MPI_Wait (&req, &stat);

Listing 3C: Even reading the send-buffer while a message is in transit violates the MPI standard. The MPI-2.2 standard may relax this restriction.

   MPI_Isend (&buf, 1, MPI_INT, 1, 0, MPI_COMM_WORLD, &req);
   var = buf;
   MPI_Wait (&req, &stat);

Listing 3D shows how race conditions are also possible when using buffered communication. The program in Listing 3D runs correctly most of the time because the attached buffer is almost large enough to accommodate all of the data being sent. “Almost” is never good enough in parallel programming. The program is prone to failure if the previously buffered sends have not completed by the time the final message is sent. Attaching and reattaching the send-buffer inside the loop solves the race condition but hurts performance. Increasing the buffer size is a better solution.

Listing 3D: Performing buffered sends inside of a loop can also cause a race condition if the attached buffer is too small. This example is adapted from the book Using MPI (2nd Edition) by W. Gropp, E. Lusk, and A. Skjellum.

#include "mpi.h"

int main (int argc, char *argv[])
   int i, rank, bsize, s, r;
   int sbuf[1*MPI_BSEND_OVERHEAD];
   MPI_Status stat;

   MPI_Init (&argc, &argv);
   MPI_Comm_rank (MPI_COMM_WORLD, &rank);

   if (rank == 0)
      bsize = 4 + MPI_BSEND_OVERHEAD;
      MPI_Buffer_attach (&sbuf, bsize);

      for (i = 0; i < 25; i++)
         s = i;
         MPI_Bsend (&s, 1, MPI_INT, 1, 0, MPI_COMM_WORLD);
      MPI_Buffer_detach (&sbuf, &bsize);
   else if (rank == 1)
      for (i = 0; i < 25; i++)
         MPI_Recv (&r, 1, MPI_INT, 0, 0, MPI_COMM_WORLD, &stat);
         printf ("Rank %d received %d\n", rank, r);

   MPI_Finalize ();

$ mpiexec -n 2 ./a.out

[cli_0]: aborting job:
Fatal error in MPI_Bsend: Invalid buffer pointer, error stack:
MPI_Bsend(184).......: MPI_Bsend(buf=0x7fffffffde98, count=1, MPI_INT, dest=1, tag=0, MPI_COMM_WORLD) failed
MPIR_Bsend_isend(338): Insufficient space in Bsend buffer; requested 4; total buffer size is 99
rank 0 in job 130  compute-00-00_41645   caused collective abort of all ranks
  exit status of rank 0: return code 13 

When using non-blocking communication, good MPI programming practice dictates that every pending message should be checked for completion. Failure to do so can create data races, as shown above, but it can also lead to another problem: overflowing the message queue. Most MPI implementations can handle a large number of pending messages so this problem often escapes detection during development. Over time, however, production-scale workloads fill the queue with pending messages and prevent further communication, ultimately causing the program to fail. This is akin to a memory leak.

MPI Resource Leaks

In addition to overflowing the message queue, other "leaks" are possible in the MPI library. Two such leaks concern communicators and attached message buffers.

The default communicator, MPI_COMM_WORLD, is sufficient for many MPI programs. Some applications, however, create new communicators to logically divide work. Consider a weather forecasting application, for example, that uses different communicators for its ocean and atmospheric models. An application using boss- worker parallelism can create distinct communicators for groups of worker processes assigned to different tasks. Parallel math libraries often create new communicators to keep their messages separate from the parent application. There are many good reasons for using communicators other than MPI_COMM_WORLD but it is important to realize that the MPI library can only support a finite number of communicators. A parallel math library that is called multiple times during a long-running simulation could potentially exceed the number of available communicators.

Similarly, the MPI_Bsend function requires that extra storage be allocated for the message buffer and attached with MPI_Buffer_attach. In the same way that sequential programs should free dynamically allocated memory when it is no longer needed, MPI programs must be careful to free communicators and detach message buffers (with MPI_Buffer_detach) when they are no longer needed.

Incomplete Collective Operations

A common misconception about MPI collective functions is that they behave like barriers. In the case of MPI_Bcast, for example, this would mean that no rank in the communicator can proceed until all ranks have completed the broadcast operation. This is actually implementation-dependent. (The obvious exception is MPI_Barrier, which always behaves like a barrier.) Some MPI implementations may have synchronous collectives but others may use asynchronous collectives to achieve better performance. However, an MPI program must not rely on a particular behavior. It must be correct regardless of whether the implementation uses synchronous or asynchronous collective operations. The MPI Forum provides an example to illustrate this point (Listing 4):

Listing 4: This program is incorrect because the broadcast operations are executed in reverse order. Assume that MPI_COMM_WORLD has only two MPI processes.

   switch (rank)
      case 0:
         buf1 = 1;
         MPI_Bcast (&buf1, 1, MPI_INT, 0, MPI_COMM_WORLD);
         MPI_Bcast (&buf2, 1, MPI_INT, 1, MPI_COMM_WORLD);
      case 1:
         buf2 = 2;
         MPI_Bcast (&buf2, 1, MPI_INT, 1, MPI_COMM_WORLD);
         MPI_Bcast (&buf1, 1, MPI_INT, 0, MPI_COMM_WORLD);

Correctness dictates that all ranks of a communicator must execute a collective operation in the same order. Deadlock will occur in Listing 4 if the MPI implementation uses a synchronous broadcast. This problem can escape detection during development if the developer uses an MPI implementation with asynchronous collectives but the end-user's MPI library has synchronous collectives. The MPI Forum provides several good examples of incorrect use of collective communication.

Concluding remarks

Though convenience and ease-of-use were not design goals of MPI, it is still the most effective and widely-used method for achieving scalable, parallel performance. The MPI Forum has begun working on the MPI 3.0 standard, which could make future MPI programs more fault-tolerant and easier to debug. In the meantime, avoiding common mistakes and pitfalls can go a long way to making MPI programming easier. If all else fails, several good tools are available for debugging and profiling MPI applications.


  • J.B. White III and Steve W. Bova, "Where's the Overlap? An Analysis of Popular MPI Implementations" MPI Developers Conference 1999.

  • Joe Landman, MPI in Thirty Minutes Linux Magazine, April 2008.

  • William Gropp, Ewing Lusk, and Anthony Skjellum, Using MPI: Portable Parallel Programming with the Message-Passing Interface (2nd Edition), The MIT Press, 1999.

  • Rusty Lusk and Bill Gropp, "Are We Stuck with MPI Forever?" ClusterWorld, March 2005.

Comments on "Avoiding Common MPI Programming Errors"

This website was… how do I say it? Relevant!! Finally I’ve found something that helped me. Kudos!

Hi there, I enjoy reading through your article post. I wanted to write a little comment to support you.

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I’ve got you saved as a favorite to check out new things you post…

It’s actually a nice and useful piece of information. I’m glad that you simply shared this useful information with us. Please stay us informed like this. Thanks for sharing.

Hi there colleagues, fastidious article and pleasant arguments commented at this place, I am actually enjoying by these.

It’s a pity you don’t have a donate button! I’d definitely donate to this brilliant blog! I guess for now i’ll settle for book-marking and adding your RSS feed to my Google account. I look forward to brand new updates and will talk about this site with my Facebook group. Chat soon!

If you would like to improve your knowledge only keep visiting this website and be updated with the hottest gossip posted here.

Hi there to all, the contents present at this website are truly awesome for people knowledge, well, keep up the good work fellows.

First, create a separate take into account bridal sareesand wedding sarees. After doing my research for my own white affair outfit, I have a few suggestions to help you get started with your outfit. Whether dressing for an office party, a black-tie affair, Tim Gunn will share his tips and show to ‘make it work’ all holiday season long. It has a cover strap with a trendy snap clip to securely lock in your device. One more very great advantage of short sleeve dress is nothing but their great availability in the market. Even the ladder leading up to the roof of her house is colorful and is made of pencil crayons. I am sure he will like the gift and I am very satisfied with my online shopping experience.

Do you have a spam problem on this blog; I also am a blogger, and I was wondering your situation; we have developed some nice methods and we are looking to swap solutions with other folks, why not shoot me an e-mail if interested.

What makes it intriguing is that all these are accessible in an extensive variety of sizes and designs in order to bestow an immaculate fit as well. s an occasion of special kind, then they will go for every possible way to enhance their look. The versatility of Olian Maternity makes their dresses a very worthwhile purchase. This way you will surely be the most fashionable girl at the party. The length of your plus size formal dresses must also be considered. Every breath you take makes you feel like a princess.

Really appreciate you sharing this post.Really looking forward to read more. Really Great.

Wow, that’s what I was exploring for, what a stuff! existing here at this blog, thanks admin of this website.

Hey, thanks for the blog post.Really looking forward to read more. Will read on…

No slipping in line, no crowds with any hassle, no buying parking slot. Satin is a densely woven silk with a lustrous sheen on one side. Hasidism tradition is conservative in dress, philosophy and adherence to Jewish Law. These shirts are defined by designers as garments that have buttons, a stiff and formally designed collar and a full length opening in the front, which is also called a button-front or simply a button shirt. To know the type of the event you can see the invitation because your invitation would give you the clue that what sort of event it is. Another key feature of the maxi dress is its versatility.

Most of the women are greatly satisfied and pleased with comfortability of these dresses as they are tight from upper end, while they remain bit loose from waist to the bottom. The ongoing recessional phase of the global economy has forced many people to stipulate their wedding budget. A woman is much tensed during her wedding and uncomfortable dress can keep her fidgeting, so a strapless dress can be best option to make her comfortable. In these days the women are only crazy for highly famous, nicely designed and absolutely fit dresses which carve their physical beauty and make them more appreciated among the others. Some styles are not suited to shorter women as it looks proportionally unequal. But in many cases, we find that there is similar taste for summer clothing for women with a very minor difference of opinion in style or pattern.

Hello there! This is kind of off topic but I need some guidance from an established blog. Is it very hard to set up your own blog? I’m not very techincal but I can figure things out pretty quick. I’m thinking about making my own but I’m not sure where to begin. Do you have any ideas or suggestions? Cheers

Every flower girl is of different age group and hence needs different dresses and not mandatory but they should match the theme. Belts come in every shape, size and color, creating an endless string of possibilities. They can perhaps even do significantly better when compared with regular office work when it comes to earning opportunities. Let us face it, a tight white nurses uniform with a red cross on it would be simple to spot by even the most sheltered members of society. Instead of creating a garment that is designed to be worn all day, modern designers strive to create variants on the cocktail dress that look sumptuous without the stiff formality of evening wear. Word spreads rapidly at whatever point there is a great arrangement out there.

Hello to every one, it’s in fact a pleasant for me to pay a quick visit this website, it consists of valuable Information.

You really make it seem so easy with your presentation but I find this topic to be really something which I think I would never understand. It seems too complicated and very broad for me. I am looking forward for your next post, I’ll try to get the hang of it!

Usually posts some very intriguing stuff like this. If you?re new to this site.

Good day I am so glad I found your web site, I really found you by error, while I was looking on Bing for something else, Nonetheless I am here now and would just like to say many thanks for a remarkable post and a all round entertaining blog (I also love the theme/design), I don’t have time to go through it all at the minute but I have bookmarked it and also added your RSS feeds, so when I have time I will be back to read much more, Please do keep up the awesome b.

The best idea is to wait until the last few days before the party before having them select a costume. This action will not increase the amount of product that you get onto the brush; it will only cause air to get trapped inside the bottle or tube. Instead of opting to keep yourself warm with pashminas and cardigans, opt to wear oh-so-trendy pastel blazers and accentuate it with the stark contrast of neon accessories. It helps to search for the more unique costume that other party guests aren’t likely to arrive in. Tall and slim girls can choose some of the long dresses to wear, so you can try to avoid long legs, others too small to cause the feeling of a too abrupt. It might also benefit to check the weather for the day or evening of the event.

Excellent post. I was checking continuously this blog and I am impressed! Very helpful info specifically the closing section :) I maintain such information much. I used to be looking for this particular information for a very long time. Thank you and best of luck.

This can be as straightforward as wiping the bag with a soft scarf, or as extensive as using a leather cleaner. Still finding out all through which time for more info with regards to buy Mace bombard spray readily available around the internet I found an accessible around the web retailer that has to supply you likely being the Mace pepper baton in all places inside the black. A complete woman always take care of her family ,works in the office with full vigour and at the sametime woman always wants her to be pretty.

Have you ever considered about including a little bit more than just your articles? I mean, what you say is valuable and everything. Nevertheless imagine if you added some great photos or videos to give your posts more, “pop”! Your content is excellent but with images and videos, this blog could certainly be one of the best in its niche. Excellent blog!

Alas, you still have to go out and you can’t go without. For snowy scenes, the right side of the histogram should show more pixels than the left. Despite the fact that options for the amount of clothes you are able to pair with leggings are indeed limited, their very own variations are actually plenty. Choose the color that works the best for your coloring and makes you happiest. For dessert serve a large round cake decorated as a Smiley Face or a Peace symbol. com is a beautiful soft, simple, elegant layout that features beautiful pale pastels, the picture of Easter itself. Hasidism tradition is conservative in dress, philosophy and adherence to Jewish Law.

The floral pattern here features a black foundation with flowers that feature bold, bright colors like purples, blues, pinks, oranges and even white on a modern floral pattern. Transgendered men also should be aware of the profound effect that these hormones have on the male body and should take these hormones only under a doctor’s supervision. If you’re looking to taunt the men of your city, then you could boast a shorter dress that exposes some of your thighs. If you are dressing for the office this summer, you do not have to feel left out. This is especially helpful when the winter weather gets rough. If you’re planning a very informal beach ceremony you’ll want to wear a casual and relaxed dress. This is most aptly utilized if the opposing fencer is somewhat familiar with your fencing style, including your favorite feints and defensive measures.

I’m really enjoying the design and layout of your website. It’s a very easy on the eyes which makes it much more enjoyable for me to come here and visit more often. Did you hire out a developer to create your theme? Superb work!

Young boys can wear a brighter colored t-shirt over their white one for cooler mornings. The stylish wedge heel is part of a rocker sole bottom that promotes healthy walking, making the Addison a joy to wear for a night on the town. Below is some designers brands that sell sundresses. Shirts and pants, towels, linens and sportswear were not only folded and stacked, but actually looked as though folded by automation. is yet another fabulous shoe from the Artisan Collection. “Cotton Incorporated Lifestyle Monitor Historically, most men (34%) say a good-looking woman looks sexiest in a casual sundress, according to the Cotton Incorporated Lifestyle Monitor. Taylor Swift goes for gold and ink on arm as she appears on the stage in Columbus, Ohio during her ‘Speak Now’ tour.

When the shirt sleeves aren’t peeking out from the jacket sleeves, it just gives a look of nakedness. With online marketing rapidly growing in popularity; SEO consultants, PPC specialists and anyone else in between is now trying to grab your attention and promise your business ultimate results but there are a few things you will need to be wary off before you jump into bed with an SEO agency. Bengali New Year is also popularly referred to as Poila Boisakh and is celebrated with much fanfare among the Bengali community in India. You can also integrate a variety of colors into one unified force that presents a message that is much more powerful than words. Many cultures actually prefer a white and blue wedding dress. so why not make sure everyone remembers the dress you wore to the event. If a red party dress is not good enough to add the wow factor to your look, let us go ahead and add some ruffles to it.

Cotton is one of the most common materials used in making clothes, as it is very casual and yet it can be made very beautiful to look very fashionable, and at the same time is incredibly inexpensive. They are neither too Indian oriented nor too westernized. Prices must be affordable so take time to look for a wholesale clothes supplier who can provide highly discounted products. Clothes that are too tight are also unflattering as they tend to show and cause unsightly bulges and a give a lumpy silhouette. Special bridesmaid dresses are offered at discount sale. Look out for all of the deals these sites are offering and grab yourself some bargain pieces.

I was curious if you ever considered changing the page layout of your website? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having 1 or two pictures. Maybe you could space it out better?

It’s going to be end of mine day, except before ending I am reading this great paragraph to increase my experience.

I really enjoy the blog.Thanks Again. Much obliged.

Check below, are some entirely unrelated web-sites to ours, having said that, they may be most trustworthy sources that we use.

I simply want to mention I am beginner to blogs and seriously liked this blog site. Probably I’m want to bookmark your blog post . You surely come with good posts. With thanks for sharing with us your blog.

It is extremely helpful for me. Thank you for taking the time to discuss this. I feel strongly about it and love learning more on this

topic. If possible, as you gain expertise, would you mind updating your blog with more information.

It is a small step you can take to keep your precious pup looking and feeling fantastic this winter. She launched the Material Girl clothing line with daughter Lourdes this year. These stores also offer cheap ladies apparel china brand on wholesale basis at extremely cheap rates usually there are abundant wholesalers of inexpensive ladies clothes china brand and if possible getting their direct contact with them aids one purchasing inexpensive china brand girls clothing which fits one to the most and there is the factor of buying more at less costs. There is a $35 minimum purchase if you don’t have it. You may laugh all you want, my friends, but this book is the first work of what I would like to call psychology of fashion. com’s Kindle format is that any qualified author can publish an e – Book in a matter of a day or two (assuming they own their own ISBN numbers, which I do). With her trademark pageboy bob haircut and sunglasses, Vogue’s editor-in-chief has become an institution through the fashion world.

Leave a Reply