Avoiding Common MPI Programming Errors

Convenience and ease-of-use were NOT design goals of MPI. Learn how to avoid some common pitfalls.

The Message-Passing Interface (MPI) has been around since 1994. It defines an API that allows programmers to transfer data between concurrent processes. MPI is very versatile and gives programmers fine control over all aspects of inter-process communication and synchronization. Therefore, MPI generally performs better than higher-level parallel programming methods. However, the programmer is responsible for each message and must coordinate all of the communication between processes. Many users equate MPI programming to assembly language programming. Few will claim that MPI is easy to use. Indeed, in a retrospective on MPI (“Are We Stuck with MPI Forever?” Cluster World, March 2005), Rusty Lusk and Bill Gropp, two of the original authors of the MPI standard, wrote that “convenience and ease of use were not high-priority goals of the MPI design…” In spite of this, MPI is the lingua franca for HPC because of its portability, generality, and perhaps most important, attainable performance.

This article shows some of the MPI programming errors that are frequently found in customer programs during application projects at Intel. It is assumed that the reader is already familiar with the MPI distributed-memory programming model. (Programmers who are new to MPI should take a look at Joe Landman’s excellent “MPI in Thirty Minutes” tutorial.) It is assumed that the reader knows how to send messages in MPI, understands the difference between blocking and non-blocking communication, and is familiar with MPI collective operations. This article focuses on common programming errors related to the MPI message passing model, API, and implementation-specific details.

Pitfalls in Blocking Point-to-Point Communication

Communication and synchronization are tightly coupled in MPI programs. Nothing illustrates this more clearly than blocking, point-to-point communication. For example, an MPI process sends a message to another process. In blocking communication, both the sender and the receiver wait for the message to be transferred before continuing. (Later we’ll see that there are exceptions but for now let’s assume that this is always true.) Because blocking communication is entirely synchronous, deadlock will occur unless each send has a matching receive. In Listing 1A, even numbered ranks send data to the odd ranks but a logic error causes deadlock. [Note: Complete source for the examples can be found here.] Specifically, the send and receive operations are not properly matched because message sources and destinations are not computed correctly. Similarly, the code in Listing 1B only works correctly for an even number of MPI processes. While the receivers and senders have the correct source and destination ranks, an odd number of processes means one more receiver than there are senders.

Listing 1A: Deadlock due to unmatched sends and receives

   if (rank % 2 == 0)
      dest = (rank + 1 < n_proc) ? rank + 1 : MPI_PROC_NULL;
      MPI_Send (buf, N, MPI_INT, dest, 0, MPI_COMM_WORLD);
      source = (rank + 1 < n_proc) ? rank + 1 : MPI_PROC_NULL;
      MPI_Recv (buf, N, MPI_INT, source, 0, MPI_COMM_WORLD, &stat);

Listing 1B: Another example of unmatched sends and receives when the number of MPI processes is odd

   if (rank % 2 == 0)
      source = rank + 1;
      MPI_Recv (buf, N, MPI_INT, source, 0, MPI_COMM_WORLD, &stat);
      dest = rank - 1;
      MPI_Send (buf, N, MPI_INT, dest, 0, MPI_COMM_WORLD);

Unmatched sends and receives is the first, and most obvious, pitfall of point- to-point communication. Nonetheless, the code in Listing 2A will deadlock even though the sends and receives are properly matched. This is because each MPI process performs a blocking receive before a corresponding send is posted. By the same logic, the program in Listing 2B should also deadlock because each MPI process performs a blocking send before a corresponding receive is posted. However, this is not always the case. The program does deadlock with an informative error message when run with only one process. When run with more than one MPI process it surprisingly works.

Listing 2A: MPI ranks passing data in a ring but performing blocking receives before the corresponding sends are posted, resulting in deadlock

   right = (rank + 1 == n_proc) ? 0 : rank + 1;
   left = (rank - 1 < 0) ? n_proc - 1 : rank - 1;

   MPI_Recv (buf, N, MPI_INT, left, 0, MPI_COMM_WORLD, &stat);
   MPI_Send (buf, N, MPI_INT, right, 0, MPI_COMM_WORLD);

Listing 2B: Performing a blocking send before a corresponding receive should cause deadlock but sometimes it does not. The error output is from the Intel MPI Library, which is based on MPICH2.

#include "mpi.h"

#define N 65536

int main (int argc, char *argv[])
   int rank, n_proc, right, left, buf[N];
   MPI_Status stat;

   MPI_Init (&argc, &argv);
   MPI_Comm_rank (MPI_COMM_WORLD, &rank);
   MPI_Comm_size (MPI_COMM_WORLD, &n_proc);

   right = (rank + 1 == n_proc) ? 0 : rank + 1;
   left = (rank - 1 < 0) ? n_proc - 1 : rank - 1;

   MPI_Send (buf, N, MPI_INT, right, 0, MPI_COMM_WORLD);
   MPI_Recv (buf, N, MPI_INT, left, 0, MPI_COMM_WORLD, &stat);

   printf ("Rank %d sent message to rank %d\n", rank, right);
   printf ("Rank %d received message from rank %d\n", rank, left);

   MPI_Finalize ();

$ mpicc listing-2b.c -o listing-2b.exe
$ mpirun -np 1 ./listing-2b.exe
[cli_0]: aborting job:
Fatal error in MPI_Send: Other MPI error, error stack:
MPI_Send(175): MPI_Send(buf=0x7ffffffbde50, count=65536, MPI_INT, dest=0, tag=0, MPI_COMM_WORLD) failed
MPID_Send(57): DEADLOCK: attempting to send a message to the local process without a prior matching receive
rank 0 in job 51  compute-00-00_41645   caused collective abort of all ranks
  exit status of rank 0: killed by signal 9 

$ mpiexec -n 4 ./listing-2b.exe
Rank 2 sent message to rank 3
Rank 2 received message from rank 1
Rank 0 sent message to rank 1
Rank 0 received message from rank 3
Rank 3 sent message to rank 0
Rank 3 received message from rank 2
Rank 1 sent message to rank 2
Rank 1 received message from rank 0

Remember the previous assertion that blocking communication is entirely synchronous and that neither the sender nor receiver can continue until message transfer is complete? This is not entirely true for most MPI implementations, which brings us to the topic of internal buffering. Notice the size of the messages being sent in Listing 2B. In most MPI implementations, MPI_Send actually behaves like a non-blocking send when the message size is below a certain threshold. However, the MPI specification is very clear that programs must not rely on this behavior:

"The reluctance of MPI to mandate whether standard sends are buffering or not stems from the desire to achieve portable programs. Since any system will run out of buffer resources as message sizes are increased, and some implementations may want to provide little buffering, MPI takes the position that correct (and therefore, portable) programs do not rely on system buffering in standard mode." [source]

Since this threshold can vary, incorrect programs may work fine with one MPI library but deadlock with another. Such problems often escape detection because debugging workloads tend to have smaller messages than production workloads or the MPI library used during development uses a larger internal buffer than an end-user's library. This error is also common in MPI programs that were translated from PVM because programmers mistakenly assume that because PVM_Send is a non-blocking function, MPI_Send is also non-blocking. Seeing MPI_Send behave like a non-blocking function confirms their assumption. Nevertheless, their program now contains a dormant deadlock.

Comments on "Avoiding Common MPI Programming Errors"

This website was… how do I say it? Relevant!! Finally I’ve found something that helped me. Kudos!

Hi there, I enjoy reading through your article post. I wanted to write a little comment to support you.

I wanted to thank you for this excellent read!! I definitely loved every little bit of it. I’ve got you saved as a favorite to check out new things you post…

It’s actually a nice and useful piece of information. I’m glad that you simply shared this useful information with us. Please stay us informed like this. Thanks for sharing.

Hi there colleagues, fastidious article and pleasant arguments commented at this place, I am actually enjoying by these.

It’s a pity you don’t have a donate button! I’d definitely donate to this brilliant blog! I guess for now i’ll settle for book-marking and adding your RSS feed to my Google account. I look forward to brand new updates and will talk about this site with my Facebook group. Chat soon!

If you would like to improve your knowledge only keep visiting this website and be updated with the hottest gossip posted here.

Hi there to all, the contents present at this website are truly awesome for people knowledge, well, keep up the good work fellows.

First, create a separate take into account bridal sareesand wedding sarees. After doing my research for my own white affair outfit, I have a few suggestions to help you get started with your outfit. Whether dressing for an office party, a black-tie affair, Tim Gunn will share his tips and show to ‘make it work’ all holiday season long. It has a cover strap with a trendy snap clip to securely lock in your device. One more very great advantage of short sleeve dress is nothing but their great availability in the market. Even the ladder leading up to the roof of her house is colorful and is made of pencil crayons. I am sure he will like the gift and I am very satisfied with my online shopping experience.

Do you have a spam problem on this blog; I also am a blogger, and I was wondering your situation; we have developed some nice methods and we are looking to swap solutions with other folks, why not shoot me an e-mail if interested.

What makes it intriguing is that all these are accessible in an extensive variety of sizes and designs in order to bestow an immaculate fit as well. s an occasion of special kind, then they will go for every possible way to enhance their look. The versatility of Olian Maternity makes their dresses a very worthwhile purchase. This way you will surely be the most fashionable girl at the party. The length of your plus size formal dresses must also be considered. Every breath you take makes you feel like a princess.

Really appreciate you sharing this post.Really looking forward to read more. Really Great.

Wow, that’s what I was exploring for, what a stuff! existing here at this blog, thanks admin of this website.

Hey, thanks for the blog post.Really looking forward to read more. Will read on…

No slipping in line, no crowds with any hassle, no buying parking slot. Satin is a densely woven silk with a lustrous sheen on one side. Hasidism tradition is conservative in dress, philosophy and adherence to Jewish Law. These shirts are defined by designers as garments that have buttons, a stiff and formally designed collar and a full length opening in the front, which is also called a button-front or simply a button shirt. To know the type of the event you can see the invitation because your invitation would give you the clue that what sort of event it is. Another key feature of the maxi dress is its versatility.

Most of the women are greatly satisfied and pleased with comfortability of these dresses as they are tight from upper end, while they remain bit loose from waist to the bottom. The ongoing recessional phase of the global economy has forced many people to stipulate their wedding budget. A woman is much tensed during her wedding and uncomfortable dress can keep her fidgeting, so a strapless dress can be best option to make her comfortable. In these days the women are only crazy for highly famous, nicely designed and absolutely fit dresses which carve their physical beauty and make them more appreciated among the others. Some styles are not suited to shorter women as it looks proportionally unequal. But in many cases, we find that there is similar taste for summer clothing for women with a very minor difference of opinion in style or pattern.

Hello there! This is kind of off topic but I need some guidance from an established blog. Is it very hard to set up your own blog? I’m not very techincal but I can figure things out pretty quick. I’m thinking about making my own but I’m not sure where to begin. Do you have any ideas or suggestions? Cheers

Every flower girl is of different age group and hence needs different dresses and not mandatory but they should match the theme. Belts come in every shape, size and color, creating an endless string of possibilities. They can perhaps even do significantly better when compared with regular office work when it comes to earning opportunities. Let us face it, a tight white nurses uniform with a red cross on it would be simple to spot by even the most sheltered members of society. Instead of creating a garment that is designed to be worn all day, modern designers strive to create variants on the cocktail dress that look sumptuous without the stiff formality of evening wear. Word spreads rapidly at whatever point there is a great arrangement out there.

Hello to every one, it’s in fact a pleasant for me to pay a quick visit this website, it consists of valuable Information.

You really make it seem so easy with your presentation but I find this topic to be really something which I think I would never understand. It seems too complicated and very broad for me. I am looking forward for your next post, I’ll try to get the hang of it!

Usually posts some very intriguing stuff like this. If you?re new to this site.

Good day I am so glad I found your web site, I really found you by error, while I was looking on Bing for something else, Nonetheless I am here now and would just like to say many thanks for a remarkable post and a all round entertaining blog (I also love the theme/design), I don’t have time to go through it all at the minute but I have bookmarked it and also added your RSS feeds, so when I have time I will be back to read much more, Please do keep up the awesome b.

The best idea is to wait until the last few days before the party before having them select a costume. This action will not increase the amount of product that you get onto the brush; it will only cause air to get trapped inside the bottle or tube. Instead of opting to keep yourself warm with pashminas and cardigans, opt to wear oh-so-trendy pastel blazers and accentuate it with the stark contrast of neon accessories. It helps to search for the more unique costume that other party guests aren’t likely to arrive in. Tall and slim girls can choose some of the long dresses to wear, so you can try to avoid long legs, others too small to cause the feeling of a too abrupt. It might also benefit to check the weather for the day or evening of the event.

Excellent post. I was checking continuously this blog and I am impressed! Very helpful info specifically the closing section :) I maintain such information much. I used to be looking for this particular information for a very long time. Thank you and best of luck.

This can be as straightforward as wiping the bag with a soft scarf, or as extensive as using a leather cleaner. Still finding out all through which time for more info with regards to buy Mace bombard spray readily available around the internet I found an accessible around the web retailer that has to supply you likely being the Mace pepper baton in all places inside the black. A complete woman always take care of her family ,works in the office with full vigour and at the sametime woman always wants her to be pretty.

Have you ever considered about including a little bit more than just your articles? I mean, what you say is valuable and everything. Nevertheless imagine if you added some great photos or videos to give your posts more, “pop”! Your content is excellent but with images and videos, this blog could certainly be one of the best in its niche. Excellent blog!

Alas, you still have to go out and you can’t go without. For snowy scenes, the right side of the histogram should show more pixels than the left. Despite the fact that options for the amount of clothes you are able to pair with leggings are indeed limited, their very own variations are actually plenty. Choose the color that works the best for your coloring and makes you happiest. For dessert serve a large round cake decorated as a Smiley Face or a Peace symbol. com is a beautiful soft, simple, elegant layout that features beautiful pale pastels, the picture of Easter itself. Hasidism tradition is conservative in dress, philosophy and adherence to Jewish Law.

The floral pattern here features a black foundation with flowers that feature bold, bright colors like purples, blues, pinks, oranges and even white on a modern floral pattern. Transgendered men also should be aware of the profound effect that these hormones have on the male body and should take these hormones only under a doctor’s supervision. If you’re looking to taunt the men of your city, then you could boast a shorter dress that exposes some of your thighs. If you are dressing for the office this summer, you do not have to feel left out. This is especially helpful when the winter weather gets rough. If you’re planning a very informal beach ceremony you’ll want to wear a casual and relaxed dress. This is most aptly utilized if the opposing fencer is somewhat familiar with your fencing style, including your favorite feints and defensive measures.

I’m really enjoying the design and layout of your website. It’s a very easy on the eyes which makes it much more enjoyable for me to come here and visit more often. Did you hire out a developer to create your theme? Superb work!

Young boys can wear a brighter colored t-shirt over their white one for cooler mornings. The stylish wedge heel is part of a rocker sole bottom that promotes healthy walking, making the Addison a joy to wear for a night on the town. Below is some designers brands that sell sundresses. Shirts and pants, towels, linens and sportswear were not only folded and stacked, but actually looked as though folded by automation. is yet another fabulous shoe from the Artisan Collection. “Cotton Incorporated Lifestyle Monitor Historically, most men (34%) say a good-looking woman looks sexiest in a casual sundress, according to the Cotton Incorporated Lifestyle Monitor. Taylor Swift goes for gold and ink on arm as she appears on the stage in Columbus, Ohio during her ‘Speak Now’ tour.

When the shirt sleeves aren’t peeking out from the jacket sleeves, it just gives a look of nakedness. With online marketing rapidly growing in popularity; SEO consultants, PPC specialists and anyone else in between is now trying to grab your attention and promise your business ultimate results but there are a few things you will need to be wary off before you jump into bed with an SEO agency. Bengali New Year is also popularly referred to as Poila Boisakh and is celebrated with much fanfare among the Bengali community in India. You can also integrate a variety of colors into one unified force that presents a message that is much more powerful than words. Many cultures actually prefer a white and blue wedding dress. so why not make sure everyone remembers the dress you wore to the event. If a red party dress is not good enough to add the wow factor to your look, let us go ahead and add some ruffles to it.

Cotton is one of the most common materials used in making clothes, as it is very casual and yet it can be made very beautiful to look very fashionable, and at the same time is incredibly inexpensive. They are neither too Indian oriented nor too westernized. Prices must be affordable so take time to look for a wholesale clothes supplier who can provide highly discounted products. Clothes that are too tight are also unflattering as they tend to show and cause unsightly bulges and a give a lumpy silhouette. Special bridesmaid dresses are offered at discount sale. Look out for all of the deals these sites are offering and grab yourself some bargain pieces.

I was curious if you ever considered changing the page layout of your website? Its very well written; I love what youve got to say. But maybe you could a little more in the way of content so people could connect with it better. Youve got an awful lot of text for only having 1 or two pictures. Maybe you could space it out better?

It’s going to be end of mine day, except before ending I am reading this great paragraph to increase my experience.

I really enjoy the blog.Thanks Again. Much obliged.

Check below, are some entirely unrelated web-sites to ours, having said that, they may be most trustworthy sources that we use.

I simply want to mention I am beginner to blogs and seriously liked this blog site. Probably I’m want to bookmark your blog post . You surely come with good posts. With thanks for sharing with us your blog.

It is extremely helpful for me. Thank you for taking the time to discuss this. I feel strongly about it and love learning more on this

topic. If possible, as you gain expertise, would you mind updating your blog with more information.

It is a small step you can take to keep your precious pup looking and feeling fantastic this winter. She launched the Material Girl clothing line with daughter Lourdes this year. These stores also offer cheap ladies apparel china brand on wholesale basis at extremely cheap rates usually there are abundant wholesalers of inexpensive ladies clothes china brand and if possible getting their direct contact with them aids one purchasing inexpensive china brand girls clothing which fits one to the most and there is the factor of buying more at less costs. There is a $35 minimum purchase if you don’t have it. You may laugh all you want, my friends, but this book is the first work of what I would like to call psychology of fashion. com’s Kindle format is that any qualified author can publish an e – Book in a matter of a day or two (assuming they own their own ISBN numbers, which I do). With her trademark pageboy bob haircut and sunglasses, Vogue’s editor-in-chief has become an institution through the fashion world.

Leave a Reply