dcsimg

Concurrent and Parallel Are Not The Same

Pick one: portability or efficiency. Neither is guaranteed when writing explicit parallel code

In case you did not have a chance to read the column from last week, I am taking my yearly vacation at the Jersey Shore. Please refrain from the jokes, lest I pull out the Bruce Springsteen trump card. I try to spend two continuous weeks with family and friends each year. I have found that one week is just too short. I need two weeks. The first week is used to try and forget about all the stuff I did not get done before I threw the laptop in the car and say “let’s go.” The second week is used to try and remember and organize all the stuff I have to do when I get back. My plan usually breaks down somewhere around 10 AM my first day back to work.

This year I had a bunch of writing to do (including this column), so it was kind of a working vacation. Not to worry, I’ll have my feet in the Atlantic Ocean in few hours. In any case, my dilemma is as follows. Write an insightful column quickly and get to the beach. It may surprise some readers, but I do like to research some of the topics I write about. At a minimum, I like to include enough URLs so that if you actually want to investigate a topic further, more information is just a click away. As an aside, I am constantly amazed at how much content on the web has absolutely no external links to supporting material. I thought that was the whole idea. I mean how hard is it to add a Wikipedia link to a discussion of Clos Networks or some other networking technology.

Back to my dilemma. What can talk about that will get me to beach before the water ice guy packs up for day? Although, I don’t to like to rehash things I have written about in the past, I will be making a an exception this week. Not necessarily because it is easy, but because I think some messages need reinforcing. Therefore, all I have to decide is what message I should I hammer home on this July morning.

The answer is simple — understanding the difference between concurrent and parallel. I believe these two terms are often used interchangeably while, in my opinion, they are represent two different concepts.

Let’s start with concurrency. A concurrent program or algorithm is one where operations can occur at the same time. For instance, a simple integration, where numbers are summed over an interval. The interval can be broken into many concurrent sums of smaller sub-intervals. As I like to say, concurrency is a property of the program. Parallel execution is when the concurrent parts are executed at the same time on separate processors. The distinction is subtle, but important. And, parallel execution is a property of the machine, not the program.

If execution efficiency is important (i.e. you want things to go faster by adding more cores), then the question you need to ask is “If I run everything that is concurrent in parallel, will my code run faster?” If the answer were “yes” then we would not be having this discussion. And, since the answer, is “no”, then the question is “What should run in parallel?” which is obviously, the portions of code that lower execution time.

This decision is one of the reasons cluster parallel computing is hard. It really does depend on the machine. Take our integration case. If the integration interval is small, then breaking it up into small sub-intervals and sending them out to other nodes will result in extending the execution time of the program due to parallel overhead. If the integration interval is huge, then parallel execution may make sense. Because parallel overhead can vary from cluster to cluster, there is no easy way to predict overhead beforehand. (i.e. The parallel overhead is larger for GigE vs InfiniBand when sending small packets.)

The same applies to multi-core. The overhead for thread communication is lower, but there is still overhead (see my HPC Hopscotch for background on SMP memory). There is no free lunch — everyone has to deal with overhead.

In summary, the point I want to make is this, Concurrency is a property of the program and parallel execution is a property of the machine. What concurrent parts should and should not be executed in parallel can only be answered when the exact hardware is known. Which I might like to add leads to the most unhappy conclusion when dealing with explicit parallel programming, There is no guarantee of both efficiency and portability with explicit parallel programs. Yes, I know, a sad state of affairs. I’ll let you wrestle with that for a while, in the mean time, I’m going to the beach.

Comments on "Concurrent and Parallel Are Not The Same"

very nice
i want to ask that “if two processes are concurrently executing then they holds efficiency and portability or not?”

Reply

Great blog post.Thanks Again. Really Cool.

Reply

Say, you got a nice post.Thanks Again. Will read on…

Reply

Yesterday, while I was at work, my sister stole my iphone and tested to see if it can survive a forty foot drop, just so she can be a youtube sensation. My iPad is now broken and she has 83 views. I know this is completely off topic but I had to share it with someone!

Reply

I am just truly influenced using your writing skills together with with the design and style to your weblog. Is this the settled topic and also have you customize it yourself? In any event .. wordpress themes keep up to date the wonderful high-quality composing, it can be unusual to see a terrific website like this one right now.

Reply

Hello. excellent job. I did not anticipate this. This is a remarkable story. Thanks!

Reply

Woh I like your posts, saved to bookmarks! .

Reply

Useful info. Fortunate me I found your site unintentionally, and I’m surprised why this coincidence didn’t came about earlier! I bookmarked it.

Reply

I will right away snatch your rss as I can at find your email subscription hyperlink or e-newsletter service. Do you have any? Please permit me understand in order that I may just subscribe. Thanks.

Reply

Below you will locate the link to some internet sites that we consider you should visit.

Reply

Always a major fan of linking to bloggers that I love but don?t get a good deal of link like from.

Reply

Very handful of internet websites that happen to be in depth beneath, from our point of view are undoubtedly properly really worth checking out.

Reply

Hi there. I found your web site by means of Google whilst looking for a similar subject, your website came up. It looks good. I have bookmarked it in my google bookmarks to visit then.

Reply

wIBuEN Svens Bilder Laufen Marathon Triathlon Fotos

Reply

Simply a smiling visitor here to share the love (:, btw outstanding pattern.

Reply

Howdy! This is my first visit to your blog! We are a team of volunteers and starting a new initiative in a community in the same niche. Your blog provided us valuable information to work on. You have done a wonderful job!

Reply

We came across a cool web page that you just may delight in. Take a search when you want.

Reply

Usually posts some very interesting stuff like this. If you?re new to this site.

Reply

Here are a few of the web pages we recommend for our visitors.

Reply

Here are some links to internet sites that we link to simply because we feel they may be worth visiting.

Reply

Check beneath, are some entirely unrelated websites to ours, even so, they may be most trustworthy sources that we use.

Reply

Although websites we backlink to below are considerably not related to ours, we feel they may be in fact really worth a go via, so possess a look.

Reply

Although web-sites we backlink to below are considerably not connected to ours, we feel they are basically really worth a go by, so possess a look.

Reply

Check beneath, are some completely unrelated web-sites to ours, even so, they’re most trustworthy sources that we use.

Reply

Although websites we backlink to below are considerably not connected to ours, we feel they are really worth a go by means of, so have a look.

Reply

We like to honor quite a few other online internet sites on the net, even though they aren?t linked to us, by linking to them. Underneath are some webpages worth checking out.

Reply

Just beneath, are many entirely not connected websites to ours, however, they are certainly really worth going over.

Reply

Here are some links to websites that we link to due to the fact we feel they’re worth visiting.

Reply

You got a very fantastic website, Sword lily I detected it through yahoo.

Reply

The details mentioned inside the report are a few of the most beneficial readily available.

Reply

That may be the end of this report. Here you will find some web pages that we consider you?ll enjoy, just click the links.

Reply

We like to honor lots of other net web pages around the internet, even when they aren?t linked to us, by linking to them. Beneath are some webpages worth checking out.

Reply

That is the end of this post. Right here you?ll locate some web sites that we believe you will enjoy, just click the hyperlinks.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>