I have seen (and designed languages for) functional, object-oriented, logic and procedural parallel models. About the most promising approach I have seen so far is a hybrid approach, where major computational elements are sewn together using a data-flow, or functional, or object-oriented high-level language. The individual computational elements can then be expressed in a language that best suits the architecture.

For example, your program might read in a set of matrices, perform an FFT to translate the data from the time to the frequency domain, pass the results through several high-pass and low-pass filters, then pass those results through an inverse FFT to translate it back to the time domain.

The high-level operations — FFT->filters->inverse FFT — are invariant. They, along with descriptions of your data, represent the semantic content of your program. The individual implementations of the FFT, etc., may vary from one system to the next to take advantage of specific hardware. Think of it as a really fast implementation of LAPACK or math.h if you like.

The idea that you can specify the low level details of an algorithm (e.g., matrix inversion) in a machine independent form that can be executed efficiently on architectures as different as GPGPUs and large-scale distributed clusters and BlueGene and Larrabee and TMC\’s CM-1 (remember them?) is … well … a nice idea but, IMHO, not much more than that. In order for the compiler to have enough flexibility to make efficient choices for the architecture in question, the algorithm *must* be expressed at a fairly high level, the higher the better.

I really think that directives can solve most problems efficiently, with a top-down structure MPI-OpenMP-Accelerator, but the bottom-up link, from Accelerator to MPI, might be even more important in order to decide at MPI level what the most efficient solution might be. If Accelerator is intelligent enough to do a break-even code analysis at CUDA level, it should be possible to to the same at MPI level, with or without the use of OpenMP and other tools. And now back to my DEadline :-)

Maybe in another decade or two, processor cores will be managed automatically by higher-level language compilers the way registers are now, and we won\’t care how many general or special purpose cores (or nodes!) a system has. It will probably mean moving to a higher-level language than what\’s commonly used now, so we\’re not programming as close to the hardware as we are now, and leaving the details of what depends on what results and what can be done in parallel (and on which cores) to the compiler and/or OS. Worrying about such details may seem as quaint as worrying about CPU registers today. Of course, the big question is how to get to that point. A lot of hard work went into making today\’s compilers as sophisticated and efficient as they are, and this next challenge seems even more daunting. But I wouldn\’t doubt that it\’s possible.

Given the cost of modifying working code, and the lack of talented programmers who know how to do so, many researchers prefer to just run lots of single core jobs to get their work done. Sure, many of the savvy researchers with grant based funding have gotten their code to scale on multi cores, but none are even considering the daunting task of migrating to a GPU, if they even understood how to do it.

So your observations are very correct. In reality, very large supercomputers, like those at Los Alamos and Livermore, are very efficient at generating heat, but very inefficient at generating results due to the enormous programming effort required to take advantage of such scale.

The challenge becomes integrating a heterogeneous computing solution into your current favorite computing platform. If you have to use some new programming language or a dialect that\’s harder than actually restructuring your app to take advantage of a multi-core processor, it\’s probably not worth it. If it\’s relatively easy, then you get performance/watt increases beyond what\’s possible with off-the-shelf processors, without totally rewriting your application.

I compare it to using an attached array processor back in the 80\’s. Worked great, but most of the time wasn\’t worth it. It wasn\’t until minisupercomputers came along, with integrated vector instructions, that you could get performance increases without working in two different development & runtime environments.

So, if the HLL for parallel programming were to support higher abstractions for operators, such as reductions, partial reductions, etc. — operators that could be parallelized across a wide variety of parallel architectures — then I can see how that might be a path for success. It takes advantage of the best features of the hybrid approach without some of its complexity.

The success of HLLs over assembly came about because they reduced the complexity of programming. It allowed the programmer to stop thinking about low level details that were incidental to the algorithm. Examples of those details might include the number of registers available, whether the data to be operated on was in this register or that, whether branch conditions were handled through direct operators or by testing a condition in one instruction and branching based on a condition register in the next.

In short, the level of abstraction was raised and this helped the programmer, while at the same time it stayed fairly close to the architectures they needed to support. The range in architectures is also pretty narrow, so that helps a lot.

So, could a parallel language be created that would span the spectrum of parallel architectures? I\’m still a bit skeptical, because there are huge differences in parallel architectures, much larger than what we see separating, say, CISC from RISC. But if there is hope it is in identifying a useful set of operators, expressive enough to easily describe a large set of programs, while at the same time compilable into low level code for the available spectrum of parallel architectures. (The hybrid approach assumes no such operators exist and allows the user to define their own.)