Comments on: Small HPC http://www.linux-mag.com/id/7362/ Open Source, Open Standards Sat, 05 Oct 2013 13:48:18 +0000 hourly 1 http://wordpress.org/?v=3.1 By: kalloyd http://www.linux-mag.com/id/7362/#comment-6544 kalloyd Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7362/#comment-6544 Doug,<br /> <br /> This is a great start on what's going on in HPC. But there is much more that multi-core CPUs. Consider the MPI implications between clusters of multi-core CPUs, each core spawning MPP kernels on hundreds of GPU process cores. Some of those simple global or shared memory sections now have multiple routing options - such as xGigE or InfiniBand.<br /> <br /> While I see a great paradigm shift in learning parallel programming, the compute fabrics that result from hybrid, heterogeneous compute clusters means that networked parallel/serial fabrics offer breathtaking power, but come with great complexity.<br /> <br /> This is playing field for serious HPC software developers today.<br /> <br /> Ken Lloyd<br /> Director of Systems Science<br /> Watt Systems Technologies Inc. Doug,

This is a great start on what’s going on in HPC. But there is much more that multi-core CPUs. Consider the MPI implications between clusters of multi-core CPUs, each core spawning MPP kernels on hundreds of GPU process cores. Some of those simple global or shared memory sections now have multiple routing options – such as xGigE or InfiniBand.

While I see a great paradigm shift in learning parallel programming, the compute fabrics that result from hybrid, heterogeneous compute clusters means that networked parallel/serial fabrics offer breathtaking power, but come with great complexity.

This is playing field for serious HPC software developers today.

Ken Lloyd
Director of Systems Science
Watt Systems Technologies Inc.

]]> By: endawobrien http://www.linux-mag.com/id/7362/#comment-6545 endawobrien Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7362/#comment-6545 I expect that shared-memory, multi-threaded HPC will become quite democratic and ubiquitous, since multi-core will be everywhere and the programming tools for this (e.g., OpenMP) are relatively simple. I think of this as "supply-led" HPC.<br /> <br /> However, the real heavyweight macho HPC world is more "demand led", and will continue to exploit the power of distributed-memory systems, while still press-ganging multi-core processors, GPGPUs, FPGAs and whatever other accelerator or configuration will let them run ever bigger models, ever faster.<br /> <br /> Difficulty with programming may limit the adoption (and business success) of some "accelerators", but there will probably always be a hard core (pardon the pun...) of deviant weirdos for whom performance is everything, and who are prepared to go to heroic lengths to exploit parallelism on even the most unwieldy hardware.<br /> <br /> Certainly, a "holy grail" for HPC would be a high-level programming language that could compile an executable to run "transparently" over distributed memory. Some such grails, of intermediate holiness, already exist. Otherwise, a tool to automatically decompose a serial application for MPI would be nice; something analogous to what OpenMP does for multi-threading.<br /> <br /> -Enda I expect that shared-memory, multi-threaded HPC will become quite democratic and ubiquitous, since multi-core will be everywhere and the programming tools for this (e.g., OpenMP) are relatively simple. I think of this as “supply-led” HPC.

However, the real heavyweight macho HPC world is more “demand led”, and will continue to exploit the power of distributed-memory systems, while still press-ganging multi-core processors, GPGPUs, FPGAs and whatever other accelerator or configuration will let them run ever bigger models, ever faster.

Difficulty with programming may limit the adoption (and business success) of some “accelerators”, but there will probably always be a hard core (pardon the pun…) of deviant weirdos for whom performance is everything, and who are prepared to go to heroic lengths to exploit parallelism on even the most unwieldy hardware.

Certainly, a “holy grail” for HPC would be a high-level programming language that could compile an executable to run “transparently” over distributed memory. Some such grails, of intermediate holiness, already exist. Otherwise, a tool to automatically decompose a serial application for MPI would be nice; something analogous to what OpenMP does for multi-threading.

-Enda

]]> By: indivar http://www.linux-mag.com/id/7362/#comment-6546 indivar Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7362/#comment-6546 Hi <br /> <br /> One of the biggest challenge for any Distributed Shared Memory (DSM) system is 'Thread Migration'. While most DSM systems have managed to migrate complete processes from one node to another (and thereby spread the load), almost all of them are struggling with individual thread migration. Now since most of today's applications are multi-threaded as opposed to multi-process, the effectiveness of such a solution is limited. DSM developers are getting there, but it will take a while to perfect it.<br /> <br /> In any case, App Developers will have to now look at writing Hybrid parallel applications, i.e. applications that are multi-process as well as multi-threaded (e.g. Apache HTTP Server, worker module) to really exploit the power of today's systems, be it Hardware based SMP (mulit-core) or Software based SMP (DSM). <br /> <br /> In short, both DSM and MPI developers face the same set of challenges. <br /> <br /> In future, DSM will be used for day-to-day to applications while MPI, because of its sheer ability to scale massively will continue to be used for hardcore number crunching stuff. <br /> <br /> Note: <b>HPC</b> is a relative term. What you call 'Small HPC' or as Microsoft calls it - 'Personal SuperComputer', will simply remain to be called the humble <b>'Personal Computer'</b> or <b>'PC'</b> in future.<br /> <br /> - Indivar Nair Hi

One of the biggest challenge for any Distributed Shared Memory (DSM) system is ‘Thread Migration’. While most DSM systems have managed to migrate complete processes from one node to another (and thereby spread the load), almost all of them are struggling with individual thread migration. Now since most of today’s applications are multi-threaded as opposed to multi-process, the effectiveness of such a solution is limited. DSM developers are getting there, but it will take a while to perfect it.

In any case, App Developers will have to now look at writing Hybrid parallel applications, i.e. applications that are multi-process as well as multi-threaded (e.g. Apache HTTP Server, worker module) to really exploit the power of today’s systems, be it Hardware based SMP (mulit-core) or Software based SMP (DSM).

In short, both DSM and MPI developers face the same set of challenges.

In future, DSM will be used for day-to-day to applications while MPI, because of its sheer ability to scale massively will continue to be used for hardcore number crunching stuff.

Note: HPC is a relative term. What you call ‘Small HPC’ or as Microsoft calls it – ‘Personal SuperComputer’, will simply remain to be called the humble ‘Personal Computer’ or ‘PC’ in future.

- Indivar Nair

]]> By: heymanj http://www.linux-mag.com/id/7362/#comment-6547 heymanj Wed, 30 Nov -0001 00:00:00 +0000 http://www.linux-mag.com/id/7362/#comment-6547 Memory on a per node basis will become the cost point. With the 6core AMD processor supporting upwards of 256GB of memory, one has to think of memory cost when building a large multi-core system.<br /> This fact will come to play with application development as another possible divergence comes into play - small number of multi-core processors with large local memory or large number of multi-core processors with significantly smaller amounts of local memory.<br /> <br /> OpenMP benefits from the large shared memory w/any number of multi-core processors. MPI can do both but will definitely succeed in a large multi-core with smaller individual node memory footprint.<br /> <br /> There is a definite need for something to assist MPI programming along the lines of OpenMP pragmas or compiler smarts to hide all the details. Otherwise in the case of 10Ks of MPI tasks (one task per core) you wind up with the old adage of "not being able to see the forest because you're lost with the individual trees"<br /> <br /> Jerry Heyman Memory on a per node basis will become the cost point. With the 6core AMD processor supporting upwards of 256GB of memory, one has to think of memory cost when building a large multi-core system.
This fact will come to play with application development as another possible divergence comes into play – small number of multi-core processors with large local memory or large number of multi-core processors with significantly smaller amounts of local memory.

OpenMP benefits from the large shared memory w/any number of multi-core processors. MPI can do both but will definitely succeed in a large multi-core with smaller individual node memory footprint.

There is a definite need for something to assist MPI programming along the lines of OpenMP pragmas or compiler smarts to hide all the details. Otherwise in the case of 10Ks of MPI tasks (one task per core) you wind up with the old adage of “not being able to see the forest because you’re lost with the individual trees”

Jerry Heyman

]]>