Adding Virtualization To My Anxiety Closet

Doug Eadline takes a break from his ranting about multi-core CPUs to rant about another technology that gives him fits: Virtualization.

This month, I thought I’d take a break from my usual multi-core ranting and talk about another new technology that’s starting to give me fits. No, it’s not the somewhat confusing General Purpose Graphical Processing Units (GP-GPUs) acronym, which is not to be confused with the GPU that has something to do with putting colored dots on your screen. No, the thing that’s raising my dander is the virtualization thing, or “virt” for short.

Now, virt is a good idea, even if it’s become an annoying buzzword of late. It kind of makes sense with all the extra cores and all. And while I’m not expert enough to talk about how virt will play out in the larger server market, I do have some thoughts about virtualization in high-performance computing (HPC).

Before Virt Came to Town

Back in the good old days of single-core processors, when HPC clustering was in its infancy, getting the application as close to the hardware was very important. In many cases, it still is. Communication between nodes can take place through the operating system using TCP/IP, or outside the operating system using a user space zero-copy protocol. With the exception of pinning down memory, a user space protocol removes the operating system from the communication, resulting in better latency and often better application performance.

If your application is sluggish over Fast Ethernet and TCP, you must get closer to the hardware and use a specialized interconnect like Myrinet. All high performance interconnects are as close as possible to the hardware. Indeed, the hardware actually assists in the communication.

Let’s recap how programs operate on a cluster. A parallel MPI program is essentially a collection of individual programs (processes) running on the same node or on different nodes. Messages are essentially a way to copy memory from one program to another. Programs are “placed” on specific idle nodes by a scheduling program (such as PBS, Sun Grid Engine, or Torque).

The programs are managed by the all-knowing scheduler and once a program is placed on a node, it remains there until complete. Communication takes place over an interconnect, but as far as the scheduler is concerned, what you do with your nodes is your business. This model has worked well for HPC clustering, although it is not the only scheme available.

The other important issue is MPI or parallel applications must be explicitly written. Communication between processes is what makes it “parallel”. The goal is to make your program run faster.

R.I.P. OpenMosix

Recently, the OpenMosix project (http://openmosix.sourceforge.net/) announced it was shutting down. For those that don’t recall or have never heard of OpenMosix, it was an open implementation of the Mosix software originally developed by Amnon Barak. Mosix team member, Moshe Bar started and lead the OpenMosix effort. And what is Mosix? Very simply, it’s a method to migrate running processes to other computers.

For instance, if you had two computers running Mosix (or more properly, a Mosix-modified kernel), and one computer noticed that its load was very high, it could transparently migrate a process to the less-loaded node.

Now think of Mosix running on a cluster of nodes. A user logs into the head node, starts jobs, and as the load increases, the jobs migrate off the head node to less loaded nodes in the cluster. To the user, the job still looks as if it’s running on the head node, as it appears in the process table and can be manipulated as such.

The unique feature of Mosix is the ability to migrate jobs transparently. Mosix contained heuristics (rules) that determine when and where to move a process. A process could be moved several times during it’s execution as means to equalize the workload among all nodes. In short, Mosix and OpenMosix are able to make a collection of nodes look like a big fat SMP machine, or as it is often called, a Single System Image (SSI). (There are other SSI efforts as well. If you are interested, have a look at OpenSSI, found online at http://wiki.openssi.org; Kerrighed, hosted at http://www.kerrighed.org; and Scyld, found at http://www.penguincomputing.com. Each does user-directed process migration as well.)

There are some issues with Mosix migration, however. Things like I/O and threads make migration difficult or in some cases impossible.

  • Jobs requiring I/O are often returned to the head node to access I/O directly. Recent versions of Mosix are considering global file systems to help solve this problem.

  • Jobs that are threaded cannot be migrated because managing threads across multiple nodes requires fine-grained synchronization (for instance, shared memory access) that would lead to large inefficiencies.

It is also possible for some MPI versions (and PVM) to work under Mosix, provided they use the kernel for communication. Migrating user space communication is not so clear as migration is kernel-directed. If you are bypassing the kernel – well, you’re kind of asking for trouble when Mr. Migrater comes to visit.

The point about Mosix is that process migration is done dynamically with no user control. There is no need for a global scheduler, as Mosix takes care of load balancing. Compared to a traditional cluster where processes are placed in queues and eventually nailed to specific nodes, Mosix kind of dumps all the processes in a bucket and sorts things out at run-time.

While OpenMosix is used on many clusters, OpenMosix doesn’t make your applications run in parallel. That task is up to the programmer. According to project leader Moshe Barr, the advent of multi-core and virtualization has reduced the need for OpenMosix. Where once OpenMosix could unify sixteen nodes into a low-cost SMP resource, multi-core has been doing this for the past year. Today, servers with eight cores are not uncommon or that expensive. In the near future, sixteen cores will be the norm.

Virtualization Joy

So what does all this have to do with virtualization? Nothing. Well almost nothing. Virtualization is about moving operating system instances around. Cluster parallel computing is about placing or moving processes on or to nodes.

Yet interesting things happen when you mix the two concepts.

Virtualization allows the operating system to run on top of a “Super OS” or a hypervisor. This standardization or abstraction of the hardware presents some interesting possibilities. It allows multiple operating system instances to run at the same time on the same processor.

Remember all those cores you have? Using virtualization on such a machine makes sense if you want to run different distributions at the same time. Maybe you want to run Red Hat and Novell/Suse on the same machine. Or maybe you want to create a sandbox machine that runs a new or different kernel version. Or maybe you are selling co-locations space and you want to charge each customer for an individual machine. Of course, you, the clever one, just bought an eight-core server and have eight virtual machines running on it. If a customer wants to reboot their virtual machine, no problem. You just saved a bundle on power and hardware costs.

Because the hypervisor insulates the operating system instance from the hardware, it allows the instance to be migrated from one hypervisor to another. Think about this for a minute. Need to take down a server to fix/add a hard drive? No problem. Start a new virtualization server, move the operating system instance (s) to the new server, fix the box, then move the instance (s) back. The operating system instance has no idea that this is happening. It is almost like the hypervisor is a light bulb socket. An operating system instance, like a light bulb, can vary in different ways, wattage or color, but all lightbulbs fit into the same socket. So if you need to move a light bulb, you can do it easily.

In the virtualization world, there are two ways to move a light bulb. The first way is to turn it off, unscrew it, maybe even wait a bit, and then put it in a new socket. The other way is to move it while it’s on. (Stick with me – this is a thought experiment.) If you unscrew the bulb really fast, and put it in the new socket before the electrons stop moving (I did say “really fast”), no one notices the light going out.

What I’ve described is the two forms of migration that are available with virtualization. The first is kind of a “halt and move” while the second is live migration. There are practical applications for both. Halting an operating system instance, let’s call it check-pointing, allows the current image to be suspended and preserved until it is restarted. Live migration allows real-time movement of the operating system image. I don’t have space to go into all the interesting ways this could be used, so I invite you to let your mind wander a bit.

Done wondering? Just in case, here comes some cold water. For virtualization to work, it must add an abstraction layer over the hardware. If you are a true cluster geek, an alarm should have just gone off in you head. Layers add overhead. HPC requires minimal overhead. There is a cost for virtualization goodness. In addition, migrating a single operating system instance is a neat trick. Migrating operating system instances that are synchronously or asynchronously communicating adds another layer of difficulty.

Virtual HPC

Even with the overhead issue, there is still something alluring about virtualization in HPC. If you think about it, what is running on an cluster compute node? Well, let’s see: You have your MPI process (es), the operating system, and hopefully nothing else. If you provision your cluster in an efficient manner, the operating system instance should be pretty minimal and maybe even lives in a RAM disk. Not that much to migrate.

The Mosix and OpenMosix approach heavily modified the kernel to allow process migration. With virtualization, the kernel still needs some modification, but it to is scooped up in the migration. This approach could be valuable to HPC in a number of ways:

  • Check-pointing node instances is one possibility. Just dump each operating system instance to disk.

  • Similarly, whole cluster hogging jobs could be swapped out of the cluster and run at a later time or on another cluster.

  • Another possibility is running unique node instances. Suppose one of your codes requires a specific kernel, libc, or distro version not used on the other nodes. No problem, start the specific operating system instance you need on the nodes you need.

  • Schedulers could be crafted to work with operating system instances, migrating an instance to help load balance the cluster in real time.

It all starts to sound a little crazy at some point. The thought of moving a large N- way MPI code from one cluster to a hard drive (or at some point a USB stick), then on to another cluster definitely makes my head spin.

As you will keenly note, I intentionally glossed over many details and looked at the big picture. Details, I’ve learned, are important in the end and make a nice home for the devil.

Virtualization is overhead expensive and still maturing. What we do know is that processors and networks will get faster, cores will multiply, and memory sizes will grow. Approaches that are overhead-intensive and- expensive today may not be as costly tomorrow. Virtualization may act a kind of glue that may bring Mosix-like migration to MPI programs.

But then, my head still spins when I wonder how it will all work out in the end.

Comments on "Adding Virtualization To My Anxiety Closet"


I’m thinking about that for a while. It’s a interesting research topic and perhaps a good starting point for a PhD thesis.

I’m happy that I’m not alone in this madness, and that the head of more experienced people spin like mine when thinking about all this mess.


Great Stuff. Good write up for a general audience.


.smallpic { display: block; float: left; }

Click image to enlarge

A line Strapless Chapel Train Wedding Dress (yz0423)

Starting at: $309.00

Crystals & Beads:

Please Select
Economy Plan
Deluxe Plan ( +$30.00 )

What’s this?


Please Select
Regular Delivery ?C 25-28 Days
Rush Delivery ?C 15-20 Days ( +$30.00 )


As Shown
Dark Green
Dark Navy
Hunter Green
Light Sky Blue
Light Slate Gray
Royal Blue

<a style="text-decor Other About Click image to enlarge blog About dearweddingdress.com blog


Are you feeling launching a dollar save? If you are you have to remember that retailing on shop is needed to take care of and in some cases grow whole profits. Absolutely sure it will take hard work, and you are going to be paid seeing that business grow.

Essentially the most essential places cheap wholesale nfl jerseys just for long-term routine maintenance is normally end top showcases. While adequately was able conclusion truck caps are the utmost merchandising facets of your keep. The reality is through correct supervision stop truck caps is odds of delight just as potential customers acquire brand-new and different items and be able to choose those items. When initial any money hold it is usually which often excitement that will actually also create patrons here we are at ones hold over and over

There are many methods that need to be transported to correctly manage your end covering demonstrations when ever cracking open $ 1 retailer. Begin by turning when it comes to latest products which are based on which aisle in which the close limitation is found. As an example, shows involved with family members purifiers might positioned a finish level when they get home on the residential clearer section. An alternative buy cheap nike free run variation could be to cycle within fresh stuff right onto close shelves that go along well with this section. As an example sponges and also scrubbers might added to which equal finish cap in the household tidier section.

You will find various approaches to include things like in the last part cover control process whenever best a buck shop. Do not forget to benefit from signage for you final a terrific way to. This tends to catch the attention of awareness of those things that are simply being available. Thoroughly clean, dazzling, adequately reproduced clues in addition brighten up your main store.

Ensure that you add a particular sometimes. That would insert appeal together with exhilaration any time clients find out the further cost savings. Get goods for the stop hats when ever you’ll find distinctive charges offered. As soon as starting any money retailer you will additionally discover that you can acquire products designed for finish cap monitors in trade events that you enroll.

If you are cutting open about $ 1 retail store won’t price cut benefit that twisting different collections at a stretch a terrific way to is capable of having for the achieving success. Doing it gives your own keep a new and different overall look while individuals revisit. cheap wholesale nike shoes Them comes with certain enjoyment. It generates unique sales.

It is advisable for sports enthusiast for getting some things that will characterize the amount of they go mad in excess of physical activities and also matches. A lot of go for jerseys; individuals require sack, groups, baseball bats, and etc; and several choose paper prints or perhaps action figures of these favored runners

Even so, in the case of an excellent college what person uses greater than a 1 fourth about his own beginner lifestyle fantasizing, paying attention to authentic nfl jersey in addition to involved in a number of sporting activities, a hot choices an excellent sports activities relax and watch. In essence, looks essential for almost every college chunk to create a advanced schooling activities enjoy that may show the level of they is in love with as a world regarding sporting activities.

Best places if you buy one

There are plenty of activities classic suppliers and additionally surprise boutiques that offer institution competitive sports designer watches notably if you get lucky and inhabit this town. Yet, however, if people that can be found nside your nearest usually do not match your sample together with guidelines, you can decide check the internet for lots more wide-ranging decisions.

Glance at the total price

The typical range of university sports monitors will be with the collection of 35 greenbacks with a hundred dollars. Basically, the cost s dependent upon materials employed for this wrist watch, and also pattern. wholesale jerseys nfl If you’re planning to find over the internet, you’ve got to incorporate slightly for your afford the transporting charge as well as florida sales tax. Virtually all timepieces may be mailed within a couple of weeks. If you prefer a powerful instantaneous and even race distribution, that might could cost more. Yet, you can also get various online businesses which offer free freight expense.

Decide on your own layout

There are lots of university or college sporting monitors generated for people young and old. The majority college wristwatches offers the college or university or or even custom logo design in the face within the activities sit back and watch. Common offers include a date together with stable website synthetic leather and / or material companies. One common option for university or college sports timepieces will be stainless sport having revolving frame and quartz analog initiative.

Remember that a lot of university or college activities different watches are actually pre-designed, there’s also quite a few enterprises that serve competitive sports check out creating to order. On the other hand nfl new jerseys 2011 a majority of this kind of expertise can be found from a higher price.


The North Face Windstopper Soft Shell Jacket Men Beige [13003] – $160.00 : Northface Jackets,North Face, North Face Outlet,North Face Store,North Face Clothing,North Face Jacket,Northface sale, Northf


Login : Zen Cart!, The Art of E-commerce


New Products : Zen Cart!, The Art of E-commerce


The North Face Apex Bionic Jacket – Womens white TNF00035 – $98.00 : Zen Cart!, The Art of E-commerce


The North Face Triclimate Jacket Men Black [12951] – $125.95 : Northface Jackets,North Face, North Face Outlet,North Face Store,North Face Clothing,North Face Jacket,Northface sale, Northface Jackets,

hello!,I like your writing so much! share we communicate more about your article on AOL? I require a specialist on this area to solve my problem. Maybe that’s you! Looking forward to see you.

Adding Virtualization To My Anxiety Closet | Linux Magazine. What’s up Jackson, if you are a new web user then you must visit every day this web page and read the updated articles at here.

Uj09Lq uyfvtcdseeup, [url=http://kilyjzzxiucg.com/]kilyjzzxiucg[/url], [link=http://xjmwwkysapat.com/]xjmwwkysapat[/link], http://pwobnsdnewcu.com/

Leave a Reply