Cluster Tools for the Enterprise

Enterprise-level clusters? It's time we start thinking about them.

The Cry from the Wilderness

For whatever reason lately I’ve become sensitive to getting clusters to be considered “Enterprise-ready.” I’ve been talking to various people who either have clusters that are managed outside the mainstream Enterprise or who are new to clusters and want to find a way to get the clusters into the IT shop (so they don’t have to manage them). But a common theme from both sides is that Cluster Tools are missing what be termed “Enterprise Class” tools.

So how do we make existing Cluster tools into Enterprise-Class Cluster Tools? One approach is to add the pieces that are missing. I think there are some pieces that can easily be added to existing tools or just better installation and management tools to make them more Enterprise ready. However, I think there are some tools that need to be developed. Regardless let’s look at some things that are needed to move Cluster Tools into the Enterprise realm.

Bare Metal Backup

One of the key components that the Enterprise IT groups require in servers is a bare metal backup. This can be thought of a Disaster Recovery (DR) requirement. The idea is that if the server goes down and either a new OS drive is required or a complete new server is required, you can quickly restore it to the originally configured state without having to work through the installation, configuration, etc. that you originally had to do. It’s fairly easy to see the usefulness of a bare metal backup.

A bare metal backup is a bit different than an image. With an image you will need the same drive(s) as you had before. With a backup you don’t necessarily have to have the same drives as before. They have to be at least the same size, but not necessarily the same type or the same capacity. However, to make life easier it’s probably a good idea to have the drive(s) be the same type as before.

A bare metal backup for a cluster only applies to the critical components of a cluster. This includes the head node, any login nodes, IO nodes, and possibly storage nodes (for example, IO nodes for a parallel file system such as Lustre, GPFS, or GFS). Basically you create bare metal backups for anything but the compute nodes. Typically people think of the compute nodes as disposable. That is, if a compute node fails you just replace it and restart the job that was using the node. This also means that you don’t place anything critical on the node. For example, if you put the OS or a subset of the OS in a ramdisk in the compute node then if there is a hard drive in the node, it can be used for local storage or for swap space.

Using bare metal backups in practice is fairly easy, but as will all backups, requires some discipline. When a cluster is first configured and tested, but perhaps before it goes into full production, the first bare metal backup should be made of the head node, the login nodes, and possibly storage nodes (not the storage itself but the OS on the storage nodes). Then if any configuration changes are made to any of these nodes, a new bare metal backup is made of the entire set of nodes. These backups should be get a safe place and even better, extra copies are made and kept off-site (a simple safety deposit box can be used for this).

To make the process of creating bare metal backups easier it should be integrated with the cluster management tool. Then when a configuration change is made a bare metal backup can be created. This is also true for any firmware or BIOS upgrades. However, you will need to keep track of the current firmware versions so any replacement hardware is configured with identical firmware versions. In addition, if the image for the nodes is changed, the cluster management tool can prompt the administrator to make the backup.

One of my favorite tools for creating a bare metal backup tool called Mondorescue (www.mondorescue.org). It’s a GPL licensed bare metal backup tool that is also scriptable. It has a variety of ways to store the backups including NFS and is very easy to use.

Reporting Tool

Another enterprise class tool that is absent from virtually every cluster that I know of is a reporting tool. A reporting tool takes information about the system and creates a summary report. This report is very useful to get an idea at a reasonable high level but with some detail, as to how the system was used. The report can be a file or a delivered via the web or embedded in applications. it can include tables (including a spreadsheet), or graphs.

These reports can be used for visualization and reporting on the cluster. For example, it can contain information such as the number of jobs run in a certain time period, total uptime of the nodes, number of nodes down, the total down time for the cluster (sum of the down time of the nodes that were down), file space usage, log in times, administration time, any security information, such as failed login attempts, on the head node, any network traffic information you have gathered, and just about anything else you can measure or are interested in seeing a report.

I’m sure there are some people who are reading this column and asking if I’m on crack. Well the obvious answer is no and it’s not the coffee either. Let me explain why I think reports are a very useful and really a required feature for enterprise class clusters.

I think clusters are cool in and off themselves, but they are very useful for solving large scale problems. Many times, people get a cluster up and running and declare victory and start using it. However, as they will find out, that getting it up and running is only half the problem. The other half is keeping the system running. Simple day to day activities on the cluster such as watching the queues, killing hung jobs, starting jobs that need to be started, juggling project priorities, adding users, removing users, recovering or restoring data, debugging nodes that are down, bringing up repaired nodes, restarting hung nodes, and fixing user problems (“why won’t my script work?”) can take a great deal of time, particularly when you have a number of users with varied experience running on HPC class systems. Good administrators will fix these problems but the better administrators will try to determine why they happened in the first place and how to make sure they don’t happen again. The best administrators won’t necessarily through up barriers to prevent users from doing things but will give users better tools and better education to help them solve their problems.

If an administrator had a way to summarize activity on the cluster and correlate that with problems, then it’s much easier to see cause and effect and take corrective action. I know this sounds awfully “enterprise” like and some people are cringing and wanting to strangle me, but the fact of the matter is that having this information can make it easier to understand what is happening in the cluster and fix the problem. The enterprise world uses the phrase “business intelligence.” While I’m not hot on that phrase (it does have an oxymoron ring to it), it does hint at what you’re trying to do — more efficient use of the resources and making your cluster run more smoothly.

Comments on "Cluster Tools for the Enterprise"

How can I obtain the computer in a single room to deliver stuff towards the printer on my pc in one more room. Internet?

In these busy times it is sometimes hard to squeeze in quality time with our children. A stand up blender,
similar to the kind that is used at malt shops, is easy to learn how
to use, and can be useful for all sorts of different drinks.
If you do find yourself in over your head a bit when it comes time to cook and plan ingredients, head online for some virtual cooking games.

Spend a day with Ayurvedic teacher Richard La – Marita to learn about the oldest philosophy of health care based on prevention in the entire
world. These products are great for when you are
cooking for large family get-togethers, and can make
cooking easier by saving baking time. Of course, unsanitary conditions
can lead to food-borne illnesses as well.

My programmer is trying to persuade me to move to .net from PHP.
I have always disliked the idea because of the costs. But he’s tryiong
none the less. I’ve been using Movable-type on numerous websites
for about a year and am anxious about switching to another
platform. I have heard good things about blogengine.net.

Is there a way I can import all my wordpress content into it?

Any kind of help would be greatly appreciated!

Together with every little thing that appears to be building inside this subject material, a significant percentage of opinions are generally very exciting. Having said that, I am sorry, but I can not subscribe to your entire theory, all be it exhilarating none the less. It appears to me that your opinions are not completely rationalized and in reality you are your self not completely convinced of your point. In any event I did enjoy reading it.

A large percentage of of the things you point out is supprisingly precise and it makes me ponder why I had not looked at this with this light before. This particular piece truly did turn the light on for me personally as far as this particular subject matter goes. Nonetheless at this time there is 1 factor I am not necessarily too comfortable with and whilst I make an effort to reconcile that with the core idea of the position, permit me see what the rest of your visitors have to say.Nicely done.

Almost all of the feed-back on this web site dont crank out come to feel.

I do enjoy the way you have framed this particular concern plus it does give me personally a lot of fodder for thought. Nevertheless, through just what I have witnessed, I only trust as the actual responses pile on that individuals remain on issue and not embark upon a tirade regarding the news du jour. Still, thank you for this excellent point and though I do not necessarily agree with the idea in totality, I respect your viewpoint.

Along with almost everything that appears to be building within this specific subject matter, a significant percentage of perspectives are actually rather stimulating. Nonetheless, I appologize, because I do not give credence to your entire idea, all be it exciting none the less. It appears to me that your commentary are not totally validated and in actuality you are yourself not even entirely convinced of your assertion. In any case I did take pleasure in reading through it.

Here are a few of the web-sites we advise for our visitors.

Kia es un fabricante surcoreano de automóviles con la oficina central en Yangjae-dong, Seocho-gu, Seúl, Corea del Sur, que inició su actividad en 1944, como fabricante de bicicletas y tuberías de acero. OSLO / 2012-10-18 / Razón tuvo Álvaro Leiva cuando dijo que el gavirista Humberto de la Calle, fue enviado en 1990 por el Presidente Cesar Gaviria al proceso de paz iniciado en Caracas, con el fin único de destruirlo para que la Constituyente del 91 que dio inicio a la apertura neoliberal de bienvenidos al futuro”, se pudiera imponer mediante el presupuesto militar de la guerra contrainsurgente.

One of our visitors lately advised the following website.

Astounding write-up ! Cheers for, article on this web site page mate! Ill message your self all over again. I didnt realise that.

Here is a superb Blog You might Discover Intriguing that we encourage you to visit.

Along with everything that seems to be developing throughout this specific subject matter, all your perspectives are rather stimulating. Even so, I beg your pardon, because I do not give credence to your entire idea, all be it exciting none the less. It would seem to everybody that your remarks are generally not completely justified and in actuality you are generally yourself not really fully confident of your assertion. In any case I did appreciate looking at it.

hello!,I love your writing so much! proportion we communicate extra approximately your post on AOL? I need an expert on this space to solve my problem. May be that’s you! Looking forward to see you.

With the whole thing that seems to be building throughout this specific subject material, a significant percentage of points of view happen to be very exciting. Nevertheless, I beg your pardon, but I do not subscribe to your entire plan, all be it exciting none the less. It would seem to me that your comments are generally not totally justified and in fact you are generally yourself not even entirely certain of your assertion. In any case I did take pleasure in examining it.

The information talked about within the post are some of the best available.

Just about all of whatever you claim is supprisingly appropriate and it makes me ponder the reason why I hadn’t looked at this in this light before. This particular piece really did turn the light on for me personally as far as this specific topic goes. Nonetheless at this time there is one particular position I am not too comfy with and while I attempt to reconcile that with the actual core theme of your position, let me see just what all the rest of your subscribers have to say.Very well done.

Very neat post.Thanks Again. Cool.

I do love the manner in which you have presented this particular problem and it does indeed give me a lot of fodder for thought. Nevertheless, because of what I have witnessed, I simply just wish when the feed-back pile on that men and women keep on point and don’t get started on a soap box of the news of the day. Yet, thank you for this excellent piece and while I can not concur with it in totality, I respect the standpoint.

Nearly all of what you claim happens to be supprisingly precise and it makes me ponder the reason why I had not looked at this with this light previously. This article really did switch the light on for me personally as far as this specific issue goes. But at this time there is actually one particular factor I am not too comfortable with and while I try to reconcile that with the central theme of your position, permit me see what the rest of your subscribers have to point out.Very well done.

Very good article and a great summation of the problem.. Make sure you hold us up in the direction of date such as this. A great deal further more Extraordinary A lot far more.

Very good post.Much thanks again. Great.

Thank you, I have just been searching for information approximately this subject for a while and yours is the best I’ve discovered till now. However, what about the conclusion? Are you certain concerning the source?

Its like you read my mind! You seem to know a lot about this, like you wrote the book in it or something. I think that you could do with some pics to drive the message home a bit, but instead of that, this is great blog. A great read. I’ll definitely be back.

I was trying to get this exclusive information for a Pretty very long time. Thank by yourself and excellent of luck.. Cheers! Added A shapely Agreeable.

Just beneath, are quite a few absolutely not related websites to ours, on the other hand, they may be surely worth going over.

Hello there, just was alert to your blog thru Google, and found that it’s really informative. I am gonna watch out for brussels. I’ll be grateful if you proceed this in future. A lot of other people shall be benefited out of your writing. Cheers!

Wow, yourself seem to be exceptionally knowledgable concerning this variety of issue.. Properly this is seriously attention-grabbing certainly. A shapely Supplemental A rise within.

Hello would you mind sharing which blog platform you’re working with?

I’m planning to start my own blog in the near future but I’m having a difficult time deciding between BlogEngine/Wordpress/B2evolution and
Drupal. The reason I ask is because your design and style seems different then most blogs and I’m looking for something unique.
P.S My apologies for getting off-topic but I had to ask!

My coder is trying to convince me to move to .net from PHP. I have always disliked the idea because of the expenses. But he’s tryiong none the less. I’ve been using Movable-type on a variety of websites for about a year and am nervous about switching to another platform. I have heard fantastic things about blogengine.net. Is there a way I can transfer all my wordpress posts into it? Any kind of help would be really appreciated!

Here is a great Weblog You may Uncover Intriguing that we encourage you to visit.

Wonderful story, reckoned we could combine a few unrelated data, nevertheless definitely worth taking a appear, whoa did one study about Mid East has got a lot more problerms also.

Simply just learned this internet site via Yahoo, what a pleasant shock!

Wonderful goods from you, man. I have understand your stuff previous to and you’re just extremely wonderful. I really like what you have acquired here, really like what you’re saying and the way in which you say it. You make it entertaining and you still take care of to keep it sensible. I cant wait to read much more from you. This is actually a great website.

Im taking a browser mistake, is absolutely everyone else?

gerçekten ba?ar?l? olmu? ?ahane

This is a great website. Bookmarked!

Hi, can I employ the service of your report upon my blog with a linkback?

I have mastered some points through your blog post. One other thing I would like to say is that there are plenty of games available on the market designed mainly for preschool age young children. They include things like pattern acceptance, colors, animals, and shapes. These commonly focus on familiarization as an alternative to memorization. This makes a child occupied without having the experience like they are learning. Thanks

Another thing I have noticed is the fact that for many people, poor credit is the reaction of circumstances beyond their control. Such as they may have already been saddled with an illness and because of this they have substantial bills for collections. Maybe it’s due to a employment loss or perhaps the inability to go to work. Sometimes divorce proceedings can really send the financial situation in an opposite direction. Many thanks sharing your notions on this site.

Please pay a visit to the web pages we follow, such as this a single, as it represents our picks from the web.

I have recently started a site, the info you provide on this site has helped me tremendously. Thanks for all of your time & work.

After all, what a great site and informative posts, I will upload inbound link – bookmark this web site? Regards, Reader.

An insightful weblog write-up right there close friend ! Cheers for it !

Heya i am for the first time here. I came across this board and I find It really
useful & it helped me out a lot. I hope to give something back and
help others like you aided me.

Here is my homepage :: slot777.trade

You could certainly see your expertise within the work you write. The arena hopes for even more passionate writers such as you who are not afraid to say how they believe. At all times go after your heart. “In America, through pressure of conformity, there is freedom of choice, but nothing to choose from.” by Peter Ustinov.

Lots of because of for your review! I exceptionally relished it.. Hi,I really including examining in the course of your website, I desired toward depart a small comment in the direction of aid yourself and wish on your own a Great continuation. A large amount much more A shapely A rise within.

Leave a Reply