From Russia with Love: POHMELFS – A New Distributed Storage Solution

There is a new file distributed file system in the staging area of the 2.6.30 kernel called POHMELFS. Sporting better performance than classic NFS, it's definitely worth a look.

Distributed storage solutions are almost ubiquitous today. They are used in HPC systems, corporate desktops, corporate laptops, and even typical laptop and home users are starting to use servers to provide centralized storage for their homes and families. The most common file system used in these situations is NFS. It has been in use for many years, comes with virtually every OS, is well understood, and it just works. In addition, it’s the only standard file system. This allows you to use a single central server for many different Operation Systems (OS).

However, NFS is not without it’s limitations. Evgeniy Polyakov, a long time Linux hacker, has recently contributed a new distributed file system, called POHMELFS (Parallel Optimized Host Message Exchange Layered File System). It has appeared in the “chock-full-of-filesystems” kernel version 2.6.30 in the staging area. It is ready for testing and can give you a boost in performance (remember – it’s parallel!). This article will discuss POHMELFS and where it is headed.

An Oldie but a Goody – NFS

NFS has been the dominant file system protocol for distributed storage needs because it is “there” and is pretty much “plug and play” on most *nix systems. It was the first widespread file system that allowed distributed systems to share data effectively. In fact, it’s the only standard network file system.

While NFS is likely to be the most ubiquitous distributed file system, it has gotten a little long in the tooth, so to say, and has some limitations. For example, it doesn’t scale well for large number of clients and has limited performance. It also used to have some security issues, but these were addressed in Version 4 of the NFS protocol. Despite these limitations, it remains the most popular distributed file system because:

  • It comes with virtually every OS (it can even be an add-on to Windows)
  • Easy to configure and manage
  • Well understood
  • Works across multiple platforms
  • Usually requires little administration (until it goes down)
  • It just works

NFS is a fairly easy protocol to follow. All information, data and metadata, flows through the file server. This is commonly referred to as an “in-band” data flow model shown in Figure One below.

Figure 1 - In-Band Data Flow Model (Courtesy of Panasas)
Figure 1 – In-Band Data Flow Model (Courtesy of Panasas)

Notice that the file server touches and manages all data and metadata. This model does make things a bit easier to configure and monitor. Moreover, it has narrow, well defined failure modes. Some drawbacks include an obvious bottleneck for performance, has problems with load balancing, and security is a function of the server node and not the protocol (this situation means that security features can be all over the map).

With NFS, at least one server “exports” some storage space to the nodes of the cluster. These nodes mount the exported file system(s). When a file request is made to one of these mounted file systems, the mount daemon transfers the request to the NFS server, which then accesses the file on the local file system. The data is the transferred from the NFS server to the requesting node, typically using TCP, but can be UDP. Notice that NFS is “file” based. That is, when a data request is made, it is done on a file, not blocks of data or a byte range. So we say that NFS is a file based protocol.

For typical NFS systems (not clustered NFS), all metadata and data operations go through a single server. As you increase the number of clients that are addressing the server, the more load the server must carry. Consequently, NFS can have limited performance (it depends upon the number of clients accessing the storage, the workload, the performance capabilities of the server, and the network).

Introduction to POHMELFS

POHMELFS is a new file system that is designed to take a step beyond classic NFS focusing on improved performance. It’s name even contains the term “Parallel” that indicates that the clients interact with multiple servers. In particular, it has the ability to balance reading from multiple servers and also do simultaneous writes to different remote servers. It is designed in the classic server-client model. As with NFS, POHMELFS exports a directory from each server so it relies on an underlying file system to read and write data to the physical devices themselves. In addition, it is designed as an object based file system (more on that below).

The POHMELFS website has a very comprehensive list of features. For the sake of completeness these features are summarized here:

  • One of the most important attributes is the ability to write to multiple servers and balance reading between multiple servers.

  • POHMELFS has a local coherent cache for data and metadata (this basically adds some of the features of FS-Cache and CacheFS to the network file system.)

  • It includes locking (a necessary feature for a shared file system). It was originally designed for byte-range locking but according to the website, all Linux file systems lock the whole inode. So the developers decided to lock the whole object during writing. But POHMELFS has the ability to allow different clients to simultaneously write into the same page via different offsets with the result that the file will be coherent on all clients and all servers (not a small feat).

  • All events are completely asynchronous with the only exceptions being hard links and symlinks. These events include the creation of the objects as well as data reading and writing.

  • POHMELFS is designed to have a flexible object architecture that is optimized for network processing. Network processing is a potential weak point for distributed file systems since file systems can be “chatty” and create many small messages that are not always optimal for networks. The design of the object architecture allows for very long paths to the objects and the ability remove arbitrary size directories with a single network command.

  • The server portion is multi-threaded and scalable and, perhaps more importantly, is in user space. There is only a driver for POHMELFS in the kernel. The client and the server are all in user-space and interact with the driver. Assuming that the driver does not change very much, then as POHMELFS evolves only the user-space tools evolve. Consequently, new evolutions don’t require new kernels. This also means that development can progress at a very fast rate.

  • POHMELFS utilizes a transaction model for all its operations. Each transaction is an object which may embed multiple commands that are to be completed atomically. This design also means that it will resend transactions to different servers if there is a timeout or an error on the initially contacted server. This design maintains high data integrity and does not desynchronize the file system state in the event of a server failure or a network failure. An end result of this design is that if a server goes down the clients can switch to a different one automatically.

  • It has the ability for the clients to dynamically add or remove servers from a working set.

  • POHMELFS is also designed for strong authentication with the possibility of data encryption in the network channel.

  • It has extended attribute support

  • It can do read-only mounts and also has the ability to limit maximum size of the exported directory.

Comments on "From Russia with Love: POHMELFS – A New Distributed Storage Solution"

You’ve made some decent points there. I looked on the web
to find out more about the issue and found most people will go
along with your views on this website.

It’s actually very complicated in this busy life to listen news on TV,
therefore I simply use world wide web for that reason, and take the latest news.

WOW just what I was searching for. Came here by searching
for puma golf hats

Thank you for the auspicious writeup. It in reality was a amusement account it.
Glance complex to more added agreeable from you! However,
how can we keep up a correspondence?

Also visit my site Cheap car insurance

Touche. Outstanding arguments. Keep up the amazing effort.

Feel free to surf to my web blog :: cheap car insurance in pa

It’s great that you are getting thoughts from this piece of
writing as well as from our dialogue made at this time.

My blog post; cheap car insurance in ct

Hey! This is my first comment here so I just wanted to give
a quick shout out and tell you I really enjoy reading through your posts.

Can you suggest any other blogs/websites/forums that deal with the same topics?

My site: ?????

I have read so many content about the blogger lovers but this post is actually a good article, keep it up.

My web site: cheap car insurance

Florida is one of sеvеn ѕtates iin tҺᥱ United Ѕtates օff America tҺаt prohibbit tһе
օpen carrying оf handguns. Ꭺге үοu currently starting a buiness іn tһе Philippines.
A wide variety ⲟf Νew York office space сɑn Ƅᥱ rented tο suit thee neеds off large
companies аnd emerging start սⲣ businesses.

Μy рage – office commercial space for rent

Ridiculous story there. What occurred after? Good luck!

Appreciate the recommendation. Will try it out.

Take a look at my web-site: Cheap car insurance

Wow! Finally I got a webpage from where I know how to really get valuable data regarding my study and knowledge.

Feel free to visit my web page – cheap car insurance quotes

If you would loke to get a great deal from thios paragraph then you have to apply such techniques to your won weblog.

Feel free to surf to my web-site … very cheap car insurance


The other day, while I was at work, my sister stole my apple ipad and
tested to see if it can survive a 30 foot drop, just so she can be a youtube sensation. My iPad is now
broken and she has 83 views. I know this is entirely off topic but I had to share it with someone!

Great work! That is the kind of information that should be shared around the internet.
Shaame on the search engines for no longer positioning this publish upper!
Come on over and consult wiyh my site . Thanks =)

Feel free to surf to my weblog – Cheap Car Insurance

Yes! Finally something about cheap car insurance for

I have read so many posts regarding the blogger lovers but this article is truly a
good post, keep it up.

Stop by my page – satellite internet gerton north carolina

This is the perfect blog for everyone who wishes
to find out about this topic. You realize a whole lot its almost tough to argue with you
(not that I personally will need to…HaHa). You certainly
put a new spin on a topic which has been written about for many
years. Excellent stuff, just excellent!

I’m extremely impressed with your writing skills as well as with
the layout on your weblog. Is this a paid theme orr did you customize it yourself?

Anyway keep up the nice quality writing, it’s rare to see a great
blog like this one nowadays.

my web-site: Cheap Car Insurance

This piece of writing will help the internet visitors for setting up new website or even a blog from start to end.

Feel free to surf to my blog post … Cheap car insurance

Hey There. I discovered your blog using msn. That is an extremely neatly written article.
I will make sure to bookmark it and come back to learn extra of your helpful info.
Thanks for the post. I will certainly return.

Great article! We wiol bee linkimg to this particularly great
content on our site. Keep up the goiod writing.

Here is my web blkog :: cheap liability car insurance

I am sure this piece of writing has touched all the internet users,
its really really pleasant paragraph on building up new website.

my web site: cheap car insurance

Its like you read my mind! You appear to know so much about
this, like you wrote the book in it or something. I think that you can do with a few pics to
drive the message home a bit, but other than that, this is wonderful blog.
An excellent read. I will certainly be back.

Also visit my web-site … car insurance companies in michigan

Hmm it appears like your site ate my first comment (it was extremely
long) so I guess I’ll just sum it up what I submitted
and say, I’m thoroughly enjoying your blog. I too am
an aspiring blog writer but I’m still new to the whole thing.
Do you have any helpful hints for beginner blog writers? I’d certainly appreciate it.

Hi there friends, its great post on the topic of cultureand
completely explained, kep iit up all the time.

Feel free tto surf to my site; Cheap car insurance

Valuable information. Fortunate me I discovered your site accidentally, and I am stunned why this twist
of fate did not came about earlier! I bookmarked it.

Feel free to surf to my website :: Joie

Greetings! Very helpful advice within this post! It is the little changes that
produce the most important changes. Many thanks for sharing!

Also visit my blog – Cheap car insurance

Hi there mates, how is the whole thing, and what you wish for to say
regarding this post, in my view its really
amazing in support of me.

Here is my blog; free home security s

After I initially commented I appear to have
clicked on the -Notify me when new comments are added- checkbox
and now each time a comment is added I get 4 emails with the exact same comment.
There has to be a means you can remove me from that service?
Thanks a lot!

I simply want to tell you that I am beginner to blogging and actually savored this website. Probably I’m want to bookmark your blog post . You surely have awesome posts. Kudos for sharing with us your website.

Heya exceptional blog! Does running a blog such as this take a massive
amount work? I’ve very little expertise in programming but I had been hoping to start my
own blog soon. Anyway, if you have any suggestions or techniques for new blog owners
please share. I understand this is off subject nevertheless I simply wanted to ask.
Appreciate it!

My brother recommended I might like this blog. He used to be entirely right.
This submit truly made my day. You cann’t consider simply
how a lot time I had spent for this information! Thanks!

Hmm is anyone else encountering problems with the pictures on this blog loading? I’m trying to figure out if its a problem on my end or if it’s the blog. Any feed-back would be greatly appreciated.

When some one searches for his required thing, so he/she wishes to be available that Living In the complex detail, thus that thing is maintained over here.

I have been exploring for a bit for any high quality articles or blog posts on this kind
of space . Exploring in Yahoo I ultimately stumbled upon this site.

Studying this information So i’m happy to show that I
have an incredibly good uncanny feeling I found out just what I needed.

I so much without a doubt will make sure to don?t fail to remember this website and
provides it a glance regularly.

0 inch flip out touch screen display allows quick spot focusing
and access to function menus as well as offering a crisp 230K pixel display and a
270 degree rotational angle for self documenting. Hailed as “one of the best new puzzle engines to come around in a decade”, this unique puzzler finds inspiration from
the popular pastime ‘Picross’ and blends it with mechanics from the
popular ‘Bust-A-Move’ series and a little dash of ‘Tetris’.
Prognosis: Good – again, the scratch may be annoying but in many cases won’t be noticeable when a
show is on.

my web site; Ourworld Gem Codes

Hi there all, here every one is sharing these know-how, so it’s
good to read this web site, and I used to go to see this web site every day.

Also visit my web site – The units

Hello! I could have sworn I’ve been to this website before but after browsing through
some of the articles I realized it’s new to me.

Nonetheless, I’m definitely delighted I came across it and
I’ll be bookmarking it and checking back regularly!

Good answer back in return of this matter with solid arguments and explaining all concerning that.

I think this is one of the most important info for me.
And i am satisfied studying your article. But should remark on few general issues, The site style is wonderful, the articles is really great
: D. Good activity, cheers

Take a look at my page; canadian pharmacy

What’s up, this weekend is nice for me, since this occasion i am reading this enormous educational article here at my residence.

Hello, after reading this amazing post i am as well glad to share my knowledge here with friends.

Also visit my site – my canadian pharmacy

We can play with the color of the blouse and adapt it to the
style and workplace of each person. com Sort your laundry neatly on the low wicker basketss, sorters, laundry bags and
draw attention away from your rooms clean. These cases were
effective,but because going to be the case required to obtain opened its doors at these point,
going to be the player was however.

Visit my web blog :: prada bags outlet

Very good write-up. I absolutely appreciate
this website. Thanks!

my web page; dyson dc58

Good day! I could have sworn I’ve visited this website before but after going
through some of the articles I realized it’s new to me.
Anyhow, I’m definitely delighted I came across it and I’ll be book-marking it and checking back regularly!

Here is my site: dyson vacuum

Leave a Reply