Harping on Metadata Performance: New Benchmarks

Metadata performance is perhaps the most neglected facet of storage performance. In previous articles we've looked into how best to improve metadata performance without too much luck. Could that be a function of the benchmark? Hmmm...

If you have been reading the articles in this column for any amount of time you will see a number of them focus on metadata performance. The primary reason for focusing on metadata performance is that it is one of the most critical aspects of overall performance and is one of the most neglected aspects. When people talk about drive or storage performance it is invariably given in terms of MB per second or something similar. In other words – throughput. However, people do not realize how important metadata performance is to overall performance.

Metadata performance refers to the how quickly files and directories can be created, removed, their status checked (stat), as well as other data functions. This aspect of storage performance is becoming more important because of the increasing number of files and directories on systems. Creating files, deleting them, and performing a status check of them is important for more applications than ever before. There are applications that can produce millions of files in a single directory and applications that create very deep and wide directory structures. As the number of cores increases the number of files and number of directories increases putting more and more pressure on the metadata performance of storage solutions.

In previous articles a simple metadata benchmark, fdtree, was used to measure benchmark performance. We saw that many times there was not a lot of change in performance when the file system metadata performance was changed. This led me to ask the question, is fdtree a good enough benchmark to show metadata performance as one tweaks a system for performance? If you like analogies, it’s like wondering if you doctor has the right diagnosis. Since I don’t always trust single opinions, I always like to get second and even third opinions. So in this article I want to examine a new metadata performance benchmark, run it on the same hardware as past articles that used fdtree, and see if they both show the same trends.


A common benchmark used for HPC storage systems is called metarates. Metarates was developed by the University Corporation for Atmospheric Research (UCAR). It is a MPI application that tests metadata performance by using POSIX system calls:

  • creat() – (open and possibly create a file)
  • stat() – (get file status)
  • unlink() – (delete a name and possibly the file it refers to)
  • fsync() – (synchronize a file’s in-core state with storage device)
  • close() – (close a file descriptor)
  • utime() – (change file last access and modification times)

Using these system calls, the main analysis options for metarates are the following:

  • Measure the rate of file creates/closes (create/close per second)
  • Measure the rate of utime calls (utimes per second)
  • Measure the rate of stat calls (stats per second)

Metarates has options for the number of files to write per MPI process (remember that you will have N processes with a MPI application where N is a minimum of 1) and if the files are to be written to a single directory or to many directories. It also has the option of using the system call fsync() to synchronize the file’s in-core state with the storage device. But fsync has been something of a controversial subject in Linux so let’s cover fsync() for a moment.

The system call fsync is really designed to flush the buffer cache data to the storage devices (e.g. disks) to make sure that the data is actually on the disk and not in a buffer where a power lose would mean losing data. According to the eminent Theodore Ts’o fsync() does the following:

"... the only safe way according that POSIX allows for requesting data written to a particular file descriptor be safely stored on stable storage is via the fsync() call. ..."

However, if you read the man pages for fsync() (“man 2 fsync”), you will see the following,

"Calling fsync() does not necessarily ensure that the entry in the directory containing the file has also reached disk. For that an explicit fsync() on a file descriptor for the directory is also needed."

So fsync appears to be the best way for an application to ensure that the data has actually reached the disk but you still have to be careful if you want to be 100% sure that the data is actually on the disk.

If you want to try a simple experiment examine the IO output of your applications and look for calls to fsync. I think you would be surprised at how many applications do not use use it.

Test System

The test system used for these experiments was a stock CentOS 5.3 distribution but with a 2.6.30 kernel and e2fsprogs was upgraded to 1.41.9. The tests were run on the following system:

  • GigaByte MAA78GM-US2H motherboard
  • An AMD Phenom II X4 920 CPU
  • 8GB of memory
  • Linux 2.6.30 kernel
  • The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  • /home is on a Seagate ST1360827AS
  • There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb for all of the tests. The second drive, /dev/sdc was used only for the tests where the journal was placed on a drive.

As with previous articles, ext4 is used as the file system. The journal can be easily moved to a different device so that the impact of the journal location on performance can be determined. The steps for creating the file system and the journal are covered in previous articles.

Metarates Command Parameters

The test system has 4 cores and Metarates is an MPI application allowing us to choose the number of processes (cores) we use during the run. So for this benchmark 1, 2, and 4 cores were used (three independent tests).

Not forgetting our good benchmarking skills, the run time (wall clock time) of the runs should be greater than 60 seconds if possible. So the number of files was varied for 4 MPI processes until a run time of 60 seconds was reached. The resulting number of files from the test was found to be 1,000,000 and was fixed for all tests. Also it was arbitrarily decided to have all files are written to the same directory with the goal of really stressing the metadata performance.

The final command line used for metarates for all three numbers of processors (1, 2, and 4) is the following.

time mpirun -machinefile ./machinefile -np 4 ./metarates -d junk -n 1000000 -C -U -S -u >> metarates_256MB_ramdisk.np_4.1.out

Where the “-np” option stands for number of processes (in this case 4), “-machinefile” refers to the list of hostnames of systems to be used in the run (in this case it is a file name “./machinefile” that contains the name of the test machine repeated 4 times – once for each process), and the results to stdout are sent to a file “metarates_256MB_ramdisk.np_4.1.out” which is an example of how the output files were named.

Notice that fsync wasn’t used in the benchmark run so that the results would be comparatively similar to those of fdtree which didn’t explicitly fsync() files. In addition, the directory where the files are written is the current directory where the binary (“./metarates”) is located. Also notice that three different performance measures are used:

  • File create and close rate (how many per second)
  • File stat rate (how many “stats” per second)
  • File utime rate (how many “utimes” per second)

Next: Running Metarates

Comments on "Harping on Metadata Performance: New Benchmarks"

Do you have any video of that? I’d love to find out some additional

Hi! I just wanted to ask if you ever have any trouble with
hackers? My last blog (wordpress) was hacked and I ended
up losing many months of hard work due to no data backup.

Do you have any solutions to stop hackers?

I love what you guys are up too. This sort of clever work and coverage!

Keep up the great works guys I’ve incorporated you guys to
my blogroll.

It’s remarkable to go to see this website and reading the views of
all mates regarding this post, while I am also eager of getting knowledge.

This is the right site for anyone who hopes to understand this topic.
You understand so much its almost hard to argue with
you (not that I actually would want to…HaHa). You certainly put a brand new spin on a subject which has been discussed for many years.
Wonderful stuff, just excellent!

Yes! Finally someone writes about kroger.

Thank you for your post. Want more.

Good article! We are linking to this particularly great content on our site.
Keep up the good writing.

Ahaa, its good dialogue concerning this article at this place at this website, I have read all that, so now me also commenting here.

Review my web-site … hermes outlet online

I’m extremely inspired along with your writing abilities and
also with the format to your blog. Is that this a paid subject or did you modify it
yourself? Either way keep up the nice high quality writing, it is rare to
look a nice blog like this one nowadays..

I loved as much as you’ll receive carried out right here.
The sketch is attractive, your authored subject matter stylish.
nonetheless, you command get got an edginess over that you wish be delivering the following.

unwell unquestionably come further formerly again as exactly the
same nearly a lot often inside case you shield this hike.

Woah! I’m really enjoying the template/theme of this website.
It’s simple, yet effective. A lot of times it’s very hard to
get that “perfect balance” between usability and visual appeal.

I must say you’ve done a very good job with this. In addition,
the blog loads super fast for me on Firefox. Exceptional Blog!

I visit each day some blogs and information sites to read posts, except this blog
presents feature based articles.

One of our visitors not too long ago advised the following website.

If you are going for finest contents like me, simply pay a visit this
website all the time as it provides feature contents,

It’s very trouble-free to find out any matter on net
as compared to books, as I found this article at this web site.

Here is my weblog … The Lost Ways Scam

Keep on writing, great job!

As the Earth revolves about herself and the Sun rises and falls, the forex markets all more than the planet, hold on opening and closing, in order to facilitate uninterrupted currency trade.

Feel free to surf to my site; valutaomvandlare forex

To begn with, there is nothing incorrect with supportive to enjoy.
There are some people who live to consume as well as there are a few
who consume to call home. Even so the overall people does not
remain in possibly of such categories. There are a few that really like excellent
meal, although their particular life be based upon the food.
Baar meal Cheshire is beloved in several parts of the country
in addition too in case youu are the meals lover, you must expertise the type of meals you obtain the following.

Go to a loved onhes club Cheshire and you can take
advantage of the ideal foods as well as refreshments together with your complete
loved ones. There’s nothing wrong along with caring to eat.
There are many individuals who live tto nibble on as well
as therde are numerous that take to reside in. Though the complete human race
does not remain inn possibly of those categories. There are several
that love great foods, nevertheless their own lives rely oon thier food.
Bar food Cheshire can be loved in many regions and also for anyone who iis any food partner, you have to expertise the kind of foodstuff yoou receive
here. Visit a loved ones bar Cheshire and you can benefit from thhe greatest food in addition to cocktails with your
overall loved ones. The worfd club is frequently connected with grown ups.
The most important pubs near your vicinity aree generally filled by means of athletics
addicts exactly who adore to observe his or her competitors battle the idea out there becaause they glass drauhght beer as well as lager and so on. Nevertheless it
isn’t essential your little ones aare not able to check out a pub.
Indeed, thgere are numerous brs that you just won’t prefer to take your kidds to.
Nevertheless a family group tavern Cheshire isn’t amid these kinds of institutions.
The particular pub foodstuff Cheshire in this particular nightclub is indeed great which anyone reaches take pleasure in in some way or even another.
Thhis spouse and children tavern Cheshire supplies the finest within British foods.
You’ll find some people saying that will United kingdom food isn’t just like the meal
that’s reacy in the some other places like France and Sain while others.
Moreover, this increase associated with American indian curries provides more improved the actual colour pallette of the British
isles foodies. Yet tthe fact is in which United kingdom
food possesses its very own good style. Consume standard English foods and you also are unable tto guide although nearly lick your current palms.
Indeed, the item issues thqt the chief cook is a plus, akthough whenever you check outt the best place
intended for nightclub food Cheshire, yoou can expect tgis cocinero to be
at the top of his or her preparig food skill.

In this family members pub Cheshire you have eevery thing you wish to take in and ingest.
Inside the products area you’ve gott optimum cask beers, United
kingdom as well as imported lager, cider and also a thorough numbner of
bottles of wine. The meal selection is additionally extensive andd also
you have the option from which to choose the actual afe food list, the actual
2-for-1 menu, the particular pub selection plus the Sunday food
list. The mouth-watering polts and pans are generally
destined for making you are feeling greater than pleased.
In the event that you are searching for obtaining this kind
of club foods Cheshire, go to tthe web page in addition to take a
look at thee food selection. All thhe information that you
need cann be presenterd in the site suhh aas the number in order to contact to make thee reservation. Upon weekdays you may go to your restaurant and find a spot, but
throughout weekends, it truly is foolhardy searching for the place with out
a reservation. Whatever the case, it seems sensible to be able to book befre
hand irrespective of when you are. Find out
more on thee internet to help trial exceptional club foods Cheshire.
At this family club Cheshire you will get dozens of foodstuffs that leave an individual satisfied.

The prices tend to be effectively within your budget, nevertheless the foods is
actually out of the planet.

Feel free to surf to my web site … foto ngentot

So with a motivation to find out and a desire to succeed, there is nothing to stop you generating huge income trading international currencies.

Also visit my weblog … hvor mye er 1 euro i norske kroner 2014

Trading currency futures at the CME are undoubtedly a lot much better suited to
the on the internet day trader who is generally working with a little quantity of capital in his or her trading account along with low margin needs also.

my homepage – forex valutakalkulator

For the objective of guaranteeing profitable forex trades, one wants
to be in a position to interpret the top and
lagging indicators.

Stop by my weblog; forex valuta

Before any investment in Forex you want to carefully
consider your targets, prior experience, and danger level.

Feel free to visit my blog post – 50 euro nok

Forex robot systems need to also have an inbuilt loss protection mechanism given that these systems are not foolproof.

Take a look at my web page … reisetips malta

I just want to say I am just all new to blogging and site-building and actually loved this page. Probably I’m going to bookmark your blog post . You certainly come with great posts. Appreciate it for sharing with us your blog.

I like the valuable info үⲟu provide in ʏօur articles.
Ⅰ’ll bookmark ʏоur blog аnd chefk again here frequently.

Ӏ аm quite ϲertain Ӏ willl learn ɑ ⅼot օf neա stuff right
һere! Good luck fⲟr tҺе neⲭt!

Visit my blog post: prostitute

Muchos Gracias for your blog. Will read on…

I just want to say I am beginner to weblog and definitely savored you’re web blog. Almost certainly I’m planning to bookmark your blog . You actually come with really good well written articles. Kudos for revealing your web site.

Pretty nice post. I just stumbled upon your weblog and wished to say that I have really enjoyed browsing your weblog posts.

In any case I will be subscribing on your feed and
I’m hoping you write once more very soon!

Here is my web site :: Val

Good article. I’m experiencing a few of these issues
as well..

Thanks for the marvelous posting! I genuinely enjoyed
reading it, you happen to be a great author.I will
be sure to bookmark your blog and will come back someday.
I want to encourage you to ultimately continue your great
work, have a nice morning!

Hi there to every single one, it’s in fact a fastidious for me to pay a quick visit this website, it includes
useful Information.

I read this paragraph fully regarding the comparison of latest and preceding technologies, it’s awesome article.

Hello! I could have sworn I’ve been to this web site before but after going through many of the articles I realized it’s new to me.
Nonetheless, I’m certainly pleased I discovered it and I’ll be
bookmarking it and checking back regularly!

If you wish for to grow your know-how just keep visiting this website and be updated with the hottest news posted here.

I like the helpful information you provide in your articles.
I will bookmark your weblog and check again here
frequently. I’m quite sure I’ll learn many new stuff right here!
Best of luck for the next!

To start with, you’ll find nothing wrong hwving oving to enjoy.
There are numerous people who are living to enjoy
in addition too there are a few that take to live on. Nevertheless the entire people isn’t going
to fit in with possibly of these groups. There are a few whbom appreciate
great meal, yet their own life be determined byy the food.

Club foodstuff Cheshire will be cherished in nunerous regions and if
you are any meals enthusiast, you must experience the type of
meals you obtain below. Visit a spouse andd children bbar Cheshire and you will experience the best meals along
with products with yourr total household. There is nothing wrong using adoring to enjoy.
There are some those whho reside to enjoy and there are several who eat to reside.
Even so the whole people isn’t going to are part of often these categories.

There are many which really like very good foods, although their own lifestyles rely on thier food.
Club foods Cheshire is liked in many areas in addition tto
if you are a eal partner, you should practical knowledge the sort
of food you obtain right here. Visit a spouse and children pub Cheshire and you will take advantage of the finest meals inn addition to cocktails together wijth your overall spouse and children.
The phrwse club is frequently associated wwith parents. The greatest pubs in the united kingdom arre often buisy simply by activities addicts which love to discover their own groups struggle that
out annd about while they glass draught beer along with lager and many others.
Although it’s not at all essential that the kids
cannot go to a club. Sure, there are a few cafes you would not love to consider
the kids in order to. Although a family bar Cheshire is just not amid these types of companies.
This pub foods Cheshire in this particular club is really wonderful that will everyone reaches appreciate
somewhat or another. This specific household nightclub Cheshire supplies the very best throughout English
meals. You’ll ffind many individuals indicating that will British foods
just isn’t as well as thier food which is well prepared in the various
other nations aroujd the world just like Italy ass well as Spain while others.
Also, the increase of Indian curries possesses more altered the particular colour scheme with the Uk foodies.
Although the fact is that United kingdom meals possesses its fantastic preference.
Consumed normal British isles food and you also annot support buut virtually coat your hands.

Sure, the item is impportant which the chief cook is excellent, although iif you stop
by the best place pertaining to bar food Cheshire, you can expect the cooking being over the rest the preparing food ability.
In this family tavern Cheshire you get every thing you intend to ake and also take in. In the
beverages portion you have the best possible cask ales, Brktish isles and also imported beer, cider along with a thorough collection of wine drinks.
The meals food selection iis additionally comprehensive and you also have the choice tto choose from the particular restaurant menu, your
2-for-1 food list, this pub menu along with the Weekend food selection.
The mouth-watering dinners tend to bee sure for making you’re feeling over pleased.
In the event you are looking at getting thjis particular pubb foodstuff Cheshire, visit the web
site and chjeck out the food list. Alll the info
that you need can be granted in the web site such as the range to
help call for making your current booking. With weekdays you may get to the actual restaurant in order to find an area, but
throughout weekends, it can bbe foolhawrdy looking to find a spot with no booking.
No matter the reason, it seems sensible to be
aboe to arrange upfront no mastter what when you’re. Geet
more information on-line to test outstanding club foo Cheshire.

When itt reaches this loved ones nightclub Cheshire you might have all
those meas which make an individual happy. The values usualy are properly within your budget, but the food is merely out of your globe.

Look at my web-site sinen8.com

An outstanding share! I’ve just forwarded this onto a coworker who
had been doing a little research on this. And he in fact bought me breakfast due to the fact that I found it for him…
lol. So let me reword this…. Thank YOU for the meal!!
But yeah, thanks for spending some time to talk about this topic here on your
web site.

You should be a part of a contest for one of the
best blogs on the net. I’m going to highly
recommend this web site!

Very good post. I’m going through a few of these issues as

Awesome! Its in fact awesome piece of writing,
I have got much clear idea concerning from this paragraph.

Keep on working, great job!

great issues altogether, you just won a new reader. What would you recommend
in regards to your submit that you made some days in the past?
Any sure?

I was recommended this blog by my cousin. I am not sure whether this post is written by him as no one else know such detailed about
my problem. You are wonderful! Thanks!

Have you ever thought about adding a little bit more than just your articles?

I mean, what you say is important and all.

But just imagine if you added some great pictures or video clips to give your posts more,
“pop”! Your content is excellent but with pics and clips, this website could certainly be one of the best in its field.
Terrific blog!

I am really impressed with your writing skills and also
with the layout on your blog. Is this a paid theme or did you customize it yourself?
Anyway keep up the nice quality writing, it is rare to see a great blog
like this one nowadays.

Cuando recuerdas el día de tu boda , miras el álbum de fotos, la mayoría de las parejas lo que hecha en falta son fotos de sus amigos y familiares, y es que con el ajetreo que los novios llevan ese día, no son capaces, muchas veces, ni de recordar quién fue a la boda, y muchos menos como iban vestidas sus invitadas, alguna imagen graciosa que ellos vivieran en tu día.

WOW just what I was searching for. Came
here by searching for quest bars

Pretty section of content. I simply stumbled
upon your website and in accession capital to say that I acquire
in fact loved account your weblog posts. Anyway I’ll be subscribing in your augment and
even I achievement you get admission to persistently rapidly.

It’s perfect time to make some plans for the longer term and it’s time to be happy.
I have read this put up and if I could I desire to recommend you some fascinating issues or suggestions.
Perhaps you could write next articles relating to this article.
I want to learn even more things approximately it!

Leave a Reply