Size Can Matter: Ramdisk Journal Metadata Performance – Part 2

Previously, we examined the impact of journal size using a separate disk on metadata performance as measured by fdtree. In this follow-up we repeat the same test but use a ramdisk for the journal, thereby boosting the best performance. Or does it?

In a previous article we took a look at metadata performance and found that regardless of whether the journal was on a separate disk or a ramdisk performance was almost the same. However, the size of the journal was very small compared to the size of the file system (0.0032%) perhaps impacting performance.

The article linked above was all about metadata performance using Ext4 with a journal device on a separate disk. The key variable in the quest for ultimate performance, the journal size, was varied for a fixed file system size. Metadata performance was measured using the fdtree benchmark which has been used in a number of our previous metadata-focused articles.

In this article the journal is placed on a ramdisk and its size is varied to understand the impact on metadata performance measured by the metadata benchmark fdtree. The goal of placing the journal on a ramdisk is to test the performance when the fastest possible device is used for the journal. However, be warned that the journal will not survive a reboot so the file system cannot be mounted normally. The only practical way the journal can be placed on a ramdisk and survive a reboot is to sync the file system, unmout it, and then shutdown the system. When the system is rebooted, the ramdisk based journal needs to be recreated and the file system is remounted. This process is likely to be a manual process but could be automated in a script.

Testing Discussion

As with the previous article, four journal sizes are tested to understand the impact of journal size on metadata performance. The four journal sizes are:

  • 16MB (0.0032% of file system size)
  • 64MB (0.0128% of file system size)
  • 256MB (0.0512% of file system size)
  • 1GB (0.2% of file system size)

A ramdisk of the appropriate size is created and is then utilized for the journal for the ext4 file system.

CentOS 5.3 was used as the OS for the testing. By default, the maximum ramdisk size is 16MB. To increase the ramdisk size, a parameter is passed to the kernel during boot. The command line in /etc/grub.conf is added to the kernel boot line so it looks like the following:

kernel /vmlinuz-2.6.30 ...  ramdisk_size=64000

The parameter was changed to the appropriate size and the system was rebooted. Then the device, /dev/ram0, was used as the journal device.

The details of the metadata testing can be found in the previous article and won’t be repeated here except to say that four scenarios were tested.

Each test was run 10 times for the four journal sizes. The test system used for these tests was a stock CentOS 5.3 distribution but with a 2.6.30 kernel. In addition, e2fsprogs was upgraded to 1.41.9. The tests were run on the following system:

  • GigaByte MAA78GM-US2H motherboard
  • An AMD Phenom II X4 920 CPU
  • 8GB of memory (DDR2-800)
  • Linux 2.6.30 kernel
  • The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  • /home is on a Seagate ST1360827AS drive
  • There are two drives for testing. They are both Seagate ST3500641AS-RK drives with a 16 MB cache each. These drives show up as devices, /dev/sdb and /dev/sdc.

The first Seagate drive, /dev/sdb, was used for the file system and was used exclusively in these tests.

The details of creating an ext4 file system with a journal on a separate device are contained in a previous article. The basic steps are to first create the file system assuming the journal is located with the file system on the drive. Second, a new journal is created on a ramdisk (/dev/ram0). Finally, the file system is told that that it no longer has a journal and then it is told that it’s journal is on the specific device (the ramdisk).

Benchmark Results

This section presents the results for the four scenarios. The results are plotted along with the error bars to allow easy comparison. However, the full results are available in tabular form at the end of the article after the summary.

The first test is for the “small file, shallow structure” scenario for the four journal sizes. The “file create” test ran for over 300 seconds and the “file remove” test ran for a bit over a minute. Consequently, these tests are considered worthwhile for examination. Figure 1 below plots the average file create performance in KiB per second for the four journal sizes. Also note that error bars representing the standard deviation are shown.

small_shallow_file_creates.png
Figure 1: Average File Create Performance (KiB per second) for the Small File, Shallow Structure Test for the Four Journal Sizes

Figure 2 below plots the average “File Remove” results in “File Removes per second” for the four journal sizes for the small file, shallow structure scenario. Again, there are error bars representing the standard deviation in the plot as well.

small_shallow_file_removes.png
Figure 2: Average File Remove Performance (File Removes per second) for the Small File, Shallow Structure Test for the Four Journal Sizes

The next test uses small files but with a deep directory structure. For this scenario all four tests had run times long enough for consideration. The “Directory Create” test ran from an average of 120.1 seconds to 312.4 seconds depending upon the journal size. The “File Create” test ran an average of 328.5 seconds to 566.6 seconds. The “File Remove” test ran an average of 136.6 seconds to 333 seconds. And finally the “Directory Remove” test ran from an average of 52.5 seconds to 253.0 seconds.

Figure 3 below plots the average “Directory Create” results in “creates per second” for the four journal sizes for the small file, deep structure scenario. Again, there are error bars representing the standard deviation in the plot as well.

small_deep_dir_creates.png
Figure 3: Average Directory Create Performance (creates per second) for the Small File, Deep Structure Test for the Four Journal Sizes

Figure 4 below plots the average “File Create” results in KiB per second for the four journal sizes for the small file, deep structure scenario. Again, there are error bars representing the standard deviation in the plot as well.

small_deep_file_creates.png
Figure 4: Average File Create Performance (creates per second) for the Small File, Deep Structure Test for the Four Journal Sizes

Figure 5 below plots the average “File Remove” results in removes per second for the four journal sizes for the small file, deep structure test.

small_deep_file_removes.png
Figure 5: Average File Remove Performance (removes per second) for the Small File, Deep Structure Test for the Four Journal Sizes

Figure 6 below plots the average “Directory Remove” results in removes per second for the four journal sizes for the small file, deep structure test.

small_deep_dir_removes.png
Figure 6: Average Directory Remove Performance (removes per second) for the Small File, Deep Structure Test for the Four Journal Sizes

The next test was the medium files, shallow directory structure scenario. The only result that had a meaningful run time was the file create test (154.2 seconds to 154.4 seconds). Figure 7 below plots the the file create performance in KiB per second for the four journal sizes. Also note that the error bars are plotted as well.

medium_shallow_file_creates.png
Figure 7: Average File Create Performance (KiB per second) for the Medium File, Shallow Structure Test for the Four Journal Sizes

The final test was the medium files, deep directory structure scenario. The only result that had meaningful times was the file create test (220.4 seconds to 225.9 seconds). Figure 8 below plots the the file create performance in KiB per second for the four journal sizes. Also note that the error bars are plotted as well.

medium_deep_file_creates.png
Figure 8: Average File Create Performance (KiB per second) for the Medium File, Deep Structure Test for the Four Journal Sizes

Benchmark Observations

The benchmark results are very interesting since we actually see some variation in the results whereas in the first article we did not seem much variation. A quick summary of the results is given below.

  • Small files, shallow directory structure:
    • From Figure 1, increasing the journal size to 256MB increased the average file creation by 6% (from 3,837.2 KiB/s to 4,080.1 KiB/s). Increasing the journal size to 1GB did not increase the file create performance by any measurable amount.
    • From Figure 2, the file remove performance increased by 25% when the journal size went from 16MB to 256MB. But increasing the journal from 256MB to 1GB didn’t make an appreciable change in performance.
  • Small files, deep directory structure:
    • The directory creation performance increased dramatically as the journal size increased (Figure 3). The average performance increased by 163% (from 282.8 KiB/s at 16MB to 743.6 KiB/s at 1GB).
    • The file creation performance also increased fairly remarkably as seen in Figure 4. It increased by 58% (2,267.8 KiB/s at 16MB to 3,572.8 KiB/s at 1GB).
    • The file removal rate (removes per second) increased by 144% in going from a 16MB journal to a 1GB journal (1,063.6 removes/s to 2,599.6 removes/s). See Figure 5 for the image.
    • The most remarkable change in performance was for the directory removal result as seen in Figure 6. The directory removal rate (removes per second) increased by 410% in going from a 16MB journal to a 1GB journal (504.46 removes/s to 1,849.4 removes/s). However, the variation in results at 1GB is very large.
  • Medium files, shallow directory structure
    • The file create performance actually decreased slightly going from 16MB to 1GB (Figure 7). The decrease in average performance was only about 2% but the results are within the error band of the others. The reason for the decrease is unknown at this time.
  • Medium Files, deep directory structure
    • The file creation performance (in KiB/s) only really changed in going from a 16MB journal size to a 64MB journal size (see Figure 8). The performance with a 64MB journal size is about 2% better than with 16MB (73,971.3 KiB/s vs. 72,495 KiB/s).
    • After 64MB the file creation performance did not change appreciably and the average performance at 1GB actually decreased slightly but is within the error bounds of the 64MB and 256MB journal sizes.

Summary

This is the next article in a series examining the impact of journal size and device on ext4 performance. The previous article examined the performance when the journal is on a separate disk. This article examines the performance when the journal is placed on a ramdisk. It is expected that the ramdisk based journal will produce the best metadata performance results.

As with the previous test the metadata benchmark, fdtree, was used for testing the performance of four journal size: 16MB, 64MB, 256MB, and 1GB. For all four sizes, the journal was placed on a ramdisk. As with previous metadata testing, four scenarios were tested: (1) small files (4 KiB) and a shallow directory structure, (2) small files (4 KiB) and a deeper directory structure, (3) medium sized files (4 MiB) and a shallow directory structure, and (4) medium sized files (4 MiB) and a deeper directory structure.

The results are as interesting as they were for the journal device on a second disk. Increasing the journal size generally results in an increase in the file create performance for all scenarios as it did with the disk based journal. The performance increased from a little over 2% to over 58% depending upon the scenario. However for the medium file (4 MiB), shallow directory structure scenario, the average performance actually decreased slightly.

The file remove performance was greatly affected by journal size. Increasing the journal size increased the file removal performance by 25% for the small files, shallow directly structure scenario and 144% for the small files, deep directory structure scenario.

The most remarkable change in performance was for the directory remove test for the small files, deep directory structure scenario where the performance increased by 410% when going from a 16MB journal to a 1GB journal.

Will Batman escape the Catwoman’s Cluthes? Will Gotham City ever switch to Linux from the evil Joker’s Windows domination? Will Jeff ever finish the testing of ext4 journal options? Say tuned to the next article to find out.

Next: Table of Results

Comments on "Size Can Matter: Ramdisk Journal Metadata Performance – Part 2"

djekels

I tried creating ram drives on a 3TB ext4 File System.

pvcreate /dev/ram[0-20]
vgcreate jrnlvg /dev/ram[0-20]
lvcreate -L1G -n lv0 jrnlvg

mkfs -O journal_dev /dev/jrnlvg/lv0
tune2fs -L /opt journal device /dev/jrnlvg/lv0

tune2fs -j -J device=/dev/jrnlvg/lv0 /dev/myTBvg/lv10

this all works great

the problem is when the system reboots, you must go through the whole process and 1+TB file systems regenerating their journals takes forever.

Q: how can I dd or copy the ram device data to a regular disk.
so when the system reboot, you setup the ram devices then copy the old jourabck before you fire up the FS?

Reply

thank you for share!

Reply

thank you for share!

Reply

Thanks again for the blog post.Really thank you! Awesome.
russian humor

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

Size Can Matter: Ramdisk Journal Metadata Performance – Part 2 | Linux Magazine

Reply

I think other site proprietors should take this website as an model, very clean and great user genial style and design, let alone the content. You’re an expert in this topic!

Reply

J6LGYH Say, you got a nice post. Great.

Reply

Thank you ever so for you post. Cool.
fiverrr23Jz

Reply

I appreciate you sharing this blog article.Really thank you! Fantastic.
fiverrr23Jz

Reply

DxOxNS Appreciate you sharing, great post.Much thanks again. Cool.

Reply

H7s6bU Thanks a lot for the blog post.Really thank you! Really Cool.

Reply

2E51qs Appreciate you sharing, great post.Really looking forward to read more. Much obliged.

Reply

TdsCQE Thanks a lot for the article post.Much thanks again. Great.

Reply

6jvEVw I am so grateful for your article post.Thanks Again.

Reply

tJeayR Awesome blog post.Really thank you! Fantastic.

Reply

If the business enterprise is totally new or a novice, it really is unlucky
that the business has any credit of the own and so the company owners credit will likely be utilized to determine the additional value for a loan prince royce concert this effort
is the investigating online what type are able to do for the low
credit score secured personal loan.

Reply

zjifiT This is one awesome post. Cool.

Reply

Take a try and have as many questions when you want to
katy perry tickets it may be worth
to notice that folks with assorted levels of education need being
motivated in order to operate optimally.

Reply

whoah this weblog is fantastic i like studying your articles. Stay up the great work! You realize, many individuals are searching round for this info, you can help them greatly. |

Reply

OUJ4jz fcfsmsvazowy, [url=http://crqnxnthcgsl.com/]crqnxnthcgsl[/url], [link=http://cqtevttvnhrc.com/]cqtevttvnhrc[/link], http://zfmhyjmexuip.com/

Reply

Hi there, just wanted to tell you, I liked this blog post.
It was inspiring. Keep on posting!

Here is my homepage :: Vilma R. Clausing

Reply

qpl2Ir sroqceyidzsf, [url=http://axbyjimlwdvq.com/]axbyjimlwdvq[/url], [link=http://feisdbjygkxm.com/]feisdbjygkxm[/link], http://rbdrizrhymxu.com/

Reply

hgo0cU qaxazutzrjfu, [url=http://szloykgxbytj.com/]szloykgxbytj[/url], [link=http://wrirsrlkurur.com/]wrirsrlkurur[/link], http://ztkykzvmjlog.com/

Reply

kH1P3I everxjxtcuyc, [url=http://ixtrfskboumy.com/]ixtrfskboumy[/url], [link=http://xucsilowajua.com/]xucsilowajua[/link], http://avzgxroblblw.com/

Reply

PFOLwD apnhqutorqfk, [url=http://dbhecetaqkik.com/]dbhecetaqkik[/url], [link=http://eszveijhhpgp.com/]eszveijhhpgp[/link], http://xrvyzawgszor.com/

Reply

nPtrEr thfgdahtqzzl, [url=http://edaycwyytwlt.com/]edaycwyytwlt[/url], [link=http://xshdeaghqhxf.com/]xshdeaghqhxf[/link], http://nixagvhnqhqd.com/

Reply

got several concerns considering the survey shit, but this time it payed back :))

Reply

the 101th cartel coin hack that I am trying, so.. 1 out of 100 are working :|

Reply

spSlTK xkwhlizhltgw, [url=http://mmkhpinjmnru.com/]mmkhpinjmnru[/url], [link=http://cwwuutxnodlr.com/]cwwuutxnodlr[/link], http://pcfejfnwroag.com/

Reply

FED8R5 dvdiktddfvij, [url=http://vtgtjxlwpidi.com/]vtgtjxlwpidi[/url], [link=http://ifucjvsyunjl.com/]ifucjvsyunjl[/link], http://qxiyfqfdihjq.com/

Reply

BbST8o wjzhtxzxiexs, [url=http://fofumqchkjpf.com/]fofumqchkjpf[/url], [link=http://ogalhusdffyn.com/]ogalhusdffyn[/link], http://inzantgjqqbd.com/

Reply

WpNjF7 bwqbbmtpvzqm, [url=http://cbdtdlkdnktj.com/]cbdtdlkdnktj[/url], [link=http://mfdnokugbzpk.com/]mfdnokugbzpk[/link], http://fyulrnyuhrwo.com/

Reply

CCPh62 gzexfatbnqdz, [url=http://fbizsmqumeth.com/]fbizsmqumeth[/url], [link=http://piaylfipgulg.com/]piaylfipgulg[/link], http://iepdfyzihicc.com/

Reply

oWJSAi ecsbgtybhnwo, [url=http://rueoocwwkfpq.com/]rueoocwwkfpq[/url], [link=http://vbprbssnscma.com/]vbprbssnscma[/link], http://akflywuvvykw.com/

Reply

OrJDyx ttsvxdvidzei, [url=http://zlkddrojvxob.com/]zlkddrojvxob[/url], [link=http://nmqaoillbqik.com/]nmqaoillbqik[/link], http://vaifbweplnob.com/

Reply

YRWFgo dtgnldntdjlv, [url=http://gprubczvqjlf.com/]gprubczvqjlf[/url], [link=http://iygqujhkvlim.com/]iygqujhkvlim[/link], http://mqomlkemcnbs.com/

Reply

nNehhc naelegcjaqus, [url=http://vguvumpzgfth.com/]vguvumpzgfth[/url], [link=http://rnmpkdvxxktv.com/]rnmpkdvxxktv[/link], http://tqhbjjytpkad.com/

Reply

EtnjuV gvtqynlyzzcq, [url=http://zozgdslvguok.com/]zozgdslvguok[/url], [link=http://ltvxgphlkjld.com/]ltvxgphlkjld[/link], http://hxwhlrdaykmc.com/

Reply

I am curious to find out what blog system you are working with?
I’m having some minor security problems with my latest website and I would
like to find something more secure. Do you have any recommendations?

Reply

For fashionable women high heel shoes are a must for almost all occasions except maybe the daily grind on the treadmill. Since these high end trendy shoes are donned by many a famous Hollywood star and celebrity, it is hard to believe that you can actually buy Wholesale Christian Louboutin shoes.

Reply

A untroubled old majority stands out as the award of your well-spent youth. Rather then its bringing sad and low prospects of decay, it will yield to defeat us hopes of eternal youth in a less ill globe

Reply

461251 826948light bulbs are great for lighting the home but stay away from incandescent lamps because they create so a lot heat;; 676635

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>