Metadata Performance Exploration Part 2: XFS, JFS, ReiserFS, ext2, and Reiser4

Last week we tested four Linux file systems — ext3, ext4, nilfs2, and btrfs — for metadata performance using a benchmark called fdtree. The point of the benchmarks was not really to comparison the performance of the file systems per say, although comparisons are inevitable. Rather, the benchmarks were performed as part of an exploration into the metadata performance of Linux file systems.

We’re using the same benchmark from the last article and applying it to additional Linux file systems - xfs, jfs, reiserfs, ext2, and resier4. As you are probably aware there are a large number of file systems available in Linux from some fairly old ones such as ext2, to some that are still considered “experimental” in the latest kernel (2.6.30 as of this writing) such as btrfs and nilfs2. But these other file systems, ext2, xfs, jfs, reiserfs are still in production use in a number of places. Consequently this article performs the same benchmarks as the previous article on these additional file systems. It also adds in reiser4 which, believe it or not, is still moving ahead thanks to the determination of some of the developers.

Quick Review of Benchmark

The benchmark used in the previous article is fdtree. It’s perhaps not the best known metadata benchmark for file systems but it is fairly common in HPC circles. It stresses the creation and removal of directories and files using a simple bash script and *nix utilities. It builds a tree structure of directories. A number of files of a given size is created in each directory. The number of directories (branches) at each level in the tree, the depth of the tree, the number of files at each point, and the size of the files are all under user control.

For the specified benchmark two different directory tree structures were used: (1) shallow tree (not much depth) with a larger number of directories at each level, and (2) a deep tree structure with only a few directories at each level but many levels. There are also two file system sizes used: (1) a small size, 4 KiB (1 block), (2) a medium size, 4 MiB (1,000 blocks). This makes a total of four (4) benchmark sets run.

• Small files (4 KiB)
  
  
  
  • Shallow directory structure
    
  • Deep directory structure
    
  
  
• Medium files (4 MiB)
  
  
  
  • Shallow directory structure
    
  • Deep directory structure
    
  
  

To create the specific parameters for fdtree used in the exploration, there were three overall goals:

• Keep the total run time to approximately 10-12 minutes at a maximum


• Keep the total data for the two directory structures approximately the same


• Keep the run time for each of the four functions greater than 1 minute if possible

All four functions were not always run for 1 minute, sometimes only for a few seconds. These will be noted in the results

In keeping with the good benchmark behavior laid out in previous article, the test was run 10 times with the four combinations for the file systems. The test system used for these tests was a stock CentOS 5.3 distribution but with a 2.6.30 kernel and e2fsprogs was upgraded to the latest version as of the writing of this article, 1.41.9. In addition, the following file system tools were used:

• xfs: xfsprogs pulled via git as of 9/5/2009
  
• jfs: jfsutils-1.1.14
  
• reiserfs: reiserfsprogs-3.6.21
  
• ext2: e2fsprogs 1.41.9
  
• reiser4: resier4progs-1.0.7
  

The defaults for all the file systems were used (tuning is a subject for a whole series of articles and tons of additional work).

The tests were run on the following system:

• GigaByte MAA78GM-US2H motherboard
  
• An AMD Phenom II X4 920 CPU
  
• 8GB of memory
  
• Linux 2.6.30 kernel
  
• The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  
• /home is on a Seagate ST1360827AS
  
• There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb, for all of the tests.

Further details of the specific parameters for fdtree that were run are in the previous article.

Results

The results for the new file systems are included in the tables with the results from the previous study. The new results are presented first followed by the previous data. There are tables for each of the four tests (making a total of 8 tables).

The first table for each test is the time for each test. This data is used to help validate the test since the desire is to have the test run for more than a few seconds. Ideally, it should run for at least 60 seconds (1 minute). The second table presents the actually performance data. Larger values are better than smaller values.

Table 1 - Benchmark Times Small Files (4 KiB) - Shallow Directory Structure

Table 2 - Performance Results of Small Files (4 KiB) - Shallow Directory Structure

Table 3 - Benchmark Times Small Files (4 KiB) - Deep Directory Structure

Table 4 - Performance Results of Small Files (4 KiB) - Deep Directory Structure

Table 5 - Benchmark Times Medium Files (4 MiB) - Shallow Directory Structure

Table 6 - Performance Results of Medium Files (4 MiB) - Shallow Directory Structure

Table 7 - Benchmark Times Medium Files (4 MiB) - Deep Directory Structure

Table 8 - Results of Medium Files (4 MiB) - Deep Directory Structure

Discussion of Results

In general, only the file create and file removal tests ran long enough to be useful. In the small file, deep directory test, the directory creation step ran long enough to produce meaningful results, but it is the only test where this happens. Consequently it won’t be discussed here.

The observations from the previous article are still valid with some accommodation for the new file systems. These observations are:

• Small files put extreme pressure on metadata performance regardless of the file system. Compare the file create and removal rates for the small files versus the medium files. The rates are about an order of magnitude smaller for small files. However, this is to be expected because there are simply many more files.


• For small files, a shallow or deep directory structure did not appreciably impact metadata performance. However, the deep directory structure did produce slower results in general.


• For larger files, a shallow or deep directory structure also did not appreciably impact metadata performance. However, again, for deep directories, the performance was slightly slower than shallow directories.


• There can be a great deal of variation in metadata performance for some of the file systems. The reason for this is unknown at this time.

These are general observations. However, I’m sure most readers are comparing the file systems even before they reach this point in the article. In keeping with the crowd, let’s do a little contrasting of the file systems (and I mean a little).

• Small Files:
  
  
  
  • A number of file systems had about the same performance on fdtree: btrfs, ext4, ext2, reiser4, and nilfs2.
  
  
  • Surprisingly, xfs did not have good performance falling down badly relative to the others.
  
  
  • Reiserfs did well on the shallow test but not so well on the deep test.
  
  
  
  
• Medium Files:
  
  
  
  • btrfs had the top performance by a fairly wide margin - approximately 10%
    
  • In second place, xfs and resier4 did very well. So xfs has redeemed itself from small file performance
  
  
  • ext4 is close behind xfs and reiser4.
    
  • The remaining file systems, jfs, reiserfs, ext3, and nilfs23 all drop down a bit below xfs, reiser4, and ext4.
  
  
  • As a secondary consideration, jfs and reiser4 actually improve in performance when going to a deep directory structure. They are the only file systems to do this.
  
  
  
  

And finally a few quick observations about the file systems in general:

• Log-base file systems such as nilfs2 should work well with metadata tests. But the developers are evolving the garbage collection (gc) algorithm which should improve performance.


• Reiserfs is going through some changes to remove some locks. This should help performance


• The problems with xfs and small files is unknown at this time. If you any suggestions as to options for improving performance, please let me know.


• btrfs has really good performance at this stage and it is still experimental

One final word of caution. Do not pick your file system based on these results alone. However, let me know if this “type” of article is useful. And please, if you are upset that your file system isn’t included or if your file system didn’t do as well as you would like, please try out the benchmark yourself and post the results in the forums for everyone.