Metadata Performance of Four Linux File Systems

Each test was run 10 times with the four combinations of the file systems (ext3, ext4, btrfs, nilfs). The test system used for these tests was a stock CentOS 5.3 distribution but with a 2.6.30 kernel and e3fsprogs was upgraded to the latest version as of the writing of this article, 1.41.9. The tests were run on the following system:

• GigaByte MAA78GM-US2H motherboard
  
• An AMD Phenom II X4 920 CPU
  
• 8GB of memory
  
• Linux 2.6.30 kernel
  
• The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  
• /home is on a Seagate ST1360827AS
  
• There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb for all of the tests.

For all 4 file systems, the defaults were used in building the file systems. For btrfs, version btrfs-progs v0.18. For nilfs2, nilfs-utils-2.0.14 was used. Both ext3 and ext4 were mounted with “data=ordered” since this is recommended practice to prevent data loss.

Benchmark Results

This section presents the results of the testing (exploration). The results are presented in tables listing the average value and just below it, in red, is the standard deviation. This is done for each of the four combinations and each of the four file systems.

The first combination tested was for small files (4 KiB) with a shallow directory structure. Table 1 below lists the results with an average value and just below it, in red, is the standard deviation.

Table 1 – Benchmark Times Small Files (4 KiB) – Shallow Directory Structure

The first test, directory creates, had an average run time of 12 seconds for all four file systems, so the results may not be that meaningful. In addition, the directory remove test ran in about 1 second or less. Consequently, this test may not have much value.

Table 2 below lists the performance results with an average value and just below it, in red, is the standard deviation.

Table 2 – Performance Results of Small Files (4 KiB) – Shallow Directory Structure

The second combination tested was for small files (4 KiB) with a deep directory structure. Table 3 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.

Table 3 – Benchmark Times Small Files (4 KiB) – Deep Directory Structure

For these tests, the first test, directory creates, took about 40 seconds for ext3 (the fastest). This time is fairly small and, consequently, the results may not be as applicable the other tests which had a much longer run time. The last test, directory removes, took 11 seconds for ext3 (the fastest). Again, this time is fairly quick so it may not be as useful because the time is so short.

Table 4 below lists the performance results with an average value and just below it, in red, is the standard deviation.

Table 4 – Performance Results of Small Files (4 KiB) – Deep Directory Structure

The third combination tested was for medium files (4 MiB) with a shallow directory structure. Table 5 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.

Table 5 – Benchmark Times Medium Files (4 MiB) – Shallow Directory Structure

For these tests, the first test, directory creates, took less than 1 second. This time is very small and, consequently, the results are not as applicable as some of the other tests. The file removes test took about 10-15 seconds. Again this is a very short time and the results may not be as applicable. The last test, directory removes, took 0-1.4 seconds. This time too, is very short.

Table 6 below lists the performance results with an average value and just below it, in red, is the standard deviation.

Table 6 – Performance Results of Medium Files (4 MiB) – Shallow Directory Structure

The fourth and final combination tested was for medium files (4 MiB) with a deep directory structure. Table 7 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.

Table 7 – Benchmark Times Medium Files (4 MiB) – Deep Directory Structure

The first test, directory creates, took 2-3 seconds, which is very short. The time for the third test, file removal, was also fairly short at 11-19 seconds. The last test, directory removes, was extremely fast at less than 2 seconds. These three results are somewhat suspect because of short run time.

Table 8 below lists the performance results with an average value and just below it, in red, is the standard deviation.

Table 8 – Results of Medium Files (4 MiB) – Deep Directory Structure

Discussion of Results

There are four different combinations that were tested for the four file systems. Comparing the file systems is interesting but also comparing the same file system for the different tests is interesting as well.

Firstly, let’s examine the shallow directory structure results (Tables 2 and 6). For small files (4 KiB) the four file systems performed about the same for the file create and file removal tests (the directory create and remove tests ran too quickly to be really useful). All four file systems achieved about 1,000 file creates per second or about 4,000 KiB per second (see Table 2). But medium files (4 MiB) produced very different results. For this case, btrfs was almost twice as fast as nilfs and 50% faster than ext3 or ext3 with respect to the number of files creates per second or throughput (KiB/s) (see Table 6).

However, the small file case produced 109X more files, 27X more directories, but only produced about 1/10-th the total amount of data. This points out the extreme pressure that is put on metadata performance of file systems for small files.

Second, we can perform the same comparison for deep directory structure (Tables 4 and 8). Small files (1 KiB) put extreme pressure on file system metadata performance as with the shallow directory structure.

Examining the results, the following observations are made:

• Small files put extreme pressure on metadata performance regardless of file systems


• For small files, a shallow or deep directory structure did not appreciably impact metadata performance


• For larger files, a shallow or deep directory structure also did not appreciably impact metadata performance


• For small files, btrfs has good file creation performance but file removal performance is not as good as ext3 and ext4 at this time


• For larger files, btrfs has both excellent file creation and removal performance relative to the other 3 file systems


• Log-base file systems such as nilfs2 should work well with metadata tests. But the developers are evolving the garbage collection (gc) algorithm which should improve performance.


• The standard deviation, or the spread in the data was much greater for ext3 than the other three file systems. The reason(s) for this are not known.

This is the first attempt at useful benchmarks and analysis for Linux file systems using the approach outlined in a previous article around benchmarking. There will be future articles that use the same basic tenants. But be ready because this approach introduces a great deal of data into the article. But overall it does give much more information that a quick table and a conclusion “file system X is better” (usually followed by a run for cover). Please let me know in the forums if this type of article is useful.