dcsimg

IOzone Performance Exploration, Part 2: The Rest of the Crowd (Almost)

We finish off our IOzone performance exploration of the major Linux file systems. This time adding ext2, jfs, xfs, btrfs, and reiserfs. Let's take a look at the numbers.

In our previous Part 1 of this series we looked at the IOzone performance for ext3, ext4, and reiser4 using larger size record lengths (1MB, 4MB, 8MB, 16MB). The results are interesting because they unexpectedly showed some trends in performance that are independent of the file system. These observations are important if you take the results from a single record size run of IOzone and apply it to other record sizes (which you shouldn’t do).

In this article the number of file systems is extended to add most of the major file systems in Linux, even the experimental ones. This article adds ext2, jfs, xfs, btrfs, and reiserfs. As with the metadata benchmarks the purpose of this study is not to compare file systems and pick the “best” one (insert your definition of “best”). Rather, this study is an exploration of the performance of various Linux file systems using a single throughput benchmark. So the focus of this article is to explore how Linux file systems perform when IOzone is the benchmark.

Review of Last Results

This study uses IOzone to examine throughput of various access patterns for various Linux file systems. Recall that IOzone has a number of tests (access patterns) that stress the file system:


  • Write
  • Re-write
  • Read
  • Re-read
  • Random Read
  • Random Write
  • Backwards Read
  • Record Rewrite
  • Strided Read
  • Fwrite
  • Frewrite
  • Fread
  • Freread

The general access pattern was discussed in the previous article.

A few observations were made in the previous article about the general behavior of the tested file systems:


  • Write and re-write performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  • Random write got much better as the record size increased from 1MB to 16MB. For example, ext3 went from 53,709.90 KB/s to 62,060.20 KB/s.
  • Record re-write got worse as the record size increased. This performance can be driven by various cache and buffer sizes. Perhaps as the size increases, some cache sizes are exceeded and performance decreases.
  • Read and re-read performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  • Random read got dramatically better as the record size increased. For example at 1MB the random read performance for ext4 was 52,620.10 KB/s and at a record size of 16MB it was 105,651.60 KB/s. This is about a 100% increase in throughput performance! Some increase in performance is expected because there are fewer records as the record size is increased, but the change in performance is fairly dramatic.
  • Backward read performance also dramatically improved with increasing record size. This is to be expected since there is less drive head movement because there are fewer total records.
  • Strided read throughput performance also dramatically improved as the record size was increased. Again, it is about a 100% increase in performance from a record size of 1MB to a record size of 16MB.
  • fread and freread are relatively insensitive to a change in record size.

Test System

The tests were run on the same system as the previous tests. The highlights of the system are:


  • GigaByte MAA78GM-US2H motherboard
  • An AMD Phenom II X4 920 CPU
  • 8GB of memory (DDR2-800)
  • Linux 2.6.30 kernel (with reiser4 patches only)
  • The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ulta ATA/100)
  • /home is on a Seagate ST1360827AS
  • There are two drives for testing. They are Seagate ST3500641AS-RK with 16 MB cache each. These are /dev/sdb and /dev/sdc.

Only the first Seagate drive was used, /dev/sdb, for all of the tests.

IOzone Command Parameters

For this article, as with the last, only four record sizes are tested: (1) 1MB, (2) 4MB, (3) 8MB, and (4) 16MB. For a file size of 16GB that results in (1) 16,000 records, (2) 4,000 records, (3) 2,000 records, (4) 1,000 records.

The command line for the first record size (1MB) is,

./IOzone -Rb spreadsheet_output_1M.wks -s 16G -r 1M > output_1M.txt

The command line for the second record size (4MB) is,

./IOzone -Rb spreadsheet_output_4M.wks -s 16G -r 4M > output_4M.txt

The command line for the third record size (8MB) is,

./IOzone -Rb spreadsheet_output_8M.wks -s 16G -r 8M > output_8M.txt

The command line for the fourth record size (16MB) is,

./IOzone -Rb spreadsheet_output_16M.wks -s 16G -r 16M > output_16M.txt

Test Results

As mentioned previously, there are 13 tests that are each run 10 times and there are 4 record sizes. This makes a total of 520 tests that were run per file system.

Because of the large number of tests that are run, the results are split into two tables. The first table is for the write tests: write, re-write, random write, record re-write, fwrite, frewrite. The second table is for the read tests: read, re-read, random reads, backwards read, strided read, fread, and freread. Each group below has two tables for a specific record size group (1MB, 4MB, 8MB, 16MB). So that means there are 8 tables of results.

The first two tables of results are for the 1MB record size. Table 1 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 1 – IOzone Write Performance Results with a Record Length of 1MB and a File Size of 16GB

File System Write
KB/s
Re-write
(KB/s)
Random write
(KB/s)
Record re-write
(KB/s)
fwrite
(KB/s)
frewrite
(KB/s)
Reiser4 100,476.50
167.31
94,290.90
231.34
62,220.00
98.06
3,258,345.00
372,536.39
100,095.70
197.08
94,258.60
156.25
ext3 72,938.10
272.06
75,759.60
435.48
53,709.90
712.81
2,715,427.60
22,943.63
72,705.10
573.25
75,250.30
492.69
ext4 109,339.90
370.61
103,843.50
8,348.59
66,683.50
445.16
2,980,795.70
42,895.82
109,147.50
180.18
108,184.30
165.20
ext2 110,180.20
455.65
101,163.80
76.83
77,323.50
437.84
3,398,865.80
32,323.03
110,282.60
380.54
99,474.90
1,611.26
jfs 39,500.40
1,182.29
105,684.20
2,087.95
76,246.70
395.68
3,478,208.70
52,112.39
39,040.50
190.87
102,491.90
702.24
xfs 113,667.70
356.55
113,684.80
518.88
78,354.30
902.01
3,357,432.40
47,951.78
113,469.00
183.75
114,363.40
275.10
btrfs 105,828.00
2,362.61
108,022.80
1,577.79
107,099.00
502.51
3,309,137.50
216,655.75
98,466.10
961.72
101,230.60
377.78
reiserfs 107,032.90
1,261.99
112,049.80
491.51
91,433.70
4087.92
3,272,360.30
75,848.56
107,884.80
1,519.54
112,040.80
301.72

Table 2 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 1MB.

Table 2 – IOzone read Performance Results with a Record Length of 1MB and a File Size of 16GB

File System Read
(KB/s)
Re-read
(KB/s)
Random read
(KB/s)
Backwards read
(KB/s)
Strided read
(KB/s)
fread
(KB/s)
freread
(KB/s)
Reiser4 106,777.20
61.33
106,861.50
22.18
61,980.50
78.35
66,201.50
144.74
49,071.90
122.04
106,719.80
39.03
106,787.60
61.63
ext3 71,903.60
2,688.14
76,442.60
4,827.36
52,851.20
79.31
59,207.50
61.13
49,839.60
56.14
71,746.00
3,857.21
70,165.50
2,430.79
ext4 97,969.20
46.97
98,050.60
55.21
52,620.10
104.39
75,419.50
135.84
51,048.90
35.03
98,039.90
15.35
98,041.30
54.90
ext2 97,124.00
6,384.66
101,119.50
5,478.30
61,818.00
349.21
78,147.80
127.17
54,456.70
1,647.81
92,690.10
5,961.05
87,314.90
763.69
jfs 106,320.00
259.99
106,651.50
183.53
53,182.10
190.01
78,038.80
98.96
66,076.50
193.06
106,558.10
48.72
107,051.00
33.69
xfs 106,965.30
64.74
106,9992.0
80.59
63,641.50
206.79
79,097.80
55.86
54,481.20
53.43
107,015.50
135.41
107,034.90
80.91
btrfs 102,851.30
957.26
103,123.10
953.70
56,769.90
178.35
56,224.70
567.83
57,826.10
384.28
99,655.80
337.50
99,914.50
377.78
reiserfs 93,853.70
1,319.91
95,643.70
858.44
62,614.80
1,643.73
84,807.30
292.07
54,212.00
58.80
92,957.10
1,819.37
94,928.40
2,627.24

The next two tables of results are for the 4MB record size. Table 3 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 3 – IOzone Write Performance Results with a Record Length of 4MB and a File Size of 16GB

File System Write
KB/s
Re-write
(KB/s)
Random write
(KB/s)
Record re-write
(KB/s)
fwrite
(KB/s)
frewrite
(KB/s)
Reiser4 100,360.50
181.21
94,389.00
217.43
71,415.40
306.27
2,711,385.40
269,436.91
99,821.10
165.40
94,180.20
197.30
ext3 73,167.60
478.11
75,142.20
726.40
58,419.20
680.76
2,275,931.10
69,312.23
72,502.40
464.13
75,054.70
999.82
ext4 109,407.30
142.35
101,720.10
9,752.95
76,935.90
406.46
2,496,937.70
91,639.95
109,115.20
220.57
105,865.70
6,536.20
ext2 110,059.40
262.16
99,688.70
2,284.40
86,990.10
310.92
2,840,720.10
90,951.13
110,258.70
398.58
99,706.00
1,521.29
jfs 41,102.60
307.26
105,831.70
410.71
93,563.50
626.75
2,923,355.60
71,123.01
40,428.70
151.28
104,266.00
600.51
xfs 113,051.40
686.60
114,455.00
609.82
94,614.60
904.96
2,736,998.50
71,863.30
113,448.50
356.05
114,556.70
134.01
btrfs 110,230.30
638.29
110,334.10
603.85
98,633.60
539.74
2,223,944.30
419,578.36
98,449.00
517.37
104,320.90
539.80
reiserfs 106,766.00
1,969.39
111,738.20
410.76
102,332.80
4,301.85
2,627,223.20
86,466.86
107,329.30
912.46
111,707.40
342.10


Table 4 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 4MB.

Table 4 – IOzone Read Performance Results with a Record Length of 4MB and a File Size of 16GB

File System Read
(KB/s)
Re-read
(KB/s)
Random read
(KB/s)
Backwards read
(KB/s)
Strided read
(KB/s)
fread
(KB/s)
freread
(KB/s)
Reiser4 106,758.60
63.64
106,772.40
63.60
95,476.80
63.67
99,972.00
162.45
88,978.80
109.89
106,834.40
34.88
106,828.60
74.42
ext3 72,267.70
3,011.67
73,857.90
4,154.39
71,447.60
122.82
82,327.20
195.91
81,040.40
221.83
72,087.80
3,449.44
71,736.10
3,411.92
ext4 97,929.80
43.56
98,041.80
44.83
86,095.00
108.35
102,674.20
88.16
88,711.90
113.00
98,012.70
45.39
98,054.10
37.33
ext2 94,367.70
5,209.01
98,928.40
7,427.57
88,569.70
310.92
106,502.00
161.75
100,847.50
243.85
91,433.50
51,97.23
88,325.40
3,614.90
jfs 106,739.80
706.54
106,771.50
703.42
95,810.40
1,344.66
107,407.80
245.18
96,022.50
1,040.32
106,958.30
69.79
107,021.70
70.26
xfs 106,246.60
110.12
106,595.60
160.98
85,248.00
158.14
106,079.10
328.76
91,622.00
148.41
106,563.40
66.54
106,998.70
83.28
btrfs 105,535.20
194.36
105,753.80
292.67
87,351.90
640.83
87,047.80
888.18
87,179.80
790.91
101,232.60
262.23
101,621.40
234.68
reiserfs 93,466.10
1,186.53
95,317.70
768.62
87,430.20
486.30
95,647.10
485.30
88,776.80
994.89
93,542.60
1,353.85
95,651.70
581.12

The next two tables of results are for the 8MB record size. Table 5 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 5 – IOzone Write Performance Results with a Record Length of 8MB and a File Size of 16GB

File System Write
KB/s
Re-write
(KB/s)
Random write
(KB/s)
Record re-write
(KB/s)
fwrite
(KB/s)
frewrite
(KB/s)
Reiser4 100,374.50
202.35
94,419.50
189.81
74,282.60
483.86
2,435,265.10
74,257.84
99,884.00
158.36
94,109.20
191.79
ext3 73,112.90
298.23
75,521.10
500.19
60,274.30
294.63
1,402,615.20
38,937.18
72,796.30
412.19
75,159.00
651.18
ext4 109,327.10
171.36
104,004.90
8,327.54
79,617.30
647.18
1,532,668.10
33,213.47
109,296.60
95.56
107,530.10
1,743.57
ext2 110,225.00
290.69
100,773.20
1,229.66
89,755.40
324.99
1,630,790.80
41,460.64
110,287.50
370.52
99,253.60
2,126.92
jfs 40,911.90
264.20
104,983.90
425.36
100,718.30
515.33
1,698,199.00
30,143.23
40,528.50
268.13
105,610.10
379.17
xfs 113,075.40
382.02
114,553.90
169.71
98,860.70
814.56
1,637,958.70
32,970.06
114,542.30
425.29
114,542.30
197.63
btrfs 106,660.80
381.24
108,358.30
838.67
94,798.90
903.87
1,626,021.50
47,664.79
101,264.90
196.78
101,432.50
201.40
reiserfs 107,640.90
1,198.96
111,642.40
424.64
105,762.60
2,581.85
1,634,817,80
40,739.76
107,392.60
1,256.77
111,940.00
189.59

Table 6 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 8MB.

Table 6 – IOzone Read Performance Results with a Record Length of 8MB and a File Size of 16GB

File System Read
(KB/s)
Re-read
(KB/s)
Random read
(KB/s)
Backwards read
(KB/s)
Strided read
(KB/s)
fread
(KB/s)
freread
(KB/s)
Reiser4 106,770.60
63.29
106,837.40
59.29
106,224.90
98.67
107,486.60
126.50
101,737.00
183.12
106,792.90
52.36
106,862.30
26.13
ext3 74,145.80
4,309.27
77,749.70
3,741.80
76,112.30
66.45
81,136.90
221.88
83,584.60
147.11
71,949.30
2,636.84
72,275.10
3,634.02
ext4 97,985.40
40.32
98,031.20
55.91
98,299.90
195.36
108,635.40
159.98
100,060.90
128.50
97,968.30
43.66
98,053.30
18.49
ext2 94,353.90
6,807.06
100,562.10
7,061.81
95,477.40
320.77
102,138.40
152.15
104,958.50
232.54
90,254.40
4,507.88
87,060.20
83.14
jfs 106,227.90
95.68
106,571.70
153.71
99,805.30
154.93
113,392.80
72.09
102,055.00
87.36
106,453.70
143.17
106,956.90
68.83
xfs 106,965.00
52.79
107,012.80
113.72
107,984.80
146.37
114,795.70
107.12
110,353.50
102.99
106,963.20
49.48
107,043.60
55.45
btrfs 104,272.20
497.24
104,543.30
541.43
84,625.70
612.39
95,962.20
982.57
89.078.40
1,034.25
101,264.90
196.78
101,432.50
201.40
reiserfs 93,507.60
835.92
95,664.10
590.81
96,438.70
479.78
101,773.50
847.89
98,437.40
513.42
93,781.80
1,396.35
95,851.90
610.12

The final two tables of results are for the 16MB record size. Table 7 below presents the throughput in KB/s for the file systems for the 6 write tests.

Table 7 – IOzone Write Performance Results with a Record Length of 16MB and a File Size of 16GB

File System Write
KB/s
Re-write
(KB/s)
Random write
(KB/s)
Record re-write
(KB/s)
fwrite
(KB/s)
frewrite
(KB/s)
Reiser4 100,590.20
784.99
94,398.70
232.84
78,827.60
460.14
2,346,101.20
38,954.92
99,920.00
192.24
94,045.80
393.75
ext3 72,999.50
397.39
75,406.00
452.57
62,060.20
457.09
1,343,648.10
30,995.74
72,988.20
673.37
74,796.40
650.74
ext4 109,348.90
141.87
103,807.50
8,477.20
81,682.40
1,304.42
1,505,397.20
41,948.58
109,129.30
201.36
101,880.90
9,679.37
ext2 110,259.20
423.93
99,366.20
4,213.22
89,199.20
3,633.16
1,601,081.40
31,263.97
110,480.90
454.10
99,639.60
1,105.63
jfs 40,684.50
426.70
106,511.50
2,794.47
104,702.90
758.06
1,661,039.50
24,962.35
40,579.20
270.64
106,951.50
116.81
xfs 113,341.40
347.11
111,645.40
174.08
105,119.00
763.83
1,602,714.00
28,390.65
106,979.80
52.71
107,076.00
27.38
btrfs 108,148.30
703.04
109,098.00
373.92
99,521.70
444.04
1,553,258.50
22,812.04
103,457.40
823.76
105,485.90
623.77
reiserfs 107,323.70
1,508.79
111,468.60
785.59
108,354.50
1,198.54
1,577,681.20
21,635.30
107,238.60
1,225.59
111,754.00
153.40

Table 8 below presents the throughput in KB/s for the file systems for the 7 read tests for a record length of 16MB.

Table 8 – IOzone Read Performance Results with a Record Length of 16MB and a File Size of 16GB

File System Read
(KB/s)
Re-read
(KB/s)
Random read
(KB/s)
Backwards read
(KB/s)
Strided read
(KB/s)
fread
(KB/s)
freread
(KB/s)
Reiser4 106,780.70
39.83
106,821.40
44.57
112,629.90
67.25
112,646.50
178.95
110,757.00
149.74
106,736.50
66.59
106,850.50
16.02
ext3 74,781.80
4,466.66
76,942.70
4,478.32
78,993.80
83.48
81,133.80
114.28
84,995.70
266.05
73,546.40
3,656.29
71,958.20
3,442.78
ext4 97,943.60
31.41
98,040.20
39.20
105,651.60
151.52
111,708.70
219.17
106,597.60
127.14
97,944.60
53.76
98,053.00
22.69
ext2 92,628.70
3,691.33
103,561.10
5,073.18
99,302.60
216.93
101,995.80
1,227.96
107,690.30
692.39
92,169.30
5,900.54
87,643.90
1,250.59
jfs 106,361.80
287.72
106,690.40
220.64
108,949.50
106.43
117,216.30
113.33
109,621.60
106.51
106,557.90
116.81
107,022.50
42.67
xfs 106,939.60
65.73
107,022.00
56.59
114,172.40
88.58
118,130.60
115.89
117,923.00
78.56
106,979.80
52.71
107,076.00
27.38
btrfs 104,048.10
370.22
104,312.20
360.24
97,223.50
444.04
102,410.40
685.46
97,855.20
882.64
102,014.20
537.21
102,130.70
643.75
reiserfs 93,702.50
1,320.49
95,530.40
895.24
101,546.50
930.53
104,815.80
1,125.34
103,938.60
765.59
93,088.20
1,171.06
95,671.60
745.50

There are a large number of tables with a great deal of data. These tables allow you to compare file systems but it may be a bit more difficult to see the trends as a function of record length. So the data in tables 1-8 can be tabulated for each file system. Then the performance for each test is tabulated as a function of the record size.

In general all of the file systems but btrfs have the same basic trend. Btrfs is slightly different. So only the data for reiser4, which is representative of all the file systems, and btrfs and are contained in the tables below. Table 9 contains the write throughput in KB/s for the 6 write tests for the various record lengths for reiser4.

Table 9 – IOzone Write Performance Results for Reiser4 with Various Record Lengths with a File Size of 16GB

Record Size Write
(KB/s)
Re-write
(KB/s)
Random write
(KB/s)
Record re-write
(KB/s)
fwrite
(KB/s)
frewrite
(KB/s)
1 MB 100,476.50 95,290.90 61,980.50 3,258,345.00 100,095.70 94,258.60
4 MB 100,360.50 94,389.00 71,415.40 2,711,395.40 99,821.10 94,180.20
8 MB 100,374.50 94,419.50 74,282.60 2,435,265.10 99,884.00 94,109.20
16 MB 100,590.20 94,398.70 78,827.60 2,346,101.20 99,920.00 94,045.80

Table 10 contains the write throughput results in KB/s for btrfs for the 6 write tests for various record lengths.

Table 10 – IOzone Write Performance Results for Btrfs with Various Record Lengths with a File Size of 16GB

Record Size Write
KB/s
Re-write
(KB/s)
Random write
(KB/s)
Record re-write
(KB/s)
fwrite
(KB/s)
frewrite
(KB/s)
1 MB 105,828.00 108,022.80 107,099.00 3,309,137.50 98,466.10 101,230.60
4 MB 110,230.30 110,334.10 98,633.60 2,223,944.30 98,449.00 104,320.90
8 MB 106,660.80 108,258.30 94,798.90 1,626,021.50 101,340.80 104,148.90
16 MB 108,148.30 109,098.00 99,521.70 1,553,258.50 103,457.40 105,485.90

Table 11 contains the read throughput results in KB/s for Reiser4 for the 7 read tests for various record lengths.

Table 11 – IOzone Read Performance Results for Reiser4 with Various Record Lengths with a File Size of 16GB

Record Size Read
(KB/s)
Re-read
(KB/s)
Random read
(KB/s)
Backwards read
(KB/s)
Strided read
(KB/s)
fread
(KB/s)
freread
(KB/s)
1 MB 106,777.20 106,861.50 61,980.50 62,220.00 49,071.90 102,014.20 102,130.70
4 MB 106,758.60 104,312.20 95,7476.80 71,415.40 88,978.80 106,834.40 106,828.60
8 MB 106,770.60 106,837.40 106,224.90 74,282.60 101,737.00 106,792.90 106,862.30
16 MB 106,780.70 106,821.40 112,629.90 78.827.60 110,757.00 106,736.50 106,850.50

Table 12 contains the read throughput results in KB/s for Btrfs for the 6 read tests for various record lengths.

Table 12 – IOzone Read Performance Results for Btrfs with Various Record Lengths with a File Size of 16GB

Record Size Read
(KB/s)
Re-read
(KB/s)
Random read
(KB/s)
Backwards read
(KB/s)
Strided read
(KB/s)
fread
(KB/s)
freread
(KB/s)
1 MB 102,851.30 103,123.10 56,769.90 56,224.70 57,826.10 99,655.80 99,914.50
4 MB 105,535.20 105,753.80 87,351.90 87,047.80 89,078.40 101,232.60 101,621.40
8 MB 104,272.20 104,543.30 84,625.70 95,962.20 89,078.40 101,264.90 101,432.50
16 MB 104,048.10 104,312.20 97,233.50 102,410.40 97,855.20 102,014.20 103,130.70

For most of the tests, btrfs has the same trends as for the other file systems. For example, the trends for all of the read performance tests as a function of the record size is the same for reiser4 and btrfs. However, the random write performance test is a bit different. For reiser4 as the record size increases, the random write performance improves. This is to be expected since there are fewer writes and they increase in size as the record size increases. However, for btrfs, the trend is the opposite. As the record size increases, the random write performance of btrfs decreases. It’s not a large decrease, approximately 7.5%, but it is much more noticeable when reiser4 increases by approximately 28%. At this time, there is no reason for this trend in btrfs. But recall that btrfs is still under heavy development.

Observations

The observations have not changed too much with the addition of the other file systems. Most of the overall observations remain pretty much the same:


  • Write and re-write performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  • The random write performance gets better as the record size is increased from 1MB to 16MB. For example, jfs went from 53,182.10 KB/s to 108,949.50 KB/s, the greatest increase of all the file systems. However, the random write performance for btrfs remained about the same as the record size is increased.
  • Record re-write performance decreased as the record size increased. This performance can be driven by various cache and buffer sizes. Perhaps as the size increases, some cache size are exceeded and performance decreases.
  • Fwrite and frewrite performance are relatively insensitive to record size over the range of record sizes tested for the parameters of this test for all of the file systems tested.
  • Read and re-read performance are relatively insensitive to record size over the range tested (1MB to 16MB) for the parameters of this test.
  • Random read performance improved dramatically as the record increased. For example at 1MB the random read performance for ext4 was 52,620.10 KB/s and at a record size of 16MB it was 105,651.60 KB/s. This is about a 100% increase in throughput performance! An increase in performance is expected because there are fewer records, but the change in performance is fairly dramatic.
  • Backward read performance also dramatically improved with increasing record size. This is to be expected since there is less drive head movement because there are fewer total records.
  • Strided read throughput performance also dramatically improved as the record size was increased. For reiser4 the strided read performance increased from 49,071.90 KB/s at 1MB record size to 110,757.00 KB/s at 16MB.
  • Fread and freread are relatively insensitive to a change in record size.

We are all human so it’s rather difficult to resist the temptation to compare the file systems. For the most part the file systems are tested with their default creation and mount options (ext3 and ext4 are mounted with “data=ordered”). So the above results may not show the file system in their best light. However, it may also reflect what many admins do – use the default options because they may not know the interaction between the options and the performance. At least many admins start with the default options and tune them based on testing. Therefore, drawing any conclusions from these test results about which file system is “superior” to the others is impossible. However, they can be contrasted.

There is quite a bit of data to be examined, but here are some observations:


  • For write performance, most file systems are pretty good with the exception of ext3 and jfs. Jfs is by far the worst performing file system on this test compared to the top performing by about a factor of 2.5. Xfs is about the best, very closely followed by btrfs and reiserfs and then ext4, ext2, and reiser4.
  • For rewrite performance, ext3 is the worst performing file system while jfs has about the best performance over the range of record sizes. Xfs, ext4, and reiserfs were close to the performance of jfs, but reiser4 and btrfs are not far behind.
  • Reiserfs has about the best random write performance followed by btrfs, particularly at the smaller record sizes. At the larger record sizes, xfs and jfs have similar performance to btrfs. On the other hand, ext3, ext4, and reiser4 have about the worst performance.
  • For record re-write, reiser4 is by far the best, particular at larger record sizes. The other file systems have about the same level of performance.
  • For fwrite and frewrite, jfs is the worst performer by far – about a factor of 2. Ext3 is second from the bottom, fairly far behind the other file systems. The others are near or at the top in terms of performance but at smaller record sizes, xfs is the better performer.
  • For read and re-read performance the ext file systems are behind the other file systems. Also, reiserfs has about the same performance as the ext file systems as well. Reiser4, jfs, xfs, and btrfs are the top performers and have about the same performance.
  • For random read, ext3 has the worst performance – far less than the others. The others have somewhat similar performance to each other which is much better than ext3. Over the range of record sizes xfs seems to have the best performance, but at various record sizes, jfs, ext2, btrfs, reiserfs, and reiser4 are very close to xfs.
  • For backwards read, ext3 is the slowest followed by btrfs. On the other hand, xfs and jfs have about the best performance but the others aren’t too far behind. In general reiserfs is just behind the others, but at the 1 MB record size it has the best performance by far.
  • In the case of strided read, ext3 is the slowest by a reasonable margin. Near the top are jfs and xfs.
  • For fread, ext3 has the worst performance but ext2 is fairly close. Xfs, jfs, and reiser4 are about the best performers but btrfs and reiser4 are very close. Ext4, reiserfs, and ext2 are a little further back.

Based on these observations I think it’s difficult to declare an overall “winner”. Several of the file systems perform very well over the range of tests and others don’t perform well in certain tests.

Next Time

One file system that is missing from this article is nilfs2. Recall that nilfs2 is a log-structured file system which is different in design than the others. It also has the feature of creating checkpoints after each sync system call. IOzone does a great deal of syncs during it’s tests. Consequently, nilfs2 is a bit more difficult to compare to the other file systems.

In an upcoming article we’ll test nilfs and explain the testing methodology using IOzone. It should be interesting so keep an eye out for this article.

Comments on "IOzone Performance Exploration, Part 2: The Rest of the Crowd (Almost)"

duncan

Record lengths of 1-16 MB? I think it would be a little more realistic to test with record lengths of 1-16KB. Does anyone really need to keep 1,048,576 bytes of info on a customer? Invoice? etc? I guess this test was for a home computer used for playing games.

laytonjb

@duncan,

Please read the first article – it explains why I chose the record lengths.

Jeff

mxcreep

I\’ve been playing with IOZone for some time and noticed the performance differences for different record sizes. But I don\’t know how to map these results to real life situations, what I mean is how to determine the record size applications use when doing disk io ? That info is needed when choosing a specific filesystem for specific tasks.

laytonjb

@mxcreep,

Ahh… grasshopper, welcome to the fray. The question you ask is the perfect one – how are benchmarks relevant to my workload. The answer is that – it ain\’t easy and usually impossible to determine that.

One of the projects I\’ve been working on is a way to do IO profiling of your application (look at some of the early columns I wrote – it\’s called strace_analyzer). From the output of this tool you can get an _idea_ of whether your app is IOPS driven or perhaps throughput driven. It also tells you the size of the records during reads and writes.

So \”theoretically\” with this information you can look at benchmarks that sort of mimic that behavior.

What I\’m working on next is to take the strace information and actually create a pseudo-application that just mimics the IO behavior. Then you have a real benchmark – your application. (I can\’t solve the problem of test hardware though – I\’m trying to solve that problem for me personally :) ).

Jeff

richarson

Hello, Jeff

Thank you for this series of articles, they\’ve been very enlightening.

One suggestion to possibly avoid memory cache effects and surely reduce the time needed for the tests could be using the \”mem=\” kernel parameter so you don\’t have to pull the DIMMs out.

That way you might be able to use a smaller file and smaller record sizes but still get relevant numbers from the benchmark.

laytonjb

@richardson,

Thanks for the suggestion. It definitely would help things :) These benchmarks take a great deal of time to run :)

Thanks!

Jeff

Hi Jeff, Its really great results you published. It helped me in understanding IOZone and tests to run.

Do you any idea, how do I use IOZone on a customized file system, where there are specific APIs exists. Like xxx_open, xxx_close, xxx_read, xxx_write …

Is porting iozone is the only solution to support new FS? Does anybody ported already?

regards
Prasad

Hi Jeff,

I’ve been using Iozone benchmark to analyse disk performance on virtualized environments. I have a very basic doubt, what exactly is the significance of record size parameter in IOzone? As far as I understand, the actual file write commands (system calls) will be in terms of the file system block size which is usually between 1K to 4K on most linux systems, so how does record size affect the performance ? And why does iozone impose a limit of 16M on the max record size saying that this is the max buffer cache size although buffer cache size could be more than that depending on the system memory parameters…

Really appreciate you sharing this blog article.Really thank you! Will read on…

Does your site have a contact page? I’m having problems locating it but, I’d like to shoot you an e-mail. I’ve got some recommendations for your blog you might be interested in hearing. Either way, great blog and I look forward to seeing it develop over time.

Hplrwu Normally I really do not study post on blogs, but I must say until this write-up really forced me to try and do thus! Your creating style continues to be amazed us. Thanks, very wonderful post.

“I value the blog article.Really thank you! Want more.”

“I’m not sure exactly why but this web site is loading very slow for me. Is anyone else having this issue or is it a problem on my end? I’ll check back later and see if the problem still exists.”

My brother suggested I would possibly like this website.
He was once totally right. This submit actually made my day.

You can not believe simply how much time I had spent for this info!
Thanks!

This post have resolved my problem,thank you very much and hope you writting more good articles.
Wholesale Oakley sunglasses polished black fire iridium seize the deal outlet http://www.fleetsale.ru/new-arrival-oakleys-780.html

I like reading a post that may make women and men think.
Also, thanks a lot for allowing me to comment!

Here is my blog OnaSDuerst

What’s up, the whole thing is going fine here and ofcourse every
one is sharing facts, that’s really fine, keep up writing.

My web page – ConradJEagan

Today, I went to the beach with my kids. I found a sea shell and gave it to my 4 year old daughter
and said “You can hear the ocean if you put this to your ear.” She placed the shell to her ear and screamed.
There was a hermit crab inside and it pinched her ear. She never wants to go back!
LoL I know this is completely off topic but I had to tell someone!

Also visit my page – AmalOStotler

I’ll immediately snatch your feed as I can’t find your e-mail subscription hyperlink or e-newsletter service.
Do you possess any? Please allow me understand in order that I may
subscribe. Thanks.

Also visit my blog WernerJGrosz

Wow, this article is pleasant, my younger sister is analyzing such things,
so I am going to inform her.

Look at my page – AntonXDrakes

Leave a Reply