Recently we saw that the journal device location, unfortunately, didn't make much of a difference on ext4 metadata performance. But can the size of the journal will have an impact on metadata performance? The first in a series of articles examining the journal size and performance.
Appendix – Table of Results
The first combination tested was for small files (4 KiB) with a shallow directory structure. Table 1 below lists the results with an average value and just below it, in red, is the standard deviation.
Table 1 – Benchmark Times Small Files (4 KiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
28.40
1.28 |
346.70
2.53 |
70.90
0.94 |
6.80
3.89 |
| 64MB |
13.70
1.10 |
338.40
5.22 |
65.80
2.40 |
4.10
1.76 |
| 256MB |
10.0
0.45 |
330.80
0.98 |
53.80
1.17 |
15.70
0.64 |
| 1GB |
10.20
1.33 |
330.70
1.27 |
53.40
0.92 |
2.70
2.83 |
Table 2 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 2 – Performance Results of Small Files (4 KiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
296.50
13.30 |
971.10
7.06 |
3,885.90
28.73 |
4,751.50
63.66 |
1,529.50
513.69 |
| 64MB |
617.90
47.78 |
995.70
15.32 |
3,982.10
61.59 |
4,665.50
1,399.38 |
2,554.00
1,200.46 |
| 256MB |
843.60
38.15 |
1,017.60
2.94 |
4,072.50
11.94 |
6,263.40
134.59 |
6,315.60
2,681.55 |
| 1GB |
837.90
99.13 |
1,017.90
3.81 |
4,073.70
15.44 |
6,309.10
108.52 |
5,698.00
2,928.20 |
The next test was for small files (4 KiB) with a deep directory structure. Table 3 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.
Table 3 – Benchmark Times Small Files (4 KiB) – Deep Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
324.50
3.17 |
633.30
7.09 |
330.60
2.15 |
214.40
1.36 |
| 64MB |
218.60
7.31 |
500.50
8.86 |
224.70
5.92 |
75.60
4.27 |
| 256MB |
148.10
3.45 |
426.00
15.49 |
169.80
8.94 |
54.20
3.19 |
| 1GB |
127.60
11.09 |
406.20
11.68 |
147.40
13.17 |
47.80
10.03 |
Table 4 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 4 – Performance Results of Small Files (4 KiB) – Deep Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
272.50
2.80 |
559.00
6.36 |
2,237.70
25.28 |
1,071.30
6.96 |
412.40
2.42 |
| 64MB |
405.00
13.54 |
707.50
12.70 |
2,831.80
50.11 |
1,577.20
41.77 |
1,175.10
67.83 |
| 256MB |
597.90
14.59 |
832.10
30.30 |
3,330.60
121.08 |
2,091.90
110.24 |
1,639.70
102.91 |
| 1GB |
698.90
57.52 |
872.40
23.33 |
3,491.10
93.44 |
2,420.10
191.32 |
1,924.00
345.24 |
The third combination tested was for medium files (4 MiB) with a shallow directory structure. Table 5 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.
Table 5 – Benchmark Times Medium Files (4 MiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
0.20
0.40 |
154.40
3.41 |
12.20
3.49 |
0.10
0.30 |
| 64MB |
0.30
0.46 |
154.10
3.27 |
11.40
2.69 |
0.00
0.00 |
| 256MB |
0.50
0.67 |
156.80
2.09 |
9.40
3.01 |
0.10
0.30 |
| 1GB |
0.30
0.46 |
154.20
1.54 |
11.40
3.56 |
0.20
0.40 |
Table 6 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 6 – Performance Results of Medium Files (4 MiB) – Shallow Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
61.40
122.80 |
19.40
0.66 |
79,570.40
1,682.43 |
271.60
72.43 |
30.70
92.10 |
| 64MB |
92.10
140.69 |
19.50
0.67 |
79,722.90
1,644.04 |
282.50
58.21 |
0.00
0.00 |
| 256MB |
107.30
138.10 |
19.10
0.30 |
78,329.70
1,050.68 |
386.60
219.08 |
.00
0.00 |
| 1GB |
92.10
140.69 |
19.30
0.46 |
79,644.20
784.75 |
305.00
123.77 |
61.40
122.80 |
The fourth and final combination tested was for medium files (4 MiB) with a deep directory structure. Table 7 below lists the benchmark times with an average value and just below it, in red, is the standard deviation.
Table 7 – Benchmark Times Medium Files (4 MiB) – Deep Directory Structure
| Journal Size |
Directory Create
(secs.) |
File Create
(secs.) |
File Remove
(secs.) |
Directory Remove
(secs.) |
| 16MB |
4.20
0.60 |
225.90
1.58 |
15.30
2.69 |
1.50
0.50 |
| 64MB |
3.00
0.00 |
220.30
1.55 |
14.10
3.51 |
1.80
1.25 |
| 256MB |
2.80
1.47 |
220.90
0.94 |
14.20
3.71 |
3.30
3.16 |
| 1GB |
2.90
0.83 |
220.40
1.28 |
14.30
3.00 |
2.50
1.20 |
Table 8 below lists the performance results with an average value and just below it, in red, is the standard deviation.
Table 8 – Results of Medium Files (4 MiB) – Deep Directory Structure
| Journal Size |
Directory Create
(Dirs/sec) |
File Create
(Files/sec) |
File Create
(KiB/sec) |
File Remove
(Files/sec) |
Directory Remove
(Dirs/sec) |
| 16MB |
497.50
76.57 |
17.80
0.40 |
72,495.30
507.54 |
225.90
1.58 |
1,535.00
512.00 |
| 64MB |
682.10
0.30 |
18.00
0.00 |
74,338.20
523.04 |
311.40
90.36 |
1,541.90
639.70 |
| 256MB |
847.60
239.91 |
18.20
0.40 |
74,133.90
317.48 |
309.60
85.20 |
1,210.80
697.18 |
| 1GB |
767.10
219.19 |
18.10
0.30 |
74,303.30
430.36 |
300.10
70.44 |
1,108.40
639.67 |
Jeff Layton is an Enterprise Technologist for HPC at Dell. He can be found lounging around at a nearby Frys enjoying the coffee and waiting for sales (but never during working hours).
Comments on "Size Can Matter: Improving Metadata Performance with Ext4 Journal Sizing – Part I"
I dont get the 256MB recommendation. Am I reading the results wrong? It seems that there is no negative to going 1GB, other than the 744MB extra storage. These days, with 1TB going for < $100, looks like its worth the space.
Well it\’s not really a recommendation but if someone held a gun to my head and said, \”pick a journal size,\” I would pick 256MB. The simple reason is that I don\’t want to waste the space. But as you point out it doesn\’t hurt at all to go to 1GB for the configurations tested.
I guess I\’m old enough where I compress data that isn\’t being actively used and I try to reduce the number of variables in my code to save space, etc. Really what I\’m saying that is I\’m old and I remember when we did everything we could to save space. Unfortunately, old habits die hard :)
Jeff
man man mkfs.ext4 shows maximum journal size of 102400 fs blocks (whatever size block has). Do you have any evidence that 1GB of size was really used or only 400MB of the partition?
My man page says 10,240,000 _fs_ blocks (ususally of 4KB each), which would be 40 GB.