SandForce 1222 SSD Testing, Part 5: Detailed Throughput and IOPS Analysis with a 2.6.38.2 Kernel

While you can go back and read the first four articles, here, here, here, and here, to read the details on the testing process, to make this article a little more complete the testing process is briefly reviewed.

For the throughput testing, this article uses IOzone as the benchmarking tool. Recall that IOzone is one of the most popular throughput benchmarks. It’s open-source and is written in very plain ANSI C (not an insult but a compliment). It is capable of single thread, multi-threaded, and multi-client testing. The basic concept of IOzone is to break up a file of a given size into records. Records are written or read in some fashion until the file size is reached. Using this concept, IOzone has a number of tests that can be performed. The thirteen tests listed below, are performed to measure the throughput capability of the drives.

• Write
  
• Re-write
  
• Read
  
• Re-read
  
• Random Read
  
• Random Write
  
• Backwards Read
  
• Record Rewrite
  
• Strided Read
  
• Fwrite
  
• Refwrite
  
• Fread
  
• Refread
  

You can refer to the first article for more details on these tests.

For the IOPS testing, IOzone was also used. The four tests listed below, are performed to measure the IOPS capability of the drives.

• Write IOPS
  
• Read IOPS
  
• Random Read IOPS
  
• Random Write IOPS
  

You can refer to the first IOPS article for more details on these tests.

The tests were run on the same system as previous tests but as mentioned previously a 2.6.38.2 kernel (the most recent kernel when the tests were performed) was used. The highlights of the system are:

• CentOS 5.4 with either a 2.6.32 kernel or a 2.6.38.2 kernel
  
• GigaByte MAA78GM-US2H motherboard
  
• An AMD Phenom II X4 920 CPU
  
• 8GB of memory (DDR2-800)
  
• The OS and boot drive are on an IBM DTLA-307020 (20GB drive at Ultra ATA/100)
  
• /home is on a Seagate ST1360827AS
  
• Micro Center SandForce 1222, 64GB SSD. This is mounted as /dev/sdd
  
• Additionally, a 64GB Intel X-25E SSD is used, courtesy of Intel. This is mounted as /dev/sdd.
  
• ext4 is used as the file system with the default options (Note: To get TRIM capability, the “discard” mount option has to be used).
  

For the throughput testing five record sizes were used: (1) 64KB, (2)1MB, (3) 4MB, (4) 8MB, and (5) 16MB. The command line for the first record size (64KB) is,

./IOzone -Rb spreadsheet_output_64K.wks -s 16G -+w 98 -+y 98 -+C 98 -r 64K > output_64K.txt

The command line for the second record size (1MB) is,

./IOzone -Rb spreadsheet_output_1M.wks -s 16G -+w 98 -+y 98 -+C 98 -r 1M > output_1M.txt

The command line for the third record size (4MB) is,

./IOzone -Rb spreadsheet_output_4M.wks -s 16G -+w 98 -+y 98 -+C 98 -r 4M > output_4M.txt

The command line for the fourth record size (8MB) is,

./IOzone -Rb spreadsheet_output_8M.wks -s 16G -+w 98 -+y 98 -+C 98 -r 8M > output_8M.txt

The command line for the fifth record size (16MB) is,

./IOzone -Rb spreadsheet_output_16M.wks -s 16G -+w 98 -+y 98 -+C 98 -r 16M > output_16M.txt

The options “-+w”, “-+y” and “-+C” define the how much “dedupability” there is in the data and buffers. In the above command lines “98″ means that the data can be deduplicated (compressed) 98%. Note that these options are only available with a later version of IOzone.

For the IOPS testing, only one record size was used corresponding to the most common size used for testing IOPS – 4KB. The command line for this record size is,

./iozone -Rb spreadsheet_output_4K.wks -O -i 0 -i 1 -i 2 -e -+n -+w 98 -+y 98 -+C 98 -r 4K -s 16G > output_4K.txt

The options “-+w”, “-+y” and “-+C” define the how much “dedupability” there is in the data and buffers. In the above command lines “98″ means that the data can be deduplicated (compressed) 98%. Note that these options are only available with a later version of IOzone.

For this article, six compressibility levels: (1) 98%, (2) 75%, (3) 50%, (4) 25%, (5) 2%, and (6) random, were used. The last compressibility level is one that allows IOzone to randomly pick the compressibility level of the data, presumably randomly, throughput the test run. That option is “-+d” and the “-+w”, “-+y” and “-+C” are dropped from the command line.

Results

Presenting the results in a clear easy to understand manner has proven to be challenging. The results could have been presented in side-by-side charts, one for the 2.6.32 kernel, and one for the 2.6.38.2 kernel, but that data was so small that it was difficult to discern which bar corresponded to which record size and compressibility level. Consequently, it was decided to present the results as a “percentage difference” between the performance for the 2.6.32 kernel and the 2.6.38.2 kernel since the intent of the testing was to examine any performance changes.

All the subsequent plots use the percent difference in performance between the 2.6.38.2 kernel and the 2.6.32 kernel. More specifically, the difference in the average performance is reported. A positive number (> 0) indicates that the average performance over the 10 tests is faster for the 2.6.38.2 relative to the 2.6.32 kernel (the performance difference is normalized by the 2.6.32 average performance results). And obviously, if a negaive number indicates that the average performance over the 10 tests is slower with the 2.6.38.2 kernel relative to the 2.6.32 kernel.

In addition to reporting the difference in the average performance, the percentage standard deviation is also plotted as error bars. The percent variation in performance is computed for the 2.6.38.2 kernel and is used as the “error” or standard deviation for the percent difference in average performance. The intent for displaying the standard deviation is that when examining the performance difference between two performance results, the standard deviation can tell you if the difference is statistically meaningful. So, for these plots, if the standard deviation crosses the zero axis (positive or negative), then the difference is not statistically meaningful. The reason this is true is that the standard deviation is greater than the performance difference so the results are not statistically meaningful.

This article will start with the throughput performance followed by the IOPS performance. There will be a number of plots so hang in there! (you should be congratulated for making it this far).

Figure 1 below is the performance difference for the sequential write test for the five record sizes for the six levels of dedupability (compressibility) for the SandForce SSD for the two kernels (2.6.32 and 2.6.38.2). Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 1: Average Write Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

Notice that in Figure 1, at 75% compressibility and a record size of 64KB, that the error bar does not cross the zero axis. This means that the performance difference for this test is acceptable (i.e. the standard deviation is not greater than the performance difference). On the other hand, if you look at the “RANDOM” compressibility and a 4MB record size you will see that the standard deviation (the error bars) is much greater than the performance difference indicating that the performance change is not statistically meaningful (i.e. one cannot say that the performance difference is actually “real” or is within the standard deviation of the tests).

In examining the data in Figure 1, a few observations stand out. The first one (IMHO) is that except for the 98% compressible case, as the record size increases, the performance of the 2.6.38.2 kernel is slightly worse than the 2.6.32 kernel although some of the differences are statistically meaningless (i.e. the difference is within the standard deviation). Table 1 below lists the changes that appear to have more improvement in performance (2.6.38.2 is better than 2.6.32) and worst changea in performance (2.6.32 is better than 2.6.38.2).

Table 1 – Sequential Write Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 2 below is the performance difference for the sequential re-write test for the five record sizes for the six levels of dedupability (compressibility) for the SandForce SSD for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 2: Average Re-write Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 2, one obvious thing stands out – overall the sequential rewrite performance went down for the 2.6.38.2 kernel relative to the 2.6.32 kernel and they are really no cases where it improved (except one where the improvement was minor). Table 2 below highlights some of the more significant changes.

Table 2 – Sequential Rewrite Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 3 below is the performance difference for the random write test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 3: Average Random Write Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 3, it appears that there are a couple of good performance gains but in general random write throughput performance decreases for the 2.6.38.2 kernel relative to the 2.6.32 kernel, but the decreases are fairly small. Table 3 below highlights some of the more significant changes.

Table 3 – Random Write Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 4 below is the performance difference for the record rewrite test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 4: Average Record Rewrite Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 4, one of the more obvious trends is that the record rewrite performance improved when moving from the 2.6.32 to the 2.6.38.2 kernels. However, the standard deviations are fairly large so it’s difficult to say that the differences are statistically meaningful. But there are some notable performance differences as highlighted below in Table 4.

Table 4 – Record rewrite Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 5 below is the performance difference for the fwrite test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 5: Average Fwrite Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 5, it appears that there is not a general trend and perhaps not much change because of the large standard deviations making much of the performance change statistically insignificant. However there are some small gains, particularly with the 64KB record size, and some losses, particularly at the 1MB size as highlighted below in Table 5.

Table 5 – Fwrite Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 6 below is the performance difference for the frewrite test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 6: Average Frewrite Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

Based on the data in Figure 6 the only real general trend is that overall the performance decreased for the 2.6.38.2 kernel relative to the 2.6.32 kernel. But there was one very nice performance improvement but there are several performance decreases as highlighted in Table 6 below.

Table 6 – Frewrite Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 7 below is the performance difference for the sequential read test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 7: Average Read Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 7, it’s fairly apparent that overall there isn’t much change in performance since the differences are small. Moreover, the standard deviations are fairly large so it is difficult to find substantial changes. Table 7 below lists the few reasonable performance changes due to the 2.6.38.2 kernel.

Table 7 – Read Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 8 below is the performance difference for the reread test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 8: Average Reread Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

Much like the read performance differences there is little trend in the reread performance differences. Plus the standard deviations are fairly large making any use of the performance differences difficult. Table 8 below highlights a few changes.

Table 8 – Reread Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 9 below is the performance difference for the random read test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 9: Average Random Read Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 9, it is noticed that the performance changes are fairly small (less than 2%) with many of them being negative changes (i.e. worse performance for the 2.6.38.2 kernel). Table 9 has some of the highlights from Figure 9.

Table 9 – Random Read Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 10 below is the performance difference for the backward read test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 10: Average Backward Read Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

Based on the data in Figure 10 the biggest thing that stands out is that there are some fairly substantial performance decreases for the 2.6.38.2 kernel for the backward read test but some of these decreases have fairly large standard deviations indiciating that they are not statistically significant. Table 10 below highlights some of the more significant changes.

Table 10 – Backward Read Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 11 below is the performance difference for the stride read test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs wi

Figure 11: Average Stride Read Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 11, it is fairly obvious that performance differences are fairly small and there really isn’t a pattern to them. Table 11 below highlights some of the more significant performance changes going from 2.6.32 to 2.6.38.2.

Table 11 – Stride Read Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 12 below is the performance difference for the fread test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 12: Average Fread Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 12, it appears that there really isn’t much of a general trend in the performance differences. The performance differences are small and the standard deviations are fairly large. The more prominent performance differences are listed below in Table 12.

Table 12 – Fread Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 13 below is the performance difference for the freread test for the five record sizes for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 13: Average Freread Throughput Difference between the 2.6.32 and 2.6.28.2 kernels for the Five Record Sizes for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

Much like the fread tests, the performance differences for the 2.6.38.2 and 2.6.32 kernels for the freread test are fairly small with no real discernible trend and fairly large standard deviations. Table 13 below contains the notable performance differences.

Table 13 – Freread Performance Difference Highlights for the 2.6.38.2 and 2.6.32 kernels

Figure 14 below is the performance difference for the sequential write IOPS test for a 4KB record size for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 14: Average Sequential Write IOPS Difference between the 2.6.32 and 2.6.28.2 kernels for a 4KB Record Size for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

It’s fairly obvious from the data, that updating the kernel to 2.6.38.2 from 2.6.32 really improved Sequential Write IOPS performance. The largest improvement in performance was for 98% data compressibility where the average performance improved by 21.02% (+/- 4.50%). The smallest improvement was for the random data compressibility case where the average performance improved by 5.88% (+/- 2.20%). Even for the case where the data is almost incompressible (2% data compressibility), the average performance improved by about 6.5%.

Figure 15 below is the performance difference for the random write IOPS test for a 4KB record size for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 15: Average Random Write IOPS Difference between the 2.6.32 and 2.6.28.2 kernels for a 4KB Record Size for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

Recall from the this article that the random Write IOPS performance was a hallmark of the SandForce based SSD tested so one should not expect much improvement in the random Write IOPS performance with a kernel change. However, the performance did improve by a fairly large amount except at 98% data compressibility.

At 98% data compressibility, the average random write IOPS performance decreased by 3.11% but the standard deviation is larger than the performance difference so it’s difficult to state the performance did actually decrease. At the 75%, 25%, 2%, and random data compressibility levels, the average random Write IOPS performance improved by varying amounts. At the 75% the performance improvement is about 28%, at 25% data compressibility the performance improvement is a little over 15%, and at random data compressibility the performance improvement is almost 16%. You can see the exact changes in Table 16.

Figure 16 below is the performance difference for the sequential read IOPS test for a 4KB record size for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 16: Average Sequential Read IOPS Difference between the 2.6.32 and 2.6.28.2 kernels for a 4KB Record Size for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 16, it’s obvious that only at certain compressibility levels has the performance improved and for other compressibility levels the average performance barely changed. At 75% data compressibility, the sequential Read IOPS improved by 6.94% (+/- 5.20%) which was largest performance change. However even at 25% data compressibility level, where the data is close to being incompressible, the average performance improved by about 6.3%. Finally, for the random data data compressibility level, the average performance improved by about 5.24%.

Figure 17 below is the performance difference for the random read IOPS test for a 4KB record size for the six levels of dedupability (compressibility) for both the SandForce for the two kernels. Note that the “error” bars are the standard deviation for the 10 test runs with the 2.6.38.2 kernel.

Figure 17: Average Random Read IOPS Difference between the 2.6.32 and 2.6.28.2 kernels for a 4KB Record Size for the Six Levels of Dedupability (Compressibility) for the SandForce SSD

In examining the data in Figure 17 the general trend is similar to that for sequential Read IOPS – there are some fairly big improvements but only for certain data compressibility levels. For data compressibility levels of 75%, 25%, and random, the average random Read IOPS improved by a fair amount. The average performance for these data compressibility levels improved in the range of about 12.5% to 14.5% while the other data compressibility showed no real signs of changing.

Summary

The first four articles in this series focused on testing the SandForce based SSD using the 2.6.32 kernel. However, this kernel didn’t necessarily have the features that take advantage of SSDs. So this article repeats all of the tests in the first four articles but with a 2.6.38.2 kernel (the most recent non-rc kernel when the testing was started). Both throughput and IOPS performance were examined using IOzone which allows control over the data compressibility levels.

In general, there were some noticeable changes in performance. Specifically,

• Write Throughput Tests:
  
  
  
  • Sequential write throughput improved somewhat for smaller record sizes
    
  • Sequential rewrite throughput performance decreased pretty much across the board (data compressibility levels and record sizes)
    
  • Random write throughput performance decreased but by a fairly small amount
    
  • Record rewrite throughput performance improved by a fairly sizable amount
    
  • Fwrite throughput performance didn’t show a lot of change
    
  • Frewrite throughput performance generally decreased except at 98% data compressibility (very compressible data)
    
  
  
• Read Throughput Tests:
  
  
  
  • Sequential Read, Reread, and Random Read throughput performance didn’t change too much for the record sizes tested
    
  • Backward read throughput performance decreased by a fair amount, particularly for the smaller record sizes tested
    
  • The stride read, fread, and freread throughput performance didn’t change much with the new kernel.
    
  
  
• IOPS:
  
  
  
  • The Sequential Write IOPS performance increased by quite a bit for the 2.6.38.2 kernel – up to about 22% for the 98% data compressibility case
    
  • The Random Write IOPS performance improved by a large amount for some data compressibility cases. For the 75% data compressibility case, the performance improved by just over 28%!!
    
  • The Sequential Read IOPS performance improved by a reasonable amount but only for certain data compressibility levels – 75%, 25%, and random
    
  • The Random Read IOPS performance exhibited the same general behavior as sequential read IOPS but the levels of improvement were much greater, reaching just over 14% improvement for the 2.6.38.2 kernel for the 75% data compressibility case
    
  
  

Overall the performance improved with the 2.6.38.2 kernel with one or two exceptions. Perhaps most importantly, the overall IOPS performance improved greatly.

I hope you have enjoyed this series on the SandForce SSD testing. I’ve tried to follow good benchmarking practices while also exploring aspects of a really cool technology – real-time data compression in the SandForce SSD controllers.

Final note: While not bragging, doing good testing involves a great deal of work. For this article series, about 36,000 individual tests were run. But no electrons were harmed during the testing (although my power bill may have taken a noticeable hit).

Appendix

This section contains the data from the plots in tabular form in case you need or want exact values from the figures.

Table 14 – Difference in IOzone Write Performance Results from the 2.6.38 to 2.6.32 kernels with data compression levels of 98%, 75%, 50%, 25%, 2% and Random, for the SandForce SSD

Table 15 – Difference in IOzone Read Performance Results from the 2.6.38 to 2.6.32 kernels with data compression levels of 98%, 75%, 50%, 25%, 2% and Random, for the SandForce SSD

Table 16 – Difference in IOzone IOPS Performance Results from the 2.6.38 to 2.6.32 kernels with data compression levels of 98%, 75%, 50%, 25%, 2%, and Random, for the SandForce SSD