Not much sense writing the data Really Really Fast, if you end up losing the entire file system and all the data when someone trips over the power cord…

The manipulation of this data adds to the latency of the SSD. The larger the block size, the better the write latency results, because the control information is relatively low.

Reads, of course, remain almost instantaneous.

Since SSD\’s are still very expensive, a customer needs to correctly characterize his transaction processing to make sure solid state storage will pay for itself in the long run.

Some popular SSD manufacturers don\’t tell you up front that for every 40 Gig of SSD storage, you need to allocate 4 Gig of system ram to process the metadata overhead of small block writes.

On top of that, systems are tuned to allow for the latency of mechanical disk drives. So, routines for buffering need to be identified and turned off for SSD\’s. My question is: Was that done for these tests?

I\’m not advocating using ramdisks for the journal. You can do it if you like and there are things you need to do ensure data integrity if you do it. But it is possible. There are some DRAM devices you can use for this approach: ACARD has a cool box, Texas Memory, Violin Memory, etc. All of them should be coupled with a UPS and a mechanism to flush the journal completely, unmounting the file system, and then dumping the contents of the DRAM device to permanent media all in the event of a power failure. Reversing the process on start up involves bringing up the DRAM device, restoring the journal contents from permanent media to the DRAM device, bringing up the file system and mounting it. I\’ve tried this process one as an experiment and ext4 didn\’t mind so I\’m assuming everything went correctly (didn\’t do an fsck but I should have).

But again, it is possible to use a DRAM device. As with anything though, there are tradeoffs: you can potentially get better performance but it\’s much more of an administrative task.

The reason I haven\’t tested an SSD is, well, I don\’t have one. I have looked at buying one but my price range is fairly low right now and I didn\’t want to test a substandard SSD (then we get the ensuing argument about \”… that SSD is a piece of garbage and doesn\’t reflect what a REAL SSD can do.. blah, blah\”.

Some specific answers:

@genghiskhat – you are correct. I\’m surprised that slipped by – my bad. I will get those fixed.

@jmoondoogle – I don\’t know of any file systems where the buffering is tuned for mechanical drives. I don\’t think they get that specific although I can ask Eric Sandeen or Theodor Ts\’o. Just remember that there are a number of layers and buffering that happen between the application and the actual drive. Flipping on/off these buffers is not always trivial and does not always produce the desired affects (unless you are a kernel hacker and can read the code – Larry McVoy is quite good at doing this). There are buffers in the file system, there are buffers in the actual VFS as controlled by the kernel, there are buffers in the IO scheduler, there can potentially be buffers in the driver layer, and there are buffers (cache) in the drives themselves. Determining how they all interact is, well, difficult.

Also, I wanted to examine the impact on performance using just the default options. Trying to determine the impact of \”tuning\” is load dependent and difficult to analyze a priori\’.

@jmoondoggie: I\’m not aware of any SSD drives using system RAM for buffering. I\’m not exactly sure how they do that because it would have to be in the driver and I haven\’t seen any drivers with buffering but I could definitely be wrong. Can you point to some examples of this behavior?

Jeff

Fusion-io is such a product. But the problem with NAND Flash is that it inherently has latency on writes due to the way it commits the write internally. So, the smaller the write block, the more control overhead to handle, thus the need to overflow into system resources because it can\’t handle it on such a small card. Again, reads are not a problem. It is the write process inherent in NAND Flash.

One area Fusion-io is different, is that they don\’t use standard disk drive protocol like SCSI or SATA. They pass data directly from the \”drive\” to the PCIe bus. So there is no latency injected from the standard disk channel protocol. To compensate for the increased speed through PCIe, it is necessary to set O_Direct.

This is a double edged sword, because although the performance is fast, the drive is not SNIA-compliant or SMART compliant, and a disk controller card such as Promise can\’t manage it, or use it for hardware based RAID. It also means it\’s not bootable (it can\’t be used to load an OS).

Whenever I run an I/O benchmark I always run collectl in parallel, measuring a wide variety of metrics every 10 seconds, and graph the output. Often times something jumps out to indicate an invalid test that might reveal a kernel bug or mistuned system. If you can\’t get a relatively smooth I/O rate, you\’re not reporting valid numbers in your result OR simply identifying a system limitation that itself can be important to note.

-mark