Yet another arrangement that can be a practical compromise is to place any cold storage (rarely accessed and typically in large contiguous blocks when it is accessed) on the same physical drive with the journal
for your hot storage FS. The data for the hot storage FS is on a separate dedicated drive. This way, you\’re not wasting drive space or bays but you do have to do the homework up front of figuring out which of our data fits the hot access profile and which fits the cold access profile.

Am I wrong to assume that any hard drive worth a salt has a capacitor on it that can power the drive long enough to commit any uncommitted parts of the local drive cache upon power failure?

With a properly designed hardware device, the need for write barriers should be redundant. With fewer cache flushes on the device, sustained performance should be much higher.

Also, using SSD drives could be another way to get around this, at least having one small SSD drive to hold the filesystem journal boosts performance a whole lot.

1. Does the on-board drive cache add that much in performance? Does the extra 3GB of system RAM, that I have, not trump whatever cache improvements I would get from the on-drive cache? I thought the schedulers, especially with the ability to pick the one that best fits your work load would do a better job of caching and ordering writes than whatever mechanism is on the drive. I think I\’m going to investigate turning off the on-drive caches on my system just to find out.

2. It seems from a theoretical standpoint that caching should not be done on the drive if you have an OS that is designed well at all. For which I\’m fairly confident that Linux is. :-) The drive can never know the correct ordering required to push the data to platter to maintain data integrity. Nor can drives bring the kind of cache sizes to bare that a computer can.

3. Putting a file system log on another drive without caching ensures the integrity of the log but not the data that it represents. Specifically in the case of expanding files or new files. The data written to the file may have not made it to the data drive while the log says it did. So even though you have space allocated the data within it is erroneous. And then again there are no guarantees with the log on the same drive and on-drive caching enabled.

4. Are the drives guaranteed to flush the caches before the system powers off? I would assume that the drive would flush the cache when it receives a power down signal. But that also assumes that the drive is given enough time to do so before the system power goes off. Perhaps the caps mentioned by a previous poster are real indeed?

5. Do all drives with cache support write barriers and cache control?

I hope this sparks some interesting discussion. But it seems to me that regardless of whatever slowdown you might get you are better off letting Linux figure out the best way to order writes then you are letting the drive, except for the case of battery backed caches.

I am a firm believer in decently long lived battery backups regardless. Any computer handling data that is worth anything, and especially if its shared on a network, MUST have a decent battery backup, which is checked regularly to make sure the battery is still viable. But then during the spring and fall in my area the 60 MPH winds will cause numerous power failures in a single day.

Thanks again for the article.

Now on to the questions… But before I answer them let me say that I don\’t know that much about how drives work internally. I don\’t know how the cache interacts with the kernel, etc. (what is really interesting is that the new Seagate drive with the built-in SSD apparently can \”learn\” and move certain blocks onto the SSD so drives have far more intelligence that I knew about).

1. It\’s always good to experiment with various options (write barriers, disabling the drive cache). It helps us understand the performance implications and then we can make an informed decision about choices we make. But to be honest, I don\’t know the relative impact of drive cache vs. system memory. The only \”gotcha\” is that system memory can\’t generally be allocated for IO – the kernel will do what it wants with it (I wish there was a way to force that – need to ask the kernel gods about that). BTW – if you have any data to share, let\’s hear about it! You can always write a quick article for Linux Mag :)

2. Try turning off the drive cache. Performance isn\’t always that great. Plus if the cache is disabled then the kernel will have to \”pause\” more often for the drive to return. This will definitely impact performance.

3. Great observation.

4. The drives will flush as part of the shutdown. I forgot the details but there was some discussion on the ext4 mailing list where Larry McVoy (I hope I got his name correct from memory) pointed out that until the file system is unmounted, the drive may not truly flush it\’s cache. I admit that I don\’t know the details of sync() or fsync() to know when data is truly flushed from cache.

5. I believe that all drives (within reason) obey the write barrier.

Thanks for the comments! Greatly appreciated. And again, if you have particular sentences or passages that seem goofy, let me know. Either I didn\’t explain things well or I made mistakes.

Jeff

#3: There are different levels of journaling. The basic level of journaling is to log the file system meta data; you take a small performance hit for some security. You\’re asking about full-journaling in which ALL user data is written to the log first. This can be done with some file systems, but you take a much bigger hit as the data has to be written to the disk twice.

For people who are really worried about their data, you could invest some time in parity files (PAR2 for example). I\’m really shocked that the ext3 journal didn\’t have checksums!

In a slightly ironic twist, not long after they finally got the I/O barriers working also with LVM/MD in 2.6.33, now they’re getting rid of them. See http://lkml.org/lkml/2010/9/3/199

No need to worry about data integrity with volatile write caches though; the barriers are being replaced with a (simpler) interface to issue cache flush and FUA commands to the devices. Beyond being simpler, the big win with the new code is avoiding IO queue draining. In the current ordered barrier code, ordering is ensured by draining the queue before flushing the cache, but in practice all file systems (well, reiserfs could optionally use the ordered barrier code for ordering) already take care of ordering themselves by waiting for completion before submitting dependent IO’s. So the draining turns out to be unnecessary, and it can have a rather big performance impact.

1) It’s not so much about increasing the amount of cache memory, which as you note is rather insignificant compared to all the RAM the kernel can use for the page cache. Rather it’s about reducing command latency. I.e. instead of “kernel issues write -> device writes stuff to disk -> device signals completion” you have “kernel issues write -> device writes to cache memory -> device signals completion”. Also, presumably the drive has the best information about where it’s head is at the moment, and is thus in the best position to decide in which order to serve I/O commands. Another way to reduce the impact of command latency is to have multiple outstanding commands; SCSI has had this since, well, forever with something called TCQ. Which perhaps explains why SCSI devices typically don’t have volatile write caches. SATA nowadays has something roughly equivalent called NCQ, but it came on the scene after volatile write caches were already the norm in the (S)ATA world.

2) In Linux ordering is handled by the filesystems waiting for IO’s to complete before issuing dependent IO’s. The block layer, and the device command queue for devices supporting such a thing, are free to reorder IO’s in any way they see fit.

4) Sadly, I know of no traditional drive with capacitors. If it had that, it would be awesome; we could safely mount our filesystems with barrier=0 and still be safe. But, I’m sure there are mechanisms to ensure caches are flushed when shutting down; I don’t know if the OS explicitly has to do that when unmounting, or if hw handles it itself.

5) There are rumors about drives which don’t honor cache flush commands, but AFAIK more or less all drives nowadays do honor them. Also, to be pedantic, neither the SCSI nor SATA standards know anything about write barriers, they are purely a software concept in the Linux kernel (implemented via queue draining and cache flushing). However see my previous post about how they are being replaced.