Tuning CFQ – What Station is That?

The last article was a quick overview of the 4 schedulers in the Linux kernel. This article takes a closer look at the Completely Fair Queuing (CFQ) scheduler and how you can tune it.

The previous article was an overview of the four standard IO schedulers in the Linux kernel. These IO schedulers are very crucial in determining the IO performance of your hand-held, laptop, desktop, server (of various flavors), or supercomputer nodes. It’s possible to tune the IO scheduler for, possibly, better performance.

This article presents an example of one possible tuning option for the default IO scheduler, CFQ. It adjust just one parameter but the point is to show you that it’s possible and, more importantly, remind you to measure the performance both before and after the changes to determine the effect of the parameter change.

CFQ IO Scheduler

Recall that the CFQ scheduler was written in response to some potential problems with the deadline controller. In particular, from an interview with Jens Axboe, the developer of CFQ, deadline, and a few other IO schedulers, Jens stated, “While deadline worked great from a latency and hard drive perspective, it had no concept of individual process fairness.” So one process that was doing a great deal of IO could cause other applications to have their IO starve.

The CFQ scheduler created the concept of having queues for each process. These queues are created as needed for a particular process. Also, while perhaps not new, the scheduler divided the concept of IO into two parts, Synchronous IO and Asynchronous IO (AIO). Synchronous IO is important because the application stops running until the IO request is finished. In other words, synchronous IO “blocks” the execution of the application until it is done. This is fairly common in read operations because an application may need to read some input data prior to continuing execution.

On the other hand, AIO allows an application to continue operation because the result of the IO operation returns immediately. So rather than wait for confirmation that the application’s IO request has succeeded as in the case of synchronous IO, for AIO the result is returned immediately even if the IO operation has not yet finished. This is potentially very useful for write operations and allows the application to overlap IO with computation (helps efficiency – that is, it can improve run time).

CFQ was designed to separate out synchronous and asynchronous IO operations, favoring synchronous operations (naturally). It also favors read operations for a couple of reasons: (1) reads have a tendency to block execution because the application needs the data to continue, (2) it’s possible with the elevator approach for schedulers to “starve” a read operation that is fairly far out on disk geometry (near the outside of the disk). By favoring read operations it improves read responsiveness and greatly reduces the possibility of a far-out read starvation.

CFQ goes even further by keeping the concept of deadlines from the Deadline IO Scheduler to prevent IO operations from being starved. Jens’ wrote the deadline scheduler and realized that for good performance for some applications it needed the concept of an IO operation “timing out.” That is, an IO operation may be put into a queue for execution but subsequent IO operations may be put ahead of it in the queue. Therefore the IO request at the end of the queue may never get executed or it’s execution may be seriously delayed. The deadline IO scheduler has the concept of a “time-out” period. If the IO request is not executed in during this period, it will be executed immediately. This keeps IO operations from starving in the queue.

Jens combined all of these concepts along with the per-process concept to create CFQ. It can be rather complicated to define exactly how these concepts interact and it’s beyond this article to go into detail, but understanding the concepts that go into CFQ are very important. This is particularly important if we are going to try tuning the scheduler for performance.

Tunable Parameters in CFQ

In addition to open-source being a great way to have access to the code to make changes yourself to adapt it to your requirements, many times the developers of software allowing you to “tune” the application for your situation without having to hack the code base yourself. The IO schedulers in the Linux kernel are no exception. In particular, the CFQ scheduler has 9 parameters for tuning performance. While discussing the parameters can get long, it is worthwhile to take a look at these parameters in a bit more depth:


  1. back_seek_max
    This parameter, given in Kbytes, sets the maximum “distance” for backward seeking. By default, this parameter is set to 16 MBytes. This distance is the amount of space from the current head location to the sectors that are backward in terms of distance. This idea comes from the Anticipatory Scheduler (AS) about anticipating the location of the next request. This parameter allows the scheduler to anticipate requests in the “backward” or opposite direction and consider the requests as being “next” if they are within this distance from the current head location.
  2. back_seek_penalty
    This parameter is used to compute the cost of backward seeking. If the backward distance of a request is just (1/back_seek_penalty) from a “front” request, then the seeking cost of the two requests is considered equivalent and the scheduler will not bias toward one or the other (otherwise the scheduler will bias the selection to “front direction requests). Recall, the CFQ has the concept of elevators so it will try to seek in the current direction as much as possible to avoid the latency associated with a seek. This parameters defaults to 2 so if the distance is only 1/2 of the forward distance, CFQ will consider the backward request to be close enough to the current head location to be “close”. Therefore it will consider it as a forward request.
  3. fifo_expire_async
    This particular parameter is used to set the timeout of asynchronous requests. Recall that CFQ maintains a fifo (first-in, first-out) list to manage timeout requests. In addition, CFQ doesn’t check the expired requests from the fifo queue after one timeout is dispatched (i.e. there is a delay in processing the expired request). The default value for this parameter is 250 ms. A smaller value means the timeout is considered much more quickly than a larger value.
  4. fifo_expire_sync
    This parameter is the same as fifo_expire_async but for synchronous requests. The default value for this parameter is 125 ms. If you want to favor synchronous request over asynchronous requests, then this value should be decreased relative to fifo_expire_asynchronous.
  5. slice_sync
    Remember that when a queue is selected for execution, the queues IO requests are only executed for a certain amount of time (the time_slice) before switching to another queue. This parameter is used to calculate the time slice of the synchronous queue. The default value for this parameter is 100 ms, but this isn’t the true time slice. Rather the time slice is computed from the following: time_slice = slice_sync + (slice_sync / 5 * 4 – io_priority)). If you want the time slice for the synchronous queue to be longer (perhaps you have more synchronous operations), then increase the value of slice_sync.
  6. slice_async
    This parameter is the same as slice_sync but for the asynchronous queue. The default is 40 ms. Notice that synchronous operations are preferred over asynchronous operations.
  7. slice_asyn_rq
    This parameter is used to limit the dispatching of asynchronous requests to the device request-queue in queue’s slice time. This limits the number of asynchronous requests are executed (dispatched). The maximum number of requests that are allowed to be dispatched also depends upon the io priority. The equations for computing the maximum number of requests is, max_nr_requests = 2 * (slice_async_rq + slice_async_rq * (7 – io_priority)). The default for slice_async_rq is 2.
  8. slice_idle
    This parameter is the idle time for the synchronous queue only. In a queue’s time slice (the amount of time operations can be dispatched), when there are no requests in the synchronous queue CFQ will not switch to another queue but will sit idle to wait for the process creating more requests. If there are no new requests submitted within the idle time, then the queue will expire. The default value for this parameter is 8 ms. This parameters can control the amount of time the schedulers waits for synchronous requests. This can be important since synchronous requests tend to block execution of the process until the operation is completed. Consequently, the IO scheduler looks for synchronous requests within the idle window of time that might come from a streaming video application or something that needs synchronous operations.
  9. quantum
    This parameter controls the number of dispatched requests to the device queue, request-device (i.e. the number of requests that are executed or at least sent for execution). In a queue’s time slice, a request will not be dispatched if the number of requests in the device request-device exceeds this parameter. For the asynchronous queue, dispatching the requests is also restricted by the parameter slice_async_rq. The default for this parameter is 4.

You can see that the CFQ scheduler prefers synchronous IO requests. The reason for this is fairly simple – synchronous IO operations block execution. So until that IO operation is executed the application cannot continue to run. These applications can include streaming video or streaming audio (who wants their movie or music to be interrupted?), but there are a great deal more applications that perform synchronous IO.

On the other hand, Asynchronous IO (AIO) can be very useful because execution immediately returns to the application immediately without waiting for confirmation that the operation has completed. This allows the application to “overlap” computation and IO. This can be very useful for many operations depending upon the goals and requirements. There is quite a good article that talks about synchronous and asynchronous and blocking and non-blocking IO requests.

Quick Tuning Example

Comments on "Tuning CFQ – What Station is That?"

sgtaaron

The grammar in this story is terrible. I read the previous article just now and note that the grammar in that story is bad, too. Does anybody at Linux Magazine bother to proofread their stories?

Reply
porridge

Wow, what a useless article.. should have finished after listing the knobs, if tweaking them didn\’t bring any improvements.

Reply
laytonjb

@porridge,

Sorry – but a \”no\” answer can be as good as a \”yes\” answer. At least I showed for the test configuration I used and the tests I ran that I saw little improvement. Guess what – at least I now know. I doubt you could have answered that question before the testing.

Plus, I hope you understand that you can make tweaks but you don\’t know if they help or not, until you actually test them. Plus you should test them with applications that have some meaning to you and you situation.

If I had found a \”magic\” tweak combination would that have meant anything to you? Maybe or maybe not. Is your hardware the same? Did you run the same tests? Are the tests meaningful to you? Sometimes coming up with a magic tweak doesn\’t help all situations.

I\’m hoping that people reading the article will learn from it. Obviously you didn\’t. Remember the Thomas Edison phrase, \”If I find 10,000 ways something won\’t work, I haven\’t failed. I am not discouraged, because every wrong attempt discarded is another step forward\”

Jeff

Reply
nouiz

Do someone know something similar to the nice program for the cpu, but for the disk?

It happen from times to times that I must do some very heavy io operation on a nfs server(like du -ch –max-depth 1 …). I do it locally, but this slow down all user of this nfs server. I don\’t see any way to do this in all the IOScheduler that you presented or in how we can tune them.

Thanks

Frédéric Bastien

Reply

Definitely would love to start a website like yours. Wish I had the time. My site is so amateurish compared to yours, feel free to check it out: http://tinyurl.com/o55af8p Alex :)

Reply

Tuning CFQ – What Station is That? | Linux Magazine
[url=http://www.g35j6am145avzr1oq842svh9371m6y9rs.org/]ujlrcnyzxe[/url]
jlrcnyzxe http://www.g35j6am145avzr1oq842svh9371m6y9rs.org/
ajlrcnyzxe

Reply

Hollister V?tements Rang dans Coups de coeur Pour les ados

Reply

People in this world have a common fever towards fashion; literally, it is the truth that you can’t oppose
sac louis vuitton soldes

Reply

You can do some research the current trends in fashion, as it will help you a lot. In this current fashionable world
cheap chanel

Reply

People in this world have a common fever towards fashion; literally, it is the truth that you can’t oppose
chanel factory outlet

Reply

If you need the looking-good appearance, then it is assured that you will look out for a top clothing brand.
chanel replica watches

Reply

People in this world have a common fever towards fashion; literally, it is the truth that you can’t oppose
Fake Patek Philippe

Reply

The fashion clothing is a major fact that brings some unique feature in this emerging state of reality
chanel sunglasses sale

Reply

The trendiness in dressing is one of the most essential practices for all, as it plays a main role in personality
Replica Rolex Yachtmaster Watches

Reply

Avoid all those inexpensive packages by spending more on lavish costumes. You can find a fashion flare in yourself for sure.
louis vuitton homme

Reply

The trendiness in dressing is one of the most essential practices for all, as it plays a main role in personality
sac louis vuitton soldes

Reply

The trendiness in dressing is one of the most essential practices for all, as it plays a main role in personality
ralph lauren soldes

Reply

The trendiness in dressing is one of the most essential practices for all, as it plays a main role in personality
chanel handbags outlet

Reply

Avoid all those inexpensive packages by spending more on lavish costumes. You can find a fashion flare in yourself for sure.
replica Breitling

Reply

profit the merchandise within school. i find alot of encouragement. i have been the initial within my education to gain 20244.pink2.net!

Reply

feeling associated with “Quarter in order to 6. inch
furla purse

Reply

Residents have expressed frustration at the byzantine system through which they have to go to get approvals, inspections and, finally, the grant money, all while the boardwalks and businesses lining the Jersey Shore reopened much more quickly.

Reply

Your Pentium D is 4 years old in a few months, anything but new technology. It ran hot, consumed a lot of power, and performed poorly compared to the competition. If your happy with it’s performance, than that’s all that matters, but I would never use one in this day and age.?Whats the best enema recipe for constipation

Reply

Stay in touch with your potential customers. Make them respond to reports about the standard of the service or product they bought. The majority of people tend not to improve with studies: be inventive for making your internet surveys extra fascinating. You can have people gain rewards or create your reviews appear to be game titles.

Reply

You’ve made some decent points there. I looked on the web to learn more about the issue and found most individuals will go along with your views on this web site.|

Reply

Any posting seemed to be very well developed, undoubted viewpoint, abundant in articles, I personally appreciation for aggressive want to be someone

Reply

I’m impressed, I should say. Genuinely rarely do I encounter a weblog that’s both educative and entertaining, and let me tell you, you’ve hit the nail on the head. Your notion is outstanding; the issue is something that not sufficient persons are speaking intelligently about. I’m rather pleased that I stumbled across this in my search for some thing relating to this.
jordans cheap

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>