Kernels 2.6.35 and 2.6.36: Storage Updates

Two kernel releases have gone by since we last checked in with the check-ins. While the storage related changes are seemingly minimal, it's always good to review what changed; you might be surprised.


Recently, I have spent some time reviewing Linux kernels from a storage perspective; starting with 2.6.30 (actually going back into the 2.6.2x series) and ending with the 2.6.34 kernel. That article was published on June 8 of this year – just a short time ago. In the meantime, the pace of kernel development has continued and as of the writing of this article the merge window for the 2.6.37 kernel has just closed signaling the beginning of the great bug hunt before the release of the 2.6.37 kernel. So now seems like a good time to go over the 2.6.35 and 2.6.36 kernels to review what changes have happened that affect the Linux storage world.


One of the best sources of information about Linux kernel versions is kernelnewbies.org”. It has a very nice writeup of the higher level kernel changes but the writeup can be a little terse. However, they can be very useful as a starting point. In particular, let’s start with the 2.6.35 highlights as a starting point for a discussion about the storage changes in the 2.6.35 kernel.

One of the first items listed in the kernel newbies 2.6.35 review is about changes to one of the favorite Linux file systems, btrfs. In 2.6.35, btrfs added support for direct I/O. Direct I/O is a method that allows the data to go from the application buffers directly to the storage devices (hence the term “direct”). This bypasses all of the caching and other IO techniques in the VFS (e.g. read-ahead, write coalescence). For some workloads Direct I/O can result in improved performance and reduced CPU usage on the host. A classical example of a workload that can benefit from Direct I/O is a database workload.

The second updated file system that was highlighted is xfs. XFS has been around a fairly long time and was originally ported to Linux by SGI (the “old” SGI) a while ago. For many years xfs was the file system of choice for high performance but it is also known to have less than stellar metadata performance. However, thanks to Redhat and other developers, xfs development has been revived and there are a number of new features/capabilities in today’s xfs (i.e. “it’s not your father’s xfs”).

In the 2.6.35 kernel, a new journaling technique was added to xfs that is somewhat based on ext3 and reiserfs. This journal approach collects asynchronous transactions in memory. This reduces the I/O bandwidth on the log by several orders of magnitude, allowing much better metadata performance for heavy metadata workloads. In addition, the I/O bandwidth used for the log decreases by orders of magnitude. This patch didn’t change the log format on the disk but only changed the in-memory potion of the logging. However, this journal mode, called “delayed logging”, is still experimental, but you can test it using a mount option: “-o delaylog”. If you are using xfs and have heavy metadata workloads, you might want to consider trying this logging option in a test environment.

The kernel newbies review listed the btrfs and xfs changes as fairly major, but there were some other changes to other file systems that are fairly important as well. To make things easier, I’ve create a list of the file system and the significant changes.

  • Squashfs: xattr (extended file attributes) support was added. This allows squashfs to retain xattr values associated with data and it also allows squashfs to be better combined (“unioned”) with other file systems so that a single data tree can seemingly be both read and write.
  • Ext2: The remaining pieces of the BKL (Big Kernel Lock) were removed from ext2 in 2.6.35. The BKL was originally used to get SMP (Symmetric Multi-Processing) capability into the kernel fairly quickly but it also reduced the scalability of the kernel as well as imposing some performance penalties. These changes mean that the last remnants of the BKL in ext2 have been removed from the Linux kernel. While you may scoff at people still focusing on ext2, it is still widely used and, believe it or not, it has good performance.
  • Ext4: The big change with ext4 in 2.6.35 was the addition of a check for a good block group before loading “buddy pages”. This change speeds up allocations particularly for the case where partitions are relatively full.
  • NILFS: This log-structured file system added a subtle change in 2.6.35. It changed the default of “errors” mount option to “remount-ro” mode. Previously when nilfs encountered an error it would allow operations to continue, possible further causing errors. With this change, the default behavior in nilfs is to mount the file system as read-only (“ro”) so that further damage is minimized.
  • OCFS2: In 2.6.35 there were several changes to OCFS2 (one of the under appreciated file systems in Linux). The first change was to implement allocation reservations, which can greatly reduce fragmentation because allocations for a specific file can be done much more sequentially (this has implications for performance as well). Several smaller changes were made as well and they are listed below (from the kernel newbies list)
    • The “punch-hole” code was further optimized to speed up some rare operations.
    • A dis-contiguous block group was created to improve some types of allocations. It marks an incompatible bit which makes it a forward-compatible change.
    • The no file interruption (nointr) mount option was made default.

There were also some additions to the block I/O layer for the kernel (blkio) that impact storage. These changes can be important because they affect how I/O operations are done within the kernel including monitoring of I/O operations. The first set of changes are the addition of some statistics that can help with monitoring. These statistics are:

  • io_service_time
  • io_wait_time
  • io_serviced
  • io_service_bytes
  • io_merged
  • io_queued
  • avg_queue_size

The first four statistics are accumulated per operation type (I/O operation) helping us to distinguish between read and write operations and sync and async I/O operations. The fifth statistic, io_merged, is mostly used for debugging. The last two statistics are also primarily used for debugging as well. They keep statistics of the queue depth of the cgroup (a part of the I/O controller). The authors, Jens Axboe and Divyesh Shah, state that they can be used for debugging user I/O problems.

A few more patches were added to gather the per-cgroup statistics which can be used for debugging problems as well. One of the more important patches was to add a “reset_stats” interface so that all of these statistics can be reset (great for debugging).

Overall, these kernel changes may not result in obvious changes to Linux storage, but actually they are fairly important. We’re seeing the continued development of btrfs, albeit somewhat slower than we like. This could possibly point to the stabilization of btrfs but people are still asking for the additional features. Similarly we’re seeing changes to ext4 that capture more “corner cases” which can affect performance.

Finally, one change that is important whether you might not think so, is the addition of the statistics to the I/O controller. These statistics can greatly help developers when debugging problems or when users encounter problems. Scheduling I/O operations can be very complex and greatly depend upon the details of the workload so it’s almost impossible a priori to know how workloads will interact with the I/O scheduler. Being able to understand what the I/O scheduler is actually doing is important in finding and debugging problems or for improving performance (remember that bad performance is considered by many to be a “bug”).


Comments on "Kernels 2.6.35 and 2.6.36: Storage Updates"

I was suggested this website via my cousin. I am no longer
sure whether this submit is written via him as no one else
recognize such specified approximately my problem. You are
wonderful! Thanks!

I need to to thank you for this fantastic read!! I definitely enjoyed every bit of
it. I have you bookmarked to check out new stuff you post?|

Every weekend i used to pay a quick visit this website, because i wish for enjoyment,
for the reason that this this site conations really pleasant funny information too.|

You made some good points there. I looked on the
internet to learn more about the issue and found most individuals will go
along with your views on this site.|

Great article, exactly what I needed.|

Hey there! I know this is sort of off-topic but I needed to ask.
Does managing a well-established blog like yours require a large amount of work?
I am completely new to operating a blog but I do write in my journal everyday.
I’d like to start a blog so I can share my personal
experience and feelings online. Please let me know if you
have any recommendations or tips for new aspiring blog owners.

Hey there are using WordPress for your blog platform? I’m new to the blog world but I’m
trying to get started and set up my own. Do you need
any html coding expertise to make your own blog? Any help would be really appreciated!|

Terrific post but I was wanting to know if you
could write a litte more on this subject? I’d be very
grateful if you could elaborate a little bit further.
Bless you!|

all the time i used to read smaller posts that also clear their
motive, and that is also happening with this post which I
am reading now.

Thank you for the auspicious writeup. It in fact
was a amusement account it. Look advanced to far added agreeable from you!

By the way, how can we communicate?

I do not even know the way I stopped up right
here, but I believed this publish was good. I do not understand who you might be however certainly you are going to a well-known blogger in the event you aren’t already.

At this time I am going away to do my breakfast, afterward having
my breakfast coming again to read other news.

hello!,I love your writing so so much! share we keep in touch extra about your post on AOL?
I require a specialist in this area to resolve my problem.
May be that’s you! Having a look ahead to see you.

Wonderful website. Lots of helpful information here.

I am sending it to several pals ans also sharing in delicious.
And of course, thanks for your sweat!

Really enjoyed this post.Really thank you! Really Cool.

I’d like to thank you for the efforts you’ve put in penning this blog.
I am hoping to view the same high-grade blog posts by you in the future as well.
In truth, your creative writing abilities has inspired me to get my very own blog now ;)

I loved your article post.Really thank you! Will read on…

I’m not sure why but this weblog is loading very slow for me.
Is anyone else having this problem or is it a problem
on my end? I’ll check back later on and see if the problem still exists.

You actually make it seem so easy with your presentation but
I find this topic to be really something which I think I would never
understand. It seems too complicated and extremely broad for me.
I am looking forward for your next post, I will try
to get the hang of it!

my homepage :: cheap car insurance in michigan

Thanks in favor of sharing such a fastidious opinion, article
is good, thats why i have read it completely

my web blog Cheap Car Insurance

Good article. I’m dealing with a few of these issues as well..

Look at my website very cheap car insurance

Good day! Do you know if they make any plugins to safeguard against hackers?
I’m kinda paranoid about losing everything I’ve worked hard on. Any tips?

Review my web site :: cheap car insurance

Good day! This is my 1st comment here so I just wanted to
give a quick shout out and tell you I genuinely enjoy reading your
posts. Can you recommend any other blogs/websites/forums that go over the same topics?
Thanks a ton!

Here is my website – cheap car insurance in va

Everyone loves what you guys are usually up too. This type
of clever work and reporting! Keep up the amazing works guys I’ve you guys to blogroll.

Also visit my web page … cheap car insurance in florida

Hi there to every one, it’s actually a good for me to
pay a visit this site, it contains precious Information.

Also visit my weblog; cheap car insurance

For latest news you have to go to see web and on world-wide-web I found this site as a best web page for newest updates.

Stop by my web page :: cheap car insurance

My partner and I stumbled over here coming from a different web address and thought I
may as well check things out. I like what I see so i am just following you.
Look forward to looking into your web page repeatedly.

Here is my website :: cheap car insurance in nj

Hey there this is somewhat of off topic but I was wondering if blogs use
WYSIWYG editors or if you have to manually code with HTML.

I’m starting a blog soon but have no coding know-how so I wanted to get guidance from someone
with experience. Any help would be greatly appreciated!

My website: cheap car insurance in va

We are a roup of volunteers and starting
a new scheme in our community. Your website offered us with valuable
info to woork on. You’ve dine a formidable jobb
and our whole community will be grateful to you.

Also visot my weblog … cheap car insurance

Hi there, I want to subscribe for this blog
to take most up-to-dateupdates, therefore where can i do
iit please help.

Here is mmy webb blog: cheap car insurance in ga

I always used to study article in news papers but now as I am a user
oof net so from now I am using net for posts, thanks to web.

Also viwit myy blog post; cheap car insurance quotes

This is mmy first time visit at here and i am actually happy to read
all at one place.

Here is mmy homepage :: Cheap Car Insurance

Helloo colleagues, its great paragraph on the topic of educationand entirely defined, keep it up all
the time.

Feel free tto surf to my site; cheap car insurance in california

Have you ever thought about including a little biit more
than juswt your articles? I mean, what you say is imortant and all.
Nevertheless think about if you added some great pictures or videos to give
your posts more, “pop”! Your conten is excellent but with images and videos,
this blog ciuld undeniably be one of the best in its field.

Superb blog!

Look into my blog: Cheap Car Insurance

I just like tthe helpful information you provide to your articles.
I will bookmark your blog and check again here frequently.
I am fairly sure I wioll learn many new stuff proper here! Best of luck for
the next!

My blog post cheap car insurance

It’s an amazing article in ffavor off all the internet people;
they will take advantage from it I am sure.

Also visit my blog post … cheap car insurance

I enjoy reading an article that can make people
think. Also, thank you for allowing me to comment!

Have a look at my website Cheap car insurance

Its like you read myy mind! Youu seem to understand so
much about this, likke you wrote the e book in itt or
something. I think that you could do with some percent to pressure the message home a little bit, but othr than that, this is magnificent blog.
A great read. I’ll definitely be back.

Also visit my web site :: cheap car insurance michigan

Hello there! I know thijs is somewhat off topic but I was wondering if you knew where
I ould loczte a captcha plugin for my comment form?
I’m using the same blog platform as yours aand I’m havingg problems
finding one? Thanks a lot!

my web-site … Cheap Car Insurance

Hello there! Would you mind if I share your blolg with my twitter group?
There’s a lot of folks that I think would really enjoy your content.
Please let me know. Thanks

my page: Cheap Car Insurance

Hmm is anyone else encountering problems with the
pictures on this blog loading? I’m trying to determine if its a problem on my end or if it’s the blog.

Any feed-back would be greatly appreciated.

Feel free to visit my website nj car insurance companies

I am really inspired with your writing talents as well
as with the layout to your weblog. Is this a paid subject or did you
customize it yourself? Anyway keep up the excellent high
quality writing, it is rare to look a great weblog like this one today..

My web-site – car insurance companies

This website really has all of the info I wanted about this subject and didn’t know who to ask.

Review my website … cheap car insurance florida

I’m impressed, I have to admit. Rarely do I encounter a blog that’s both educative aand
amusing, and without a doubt, you have hit the nail on the head.

The problem is something that not enough folks are speaking intelligently about.
I’m very happy I found this during my hunt for something
concerning this.

My blog … cheap car insurance in texas

Hi there would you mind letting me know which hosting company you’re working
with? I’ve loaded your boog in 3 dirferent internet
browsers and I must say this blog loads a lot faster then most.

Can you recommend a good internet hosting provider at
a honest price? Thanks a lot, I appreciate it!

my web page :: cheap car insurance in florida

My partner and I stumbled over here from a different web page and thought I
may as well check things out. I like what I see so now i’m following you.
Look forward to looking at your web page for a second time.

Check out my blog :: cheap car insurance for students

Really enjoyed this blog.Really looking forward to read more. Really Cool.

whoah this blog is wonderful i love reading your articles. Keep up the great work! You know, many people are hunting around for this information, you could aid them greatly.

Leave a Reply