dcsimg

FS-Cache & CacheFS: Caching for Network File Systems

FS-Cache along with CacheFS is now in the 2.6.30 kernel and can be used for local caching of AFS and NFS.

When *nix OS’s were developed, systems could be a bit on the slow side. Typical networks were either 10 Mb/s or, if you were lucky, 100 Mb/s. Accessing network based file systems such as NFS and AFS could be rather slow over these networks. In response to sometimes slow access, a local caching mechanism, called CacheFS, was developed to provide local caching for distributed file systems. This mechanism caches data (either complete files or parts of files) on local storage so that data can be possibly accessed from a local storage device instead of a network based file system.

As part of the 2.6.30 kernel bonanza of file systems, a CacheFS implementation for Linux was added. Currently it supports NFS and AFS, but other file systems could benefit from such a caching mechanism (more on that later in the article). This article is a brief introduction to CacheFS, which is still undergoing a few growing pains, but is showing great progress for helping network file system performance with an eye toward other uses. In addition, the article will discuss FS-Cache which is a thin layer between CacheFS and the file system itself.

What is FS-Cache and CacheFS?

If you have read about FS-Cache or CacheFS you can easily get confused by the terminology. FS-Cache is the interface between the file system and the cache, allowing the file system to be cache agnostic. CacheFS is the caching backend for FS-cache. CacheFS does the actual handling of data storage and retrieval (i.e. the caching mechanism) and uses a partition on a block device. There is also an alternative caching mechanism called CacheFiles that uses a directory in an existing file system. This architecture allows alternative caching mechanisms to be used with FS-Cache. The first implementation of CacheFS was developed by Sun in 1993 for the Solaris 2.3 operating system. After that, other versions soon appeared, notably for Irix in 1994.

A number of years ago, some Linux kernel developers, notably Dave Howells and Steve Dickson, began to develop FS-Cache and CacheFS for Linux. In August 2004, Dave posted a set of patches to the Linux kernel mailing list for CacheFS. FS-Cache patches have been posted along with CacheFS. A good overview of CacheFS is an article on lwn.net. With the 2.6.30 kernel, CacheFS is included and is available for NFS and AFS.

CacheFS cannot be used for just any file system. The file system must be written to use FS-Cache. FS-Cache can use any caching mechanism it wants (using the caching interface) and the file system does not care. Both AFS and NFS have been modified to use FS-Cache. The 2.6.30 kernel contains these modified versions that utilize FS_Cache.

The diagram below gives you a general view of the data flow when using FS-Cache and CacheFS.

Figure 1: Data Flow showing FS-Cache, CacheFS, and CacheFiles
Figure 1: Data Flow showing FS-Cache, CacheFS, and CacheFiles

In the diagram, NFS or AFS or ISOFS, which are generically called the netfs, make function calls to FS-Cache which in turn calls either CacheFS or CacheFiles. The reason for FS-Cache is to allow the file systems on the left hand side to use caching without having to know about the implementation. This keeps the API a constant for the file systems (NFS, AFS, etc.) while allowing developers to slip in different caching schemes as needed (or desired). Notice that CacheFS uses a partition on a block device, in the diagram this is /dev/hda5, while CacheFiles, uses /var/fscache by default to store the locally cached files.

Figure 2 below shows a bit more detail for the data flow of netfs and FS-Cache.

Figure 2: More Detailed Data Flow showing FS-Cache, CacheFS, and CacheFiles along with netfs
Figure 2: More Detailed Data Flow showing FS-Cache, CacheFS, and CacheFiles along with netfs

In this diagram, it shows how the netfs file system communicates with FS-Cache, which in turn communicates to CacheFS or CacheFiles. Notice too that the netfs also communicates with the VFS as needed (CacheFS is just for caching and is not intended to be a complete file system by itself. Consequently, accessing the VFS by the netfs is necessary). The kernel documentation for FS-Cache is here and the kernel documentation for the api for the netfs is here. Both provide a wealth of information.

The key concept behind FS-Cache is that it does not itself require a file from the netfs to be completely loaded into the cache before the file can be accessed through the cache. This is because:


  • It must operate without a cache

  • It must be possible to open a remote file that’s larger than the cache

  • The combined size of all open remote files must not be limited to the size of the cache

  • The user should not be forced to download an entire file just to access a portion of the file for one time

Consequently, FS-Cache is built around the concept of pages. So pages, rather than files, are put into the cache and are used when the netfs requires it. The FS-Cache implementation in Linux provides the following features (from the kernel documentation):

  • More than one cache can be used. They are identified by tags to allow them to be individually identified

  • A cache can be shared between multiple netfs. In the event that they are shared, the cached data of each netfs is independent of the other. In addition, it does not associate two different views of the same file. For example, this means that if a file is read by both NFS and CIFS (if CIFS is FS-Cache aware), then two copies of the file will end up in the cache.

  • The caches can be removed or added at any time

  • The netfs is provided with an interface that allows either party (netfs or FS-Cache) to withdraw caching facilities from a file

  • The interface to the netfs returns as few errors as possible, preferring to let the netfs remain oblivious

  • Data IO is done directly to and from the netfs’s pages

  • As much as possible is done asynchronously

Dave Howell has also published a paper that discusses FS-Cache more in-depth. It also talks about CacheFS and CacheFiles.

Just as with everything in life, there are some positives and negatives to things. FS-Cache and CacheFS follow this rule of thumb. Dave Howells has done a good job explaining what the trade-offs are in terms of performance. Summarizing from his paper:


  • File Lookup Time will be:


    • Increased because the cache will be checked before resorting to the network.

    • Increased because the a looked-up object in the cache must be noted.

    • Decreased because the metadata is cached locally


  • File Read Time will be:


    • Increased because the cache much be checked before resorting to getting the data over the network

    • Increased because the data obtained must be copied back to the cache

    • Decreased because of the local caching which should be faster than going over the network


  • File Write Time will be:


    • Decreased because of writeback caching using the disk

    • Neutral because write-through caching should be able to write to the network and the cache at the same time


  • File Replacement Time will be:


    • Increased because an object or tree of objects will have to be retired from the disk


The performance of the network and the server are also affected. The goal of FS-Cache and CacheFS is to reduce network traffic because some of the data requests will be satisfied by local storage (CacheFS) reducing the amount of network traffic. The load on the server should also be reduced since it will not have to satisfy all data requests. Consequently, this reduction may make up for the increased file lookup time and file read time due to the cache.

Keep in mind that the local caching might not produce an overall throughput – it’s all up to the particular load and IO pattern. It is not well suited for random access IO such as that from databases because it is unlikely that the data will be in cache. But for IO patterns that are more sequential in nature, CacheFS could help performance.

Moreover, files that are opened with O_DIRECT should not go through the cache. It is up to the netfs to take care of this so that FS-Cache shouldn’t even be contacted. But if your application uses this flag when opening files, caching may not work as well as you had hoped (it shouldn’t work at all).

Comments on "FS-Cache & CacheFS: Caching for Network File Systems"

Below you will come across the link to some web pages that we consider you need to visit.

Every the moment inside a though we opt for blogs that we read. Listed below are the newest websites that we pick.

Very couple of sites that come about to become comprehensive below, from our point of view are undoubtedly properly worth checking out.

Here are a few of the websites we advise for our visitors.

Check below, are some absolutely unrelated websites to ours, nonetheless, they are most trustworthy sources that we use.

Major thanks for the post.Really looking forward to read more. Really Great.

Very handful of web sites that occur to become comprehensive below, from our point of view are undoubtedly well worth checking out.

Just beneath, are many totally not related web sites to ours, however, they are surely worth going over.

The information mentioned within the article are a few of the very best out there.

Here is an excellent Weblog You may Locate Fascinating that we encourage you to visit.

That is the finish of this report. Here you will find some sites that we feel you will appreciate, just click the hyperlinks.

Below you will find the link to some web pages that we consider you ought to visit.

Here are several of the sites we advise for our visitors.

That is the finish of this post. Here you will find some websites that we think you?ll enjoy, just click the links.

Please take a look at the web sites we comply with, including this one particular, because it represents our picks from the web.

Every once in a whilst we pick blogs that we read. Listed below would be the most up-to-date web sites that we pick.

Usually posts some pretty intriguing stuff like this. If you?re new to this site.

The time to read or visit the material or web pages we have linked to below.

The facts talked about inside the article are a few of the most effective accessible.

Although web-sites we backlink to below are considerably not related to ours, we feel they are truly worth a go by means of, so have a look.

Please take a look at the sites we adhere to, like this one, because it represents our picks from the web.

One of our visitors just lately recommended the following website.

Always a significant fan of linking to bloggers that I love but don?t get lots of link like from.

Very couple of web-sites that transpire to be in depth below, from our point of view are undoubtedly effectively really worth checking out.

Always a large fan of linking to bloggers that I enjoy but really don’t get a lot of link love from.

Very handful of sites that happen to become in depth beneath, from our point of view are undoubtedly properly worth checking out.

Very few internet websites that come about to become in depth beneath, from our point of view are undoubtedly properly really worth checking out.

Always a significant fan of linking to bloggers that I like but really don’t get a great deal of link appreciate from.

Here are some hyperlinks to websites that we link to because we believe they are worth visiting.

I just want to mention I’m beginner to blogs and absolutely loved you’re web site. Almost certainly I’m likely to bookmark your website . You surely have great articles and reviews. With thanks for revealing your website page.

The time to study or pay a visit to the content material or web sites we have linked to beneath.

Very handful of web-sites that happen to become detailed below, from our point of view are undoubtedly properly worth checking out.

One of our guests not long ago proposed the following website.

Please pay a visit to the websites we stick to, including this one, as it represents our picks in the web.

Please stop by the web-sites we comply with, including this a single, as it represents our picks through the web.

Wonderful story, reckoned we could combine a few unrelated information, nevertheless seriously worth taking a appear, whoa did 1 study about Mid East has got extra problerms as well.

We came across a cool website that you simply might take pleasure in. Take a look should you want.

Usually posts some very interesting stuff like this. If you are new to this site.

Although internet websites we backlink to beneath are considerably not associated to ours, we really feel they’re truly worth a go by means of, so have a look.

That would be the end of this article. Here you will obtain some web pages that we assume you will value, just click the hyperlinks.

Usually posts some really fascinating stuff like this. If you?re new to this site.

Here is an excellent Weblog You might Locate Fascinating that we encourage you to visit.

Although websites we backlink to beneath are considerably not related to ours, we feel they may be really worth a go by means of, so have a look.

We like to honor many other net web-sites on the web, even when they aren?t linked to us, by linking to them. Under are some webpages really worth checking out.

Here is an excellent Blog You might Discover Fascinating that we encourage you to visit.

One of our guests lately proposed the following website.

Usually posts some quite exciting stuff like this. If you are new to this site.

Here are a few of the internet sites we advise for our visitors.

Just beneath, are several completely not related internet sites to ours, however, they may be surely really worth going over.

Here is a great Weblog You might Come across Interesting that we encourage you to visit.

Leave a Reply