dcsimg

FS-Cache & CacheFS: Caching for Network File Systems

FS-Cache along with CacheFS is now in the 2.6.30 kernel and can be used for local caching of AFS and NFS.

When *nix OS’s were developed, systems could be a bit on the slow side. Typical networks were either 10 Mb/s or, if you were lucky, 100 Mb/s. Accessing network based file systems such as NFS and AFS could be rather slow over these networks. In response to sometimes slow access, a local caching mechanism, called CacheFS, was developed to provide local caching for distributed file systems. This mechanism caches data (either complete files or parts of files) on local storage so that data can be possibly accessed from a local storage device instead of a network based file system.

As part of the 2.6.30 kernel bonanza of file systems, a CacheFS implementation for Linux was added. Currently it supports NFS and AFS, but other file systems could benefit from such a caching mechanism (more on that later in the article). This article is a brief introduction to CacheFS, which is still undergoing a few growing pains, but is showing great progress for helping network file system performance with an eye toward other uses. In addition, the article will discuss FS-Cache which is a thin layer between CacheFS and the file system itself.

What is FS-Cache and CacheFS?

If you have read about FS-Cache or CacheFS you can easily get confused by the terminology. FS-Cache is the interface between the file system and the cache, allowing the file system to be cache agnostic. CacheFS is the caching backend for FS-cache. CacheFS does the actual handling of data storage and retrieval (i.e. the caching mechanism) and uses a partition on a block device. There is also an alternative caching mechanism called CacheFiles that uses a directory in an existing file system. This architecture allows alternative caching mechanisms to be used with FS-Cache. The first implementation of CacheFS was developed by Sun in 1993 for the Solaris 2.3 operating system. After that, other versions soon appeared, notably for Irix in 1994.

A number of years ago, some Linux kernel developers, notably Dave Howells and Steve Dickson, began to develop FS-Cache and CacheFS for Linux. In August 2004, Dave posted a set of patches to the Linux kernel mailing list for CacheFS. FS-Cache patches have been posted along with CacheFS. A good overview of CacheFS is an article on lwn.net. With the 2.6.30 kernel, CacheFS is included and is available for NFS and AFS.

CacheFS cannot be used for just any file system. The file system must be written to use FS-Cache. FS-Cache can use any caching mechanism it wants (using the caching interface) and the file system does not care. Both AFS and NFS have been modified to use FS-Cache. The 2.6.30 kernel contains these modified versions that utilize FS_Cache.

The diagram below gives you a general view of the data flow when using FS-Cache and CacheFS.

Figure 1: Data Flow showing FS-Cache, CacheFS, and CacheFiles
Figure 1: Data Flow showing FS-Cache, CacheFS, and CacheFiles

In the diagram, NFS or AFS or ISOFS, which are generically called the netfs, make function calls to FS-Cache which in turn calls either CacheFS or CacheFiles. The reason for FS-Cache is to allow the file systems on the left hand side to use caching without having to know about the implementation. This keeps the API a constant for the file systems (NFS, AFS, etc.) while allowing developers to slip in different caching schemes as needed (or desired). Notice that CacheFS uses a partition on a block device, in the diagram this is /dev/hda5, while CacheFiles, uses /var/fscache by default to store the locally cached files.

Figure 2 below shows a bit more detail for the data flow of netfs and FS-Cache.

Figure 2: More Detailed Data Flow showing FS-Cache, CacheFS, and CacheFiles along with netfs
Figure 2: More Detailed Data Flow showing FS-Cache, CacheFS, and CacheFiles along with netfs

In this diagram, it shows how the netfs file system communicates with FS-Cache, which in turn communicates to CacheFS or CacheFiles. Notice too that the netfs also communicates with the VFS as needed (CacheFS is just for caching and is not intended to be a complete file system by itself. Consequently, accessing the VFS by the netfs is necessary). The kernel documentation for FS-Cache is here and the kernel documentation for the api for the netfs is here. Both provide a wealth of information.

The key concept behind FS-Cache is that it does not itself require a file from the netfs to be completely loaded into the cache before the file can be accessed through the cache. This is because:


  • It must operate without a cache

  • It must be possible to open a remote file that’s larger than the cache

  • The combined size of all open remote files must not be limited to the size of the cache

  • The user should not be forced to download an entire file just to access a portion of the file for one time

Consequently, FS-Cache is built around the concept of pages. So pages, rather than files, are put into the cache and are used when the netfs requires it. The FS-Cache implementation in Linux provides the following features (from the kernel documentation):

  • More than one cache can be used. They are identified by tags to allow them to be individually identified

  • A cache can be shared between multiple netfs. In the event that they are shared, the cached data of each netfs is independent of the other. In addition, it does not associate two different views of the same file. For example, this means that if a file is read by both NFS and CIFS (if CIFS is FS-Cache aware), then two copies of the file will end up in the cache.

  • The caches can be removed or added at any time

  • The netfs is provided with an interface that allows either party (netfs or FS-Cache) to withdraw caching facilities from a file

  • The interface to the netfs returns as few errors as possible, preferring to let the netfs remain oblivious

  • Data IO is done directly to and from the netfs’s pages

  • As much as possible is done asynchronously

Dave Howell has also published a paper that discusses FS-Cache more in-depth. It also talks about CacheFS and CacheFiles.

Just as with everything in life, there are some positives and negatives to things. FS-Cache and CacheFS follow this rule of thumb. Dave Howells has done a good job explaining what the trade-offs are in terms of performance. Summarizing from his paper:


  • File Lookup Time will be:


    • Increased because the cache will be checked before resorting to the network.

    • Increased because the a looked-up object in the cache must be noted.

    • Decreased because the metadata is cached locally


  • File Read Time will be:


    • Increased because the cache much be checked before resorting to getting the data over the network

    • Increased because the data obtained must be copied back to the cache

    • Decreased because of the local caching which should be faster than going over the network


  • File Write Time will be:


    • Decreased because of writeback caching using the disk

    • Neutral because write-through caching should be able to write to the network and the cache at the same time


  • File Replacement Time will be:


    • Increased because an object or tree of objects will have to be retired from the disk


The performance of the network and the server are also affected. The goal of FS-Cache and CacheFS is to reduce network traffic because some of the data requests will be satisfied by local storage (CacheFS) reducing the amount of network traffic. The load on the server should also be reduced since it will not have to satisfy all data requests. Consequently, this reduction may make up for the increased file lookup time and file read time due to the cache.

Keep in mind that the local caching might not produce an overall throughput – it’s all up to the particular load and IO pattern. It is not well suited for random access IO such as that from databases because it is unlikely that the data will be in cache. But for IO patterns that are more sequential in nature, CacheFS could help performance.

Moreover, files that are opened with O_DIRECT should not go through the cache. It is up to the netfs to take care of this so that FS-Cache shouldn’t even be contacted. But if your application uses this flag when opening files, caching may not work as well as you had hoped (it shouldn’t work at all).

Comments on "FS-Cache & CacheFS: Caching for Network File Systems"

Every when in a whilst we select blogs that we read. Listed below are the most recent web-sites that we choose.

Wonderful story, reckoned we could combine several unrelated data, nevertheless genuinely really worth taking a appear, whoa did a single find out about Mid East has got far more problerms also.

We came across a cool web-site that you may get pleasure from. Take a look in the event you want.

Usually posts some very intriguing stuff like this. If you are new to this site.

Wonderful story, reckoned we could combine a few unrelated data, nonetheless definitely worth taking a appear, whoa did a single understand about Mid East has got much more problerms also.

Every the moment inside a whilst we pick blogs that we read. Listed below would be the most recent web sites that we pick out.

Every the moment in a though we select blogs that we study. Listed beneath would be the newest web pages that we pick.

One of our guests recently proposed the following website.

The time to study or check out the content or internet sites we’ve linked to beneath.

Very couple of websites that take place to be in depth below, from our point of view are undoubtedly effectively worth checking out.

Although internet websites we backlink to beneath are considerably not related to ours, we feel they’re truly worth a go as a result of, so have a look.

That could be the end of this article. Here you?ll discover some web sites that we assume you will enjoy, just click the hyperlinks.

Please check out the websites we stick to, such as this 1, because it represents our picks from the web.

Below you?ll uncover the link to some sites that we feel you need to visit.

Very few web-sites that transpire to be detailed beneath, from our point of view are undoubtedly properly worth checking out.

Usually posts some quite interesting stuff like this. If you are new to this site.

We came across a cool internet site that you simply might enjoy. Take a appear should you want.

That would be the finish of this write-up. Right here you?ll locate some web sites that we believe you?ll appreciate, just click the hyperlinks.

Below you will find the link to some sites that we feel you need to visit.

Wonderful story, reckoned we could combine a number of unrelated data, nonetheless actually worth taking a look, whoa did one understand about Mid East has got extra problerms too.

Check below, are some entirely unrelated sites to ours, nonetheless, they are most trustworthy sources that we use.

Please visit the web pages we adhere to, which includes this 1, as it represents our picks through the web.

Check below, are some entirely unrelated sites to ours, nevertheless, they are most trustworthy sources that we use.

Very few internet websites that happen to be comprehensive beneath, from our point of view are undoubtedly very well worth checking out.

Every after in a whilst we decide on blogs that we read. Listed beneath would be the most current sites that we pick out.

The time to read or stop by the content material or web pages we’ve linked to beneath.

We came across a cool web site that you simply could possibly enjoy. Take a look should you want.

The facts talked about inside the post are a few of the most effective offered.

Just beneath, are various totally not connected web-sites to ours, even so, they’re certainly worth going over.

Usually posts some extremely exciting stuff like this. If you are new to this site.

Although web-sites we backlink to beneath are considerably not associated to ours, we really feel they are in fact worth a go by means of, so have a look.

Wonderful story, reckoned we could combine a number of unrelated information, nonetheless genuinely really worth taking a look, whoa did one study about Mid East has got far more problerms also.

Just beneath, are various totally not associated web sites to ours, nevertheless, they’re certainly really worth going over.

The info talked about within the write-up are some of the most beneficial accessible.

The facts mentioned inside the article are several of the ideal offered.

One of our guests lately advised the following website.

Here are some links to web sites that we link to simply because we believe they are worth visiting.

Very few internet sites that take place to become in depth beneath, from our point of view are undoubtedly effectively worth checking out.

That is the finish of this article. Here you will find some web sites that we believe you?ll appreciate, just click the links.

Usually posts some extremely interesting stuff like this. If you are new to this site.

One of our visitors lately advised the following website.

Please go to the web pages we follow, including this one particular, because it represents our picks in the web.

Very handful of websites that come about to be comprehensive below, from our point of view are undoubtedly properly really worth checking out.

Just beneath, are many absolutely not connected web sites to ours, even so, they may be certainly worth going over.

The information and facts mentioned in the article are some of the best offered.

We like to honor quite a few other world-wide-web web sites on the web, even though they aren?t linked to us, by linking to them. Beneath are some webpages worth checking out.

Just beneath, are many totally not connected web pages to ours, however, they are certainly really worth going over.

Always a massive fan of linking to bloggers that I appreciate but really don’t get quite a bit of link love from.

Every when in a whilst we pick out blogs that we study. Listed beneath would be the most current sites that we select.

That will be the finish of this report. Here you?ll uncover some websites that we assume you?ll enjoy, just click the links.

Leave a Reply