dcsimg

FS_scan: Getting Detailed with Your Data

Need details on your file system's data? FS_scan allows you dig deep into your storage, giving you the ability to perform trend analysis on the results.

Just last week we walked you though a new tool, agedu, that allows you to get a snapshot view of your file system. agedu produces a very nice graphical display that provides an overview of the age and size of your data (either change or access time). However, there are times when you need or want more detail on the data that’s sitting in your storage. This time around we’ll look at a new tool, FS_scan, that does precisely that.

FS_scan allows you to recursively scan a directory tree to get a detailed view of your data. In particular, it will tell you the dates and ages of your files, the average ages of the files in a given directory, and it will tell you the oldest files in the directory tree. It also produces a CSV file that you can open in a spreadsheet. With this information you can get a very detailed view of the state of your storage with the ability to do a trend analysis of the resulting data (i.e. How fast is it changing? How often are files accessed? How often is data modified?).

Let’s dive in and see what our data’s doing.


When You Just Need More Details

Remember that when talking about the data on your storage there are three dates (or three ages) that need to be considered: (1) Last date accessed or the access age, (2) Last date modified or the modify age, (3) Date last changed or the change age. So when examining a file system it becomes much more difficult to quantify how data is being used because all three dates or ages can be very important. Agedu is a great tool for getting a quick glimpse of the access age or change age of the file system being examined, but it is only a glimpse of the state of the filesystem. If you want to create a more detailed report or monitor the file system over time for a trend analysis then you need more detailed information than what agedu can provide at this time.

One option for getting more detailed information is to use the stat command in Linux. It can be used to get the status of files or even the file system. For example the output from stat looks like the following,

$ stat *
  File: `~storage002.html'
  Size: 11472     	Blocks: 24         IO Block: 4096   regular file
Device: 811h/2065d	Inode: 3220767     Links: 1
Access: (0600/-rw-------)  Uid: ( 1000/laytonjb)   Gid: ( 1000/laytonjb)
Access: 2009-05-24 17:19:52.000000000 -0400
Modify: 2009-05-24 17:19:52.000000000 -0400
Change: 2009-05-24 17:19:52.000000000 -0400
  File: `storage002.html'
  Size: 11285     	Blocks: 24         IO Block: 4096   regular file
Device: 811h/2065d	Inode: 3220766     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/laytonjb)   Gid: ( 1000/laytonjb)
Access: 2009-05-24 17:13:13.000000000 -0400
Modify: 2009-05-24 16:02:27.000000000 -0400
Change: 2009-05-24 16:02:27.000000000 -0400

Or you can get a glimpse of the file system status using the “-f” option.

$ stat -f *
  File: "~storage002.html"
    ID: f11c91747fe09927 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 37263886   Free: 33094551   Available: 31216553
Inodes: Total: 9396224    Free: 9106153
  File: "storage002.html"
    ID: f11c91747fe09927 Namelen: 255     Type: ext2/ext3
Block size: 4096       Fundamental block size: 4096
Blocks: Total: 37263886   Free: 33094551   Available: 31216553
Inodes: Total: 9396224    Free: 9106153

Both options provide useful information. The first option, stat, gives the access, modify, and change dates for the file, as well as the uid, gid, the size of the files, and the permissions. The second option, stat -f gives additional information including the file system type and the fundamental block size. However, if you want to use the stat command to gather detailed information you will have to perform these commands for the directory tree, parse the information, and assemble it into a usable form.

Python has a nice module, called the os module that can easily walk a file system and gather virtually all of the same information that the stat command produces. Even better is that this module is part of the standard library for many of the python packages in many of the distributions. This can easily form the basis of a tool to walk a file system and gather detailed file information.

Python Modules to the Rescue

One of the functions in the os module is called “walk” (os.walk). This function allows you to easily walk a directory tree (i.e. examine the files recursively in a directory tree) and get information on the directories and the files. From the Python 2.6.2 documentation there is a simple example that has been modified and presented below.

#!/usr/bin/python

import os
from os.path import join, getsize

for root, dirs, files in os.walk('.'):
    print root, "consumes",
    print sum(getsize(join(root, name)) for name in files),
    print "bytes in", len(files), "non-directory files"

This quick code snippet displays the number of bytes taken by non-directory files in each directory under the starting directory (current working directory). This simple snippet can form the basis of a script that can walk through a directory tree and gather information about the files. A quick note – this code snippet does not have any exception handling and it is definitely possible you can encounter exceptions.

With the ability to walk a directory tree, you can open the files in the directory and gather statistics on each file. The os module also has a function (method) called os.fstat that can give you most of the information that the stat command produces. Taking the previous example and extending it a bit results in the following example.

#!/usr/bin/python

import os
from os.path import join, getsize

for root, dirs, files in os.walk('.'):
    print root, "consumes",
    print sum(getsize(join(root, name)) for name in files),
    print "bytes in", len(files), "non-directory files"
    for file in files:
       fileloc = root + "/" + file
       FILE = os.open(fileloc, os.O_RDONLY)
       junk = os.fstat(FILE)
       size = junk[6]
       atime = junk[7]
       mtime = junk[8]
       ctime = junk[9]
       uid = junk[4]
       gid = junk[5]
       print "   File: %s size: %s atime: %s mtime: %s ctime: %s" % (file,size,atime,mtime,ctime)
       os.close(FILE)

In the second for loop, the full path to the file is created (fileloc) using the root of the director tree (root) and the file name (file). Notice that os.fstat function returns a list of attributes. For example, it returns the access time (atime), the modify time (mtime), and the change time (ctime), which are all in seconds since the epoch. There are other attributes as well includes the size in bytes (size), the uid (uid) and gid (gid).

The previous example serves as a quick introduction to what you can do with Python using the modules in the standard library. In particular the os module has many functions that are useful for getting detailed information about files.

Down to Details »

FS_scan: A Tool For Detailed File System Information

Comments on "FS_scan: Getting Detailed with Your Data"

Just beneath, are quite a few completely not associated sites to ours, nevertheless, they’re certainly worth going over.

The facts talked about inside the write-up are some of the best readily available.

Very couple of internet websites that occur to become comprehensive below, from our point of view are undoubtedly properly worth checking out.

Check below, are some absolutely unrelated internet websites to ours, nonetheless, they’re most trustworthy sources that we use.

Wonderful story, reckoned we could combine several unrelated information, nonetheless definitely worth taking a appear, whoa did 1 learn about Mid East has got far more problerms as well.

Please pay a visit to the internet sites we comply with, including this one particular, because it represents our picks in the web.

Just beneath, are a lot of completely not related sites to ours, having said that, they’re certainly worth going over.

Always a big fan of linking to bloggers that I enjoy but do not get a great deal of link like from.

One of our visitors lately advised the following website.

I really enjoy simply reading all of your weblogs. Simply wanted to inform you that you have people like me who appreciate your work. Definitely a great post I would like to read this http://jofaorg.tumblr.com/post/145756938847/choose-the-right-way-to-detox-your-body-and

Although internet sites we backlink to beneath are considerably not associated to ours, we feel they may be essentially really worth a go through, so possess a look.

We prefer to honor several other web web-sites on the web, even though they aren?t linked to us, by linking to them. Underneath are some webpages worth checking out.

Just beneath, are quite a few absolutely not related web sites to ours, on the other hand, they may be surely really worth going over.

Really rarely do I encounter a blog that’s both educative and entertaining, and let me tell you, you could have hit the nail on the head. Your idea is outstanding; the issue is something that not enough people are speaking intelligently about. I am very pleased that I stumbled across this in my search for something relating to this hcg

Below you?ll discover the link to some web pages that we believe you ought to visit.

Here are some hyperlinks to web-sites that we link to because we consider they are worth visiting.

Usually posts some incredibly interesting stuff like this. If you are new to this site.

I appreciate that you produced this wonderful article to help us get more knowledge about this topic. I know, it is not an easy task to write such a big article in one day, I’ve tried that and I’ve failed. But, here you are, trying the big task and finishing it off and getting good comments and ratings. That is one hell of a job done! http://steezmatic-designs.com/forums/

One of our visitors not too long ago advised the following website.

Always a big fan of linking to bloggers that I appreciate but really don’t get a whole lot of link like from.

We came across a cool web-site which you might delight in. Take a look for those who want.

Here are some links to web sites that we link to mainly because we believe they’re really worth visiting.

We came across a cool web site that you might take pleasure in. Take a look should you want.

Please check out the web pages we comply with, such as this 1, because it represents our picks in the web.

Although sites we backlink to beneath are considerably not related to ours, we really feel they’re essentially worth a go by, so have a look.

I simply want to tell you that I am new to weblog and definitely liked this blog site. Very likely I’m going to bookmark your blog . You absolutely have wonderful stories. Cheers for sharing with us your blog Schlüsseldienst Berlin

The time to study or take a look at the material or internet sites we’ve linked to beneath.

Here are some links to web sites that we link to since we consider they’re really worth visiting.

Here are some of the web-sites we advise for our visitors.

Here are several of the sites we advocate for our visitors.

Its like you read my mind! You seem to know so much about this,like you wrote the book in it or something. I think that you could do with some pics to drive the message home a bit, but other than that, this is great blog. this https://www.youtube.com/watch?v=I_Y1O8yHXTE

Please stop by the sites we stick to, which includes this 1, because it represents our picks in the web.

Please take a look at the web-sites we comply with, like this one, as it represents our picks from the web.

Here is a superb Blog You might Come across Interesting that we encourage you to visit.

Although internet websites we backlink to beneath are considerably not associated to ours, we really feel they may be in fact worth a go by, so have a look.

Very couple of internet websites that transpire to be comprehensive beneath, from our point of view are undoubtedly very well really worth checking out.

Really great post, Thank you for sharing This knowledge.Excellently written article, if only all bloggers offered the same level of content as you, the internet would be a much better place. Please keep it up! https://www.youtube.com/watch?v=IXG4dC_cHXM

That is the finish of this write-up. Right here you will discover some sites that we feel you?ll value, just click the hyperlinks.

Leave a Reply