dcsimg

Disk Space: The Final Frontier

Disk space is always at a premium but how do you handle it when you have a disk or filesystem that's full? After today, you'll know how to boldly go into that dark realm.

Your prime directive as a system administrator is to keep those systems whipped into shape and running without interfering with user’s daily operations. Sometimes the systems put you in a difficult position when their disks overflow with files. The simple solution is to remove files. But, what if you can’t do that? You have to add disk to your ailing system. But, how and where do you add that space? And, how do you keep this from happening again?

This article leads you through the discovery and remediation of disk space problems. Start your favorite virtual machine and follow along.

The Basics

The current scenario includes a Debian 5 virtual machine (VM) in VirtualBox 3.2 that currently uses two virtual disks: /dev/hda1 (5 GB) for / and /dev/hdb1 for /var (2 GB). The original 5GB wasn’t enough and the addition of the 2 GB /var disk came later to relieve those issues.

As you can see from the df, the original disk is full again.

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             4.7G  4.4G  110M  98% /
tmpfs                 124M  8.0K  124M   1% /lib/init/rw
udev                   10M  684K  9.4M   7% /dev
tmpfs                 124M  140K  124M   1% /dev/shm
/dev/hdb1             2.0G  262M  1.7G  14% /var

This VM is a web server, therefore, it made sense to expand /var to alleviate any future problems with disk space for that service. Can you think of any other directories that would benefit from more space on this system? Let’s explore how a system administrator can fix this problem.

Space Exploration

The most likely candidates for expansion are /usr, /home and /opt. Current disk usage in each of these directories is as follows:

~$ du -sk /usr
3037496 /usr

$ du -sk /home
957060  /home

$ du -sk /opt
229468  /opt

Clearly, /usr is the space criminal in this scenario and the best candidate for expansion into its own virtual disk. To create a new disk for /usr, would also return 3GB of space to the / filesystem that includes /home and /opt for this system.

The ten steps involved in this expansion are:

  1. Create a new 5 GB virtual disk.
  2. Associate the new disk with the Debian VM.
  3. Boot the VM.
  4. Initialize the new disk.
  5. Format the disk.
  6. Temporarily mount the disk.
  7. Copy /usr to the mount point.
  8. Make the mount permanent with an entry in /etc/fstab.
  9. Remove the contents of /usr.
  10. Mount the new disk on /usr.

Step One: Create a 5 GB Virtual Disk

# VBoxManage createhd --filename Debian3.vdi --size 5000 --variant Fixed

Oracle VM VirtualBox Command Line Management Interface Version 3.2.4
(C) 2005-2010 Oracle Corporation
All rights reserved.

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: f88dc8a9-dbf5-4ac4-845a-c73bc0de1f53

Step Two: Associate the new disk with the Debian VM (Debian5-1).

# VBoxManage storageattach Debian5-1 --storagectl "IDE Controller" --port 1 --device 0 --type hdd --medium Debian3.vdi

Step Three: Boot the VM.

# VBoxManage startvm Debian5-1

Login to the VM and become root or use sudo for these privileged command steps.

Step Four: Initialize the New Disk. In this step, you’ll use the sfdisk command to list the new disk device. Then, you’ll step through the simple initialization process.

Note: The following steps only display relevant output to decrease the length and complexity of the screens.

# sfdisk -l

Disk /dev/hdc: 10158 cylinders, 16 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/hdc: unrecognized partition table type
No partitions found

From the sfdiskoutput, you see that the new disk’s device name is /dev/hdc. Since we’re dedicating the entire disk to /usr, initialization is greatly simplified. The following screen is an interactive command line session using fdisk.

# fdisk /dev/hdc

Command (m for help):n

Command action
   e   extended
   p   primary partition (1-4)
p

Partition number (1-4): 1
First cylinder (1-10158, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-10158, default 10158): 10158

Command (m for help): p

Disk /dev/hdc: 5242 MB, 5242880000 bytes
16 heads, 63 sectors/track, 10158 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0x761001bb

   Device Boot      Start         End      Blocks   Id  System
/dev/hdc1               1       10158     5119600+  83  Linux

Command (m for help):  w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

# sfdisk -l

Disk /dev/hdc: 10158 cylinders, 16 heads, 63 sectors/track
Units = cylinders of 516096 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/hdc1          0+  10157   10158-   5119600+  83  Linux
/dev/hdc2          0       -       0          0    0  Empty
/dev/hdc3          0       -       0          0    0  Empty
/dev/hdc4          0       -       0          0    0  Empty

Step Five: Format the new Disk (Create the filesystem).

# mkfs.ext3 /dev/hdc1

mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
320000 inodes, 1279900 blocks
63995 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1312817152
40 block groups
32768 blocks per group, 32768 fragments per group
8000 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

The new disk is ready to mount and use.

Step Six: Temporarily Mount the New Disk.

# mount /dev/hdc1 /mnt

# mount

/dev/hdc1 on /mnt type ext3 (rw)

Step Seven: Copy /usr to the Mount Point.


# cp -Rp /usr/* /mnt

Step Eight: Make the Mount Permanent with an Entry in /etc/fstab. Edit /etc/fstab and create the following entry.


/dev/hdc1       /usr            ext3    errors=remount-ro 0       1

Step Nine: Remove the contents of /usr.


# rm -rf /usr/*

Step Ten: Mount the New Disk on /usr.


# umount /mnt

# mount /dev/hdc1 /usr

mount

/dev/hdc1 on /usr type ext3 (rw)

The true test of your handiwork is to reboot the system and confirm that everything went as planned. If something goes wrong, your system will panic. The most likely culprits, should you have any problems, are that the /etc/fstab isn’t correctly setup or that the copy and mount didn’t work correctly. Follow the ten steps exactly as shown and you should have no problems.

If your system panics and doesn’t boot, boot the system using a rescue CD, mount the disks to see what went wrong, work through the steps again and take corrective action.

A quick look at the entire system now shows that the filesystems are happy and have plenty of useable space.


$ df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             4.7G  1.5G  3.1G  32% /
tmpfs                 124M  8.0K  124M   1% /lib/init/rw
udev                   10M  684K  9.4M   7% /dev
tmpfs                 124M     0  124M   0% /dev/shm
/dev/hdb1             2.0G  263M  1.7G  14% /var
/dev/hdc1             4.9G  3.1G  1.6G  67% /usr

Warp Speed Ahead

Taking what you’ve learned here, you can create a new disk for /home and one for /opt, if necessary. Systems run out of space. Filesystems fill up. And, it’s your job to fix it before it’s critical.

In the old days (1990s), systems could survive on two disks (not including any mirroring or RAID): A system disk and a “home” disk. The system disk held the standard filesystems, which were partitions or slices created by the system administrator to limit users from killing the system by filling /tmp or /home. The “home” disk, mounted on /home or /h kept user’s home directories isolated from the rest of the system. For a system administrator, managing user’s misuse of space is just another annoyance. These practices helped resolve that annoyance.

Some “old timers” still practice this kind of user isolation today. The rest of us use quotas (A topic of a future article).

Next week, you’ll pay a visit to Father Time and learn to lean on NTP to keep you in sync with the rest of the world.

Comments on "Disk Space: The Final Frontier"

eperry

Step 9 I personally believe that you should not remove the directory until you reboot at least once.

To be safe it would be better to do the following
mv /usr /usr1
mkdir -p 755 /usr

then after step 10 (the reboot) just clean up after your self
rm -rf1 /usr1

Reply
palov@bio.bas.bg

I\’d better mount the already populated \”new\” /dev/hdc1 over existing /usr, after step 7 and make some checks, and then fix /etc/fdisk in step 8; then I will mount /dev/hda1 on a temporary mount point, e.g. /mnt/tmp and there will remove /mnt/tmp/usr/* – cause I\’m not sure how happy will be a running system with an empty /usr between steps 9 and 10! Maybe, before removing the old /usr/* I\’d make an \”rsync -avn /mnt/tmp/usr/ /usr/\” as a last check?

Regards,

Petko

Reply
dbayer

I\’m with eperry on this one. If something were to go wrong, it makes for a much quicker & easier fix, while at the same time adding only slightly to the work required to add the disk.

Reply
symeonb

Interesting perhaps, but nowhere in the title or resume does this mention it is infact for a vbox ! ? so perhaps not all that useful for many linux admins.

Reply
dragonwisard

@symeonb: I agree, it seems misleading. However I think it\’s implied that the same technique he demonstrates with virtual disks could be replicated in meat-space with physical disks.

Reply
cldavisjr

Great article! Simple and straight forward. Many different scenarios exist for this problem. Your example of it happening with a VM is appropriate. Once adding a new \”disk\” for your VM your solution follows and given no errors it is simple and easy to verify. With a VM (at least ours) you can take a snapshot of the current VM image and should a problem arise after editing the /etc/fstab or related to the newly created partition you can reboot into the backup VM image, mount the new partition on a temporary mount point and troubleshoot.
How about an additional article on the new file system features that allow dynamically growing a partition in a non-VM environment? They are a boon to the Linux community and have added to Linux\’ ability to stay at the top of the server world.

Reply
knightperson

While a reasonable option in some situations, this solution only works if it\’s a system that can be shut down while you do the maintenance. If you don\’t boot to single user mode, there\’s the possibility that the old and new copies of /usr will end up different, or if you use mv then you will have parts of the OS not working when their /usr files suddenly disappear. In my opinion, a better solution, although it requires some forethought when building the machine, is to use LVM. With LVM and modern filesystems (most of the ones newer than ext3 can do this), you can do this all without rebooting the system or even unmounting anything.

Going from memory, the steps would be
1) create the new partition with a hex ID of LVM physical volume (sfdisk or similar)
2) Put an LVM physical volume on the partition (pvcreate)
3) add it to the LVM system (vgchange, I think)
4) extend the volume group used by / to include the new physical volume (lvextend)
5) resize and expand the root filesystem (depends on the filesystem)

It\’s still a good idea to have your partitions separated when you build the system, of course, but LVM gives you much more flexibility in growing their sizes when necessary.

Reply

I appreciate you sharing this blog article.Really thank you! Really Great.

Reply

I discovered your blog site on google and verify a number of of your early posts. Continue to keep up the excellent operate. I simply extra up your RSS feed to my MSN News Reader. Searching for forward to studying extra from you in a while!…

Reply

There are some attention-grabbing deadlines in this article but I don’t know if I see all of them center to heart. There is some validity however I’ll take maintain opinion until I look into it further. Good article , thanks and we wish more! Added to FeedBurner as effectively

Reply

    je ne vois pas où est le débat. si l’on enlève les deux mois d’été -non rémunérés donc-, il nous reste 7 semaines et demi. soit moins que certains de mes amis dans le privé, qui, une fois en vacances, sont réellement en vacances et ne s&;uasorqvancent pas dans leur boulot. mais je ne les envie pas, ne les étiquette pas comme étant privilégiés, parce que j’ai moi aussi eu la possibilité de choisir mon métier, P.E., et j’ai aussi celle de le quitter si je le souhaite.

    Reply

This weibste makes things hella easy.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *

*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>