Disk Space: The Final Frontier

Disk space is always at a premium but how do you handle it when you have a disk or filesystem that's full? After today, you'll know how to boldly go into that dark realm.

Your prime directive as a system administrator is to keep those systems whipped into shape and running without interfering with user’s daily operations. Sometimes the systems put you in a difficult position when their disks overflow with files. The simple solution is to remove files. But, what if you can’t do that? You have to add disk to your ailing system. But, how and where do you add that space? And, how do you keep this from happening again?

This article leads you through the discovery and remediation of disk space problems. Start your favorite virtual machine and follow along.

The Basics

The current scenario includes a Debian 5 virtual machine (VM) in VirtualBox 3.2 that currently uses two virtual disks: /dev/hda1 (5 GB) for / and /dev/hdb1 for /var (2 GB). The original 5GB wasn’t enough and the addition of the 2 GB /var disk came later to relieve those issues.

As you can see from the df, the original disk is full again.

$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             4.7G  4.4G  110M  98% /
tmpfs                 124M  8.0K  124M   1% /lib/init/rw
udev                   10M  684K  9.4M   7% /dev
tmpfs                 124M  140K  124M   1% /dev/shm
/dev/hdb1             2.0G  262M  1.7G  14% /var

This VM is a web server, therefore, it made sense to expand /var to alleviate any future problems with disk space for that service. Can you think of any other directories that would benefit from more space on this system? Let’s explore how a system administrator can fix this problem.

Space Exploration

The most likely candidates for expansion are /usr, /home and /opt. Current disk usage in each of these directories is as follows:

~$ du -sk /usr
3037496 /usr

$ du -sk /home
957060  /home

$ du -sk /opt
229468  /opt

Clearly, /usr is the space criminal in this scenario and the best candidate for expansion into its own virtual disk. To create a new disk for /usr, would also return 3GB of space to the / filesystem that includes /home and /opt for this system.

The ten steps involved in this expansion are:

  1. Create a new 5 GB virtual disk.
  2. Associate the new disk with the Debian VM.
  3. Boot the VM.
  4. Initialize the new disk.
  5. Format the disk.
  6. Temporarily mount the disk.
  7. Copy /usr to the mount point.
  8. Make the mount permanent with an entry in /etc/fstab.
  9. Remove the contents of /usr.
  10. Mount the new disk on /usr.

Step One: Create a 5 GB Virtual Disk

# VBoxManage createhd --filename Debian3.vdi --size 5000 --variant Fixed

Oracle VM VirtualBox Command Line Management Interface Version 3.2.4
(C) 2005-2010 Oracle Corporation
All rights reserved.

0%...10%...20%...30%...40%...50%...60%...70%...80%...90%...100%
Disk image created. UUID: f88dc8a9-dbf5-4ac4-845a-c73bc0de1f53

Step Two: Associate the new disk with the Debian VM (Debian5-1).

# VBoxManage storageattach Debian5-1 --storagectl "IDE Controller" --port 1 --device 0 --type hdd --medium Debian3.vdi

Step Three: Boot the VM.

# VBoxManage startvm Debian5-1

Login to the VM and become root or use sudo for these privileged command steps.

Step Four: Initialize the New Disk. In this step, you’ll use the sfdisk command to list the new disk device. Then, you’ll step through the simple initialization process.

Note: The following steps only display relevant output to decrease the length and complexity of the screens.

# sfdisk -l

Disk /dev/hdc: 10158 cylinders, 16 heads, 63 sectors/track

sfdisk: ERROR: sector 0 does not have an msdos signature
 /dev/hdc: unrecognized partition table type
No partitions found

From the sfdiskoutput, you see that the new disk’s device name is /dev/hdc. Since we’re dedicating the entire disk to /usr, initialization is greatly simplified. The following screen is an interactive command line session using fdisk.

# fdisk /dev/hdc

Command (m for help):n

Command action
   e   extended
   p   primary partition (1-4)
p

Partition number (1-4): 1
First cylinder (1-10158, default 1): 1
Last cylinder or +size or +sizeM or +sizeK (1-10158, default 10158): 10158

Command (m for help): p

Disk /dev/hdc: 5242 MB, 5242880000 bytes
16 heads, 63 sectors/track, 10158 cylinders
Units = cylinders of 1008 * 512 = 516096 bytes
Disk identifier: 0x761001bb

   Device Boot      Start         End      Blocks   Id  System
/dev/hdc1               1       10158     5119600+  83  Linux

Command (m for help):  w
The partition table has been altered!

Calling ioctl() to re-read partition table.
Syncing disks.

# sfdisk -l

Disk /dev/hdc: 10158 cylinders, 16 heads, 63 sectors/track
Units = cylinders of 516096 bytes, blocks of 1024 bytes, counting from 0

   Device Boot Start     End   #cyls    #blocks   Id  System
/dev/hdc1          0+  10157   10158-   5119600+  83  Linux
/dev/hdc2          0       -       0          0    0  Empty
/dev/hdc3          0       -       0          0    0  Empty
/dev/hdc4          0       -       0          0    0  Empty

Step Five: Format the new Disk (Create the filesystem).

# mkfs.ext3 /dev/hdc1

mke2fs 1.41.3 (12-Oct-2008)
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
320000 inodes, 1279900 blocks
63995 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=1312817152
40 block groups
32768 blocks per group, 32768 fragments per group
8000 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376, 294912, 819200, 884736

Writing inode tables: done
Creating journal (32768 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 35 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.

The new disk is ready to mount and use.

Step Six: Temporarily Mount the New Disk.

# mount /dev/hdc1 /mnt

# mount

/dev/hdc1 on /mnt type ext3 (rw)

Step Seven: Copy /usr to the Mount Point.


# cp -Rp /usr/* /mnt

Step Eight: Make the Mount Permanent with an Entry in /etc/fstab. Edit /etc/fstab and create the following entry.


/dev/hdc1       /usr            ext3    errors=remount-ro 0       1

Step Nine: Remove the contents of /usr.


# rm -rf /usr/*

Step Ten: Mount the New Disk on /usr.


# umount /mnt

# mount /dev/hdc1 /usr

mount

/dev/hdc1 on /usr type ext3 (rw)

The true test of your handiwork is to reboot the system and confirm that everything went as planned. If something goes wrong, your system will panic. The most likely culprits, should you have any problems, are that the /etc/fstab isn’t correctly setup or that the copy and mount didn’t work correctly. Follow the ten steps exactly as shown and you should have no problems.

If your system panics and doesn’t boot, boot the system using a rescue CD, mount the disks to see what went wrong, work through the steps again and take corrective action.

A quick look at the entire system now shows that the filesystems are happy and have plenty of useable space.


$ df -h

Filesystem            Size  Used Avail Use% Mounted on
/dev/hda1             4.7G  1.5G  3.1G  32% /
tmpfs                 124M  8.0K  124M   1% /lib/init/rw
udev                   10M  684K  9.4M   7% /dev
tmpfs                 124M     0  124M   0% /dev/shm
/dev/hdb1             2.0G  263M  1.7G  14% /var
/dev/hdc1             4.9G  3.1G  1.6G  67% /usr

Warp Speed Ahead

Taking what you’ve learned here, you can create a new disk for /home and one for /opt, if necessary. Systems run out of space. Filesystems fill up. And, it’s your job to fix it before it’s critical.

In the old days (1990s), systems could survive on two disks (not including any mirroring or RAID): A system disk and a “home” disk. The system disk held the standard filesystems, which were partitions or slices created by the system administrator to limit users from killing the system by filling /tmp or /home. The “home” disk, mounted on /home or /h kept user’s home directories isolated from the rest of the system. For a system administrator, managing user’s misuse of space is just another annoyance. These practices helped resolve that annoyance.

Some “old timers” still practice this kind of user isolation today. The rest of us use quotas (A topic of a future article).

Next week, you’ll pay a visit to Father Time and learn to lean on NTP to keep you in sync with the rest of the world.

Fatal error: Call to undefined function aa_author_bios() in /opt/apache/dms/b2b/linux-mag.com/site/www/htdocs/wp-content/themes/linuxmag/single.php on line 62