Logical Volume Management (LVM) on Linux: A great tool for creating pools of storage hardware that can be divided, resized, or used for snapshots.
LVM Snapshots
It’s impossible to go over all or many of the usage cases for LVM, but one that stands out is the snapshotting capability of LVM. If you have used LVM to create LV’s and then built file systems on those LV’s it’s very simple to create a snapshot of the file system using LVM. The key factor in the process is that you need to make sure that you have enough space on a VG that hasn’t been allocated that can be used by an LV (the snapshot LV). You should not create an LV (Logical Volume) prior to the snapshot because during the snapshot process you will be creating the LV using the lvcreate command.
In preparation for the snapshot you need to examine the file system to determine it’s size (“du -sh”). This will tell you the approximate size of the snapshot. For example, if the file system says it has 29G used (29 GB), then you should snapshot a little larger – perhaps 30-32 GB (always good to have a cushion). This cushion isn’t strictly necessary, but a little safety never hurts.
You can check how much space is used for the snapshot by using the command lvdisplay. This link discusses this. In addition as recommended in this link it's a good practice to plan for the worse and allocate the same amount of space as the original file system - just to be sure. You can always move the snapshot or erase it if you need the space back.
Each file system, ext2, ext3, xfs, reiserfs, jfs, etc. has it's own unique behavior when a snapshot is taken using LVM. It is recommended that you experiment with your file system of choice on a small scale before committing it to production. For example, when taking a snapshot of an xfs file system, you should run a command, "xfs_freeze" just prior to taking the snapshot. Also, ext3 file systems like to be mounted as read-write after a snapshot so the journal can be replayed (most of the time, a snapshot is mounted read-only so that backups of the snapshot don't have to worry about inadvertent writing).
To help understand snapshots a little better Figure 2 below is a variation of Figure 1 but the /data file system is not there and instead that space is left open on the VG for snapshots. There is also a single VG to make things a little easier.
Figure 2 - LVM Concepts for Snapshots
The snapshot will be taken using the remaining space in the VG. In this case, the remaining space available is the same as that allocated to /home. Also, as noted in the figure, the file system for /home is ext3.
For a simple test case, the 2.6.30 kernel has been copied to /home/user5. The space used in /home is,
# du -sh /home
463M /home
So there is 463 MB of space used in the file system built on top of LVM. So the snapshot should be slightly larger than this.
Now, let's take the snapshot using "lvcreate".
# lvcreate -L500M -s -n home_backup_080309 /dev/primary_vg/home_lv
Logical volume "home_backup_080309" created
The option "-s" tells lvcreate to take a snapshot. The command creates a new LV, home_backup_080309, on the VG, primary_vg. The snapshot is of the LV /dev/primary_vg/home_lv. To see if the LV was created successfully, just use "lvscan".
# lvscan
ACTIVE Original '/dev/primary_vg/home_lv' [450.00 GB] inherit
ACTIVE Snapshot '/dev/primary_vg/home_backup_080309' [500.00 MB] inherit
You can see the snapshot (it's even labeled "Snapshot") and the full path to the snapshot.
If you want to make a backup of the snapshot (one of the primary reasons for making a snapshot) then you need to mount it.
# mkdir /mnt/backup/home_backup_080309
# mount /dev/primary_vg/home_lv/home_backup_080309 /mnt/backup/home_backup_080309
# mount
/dev/hda3 on / type ext3 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
/dev/hda1 on /boot type ext2 (rw)
tmpfs on /dev/shm type tmpfs (rw)
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw)
/dev/mapper/primary_vg-home_lv on /home type ext3 (rw,data=ordered)
/dev/mapper/primary_vg-home_backup_080309 on /mnt/backup/home_backup_080309 type ext3 (rw)
It's always a good idea to check the mounted snapshot to make sure it's there.
# ls -s /mnt/backup/home_backup_080309/
total 20
16 lost+found 4 user5
# ls -s /mnt/backup/home_backup_080309/user5
total 74080
4 linux-2.6.30 74076 linux-2.6.30.tar.gz
Everything is there and is ready to be backed up. If you like, you could change the file system to ext2 and mount it read-only (the ext2 type will eliminate the need to read the journal so it will be easier to mount the snapshot read-only).
GUI Tools
The command line tools for LVM are not too complicated but for novices it can be a bit daunting. So to help here are 3 GUI tools for LVM.
It is beyond the scope of this article to review these GUI's but if you are so inclined, give them a try. EVMS, in particular, is a very powerful tool with it's own volume management capability but it can be used with LVM as well. It was developed from an IBM tool of the same name. It has some very powerful methods that make your life much easier. However, with all GUI tools it is recommended you keep detailed notes on the LVM configuration.
Summary
This is a quick review of LVM. For people new to LVM it presents some of the basic concepts and how they can be used. It's not intended to be an encompassing tutorial for LVM, but rather a quick overview. Further details are left to better more lengthy tutorials on the web. There are some GUI tools that can help gradually introduce LVM as previously pointed out.
In addition, this article reviews how to use LVM for creating snapshots. This is a very powerful tool that should be in the arsenal of all users and admins. It is straightforward to use but you need to make sure you understand the interactions with the specific file system(s). Just be sure you have enough space on your VG to store the snapshot (and don't forget to erase it when you are done).
LVM is one of the tools that people forget about and sometimes even ignore. Many installers for various distributions will use LVM as default (RHEL, CentOS, and Fedora are three that immediately come to mind but there are definitely more), and make things much easier for the novice. However, you still need to understand how the installer built the VG's and the LV's. Hopefully this article gave you a refresher on LVM (a quick one) or introduced you to a great tool for Linux.
Jeff Layton is an Enterprise Technologist for HPC at Dell. He can be found lounging around at a nearby Frys enjoying the coffee and waiting for sales (but never during working hours).
Comments on "Storage Pools and Snapshots with Logical Volume Management"
EVMS? eh? this isn\’t a gui tool it is an alternative volume management systems written mostly by IBM.
Actually, if you look at the EVMS webpage (linked in the article) you\’ll see that EVMS may be an alternative VMS, but it was also written (along with a GUI) to work with Linux LVM and Linux MD/Software RAID devices.
Great article! The only thing that I thought was slightly unclear was the size heuristic for the snapshot sizing. The referenced method allows for slightly more than 100% change in existing data and requires the VG to contain at least that much free space. While there is definitely a value in such a large COW cache, it is not always possible to have that much coverage. For sake of clarity, it bears mention that the \”size\” given to the snapshot is the cache for changes made to data existing at the time of the snapshot, and does not have to be nearly as large as the existing data, but large enough to cover the changes incurred during the expected life of the snapshot.
It also may have been good to cover the removal of the snapshot as well, although any user delving into LVM should learn lvremove quickly.
Next up LVM+MD?
This shows an example of combining physical partitions into a VG. Is there a good reason to divide a physical device (/dev/sdb1, /dev/sdb2) and then recombine those partitions into /dev/primary_vg?
This was a good review. Here is another tip if you plan to use LVM effectively. These suggestions will vary based on the distribution that you use, so KNOW that.
I tend to split up the system so that it is easier to manage file systems that tend to grow often. For instance if you take the OS, and separate the DATA, then you might end up with something like this.
/dev/os/root
/dev/os/usr
/dev/os/tmp
/dev/os/opt
/dev/os/swap
and the data you may put in a completely different VG like so.
/dev/data/home
/dev/data/var
/dev/data/srv
/dev/data/tmp * some people may choose to do this for systems that do a lot of video and sound editing.
I have experimented, and if you design your servers will then you can utilize your space better, and adapt to any storage issues that may present themselves in the future. I try to leave extra unallocated space in each volume group just in case a file system fills up, then this will allow you to extend the logical volume and the file system resides on that volume. Plan which filesystems you use carefully! Some filesystems (and distribution admin tools) will allow you to extend logical volume and filesystem in one command while others could turn out to be an ordeal.
Hope that helps someone.
http://intelliginix.com
Why doesn\’t the article mention the performance penalties involved with lvm snapshots ? For us this was the main reason not to build storage boxes based on linux, instead we use Opensolaris / ZFS for these solutions now. It is expected these problems will be solved when BRTFS is commonly available.
Thanks everyone for the comments and the suggestions I greatly appreciate.
Here are some quick replies and/or comments:
@clowenstein
The example I used in the article was a bit artificial but I wanted to show how you could combine partitions or disks into a VG. One advantage of splitting a disk into partitions and then combining them with LVM is that you could have created stripes across the PV\’s to improve performance. Then again you could have used MD to create RAID groups and also improved performance. I\’ve never dug into the interaction between LVM and MD so I don\’t know what the \”best\” configuration is (however you want to define \”best\”).
@thatblackguy:
Cool approach! I think it takes some work to setup everything as you describe but it does give you much more flexibility than a monolithic approach. Very cool – Thanks!
@mxcreep
The article wasn\’t intended to be an all-encompassing review of LVM with all of the pitfalls and benefits (I think I mentioned that twice which someone pointed out was bad writing – but oh well, I wanted to make sure my point came across). However, the performance penalty doesn\’t really come from LVM. It comes from the fact that the file system has no snapshot capability built in. Consequently, you have to reply on LVM to take the snapshot for you. This forces you to \”freeze\” the file system (so to speak) during the snapshot and means that the file system takes a performance hit during the snapshot.
I also have to be the bearer of bad tidings, but even ZFS suffers a little when a snapshot is taken. The performance penalty is less than having to use ZFS but it\’s still there.
The only time you won\’t see a performance penalty due to a snapshot is for a log based file system because it is designed for snapshots. That\’s one of the really cool features of log based file systems.
I hope that answers some questions. If it doesn\’t feel free to repost and we\’ll figure out the answer together. If it even gets too long, we can write a quick article about the question and the solution (hint, hint) :)
Thanks!
Jeff
@Jeff,
My point about performance wasn\’t really about the moment the snapshot was taken but more afterwards. A logical volume with one or more snapshots chained to it performs a lot slower.
Sorry to be late replying. You said:
- – - – -
@clowenstein
The example I used in the article was a bit artificial but I wanted to show how you could combine partitions or disks into a VG. One advantage of splitting a disk into partitions and then combining them with LVM is that you could have created stripes across the PV\’s to improve performance. Then again you could have used MD to create RAID groups and also improved performance.
- – - – -
Surely you realize that a single disk drive has only one head mechanism. Creating stripes across partitions of one drive can only cause time-consuming head-seek motion. This is definitely not a performance improver.
@clownenstein
Sorry I should have said \”disks\”.
You are correct for a disk (that is the singular form). It\’s not always a good idea (there are situations where it might be reasonable).
For multiple disks (that\’s plural indicating more than one) can give you a performance advantage but it too depends upon the exact configuration.
Jeff