“Notice that when you use \”–inplace\”, rsync writes directly on-top of the already existing file in the destination filesystem, and ONLY in the places that changed! This means that by using ZFS (which is copy-on-write) you get the BLOCK-level deduplication…”

Have you actually observed this in practice or is this just theory?

I am attempting a similar backup scheme using rsync –inplace with the destination file residing on a NetApp WAFL filesystem (which is also copy-on-write like ZFS). What I have found is that even with the –inplace option rsync still rewrites the entire file on the destination block by block (note I am also using the –no-whole-file option). It is better than without –inplace, as by default rsync would create an entirely new temporary copy of the file on the destination before overwriting the original (thus causing the COW filesystem to incur a 200% penalty in snapshot space utilization). However, I find that rsync –inplace does not update only the changed blocks on the destination file as you described, rather it still rewrites the whole file “inplace”.

The only advantage to the –inplace option that I see so far is that you don’t need double the storage space on the destination to temporarily keep 2 copies of the file being rsync’ed.

I’ve been running “almost” this exact backup scheme for my clients for years. A major limitation is exposed when a client renames a directory, the backup (in this case rsync) will see this as a new directory, and it will needlessly copy it over and delete the old directory, essentially doubling the space used by that directory.

Example: Client renames /home/user/15TB_Folder to /home/user/15Terabyte_Folder.

What would happen then? No, seriously. I’m wondering how your snapshot scenario would deal with that, since I don’t use snapshots, I use hardlinks, much like BackupPC and Rsnapshot.

NOTE: I’ve also used the nilfs2 LFS combined with the –inplace rsync option with very good results.

deduper.pl /tmp/dupeslog

…….where deduper.pl is:
#!/usr/bin/perl

$file = shift;
open FILE, $file or die \”no good $file open \\n\”;
$mode = shift;

$cnt=0;
while (<FILE>) {
chomp; # just because!
if ( /^\\s*$/ ) {
print \”save: \”, $paths[0], \”\\n\”;
for($i=1; $i<$cnt;$i++){
if($mode eq \’delete\’){ unlink ($paths[$i]); }
else { print \”delete: \” , $paths[$i], \”\\n\”; }
}
print \”\\n\\n\”;
$cnt=0;
}
else{
$paths[$cnt] = $_ ;
$cnt++;
}
}

Also, rsync uses some form of de-duplication at the file level when transferring files between remote systems, frequently used for backups as well, and also open source.

Just two open source apps that I\’ve been using for years, very mature, and work exceptionally well….

Downsides here are (special for an fs implemantation)… you have a higher (sometimes much higher) System load. Since HDD capacity is very very cheap it might be not suiteable to swap hdd space with cpu and ram load.

second think about FS crash.
the chance getting rid of more data than traditional methods is clearly much higher.

third. what is if youve to split your services to another server.
there youve to rethink youre hdd needs even for backup.

lets say youve 100gb backup space. now you split the server and youve got a second backupset… and surprise you need 150 gb now because you had many common files which are now on different backup sets.

only some quick assumptions..
shure the idea isnt new (windows has such feature a long time)
but im not a real fan of making additional much more complex changes so deep in the system.
specially for hdd capacity which is really unbeliveable cheap wen it will slowdown the performance and may complicate the menegement and failuler procedures

1. OpenSolaris backup server
2. ZFS snapshots
3. Rsync –inplace

Notice that when you use \”–inplace\”, rsync writes directly on-top
of the already existing file in the destination filesystem, and ONLY
in the places that changed! This means that by using ZFS (which is
copy-on-write) you get the BLOCK-level deduplication that you are
talking about… Taking a cron-based daily ZFS snapshot completes
the picture.

Using these tools, we are taking daily snapshots of HUGE VMWARE vmdk files that change in less than 1% of their contents on a daily basis,
using amazingly trivial space requirements (something like 3% of the size of the original vmdk is used for one month of daily backups).

I believe that OpenSolaris/ZFS/\”rsync –inplace\” is a combination
that merits a place in your article.

Kind regards,
Thanassis Tsiodras, Dr.-Ing.

The OSS BackupPC provides a crude level of dedupe via filesystem hard links . Therefore it only works on the file level and only across a single file system (but note that cheap single filesystems easily range into the 10s-100s of TB). For small to medium installations, BackupPC and the like can work well. It also can use rsync to transfer only changed blocks over the wire, which decreases bandwidth requirements.

You might note that all this proprietary dedupe technology effectively locks you to a vendor-specific implementation, which reduces your ability to escape when the vendor decides to jack prices.

Also notable is the falling-off-the-cliff price of disk. It might take more disk to ignore dedupe, but if it can be addressed by very cheap, flexible storage, that may weigh in its favor, especially if using a no-cost (tho admittedly less efficient) mechanism like hard links and rsync.

hjm

Les

It has been backing up 10+ systems to a central backup store here at our small business for years, and has compressed/deduped 1.7TB of backups into 220GB.

Better yet, I\’ve never been able to fault it.

I\’m intrigued by that other posters OpenSolaris+ZFS+rsync solution too. Very clever!