My ZFS replication script

On my new Debian GNU/kFreeBSD system, my backup strategy has changed. On the previous system, I relied on incremental dumps and a DAT160 tape drive, which has an 80GB uncompressed capacity. When you have a few hundred gigabytes of photos to back up, this is an inconvenient solution. On the new system, I am using multiple removable 3TB hard drives and a set of scripts built around zfs send/receive.

Cron runs the rep.py script 4 times a day, which does a zfs send | zfs receive pipeline for each filesystem to be backed up. On a semi-regular basis (I haven't yet decided on what schedule to do this on; with tape backups I did it less than once a month even though it was comparatively easier), I remove the drive to an off-site location, return the other drive from off-site, and insert it. (There's also fiddling with zpool import/export, of course)

The rep.py script relies on the zfs python module, also of my own creation. This module has facilities for inspecting and interacting with zfs filesystems, e.g., to list filesystems and snapshots, to create and destroy snapshots, and to run replication pipelines.

The rep.py script needs to be customized for your system. Customization items are:

TARGETS = ['bpool', 'cpool']
SRCS = ['mpool', 'rpool']

"SRCS" is a list of zpools which are replicated. "TARGETS" is a list of zpools to which backups are replicated. The first available pool out of TARGETS is chosen. (so if more than one TARGET is inserted, only one will ever be used)

You can designate individual filesystems as not replicated by setting the user property net.unpy.zreplicator:skip to the exact string "1", i.e.,

zfs set net.unpy.zreplicator:skip=1 examplepool/junkfiles

Files currently attached to this page:

rep.py1.5kB
zfs.py12.7kB

License: GPLv2+

Entry first conceived on 9 October 2013, 13:11 UTC, last modified on 11 October 2013, 18:18 UTC
Website Copyright © 2004-2024 Jeff Epler