Alex headshot

AlBlue’s Blog

Macs, Modularity and More

Crontab generated ZFS snapshots

2008, mac, zfs

Having set up ZFS on my server's partitions now, I wanted to get an automated backup going. There's a few posts on automatic ZFS snapshots kicking around (see ZFS Automatic Snapshots in Nevada 100 and Jeff's snapshot script) but the former was heavily tied into Solaris' management system, whilst the latter just created thousands of snapshots nearly indistinguishable from another.

So, without further fanfare, here is AlBlue's crontab generated ZFS snapshots (with apologies in advance for Blogger's rendering:

#Pool is called 'Data', search and replace for yours ...
@reboot /usr/sbin/zpool scrub Data
@daily  /usr/sbin/zpool scrub Data
@hourly /usr/sbin/zpool status Data | /usr/bin/egrep -q "scrub completed|none requested" && /usr/sbin/zfs snapshot -r Data@AutoH-`date +"\%FT\%H:\%M"`
@daily  /usr/sbin/zfs snapshot -r Data@AutoD-`date +"\%F"`
@weekly /usr/sbin/zfs snapshot -r Data@AutoW-`date +"\%Y-\%U"`
@monthly /usr/sbin/zfs snapshot -r Data@AutoM-`date +"\%Y-\%m"`
@yearly /usr/sbin/zfs snapshot -r Data@AutoY-`date +"\%Y"`
# do a spot of housecleaning - somewhat assumes the daily ones have run ..
@hourly /usr/sbin/zpool status Data | /usr/bin/egrep -q "scrub completed|none requested" && /usr/sbin/zfs list -t snapshot -o name | /usr/bin/grep Data@AutoH- | /usr/bin/sort -r | /usr/bin/tail -n +26 | /usr/bin/xargs -n 1 /usr/sbin/zfs destroy -r
@daily  /usr/sbin/zfs list -t snapshot -o name | /usr/bin/grep Data@AutoD- | /usr/bin/sort -r | /usr/bin/tail -n +9 | /usr/bin/xargs -n 1 /usr/sbin/zfs destroy -r
@weekly /usr/sbin/zfs list -t snapshot -o name | /usr/bin/grep Data@AutoW- | /usr/bin/sort -r | /usr/bin/tail -n +7 | /usr/bin/xargs -n 1 /usr/sbin/zfs destroy -r
@monthly /usr/sbin/zfs list -t snapshot -o name | /usr/bin/grep Data@AutoM- | /usr/bin/sort -r | /usr/bin/tail -n +14 | /usr/bin/xargs -n 1 /usr/sbin/zfs destroy -r

Caveats:

  • Use at your own risk
  • This works for a ZFS pool called 'Data', because that's what mine is called. You can use any name you want.
  • It's a bit fast-and-ugly, but that was good for me at the time. You could pull it out in a script and parameterise the pool name(s) if you wanted.
  • It would be much better to do this on individual file sets which have a particular property set to control which file systems get handed; this does everything. Again, met my needs, might not meet yours. Tim's approach is to use custom attributes; you could do this to select hourly or otherwise backups.
  • This deletes snapshots starting with Auto[HDWMY]-, so don't call your own ones that unless you want to kiss them goodbye.

The cleanup routine helps here. It uses some script kung-fu, and relies on the fact that lexicographic sorting of the snapshot names (for each subset of [HDWMY]) is also the time ordering. The plan is to keep 24xH snapshots, then 7xD snapshots, then 5xW snapshots and then don't delete yearlies at all. However, (a) there's an off-by-one in my kung-fu, and you really want to let the weekly complete before deleting the last daily. So I added a couple in each case for a safety margin.

Note that if you run this, the snapshots consume space ... be warned! Even if you delete something, it could be kicking around in your snapshot space somewhere (although they'll helpfully tell you which ones are causing the problems). Here's what my snapshots look like now:

Data/Users/alex@AutoD-2008-11-21
Data/Users/alex@AutoD-2008-11-22
Data/Users/alex@AutoH-2008-11-21T23:40
Data/Users/alex@AutoH-2008-11-22T00:21
Data/Users/alex@AutoH-2008-11-22T00:36
Data/Users/alex@AutoH-2008-11-22T00:41
Data/Users/alex@AutoH-2008-11-22T00:51
Data/Users/alex@AutoH-2008-11-22T01:00
Data/Users/alex@AutoM-2008-11
Data/Users/alex@AutoW-2008-46
Data/Users/alex@AutoY-2008

Please feel free to leave feedback, copy and improve on it. A link back here would also be appreciated; you can keep up to date with all the ZFS tagged posts. My thanks to Mike for pointing out an improvement to the hourly snapshot generation guard.