ZFS autobackup and restore

Data backup is important things we need to do. I’m using ZFS because I can do it in easy way.

There are many tools to do it:

• Sanoid – These are policy-driven snapshot management and replication tools which use OpenZFS for underlying next-gen storage.
• ZFS-autobackup – ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable
• ZnapZend – zfs backup with remote capabilities and mbuffer integration
• zrepl – One-stop ZFS backup & replication solution

I’m using ZFS-autobackup just because I have experience of it before.

I follow the docs. zfs-autobackup cli can be learn by:

$ zfs-autobackup --help
usage: zfs-autobackup [--help] [--test] [--verbose] [--debug] [--debug-output] [--progress] [--utc] [--version]
                      [--ssh-config CONFIG-FILE] [--ssh-source USER@HOST] [--ssh-target USER@HOST]
                      [--property-format FORMAT] [--snapshot-format FORMAT] [--hold-format FORMAT] [--strip-path N]
                      [--exclude-unchanged BYTES] [--exclude-received] [--no-snapshot] [--pre-snapshot-cmd COMMAND]
                      [--post-snapshot-cmd COMMAND] [--min-change BYTES] [--allow-empty] [--other-snapshots]
                      [--set-snapshot-properties PROPERTY=VALUE,...] [--no-send] [--no-holds]
                      [--clear-refreservation] [--clear-mountpoint] [--filter-properties PROPERTY,...]
                      [--set-properties PROPERTY=VALUE,...] [--rollback] [--force] [--destroy-incompatible]
                      [--ignore-transfer-errors] [--decrypt] [--encrypt] [--zfs-compressed] [--compress [TYPE]]
                      [--rate DATARATE] [--buffer SIZE] [--send-pipe COMMAND] [--recv-pipe COMMAND] [--no-thinning]
                      [--keep-source SCHEDULE] [--keep-target SCHEDULE] [--destroy-missing SCHEDULE]
                      [BACKUP-NAME] [TARGET-PATH]

zfs-autobackup v3.2 - (c)2022 E.H.Eefting ([email protected])

positional arguments:
  BACKUP-NAME           Name of the backup to select
  TARGET-PATH           Target ZFS filesystem (optional)

Common options:
  --help, -h            show help
  --test, --dry-run, -n
                        Dry run, dont change anything, just show what would be done (still does all read-only
                        operations)
  --verbose, -v         verbose output
  --debug, -d           Show zfs commands that are executed, stops after an exception.
  --debug-output        Show zfs commands and their output/exit codes. (noisy)
  --progress            show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)
  --utc                 Use UTC instead of local time when dealing with timestamps for both formatting and parsing.
                        To snapshot in an ISO 8601 compliant time format you may for example specify --snapshot-
                        format "{}-%Y-%m-%dT%H:%M:%SZ". Changing this parameter after-the-fact (existing snapshots)
                        will cause their timestamps to be interpreted as a different time than before.
  --version             Show version.

SSH options:
  --ssh-config CONFIG-FILE
                        Custom ssh client config
  --ssh-source USER@HOST
                        Source host to pull backup from.
  --ssh-target USER@HOST
                        Target host to push backup to.

String formatting options:
  --property-format FORMAT
                        Dataset selection string format. Default: autobackup:{}
  --snapshot-format FORMAT
                        ZFS Snapshot string format. Default: {}-%Y%m%d%H%M%S
  --hold-format FORMAT  ZFS hold string format. Default: zfs_autobackup:{}
  --strip-path N        Number of directories to strip from target path.

Selection options:
  --exclude-unchanged BYTES
                        Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with
                        proxmox HA replication)
  --exclude-received    Exclude datasets that have the origin of their autobackup: property as "received". This can
                        avoid recursive replication between two backup partners.

Snapshot options:
  --no-snapshot         Don't create new snapshots (useful for finishing uncompleted backups, or cleanups)
  --pre-snapshot-cmd COMMAND
                        Run COMMAND before snapshotting (can be used multiple times.
  --post-snapshot-cmd COMMAND
                        Run COMMAND after snapshotting (can be used multiple times.
  --min-change BYTES    Only create snapshot if enough bytes are changed. (default 1)
  --allow-empty         If nothing has changed, still create empty snapshots. (Same as --min-change=0)
  --other-snapshots     Send over other snapshots as well, not just the ones created by this tool.
  --set-snapshot-properties PROPERTY=VALUE,...
                        List of properties to set on the snapshot.

Transfer options:
  --no-send             Don't transfer snapshots (useful for cleanups, or if you want a separate send-cronjob)
  --no-holds            Don't hold snapshots. (Faster. Allows you to destroy common snapshot.)
  --clear-refreservation
                        Filter "refreservation" property. (recommended, saves space. same as --filter-properties
                        refreservation)
  --clear-mountpoint    Set property canmount=noauto for new datasets. (recommended, prevents mount conflicts. same
                        as --set-properties canmount=noauto)
  --filter-properties PROPERTY,...
                        List of properties to "filter" when receiving filesystems. (you can still restore them with
                        zfs inherit -S)
  --set-properties PROPERTY=VALUE,...
                        List of propererties to override when receiving filesystems. (you can still restore them with
                        zfs inherit -S)
  --rollback            Rollback changes to the latest target snapshot before starting. (normally you can prevent
                        changes by setting the readonly property on the target_path to on)
  --force, -F           Use zfs -F option to force overwrite/rollback. (Useful with --strip-path=1, but use with
                        care)
  --destroy-incompatible
                        Destroy incompatible snapshots on target. Use with care! (implies --rollback)
  --ignore-transfer-errors
                        Ignore transfer errors (still checks if received filesystem exists. useful for acltype
                        errors)
  --decrypt             Decrypt data before sending it over.
  --encrypt             Encrypt data after receiving it.
  --zfs-compressed      Transfer blocks that already have zfs-compression as-is.

Data transfer options:
  --compress [TYPE]     Use compression during transfer, defaults to zstd-fast if TYPE is not specified. (gzip, pigz-
                        fast, pigz-slow, zstd-fast, zstd-slow, zstd-adapt, xz, lzo, lz4)
  --rate DATARATE       Limit data transfer rate in Bytes/sec (e.g. 128K. requires mbuffer.)
  --buffer SIZE         Add zfs send and recv buffers to smooth out IO bursts. (e.g. 128M. requires mbuffer)
  --send-pipe COMMAND   pipe zfs send output through COMMAND (can be used multiple times)
  --recv-pipe COMMAND   pipe zfs recv input through COMMAND (can be used multiple times)

Thinner options:
  --no-thinning         Do not destroy any snapshots.
  --keep-source SCHEDULE
                        Thinning schedule for old source snapshots. Default: 10,1d1w,1w1m,1m1y
  --keep-target SCHEDULE
                        Thinning schedule for old target snapshots. Default: 10,1d1w,1w1m,1m1y
  --destroy-missing SCHEDULE
                        Destroy datasets on target that are missing on the source. Specify the time since the last
                        snapshot, e.g: --destroy-missing 30d

Full manual at: https://github.com/psy0rz/zfs_autobackup

1. On source host (pve01)

1.1. Select filesystems to backup

# zfs set autobackup:offsite1=true rpool

zfs-autobackup will backup zfs datasets with properties autobackup:firev1=true. Edit offsite1 (BACKUP-NAME) and rpool (TARGET-PATH) as you need.

Verify it by

# zfs get -t filesystem,volume autobackup:offsite1

Exclude some ZFS datasets you don’t need to backup

# zfs set autobackup:offsite1=false rpool/tmp

Verify it by

# zfs get -t filesystem,volume autobackup:offsite1

2. On backup machine (offsite1)

I’m using pull backup from backup machine.

2.1. Generate an SSH key

I made SSH key for root with no password.

# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
...

Then I copy it to source host (pve01)

# ssh-copy-id root@pve01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
Password:

Number of key(s) added: 1

2.2. Running zfs-autobackup

Create a zfs dataset for backup

# zfs create data/backup/pve01

Then run zfs-autobackup test

# zfs-autobackup --test -v --clear-mountpoint --ssh-source root@pve01 offsite1 data/backup/pve01

If everything are OK run it again without --test

# zfs-autobackup -v --clear-mountpoint --ssh-source root@pve01 offsite1 data/backup/pve01

Add above script to your crontab to run it automatic.

3. Restore from backups

When thing goes wrong I can restore data from backups with ZFS send/recv.

From backup machine (offsite1) run

# zfs send data/backup/pve01/ANY_SNAPSHOT_YOU_WANT | ssh root@pve01 "zfs recv rpool/SOME_WHERE_YOU_WANT"

Please read on ZFS docs, it is quite easy.