Mua hàng tại Link Shopee hoặc Hotline 0345-148-136.

ZFS autobackup and restore

Data backup is important things we need to do. I’m using ZFS because I can do it in easy way.

There are many tools to do it:

  • Sanoid – These are policy-driven snapshot management and replication tools which use OpenZFS for underlying next-gen storage.
  • ZFS-autobackup – ZFS autobackup is used to periodicly backup ZFS filesystems to other locations. Easy to use and very reliable
  • ZnapZend – zfs backup with remote capabilities and mbuffer integration
  • zrepl – One-stop ZFS backup & replication solution

I’m using ZFS-autobackup just because I have experience of it before.

I follow the docs. zfs-autobackup cli can be learn by:

$ zfs-autobackup --help
usage: zfs-autobackup [--help] [--test] [--verbose] [--debug] [--debug-output] [--progress] [--utc] [--version]
                      [--ssh-config CONFIG-FILE] [--ssh-source USER@HOST] [--ssh-target USER@HOST]
                      [--property-format FORMAT] [--snapshot-format FORMAT] [--hold-format FORMAT] [--strip-path N]
                      [--exclude-unchanged BYTES] [--exclude-received] [--no-snapshot] [--pre-snapshot-cmd COMMAND]
                      [--post-snapshot-cmd COMMAND] [--min-change BYTES] [--allow-empty] [--other-snapshots]
                      [--set-snapshot-properties PROPERTY=VALUE,...] [--no-send] [--no-holds]
                      [--clear-refreservation] [--clear-mountpoint] [--filter-properties PROPERTY,...]
                      [--set-properties PROPERTY=VALUE,...] [--rollback] [--force] [--destroy-incompatible]
                      [--ignore-transfer-errors] [--decrypt] [--encrypt] [--zfs-compressed] [--compress [TYPE]]
                      [--rate DATARATE] [--buffer SIZE] [--send-pipe COMMAND] [--recv-pipe COMMAND] [--no-thinning]
                      [--keep-source SCHEDULE] [--keep-target SCHEDULE] [--destroy-missing SCHEDULE]
                      [BACKUP-NAME] [TARGET-PATH]

zfs-autobackup v3.2 - (c)2022 E.H.Eefting ([email protected])

positional arguments:
  BACKUP-NAME           Name of the backup to select
  TARGET-PATH           Target ZFS filesystem (optional)

Common options:
  --help, -h            show help
  --test, --dry-run, -n
                        Dry run, dont change anything, just show what would be done (still does all read-only
                        operations)
  --verbose, -v         verbose output
  --debug, -d           Show zfs commands that are executed, stops after an exception.
  --debug-output        Show zfs commands and their output/exit codes. (noisy)
  --progress            show zfs progress output. Enabled automaticly on ttys. (use --no-progress to disable)
  --utc                 Use UTC instead of local time when dealing with timestamps for both formatting and parsing.
                        To snapshot in an ISO 8601 compliant time format you may for example specify --snapshot-
                        format "{}-%Y-%m-%dT%H:%M:%SZ". Changing this parameter after-the-fact (existing snapshots)
                        will cause their timestamps to be interpreted as a different time than before.
  --version             Show version.

SSH options:
  --ssh-config CONFIG-FILE
                        Custom ssh client config
  --ssh-source USER@HOST
                        Source host to pull backup from.
  --ssh-target USER@HOST
                        Target host to push backup to.

String formatting options:
  --property-format FORMAT
                        Dataset selection string format. Default: autobackup:{}
  --snapshot-format FORMAT
                        ZFS Snapshot string format. Default: {}-%Y%m%d%H%M%S
  --hold-format FORMAT  ZFS hold string format. Default: zfs_autobackup:{}
  --strip-path N        Number of directories to strip from target path.

Selection options:
  --exclude-unchanged BYTES
                        Exclude datasets that have less than BYTES data changed since any last snapshot. (Use with
                        proxmox HA replication)
  --exclude-received    Exclude datasets that have the origin of their autobackup: property as "received". This can
                        avoid recursive replication between two backup partners.

Snapshot options:
  --no-snapshot         Don't create new snapshots (useful for finishing uncompleted backups, or cleanups)
  --pre-snapshot-cmd COMMAND
                        Run COMMAND before snapshotting (can be used multiple times.
  --post-snapshot-cmd COMMAND
                        Run COMMAND after snapshotting (can be used multiple times.
  --min-change BYTES    Only create snapshot if enough bytes are changed. (default 1)
  --allow-empty         If nothing has changed, still create empty snapshots. (Same as --min-change=0)
  --other-snapshots     Send over other snapshots as well, not just the ones created by this tool.
  --set-snapshot-properties PROPERTY=VALUE,...
                        List of properties to set on the snapshot.

Transfer options:
  --no-send             Don't transfer snapshots (useful for cleanups, or if you want a separate send-cronjob)
  --no-holds            Don't hold snapshots. (Faster. Allows you to destroy common snapshot.)
  --clear-refreservation
                        Filter "refreservation" property. (recommended, saves space. same as --filter-properties
                        refreservation)
  --clear-mountpoint    Set property canmount=noauto for new datasets. (recommended, prevents mount conflicts. same
                        as --set-properties canmount=noauto)
  --filter-properties PROPERTY,...
                        List of properties to "filter" when receiving filesystems. (you can still restore them with
                        zfs inherit -S)
  --set-properties PROPERTY=VALUE,...
                        List of propererties to override when receiving filesystems. (you can still restore them with
                        zfs inherit -S)
  --rollback            Rollback changes to the latest target snapshot before starting. (normally you can prevent
                        changes by setting the readonly property on the target_path to on)
  --force, -F           Use zfs -F option to force overwrite/rollback. (Useful with --strip-path=1, but use with
                        care)
  --destroy-incompatible
                        Destroy incompatible snapshots on target. Use with care! (implies --rollback)
  --ignore-transfer-errors
                        Ignore transfer errors (still checks if received filesystem exists. useful for acltype
                        errors)
  --decrypt             Decrypt data before sending it over.
  --encrypt             Encrypt data after receiving it.
  --zfs-compressed      Transfer blocks that already have zfs-compression as-is.

Data transfer options:
  --compress [TYPE]     Use compression during transfer, defaults to zstd-fast if TYPE is not specified. (gzip, pigz-
                        fast, pigz-slow, zstd-fast, zstd-slow, zstd-adapt, xz, lzo, lz4)
  --rate DATARATE       Limit data transfer rate in Bytes/sec (e.g. 128K. requires mbuffer.)
  --buffer SIZE         Add zfs send and recv buffers to smooth out IO bursts. (e.g. 128M. requires mbuffer)
  --send-pipe COMMAND   pipe zfs send output through COMMAND (can be used multiple times)
  --recv-pipe COMMAND   pipe zfs recv input through COMMAND (can be used multiple times)

Thinner options:
  --no-thinning         Do not destroy any snapshots.
  --keep-source SCHEDULE
                        Thinning schedule for old source snapshots. Default: 10,1d1w,1w1m,1m1y
  --keep-target SCHEDULE
                        Thinning schedule for old target snapshots. Default: 10,1d1w,1w1m,1m1y
  --destroy-missing SCHEDULE
                        Destroy datasets on target that are missing on the source. Specify the time since the last
                        snapshot, e.g: --destroy-missing 30d

Full manual at: https://github.com/psy0rz/zfs_autobackup

1. On source host (pve01)

1.1. Select filesystems to backup

# zfs set autobackup:offsite1=true rpool

zfs-autobackup will backup zfs datasets with properties autobackup:firev1=true. Edit offsite1 (BACKUP-NAME) and rpool (TARGET-PATH) as you need.

Verify it by

# zfs get -t filesystem,volume autobackup:offsite1

Exclude some ZFS datasets you don’t need to backup

# zfs set autobackup:offsite1=false rpool/tmp

Verify it by

# zfs get -t filesystem,volume autobackup:offsite1

2. On backup machine (offsite1)

I’m using pull backup from backup machine.

2.1. Generate an SSH key

I made SSH key for root with no password.

# ssh-keygen
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
...

Then I copy it to source host (pve01)

# ssh-copy-id root@pve01
/usr/bin/ssh-copy-id: INFO: Source of key(s) to be installed: "/root/.ssh/id_rsa.pub"
/usr/bin/ssh-copy-id: INFO: attempting to log in with the new key(s), to filter out any that are already installed
Password:

Number of key(s) added: 1

2.2. Running zfs-autobackup

Create a zfs dataset for backup

# zfs create data/backup/pve01

Then run zfs-autobackup test

# zfs-autobackup --test -v --clear-mountpoint --ssh-source root@pve01 offsite1 data/backup/pve01

If everything are OK run it again without --test

# zfs-autobackup -v --clear-mountpoint --ssh-source root@pve01 offsite1 data/backup/pve01

Add above script to your crontab to run it automatic.

3. Restore from backups

When thing goes wrong I can restore data from backups with ZFS send/recv.

From backup machine (offsite1) run

# zfs send data/backup/pve01/ANY_SNAPSHOT_YOU_WANT | ssh root@pve01 "zfs recv rpool/SOME_WHERE_YOU_WANT"

Please read on ZFS docs, it is quite easy.


Comments

Một bình luận cho “ZFS autobackup and restore”

  1. […] Tự động khởi động máy backup vào 3g sáng, sử dụng SSH pull dữ liệu về mỗi khi khởi động, tự động tắt máy khi việc sao lưu hoàn thành. Tham khảo tại đây. […]

Trả lời

Email của bạn sẽ không được hiển thị công khai. Các trường bắt buộc được đánh dấu *