ARTICLES

Do your backups in the Cloud with Duplicity

By Valentin Bremond

January 4, 2020

Backups are a pain in the ass

Especially with my criteria:

Encryption of data at rest
Storage in France (at least in Europe)
No American or Chinese intermediate
Cheap (and especially: no hidden fees, like paying requests / incoming or outgoing data…)
Available everywhere and all the time
Incrementals (no need to copy 100 GB every single day)
Can configure retention and history
Preferably vendor-agnostic and open-source
Simple to configure and simple to reconfigure on a brand new PC (if it’s nearly impossible to restore backups on a new PC, we loose all the benefits of backups…)
No maintenance on whatever’s going to host this

Let’s filter out

We’ll start by removing what doesn’t comply with our criteria:

S3 / GCS / Azure: expensive and American
A dedicated / VM with some kind of OwnCloud: we would have to maintain it at the risk nothing will work the day we will need it (or that a nice hacker has messed up with our data)
A self-hosted NAS (even though building a ZFS RAID on a Raspberry Pi was very tempting)
A rsync on any standard hosting provider: no history, no encryption
Dedicated commercial software (i.e.: Blackbaze, Dropbox)

So what then?

Let’s keep it very simple: an OpenStack Swift bucket (open-source object storage, at OVH for example), a Duplicity on top of that and systemd to handle scheduling (or something else if you don’t like Lennart, but you will have to do it by yoursef then).

This meets all our criteria and it’s really cheap. For example, I have approximately 100 GB of data on my laptop that is backuped every day and it costs me ~0.09 € per month. I think we can afford 9 cents every month to backup our digital life.

TL;DR give me things to copy/paste

You need 3 things:

Duplicity
A bucket Swift, for instance here https://www.ovh.com/world/public-cloud/object-storage/
5 minutes from your life

Let’s start with installing everything

1$ sudo dnf install duplicity
2# ou
3$ sudo apt-get install duplicity

Personally, I had some Python libraries missing for Swift:

1$ sudo pip install python-swiftclient python-keystoneclient

Add the systemd configuration

You need 4 files:

~/.config/systemd/user/backups.service (the actual backup):

 1[Unit]
 2Description=Duplicity backups
 3After=network-online.target
 4Wants=network-online.target
 5OnFailure=backups-notify-failure.service
 6
 7[Service]
 8# With Type=simple, ExecStartPost commands will run a fraction of a second after the start of the backup, which will make
 9# Duplicity fail because of locks
10Type=oneshot
11TimeoutStartSec=12h
12EnvironmentFile=%h/.backups.env
13# Make sure the network is up and the OpenStack API is available
14ExecStartPre=/usr/bin/bash -c 'for i in $(seq 1 5); do test $(curl $SWIFT_AUTHURL --output /dev/null --write-out \'%{http_code}\' --silent --max-time 1) -eq 200 && exit 0 || echo "Waiting for OpenStack to be available" && sleep 3; done; echo "OpenStack is not available"; exit 1;'
15# Clean the bucket for artifacts from previous failures, if any
16ExecStartPre=/usr/bin/duplicity cleanup --force "$DEST"
17# Run the incremental backup
18ExecStart=/usr/bin/duplicity --verbosity notice --num-retries 3 --full-if-older-than "$FULL_EVERY" --asynchronous-upload --cf-backend swift $DUP_OPTS "$SRC" "$DEST"
19# Remove old backups
20ExecStartPost=/usr/bin/duplicity remove-older-than --force "$REMOVE_OLDER_THAN" "$DEST"
21ExecStartPost=/usr/bin/duplicity remove-all-inc-of-but-n-full --force "$KEEP_INCREMENTS" "$DEST"
22
23[Install]
24WantedBy=default.target

~/.config/systemd/user/backups.timer (the timer - it’s like a cron, but for systemd):

 1[Unit]
 2Description=Duplicity backups
 3After=network-online.target systemd-networkd-wait-online.service
 4Wants=network-online.target systemd-networkd-wait-online.service
 5
 6[Timer]
 7# Every 6 hours, starting at 8 AM (change this the way you like)
 8OnCalendar=*-*-* 08/6:00:00
 9Unit=backups.service
10
11[Install]
12WantedBy=default.target

~/.config/systemd/user/backups-notify-failure.service (this service will be launched in case of failure and a notification will be displayed; this will prevent you from missing backup errors and realize 6 months after that only the first backup worked)
```
1[Unit]
2Description=Displays an error notification in case of backup error
3
4[Service]
5Type=simple
6Environment=DISPLAY=:0
7ExecStart=/usr/bin/notify-send --urgency critical --icon dialog-warning "Duplicity backup failed"
```

~/.backups.env (to store your Swift token and potential Duplicity parameters):

 1# Duplicity
 2SRC="/home/val"
 3# Put here your Object Store container name - mine is called "backups"
 4DEST="swift://backups"
 5# You can exclude folders from your backups, see below
 6DUP_OPTS="--exclude /home/val/Downloads/ --exclude /home/val/.cache/ --exclude /home/val/.local/share/containers/"
 7
 8# Run a full backup every month
 9FULL_EVERY=1M
10# Remove backups older than 3 years
11REMOVE_OLDER_THAN=36M
12# Keep 3 full backups with increments, remove the increments for the older ones
13KEEP_INCREMENTS=3
14# All this means: run an incremental backup multiple times a day, run a full backup every month, keep backups for 3
15# years but only keep the last 3 incremental revisions (3 years worth of monthly backup and the last 3 months are daily
16# backups). This allows you to rollback with 1 month granularity over the last 3 years or to rollback with a 6 hours
17# granularity over the last 3 months (because in our example backups run every 6 hours).
18
19# OpenStack
20# OpenStack parameters are read by Duplicity directly from the environment, no need to add them in "DUP_OPTS"
21SWIFT_AUTHURL="https://auth.cloud.ovh.net/v3"
22SWIFT_AUTHVERSION="3"
23SWIFT_USERNAME="your username"
24SWIFT_PASSWORD="your token"
25SWIFT_TENANTNAME="your tenant name (= your project name, 16 figures)"
26SWIFT_REGIONNAME="your region, without the number (i.e.: GRA)"
27
28# GnuPG
29PASSPHRASE="the passphrase which will encrypt your data"

Make systemd aware of all this

1$ systemctl --user daemon-reload

Start the timer

1$ systemctl --user enable --now backups.timer

That’s it

You will have nice regular backups wich will be pushed to the Cloud for a ridiculous price (except if you have set up an insane retention or if you have TB of personal data).

The three last useful commands to know:

1# To check the state of the backups
2$ systemctl --user status backups
3# To know when the last backup has been run
4$ systemctl --user list-timers
5# To read your backup's logs
6$ journalctl --user -u backups