Yavook.de/kiwi-backup
1
0
Fork 0
mirror of https://github.com/yavook/kiwi-backup.git synced 2024-11-24 07:53:01 +00:00
Code Packages Activity

Just massive improvements

Browse source
This commit is contained in:
Jörn-Michael Miehe 2020-08-28 03:22:05 +02:00
parent ee36fe178e
commit 8ee3939990
2 changed files with 63 additions and 101 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

14
Dockerfile
View file

@ -77,18 +77,17 @@ RUN set -ex; \
pip3 uninstall -y wheel; \ pip3 uninstall -y wheel; \
apk del --purge .build-deps apk del --purge .build-deps
VOLUME ["/root/.cache/duplicity"] VOLUME ["/root/.cache/duplicity", "/backup/target"]
ENV \ ENV \
################# #################
# BACKUP POLICY # # BACKUP POLICY #
################# #################
SCHEDULE_BACKUP="36 03 * * *" \ SCHEDULE_BACKUP="36 02 * * *" \
SCHEDULE_CLEANUP="36 04 * * *" \ SCHEDULE_CLEANUP="36 04 * * *" \
FULL_BACKUP_FREQUENCY=4M \ FULL_BACKUP_FREQUENCY=3M \
BACKUP_RETENTION_TIME=9M \ BACKUP_RETENTION_TIME=6M \
KEEP_NUM_FULL_CHAINS=1 \ KEEP_NUM_FULL_CHAINS=2 \
BACKUP_TARGET="file:///backup/target" \
\ \
###################### ######################
# ADDITIONAL OPTIONS # # ADDITIONAL OPTIONS #
@ -96,6 +95,7 @@ ENV \
SCHEDULE_RMFULL="36 05 * * SAT" \ SCHEDULE_RMFULL="36 05 * * SAT" \
SCHEDULE_RMINCR="36 05 * * SUN" \ SCHEDULE_RMINCR="36 05 * * SUN" \
BACKUP_VOLSIZE=1024 \ BACKUP_VOLSIZE=1024 \
BACKUP_TARGET="file:///backup/target" \
OPTIONS_BACKUP="" \ OPTIONS_BACKUP="" \
OPTIONS_CLEANUP="" \ OPTIONS_CLEANUP="" \
OPTIONS_RMFULL="" \ OPTIONS_RMFULL="" \
@ -104,8 +104,6 @@ ENV \
############## ##############
# ENCRYPTION # # ENCRYPTION #
############## ##############
#
# GnuPG key-id as specified by https://www.gnupg.org/documentation/manpage.html#sec-2-6
GPG_KEY_ID="" \ GPG_KEY_ID="" \
GPG_PASSPHRASE="" GPG_PASSPHRASE=""

150
README.md
View file

@ -6,7 +6,9 @@ The backup solution for [`kiwi-scp`](https://github.com/ldericher/kiwi-scp)
## Quick start ## Quick start
Assuming the backups should be kept locally in `/var/kiwi.backup`, just add this to one of your projects' `docker-compose.yml`. kiwi-backup is an image with [duplicity](http://duplicity.nongnu.org/), tailored to backup service data of `kiwi-scp` instances.
If you want backups in the host directory `/var/kiwi.backup`, just add this to one of your projects' `docker-compose.yml` to use the default configuration.
```yaml ```yaml
backup: backup:
@ -16,14 +18,15 @@ backup:
- "/var/kiwi.backup:/backup/target" - "/var/kiwi.backup:/backup/target"
``` ```
This will use the default configuration.
- backups the entire service data directory - backups the entire service data directory
- stores all backup data on the host file system - stores all backup data on the host file system
- daily incremental backups at night (03:36 am UTC; time chosen by fair dice roll) - daily incremental backups at night
- a new full backup once every 4 months - a new full backup once every 3 months
- keeps backups for up to 9 months - keeps backups up to 6 months old
- keeps incremental backups for the most recent chain (4 months) - keeps daily backups for two recent sets (3-6 months)
- backup jobs run at 02:36 am UTC (time chosen by fair dice roll)
Be aware though -- backups will use a fair bit of storage space!
## Customization ## Customization
@ -33,18 +36,17 @@ Schedules in environment variables are to be provided [in cron notation](https:/
### Backup Scope ### Backup Scope
kiwi-backup will backup everything in its `/backup/source` directory, and you should have no incentive to change that. kiwi-backup will backup everything in its `/backup/source` directory -- change the backup scope by adjusting what's mounted into that container directory.
To change the backup scope, just change what's mounted into that container directory:
```yaml ```yaml
backup: backup:
# ... # ...
volumes: volumes:
- "$TARGETROOT:/backup/source:ro" # change me! # change scope here!
- "$TARGETROOT:/backup/source:ro"
``` ```
You may of course create additional sources below in the `/backup/source` directory to limit the backup to specific projects or services. For added safety, mount your backup sources read-only by appending `:ro`. You may of course create additional sources below the `/backup/source` directory to limit the backup to specific projects or services. For added safety, mount your backup sources read-only by appending `:ro`.
### Backup policy ### Backup policy
@ -57,28 +59,24 @@ backup:
# ... # ...
# when to run backups # when to run backups
# default: "36 03 * * *" <=> daily at 03:36 am # default: daily at 02:36 am UTC
SCHEDULE_BACKUP: "36 03 * * *" SCHEDULE_BACKUP: "36 02 * * *"
# when to remove failed transactions # when to remove leftovers from failed transactions
# default: "36 04 * * *" <=> daily at 04:36 am # default: daily at 04:36 am UTC
SCHEDULE_CLEANUP: "36 04 * * *" SCHEDULE_CLEANUP: "36 04 * * *"
# how often to opt for a full backup # how often to opt for a full backup
# default: "4M" <=> every 4 months # default: every 3 months
FULL_BACKUP_FREQUENCY: "4M" FULL_BACKUP_FREQUENCY: "3M"
# how long to keep backups at all # how long to keep backups at all
# default: "9M" <=> 9 months # default: 6 months
BACKUP_RETENTION_TIME: "9M" BACKUP_RETENTION_TIME: "6M"
# how many full backup chains with incrementals to keep # how many full backup chains with incrementals to keep
# default: "1" # default: 2
KEEP_NUM_FULL_CHAINS: "1" KEEP_NUM_FULL_CHAINS: "2"
# where to put backups
# default: "file:///backup/target" <=> likely in a host-mounted volume
BACKUP_TARGET: "file:///backup/target"
``` ```
### Additional options ### Additional options
@ -92,17 +90,21 @@ backup:
# ... # ...
# when to remove old full backup chains # when to remove old full backup chains
# default: "36 05 * * SAT" <=> every saturday at 05:36 am # default: every saturday at 05:36 am UTC
SCHEDULE_RMFULL: "36 05 * * SAT" SCHEDULE_RMFULL: "36 05 * * SAT"
# when to remove old incremental backups # when to remove old incremental backups
# default: "36 05 * * SUN" <=> every sunday at 05:36 am # default: every sunday at 05:36 am UTC
SCHEDULE_RMINCR: "36 05 * * SUN" SCHEDULE_RMINCR: "36 05 * * SUN"
# size of individual duplicity data volumes # size of individual duplicity data volumes
# default: "1024" <=> 1GiB # default: 1GiB
BACKUP_VOLSIZE: "1024" BACKUP_VOLSIZE: "1024"
# where to put backups
# default: some docker volume
BACKUP_TARGET: "file:///backup/target"
# Additional options for "duplicity --full-if-older-than" command # Additional options for "duplicity --full-if-older-than" command
OPTIONS_BACKUP: "" OPTIONS_BACKUP: ""
@ -124,7 +126,8 @@ For simplicity, this guide assumes you have a `kiwi-scp` instance with some proj
### GnuPG Key Generation ### GnuPG Key Generation
> If you already have a key you want to use for this instance, skip this section. > You'll usually want to generate a new key for each `kiwi-scp` instance.
> If you have reasons not to, skip this section.
Reasonable defaults for a backup encryption key are: Reasonable defaults for a backup encryption key are:
@ -133,84 +136,47 @@ Reasonable defaults for a backup encryption key are:
* Doesn't expire * Doesn't expire
* Secure passphrase (Don't bother memorizing it, you will save it in your `kiwi-scp` instance!) * Secure passphrase (Don't bother memorizing it, you will save it in your `kiwi-scp` instance!)
To quickly generate a key, use the following command, then enter your passphrase: To quickly generate a key, use the following command, then enter a passphrase:
```sh ```sh
docker run --rm -it -v "$(pwd)/gnupg.tmp:/root/.gnupg" ldericher/kiwi-backup gpg --quick-gen-key --yes "Administrator <root@my-hostname.com>" rsa4096 encr never docker run --rm -it -v "gnupg.tmp:/root/.gnupg" ldericher/kiwi-backup gpg --quick-gen-key --yes "Administrator <root@my-hostname.com>" rsa4096 encr never
``` ```
This creates a subdirectory "gnupg.tmp" in the current working directory, which will be discarded later. To get a more in-depth generation wizard instead, use `gpg --full-gen-key` command without any more args and follow through.
To get a more in-depth generation wizard instead, use the following command and follow its directions: ### Export the generated key
This one-liner exports your generated key into a new subdirectory "backup":
```sh ```sh
docker run --rm -it -v "$(pwd)/gnupg.tmp:/root/.gnupg" ldericher/kiwi-backup gpg --full-gen-key docker run --rm -it -v "gnupg.tmp:/root/.gnupg" -v "$(pwd)/backup:/root/backup" -e "CURRENT_USER=$(id -u):$(id -g)" ldericher/kiwi-backup sh -c 'cd /root/backup && gpg --export-secret-keys --armor > secret.asc && gpg --export-ownertrust > ownertrust.txt && chown -R "${CURRENT_USER}" .'
``` ```
### Key-ID You'll now find the "backup" subdirectory with files "secret.asc" and "ownertrust.txt" in it. Check your exported files:
> If you already have a key you want to use for this instance, skip this section.
During key generation, there's an output line `gpg: key 38CD19177F84710B marked as ultimately trusted` where `38CD19177F84710B` will be your Key-ID. If you lost it, you can list the keys using `gpg -k`:
```sh ```sh
docker run --rm -v "$(pwd)/gnupg.tmp:/root/.gnupg" ldericher/kiwi-backup gpg -k | grep -A1 '^pub' docker run --rm -v "$(pwd)/backup:/root/backup:ro" ldericher/kiwi-backup sh -c 'cd /root/backup && gpg --import --batch secret.asc 2>/dev/null && gpg --import-ownertrust ownertrust.txt 2>/dev/null && gpg -k 2>/dev/null | grep -A1 "^pub" | xargs | tail -c17'
``` ```
Output (shortened): This should output your 16-digit Key-ID, so take note of it if you haven't already! Afterwards, run `docker volume rm gnupg.tmp` to get rid of the key generation volume.
``` ### Using a pre-generated key
[...]
pub rsa4096 2020-08-27 [SC]
82BA35B0871675F78165618238CD19177F84710B
```
You can use the full fingerprint `82BA35B0871675F78165618238CD19177F84710B` or abbreviate to the last 16 digits `38CD19177F84710B`. Checking your Key-ID should succeed: To use a pre-generated key, you'll need to export it manually instead. These are the commands:
```sh ```sh
docker run --rm -v "$(pwd)/gnupg.tmp:/root/.gnupg" ldericher/kiwi-backup gpg --fingerprint 38CD19177F84710B gpg --export-secret-keys --armor [Key-ID] > backup/secret.asc
gpg --export-ownertrust > backup/ownertrust.txt
``` ```
For more possibilities of what counts as a Key-ID, refer to [the relevant GnuPG manual section](https://www.gnupg.org/documentation/manuals/gnupg/Specify-a-User-ID.html) You can still check your exported files :)
### Export the key
You now have a key to use for this instance. Export it into a new subdirectory "backup" in your project.
The following one-liner extracts the data from the previously generated "gnupg.tmp" directory:
```sh
docker run --rm -it -v "$(pwd)/gnupg.tmp:/root/.gnupg" -v "$(pwd)/backup:/root/backup" -e "CURRENT_USER=$(id -u):$(id -g)" ldericher/kiwi-backup sh -c 'cd /root/backup && gpg --export-secret-keys --armor > secret.asc && gpg --export-ownertrust > ownertrust.txt && chown -R "${CURRENT_USER}" .'
```
You'll now find the "backup" subdirectory having files "secret.asc" and "ownertrust.txt" in it.
If you did not generate your keys using the container and want to export them manually, use these commands:
```sh
gpg --export-secret-keys --armor [Key-ID] > /path/to/backup/secret.asc
gpg --export-ownertrust > /path/to/backup/ownertrust.txt
```
Optionally, check your export:
```sh ```sh
docker run --rm -v "$(pwd)/backup:/root/backup:ro" ldericher/kiwi-backup sh -c 'cd /root/backup && gpg --import --batch secret.asc && gpg --import-ownertrust ownertrust.txt && gpg -k' docker run --rm -v "$(pwd)/backup:/root/backup:ro" ldericher/kiwi-backup sh -c 'cd /root/backup && gpg --import --batch secret.asc && gpg --import-ownertrust ownertrust.txt && gpg -k'
``` ```
Output (shortened):
```
[...]
pub rsa4096 2020-08-27 [SC]
82BA35B0871675F78165618238CD19177F84710B
uid [ultimate] Administrator <root@my-hostname.com>
```
### Describe local kiwi-backup image ### Describe local kiwi-backup image
You now have a "backup" subdirectory containing your key export file and can safely discard a leftover "gnupg.tmp" subdirectory if applicable.
Now create a simple `Dockerfile` inside the "backup" directory from following template. Now create a simple `Dockerfile` inside the "backup" directory from following template.
```Dockerfile ```Dockerfile
@ -218,38 +184,36 @@ FROM ldericher/kiwi-backup
COPY secret.asc ownertrust.txt /root/ COPY secret.asc ownertrust.txt /root/
RUN set -ex; \ RUN gpg --import --batch /root/secret.asc; \
\
gpg --import --batch /root/secret.asc; \
gpg --import-ownertrust /root/ownertrust.txt; \ gpg --import-ownertrust /root/ownertrust.txt; \
rm /root/secret.asc /root/ownertrust.txt rm /root/secret.asc /root/ownertrust.txt
# Obviously, change these values to match your data! # fill in these values to match your data
ENV GPG_KEY_ID="38CD19177F84710B" \ ENV GPG_KEY_ID="changeme" \
GPG_PASSPHRASE="changeme" GPG_PASSPHRASE="changeme"
``` ```
You should add the "backup" directory to the repository backing up your `kiwi-scp` instance. If applicable, commit the "backup" directory into the `kiwi-scp` instance repository.
### Use local image ### Use local image
All that's left to do is come back to your project's `docker-compose.yml`, where you shorten one line. Old: All that's left is to come back to your project's `docker-compose.yml`, where you shorten one line. Change:
```yaml ```yaml
backup: backup:
image: ldericher/kiwi-backup image: ldericher/kiwi-backup
# [...] # ...
``` ```
New: Into:
```yaml ```yaml
backup: backup:
build: ./backup build: ./backup
# [...] # ...
``` ```
That's it! `kiwi-backup` will automatically start encrypting your new backups. That's it -- from now on, all new backups will be encrypted!
## Offsite Backups ## Offsite Backups