Backups and restores

This page tells how much data a restore can lose, how to copy the backups to a different site, how long the backups keep personal data, and how to restore the data. On this page, <domain> is the domain of your hosting provider deployment.

Know how much data a restore can lose

The recovery point objective (RPO) of a backup is the maximum period of changes that a restore of the newest backup can lose. Each backup has a daily timer with OnCalendar = "daily" (at 00:00) and Persistent = true. When the node is off at 00:00, the backup starts when the node starts again. Thus the RPO of each backup is 24 hours, plus the duration of the backup run. A run that fails does not start again before the next day, thus each failed run adds 24 hours.

mkOffsiteBackup in core/components/operator/app-resources.nix sets the daily timer of each restic backup with lib.mkDefault, except for the valkey cache. Thus a timerConfig in the configuration of the node replaces it. The valkey cache and the media backup have the daily timer as the default of their option. On the node, restic-<name> snapshots lists the snapshots of a restic backup, for example restic-mastodon_postgres snapshots. media-backup-<app> stamps lists the runs in the history of a media backup.

Each backup is a separate unit. Thus the backups of one application, for example of its database and of its media, are not of the same instant.

All these backups are in buckets of the garage of the group. The RPO applies only while this garage keeps its data, or while a copy outside the site keeps it.

garage has no offsite backup. On its node, garage makes snapshots of its metadata database in /var/lib/garage/snapshots (core/resources/garage/metadata-snapshots.nix), and keeps the two newest snapshots. The first snapshot occurs 3 hours after garage starts, then each 6 to 7.2 hours, from services.garage.settings.metadata_auto_snapshot_interval. A restart of garage starts this schedule again. A snapshot holds no data blocks, thus it is only for a corrupted metadata database, see Restore the metadata of garage. If the node of garage is lost, all the backups in these buckets are lost, unless the group has a copy outside the site. That copy holds the objects of the backup buckets, and not the metadata of garage.

These services have no backup, because a deploy makes their state again:

• The SPIRE server. A copy of its CA key can sign an identity for each host, and its CA expires after 24 hours. See the assumptions in core/specs/008-components/hosting-provider/spire.md.
• maddy. maddy only relays mail, its accounts come from the configuration, and its DKIM keys come from the vars. A lost queue loses the mail that it did not send yet.
• The audit shipper. A lost cursor makes it send the journal again from the start, thus the receiver gets the lines again.

Before an upgrade, the guard of a service makes a snapshot of the state into its restic repository, while the service is stopped. The guards of garage, OpenBao, the SPIRE server, maddy and the audit shipper, and the guards on a node with no backup provider, make a copy on the node instead. If the snapshot fails, for example because the repository is not reachable, the guard tries again every 30 seconds. After 10 minutes, the guard refuses the upgrade: <name>-migrate-pre.service fails, the service stays stopped, and its data stays at the old version. The UPGRADE_BACKUP_TIMEOUT variable in the environment of that unit sets a different time, in seconds. Make the repository reachable, then start <name>-migrate.target.

Two alerts of Grafana monitor the backups: backup-failed when a restic-backups-* or media-backup-* unit fails, and backup-stale when the timer of such a unit did not start for 36 hours. A timer with a schedule of more than 36 hours gives a backup-stale alert between its runs.

Each database backup of an application, and of lldap, Windmill, authelia and NetBox, has a daily restore drill. Between 04:00 and 05:00, the unit restore-drill-<app>_postgres restores the newest snapshot into a scratch postgres on the node, and does not change the database of the service. The drill uses the same checks of the version and the keys as a restore. It fails when the dump does not load, or when it holds no table. The scratch copy is in /var/tmp while the drill runs, thus the node must have free disk space for one more copy of the database. The scratch postgres comes from the same package as pg_dump, thus a database with an extension that this package does not have fails its drill.

The drill writes fediversity_restore_drill_duration_seconds, which is the measured time of the restore of the database (the RTO of the database), and fediversity_restore_drill_last_success_timestamp_seconds. The time to make the node again, and the restore of the files and the media, are not in this measure. Two alerts of Grafana monitor the drills: restore-drill-failed when a restore-drill-* unit fails, and restore-drill-stale when the timer of such a unit did not start for 36 hours. Run a drill with systemctl start restore-drill-<app>_postgres. Set fediversity.restoreDrill.enable = false on a node to remove its drills.

Know the limits of a single node

Each stateful service of a group runs on one node, with no replica and no standby. Thus a lost node stops the service until you make the node again and restore its data. There is no automatic failover. Do not give a customer an availability commitment that needs more than this.

You cannot change the replication_factor of garage with a setting. More garage nodes need a public RPC address and a new layout, and the code does not give them. To keep a copy of the backups if the garage is lost, see Know how much data a restore can lose.

Copy the backups to a target outside the site

The buckets of the garage are on the site of the hosting provider. To keep the backups when the site is lost, set a target at the S3 service of a different provider. Each group sets its own target, and no group sets one by default.

1. At the other provider, make a bucket and a key that can read, write and delete in that bucket. The provider is a sub-processor, see contract-templates.md. If the provider has object versioning or object lock, turn it on with a period that is longer than the time that you need to see a fault: see the threat model.

2. Keep the key at a location outside the site, for example in the password store of the administrators. A restore after the loss of the site has no OpenBao.

3. Set backups.externalCopy in the data of the group in core/setups/groups/, and deploy:

   backups.externalCopy = {
     endpoint = "https://s3.example.net";
     region = "eu-central-1";
     bucket = "fediversity-backups";
     credentialsSeed = "example-net";
   };
   

4. Put a copy of the key in OpenBao, as in notes/bootstrap-seeds.md. Until you do this, openbao-envfile-restic-backups-external-copy on the garage node waits for the entry:

   bao kv put -mount=secret vars/secret/backup-external-copy/example-net accessKeyId=<key id> secretAccessKey=<secret>
   

A daily timer on the garage node starts restic-backups-external-copy between 03:00 and 04:00. The unit copies each *-backups bucket that has objects, thus the backups of the applications and of the services, the media backups and the valkey backups. The objects stay encrypted as they are: the unit does not read or decrypt a backup, and it needs no restic password. The other provider sees the names of the buckets, the size and the time of each object, and the names of the objects of a restic repository, which are hashes.

The target has this layout, below the apex of the group:

An object that changes or goes away in the source stays in history/ for historyDays (default 7). A run removes old stamps only if the run has no fault.

The name of the unit starts with restic-backups-, thus the alerts backup-failed and backup-stale monitor it, as they do each restic backup. The unit fails when it cannot get to the target. A snapshot in the target can be up to one day older than the same snapshot in the garage, and the RPO of the target thus adds up to one day to the RPO of the table above.

The unit also fails, and changes nothing, when more than half of the copied buckets have no objects in the source. A destroy removes the buckets of one deployment, thus this occurs only if the garage lost its data or is new. This is a seatbelt, not a control: a garage that lost less than half of the buckets goes on, and these copies go into history/. If a person removes this many buckets on purpose, move current/<bucket> to history/<stamp>/<bucket> by hand for each of them, and the next run continues.

Know how long the backups keep personal data

All the backups of an application (Mastodon, Pixelfed, PeerTube) keep its data for 30 days. The backups of the databases of lldap, Authelia, the panel and api/ keep their data for the same period, because these databases hold the accounts of the users and of the operators, the configuration of the deployments, and the record of who changed each deployment. Thus data that one of these services deletes, for example a deleted account, leaves all these backups after the same period. The value is backupRetention.application in core/components/operator/app-resources.nix. A component gives the retention of its database backup in fediversity.postgresBackup.<name>.retention.

A group with a copy outside the site adds to each period of this table: the copy follows the source at the next daily run, and then history/ keeps what the run removed for historyDays. Thus with the default of 7 days, 30 days become at most 38 days at the target. This also applies to media-backup erase, and to a destroy.

A daily timer makes each backup. After each backup, restic removes the snapshots that are more than 30 days older than the newest snapshot (restic forget --keep-within 30d), and the media backup removes the history that is older than 30 days. Thus deleted data leaves the backups 30 days after the last backup that holds it, at the next backup. When the backups stop, the expiry stops too, and the backups keep their data until a backup runs again. When a local redis stops because its group gets a valkey node, the bucket stays and no backup runs again. Thus the node deletes the repository of that redis as a whole once its newest object is older than 30 days, and a return to a local redis inside this period continues the repository.

Before an upgrade of the major of PostgreSQL or of the release series of MariaDB, the guard of the server makes a snapshot of its data directory into the <server>-upgrade repository of the <app>-backups bucket of fediversity.databaseUpgradeBackup.<server>. This applies to the local PostgreSQL of Mastodon, Pixelfed and PeerTube, to the MariaDB of Forgejo when Forgejo has its forgejo-backups bucket, and to the shared PostgreSQL node, in its postgresql-backups bucket. The snapshot holds all that the databases hold. The snapshot of the shared PostgreSQL node holds the databases of all the services that use that node, and it is the only backup of that node: the scheduled backups of the databases are on the nodes of the services. This repository has no expiry by time: it keeps the snapshots of the 2 newest upgrades. Thus deleted data can stay in it for more than 30 days, until 2 newer upgrades. To remove the data before then, remove the snapshots with restic-<server>_upgrade forget <snapshot> and restic-<server>_upgrade prune on the node.

Erase personal data

An erasure removes the data from the live service. The backups that were made before the erasure keep the data until they expire. Article 17 of the GDPR does not require a separate erasure in each backup when the data in the backups is out of use until the normal expiry removes it. The data protection authorities (for example the ICO and the CNIL) and the EDPB apply this reading on these conditions:

• Use a backup only to restore a service, and for no other purpose. Do not read personal data from a backup, and do not copy it to another system.
• Tell the person that the backups keep the erased data, and the longest time, see Tell the person how long the backups keep the data.
• After a restore from a backup that is older than an erasure, the erasure must be done again, see Do the erasures again after a restore.

If one of these conditions does not apply, or the person or an authority asks for it, remove the data from the backups before the expiry, see Remove erased data from the backups before the expiry.

These erasures remove data from the live services:

The restic backups (the databases, the local storage of PeerTube and Pixelfed, the valkey caches, the local redis caches and the ACME state) have no erase command for one record. Thus an erased record stays in them until they expire. The snapshots before an upgrade of a database server expire only after 2 newer upgrades, see Know how long the backups keep personal data.

Tell the person how long the backups keep the data

The longest time follows from the retention that the code sets, see Know how long the backups keep personal data:

These times apply while the daily backups run. When the backups of a part stop, its expiry stops too.

Remove erased data from the backups before the expiry

restic cannot remove one record from a snapshot. Thus remove each snapshot that holds the data. This also removes the other data of these snapshots, thus a restore to a time before the erasure is not possible after it. On the node of the service:

1. Do the erasure in the live service.
2. Make a new backup that does not hold the data: run the operation "Back up now" of the application, or systemctl start restic-backups-<name>.service for a service with no operation. Wait until it is complete.
3. Find the snapshots that are older than the erasure: restic-<name> snapshots. For an application, "List backups" shows them.
4. Remove them: restic-<name> forget <snapshot>..., then restic-<name> prune. Do this for each repository of the service that holds the data, also restic-<server>_upgrade for the snapshots before an upgrade.
5. For an object of a media bucket, use media-backup-<app> erase <bucket> <key>.
6. With a copy outside the site, the next daily run copies the change, and history/ keeps the old objects for historyDays. Remove history/<stamp>/<bucket> of the runs after the erasure at the target to remove the data at once.

A retained *-backups bucket of a destroyed deployment has no node that runs these commands. To remove it before the end of the grace period, add the key expires_at with the current time (RFC 3339) to the custom_metadata of its retention record at secret/metadata/retained-backups/<bucket id>, and run systemctl start garage-sweep-retained-buckets.service on the garage node. The sweep deletes the bucket, its KV entries and the record.

Do the erasures again after a restore

Who does an erasure again depends on who controls the data:

• The operator controls the data of its deployment, for example the accounts and the media of its applications and the users of its lldap. The operator does the erasures of this data. After the operator erases data of a person, the operator runs the operation "Record an erasure" of the application, with the reference of the request in its own records. The record on the node holds only the time and this reference, no data of the person. A restore from a snapshot that is older than a recorded erasure shows the reference, and "List backups" shows it until the operator runs "Confirm the erasures". The restore does not do the erasure again itself: the erasure needs the restored application. After a restore of a service of a deployment, also tell the operator at once which service you restored and the date of the backup. The operator then does again each erasure that it did after this date. The record is on the node of the application: a new node starts with no record.
• The hosting provider controls the accounts of the panel and the users of its own lldap. After a restore of one of these services, and before the service is open to its users again, do again each erasure that is newer than the backup. The record of the erasure requests that the hosting provider answered tells which erasures to do.

For a deleted deployment, the audit_event table of api/ is the record. The table records each deployment_delete with the deployment, the actor and the time. api/ also writes each event as a JSON line to the journal, and Loki keeps a copy for services.fediversity-telemetry.retentionDays, 90 days by default. A restore of the database of api/ loses the events after the backup, thus read them from Loki.

Know what a destroy keeps

A destroy of a deployment, or an apply that removes an application, keeps each *-backups bucket of it for a grace period. The bucket gets the name retained-<YYYYMMDD>-<id prefix>, where the date is the day of the destroy. A copy outside the site does not read a retained bucket, and moves its copy of the bucket into history/ at the next run. The destroy keeps the KV entries that the bucket cannot be read without, <app>-backups-restic and <app>-media-backup, and writes a retention record at secret/metadata/retained-backups/<bucket id>. The record holds the old name of the bucket, the namespace of the deployment and the KV paths.

A daily timer on the garage node, garage-sweep-retained-buckets, deletes the bucket, its KV entries and its record when the grace period ends. The period is the option fediversity.garage.server.retainedBackups.gracePeriodDays of the garage node (default 7). The node reads it at each run, thus a change applies to the buckets that are retained already.

To restore from a retained bucket, give it a new global alias with garage bucket alias, and allow a key on it. Then the sweep deletes only the record, and the bucket stays until you delete it. Read the restic password at the KV path in the record before the grace period ends. The destroy does not keep the keys that a backup depends on (see Restore a backup that depends on keys), such as the secret of PeerTube or the app key of Pixelfed. Thus a database backup of these applications in a retained bucket cannot be restored into a new deployment.

The deploy token of the operator cannot read or delete the KV entries or the record. It can delete a retained bucket, because its garage admin token has all the power of the admin API. No command erases a retained bucket before the end of its grace period.

Restore OpenBao and the terraform backend

The offsite backups of OpenBao and of the terraform backend are the bootstrap backups. Their restore comes before OpenBao, thus they use a fixed restic password, and they encrypt their keys to the age recipients of fediversity.bootstrapBackupRecipients: the keys of the contributors and the cd key in core/keys.

Each file is a tar. Each restore command reads the tar on its stdin.

Restore in this order:

1. garage, which holds the backups. If the site is lost, start at Restore from the external copy.
2. OpenBao. The other services read their secrets from OpenBao.
3. The terraform backend. Do this when no apply runs, because the restore stops the backend and clears its locks.
4. The applications and the other services, from their own backups. Some of these backups depend on keys in OpenBao, see Restore a backup that depends on keys. Then do the erasures again.

Restore on the same node

The node has the key, thus the restore needs no age identity. As root on the node:

restic-openbao_raft dump latest openbao-raft | openbao-snapshot-restore
restic-terraform-backend_states dump latest terraform-backend-states | terraform-backend-restore

Restore on a new node

A new node has a different TPM and a different kmsKey. Thus the restore needs the keys from the backup. Without them, the restore commands fail and change nothing.

1. Deploy the new node. OpenBao starts with a new and empty store.

2. Get the S3 key of the bucket from the garage node of the group. garage bucket info <bucket> shows the keys of the bucket, and garage key info --show-secret <key> shows the secret.

3. As root on the new node, get the file from the backup:

   export AWS_ACCESS_KEY_ID=<key id> AWS_SECRET_ACCESS_KEY=<secret> AWS_DEFAULT_REGION=garage
   export RESTIC_REPOSITORY=s3:<S3 endpoint of the garage>/openbao-backups
   export RESTIC_PASSWORD=ReallySafePasswordTrustMe39
   restic dump latest openbao-raft > /root/openbao.tar
   

4. On your workstation, decrypt the key with your age identity, add it to the tar in plaintext, and send the tar to the restore command. The plaintext key thus does not go to the disk of the node:

   umask 077
   ssh admin@<node> sudo cat /root/openbao.tar > openbao.tar
   tar -xf openbao.tar unseal-key.age
   age -d -i ~/.ssh/id_ed25519 -o unseal-key unseal-key.age
   tar -rf openbao.tar unseal-key
   ssh admin@<node> sudo openbao-snapshot-restore < openbao.tar
   shred -u unseal-key unseal-key.age openbao.tar
   ssh admin@<node> sudo rm /root/openbao.tar
   

   The restore seals the key to the TPM of the new node, restarts the reconcilers and rotates the static roles. When a step fails, the command tells which step, and exits with a failure.

5. Do steps 2 to 4 again for the terraform backend, with the bucket terraform-backend-backups, the file terraform-backend-states, the encrypted key kmsKey.age, the plaintext key kmsKey and the command terraform-backend-restore.

Restore from the external copy

Do this when the site is lost. If the garage is good, restore from the garage: its restore is faster, and it has the newest backups.

1. First remove backups.externalCopy from the data of the group, and keep it out until the restore is complete. A copy from a new garage to the target can change or remove what you restore from.

2. Get the S3 key from the location outside the site. You do not need OpenBao for this.

3. Deploy the new nodes. This makes the buckets of the new garage again, with no data. Do the next step before the first daily backup of the new nodes. If a new node already made a backup, empty its bucket first: a new repository must not mix with the copy.

4. Copy each bucket that you need into the bucket of the same name in the new garage. Get the S3 key of a bucket as in step 2 of Restore on a new node:

   export RCLONE_CONFIG_OFFSITE_TYPE=s3 RCLONE_CONFIG_OFFSITE_PROVIDER=Other
   export RCLONE_CONFIG_OFFSITE_ENDPOINT=<endpoint> RCLONE_CONFIG_OFFSITE_REGION=<region>
   export RCLONE_CONFIG_OFFSITE_ACCESS_KEY_ID=<key id> RCLONE_CONFIG_OFFSITE_SECRET_ACCESS_KEY=<secret>
   export RCLONE_CONFIG_GARAGE_TYPE=s3 RCLONE_CONFIG_GARAGE_PROVIDER=Other RCLONE_CONFIG_GARAGE_NO_CHECK_BUCKET=true
   export RCLONE_CONFIG_GARAGE_ENDPOINT=<S3 endpoint of the garage> RCLONE_CONFIG_GARAGE_REGION=garage
   export RCLONE_CONFIG_GARAGE_ACCESS_KEY_ID=<key id> RCLONE_CONFIG_GARAGE_SECRET_ACCESS_KEY=<secret>
   rclone copy --fast-list --checksum --metadata offsite:<bucket>/<apex>/current/<name> garage:<name>
   

   Restic can also read a repository from the target with no copy. Use this to get one file, or to check the copy:

   export AWS_ACCESS_KEY_ID=<key id> AWS_SECRET_ACCESS_KEY=<secret> AWS_DEFAULT_REGION=<region>
   export RESTIC_REPOSITORY=s3:<endpoint>/<bucket>/<apex>/current/openbao-backups
   export RESTIC_PASSWORD=ReallySafePasswordTrustMe39
   restic dump latest openbao-raft > /root/openbao.tar
   

5. Restore in the order of Restore OpenBao and the terraform backend. The fixed restic password opens the bootstrap backups, and your age identity decrypts their keys. OpenBao then gives the restic and crypt passwords of the other backups: see Restore a backup that depends on keys.

6. To restore an object that a run replaced or removed, read history/<stamp>/<name>/. A restic repository is complete only in current/.

7. Do the erasures again, and set backups.externalCopy again.

The repository of an application can be at a subpath of its bucket, thus the restic path above can need that subpath. Test this restore before you depend on it: restore a repository from the target on a test node.

When the site stays lost, nothing prunes the copy. Delete it at the other provider when you do not need it, or at the end of the retention of Know how long the backups keep personal data.

Restore the metadata of garage

When the metadata database of garage is corrupted, replace it with a snapshot. The restore loses the changes after the snapshot: the objects that clients wrote after the snapshot are not in the buckets. As root on the node of garage:

ls /var/lib/garage/snapshots
systemctl stop garage.service
mv /var/lib/garage/meta/db.lmdb /var/lib/garage/meta/db.lmdb.bad
mkdir /var/lib/garage/meta/db.lmdb
cp /var/lib/garage/snapshots/<time>/db.lmdb /var/lib/garage/meta/db.lmdb/data.mdb
chown -R --reference=/var/lib/garage/meta/db.lmdb.bad /var/lib/garage/meta/db.lmdb
systemctl start garage.service
garage repair -a --yes tables

The name of each snapshot is the UTC time of the snapshot. A snapshot is one file, db.lmdb, but the database is a directory that holds the file data.mdb. garage has a DynamicUser, thus chown gives the new directory the owner of the old directory. When garage works again, remove /var/lib/garage/meta/db.lmdb.bad.

Restore a backup that depends on keys

Some data in a backup is encrypted with a key from OpenBao. For example, Mastodon and Authelia encrypt columns of their databases, lldap derives the passwords of its users from a seed, and Grafana encrypts its data sources. A backup does not hold these keys. Thus do not rotate these keys: with a new key, the restored data is not usable. If a key leaked, see Change a key after a leak.

Each snapshot of such a backup has a tag key:<path>:<digest>:v<version> for each key, with the KV path of the key, the SHA-256 digest of its value and its KV version in the secret mount. The OpenBao agent of the node writes the same data for the current keys to /run/backup-keys/<app>_<backup>. The restore script compares the two and stops when a key is different:

Snapshot <id> depends on keys that are not the keys of this node:
  <path>: the snapshot needs v<n>, the node has v<m>

To restore such a snapshot:

1. Get the old value of each key back with bao kv rollback -mount=secret -version=<n> <path>. This writes the old value as a new version, thus the digest is the same as the digest of the snapshot.
2. Wait until the OpenBao agent writes /run/backup-keys/<app>_<backup> again (at most 5 minutes), or restart the agent.
3. Restart the application, thus it uses the old key.
4. Restore again.

To restore without the check, set RESTORE_IGNORE_KEYS=1. A snapshot from before the tags has no tags and restores without a check.

A migration carries the keys of the source to the target, see Restore the apps of a migration.