Fediversity

Migrations

This page tells how to export a deployment for a migration, and how to restore the apps of the migration on the target.

Export a deployment for a migration

The export of a deployment changes the source deployment:

  1. The export stops the app services on each node of the deployment. Then it makes the snapshot of each app. Thus no write after the snapshot is lost.
  2. After the snapshots, the api suspends the deployment.
  3. If a snapshot or the suspend fails, the export starts the app services again.

The apps are down from the export until the target restores them. The source stays suspended, and its data stays. To roll back the migration, resume the source deployment. To export it again, resume it first. A configuration with no enabled app has no snapshot, thus its export does not suspend it.

Each node with an app that has state must have migration-quiesce. A node that was deployed before this command existed stops the export before a snapshot. Deploy the deployment again, then export.

When the deployment has its domain in the registrar account, the migration string gives the auth code of the domain. The api reads the code at the registrar, unlocks the domain, and removes its DNSSEC keys. The target then transfers the domain to its own registrar account. Without the keys, the DS records of the source do not stay after the transfer, thus the zone is not bogus when the target signs it with different keys or does not sign it. If the api cannot read the code at the registrar, the export fails, and does not change the source.

Until the domain is locked again and its auth code is reset, the code can transfer the domain. The api does this:

  • when the source is resumed. After a transfer, the resume fails;
  • when the domain is given to a different deployment of this hosting provider;
  • when this hosting provider imports the string. The target then uses the domain without a transfer.

If the api has no registrar credentials, or the registrar fails, lock the domain and reset its auth code in the registrar account.

The api does not set the DNSSEC keys again. If you set the keys of the domain in the registrar account, set them again after the domain is locked again.

The migration string gives the S3 credentials and the password of each snapshot repository, and the keys that decrypt the data of the apps: mastodon-keys, the app key and the Passport key of Pixelfed, and the secret of PeerTube. Thus the string gives all data of the deployment. Treat it as a secret: give it only to the target, send it on an authenticated connection, and do not keep it. The api does not keep the string, and does not write it to a log or an audit event.

A node that runs an older revision does not give its keys. The export continues, and the string has no keys for that node. To carry the keys, deploy the source with a newer revision, resume it if it is suspended, and export again.

Restore the apps of a migration

When a deployment is the target of a migration import, the deploy restores each app from the snapshot of the source:

  • The api keeps the credentials of the snapshot repository and the keys of the apps in the OpenBao KV of the deployment, at vars/operator-deployments/<deployment>/secret/migration/pointers. They are not in the state of the deploy or in the windmill job.

  • After the deploy, the deploy restores each app. Before the restore of a unit, the keys of the source replace the values that the target made, on the node and in the KV of the node. Then the deploy replaces the KV entry with a mark of the restore, and no version of the entry keeps the credentials. If the deploy stops before it records the restore, deploy again: the deploy finds the mark and does not restore again.

  • If the restore fails, the deploy fails. Deploy again to restore again.

  • The restore of Mastodon, Pixelfed and PeerTube includes the ACME state of the node (/var/lib/acme). Thus the target uses the certificates of the source and does not order new ones. This keeps the names below the rate limits of the certificate authority.

  • A node that the deploy creates as the target of a restore starts with the Incus instance key user.fediversity.restore-pending. While the key is set, the node does not contact the ACME server, and it serves a self-signed certificate. The deploy removes the key after the restore of the node, and then restarts the node.

  • If the restore of a node does not succeed, the key stays, and the node keeps the self-signed certificate. Deploy again to restore again. To let the node order new certificates without a restore, remove the key on the hypervisor and start the order units of the node again:

    incus --project fediversity config unset <instance> user.fediversity.restore-pending
    incus --project fediversity exec <instance> -- systemctl start --all 'acme-order-renew-*.service'

  • While the deployment is suspended, the deploy does not restore. Resume the deployment to restore it.

A pointer from a source with an older revision has no keys. The restore then depends on the snapshot:

  • A snapshot with no key tags restores, and the target keeps its own keys. Thus the target cannot read the data that the keys of the source encrypt. The deploy log shows restore-foreign: warning: the pointer for <unit> carries no keys. To correct this, deploy the source with a newer revision, resume it, export again, and import the new string.
  • A snapshot with key tags does not restore, and the deploy fails. Do the same steps. RESTORE_IGNORE_KEYS=1 applies only to a restore that you run by hand on the node: the deploy does not give it to the node.

After a migration, the source and the target share each key that the string carried. If the source stays in use, or if the string leaked, change the shared keys, see Change a key after a leak. Some keys cannot change without data loss, see Keys that cannot change without data loss.