State lifecycle

This page tells what happens to the state of a running system when the code of the project changes, and when a service gets a new version. State is the data that a deploy does not make again: databases, buckets, secrets, the OpenTofu state, and the records of api/ and panel/.

The rule for each type of change

The sections below give the details and the files.

Two types of OpenTofu state

The code uses OpenTofu in two layers:

• The deployment graphs are in core/effects/tf/. Their state is in the terraform-backend service of the hosting provider.
• The on-node configurations are generated from Nix and applied by a unit on the node, for example core/resources/postgresql/provision.nix, core/resources/ldap/provision.nix, core/resources/garage/provision.nix and core/components/hosting-provider/openbao/runtime/module.nix. Their state is a terraform.tfstate file on the node.

In the two layers, the state is what connects the code to the objects. Thus a change that must move or adopt an object is a block in the OpenTofu configuration, or code written for that change.

An option is renamed

The panel and api/ store the configuration of a deployment as JSON. When an option of the operator configuration gets a new path, a stored configuration still has the old path.

core/lib/renames-for-source.nix makes a table of { from; to; } entries from the option tree: each rename that is declared with lib.mkRenamedOptionModule or lib.mkAliasOptionModule, and each rename of a key in a pinned upstream JSON Schema, which a settings-renames.json file beside the component declares. core/renames-log.json keeps the entries of earlier revisions, each with the since version that added it. api/package.nix puts the table in the package as renames.json, and its build fails for a rename that the log does not have.

core/renames-log.json has 0 entries.

Before api/ validates a stored configuration, _migrate_for_validation in api/src/fediversity_api/procedures.py moves each value from its old path to its new path (api/src/fediversity_api/migrate.py). A deployment row records the version of the log that its configuration agrees with (renames_version in api/src/fediversity_api/state.py), thus only the newer entries apply. A rename that cannot apply is reported as unmigrated, and the value stays where it is.

This mechanism changes only JSON. If the renamed option is part of the name of a deployed object, the next apply sees a different object: see the next sections.

A stored deployment also records the source of its modules. api/ stores a pin of its own revision for a deployment without a source, thus a rebuild of api/ does not change the schema of a stored deployment until its next deploy. See "Default-source baseline freeze" in api/docs/design/schema-and-sources.md.

A value changes

For most values of the operator configuration, and for the environment variables of api/ and panel/, the deploy writes the value into a configuration file or into the environment of a unit. When the file or the environment changes, the unit starts again. Thus the next deploy applies the new value.

Some applications keep a value in their own state, and that value wins over the deployed value. The code prevents this for these values:

• Pixelfed writes a setting into its config_cache table at the first read, and from then on the row wins. core/components/operator/pixelfed/node.nix sets ENABLE_CONFIG_CACHE = false, thus the value of the .env file applies.
• PeerTube reads local-production.json in its configuration directory over the deployed configuration. Its web edit of the configuration writes that file. core/components/operator/peertube/node.nix turns off the web edit, and the preStart of peertube.service removes the file.
• The secrets of the operator are in OpenBao KV (api/src/fediversity_api/operator_secrets.py). When the rename table renames the field of a secret, a submit copies the value from the old name to the new name. When the operator clears a secret, the next deploy deletes its KV entry with all its versions (core/effects/tf/operator-provision/kv-clear.tf).

These values apply only to new objects, or stay with the application, by design:

The docs build generates this table and the list above from core/docs/md/technical-design/state-lifecycle.nix. The build fails when an option of a row is not in core/components/operator/operator-facing-options.csv, or when a file that a row names no longer has the code that the row describes.

A resource gets a new address

When the address of a resource changes and nothing else is done, OpenTofu destroys the object of the old address and creates an object at the new address. For an object that holds data, the new object is empty.

A moved block prevents this: it tells OpenTofu that the object of the old address is the object of the new address. The tree has these:

• core/effects/tf/operator-provision/garage.tf has the only moved block in a .tf file, for a terraform_data resource with a new name.
• core/lib/tf-labels.nix (movedBlocks) generates the moved blocks of the on-node configurations, from the labels <prefix>_<key> that the states of deployed nodes have, to the labels of the current code.
• core/resources/garage/provision.nix changes its state file with jq before the apply, because a moved block between two providers needs a provider that reads the old type.
• core/components/hosting-provider/incus.nix repairs its state file in the preStart of the unit, before the moved blocks apply.

The limits:

• A moved block and a removed block accept only static addresses. The comment before garage_bucket.bucket in core/effects/tf/operator-provision/garage.tf gives the result: a resource whose for_each keys change with each deployment cannot get such a block.
• No .tf file in the tree has a removed block.
• A moved block does not rename the object itself. A database, a bucket or a node with a new name is a new object.

An object exists, but the state does not have it

An import block adopts the object, so that the apply does not create it again:

• The on-node configurations of PostgreSQL, lldap and garage adopt an object that exists and is not in the state, for example a database that ensureDatabases made.
• core/effects/tf/incus/operator/registrar.tf adopts a domain claim in OpenBao, a domain that is in the registrar account already, and a name server group.

notes/forward-recovery.md gives the rules for an apply that failed: there is no rollback, and the same apply runs again.

A resource is dropped

When a resource leaves the configuration, the next apply destroys its object. This is the default of OpenTofu in the two layers:

• In an on-node configuration, an object that leaves the configuration is dropped. For PostgreSQL, a grant is revoked, and a database is dropped with its data (core/resources/postgresql/provision.nix).
• In a deployment graph, the destroy-time provisioners of a resource release the objects that are not in the state, for example DNS records.

Some resources have a different rule:

The docs build generates this table. It fails when a .nix or .tf file below core/ has prevent_destroy = true or destroy = false and no row names the file, and when a file of a row no longer has that code.

Gaps:

• An object that is in no state has no deletion path. The on-node configuration of PostgreSQL says so for its own objects: an object that is in neither the configuration nor the state is not touched, thus a name that left the configuration before the first apply stays.
• Only the node of the state backend has a guard against a replacement. For each other node, a change that replaces the node gives a new node, and the data on the old root disk is gone unless a volume or a backup holds it.

An object outside OpenTofu is dropped

A script or a unit makes some objects, not an OpenTofu resource. Such code follows a drop only when it lists the objects that exist, compares them with the objects that the code states, and removes the others. These do:

The docs build generates this table. It fails when a file of a row no longer has the code that removes the object. The list of backups with a sweep comes from the calls of mkRetiredRepository. The build does not find a new removal that has no row.

Gaps:

• When the rename table renames the field of a secret of the operator, the KV entry of the old name stays.
• When a role leaves core/components/operator/mastodon/role-groups.nix, the holders keep it on Mastodon (core/components/operator/mastodon/ldap-roles.rb). No path of the configuration gets to this.
• The sweep of retired restic repositories serves only the backups in the table above. Other restic repositories of a backup that stops stay, for example the repository of the dump before a PostgreSQL upgrade.

An application is disabled

The option deleteData of an application is in core/components/sharedOptions.nix. Its default is false.

• When enable becomes false and deleteData is false, the data stays: the database, the buckets, the generated secrets and the cache. A new enable uses the data again.
• When enable is false and deleteData is true, the next deployment deletes the data. purgedApplications in core/setups/wiring/node-selection.nix finds these applications. core/setups/wiring/supporting-requests.nix leaves their requests out, thus the operator graph deletes their buckets, databases and keys, and the valkey node deletes their cache. core/effects/tf/operator-provision/openbao.tf deletes their KV directories (purged_kv_dirs, from core/setups/wiring/purged-kv.nix).
• A *-backups bucket stays for the grace period, as at a destroy.

The operator documentation of the option is in the operator configuration, and its role for erasure is in Data protection.

A deployment is destroyed

A destroy of a deployment runs tofu destroy on the operator graph. It removes the nodes. core/effects/tf/operator-provision/postgresql.tf drops the databases and the login roles of the deployment on the shared PostgreSQL node: their names have the namespace of the deployment as a suffix, and this graph is the only actor that runs again when the deployment ends. terraform_data.after_instances in core/effects/tf/operator-provision/openbao.tf deletes what remains below the KV prefix of a namespaced deployment after the instances are gone. terraform_data.retire_bucket empties and deletes each bucket that is not a backup bucket.

The destroy keeps this state:

The check core/checks/integrations/destroy-reports-left-behind covers the last row. The operator steps for a retained bucket are in Know what a destroy keeps. An export for a move to a different hosting provider must be complete before the destroy: no key of the operator can read a retained bucket.

A destroy of all nodes of the hosting provider also destroys the node that keeps the OpenTofu state. Thus the run first moves the state to a local file (core/checks/integrations/destroy-detaches-state).

A service gets a new version

Each service that keeps state has a systemd.stateMigrations.<name> entry, in most cases in a file migrations.nix beside the service. The module is nix/contracts/nixos/modules/system/state-migrations.nix. An entry gives the version of the service in the current code. A unit <name>-migrate-pre runs before the service, compares that version with the version in a record file on the node, and decides:

flowchart TD
  start["The unit `<name>-migrate-pre` starts"] --> record{"Is there a record of the version?"}
  record -- "no" --> fresh["Record the version. Run `onFreshInstall` if the entry has one and the state is new."]
  record -- "yes, the same version" --> none["Do nothing"]
  record -- "yes, a later version than the code" --> down{"What is the value of `onDowngrade`?"}
  down -- "refuse" --> stop["Exit with status 78. The service stays stopped."]
  down -- "warn or ignore" --> older["Record the older version. Run nothing."]
  record -- "yes, an earlier version than the code" --> skip{"Is the step larger than `maxSkip` permits?"}
  skip -- "yes" --> stop
  skip -- "no" --> backup["Run `backup`, then `steps` and `onUpgrade`"]
  backup --> service["The service starts"]
  service --> post["The unit `<name>-migrate-post` runs `onUpgradePost` and records the version"]

The rules of the module:

• The backup. backup runs one time for each upgrade, before each other command of the upgrade.
• The downgrade. onDowngrade is refuse by default. Exit status 78 tells systemd not to start the unit again. With warn or ignore, the unit records the older version and runs nothing: these are services that also read the state of a later version.
• The largest step. maxSkip is not set by default, and then each step to a later version is permitted. With major, a step across more than one major release is refused. With minor, a step across more than one minor release is also refused.
• The restore. When the entry has a restore command, the unit records an upgrade that is not complete. A deploy of the version from before the upgrade then runs restore. A deploy of a third version is refused. Without restore, the way back is a manual restore of the backup.
• No record. When freshInstallTest is null and the node has no record, the unit records the version and does not migrate. Thus the first deploy of an entry onto state that exists does not make a backup.

What an entry does not do:

• It moves no object and changes no OpenTofu state.
• In most entries, it runs no schema command: the service migrates its own schema when it starts. The column "Upgrade command" of the table shows the entries that do run a command. For example, core/components/operator/mastodon/migrations.nix runs the database migrations of Mastodon, and the post-deployment migrations only after the services are up again.
• No entry has a command for one specific version: the list steps is empty in each entry.
• An entry whose version is a release series, for example 2.5, sees no change at a patch release. Thus a patch release makes no backup.

These files below core/components and core/resources define an entry. core/setups/nixos/postgresql.nix and core/setups/nixos/mysql.nix define the entries of PostgreSQL and MariaDB.

• core/components/dev/forgejo/migrations.nix
• core/components/hosting-provider/audit-shipper-migrations.nix
• core/components/hosting-provider/bind-migrations.nix
• core/components/hosting-provider/netbox/migrations.nix
• core/components/hosting-provider/openbao/migrations.nix
• core/components/hosting-provider/spire/migrations.nix
• core/components/hosting-provider/telemetry/migrations.nix
• core/components/hosting-provider/terraform-backend/migrations.nix
• core/components/hosting-provider/windmill/migrations.nix
• core/components/operator/mastodon/migrations.nix
• core/components/operator/peertube/migrations.nix
• core/components/operator/pixelfed/migrations.nix
• core/components/shared/authelia/migrations.nix
• core/components/shared/lldap/migrations.nix
• core/components/shared/valkey/migrations.nix
• core/resources/garage/migrations.nix
• core/resources/smtp/maddy-migrations.nix

The table shows the entries of the nodes of core with the default applications and the configuration of the tests. The docs build generates it from config.systemd.stateMigrations of each node of the groups hosting-provider and operator of core/testers/groups.nix. The column "Backup before the upgrade" is written by hand in core/docs/md/technical-design/state-lifecycle.nix, and the build fails when it is not in agreement with the backup script of the entry. A row is one entry, or the entries with the same name and the same values on more than one node. The version of each entry is the version of the package, or its release series (garage, for example), or its major version (PostgreSQL). An entry with a restic snapshot makes it on a node that has the offsite backup for that service. Without the offsite backup, the entry makes a local copy or a dump, or no backup.

These nodes of that configuration have no entry: hosting-provider/otel-collector, operator/grafana.

The table does not read the overview tag, the hypervisor and the development hosts. These entries are not on a node of that configuration, thus the table does not show them:

• fediversity-audit-shipper of core/components/hosting-provider/audit-shipper-migrations.nix. It refuses a downgrade and makes a local copy of its state directory.
• forgejo of core/components/dev/forgejo/migrations.nix.
• mysql of core/setups/nixos/mysql.nix, for a node that runs MariaDB.

Gaps:

• The postgresql entry itself does not limit the step. The guard of the PostgreSQL module, below, permits only a step of one major version.

PostgreSQL has one more rule. Each major version keeps its cluster in a directory of its own, thus a new major would start with an empty cluster. core/setups/nixos/postgresql.nix sets the major in the code. When the major goes up by one, its guard runs pg_upgrade --copy from the old cluster. For each other change of the major, it keeps PostgreSQL stopped with exit status 78. MariaDB has a rule of its own in core/setups/nixos/mysql.nix: the module runs mariadb-upgrade after each start of the server, because an upgrade is complete only after that command.

The maintainer steps for an update, and the reasons for each service, are in Update nixpkgs. Each entry has a check in core/checks/integrations/, for example garage-series and openbao-series.

The schema of the api or the panel changes

The two components keep their records in a PostgreSQL database of the hosting provider. Each applies its own migrations, and each has a systemd.stateMigrations entry that only records the version, refuses a downgrade, and makes the backup.

• api/ applies the Alembic revisions when it opens its store (_migrate in api/src/fediversity_api/state.py). A PostgreSQL advisory lock keeps a second process out of the same migration. api/ refuses a database at a revision that its release does not know. The revisions only go forward.
• panel/ runs manage migrate in the preStart of panel.service (panel/nix/configuration.nix), each time the package changes. Django does not refuse a database that later migrations changed, thus the version record is the only guard against a downgrade. The panel has no migrations that go back.
• The version of each entry is the number of the last migration file. Thus the version changes only when the schema changes. For the panel, the migrations of the Django apps from nixpkgs are not in the version.
• The backup comes from mkUpgradeBackupFor in core/components/operator/app-resources.nix: on a node with the offsite backup of core/components/shared/postgres-backup.nix, a dump of the database before the upgrade. Without that backup, the entry makes no backup.
• The largest step is not limited: the entries do not set maxSkip. Alembic and Django apply each migration between the two versions in sequence.

The two entries are not in the table of the section before, because the nodes of api/ and panel/ are not nodes of core.

An effect fails

An effect can fail, or a person or a timeout can stop it, at any point. No effect goes back to the state from before the run. Each effect is recovered forward: run it again. For a deployment graph, notes/forward-recovery.md gives the rules and tells how to read a failure.

The docs build generates this table. It fails when a file of a row no longer has the code that does what the row says. The build does not find a new retry or a new effect that has no row.

Gaps:

• No effect tries a failed run again by itself. The retries in the table are inside one run.
• A restore of a migration and some operations of an application do not say that a second run is safe.

What this page does not cover

• The list of each place where state is kept is in the part state-register.
• The backups, their retention and their restore are in the hosting provider guide.
• A move of an operator between hosting providers is in the part capability-flows.