Fediversity

Update nixpkgs

The pin of nixpkgs is in npins/sources.json. system.stateVersion does not follow the release of that pin: it is a constant in core/setups/nixos/common.nix, and core/setups/nixos/incus-base-image.nix has the same value.

The rule of the version entries of the services, and a table of the entries, are in A service gets a new version. This section gives the steps of the maintainer.

Why system.stateVersion is a constant

Some NixOS modules select the format or the location of their state from system.stateVersion. For example, the module of PostgreSQL selects the major version from it, and each major keeps its cluster in a directory of its own. A change of the value changes the default on each node, but the module does not migrate the state to the new format. Without a guard, PostgreSQL starts an empty cluster, and the applications start on empty databases.

The integration tests start each node with no state. A new format has no effect there, thus the tests pass. Only a node with state shows the problem. No check in CI starts a node on the old nixpkgs and then switches it to the new nixpkgs.

With a constant value:

  • An update of nixpkgs does not change the format of the state. You can revert the update without a migration back.
  • Each change of format is a separate change in this repository, together with its migration. A reviewer sees it in the diff.
  • The value has its upstream meaning: the release of NixOS that made the state.

Packages that follow nixpkgs

system.stateVersion does not select each package with a format of state. For example, pkgs.mariadb is the release series of MariaDB that nixpkgs selects, and an update of nixpkgs can change it. When the release series of MariaDB changes, core/setups/nixos/mysql.nix:

  • Copies the data directory to /var/lib/mysql-state-migrations before MariaDB starts on a newer release series.
  • Keeps MariaDB stopped on an older release series.
  • Runs mariadb-upgrade after each start of MariaDB.

core/checks/integrations/mariadb-series is its check.

OpenBao also starts on an older release series without an error. When the release series of OpenBao changes, core/components/hosting-provider/openbao/migrations.nix:

  • Does not restart OpenBao on a switch, as the upstream unit. stale-server.nix restarts OpenBao after a change in the recorded release series, and tells the operator to restart openbao.service after a change of the release series.
  • Copies /var/lib/openbao to /var/lib/openbao-state-migrations before OpenBao starts on a newer release series.
  • Keeps OpenBao stopped on an older release series, also after a restart that follows a failure.

core/checks/integrations/openbao-series is its check.

garage also starts on an older release series without an error. When it cannot read its cluster layout, it starts with an empty layout, and core/resources/garage/layout-bootstrap.nix assigns a new one. When the release series of garage changes, core/resources/garage/migrations.nix:

  • Copies the metadata directory /var/lib/garage/meta to /var/lib/garage-state-migrations before garage starts on a newer release series. The data blocks stay in /var/lib/garage/data: the copy of the metadata refers to them by their hash.
  • Keeps garage stopped on an older release series.

core/checks/integrations/garage-series is its check.

The preStart of forgejo.service runs forgejo migrate, which changes the schema of the database to the release series of Forgejo. When the release series of Forgejo changes, core/components/dev/forgejo/migrations.nix:

  • Backs up the database of Forgejo before Forgejo starts on a newer release series. With the offsite backup of core/components/dev/forgejo/backup.nix, it makes a restic snapshot with the tags upgrade and version:<release series>. Else, it dumps the database to /var/lib/forgejo-state-migrations. The repositories and the other files stay in /var/lib/forgejo.
  • Keeps Forgejo stopped on an older release series.

core/checks/integrations/forgejo-series is its check.

Grafana migrates its SQLite database when it starts, and also starts on a database that a later release migrated. When the version of Grafana changes, core/components/hosting-provider/telemetry/migrations.nix:

  • Backs up the database of Grafana before Grafana starts on a later version. With the offsite backup of core/components/hosting-provider/telemetry/backup.nix, it makes a restic snapshot with the tags upgrade and version:<version>. Else, it copies the database to /var/lib/grafana/state-migrations.
  • Keeps Grafana stopped on an earlier version.

core/checks/integrations/grafana-series is its check. Grafana uses SQLite, not the shared PostgreSQL node, thus its alerts about PostgreSQL do not stop when PostgreSQL stops. The comment at services.grafana.settings.database in core/components/hosting-provider/telemetry.nix gives all the reasons.

Tempo reads the blocks of each format that it knows, and writes new blocks in the format of storage.trace.block.version in core/components/hosting-provider/telemetry.nix. Thus an upgrade of Tempo does not change the format, and the earlier release reads the blocks of the later release. To use a new format, first upgrade to a release that reads it, then change block.version in a separate change. A release that cannot write the format of block.version does not start.

BIND reads the zone files, the journals and the DNSSEC keys of an earlier release when it starts, and writes them in its own format. A release can change the format of the journals or of the key state. When the version of BIND changes, core/components/hosting-provider/bind-migrations.nix:

  • Copies /var/lib/named to /var/lib/named-state-migrations before named starts on a later version. The copy holds the private keys.
  • Starts named on an earlier version, with a warning in the journal. If named does not load a zone, put the files of that zone from the copy back.

core/checks/integrations/bind-series is its check.

The OpenTofu state backend has no migration command, and keeps its locks in files in its state directory. A switch restarts it, and an apply keeps its lock through the restart. When the version of the backend changes, core/components/hosting-provider/terraform-backend/migrations.nix:

  • Runs its guard when the switch restarts terraform-backend.service. When the guard refuses, the backend stays stopped and the switch fails.
  • Copies /var/lib/terraform-backend to /var/lib/terraform-backend-state-migrations before the backend starts on a later version.
  • Keeps the backend stopped on an earlier version. The copy does not have the states that the backend wrote after the upgrade.

core/checks/integrations/tf-backend-series is its check.

maddy changes the schema of its mail database imapsql.db when it starts, and does not start on a schema that a later release made. Each release can change the schema. When the version of maddy changes, core/resources/smtp/maddy-migrations.nix:

  • Copies /var/lib/maddy to /var/lib/maddy-state-migrations before maddy starts on a later version.
  • Starts maddy on an earlier version, with a warning in the journal. If maddy does not start, put imapsql.db from the copy back.

core/checks/integrations/maddy-series is its check.

Each Windmill unit migrates the database when it starts, and an earlier release of Windmill cannot use a database that a later release migrated. When the version of Windmill changes, core/components/hosting-provider/windmill/migrations.nix:

  • Dumps the database of Windmill to /var/lib/windmill-state-migrations before Windmill starts on a later version, on a node with the offsite backup of the database.
  • Keeps the Windmill units stopped on an earlier version.

core/checks/integrations/windmill-series is its check.

lldap migrates its database when it starts, and refuses a database that a later release migrated. When the version of lldap changes, core/components/shared/lldap/migrations.nix:

  • Dumps the database of lldap to /var/lib/lldap-state-migrations before lldap starts on a later version, on a node with the offsite backup of the database.
  • Keeps lldap stopped on an earlier version.

core/checks/integrations/lldap-series is its check.

Authelia migrates its storage when it starts, and refuses a storage that a later release migrated. When the version of Authelia changes, core/components/shared/authelia/migrations.nix:

  • Dumps the database of Authelia to /var/lib/authelia-main/state-migrations before Authelia starts on a later version, on a node with the offsite backup of the database.
  • Keeps Authelia stopped on an earlier version.

core/checks/integrations/authelia-series is its check.

valkey reads its RDB snapshot and its AOF when it starts, and refuses data in the RDB format of a later release series. When the release series of valkey changes, core/components/shared/valkey/migrations.nix:

  • Copies the RDB snapshot and the AOF directory of each instance to /var/lib/redis-<namespace>-state-migrations before valkey starts on a later release series.
  • Starts valkey on an earlier release series, with a warning in the journal. To go back to a release series that refuses the data, stop redis-<namespace>.service and put the copy at the place of the data.

core/checks/integrations/valkey-series is its check.

The audit shipper keeps its journal cursor and the queue of the lines that Loki did not accept in the file_storage of the OpenTelemetry collector. An earlier release of the collector cannot read the queue of a later release. When the version of the collector changes, core/components/hosting-provider/audit-shipper-migrations.nix:

  • Copies /var/lib/fediversity-audit-shipper to /var/lib/fediversity-audit-shipper-state-migrations before the shipper starts on a later version.
  • Keeps the shipper stopped on an earlier version. To go back to the version before the last upgrade, put the copy at the place of the state, then start fediversity-audit-shipper-migrate.target. The shipper then sends again the lines after the cursor of the copy.

The collector records the cursor of a line before the line is in the queue. Thus, at each start, the shipper reads the journal again from services.fediversity-audit-shipper.rewindSeconds (60 by default) before the line of the cursor, and sends these lines again. Loki keeps one line of each duplicate.

core/checks/integrations/audit-shipper-series is its check.

SPIRE migrates the datastore of the server when the server starts. The server starts on a datastore of the release series before or after its own, and refuses a datastore that is farther away. When the release series of SPIRE changes, core/components/hosting-provider/spire/migrations.nix:

  • Copies the datastore, without the CA key keys.json, to /var/lib/spire-server-state-migrations before the server starts on a newer release series. The server is stopped during the copy. Only the last copy stays.
  • Starts the server on an earlier release series, with a warning in the journal.

core/checks/integrations/spire-series is its check.

NetBox migrates its database before netbox.service starts. The migrations go in one direction only. When the version of NetBox changes, core/components/hosting-provider/netbox/migrations.nix:

  • Dumps the database of NetBox to the state-migrations directory of its data directory before NetBox starts on a later version, on a node with the offsite backup of the database.
  • Keeps NetBox stopped on an earlier version, and on a version that skips a major release.

The applications of an operator have the same guard. When the version of an application changes, core/components/operator/mastodon/migrations.nix, core/components/operator/peertube/migrations.nix and core/components/operator/pixelfed/migrations.nix:

  • Dump the database of the application to the state-migrations directory of its state directory before the application starts on a later version, on a node with the offsite backup of the database.
  • Keep the application stopped on an earlier version. Mastodon also stays stopped on a version that skips a minor release series.

The migrations of these applications go in one direction only. Thus, the dump is the only way back to the version before.

Change system.stateVersion

Before you change the value, make sure that no node gets a new format of its state without a migration:

  1. Find the modules that read the value. Search the source of the nixpkgs pin:

    grep -rn stateVersion "$(nix-instantiate --eval --raw -E '(import ./npins).nixpkgs.outPath')/nixos/modules"

    Keep the modules of the services that a node of Fediversity enables, and the modules with a different result for the old and the new value.

  2. For each of these modules, do one of these steps:

    • Set the option in this repository, so that system.stateVersion does not select it. For example, core/setups/nixos/postgresql.nix sets services.postgresql.package. A later change of that option needs a migration. For PostgreSQL, the module has one: a change to the next major upgrades the cluster with pg_upgrade.
    • Add a migration with systemd.stateMigrations that moves the state of a running node to the new format. Add a check that starts a node on the old format, writes data, switches the node to the new format, and reads the data. core/checks/integrations/postgresql-major is an example.

  3. Change the value in core/setups/nixos/common.nix and core/setups/nixos/incus-base-image.nix in the same change.