Change a key after a leak

Change a key only when it leaked. The KV path of a key is <prefix>/secret/<instance>/<file> in the mount secret. The prefix is vars for the hosting provider and vars/operator-deployments/<namespace> for an operator deployment. The value is base64 in the field content. A redeploy does not write KV again when the key is there, thus a value that you write stays.

Keep the old versions of a key. The backups from before the change need the old key until the retention removes them (see Restore a backup that depends on keys). The OpenBao node sets max_versions = 1000000 and cas_required = true in the KV metadata of each key that a backup depends on, thus KV keeps all versions of these keys, and refuses a write that does not give the current version. Give the current version with -cas: bao read -field=current_version secret/metadata/<path> shows it.

Write the new value to a file on /run (a tmpfs), and remove the file at the end.

Authelia storage encryption key

The instance is authelia-oidc for the hosting provider and <group>-authelia-oidc for the authelia of an operator group. The file is storageEncryptionKey. Authelia can encrypt its data again with a new key, but it cannot use two keys at the same time. On the node of authelia:

Make the new key: openssl rand -hex 32 > /run/new-key.
Stop authelia: systemctl stop authelia-main.

Find the authelia binary and the --config files in ExecStart of systemctl cat authelia-main. Do not use the file oidc-jwks.yaml. Encrypt the data again:

AUTHELIA_STORAGE_ENCRYPTION_KEY_FILE=/run/vars/secret/<instance>/storageEncryptionKey \
AUTHELIA_STORAGE_POSTGRES_PASSWORD_FILE=/run/vars/secret/authelia-db/password \
<authelia> storage encryption change-key --config <config.yml> --new-encryption-key "$(cat /run/new-key)"

Write the new key to KV: bao kv put -mount=secret -cas=<version> <prefix>/secret/<instance>/storageEncryptionKey content="$(base64 -w0 < /run/new-key)".
Restart openbao-agent-default, and make sure that /run/vars/secret/<instance>/storageEncryptionKey has the new key.
Start authelia: systemctl start authelia-main. With the old key, authelia does not start.

Pixelfed `APP_KEY`

The instance is pixelfed-app and the file is appKey. Pixelfed encrypts the cookies and the S3 and captcha keys of its admin settings with this key. When it cannot decrypt the admin settings, it uses the values of its configuration. Pixelfed also decrypts with the old keys in the instance pixelfed-app-previous, file appKeys. This is a list of keys, separated by commas, and it is empty by default. Thus the sessions and the admin settings stay during the change.

While the old key is in the list, a person who has the old key can still make cookies that pixelfed accepts. Thus do the steps without a pause, and remove the old key from the list at the end.

Make sure that the list is empty: bao kv get -mount=secret -field=content <prefix>/secret/pixelfed-app-previous/appKeys shows nothing. If it shows a value, do the steps 5 and 6 of the last change first.
Make the new key: printf '%s' "$(openssl rand -hex 16)" > /run/new-key.

Write the current key to the list. The field content of appKey is base64 already:

bao kv put -mount=secret <prefix>/secret/pixelfed-app-previous/appKeys \
  content="$(bao kv get -mount=secret -field=content <prefix>/secret/pixelfed-app/appKey)"

Write the new key to KV: bao kv put -mount=secret -cas=<version> <prefix>/secret/pixelfed-app/appKey content="$(base64 -w0 < /run/new-key)". The OpenBao agent writes the new environment file, and pixelfed-data-setup restarts pixelfed with the new key (at most 5 minutes). On the node of pixelfed, grep -qxF "APP_KEY=$(cat /run/new-key)" /var/lib/pixelfed/.env && echo ok shows ok when pixelfed has the new key.

On the node of pixelfed, encrypt the admin settings again with the new key:

pixelfed-manage tinker --execute='foreach (App\Models\ConfigCache::whereIn("k", App\Services\ConfigCacheService::PROTECTED_KEYS)->whereNotNull("v")->get() as $c) { $c->v = encrypt(decrypt($c->v)); $c->save(); }'

If the command shows The MAC is invalid, a value has a key that is not in the list. Set that value again in the admin UI.

Remove the old key from the list: bao kv put -mount=secret <prefix>/secret/pixelfed-app-previous/appKeys content="". Pixelfed restarts again. A user that did not use pixelfed between the steps 4 and 6 must log in again.

The old key stays in the old versions of appKey. A backup of the database from before step 5 has the admin settings with the old key. After a restore of such a backup, do the steps 3 to 6 with the old key in the list.

Grafana `secretKey`

The instance is telemetry-grafana and the file is secretKey. Grafana encrypts the secrets in its database with data keys, for example the secrets that an admin adds in the UI to a data source or a contact point. It encrypts the data keys with this key. Grafana reads only one key, and with an incorrect key it decrypts the secrets to incorrect values without an error. The provisioned data sources and contact points do not need the old secrets, because Grafana writes them again at each start.

Before Grafana starts, grafana-secret-key makes the data keys agree with the current key. The table fediversity_secret_key in the database of Grafana holds the SHA-256 digest of the key that encrypts the data keys. When the current key has a different digest, the program finds the old key in the instance telemetry-grafana-previous, file secretKeys, decrypts the data keys with it, and encrypts them again with the current key. The file has one key on each line, and it is empty by default. When no key in the file has the digest of the database, Grafana does not start, and journalctl -u grafana shows grafana-secret-key: the key changed. Grafana tries to start again each 30 seconds. Grafana also does not start before upload-vars-openbao is active, thus it does not use the value that the node makes at boot in place of the value in KV.

Make sure that the list is empty: bao kv get -mount=secret -field=content <prefix>/secret/telemetry-grafana-previous/secretKeys shows nothing. If it shows a value, do step 5 of the last change first.
Make the new key: printf '%s' "$(openssl rand -hex 32)" > /run/new-key.

Write the current key to the list. The field content of secretKey is base64 already:

bao kv put -mount=secret <prefix>/secret/telemetry-grafana-previous/secretKeys \
  content="$(bao kv get -mount=secret -field=content <prefix>/secret/telemetry-grafana/secretKey)"

Write the new key to KV: bao kv put -mount=secret -cas=<version> <prefix>/secret/telemetry-grafana/secretKey content="$(base64 -w0 < /run/new-key)". The OpenBao agent writes the two files and restarts Grafana with the new key (at most 5 minutes). On the node of Grafana, journalctl -u grafana shows encrypted <n> legacy and <m> secret_key.v2 data keys again with the current key.
Remove the old key from the list: bao kv put -mount=secret <prefix>/secret/telemetry-grafana-previous/secretKeys content="".

The old key stays in the old versions of secretKey. A backup of the database from before step 4 has the digest of the old key. To restore such a backup, write the old key to the list with step 3 and -version=<n> on bao kv get, restore with RESTORE_IGNORE_KEYS=1, and then do step 5. A backup from before grafana-secret-key has no digest. The program then records the digest of the current key, thus if the backup has a different key, the secrets in it do not decrypt. Set them again.

To start Grafana without the old key, delete the digest: sqlite3 /var/lib/grafana/data/grafana.db 'DELETE FROM fediversity_secret_key'. Grafana then starts with the current key, and the secrets that the old key encrypts do not decrypt. Set them again.

Password of the restic repositories

The instance is <app>-backups-restic and the file is password. All repositories of an application have the same password. restic encrypts the data with a master key, and the password encrypts only the master key. Thus a new password does not change the snapshots, and the old snapshots open with the new password. An attacker that copied the key files of a repository can still decrypt it with the old password.

For each repository of the application, restic-<app>_<name> runs restic with the current password, for example restic-pixelfed_postgres.

Make the new password: printf '%s' "$(openssl rand -hex 32)" > /run/new-key.
For each repository, add the new password: restic-<app>_<name> key add --new-password-file /run/new-key. Write down the ID of the old key from restic-<app>_<name> key list.
Write the new password to KV: bao kv put -mount=secret -cas=<version> <prefix>/secret/<app>-backups-restic/password content="$(base64 -w0 < /run/new-key)".
Restart openbao-agent-default, and make sure that restic-<app>_<name> key list shows the new key as the current key (*).
For each repository, remove the old password: restic-<app>_<name> key remove <old ID>.

S3 key of a garage bucket

This procedure is for the keys that the unit garage-provision-own of a garage node holds. These are the keys of the group of that node, for example the keys of the telemetry stores and of the cache backups of valkey. The keys of an operator deployment and the keys of a node with its own garage have their own procedures, below.

The instance is <name>-s3. The access key ID is not secret, thus its path is <prefix>/public/<name>-s3/accessKey. The path of the secret is <prefix>/secret/<name>-s3/secretKey. An S3 key gives access to the bucket and encrypts no data. Thus the backups from before the change stay usable with the new key, and you do not need the old versions.

garage cannot change the secret of an access key ID, thus the change makes a new ID and a new secret. Between step 2 and the end of step 4, a consumer can have a key that garage refuses. Do the steps without a pause, and not while a backup runs.

Make the new pair: printf 'GK%s' "$(openssl rand -hex 12)" > /run/new-id and openssl rand -hex 32 > /run/new-secret.
Write the two values to KV: bao kv put -mount=secret <prefix>/public/<name>-s3/accessKey content="$(base64 -w0 < /run/new-id)" and bao kv put -mount=secret <prefix>/secret/<name>-s3/secretKey content="$(base64 -w0 < /run/new-secret)".
On the garage node, restart openbao-agent-default, then restart garage-provision-own. The unit deletes the key of the old ID with its grants, and imports the new pair with the same grants. garage key list then shows the name of the key one time, with the new ID.
On each node that uses the key, restart openbao-agent-default. The agent writes the new pair and restarts the services that use it. A restic or rclone job reads the pair at each start.
Remove /run/new-id and /run/new-secret.

S3 key of an operator deployment

The operator graph rotates the S3 keys of an operator deployment at the first apply of each month (UTC). The name of a key in garage is <name>.<YYYY-MM>.g<generation>. To rotate the keys at once, for example after a leak, increment the setting garageKeyGeneration of the deployment, and deploy it again.

At a rotation, the apply makes the new key with the grants of the old key, and writes the new pair to KV. Then it waits for garage_key_drain_seconds (600 by default), and deletes the old key and its grants. During the wait, garage accepts the old pair and the new pair, and the agent of each consumer writes the new pair and restarts the services that use it. A job that reads the pair when it starts, for example a backup, and that runs for longer than the wait, loses access when the apply deletes the old key. The next run of the job reads the new pair.

When a rotation stops before the old key goes, the old key stays in garage. The next apply waits again and then deletes it.

S3 key of a node with its own garage

The unit ensure-garage of a node with its own garage (the in-nix provisioning) imports the pair of the key files, and then deletes each other key with the same name. Thus garage refuses the old pair from the time that the unit runs until each consumer reads the new pair. Do the steps without a pause, and not while a backup runs.

Make the new pair, and write it to KV, as in steps 1 and 2 above. A node that does not read its secrets from OpenBao gets the new pair in /run/vars/public/<name>-s3/accessKey and /run/vars/secret/<name>-s3/secretKey.
On the node, restart openbao-agent-default (when the node reads its secrets from OpenBao), then restart ensure-garage. garage key list then shows the name of the key one time, with the new ID.
Restart the services that use the key, if the agent did not restart them.
Remove /run/new-id and /run/new-secret.

DKIM key of a group

The mail relay of a group signs its outbound mail with one DKIM key, as the organizational domain of the group (d=<domain>). The selector names the key: a receiver gets the public key from the TXT record <selector>._domainkey.<domain>. The DMARC record of the domain has sp=reject, thus a receiver rejects a mail whose signature it cannot verify. A key does not rotate automatically. Rotate it by hand, with a new selector beside the old selector.

The group data (core/setups/groups/<group>.nix) sets the selectors in smtp.dkim:

selectors: the list of selectors. The relay has a key for each selector, and the deploy publishes a TXT record for each selector. The default is [ "fediversity" ].
selector: the selector that the relay signs with. It must be in selectors. The default is fediversity.

The deploy makes the key of a new selector, and keeps the keys that exist. Do the rotation in three deploys:

Add the new selector to selectors, and do not change selector. For example:
```
smtp.dkim = {
  selector = "fediversity";
  selectors = [ "fediversity" "fediversity2" ];
};
```
Deploy the group. The relay makes the key of fediversity2, and the deploy publishes its record. The relay still signs with fediversity.
Wait until the receivers can get the new record, then set selector = "fediversity2" and deploy again. Do not do this step before the record is in DNS: a receiver that cannot get the record of the selector rejects the mail (sp=reject). A receiver that asked for the name before step 1 keeps the negative answer in its cache, for the time that the SOA record of the zone gives. Thus wait for the longer of the TTL of the record and this time, plus some minutes. Each authoritative server of the zone and a public resolver must give the record:
```
dig +short TXT fediversity2._domainkey.<domain> @<name server>
dig +short TXT fediversity2._domainkey.<domain> @9.9.9.9
```
The value must start with v=DKIM1;, and must be equal to the file <domain>_fediversity2.txt of the instance <smtp instance>-dkim in KV (<prefix>/public/<smtp instance>-dkim/<domain>_fediversity2.txt in the mount secret, with the prefix at the start of this page).
Wait until the mail that the relay signed with the old key has arrived, then remove the old selector from selectors and deploy again. maddy retries a mail for some days, and a receiver can check the signature again later, for example for a forwarded mail. Thus wait at least one week. The deploy removes the record of the old selector. The key file of the old selector stays on the node, and in KV.

To rotate again, add a new selector, for example fediversity3. Do not use a selector again: a receiver can have the old record in its cache.

Signing key of the bridge cache

The incus host serves its store to its containers as a binary cache on the bridge (nix-serve on http://10.0.100.1:5000), and signs each path that the cache serves. The key is in /var/lib/incus-bridge-cache/ on the host (core/components/hosting-provider/incus.nix). This key is not in KV, and it does not go out of the host. Each container gets only the public keys, as the nix.conf fragment /var/lib/incus-bridge-cache/nix.conf, through the device bridgecache of the default profile. The configuration of a node holds only the path of the fragment (!include in core/setups/nixos/incus-bridge-cache.nix), not the keys. Thus a new key needs no redeploy.

The host changes the key on a schedule:

The timer incus-bridge-cache-key-rotate runs one time each day. When the current key is fediversity.incus.bridgeCacheKey.rotationDays old (90 by default), it makes a new key and restarts nix-serve, which then signs with the new key only.
The fragment holds the new key and the previous key, until the new key is fediversity.incus.bridgeCacheKey.overlapDays old (30 by default). Then the timer removes the previous key from the fragment.
Each key has its own name, <host>-bridge-cache-<n>, with a number that increases at each change. Nix keeps only one key for each name, thus the names must be different for a node to trust the two keys.
The age of a key is the modification time of public-key.

nix-serve signs each narinfo when it serves it, thus the paths in the store need no new signature after a change of the key. But a node keeps the narinfo that it got in its disk cache (narinfo-cache-positive-ttl, 30 days by default), with the signature of the old key. In the overlap, the node can use this narinfo. After the overlap, the node refuses it, and substitutes the path from the upstream caches, or a deploy copies it from the deployer.

The containers see the new fragment at once, because the device binds the directory. A Nix command reads the fragment each time that it starts, thus a deploy uses the new keys at once. The nix daemon of a node reads nix.conf only when it starts. Thus the path unit nix-daemon-bridge-cache-key of the node restarts the daemon when the fragment changes. The restart does not stop the builds that run: a connection that is open at the restart keeps the old keys. A node that shares the store of the host does not substitute from the bridge cache.

Change the key after a compromise

When the secret key is possibly known to an attacker, change the key at once, and stop the trust in the old key at once. On the incus host:

Change the key: systemctl start incus-bridge-cache-key-revoke. The unit makes a new key, restarts nix-serve, and then writes only the new public key to the fragment. With this order, a node that gets a narinfo between the two steps gets the signature of the new key.

Make sure that the cache signs with the new key. A TTL of 0 makes Nix ignore its disk cache:

nix --extra-experimental-features nix-command store verify --no-contents \
  --option narinfo-cache-positive-ttl 0 \
  --store http://10.0.100.1:5000 "$(readlink -f /run/current-system)" \
  --trusted-public-keys "$(cat /var/lib/incus-bridge-cache/public-key)"

Make sure that the nix daemon of each container uses only the new key. The path unit nix-daemon-bridge-cache-key restarts the daemon when the fragment changes, but a virtual machine possibly does not see the change of a shared directory. Thus restart the daemon in each container. The restart does not stop the builds that run. incus list --all-projects shows the containers and their projects. For each container:
```
incus exec --project <project> <instance> -- \
  /run/current-system/sw/bin/systemctl try-restart nix-daemon.service
```
Then make sure that the container trusts only the new key:
```
incus exec --project <project> <instance> -- \
  /run/current-system/sw/bin/nix --extra-experimental-features nix-command \
  config show trusted-public-keys
```

A node keeps the narinfo that it got before the revoke in its disk cache, with the signature of the old key. The node refuses this narinfo until it expires (narinfo-cache-positive-ttl), and substitutes these paths from the upstream caches.

The schedule continues from the new key.

When `nix-serve` does not start after a change of the key

A change of the key restarts nix-serve. The unit that changes the key (incus-bridge-cache-key-rotate or incus-bridge-cache-key-revoke) also writes the new fragment when this restart fails. Thus the key and the fragment are new, and the cache can be down.

nix-serve tries to start again each 5 seconds. Its state stays activating, not failed. While the cache is down, the nodes substitute from the upstream caches only. You get these alerts:

unit-stuck for nix-serve, after approximately 40 minutes.
unit-failed for the unit that changed the key, after 10 minutes, only when the start of nix-serve fails before its process runs. When the process runs and then stops, the unit that changed the key is not failed, and unit-stuck is the only alert.

On the incus host:

Find the cause:

journalctl -u nix-serve.service -n 50 --no-pager

Correct the cause. nix-serve then starts by itself in 5 seconds, with the new key. Do not start the unit that changed the key again: it can change the key one more time.
Make sure that the cache signs with the new key, with the command in step 2 of Change the key after a compromise.
Clear the failed state, if the unit has it. Otherwise the next run of the timer clears it for incus-bridge-cache-key-rotate:
```
systemctl reset-failed <unit>
```

Keys that cannot change without data loss

The applications below cannot encrypt their data again with a new key. To change one of these keys, write a new value to KV, restart the application, and accept the loss:

PeerTube secrets.peertube: the two-factor authentication of all users that use it fails. These users must set it up again.
lldap key seed: lldap does not start with a new seed. With LLDAP_FORCE_UPDATE_PRIVATE_KEY=true it starts, and all users must set a new password.
Mastodon Active Record encryption keys: the private keys of the accounts and the two-factor secrets do not decrypt. Mastodon reads only one primary key, thus it cannot use the old and the new key at the same time.
Mastodon SECRET_KEY_BASE and VAPID keys: no data is lost, but all users must log in again, the links in password reset and confirmation emails stop working, and the push subscriptions stop until the clients subscribe again.