Feature Specification: Secret rotation

Created: 2026-10-22

Status: Partially implemented

Input: Reconstructed from the secrets contracts, the openbao component, the database roster, the resources and the components of core at 1f4c46e3d.

User Scenarios & Testing (mandatory)

A deployment has three kinds of secrets:

• Engine credentials: an OpenBao secrets engine holds them and changes them on a schedule. The consumers read them through an OpenBao agent.
• Generated secrets: a generateFiles instance makes them once and uploads them to the KV store of OpenBao. The upload does not write over a value that exists. Thus a generated secret changes only when an operator forces a new upload of it, or when its KV entry is lost and the next deploy makes a new value.
• Stored secrets: a person encrypts them in the repository with agenix (the secrets resource), or keeps them outside the deployment. A person rotates them by hand, as the guide of the secrets resource tells.

Some generated secrets encrypt data that the offsite backups hold. A backup restores only with the key that encrypted it.

User Story 1 - Know how each secret rotates (Priority: P1)

An operator or an auditor wants to know, for each secret, how it changes and which backups need it. This specification is that list.

Why this priority: a rotation of a key that a backup needs makes that backup unreadable. An operator must know this before a rotation.

Independent Test: the table in Key Entities agrees with the generators and the engine roles that it names.

Acceptance Scenarios:

1. Given a secret in the table, When an operator reads its row, Then the row tells the rotation (schedule, manual or never), the backups that depend on it, and how the consumers get the new value.

User Story 2 - Rotate the engine credentials on a schedule (Priority: P1)

The static roles of the PostgreSQL, valkey and LDAP engines rotate daily. The agent of the consumer renders the new value and reloads or restarts the consumer.

Why this priority: a long-lived database or directory password that leaks stays usable for at most one rotation period.

Independent Test: an integration test of the PostgreSQL rotation, and a service test of valkey.

Acceptance Scenarios:

1. Given a PostgreSQL static role, When the engine rotates it, Then the agent renders the new password and restarts the units that the consumer names, and the consumer connects with the new password.
2. Given the LDAP role of the authelia bind, When a bind fails because of invalid credentials, Then the LDAP engine rotates the role.

User Story 3 - Restore a backup after a change of a key that it depends on (Priority: P2)

The keys that a backup depends on do not rotate. An operator can change such a key by hand, for example by a forced upload. The backups from before the change must then stay usable for as long as the retention keeps them. KV keeps the old value, so the operator gets it back with a rollback.

Why this priority: a restore with a different key gives data that the application cannot read, and the error comes only later, when the application reads the data.

Independent Test: a service test of mastodon changes a key in KV, checks that the restore stops and names the version of the key, rolls the key back, and checks that the restore works.

Acceptance Scenarios:

1. Given a backup from before a key change, When the operator restores it, Then the restore stops before it changes data, and names each key that is different and the KV version that the backup needs.
2. Given the operator rolled each key back to the version that the backup needs, When the operator restores again, Then the restore works.

Requirements (mandatory)

Functional Requirements

• FR-001: An engine static role MUST rotate on the schedule of its roster (daily). The agent MUST render the new value and reload or restart the units that the consumer names. A role with a sidecar (pod-roles) MUST restart its service with the option restart of the secret.
• FR-002: A generateFiles upload MUST NOT write over a value that exists in KV. Only a forced upload changes a generated secret.
• FR-003: A secret that encrypts data in a backup MUST NOT change on a deploy, a redeploy or a recreate of its node.
• FR-004: The repository password of each restic backup MUST be a generated secret. A change MUST add a key for the new password to each repository before the backups read the new password, and MUST remove the key of the old password after it. The operator does these steps by hand, as the hosting provider guide tells. A restic job opens its repository only with its current password.
• FR-005: A key that a backup depends on MUST NOT rotate on a schedule. Such a key encrypts data in the backup, or data that the backup holds. A key that only gives access to the store of a backup, such as a garage S3 key, is not such a key: a backup records no digest of it, and a restore does not compare it. Each backup MUST record a digest and the KV version of each such key. A restore MUST stop on a snapshot whose keys are different from the keys of the node, and MUST name the KV version that the snapshot needs.
• FR-006: KV MUST keep the old versions of a key that a backup depends on, and MUST refuse a write to such a key that does not give the current version (check-and-set).
• FR-007: The pixelfed app key MUST have a list of old app keys beside it, which is empty by default, and pixelfed MUST also decrypt with each key in that list. A change MUST add the old key to the list before pixelfed reads the new key, and MUST remove the old key from the list after pixelfed encrypts its data again with the new key. The operator does these steps by hand, as the hosting provider guide tells.
• FR-008: The Grafana secret key MUST have a list of old secret keys beside it, which is empty by default. The Grafana database MUST record the digest of the key that encrypts its data keys. Before Grafana starts with a key that has a different digest, the node MUST encrypt the data keys again with the new key, with the old key from the list. When the list does not hold the old key, Grafana MUST NOT start. A change MUST add the old key to the list before Grafana reads the new key, and MUST remove the old key from the list after Grafana starts with the new key. The operator does these steps by hand, as the hosting provider guide tells.
• FR-009: A garage S3 key that the reconciler of a garage node holds MUST change by hand, with a new access key ID and a new secret in KV. The next run of the reconciler MUST delete the key of the old ID and MUST give the new key the same grants. Between the write to KV and the time that the reconciler and each consumer read the new pair, a consumer can have a pair that garage refuses. The operator does these steps by hand, as the hosting provider guide tells.
• FR-010: A garage S3 key of the operator graph MUST rotate at the first apply of each month (UTC), and at the next apply after the operator increments the setting garageKeyGeneration of the deployment. The name of the key in garage MUST carry the month and the generation, so the apply makes a new key beside the old key. The apply MUST give the new key the grants of the old key and MUST write the new pair to KV before it waits. It MUST wait for the drain time (garage_key_drain_seconds, default 600 seconds, about two times the interval at which an agent reads KV), and MUST delete the old key and its grants after the wait. During the wait, garage MUST accept the old pair and the new pair. A job that reads the pair when it starts, for example a backup, and that runs for longer than the wait, loses access when the apply deletes the old key.
• FR-011: A garage S3 key that the ensure-garage unit of a node with its own garage imports MUST change by hand, with a new access key ID and a new secret in KV. The next run of ensure-garage MUST import the new key and MUST delete each other key with the same name. Between this run and the time that each consumer reads the new pair, a consumer can have a pair that garage refuses. The operator does these steps by hand, as the hosting provider guide tells.
• FR-012: The DKIM key of a group MUST change by a new selector. The group data MUST give the list of selectors and the selector that the relay signs with, which MUST be in the list. The relay MUST have a key for each selector in the list, and a deploy MUST publish a TXT record for each selector in the list. A deploy MUST NOT change the key of a selector that has a key. The operator MUST NOT make the relay sign with a selector before DNS has its record, because the DMARC record has sp=reject. The operator does these steps by hand, as the hosting provider guide tells.

Key Entities

The credentials register of the technical design (core/docs/md/technical-design/credentials.md) is the source for the list of the credentials and for the state of their rotation. This table adds the backups that depend on a secret and how the consumers get a new value.

"Manual" means: the secret changes only with a forced upload. "Agent" means: the OpenBao agent renders the value and reloads or restarts the units that the consumer names.

Success Criteria (mandatory)

Measurable Outcomes

• SC-001: Each secret that a generateFiles instance or an engine role makes has a row in the table.
• SC-002: The PostgreSQL rotation tests, the service tests of valkey and mastodon, and test-monitoring-storage (a change of the Grafana secret key) pass.
• SC-003: integrations-operator-garage passes. It rotates a key by a new generation and by a new period, and shows that the old pair and the new pair in KV have access to the bucket during the wait, and that garage refuses the old pair after the apply.

Assumptions

• KV keeps older versions of a value, and an older version stays readable after a forced upload. When neither the entry nor the mount sets a limit of versions, KV keeps only a small default number of versions, and the secrets mount sets no limit. Thus the OpenBao node sets a large limit of versions on each key that a backup depends on, and makes the writes to such a key give the current version (check-and-set). The upload of generated secrets and a rollback of a key give it.
• The backups keep data for the retention periods of the application backups.

Known Deviations

• The lldap seed cannot change, also not by hand: lldap derives the OPAQUE data of each password from it, so a new seed makes each password of the live database invalid too.
• The DKIM key does not rotate automatically. Needs: A monthly switch on the relay node that adds a selector, waits until DNS has its record, then signs with it.
• The DNSSEC key does not rotate. Needs: A DS procedure: the DS record is at the registrar or the parent zone, which is part of the registrar work.
• The restic password of the bootstrap backups is fixed and public. Age protects only the unseal key and the state encryption key in these backups.
• A change of the age recipients of the bootstrap backups does not revoke a recipient that it removes. The backups from before the change hold the unseal key and the state encryption key encrypted to that recipient, and these keys do not rotate.
• A raft snapshot restore gives back the engine credentials of the time of the snapshot. The roles that rotated after the snapshot rotate again within seconds, and the consumers get the new values when their agents log in again, which can take about an hour.