Fediversity

Feature Specification: Component terraform-backend

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code of core/components/hosting-provider/terraform-backend/ and the shared tf effects at 093e68b6f.

User Scenarios & Testing (mandatory)

User Story 1 - Keep the OpenTofu state of the group (Priority: P1)

One node of the group runs an HTTP state backend for OpenTofu behind an nginx front. Each tf deployment program keeps its state and its locks there (see 004). The backend encrypts the state at rest with a key that stays on the node.

Why this priority: without the backend, no steady-state deploy can run.

Independent Test: a unit test of the node alone, and the VM tests of the tf and tf-incus deployment methods.

Acceptance Scenarios:

  1. Given the node is up, When a deployer initializes OpenTofu against it over HTTPS with its credential, Then the backend keeps the state encrypted with the key of the node.

User Story 2 - Bootstrap before public DNS and TLS (Priority: P2)

On a fresh provider, a bootstrap migrate reaches the bootstrap port of the front directly over the Incus bridge, from an internal source, with no TLS.

Why this priority: the state of the graph that repairs the cluster must not depend on public DNS, a public CA, OpenBao or SPIRE.

Independent Test: VM tests of the bootstrap and of a migrate after a failed apply.

Acceptance Scenarios:

  1. Given OpenBao attestation is broken, When the node starts, Then it makes the state key on local disk and the backend starts.

User Story 3 - Recover a stale lock (Priority: P3)

A client that stops during an apply keeps its lock. An operator releases it with the force-unlock verb of the effects library or of the CD workflows.

Why this priority: only the holder releases a lock, so a stale lock stays until an operator acts.

Independent Test: VM tests that reclaim an own lock and check the freshness of the backend.

Acceptance Scenarios:

  1. Given a lock that no client holds, When the operator forces an unlock, Then the backend releases it.

User Story 4 - Accept a request for a state only from its deployer (Priority: P1)

Three kinds of deployer use the backend: the CD workflows (cd), each person on a workstation (workstation-<person>) and the api, which deploys each operator deployment (operator). Each deployer has its own credential, so the access log of the front shows which person sent a request. The front accepts a request for a state only from a deployer of that state. The ID of each state stays the same.

Why this priority: the backend itself accepts every request. Without the front, each internal source can read and replace each state, and a state holds the secrets of its graph.

Independent Test: a VM test that sends requests with each credential to the front, and unit tests of the node configuration.

Acceptance Scenarios:

  1. Given a request with no credential or with an incorrect password, When it gets to the front, Then the front answers 401.
  2. Given the operator credential, When it reads or writes the state of the hosting provider, Then the front answers 403. A read of the addresses document of the hosting provider is accepted.
  3. Given the cd or a workstation-<person> credential, When it sends a request for the state of an operator deployment, Then the front answers 403.
  4. Given the cd credential, When it writes a state through the front, Then the backend stores the state under the ID that it had before the front.

Edge Cases

  • Each lock is a file in the state directory, so a lock stays through a restart of the backend. A switch restarts the backend, also during an apply. While the backend is stopped, a request of the apply gets no answer, and OpenTofu sends it again (retry_max). A restart that takes longer than these tries, for example the copy of the states before an upgrade, makes that request fail, and the apply stops with its lock held. The operator then releases the lock (User Story 3).
  • A restore of a backup or of the upgrade copy puts back the locks of that time. A lock that no client holds then stays until an operator releases it.
  • A new state key makes the existing states unreadable, so the node makes the key only when it is absent.
  • A restore on a node with a different key needs the key, which an administrator decrypts from the backup and adds to the archive. Without it, the restore fails before it stops the backend.
  • A state that changes during a backup makes the backup copy again, a few times at most, and then fail.
  • The backup gets its S3 key from OpenBao. When OpenBao is broken, the backup fails but the backend continues to serve.
  • A deploy that adds the front to a backend that has no front restarts the backend on loopback. The switch stops the backend before it starts nginx, so nginx can bind the bootstrap port. The administrator sets the deployer passwords only after that deploy (see notes/cd-credentials.md). Until the CD deploy publishes the operator password, the front refuses the operator deploys.
  • The deploy of the hosting provider writes its addresses document only on an http apply. After a new deploy of the hosting provider, until its first http apply writes the document, an operator deploy fails on a 404, because a terraform_remote_state read of a missing state is fatal. The operator deploy succeeds when it runs again after that apply.
  • The addresses document is written again only when an address changes. A destroy of the hosting provider does not remove it. To write it again with the same addresses, replace terraform_data.published_addresses[0].

Requirements (mandatory)

Functional Requirements

  • FR-001: The component MUST use the bootstrap certificate audience, so that its certificate never comes from a public CA.
  • FR-002: The component MUST serve the backend over HTTPS on the name that the group gives to the state backend node, and MUST request a certificate and a SPIFFE identity (see ssl, spiffe).
  • FR-003: The bootstrap port of the front MUST be open on the node only to the internal sources, and MUST NOT be published. The backend MUST bind loopback only.
  • FR-004: The backend MUST keep each lock in a file in its state directory, so that a lock stays through a restart. A switch of the node that changes the backend MUST restart it with no manual step.
  • FR-005: The node MUST make a random state key, readable only by root, through the local vars provider of generateFiles, not through OpenBao. Other generated files MUST go to the default provider.
  • FR-006: The backend MUST start after the local vars render, MUST NOT wait for an OpenBao agent, and MUST accept a forced unlock.
  • FR-007: The component MUST stream a backup as root to a bucket of garage. Each backup MUST hold a consistent copy of the states, the state key encrypted to the age recipients of the bootstrap backup, and a hash of the key.
  • FR-008: When the archive holds no plaintext key and the hash differs from the key of the node, the restore MUST fail before it stops the backend and MUST tell the administrator to add the key. Else it MUST stop the backend, replace the states, install the key when present, fix the owner, and start the backend.
  • FR-009: The node MUST write to the KV store of OpenBao only the instance that holds the operator password (FR-010). Its policy MUST grant no other KV write, also not on its per-host stem (openbao FR-024).
  • FR-010: The front MUST authenticate each request for a state with HTTP basic authentication against bcrypt hashes. For cd and workstation-<person>, the node MUST hold only the hashes that the deployment gives. The node MUST make the operator password through generateFiles, keep it in the KV store of OpenBao at terraform-backend-operator/password, and add its hash to the hashes of the front when the node starts and each time the file of the password changes. A hash for operator from the deployment MUST have no effect. The front MUST let cd and workstation-<person> in when OpenBao is not available. The front MUST answer 401 to operator while the node has no password. The users MUST be only cd, operator and workstation-<person>, where <person> has one or more of the characters [a-z0-9].
  • FR-011: The front MUST answer 403 before it checks the password when the user has no permission for the method and the state path. operator MUST have each method on project1/example-<uuid> and project1/example-<uuid>-<name>, and only GET on project1/example-hosting-provider-addresses. operator MUST NOT have a method on project1/example-hosting-provider, because that state holds the secrets of the hosting provider. cd and each workstation-<person> MUST have each method on each other state path of two names. A workstation-<person> MUST have the same permissions as cd, so that a person can do each step of the CD workflows by hand, for example when CD is not available.
  • FR-012: The front MUST send basic:fake-secret to the backend in place of the credential of the deployer, so that the ID of each state stays the same.
  • FR-013: The evaluation of the node MUST fail when the deployment gives no deployer hashes (TF_BACKEND_HTPASSWD).
  • FR-014: The CD deploy MUST authenticate as cd with TF_HTTP_PASSWORD, and the deploy runners (the api and Windmill nodes) MUST read the operator password from the KV key terraform-backend-operator/password (FR-010). Each request of a graph for a state MUST send the credential of the deployer from the deploy runtime (TF_HTTP_USERNAME, TF_HTTP_PASSWORD). This includes the read of a state with terraform_remote_state, the seed of that state and the write of the addresses document, which do not take TF_HTTP_* from OpenTofu.
  • FR-015: The http apply of the hosting provider MUST write the addresses document <its state address>-addresses: a state of version 4 that holds only the outputs ipv4, bridge_ipv4, ipv6, operator_limits_acl and spire_generation, with the values of the outputs of the hosting provider, and no resources (core/effects/tf/incus/hosts/published-addresses.tf). The operator graph MUST read only this document of the hosting provider.

Key Entities

  • State backend: the OpenTofu HTTP state backend, on loopback.
  • Front: nginx on the TLS port and on the bootstrap port, which checks the deployer of each request.
  • Deployer credential: a user and a password for each deployer. The node holds only a bcrypt hash of each password.
  • State key: encrypts the states at rest, local to the node.
  • Lock file: one file for each locked state, in /var/lib/terraform-backend/locks, with the record of the holder.
  • Addresses document: a state next to the state of the hosting provider, with only the addresses of its nodes. It is the only state of the hosting provider that operator can read.
  • Offsite backup: the states and the encrypted state key.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The VM tests of the tf and tf-incus deployment methods pass with state on this backend, and the bootstrap and migrate tests pass over the direct lane.
  • SC-002: The VM tests that reclaim a lock, check freshness, refuse a lost state and reach the state from the deployer pass.
  • SC-003: A VM test passes in which a state written with the default client credentials has the ID of the existing live states.
  • SC-004: The VM test of the bootstrap backup passes, including a restore with a different state key.
  • SC-005: A VM test passes in which the front refuses each request that is not from a deployer of the state, and in which a state written through the front has the ID of the existing live states. In the same test, the writer of the addresses document writes it through the front, and a terraform_remote_state read as operator gets its outputs.

Assumptions

  • The node has the fixed name of the state backend, and the zone publishes a served name equal to that label.
  • The firewall resource limits the backend port to the internal sources.
  • The service module comes from nixpkgs pull request 554986 until it is in the nixpkgs pin.
  • The file lock backend is a patch on the backend package (core/effects/tf/common/terraform-backend-lock-file.patch). The upstream lock backends are local, redis and postgres. local keeps the locks in memory, so a restart releases them. redis and postgres need a database service: on a different node, its credentials come from OpenBao, so the state backend depends on OpenBao; on this node, it is one more service to operate. The redis backend also holds one lock for all states, with an expiry of 12 hours. A lock file in /run does not stay through a reboot. A lock file on the node is node-local state, but a lock is only valid for the time of one apply, and a stale lock has the force-unlock path (User Story 3). The lock files are on the persistent volume of the states, so a replace of the node keeps them.

Known Deviations

  • The backend has no OpenTelemetry support.
  • The basic authentication of the backend itself accepts every request, and the password only selects the state ID, a hash of the password, the project and the name. The front checks the deployer and sends one fixed password to the backend. A process on the node can send a request to the backend on loopback without a credential. A forced unlock adds no access, because a lock answers with the record of the holder, and an unlock with that record releases it. A change of the password that the front sends, or of the auth method, changes each ID and needs a migration.
  • The backup uses a fixed restic password that is public in the source, because the restore comes before the secret store. Its encryption gives no protection. The state key encrypts the states, and age encrypts the key.