Fediversity

Feature Specification: Component spire

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code of core/components/hosting-provider/spire/ and core/resources/spiffe/ at 093e68b6f.

User Scenarios & Testing (mandatory)

User Story 1 - Give each host a SPIFFE identity from its TPM (Priority: P1)

Nodes of a deployment need a workload identity to log in to OpenBao. The spire component runs the SPIRE server of the trust domain fediversity. The server attests agents by the endorsement key (EK) of their TPM, against an allow-list of EK hashes. For each known host, the server allow-lists the EK hash and registers one SPIFFE ID for the host, whose parent is the agent of that EK. The agents on the consumer nodes come from the spiffe resource. The component provides spiffe, so the agents of the group attest to this node.

The server learns an EK hash in two ways: the configuration states it, or a deploy hands it to the server when it provisions the host.

Why this priority: without an entry, the agent gets no SVID, and each OpenBao certificate login from that node fails (see openbao).

Independent Test: a VM test of spire with openbao, and a VM test that attests a node on the hypervisor through its vTPM (see incus). An integration test registers one name against a stub server.

Acceptance Scenarios:

  1. Given the configuration states the EK hash of host-a, When the node boots, Then the server has an entry for host-a.
  2. Given a deploy that hands the server the EK hash of a host, When the registration runs, Then the server accepts that EK and has an entry for the host, whose parent is the agent of that EK and which admits only the root user on that host.
  3. Given a host whose EK hash changes, When the registration runs, Then the entry points to the new agent, and the server does not accept the old EK.
  4. Given two names with the same EK hash, When the registration runs, Then it keeps the newest name and retires the others.
  5. Given a server that cannot read the entries of a name, When the registration runs, Then it shows the error, creates no entry for that name, registers the other names, and fails at the end.
  6. Given a host whose EK hash changes and a server that cannot delete the stale entry, When the registration runs, Then it shows the error, creates no second entry, keeps the old EK hash, and fails at the end.

User Story 2 - Retire the identities of removed hosts (Priority: P2)

A host that leaves the deployment must lose its identity. Otherwise anything that holds its EK keeps the identity. The registration retires each known name that is on neither the host roster nor the other-deployment roster. A deployment that removes its own instances retires their names when it destroys them.

Why this priority: a stale identity is a security gap, but it does not stop a live deployment.

Independent Test: the VM test on the hypervisor retires the old name of a renamed node and a node that is not on the host roster, and a retire that fails does not stop the other retires. An integration test retires one name against a stub server, and checks the list of objects that a destroy leaves when its retire fails. Unit tests check the rosters that the setups render.

Acceptance Scenarios:

  1. Given a non-empty host roster, When a known name is on neither roster, Then the registration retires it.
  2. Given an other-deployment roster with app, When the server knows abc123-app, Then the registration keeps it.
  3. Given both rosters are empty, When the registration runs, Then it retires no name.
  4. Given two names with the same EK hash, When one name is retired, Then the server still accepts that EK for the other name.
  5. Given a name with no entry, When an administrator retires it, Then the retire succeeds and deletes no entry.
  6. Given a server that cannot read or delete the entry, When a name is retired, Then the retire shows the error, fails, and keeps the EK hash of the name.
  7. Given one name that the registration cannot retire, When the registration runs, Then it retires the other names, does not register that name, and fails at the end.
  8. Given a destroy whose retire of a name fails, When the destroy ends, Then the host is in the list of objects that the destroy left.

User Story 3 - Publish the trust bundle (Priority: P3)

Agents and other consumers need the CA bundle of the server. The server node writes its bundle to disk at a regular interval. The deploy reads it from the server node and gives it to each agent, and between deploys the hypervisor host does the same for its containers (see the spiffe resource). The component does not serve the bundle on the network. With an upstream root (User Story 6), the bundle holds the roots, and each agent also has the roots in its config.

Why this priority: an agent needs the bundle to attest, and after that it keeps the bundle current over its own connection to the server.

Independent Test: a VM test of a cold start, in which a node that deploys before its SPIRE server exists still finishes activation.

Acceptance Scenarios:

  1. Given a running server, When the bundle export runs, Then the server node has the current bundle on disk.
  2. Given a host that the deploy registers, When the deploy registers its EK, Then it reads the bundle on the server node and writes it to the bundle file of the agent on that host.

User Story 4 - Recover after the server loses its state (Priority: P2)

An operator recovers the fleet after the server loses its state: its datastore, its key, or the registration data. There is no offsite backup to restore (see Assumptions). The server starts with a new key, a new CA and an empty datastore. Without an upstream root (User Story 6), each agent and OpenBao trust only the old CA.

Recovery steps, on the server node unless a step says otherwise:

  1. Start the server, or redeploy its node. The node makes its state directories again, and again knows the EK hashes that the configuration states.
  2. Run the bundle export at once, so that the bundle on disk holds the new CA now and not after the next periodic export.
  3. Run the registration again when the EK hashes that deploys handed over are still there but the datastore is not. A deploy does not start the registration for a hash that did not change, so without this step the entries stay absent until the next change of a hash.
  4. Apply each deployment that registers hosts with this server, including each deployment that owns names of the other-deployment roster. The apply writes the new bundle to the bundle file of each agent. The hypervisor host does this itself within a few minutes. As an alternative to an apply, copy the bundle to the bundle file of each agent, world-readable, as an atomic replace.

With an upstream root, steps 2 and 4 are not necessary: the new CA chains to the root, which each agent and OpenBao trust already. A running agent fails its next sync, because the server does not know it, attests again, and gets an SVID from the new CA. A certificate login succeeds when the agent has that SVID.

Without an upstream root, no step on the agent nodes is necessary after the bundle arrives. A running agent fails its next sync with the server (an unknown authority) and stops, because it never bootstraps again by itself. systemd starts it again, the start removes the cached SVID that does not chain to the new bundle, and the agent attests again. Until the bundle arrives, the agent stops and starts in a loop.

OpenBao follows without a step: when the bundle of its own agent changes, it waits for that agent to attest to the new CA (see openbao), installs the bundle and writes the certificate login roles again. The listener needs no reload. Until then, each certificate login fails, and each node keeps its current token until that token expires.

Why this priority: until the recovery is complete, no node can log in to OpenBao, so no node gets a new secret or a new token.

Independent Test: a subtest of the VM test of spire with openbao deletes the state of the server, does steps 1 to 4, and makes sure that the running agent gets an SVID from the new CA with no step on its node, and that a certificate login succeeds after OpenBao has the new bundle. A separate VM test checks the wait of the OpenBao bundle refresh. The VM test of User Story 6 does the recovery with an upstream root, without steps 2 and 4.

Acceptance Scenarios:

  1. Given a server with a new CA, and a running agent that has the new bundle in its bundle file, When the agent syncs with the server, Then it stops, starts again, and holds an SVID that chains to the new bundle, with no step on its node.
  2. Given OpenBao with the old bundle, When a node logs in with an SVID from the new CA, Then the login fails and the node keeps its current token.
  3. Given OpenBao with the new bundle and new login roles, When a node logs in, Then it gets a new token, and its credential daemon reads secrets with it.

User Story 5 - Give each pod of a cluster a SPIFFE identity from its service account token (Priority: P2)

A pod of k8s-hosts has no TPM. Its agent attests with the k8s_psat plugin: it shows a projected service account token, and the server asks the kubernetes API of the cluster to review the token. A deployment of k8s gives the server a kubeconfig for its cluster, and for each node a drop-file that holds k8s_psat:<cluster>:<namespace>:<service account> in place of an EK hash.

Why this priority: a node in a cluster needs the same identity as a node on incus to log in to OpenBao.

Independent Test: the VM test deployment-k8s-hosts deploys a node to k3s, and makes sure that its agent has the SVID of its host entry and that the server lists the agent with k8s_psat.

Acceptance Scenarios:

  1. Given a kubeconfig of a cluster and a drop-file of a service account in that cluster, When the registration runs, Then the k8s_psat plugin accepts the tokens of that service account, and the host entry of the name has the alias of that service account as its parent.
  2. Given a cluster whose kubeconfig is removed, When the registration runs, Then the plugin no longer accepts tokens of that cluster.
  3. Given a change of the clusters, When the registration runs, Then the server reads the plugin configuration again, without a restart.

User Story 6 - Sign each CA with the long-lived root of the deployment (Priority: P2)

Without an upstream root, the server signs its own CAs, and each CA lives for about one day. Each agent and OpenBao trust only the bundle that the deploy gives them, thus after a loss of the server state each of them needs a new bundle (User Story 4). With upstreamRootCertificate, the UpstreamAuthority "disk" plugin signs each CA of the server with the long-lived root of the deployment. Each agent has the roots in its config at eval time (rootCertificate of the spiffe resource), and trusts them and the delivered bundle. The bundle of the server holds the roots, thus OpenBao trusts them through the bundle of its own agent. A new CA, after a CA rotation or after a loss of the server state, chains to a root that each node trusts already.

Why this priority: it removes the bundle delivery from the recovery of User Story 4. A deployment without a root continues to operate as before.

Independent Test: the VM test integrations-spire-upstream-root boots the nodes of the VM test of spire with openbao with no root. It pushes the key of the root, switches the server to the root, rotates to a second root, and deletes the state of the server. After each step, it makes sure that the agent has an SVID that chains to the expected root and that a certificate login succeeds.

Acceptance Scenarios:

  1. Given a key whose certificate is in no root of the deployment, When the deploy pushes it, Then the push fails and writes no key on the server node.
  2. Given a server node with no key, When the deploy pushes the key, Then the node has the key, which only root can read, and the server does not restart.
  3. Given a server node with the key, When the deploy pushes the same key, Then nothing changes.
  4. Given a server that signs its own CAs, When its node switches to a config with the root, Then the server restarts, the agents keep their SVIDs, the next CA chains to the root, and the bundle keeps the old CA.
  5. Given a new key whose certificate is not in the config of the server node, When the deploy pushes it, Then the server does not start with it, the push puts the old key back, and the server starts again.
  6. Given a new key whose certificate is in the config of the server node, When the deploy pushes it, Then the server restarts and signs its next CA with the new key.
  7. Given a config with the root and no key on the server node, When the server starts, Then it does not start.
  8. Given a server with the root that loses its state, When the server starts again and the registration runs, Then the agent gets an SVID from the new CA, and a certificate login gives a new token, with no new bundle for the agent or for OpenBao.

The upstream root

Custody. An operator mints the root once for each deployment, offline: a self-signed P-256 certificate and key (notes/cd-credentials.md). The certificate is the Forgejo Actions variable SPIRE_ROOT_CERT, and the key is the secret SPIRE_ROOT_KEY. The key is in these locations:

  • the secret store of the forge;
  • the runner of the deploy job of the hosting provider (cd-hosting-provider), while the job runs, as a file that cd-env.sh stages;
  • the server node, at /var/lib/spire-upstream-authority/root.key, mode 0600, owner root, outside the state of the server. spire-server reads it as a systemd credential.

No backup holds the key, because the server node has no offsite backup.

Lifetime. The mint gives the root a lifetime of 10 years. Each CA has the lifetime of the config of the server (24 hours), and does not outlive the root. Rotate the root before it expires.

Risk compared with a server that signs its own CAs. An attacker who has root on the server node can sign an SVID for each host in both cases, while the attacker has that access. After the attacker loses that access:

  • without a root, a copy of the CA key is valid until the CA expires, after about one day;
  • with a root, a copy of the root key is valid until each node and OpenBao no longer trust the root: until a rotation removes it, or for the lifetime of the root.

Thus after a compromise of the server node, of the forge secret, or of a runner during a job of the hosting provider, do a rotation of the root, with no wait between its steps. A party that has the forge secret or the runner also has INCUS_CLIENT_KEY, which reaches the server node and its CA key. The root adds only the time after that access stops.

First rollout on a deployment that has no root:

  1. Mint the root. Set SPIRE_ROOT_CERT and SPIRE_ROOT_KEY for the deployment.
  2. Apply the tf-incus deploy of the hosting provider. The push writes the key before any node deploys, and the server does not restart, because its config has no root yet. Then the deploy of the server node switches it to the root, which restarts the server. The server keeps its current CA active, thus each agent keeps its SVID. The agents of this apply get the roots in their config.
  3. Apply each other deployment that registers hosts with this server, so that its agents have the roots. Until then, these agents trust the delivered bundle, which each apply writes as before.
  4. Wait for the next CA rotation, or rotate the CA at once with spire-server localauthority x509 prepare and activate. Each CA from then on chains to the root.

Rotation of the root, with an overlap in which each node trusts the two roots:

  1. Mint a new root. Add its certificate to SPIRE_ROOT_CERT after the old certificate, keep SPIRE_ROOT_KEY, and apply each deployment. Each agent, the bundle of the server and OpenBao then trust the two roots. Wait until each node that was offline has had an apply.
  2. Set SPIRE_ROOT_KEY to the new key and apply the hosting provider. The push restarts the server, which signs its next CA with the new root. Wait one CA lifetime, or rotate the CA at once, so that no SVID chains to the old root.
  3. Remove the old certificate from SPIRE_ROOT_CERT and apply each deployment. Then remove the old root from the bundle of the server with spire-server upstreamauthority revoke -subjectKeyID <the subject key ID of the old root>.

When step 2 occurs before step 1, the server does not start with the new key, and the push puts the old key back (acceptance scenario 5).

Removal. When SPIRE_ROOT_CERT is unset again, the server signs its own CAs, and each agent trusts only the delivered bundle. The bundle of the server keeps the roots until a revoke. The deploy does not remove the key from the server node. A recovery after a loss of the server state then needs steps 2 and 4 of User Story 4 again.

Edge Cases

  • The server is not up yet. The registration waits until the server is healthy. When it fails, it tries again after a short delay, with no limit.
  • The server fails. The registration does not stop with it.
  • A retire runs twice. The second run changes nothing.
  • A retire or a registration of one name fails. It stops before it changes the state that tells the next run what to do, so a re-run finds the same state. The registration fails and tries again.
  • A name with an operator namespace prefix is gone but matches the other-deployment roster. The name stays until its owning deployment retires it.
  • The config has an upstream root, and the key on the server node is absent, or no certificate of the roots has its public key. The server does not start.

Requirements (mandatory)

Functional Requirements

  • FR-001: The component MUST register as spire and run the SPIRE server of the trust domain fediversity, with a local datastore and a key on disk.
  • FR-002: The server MUST attest agents by the TPM EK, and MUST accept only an EK whose hash is on its allow-list. The server MUST also attest agents with k8s_psat, and MUST accept only a token of a service account that a drop-file names, of a cluster that has a kubeconfig on the node (FR-017). The server MUST give each pod its own agent ID, from the UID of the pod, because a node of k8s-hosts can have more than one pod on one kubernetes node, and a second agent with one ID makes the first agent fail.
  • FR-003: The EK hashes that deploys hand over MUST be private to the server.
  • FR-004: The node MUST know at boot each EK hash that the configuration states.
  • FR-005: The registration MUST, in order, wait for server health, retire the names that are on no roster, retire the older names of a duplicate EK, and reconcile the entries. It MUST try again after a failure, with no limit.
  • FR-006: Each entry MUST be the SPIFFE ID of one host, with the agent of the EK of that host as its parent, and MUST admit only the root user on that host.
  • FR-007: The retire by roster MUST match a host roster name exactly, and an other-deployment roster name either exactly or as the suffix after a namespace prefix. It MUST be off when both rosters are empty.
  • FR-008: To retire a name MUST delete its entry and its EK hash, and MUST remove the EK from the allow-list only when no other name has that EK. A retire MUST be idempotent.
  • FR-009: An administrator on the server node MUST be able to retire a name by hand.
  • FR-010: The server node MUST export its bundle to disk at a regular interval.
  • FR-011: The component MUST NOT serve the bundle on the network. The deploy reads it over its own access to the server node.
  • FR-012: The component MUST provide spiffe, so the setups route the spiffe server address of the group to this node.
  • FR-013: A retire MUST delete each entry of the name. A retire of a name with no entry MUST succeed and change nothing. When the server cannot read or delete the entries, the retire MUST fail and keep the EK hash and the allow-list.
  • FR-014: The registration of a name MUST create the entry when the name has none, MUST point it to the new agent when the EK hash changes, and MUST update it when its DNS name does not match the namespace. When the server cannot read the entries of the name, the registration MUST fail and write no entry. A re-point MUST delete each stale entry before it creates the new one. When a delete fails, the registration MUST fail, create no entry and keep the old EK hash.
  • FR-015: A failed retire or registration of one name MUST NOT stop the others. The registration MUST NOT register a name whose retire failed, and MUST fail at the end.
  • FR-016: The node MUST record the release series of SPIRE that the datastore belongs to. Before the server starts on a newer release series, the node MUST copy the datastore, without the CA key, while the server is stopped, and MUST keep only the copy of the last upgrade. When there is no room for the copy, the server MUST stay stopped. The node MUST NOT refuse an older or a skipped release series: the server itself accepts a datastore of the release series before or after its own, and refuses a datastore of a release series farther away without a change to it.
  • FR-017: For a drop-file that holds k8s_psat:<cluster>:<namespace>:<service account>, each a DNS label, the registration MUST put the service account in the allow-list of the cluster in the configuration of the k8s_psat plugin, when k8s-clusters/<cluster> holds a kubeconfig. It MUST register the host entry with the alias spiffe://fediversity/k8s/<cluster>/<namespace>/<service account> as its parent, and MUST take the namespace of the entry from the drop-file. It MUST write the configuration only when it changes, and MUST then make the server read it again. Only root and the group spire-k8s, which the server has, MUST be able to read a kubeconfig.
  • FR-018: With upstreamRootCertificate, the server MUST sign each CA with the root whose private key is on the server node, and its bundle MUST hold each root of the config. The server MUST NOT start when the key is absent, or when no root of the config has the public key of the key. Without upstreamRootCertificate, the server MUST sign its own CAs.
  • FR-019: The deploy of the hosting provider MUST write the private key of the root to the server node before it deploys a node, over the access that it uses for the EK hashes. It MUST NOT print the key. It MUST stop before it writes a key whose certificate is in no root of the deployment. It MUST write the key only when it changes, with mode 0600 and owner root. When it replaces the key, it MUST restart a running server, and when that restart fails, it MUST put the old key back and start the server again.
  • FR-020: Each agent MUST trust the roots of its config and the delivered bundle.

Key Entities

  • EK hash: the hash of the TPM endorsement key of one host, which a deploy hands to the server under the name of the host.
  • Allow-list: the EK hashes that the TPM attestor accepts.
  • k8s_psat cluster: a cluster whose service account tokens the server reviews, with the kubeconfig of a deployment of k8s-hosts. Its name holds the UID of the namespace kube-system of the cluster and the namespace of that deployment.
  • Host entry: the SPIFFE ID of one host.
  • Host roster: the host names that this deployment provisions.
  • Other-deployment roster: name stems that other deployments register under a namespace prefix.
  • Trust bundle: the current CA of the server and the prepared next CA. With an upstream root, also the roots of the deployment.
  • Upstream root: the long-lived CA certificate and key of a deployment, which sign each CA of the server. The config of each node holds the certificate.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The VM test of spire with openbao and the VM test on the hypervisor pass.
  • SC-002: The VM test of a cold start passes.
  • SC-003: The unit tests of the rendered rosters pass.
  • SC-004: The integration test of the retire of one name passes.
  • SC-005: The VM test integrations-spire-upstream-root passes.

Assumptions

  • The deploy hands the server the EK hash of each host that it provisions, at deploy time.
  • The setups render the host roster and the other-deployment roster from the full group roster (see setups and wiring).
  • Consumer nodes run a SPIRE agent from the spiffe resource, and have a TPM 2.0 or run as a pod of k8s-hosts.
  • The server node reaches the API server of each k8s_psat cluster.
  • OpenBao trusts the bundle of this server for certificate logins. It gets the bundle from its own agent (see openbao).
  • The datastore is SQLite while the hosting provider has one SPIRE server node. SQLite needs no database server, and the deploy makes the datastore again (see the next item). A second server node needs a datastore that the server nodes share, such as PostgreSQL.
  • The retire by roster keeps each name that matches an other-deployment roster name after a namespace prefix, also when no deployment holds that name. The server cannot see which deployment namespaces are live: the namespace is a deploy-time value, and the server node has no agent, so it cannot read the deployments in OpenBao. Thus the owning deployment retires its names on destroy, and a retire that fails is reported as left behind, for an operator to retire by hand. Until then, a stale name gives no access: its EK was in the vTPM of the destroyed instance, and the OpenBao policies of a namespaced host are templates on its namespace, whose KV the destroy deletes.
  • The server removes the entry of an attested agent one week after the SVID of the agent expired, if the agent can attest again. Thus the old pods of k8s-hosts do not fill the datastore.
  • The server has no offsite backup. Its state is the datastore and the CA key, and the deploy makes all of the datastore again. A copy of the CA key can sign an SVID for each host, and thus get each secret that a host can read in OpenBao. All restic repositories use the same password, thus a backup would give this key to each node that can read a backup bucket. The CA has a lifetime of 24 hours, thus a daily copy is near its end or expired when it is restored. The server node has no SPIRE agent, thus it cannot get the S3 key of a bucket from OpenBao KV. The copy of the datastore before an upgrade stays on the server node, as the datastore. The key of the upstream root is outside this state, and the deploy writes it again from the forge secret.
  • The upstream root signs each CA directly. There is no intermediate CA between them, thus a compromise of the server node needs a rotation of the root.