Fediversity

Feature Specification: Component incus

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code of core/components/hosting-provider/incus/ and the incus modules next to it at 093e68b6f.

User Scenarios & Testing (mandatory)

User Story 1 - Run the hypervisor for the guest nodes (Priority: P1)

A hosting provider enables incus on a host node. The host runs the Incus daemon with a bridge, a storage pool and a default profile. The deployment methods then create the other nodes of the group as Incus instances on this host. The bridge gives the guests DHCP, NAT, DNS for the apex, and a binary cache of the host store.

The host makes the bridge and the pool at each start of Incus, because services on the bridge address need them at boot. A reconciler declares all other Incus state and removes what the configuration does not declare.

Why this priority: the tf-incus deployment methods need a hypervisor. Without it, no guest node exists.

Independent Test: a VM test of the Incus host. A VM test of the bridge DNS answers.

Acceptance Scenarios:

  1. Given a node with the component, When it boots, Then Incus answers, and the bridge, the storage pool and the default profile exist.
  2. Given a UI domain, When the firewall loads, Then it admits the Incus API and HTTPS.
  3. Given a guest on the bridge, When it resolves the bridge DNS name or a name that the setup aliases on the bridge, Then the bridge DNS answers it.
  4. Given the reconciler applied once, When Incus restarts and the reconciler applies again, Then nothing changes.
  5. Given a guest that substitutes from the bridge cache, When it fetches a path, Then it checks the signature with the key that the default profile gives it.
  6. Given a signing key of the bridge cache that is rotationDays old, When the rotation timer runs, Then the cache signs with a new key, and a guest trusts the new key and the previous key until the new key is overlapDays old, and only the new key after that.
  7. Given a compromise of the signing key, When the hosting provider starts the revoke unit, Then the cache signs with a new key, and a guest trusts only the new key at once.

User Story 2 - Sign in to the Incus web UI (Priority: P2)

An administrator opens the UI domain of Incus. A reverse proxy terminates TLS and forwards to the daemon. The daemon accepts an OIDC login from the identity provider of the group.

Why this priority: the UI is the operator-visible face of the hypervisor. The overview page links to it.

Independent Test: the VM test of the Incus host checks that a client gets to the UI. Unit tests check the served UI port, the published UI name and the OIDC settings of the daemon. A VM test does the OIDC login through authelia and its lldap directory, with a scripted client in place of a browser. It checks that the API admits a member of the admin group and refuses a user outside it.

Acceptance Scenarios:

  1. Given the component is on, When a browser opens the UI domain, Then the proxy forwards the request to the daemon, with websocket upgrade.
  2. Given an OIDC provider answers the request of the component, When the node activates, Then the daemon has the issuer, the client and the scopes of that provider.
  3. Given a user of the directory of the identity provider, When the user logs in and accepts the consent, Then the API trusts the session as an OIDC user. Without the session cookies, the API does not trust the client.
  4. Given an OIDC user who is not in fediversity-admins, When the user gets a session and requests a project, Then the API refuses the request.

User Story 3 - Let guests share the host store (Priority: P2)

A deployer builds a guest closure once on the host. Guests in shared-store mode read the store of the host. A scoped SSH identity lets the deployer copy signed closures in and set a profile per instance.

Why this priority: shared-store mode saves disk space and build time on the host. It is the default nixStore mode of a component.

Independent Test: a VM test of shared-store mode, and unit tests of the shared-store wiring and of the store identity.

Acceptance Scenarios:

  1. Given store serving is on, When the store user logs in over SSH, Then it can only set or remove the profile of an instance, or talk to the Nix daemon.
  2. Given the minted public keys change, When the host sees the change, Then it trusts only the new SSH key and signing key, and stops the running Nix daemon of the store user.
  3. Given the host SSH key, When the host publishes it, Then OpenBao KV holds that key, so that the deployer can pin the host.

User Story 4 - Attest the host and publish deploy tokens (Priority: P3)

The host registers the EK of its own vTPM with SPIRE. It also mints Incus trust tokens for the deploy runners and publishes them to OpenBao KV.

Why this priority: these tasks let remote deployers and the SPIRE agents of guests trust the host. A host without SPIRE or runners works without them.

Independent Test: VM tests of the host attestation, of the evidence of the EK registration and of the token publication, and a unit test of the token names, the checks and the names and KV subpaths that the prune removes.

Acceptance Scenarios:

  1. Given the host has a SPIRE server and a host name, When it boots, Then it has a vTPM, and SPIRE registers the SPIFFE ID of the host.
  2. Given a deploy runner, When the token publication runs and a KV entry or trust name of that runner is missing, Then it mints the two tokens of the runner (for hosts and for images) and writes both to KV. It first revokes the older pending tokens of those two names.
  3. Given a trusted certificate or a pending token with the trust name of a runner that the list of deploy runners does not contain, When the token publication runs, Then it removes the certificate and revokes the token.
  4. Given a KV subpath of a runner that the list of deploy runners does not contain, When the token publication runs, Then it removes the subpath and all its entries.
  5. Given a runner whose certificate is trusted under an old name, When the runner gets a new name and the token publication runs, Then the publication removes the old certificate, and the next deploy of the runner enrolls a new certificate with the token under the new name, with no manual step.

Edge Cases

  • The KV probe write fails. The token publication fails and tries again after a short delay, with no limit.
  • A runner whose KV entries and trust names all exist gets no new token.
  • A trust name without a runner after the dash, such as the bootstrap names deployer-hosts and deployer-image, the CD certificate and the metrics certificate, stays.
  • An empty list of deploy runners disables the token publication, thus nothing removes a trust name or a KV subpath.
  • A KV entry of the token instance that is not a subpath, such as the KV probe, stays.
  • A runner is named by the node key. A change of its network label does not change its trust names or its KV path.
  • A failed removal does not stop the publication of the declared runners. The token publication fails after it.
  • The reconciler fails. The host has the bridge and the pool, but no network API, no default profile devices, no bridge DNS zone and no OIDC. The reconciler tries again after a short delay, with no limit, also when the first start of Incus failed, until its apply succeeds. The bootstrap configuration applies again at each start of Incus, so it holds only the bridge and the pool. Only the reconciler removes a key.
  • The Incus package is not the LTS release. Its database migration cannot go back to an older version.
  • The host has more than one TPM resource manager device. Only the first one starts TPM units, and the host waits a bounded time for it.
  • A TPM never stays the hardware random number source of the host.
  • The keys of store serving are in volatile storage, so the host mints them again at boot. The key pair from KV then replaces them, and the host applies the change.
  • Incus has no built-in fine-grained authorization. An authorization scriptlet of the daemon checks the OIDC clients: each member of fediversity-admins is an administrator, and the scriptlet refuses all other OIDC users. The identity provider also refuses the login of a user outside fediversity-admins. TLS clients do not go through the scriptlet.
  • nix-serve signs each narinfo when it serves it, thus a new signing key needs no new signature on the paths in the store. A node keeps the narinfo that it got in its disk cache, with the signature of the previous key, for 30 days by default. The overlap lets the node use it. After the overlap, or after a revoke, the node refuses it, and substitutes the path from the upstream caches.
  • The nix daemon of a node reads the keys only when it starts. A path unit on the node restarts the daemon when the fragment changes. An instance that does not report the change of the shared directory keeps the old keys in its daemon until the daemon restarts.

Requirements (mandatory)

Functional Requirements

Component

  • FR-001: The component MUST register as incus, with the Incus logo, the default subdomain incus, and a UI that the overview page links to. Its certificate audience MUST be browser.
  • FR-002: When enabled, the component MUST request a TLS certificate for its domain (ssl), a SPIFFE identity (spiffe), its NixOS module with store serving on (nixos-module), and an oidc client.
  • FR-003: The OIDC client MUST be public, MUST use JWT access tokens, MUST ask for offline access, and MUST allow redirects on the UI domain and on the API port of that domain. The daemon MUST take its issuer from the result of the oidc contract. The provider MUST put the groups claim in the access token.
  • FR-003a: The daemon MUST send the OIDC clients to an authorization scriptlet. The scriptlet MUST admit an OIDC user who has fediversity-admins in the groups claim, and MUST refuse all other OIDC requests, also when the claim is missing.
  • FR-004: When the issuer uses HTTPS and the private PKI CA is present, the daemon MUST trust the private CA, and MUST start after the host installs it.
  • FR-005: The component MUST provide instances. The setups MUST give the public uplink, the bridge firewall, the bridge aliases, the deploy runners, the SPIRE server and the public UI record to the node that provides instances, not to a node by name.

Incus state

  • FR-006: The bootstrap configuration MUST hold only the bridge (with DHCP, NAT and no IPv6) and a directory storage pool, so that services on the bridge address have them at boot, also when the reconciler fails.
  • FR-007: The reconciler MUST declare all other runtime state of Incus: the DNS of the bridge, the network and root disk devices of the default profile, the network API listener, the OIDC settings and their authorization scriptlet, the trusted metrics certificate, and one network ACL for each /28 slot of the prefix of the operator deployments, which drops the traffic between the slot and the rest of that prefix. It MUST remove what the configuration does not declare.
  • FR-008: The reconciler MUST adopt the bridge and the pool, and MUST declare each key that the bootstrap configuration sets with the same value. Otherwise the two change the key back and forth at each start of Incus.
  • FR-009: The reconciler MUST start after Incus and its bootstrap, and MUST stop and restart with them, but MUST NOT depend on their success. It MUST try again after a failure, with no limit. A hard dependency on a failed first start cancels the start of the reconciler, and then no retry applies.

Bridge

  • FR-010: The component MUST be the single source of the bridge address and subnet, and the setups MUST read them from it.
  • FR-011: The bridge MUST serve DNS for the apex, with an alias per name that the setup asks for and an address record for the UI domain.
  • FR-012: The host MUST use the bridge DNS first, and its DHCP client MUST ignore the guest and bridge interfaces.
  • FR-013: The host MUST serve its store as a binary cache on the bridge address. The cache MUST sign each entry with a key that the host keeps across reboots, and changes when the key is fediversity.incus.bridgeCacheKey.rotationDays old (90 days by default). Each key MUST have a name of its own. The host MUST have a unit that changes the key at once, without an overlap.
  • FR-014: The default profile MUST give each container the public key of the bridge cache, read-only and optional. After a scheduled change, the fragment MUST hold the previous key too, until the new key is fediversity.incus.bridgeCacheKey.overlapDays old. A node that substitutes from the bridge cache MUST use the keys in the fragment, MUST keep signature checks on, and MUST restart its nix daemon when the fragment changes.

UI and metrics

  • FR-015: The host MUST admit the Incus API port, and HTTPS when a UI domain is set.
  • FR-016: The reverse proxy of the UI domain MUST forward to the daemon with websockets and no body size limit. It MUST verify the daemon with the server certificate of that daemon as its only trust anchor, and MUST check the DNS name in that certificate. The proxy MUST start after Incus.
  • FR-017: The metrics endpoint MUST admit only the bridge subnet, and the proxy MUST present a client certificate that the host mints. The reconciler MUST trust that certificate for metrics. When the trust store has the certificate but the reconciler state does not, the reconciler MUST adopt it, so that the apply does not fail.

Store serving

  • FR-018: With store serving on, the host MUST mint an SSH key pair and a Nix signing key pair for the store user through generateFiles.
  • FR-019: An SSH login of the store user MUST only set or remove the profile of an instance, or talk to the Nix daemon. The store user MUST linger, MUST NOT be a trusted Nix user, and MUST have no forwarding, TTY or tunnel.
  • FR-020: The profile tool MUST validate the instance name and the store path before it sets or removes a profile.
  • FR-021: When the minted public keys change, the host MUST trust only the new keys and MUST stop the running Nix daemon of the store user.
  • FR-022: The host MUST publish its SSH host key to OpenBao KV.
  • FR-023: The component MUST define the store user and the key locations once. The deploy jobs of windmill MUST use the same values.

Attestation and deploy tokens

  • FR-024: When the SPIRE server and the host name are set, the host MUST run a vTPM and MUST register its EK with SPIRE, at boot and at a regular interval. A failure of the SPIRE agent of the host MUST start a registration.
  • FR-025: With deploy runners, the token publication MUST check that Incus answers and that a KV write succeeds, then mint and publish the missing tokens of each runner under a KV path of that runner. Each runner MUST be named by the node key, and the runner MUST read its token under the same name. It MUST skip a runner only when both KV entries and both trust names of that runner are present. The mint and the check MUST use one definition of the trust names and KV paths. The publication MUST start after the reconciler, because Incus issues a token only when it listens on the network. It MUST run again at a regular interval. Before a mint, it MUST revoke the pending tokens of the trust names that it mints. Each run MUST remove the trusted certificates and revoke the pending tokens with a trust name of the form of a runner (deployer-hosts-<runner>, deployer-image-<runner>) that is not a trust name of a declared runner, and MUST NOT remove another name. Each run MUST remove each KV subpath of the token instance that is not the subpath of a declared runner, after the KV probe, and MUST NOT remove another KV entry. When the project of the deployments is not default, each token MUST be restricted to that project, and each run MUST restrict each trusted certificate with a trust name of a declared runner to that project. The CD certificate stays unrestricted.

Key Entities

  • Bridge data: the bridge address and subnet, the DHCP range of the hosting provider, and the prefix of the operator deployments in that subnet.
  • Reconciler: the declarative apply of the runtime state of Incus (FR-007).
  • Deploy tokens: one Incus trust token for hosts and one for images per deploy runner, with their trust names and KV entries. The runner is named by the node key.
  • Store identity: the store user and its SSH and signing keys.
  • Host SPIFFE ID: the SPIFFE ID of the host, from its network name.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The VM test of the Incus host passes: the daemon, network, storage pool and profile devices exist, the bridge has a DNS domain, the metrics certificate is trusted, the bridge cache signs with the key that the default profile gives the containers, changes its key on schedule with an overlap and at once with a revoke, a reapply after a restart of Incus changes nothing, SSH, HTTPS and the Incus API are open, a client gets to the UI, and the proxy verifies the daemon and forwards the API and the metrics to it.
  • SC-002: The VM test of the bridge DNS and the unit tests of the hypervisor wiring and of apex resolution pass.
  • SC-003: The VM test of shared-store mode and its unit tests pass.
  • SC-004: The VM tests of the token publication, of the host attestation and of the EK registration evidence, and the unit test of the deploy tokens pass.
  • SC-005: The unit test of the OIDC settings and the VM test of the OIDC login pass.

Assumptions

  • The node uses OpenBao for its vars, which gives the OpenBao address, token, mount, prefix and CA to the publication tasks.
  • CD reads the published host key to pin the host.
  • The deploy-side part of the deployment method gives the API trust token and the TLS material.
  • The reverse proxy and its TLS certificate come from the ssl resource.

Known Deviations

None.