Fediversity

Feature Specification: Deployment method tf-incus

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code of core/effects/tf/incus/ and core/setups/ at 14504f80d, PRs #527, #723, #1071, #1184, #1205, #1234, #1235, #1238 and #1393.

The tf-incus method deploys the nodes of one group into Incus instances on one hypervisor. One OpenTofu run creates the instances, gives them addresses, deploys NixOS into them and judges the result.

This file states only what is particular to tf-incus. The program grammar, the verbs and the state rules are in 004. The shared deploy body, its verdicts, the bootstrap wrapper and the key sets are in spec.md. The operator method tf-incus-operator.md uses the same instance and NetBox parts. The base image comes from image.md. The dev runner is in incus-bootstrap.md. The zone records of the provider come from the octodns-zone effect, not from this method.

User Scenarios & Testing (mandatory)

User Story 1 - Deploy a group into Incus containers (Priority: P1)

An operator runs the tf-incus program of a group against an Incus hypervisor. The run creates one instance per node, waits until each instance accepts commands, deploys the NixOS closure of the node into it and judges each node.

Why this priority: This is the method. Without it, no group reaches an Incus hypervisor.

Independent Test: A VM test of a full deployment onto a nested Incus hypervisor.

Acceptance Scenarios:

  1. Given a hypervisor with a managed bridge and a base image, When the operator applies the graph, Then each node exists as an instance whose name carries the namespace, and the operator can run a program from the closure of the node inside it.
  2. Given a deployed node, When a later apply replaces the instance, Then the deploy and both verdict passes run again for that node.
  3. Given a deployed group, When the operator reads the outputs, Then they hold the name and the addresses of each node.

User Story 2 - Boot containers off the hypervisor store (Priority: P2)

A node that does not need a writable store shares the store of the hypervisor read-only. It gets no image and no root disk. Its closure lives on the hypervisor under a profile per instance.

Why this priority: The shared-store mode saves a full closure copy per container on the hypervisor disk. A wrong plan in this mode destroys a live container, so the method guards it.

Independent Test: The shared-store parts of the VM test, and an integration test of the shared-store mode. A unit test of the per-node plan inputs and of the replace guard. A plan test with mocked providers of what the graph plans from those inputs.

Acceptance Scenarios:

  1. Given a shared-store node, When the graph creates it, Then it first plants the generation of the node in the host profile of the instance, and the instance boots from that profile.
  2. Given a shared-store node after a deploy, When the operator compares the host profile and the guest profile, Then both name the same generation, and the host profile keeps that generation alive in the host store.
  3. Given an instance that boots off the hypervisor store, When a plan asks for the image path for it, Then the plan fails and replaces nothing.
  4. Given a shared-store node that is a virtual machine, When the operator plans, Then the plan fails.
  5. Given a shared-store node, When the operator destroys the graph, Then the destroy removes the host profile with the credentials of the current run.

User Story 3 - Allocate public addresses in NetBox (Priority: P2)

A node that needs a public address gets one from NetBox. The graph records the node and its interface in NetBox and allocates or pins its addresses. A second interface carries that address.

Why this priority: Public nodes need stable addresses that the zone can publish. A provider that hosts its own NetBox finds it through its own state.

Independent Test: A VM test of the probe that gates the read of the own state. A unit test of which nodes ask for an address, of the pinned addresses, the NetBox coordinates and the uplink mode. A plan test with mocked providers. No test applies the NetBox part against a real NetBox.

Acceptance Scenarios:

  1. Given a fresh provider with no own state yet, When the operator applies, Then the graph reads an empty own state and continues without the NetBox and OpenBao addresses from it.
  2. Given a provider whose own state holds the address of the node that runs the netbox component, When no NetBox URL is given, Then the graph dials NetBox at that address.
  3. Given a node with a pinned address, When the graph applies, Then NetBox holds that address on the interface of the node, and the graph allocates no other address for it.

User Story 4 - Register TPM keys and judge the fleet (Priority: P2)

When the caller names a SPIRE node, the graph registers the TPM endorsement key of each other container with it. A strict verdict-only pass then judges every node after its secret chain completes.

Why this priority: A node whose units wait on SPIRE credentials cannot pass a verdict until its key is registered.

Independent Test: An integration test of SPIRE on Incus.

Acceptance Scenarios:

  1. Given a SPIRE node and container instances, When the graph applies, Then it registers the key of each node except the SPIRE node, after both nodes are deployed.
  2. Given a deployed node, When the verdict-only pass runs, Then it fails the apply when a wanted unit stays failed.

Edge Cases

  • A shared-store container has no bridge lease at creation, because the graph creates it stopped. Its address output comes from the lease of the daemon until the next apply refreshes the instance.
  • The graph looks up the base image by alias when no fingerprint is given, and only when some node needs an image.
  • The nat uplink adds no second interface. The bridged uplink requires a parent network of the bridge type, else the plan fails.
  • A filtered interface sends only from its hardware address and its own addresses. A guest that changes the hardware address loses that interface. A guest that sends from a different address loses those packets.
  • On the bridge interface without a static address, Incus reserves the current DHCP lease when it applies the filters. The node keeps that address until its interface goes away. The nodes of an operator deployment have a static address (tf-incus-operator.md).
  • Under nat, the NetBox address is an alias on the bridge interface. The filters drop traffic from that alias. No node uses it, because the nodes find each other by the DHCP names of their leases.
  • The filters drop IPv6 from temporary (privacy) addresses, so a node turns them off. When the managed bridge has no IPv6 subnet, the filters drop all IPv6 from the bridge interface, link-local too.
  • A bridged interface with no IPv6 address in NetBox sends no IPv6. Incus never filters the traffic to a guest.
  • An unreadable hypervisor makes the shared-store probe report nothing. The replace guard then passes, and the apply fails against the daemon.
  • The Incus remote can exist already from a concurrent deploy. The program accepts that and fails on any other error.
  • The NetBox token can come from the environment, from a file, or from OpenBao KV. The KV read is ephemeral, so the token stays out of the state.

Requirements (mandatory)

Functional Requirements

  • FR-001: The setup MUST mark the graph as one that holds its own state backend, and MUST give each node its application, network name and host.
  • FR-002: Before OpenTofu runs, the program MUST enroll its Incus client certificate, and MUST seed an empty own state when none exists.
  • FR-003: The graph MUST name each instance from the namespace and the network name of the node, and MUST fall back to the node key when the node has no network name.
  • FR-004: Every instance MUST have an interface on the managed bridge, MUST start with the hypervisor, and MUST allow nesting.
  • FR-005: Every container MUST map to its own block of host user and group ids, so that a process that escapes one container does not share ids with a different container.
  • FR-006: The bridge interface and every bridged public interface MUST filter by hardware address, IPv4 and IPv6. A bridged public interface MUST hold the NetBox address of each family, or no address of a family that NetBox does not hold.
  • FR-007: The graph MUST add the public interface only for public nodes under the bridged uplink.
  • FR-008: An image-path instance MUST be created running from the base image, and MUST wait for an IPv4 lease on the bridge.
  • FR-009: A shared-store instance MUST be a container, MUST be created stopped, and MUST mount the host store read-only and its host profile writable. A shared-store container MUST come from the stub image, which is an OCI image or an LXC image, and the plan MUST fail when the daemon does not have the stub image. A container MUST mount the host disks with no id shifting and MUST have no raw.lxc, because a restricted Incus project refuses both.
  • FR-010: The graph MUST give a node access to the Nix daemon of the hypervisor only when the node asks for it. That access MUST require the shared-store mode.
  • FR-011: The graph MUST plant the shared-store profile before it creates the instance, and MUST replace the profile only when the instance name changes.
  • FR-012: The plan MUST refuse to move an instance that the daemon reports as shared-store back to the image path.
  • FR-013: The graph MUST wait until each instance accepts commands before the deploy, and MUST deploy again when the instance is replaced.
  • FR-014: The per-node deploy MUST run the incus transport of spec.md in the lenient mode. The strict judgment comes from FR-016.
  • FR-015: When the caller names a SPIRE node and the instances are containers, the graph MUST register the TPM endorsement key of each other node after both are deployed.
  • FR-016: The graph MUST run a strict verdict-only pass per node after the deploy and the key registration, and MUST fail the apply on a failed wanted unit.
  • FR-017: The graph MUST read its own prior state only when a probe reports that the state is reachable.
  • FR-018: The setup MUST name the roster nodes that provide the secret store and NetBox, or none when the group runs no such node. The graph MUST take the OpenBao address and the NetBox URL from the own state of these nodes only, and MUST NOT find them by a node name.
  • FR-019: When NetBox is configured, the graph MUST record one virtual machine and interface per public node, named by its DNS label, and MUST allocate or pin its addresses there.
  • FR-020: The graph MUST dial OpenBao over HTTPS, and MUST skip TLS verification only when no CA is given.
  • FR-021: When the caller asks for it, the graph MUST publish each given deployer value to the deployer part of the provider KV.
  • FR-022: The graph MUST write no address records, and MUST publish no node names.
  • FR-023: The operator MUST be able to override at run time the hypervisor, the network, the instance type, the image, the NetBox selectors and custom fields, and the namespace.

Key Entities

  • Instance: One Incus instance per node. Its state key is the node key. Its Incus name carries the namespace.
  • Host profile: A profile per instance on the hypervisor. It is the boot profile and the only garbage collection root of the closure of a shared-store node.
  • Recreate detector: A value that changes when the instance is replaced. It makes the per-node deploy run again.
  • Own state: The state of this graph on its state backend. Its address outputs locate the openbao and netbox nodes of the provider.
  • NetBox objects: A virtual machine, an interface, one IPv4 and an optional IPv6 address per public node, in the cluster of the provider.
  • Deployer KV values: The admin sources, the NetBox and DNS tokens, the zone manager token and the registrar credentials, which the deployer programs read.

Run-time order

  1. Enroll the client certificate, then seed the own state.
  2. Probe the own state and read it, probe the shared-store mode, and look up the image.
  3. Record the NetBox objects and addresses of the public nodes.
  4. Plant the profile of each shared-store node.
  5. Create each instance, then wait until it accepts commands.
  6. Deploy each node.
  7. Register the key of each node.
  8. Run the verdict-only pass of each node.
  9. Read the bridge addresses and give the outputs.

The publication of the deployer values has no order relation with these steps.

Failure points

  • Evaluation: a value that does not match the type of the NetBox, uplink, SPIRE or deployer KV settings.
  • Plan: the two shared-store guards, and the parent network check of the bridged uplink.
  • Apply: an unreachable Incus daemon, an instance that does not accept commands, a failed deploy, a failed key registration, or a strict verdict that fails.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The VM test deploys an image-path node and a shared-store node, and runs a program from each.
  • SC-002: The host and guest generations of the shared-store node are equal, and the host profile keeps the generation alive.
  • SC-003: A shared-store node that does not ask for the host daemon has no access to the daemon and no access to the store database.
  • SC-004: The seed pass and the deploy pass of one node share one evaluation.
  • SC-005: A fresh provider falls through to DHCP-only addresses.
  • SC-006: A command in a node finds its programs when the own search path of the node resolves nothing.
  • SC-007: CD stages the hypervisor host key, the shared-store identity and the KV coordinates that this method reads.
  • SC-008: A node with a bridged public interface reaches the Incus host from its own addresses on both interfaces, and the filters drop the traffic that it sends from an address that it added itself.
  • SC-009: The plan inputs select the expected nodes, and every input that Nix renders is declared by the graph.
  • SC-010: The graph plans the shared store only for a container, the public interface only for a public node in the form of its uplink mode, and NetBox objects only for the nodes that need a public address.

Assumptions

  • The hypervisor runs Incus with a managed bridge that serves DHCP, a DNS resolver and a Nix substituter.
  • The base image exists on the hypervisor when some node uses the image path.
  • For shared-store nodes, the caller supplies an SSH identity into the hypervisor store and a Nix signing key.
  • The state backend is reachable when the caller configures one.
  • All deployments on one Incus host share the managed bridge. The firewall resource of a node of the hosting provider takes the whole subnet as an internal source, and the hypervisor accepts from the bridge only the ports that the nodes dial. The interface filters (FR-006) stop a node from sending as a node of a different deployment. The nodes of the hosting provider get DHCP leases from the provider range and have no ACL. The operator deployments are isolated from each other by an ACL per deployment (tf-incus-operator.md).

Known Deviations

None.