Fediversity

Feature Specification: Effect tf-incus-image

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code of core/effects/tf/incus/image/ and core/setups/ at 14504f80d, issue #2131.

The image effect builds the base NixOS image that the Incus deploy (tf-incus.md) starts each container from, and uploads it to the Incus server under an alias. The shared behavior of a standalone effect is in spec.md. The verbs and the state rules are in 004.

User Scenarios & Testing (mandatory)

User Story 1 - Upload the base image of a group (Priority: P1)

An operator runs the image effect for a group. The effect builds a container image, and uploads it under an alias. By default the image holds no node configuration. The Incus deploy then starts each container with a store of its own from that image, and sends the configuration of the node on the first deploy.

Why this priority: The Incus deploy needs an image on the server before it can create a container.

Independent Test: A VM test uploads the image through dev-tf-incus-image, which is this effect for the dev group, and starts containers from it.

Acceptance Scenarios:

  1. Given an Incus server and a client certificate, When the operator runs the effect, Then the server holds an image under the default alias, and the outputs give its fingerprint and alias.
  2. Given the dev group, When the catalog lists the effects, Then it does not offer the generated image effect.

User Story 2 - Give each container its own hostname (Priority: P1)

The image holds no node configuration, or the configuration of one node. Each container that starts from it gets its own hostname from Incus.

Why this priority: When the image fixes the hostname file, every container keeps the name in that file for its life, and the journal records a wrong hostname.

Independent Test: A unit test evaluates the image configuration as the effect does. It checks that the default image and an image from a node have no hostname file, that the image from a node keeps the node name in its configuration, and that the node configuration without the image changes has the file. No test starts a container from the image.

Acceptance Scenarios:

  1. Given an image made from node a, When node b starts from it, Then b does not report the hostname a.

User Story 3 - Choose the alias and the source node (Priority: P2)

An operator can set the alias and the source node. The effect then uploads under that alias, or puts the configuration of that node in the image.

Why this priority: Two groups on one Incus server need two aliases. The default image changes only when nixpkgs changes, thus CD uploads it seldom. An image from a node holds the closure of that node, thus a new container of that node needs less on its first deploy, but the image changes with each change to that node.

Independent Test: Unit tests evaluate the image with no source node and with a source node. No test sets the alias.

Acceptance Scenarios:

  1. Given an alias, When the effect applies, Then the image has that alias.
  2. Given no source node, When the effect evaluates, Then the image holds no node configuration.
  3. Given a source node that is not a node of the group, When the effect evaluates, Then the evaluation fails.
  4. Given a source node that shares the store of the host, When the effect evaluates, Then the evaluation fails, because that node has no nix.

User Story 4 - Upload the local-dev image (Priority: P3)

A developer runs dev-tf-incus-image. The effect uploads a small test image to the local Incus server. dev-tf-incus-bootstrap calls it.

Why this priority: Local development needs an image without a full group.

Independent Test: A VM test runs dev-tf-incus-image from the catalog against its Incus server. Before the run, the server has no image under the alias. After the run, the alias points at the fingerprint that the effect gives. A second run reads the local state and does not change the images on the server.

Acceptance Scenarios:

  1. Given a local Incus server, When the developer runs dev-tf-incus-image, Then the server holds the image with SSH, a test program and an operator user.

User Story 5 - Choose the container format (Priority: P2)

The images are OCI images by default. An operator can set containerFormat = "lxc" to upload LXC images. The format of an Incus container comes from the image that Incus creates it from, and the container keeps that format for its life.

Why this priority: The OCI format is portable. In both formats, a shared-store container comes from the stub image and needs no raw.lxc, thus it can go in a restricted Incus project. LXC stays available while the OCI format is new.

Independent Test: The VM test of the Incus deploy uploads the OCI images through dev-tf-incus-image, checks that both images have the type oci, and checks that the shared-store container is an OCI container with no raw.lxc. The check integrations-incus-oci starts OCI containers of both shapes, and a shared-store OCI container and a shared-store LXC container in a restricted project. A plan test of the Incus deploy asserts that a shared-store container comes from the stub image.

Acceptance Scenarios:

  1. Given the default format, When the effect applies, Then the server holds the base image under the alias and the stub image under <alias>-stub, both of type oci.
  2. Given containerFormat = "lxc", When the effect applies, Then the server holds the base image under the alias and the stub image under <alias>-stub, both of type LXC.

User Story 6 - Keep the images current from CD (Priority: P2)

The CD workflow cd-hosting-provider runs the image effect of the group before each apply or plan of the deploy, with TF_INCUS_IMAGE_STATE set to ${TF_HTTP_ADDRESS}-<group>-image. Thus the state is on the HTTP backend, and a container that the deploy creates or replaces starts from the images of that commit.

Why this priority: Without it, the images stay as an operator last uploaded them, and new containers start from an old closure.

Independent Test: A unit test reads the effect from the catalog with and without TF_INCUS_IMAGE_STATE, and checks the HTTP backend of each.

Acceptance Scenarios:

  1. Given TF_INCUS_IMAGE_STATE, When the effect applies, Then its state is on the HTTP backend at that address.
  2. Given an empty HTTP state and images under the alias from a different state, When the effect applies, Then it deletes those images and uploads its own.
  3. Given a failed upload, When the CD workflow runs, Then the deploy runs, and the workflow fails after it.

Edge Cases

  • The server has no stub image: the plan of an Incus deploy with a shared-store node fails.
  • The format changes and a container is live: the container keeps its format. Only a container that the deploy creates or replaces gets the new format.
  • The alias exists on the server and the local state does not hold it, because the state is lost or a different machine uploaded the image: the provider cannot import or update in place, so the apply fails. dev-tf-incus-bootstrap deletes the image before each upload, so its runs do not fail. A direct caller deletes the old image first. The next run builds it again.
  • The alias exists on the server and the HTTP state is empty: the effect deletes the images under the alias and <alias>-stub, then uploads. Until the upload completes, the server has no image under the alias.
  • The HTTP state holds an image that a different state replaced later: the apply fails on the alias, as for a local state. The operator deletes the images under the alias, or the image state.
  • A trust token is given and the client certificate exists: the token is not used.
  • The client certificate does not exist: the effect enrolls it with the token before OpenTofu starts, and spends the token once.
  • No client certificate and no token: the apply fails at run time.
  • A setup whose method is not tf-incus has no image part.
  • The host and the port of the server that the caller exports win over the built-in values.

Requirements (mandatory)

Functional Requirements

  • FR-001: With no source node, the effect MUST build the root file system from core/setups/nixos/incus-base-image.nix, which holds nix, DHCP on eth0 and no node configuration. With a source node, the effect MUST build it from the tf-incus NixOS configuration of that node.
  • FR-002: The image configuration MUST NOT write a hostname file.
  • FR-003: The effect MUST build the image metadata apart from the root file system. With containerFormat = "oci", the metadata MUST give the type oci and an OCI config.json with the entrypoint /sbin/init, and the root file system MUST import core/setups/nixos/oci-runtime.nix. With containerFormat = "lxc", the metadata MUST give no type and no config.json.
  • FR-004: The effect MUST create one Incus image from the root file system and the metadata, under the alias, and MUST output its fingerprint and alias.
  • FR-012: The default of containerFormat MUST be oci.
  • FR-013: In both formats, the effect MUST create the stub image under <alias>-stub, from the stub root file system and the same metadata, and MUST output its fingerprint. The stub root file system MUST hold only the mount points and /sbin/init, a symlink to /nix/var/nix/profiles/system/init.
  • FR-005: The source node MUST be the given node, else none. A given node MUST be a node of the group, and MUST NOT share the store of the host.
  • FR-006: The default alias MUST be one shared definition, which the bootstrap runner also uses.
  • FR-007: The effect MUST connect to the Incus API over HTTPS, at the standard Incus port by default. The host and the port MUST be overridable at run time.
  • FR-008: The effect MUST read the client certificate at run time, and MUST enroll it with the trust token before OpenTofu runs when it does not exist.
  • FR-009: The catalog MUST NOT offer the generated effect for the dev group.
  • FR-010: Without TF_INCUS_IMAGE_STATE, the state MUST be a keyed local file. With it, the state MUST be on the HTTP backend at that address. The effect MUST NOT read TF_HTTP_ADDRESS, the address of the state of the deploy.
  • FR-011: The effect MUST read only the alias, the source node and TF_INCUS_IMAGE_STATE from the environment, and MUST expose the OpenTofu verbs.
  • FR-014: With an HTTP state, the first apply into an empty state MUST delete the images under the alias and <alias>-stub before it uploads. A local state MUST NOT delete images.
  • FR-015: The CD workflow of a group with a tf-incus deploy MUST run the image effect before the apply and the plan of the deploy, with the state at ${TF_HTTP_ADDRESS}-<group>-image. A failed upload MUST NOT stop the deploy, and MUST fail the workflow.

Key Entities

  • Image: An Incus image of a root file system and its metadata, named by its alias.
  • Container format: OCI or LXC. Incus sets it from the image when it creates a container.
  • Stub image: An Incus image that holds no closure, named <alias>-stub. A container that shares the store of the Incus host is created from it.
  • Source node: The optional node whose configuration goes in the image.
  • Client certificate: The certificate and key with which the effect logs in to Incus.
  • Trust token: A one-time Incus token that enrolls the client certificate.
  • Image state: An OpenTofu state in a keyed local file on the machine that runs the effect, or on the HTTP backend at TF_INCUS_IMAGE_STATE.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After one apply, the Incus server lists exactly one image with the alias, and exactly one image with <alias>-stub.
  • SC-002: Every container that starts from the image reports its own hostname.
  • SC-003: The VM test of the Incus deploy passes.

Assumptions

  • The machine that runs the effect can build the closure of the image.
  • The Incus server accepts a client certificate or a trust token from the caller.
  • The image state is local by default. The image is the base of each Incus container, the state backend node included, so a cold provider uploads it before the state backend exists. dev-tf-incus-bootstrap deletes the image before each upload (FR-004 of incus-bootstrap.md), so each of its runs is idempotent and a lost or foreign state is harmless. CD keeps no file from one run to the next, thus it uses the HTTP backend, which exists by then.
  • A direct caller removes an old image under the same alias when the local state does not hold it.

Known Deviations

  • The local-dev deployment sets a deployer public key that its setup never reads, although the catalog says that the key goes to the authorized keys of root (#2131).
  • The graph declares a category input that it does not use (#2131).