Fediversity

Feature Specification: Method family deployment-methods

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code of core/effects/ and core/render/ at 14504f80d, pull requests #1531, #1705, #1712, #1784, #1790, #1848 and #1857.

This spec covers what the deployment methods share. The program grammar, the verbs, the published key sets and the state rules are in 004. This spec does not repeat them. Each method has its own item spec, listed in the Index.

User Scenarios & Testing (mandatory)

User Story 1 - Deploy a node and get a verdict on it (Priority: P1)

An operator runs a deployment program for a group. Each method gets the closure onto the node in its own way. After that, all methods that switch a closure run the same deploy body. The body builds the closure, copies it, switches the node, and then decides whether the node came up. The operator reads one verdict per node.

Why this priority: A deploy that succeeds on a broken node hides the failure. The verdict is the only signal that a closure works on the node.

Independent Test: VM tests that deploy one host over SSH with a closure that leaves a unit failed, a unit in a restart loop, and a unit that systemd cannot spawn. Unit tests of the parsers that read the unit states.

Acceptance Scenarios:

  1. Given a closure that leaves a unit failed, When the deploy finishes, Then the verdict names the unit and the program fails.
  2. Given the same closure and a lenient mode, When the deploy finishes, Then the verdict still names the unit and the program succeeds.
  3. Given a unit that systemd never spawned because of a host fault, When the verdict runs, Then it names the unit as a host fault and does not blame the closure.
  4. Given a unit in a restart loop, When the deploy finishes, Then the verdict reports the loop and does not fail the deploy for it.

User Story 2 - Deploy the submitted configuration, and only that (Priority: P1)

The api submits a configuration. The deploy program reads the submission when it is evaluated, merges it into the baked configuration, and makes the plan of each node. The apply deploys only the nodes that the merged configuration enables.

Why this priority: If the node list and the node configurations disagree, a container gets no applications and fails during its attestation. An apply with a different submission would deploy a configuration that the operator did not submit.

Independent Test: Unit tests of the merge and of the plan writer.

Acceptance Scenarios:

  1. Given a submission that names an application without enabling it, When the merge runs, Then the application is off.
  2. Given a submission that does not name an application, When the merge runs, Then the baked entry for that application stays.
  3. Given a submission that puts the applications, the domain and the email inside the branch of the picked method, When the merge runs, Then these values move to the top level and the method branch goes away.
  4. Given an apply with a submission that is not the one the program was evaluated with, When the apply starts, Then it refuses.

User Story 3 - Redeploy a node when its inputs change, and only then (Priority: P2)

The plan of each node holds every coordinate that the plan reads from the environment. A change in such a coordinate redeploys the node. A change in a per-run value, such as the location of a staged credential, does not.

Why this priority: A missed input leaves a node on a stale closure. A per-run input replaces every node on every apply.

Independent Test: A unit test that finds every environment read in the files that make a node plan, and checks that each one is a declared node identity key. A unit test that two runs with different credential locations give equal plans. A VM test that the seed pass and the deploy pass evaluate a node once.

Acceptance Scenarios:

  1. Given a change in the DNS domain, When the program makes the plan of a node, Then the plan changes and the node is redeployed.
  2. Given two runs that stage a credential at different temporary locations, When the plans are made, Then the two plans are equal.
  3. Given a node that the seed pass already evaluated, When the deploy pass evaluates the same node with the same inputs, Then it reuses the result.

User Story 4 - Judge a running node without a switch (Priority: P3)

The incus graphs run a second, verdict-only pass on a node after its attestation step. This pass does not build, copy or switch. It waits for the secret chain of the node, restarts failed units that the closure wants, and then gives the verdict.

Why this priority: The first pass judges a node before its secrets arrive. Only the second pass can say if the node converged.

Independent Test: VM tests of the verdict-only pass on a chain that converges, a chain that fails, and a unit that keeps looping.

Acceptance Scenarios:

  1. Given a secret chain that converges, When the verdict-only pass runs, Then it waits for the chain and restarts units only after the chain is up.
  2. Given a chain member that failed, or that shows no progress for a while, When the pass samples the chain, Then it stops early and names that member.
  3. Given a unit that still loops after its loop budget, When the verdict-only pass runs, Then the pass fails and names the unit and its budget.

Edge Cases

  • A node that the deployer cannot reach gets a few samples in the chain wait. Then the verdict reports the node as unreachable.
  • A severed system bus leaves the unit sets unread. The report shows them as unread, not as empty.
  • A closure whose network link files expect a permanent hardware address that the node does not have is refused before the copy. A closure with no such file is not gated.
  • A switch that never activates makes the transport revert the boot entry to the running generation.
  • A failed unit that the deployed configuration does not define is set aside and does not fail the deploy (#1712).
  • A unit that systemd skips on a condition is not restarted, and the verdict still names it.
  • A cached evaluation result that names a derivation the store does not hold is ignored. A cache that cannot be written is not an error.
  • A node whose own command search path resolves nothing still runs remote commands, because each remote command can reach the tools of the deployed closure.

Requirements (mandatory)

Functional Requirements

  • FR-001: The catalog MUST expose the methods ssh, tf, tf-incus, tf-incus-operator, k8s and k8s-operator, each with its setup, setup arguments and description. A method MAY declare that it is available only for some groups, and every consumer MUST take the methods of a group from one shared per-group lookup. tf-incus-operator and k8s-operator MUST be available only for a group that runs an application.
  • FR-002: Every method that switches a closure on a node MUST deploy the node through one shared deploy body. A transport layer per method family (SSH for ssh and tf, incus for tf-incus and tf-incus-operator) MUST supply how to reach the node, run a remote command, copy a closure, limit the switch run time, finalize the boot entry and revert it. The deploy body MUST NOT depend on a transport.
  • FR-003: The deploy body MUST run its phases in this order: build, wait for the node, push the closure, wait for the secret chain, snapshot the unit states, switch, read the switch result, detect a switch that never confirmed, collect the units the closure owns, restart failed owned units, let the node settle, watch for cycling and looping units, classify, give the verdict, report, finalize the boot entry, conclude.
  • FR-004: The deploy body MUST set the system profile and the boot entry before it fails for a failed unit, so that a node with a failed unit keeps the new boot entry.
  • FR-005: The verdict MUST name failed units, restart loops, units that cycle, start jobs that are too old, and units that systemd never spawned. It MUST fail the program for a failed unit, unless the operator selects the lenient mode.
  • FR-006: A unit MAY declare its own loop budget. The budget MUST delay the loop verdict for that unit and MUST NOT remove it.
  • FR-007: In verdict-only mode, the deploy body MUST NOT build, copy or switch. It MUST wait for the secret chain, for a bounded time, before the snapshot, and MUST restart only failed units that the closure wants.
  • FR-008: Every timing and threshold of the deploy body MUST have a default, and the operator MAY tune it.
  • FR-009: The deploy body MUST cache the evaluated system of a node, keyed on the node plan alone. The cache MUST NOT be a source of truth.
  • FR-010: The merge of a submission into the baked configuration MUST be a pure data merge. It MUST raise the applications, the domain and the email out of the branch of the picked method, remove the method branch, and turn off each application that the submission names but does not enable.
  • FR-011: The program MUST read the submission when it is evaluated, and MUST make the plan of each node from it. The program MUST record a digest of that submission, and the apply MUST refuse a different submission.
  • FR-012: The caller of an effect names the setup that makes the plans of the nodes. A caller relative to the operator tree MUST resolve in the operator tree, thus a separately pinned operator tree uses its own copy of the setup. An absolute caller MUST be used as it is, thus a setup outside the operator tree, for example in a flake that has this repository as an input, can be a caller. Such a setup does not follow the operator pin. Its node plans MUST still name only files of the operator tree, as FR-014 states.
  • FR-013: The plan arguments MUST NOT hold the locations of deployer credentials. These are per-run values and are not part of the node identity.
  • FR-014: The plan of a node MUST be data: attribute sets, lists, plain values, and references to files of the operator tree. The writer MUST store a file reference relative to the operator tree, and MUST fail on a value that is not data. The reader MUST resolve the reference in the operator tree.
  • FR-015: The node configuration MUST take the plan of the node from the written plan and MUST NOT evaluate the plan again. The addresses that NetBox allocates MUST NOT be in the plan. An incus node MUST read them at run time from the incus guest API.
  • FR-016: Each effect that bakes a node map MUST publish the names of its enabled nodes and of its disabled applications, from the same submitted plan as its node configurations. The published node names MUST NOT hold a disabled node. The api MUST treat these names as optional. The apply MUST refuse a submission for a node map that publishes no names.
  • FR-017: The node identity keys MUST list every coordinate that the files that make a node plan read from the environment, so that the front-ends forward it. The plan holds the values of these coordinates, thus the node identity covers them.
  • FR-018: Each method MUST declare each coordinate it reads, with a description and a default text. The declared coordinates, plus the self pins and the hosting and host coordinates, MUST make up the key set of each method.
  • FR-019: Each OpenTofu method MUST give one program per verb of 004, plus a run program. Every verb MUST go through the shared bootstrap wrapper. A verb program MUST turn off the bootstrap mode.
  • FR-020: Before apply, destroy, plan and refresh, the OpenTofu methods MUST seed an empty self-state, so that the read of its own state on a first apply reads empty and does not fail.
  • FR-021: Each operator-facing effect MUST declare its settings schema as effect metadata. The kind of the effect MUST be deployment when the effect has a node roster, and effect otherwise. An effect that the form does not offer MUST declare no metadata and MUST be a building block.
  • FR-022: Each settings leaf of the effect metadata MUST have a default.
  • FR-023: The node roster in the effect metadata MUST hold the nodes of each group that runs an application, and MUST NOT name a group.
  • FR-024: A method MAY roll out an image of the node configuration in place of a closure switch. Such a method MUST name the image by its digest, MUST wait for the node to become ready for a bounded time, and MUST set the node back to the previous image and fail when the node does not become ready. It does not run the deploy body of FR-002 and FR-003.
  • FR-025: The effect registry MUST be a set of modules with one set of options (effects, buildingBlocks). The in-tree effects MUST be one of these modules. A consumer MUST be able to add effects with more modules, through the effectModules argument of the registry, of core/lib/lib.nix, of a setup or of render, or through extend on a registry. A module MUST NOT replace an effect that another module defines, and MUST NOT use a name of the registry itself.
  • FR-026: A deployment <method> MUST dispatch to the effect <method>-hosts of the registry, for an in-tree effect and for a consumer effect alike. When the registry has no such effect, the evaluation MUST fail with a message that names the deployment and the deployment effects of the registry. A consumer method MUST be able to give its setup as a path or a function in the methods argument of render. A consumer method MUST NOT replace an in-tree method, and an unknown setup name MUST fail with a message that names the in-tree setups.

Key Entities

  • Method: One catalog entry. It names a setup, its arguments, its verbs, a description, and optionally the groups it is available for.
  • Transport layer: The part of a method that reaches the node and supplies the hooks of the deploy body.
  • Deploy body: The shared sequence of phases that builds, copies, switches and judges a node.
  • Image rollout: The deploy of a method that runs the node from an image: push the image, roll the node out to it, and set it back to the previous image if the node does not become ready.
  • Verdict: The report per node. It lists the switch result, the system state, the unit sets and the strict mode.
  • Plan file: The plan of a node as data. The program writes it when it is evaluated, and the node configuration reads it.
  • Node identity keys: The coordinates that a node plan reads, which the front-ends forward.
  • Effect metadata: The settings schema of an operator-facing effect, which the form reads.
  • Building block: An effect that the form does not offer, such as a registrar or NetBox effect. A setup, a check or the catalog composes it into a program.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: A deploy that leaves a unit failed fails on that apply and on the next.
  • SC-002: Two applies with the same submission and different credential locations produce the same node plans.
  • SC-003: The seed pass and the deploy pass of one node evaluate the node once.
  • SC-004: Every environment variable that a method reads is in its declared key set, as a unit test shows.
  • SC-005: A verdict-only pass on a failed chain member stops before the chain wait times out.

Assumptions

  • The nodes run systemd and NixOS.
  • The caller of an incus graph schedules the verdict-only pass after the step that publishes what the units wait for (the attestation step).
  • The api passes the submission in the shape that the merge reads.
  • The deployer can evaluate the source tree that the node configuration and its plan name.

Known Deviations

  • Nix cannot serialize a NixOS configuration (NixOS/nix#4000). The plan is data, and the node configuration imports the modules that the plan names. Each effect makes the plans of its nodes once, when the program is evaluated.

Index

  • ssh.md: ssh, a deploy over SSH to machines that already exist. It provisions nothing.
  • tf.md: tf, the same SSH deploy inside an OpenTofu graph with tracked state.
  • tf-incus.md: tf-incus, which creates the Incus containers of a group and deploys onto them in one OpenTofu graph.
  • tf-incus-operator.md: tf-incus-operator, which adds the per-tenant shared resources, DNS records, the domain and SPIRE registration to tf-incus.
  • k8s.md: k8s, which runs each node as a pod of a Kubernetes cluster, from an image, in one OpenTofu graph, and k8s-operator, which adds the DNS lane of an operator deployment to k8s.