Fediversity

Feature Specification: Deployment method ssh

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code of core/effects/ssh/ and core/setups/ at 14504f80d, issue #723.

The ssh method updates NixOS hosts that already exist. It reaches each host over SSH, copies the closure and switches the host to it. It creates no machine, no address and no state.

The deploy body, its phases and its verdict are in spec.md. The program grammar and the verbs are in 004. The tf method (tf.md) runs the same SSH transport for each node from OpenTofu.

User Scenarios & Testing (mandatory)

User Story 1 - Update existing hosts in order (Priority: P1)

An operator has a roster of NixOS hosts that they can reach over SSH. They run the ssh program. Each host gets its new configuration, one host after the other.

Why this priority: This is the only purpose of the method.

Independent Test: A VM test deploys two targets from a deployer. It checks that the closure's file rules apply before the units start, that both targets serve HTTPS to the deployer, and that TLS ends on the ssl resource. Each target takes a login only on a non-standard port: one through the SSH options of its own node entry, one through an SSH client configuration. A second VM test boots the same configuration with no transport, as the baseline.

Acceptance Scenarios:

  1. Given two reachable hosts in the node list, When the operator runs the program, Then the program announces and switches each node in list order.
  2. Given a node that fails, When the program runs, Then the later nodes still deploy, the program names each failed node, and the program fails at the end.
  3. Given a successful switch, When the deploy ends, Then the boot entry of the host points at the new generation.

User Story 2 - Reach each host with its own identity (Priority: P2)

Hosts of one roster can differ in address, user, key and SSH options. The operator states one shared identity and, per node, the overrides.

Why this priority: Without overrides, all hosts must share one user, one key and the roster address.

Independent Test: A unit test of the layering. A unit test that the ssh and tf readers parse the shared user and the node map. A unit test that an SSH option in command-line form fails at evaluation, and that the form schema refuses it.

Acceptance Scenarios:

  1. Given no overrides, When the setup builds the SSH settings of a node, Then they use the root user, the deployer key and the roster host.
  2. Given shared defaults and a node entry, When the setup builds the SSH settings of that node, Then the node entry wins over the defaults, and the defaults win over the built-in values.
  3. Given a node entry with SSH options, When the setup merges it, Then the list replaces the default list and does not append to it.
  4. Given a node entry or a field that is empty, When the setup merges it, Then it inherits the layer below, and an empty host gives the roster host.
  5. Given an SSH option that is not a bare Key=Value, such as a command-line flag, When the deploy is evaluated, Then the evaluation fails with a message that names the form, and the form schema refuses the same item.
  6. Given a shared user from the environment, When the setup builds the SSH settings of a node, Then it uses that user in place of root, below both override layers.

User Story 3 - Get through a slow or broken target (Priority: P2)

A target can be slow, can hold the bootloader lock, or can lose its system bus during the switch. The deploy works around these faults or reports them.

Why this priority: A deploy that hangs or gives a false success hides the state of the host from the operator.

Independent Test: A VM test breaks the target from outside: a blocked start job, a switch that cannot be scheduled, a hanging transient unit, a dead bus, a stuck login manager, and a held bootloader lock with its retry budget. The verdict tests of spec.md run on the same transport.

Acceptance Scenarios:

  1. Given a switch that still holds the bootloader lock, When the bootloader install finds the lock, Then it tries again at a short interval, up to a bounded number of attempts.
  2. Given a target in a container, When the switch runs, Then the switch has a run time limit. On bare metal it has no limit.
  3. Given a container target, When the bootloader install fails, Then the failure hint tells the operator to use tf-incus.

User Story 4 - Refuse a host that shows a different key (Priority: P2)

The operator can pin the public host key of a node. The deploy then logs in only to a host that shows that key.

Why this priority: The deploy logs in as a user that can switch the system. A host that is not the node must not get that login.

Independent Test: A VM test reads the effective SSH options of each connection, with and without a pinned key, with a host-key option in the node options or the client configuration, and for a jump host. The two-target VM test pins each target key, and shows that a different pinned key stops the deploy before the login.

Acceptance Scenarios:

  1. Given a node with a pinned key, When the deploy connects, Then it accepts only that key, and a host that shows a different key stops the deploy of that node before the login.
  2. Given a node with no pinned key, When the deploy connects for the first time, Then it accepts the key and saves it. A later deploy that reads the same saved keys refuses a different key.
  3. Given a host-key option in the node SSH options or in the client configuration, When the deploy connects, Then that option wins over the values of the deploy.

Edge Cases

  • No key file, no deployer key and no agent: SSH fails at once and does not prompt.
  • A client configuration sends the host to a different address: the program logs the change.
  • A jump host: the direct SSH calls and the closure copy both reach the node through it. The jump host itself gets trust on first use with the same saved keys, also when the node key is pinned. The connection to the node runs end to end and checks the key of the node.
  • A remote command that hangs: a keepalive ends it after a bounded time.
  • A target without the tools of the deployer on its search path: remote commands still run (see spec.md).
  • The combined environment: all nodes go into one configuration on the first host of the roster.
  • A changed host key: with a pinned key, the deploy of that node stops. Without it, the deploy refuses a key that differs from the saved one. A fresh CI runner has no saved keys, so it accepts the key again.
  • A node that is installed again gets a new host key: the operator changes the pinned key, or removes the saved key.

Requirements (mandatory)

Functional Requirements

  • FR-001: The method MUST deploy the nodes one at a time, in list order, in the foreground.
  • FR-002: A failed node MUST NOT stop the later nodes. The program MUST fail when one or more nodes failed.
  • FR-003: Each node MUST run the SSH transport with the shared deploy body of spec.md, with its node plan, address, user, key, pinned host key and SSH options.
  • FR-004: The key MUST come from the node, else from the deployer key in the environment, else from the SSH agent.
  • FR-005: Every SSH call MUST be non-interactive. A client configuration that the operator gives MUST apply to every SSH call.
  • FR-006: A remote command MUST have a bounded connect time and a keepalive that ends a hung command after a bounded time.
  • FR-007: The closure copy MUST compress the transfer and MUST let the target substitute paths.
  • FR-008: In a container, the switch MUST run with a run time limit.
  • FR-009: The boot entry finalization MUST be bounded in time. It MUST retry while the bootloader lock is held, at a short interval, up to a bounded number of attempts.
  • FR-010: The SSH transport MUST NOT revert the boot entry.
  • FR-011: The setup MUST layer the SSH settings of a node as built-in defaults, then the shared defaults, then the node entry, and fall back to the roster host. The built-in user MUST be the shared user from the environment, else root. An empty or absent field MUST inherit the layer below. A pinned host key MUST come only from the node entry.
  • FR-012: The method MUST read its coordinates (the deployer key, the shared user, the node map and the others) through its declared key set.
  • FR-013: With a pinned host key, every SSH call to the node MUST trust only that key, and MUST ignore any global known-hosts list. The pinned key MUST live in a temporary file that the program removes when it exits.
  • FR-014: Without a pinned host key, every SSH call to the node MUST trust a key on first use and refuse a changed key later. The saved keys MUST live next to the key file, else in the temporary directory. When neither is writable, SSH MUST use its default known-hosts file.
  • FR-015: A host-key option in the node SSH options MUST win over the host-key options of FR-013 and FR-014. The program MUST NOT override a host-key option that the client configuration sets.
  • FR-016: A jump host MUST get trust on first use, with the saved keys of FR-014.
  • FR-017: Each SSH option of a node MUST be a bare Key=Value option, such as Port=2222. The type MUST refuse an item that starts with - or has no key. The type MUST carry a pattern that admits the same items in the form schema, so that a front-end refuses the same items as the evaluation.

Key Entities

  • Node: A deployment name, a NixOS configuration and its SSH settings. The list order is the deploy order.
  • Host SSH settings: The address, the user, the key, the optional pinned host key (one public key line of the host) and the SSH options (bare Key=Value items, FR-017).
  • SSH node map: Shared defaults and per-node entries, which the operator gives as JSON in the environment. Every field is optional. A pinned host key is a field of a node entry only. The effect metadata describes the map for a front-end.
  • Transport hooks: Remote command, closure copy, switch run time limit, boot entry finalization, boot entry revert and the failure hint.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After a run, every reachable node runs its new closure.
  • SC-002: One unreachable node makes the program fail and names that node, and does not block the other nodes.
  • SC-003: A held bootloader lock costs at most the bounded retry budget, then the deploy fails.
  • SC-004: The recovery and verdict VM tests pass on the SSH transport.
  • SC-005: A node whose pinned key the host does not show gets no login.
  • SC-006: A deploy reaches a node on a port that only its node entry names, and a node on a port that only the client configuration names.

Assumptions

  • Each host already runs NixOS and accepts SSH as a user that can switch the system.
  • The deployer can build every closure, or substitute it.
  • A node with no pinned key is reached over a trusted network for its first connection, because that connection accepts any key.
  • A host that is a container is deployed with tf-incus, not with this method.