Fediversity

Feature Specification: Test harness, checks and documentation

Created: 2026-09-22

Status: Implemented

Input: Reconstructed from the testers, the checks and the docs of core at 0bf4ecd36, pull requests #1210, #1790 and #1833, and the design notes of the maintainers.

User Scenarios & Testing (mandatory)

User Story 1 - Test a deployment end to end (Priority: P1)

A developer of core or of a front end wants to prove that a deployment works. The shared harness starts the machines of a deployment in a NixOS test (containers, or VMs where a machine needs them), deploys the applications with a deployment method, and checks them.

Why this priority: the deployment programs change machines. Only an end to end test shows that they work.

Independent Test: a VM test that deploys the applications with the tf method and checks each one.

Acceptance Scenarios:

  1. Given the default configuration with the applications on, When the end to end test runs, Then each application answers on its domain.
  2. Given a front end with its own deployment driver, When it uses the harness, Then it gets the same machines and the same deployment steps as core.

User Story 2 - Run one check by name, and route every check in CI (Priority: P1)

Each check has a name that starts with its family (application, deployment, integration, contract, service and unit tests), a description, and a route to a CI runner. A developer runs one check by name.

Independent Test: add a check without a route. The check that annotates the checks fails.

Acceptance Scenarios:

  1. Given a check name, When a developer builds that check, Then only that check builds.
  2. Given a unit test suite, When CI runs, Then it builds the group check of the directory. The check per suite stays available to run one suite.

User Story 3 - Read the documentation of core (Priority: P2)

A reader opens the documentation site of core. It shows the concepts, the data model, the operator configuration, the hosting resources, the names of the deployment programs, the module options and the reference of the library from its doc comments.

Independent Test: build the docs site. The page of the flake attributes lists each family of program names.

Edge Cases

  • An environment variable is documented next to the code that reads it. The docs take the description from its declaration, so they cannot go out of date.
  • A machine needs a VM, not a container, only for a feature that a container cannot give: a TPM, the KVM device or fanotify (as the incus daemon needs), a reboot, a crash, a boot loader or a hardware watchdog, or a kernel or a kernel log of its own.

Requirements (mandatory)

Functional Requirements

  • FR-001: core MUST publish a test harness that front ends can use for their own end to end checks.
  • FR-002: The harness MUST take the modules and drivers of a front end as parameters, and MUST NOT import a front end.
  • FR-003: Each check MUST have a family prefix, a description and a CI runner route.
  • FR-004: core MUST build the docs site from its doc pages, the module options and the doc comments of the library.
  • FR-005: The docs MUST take the names of the deployment programs from the same grammar that renders them.
  • FR-006: The docs MUST include a guide that tells a hosting provider how to find and use the web UIs of a hosting-provider deployment.
  • FR-007: The docs MUST tell a maintainer how to check a change without a deploy, how to deploy from CI and from a workstation, and where to find the deploy logs. A template MUST list each repository secret and variable that a CD job reads.
  • FR-008: A unit test of core MUST import the entrypoint of core, not the entrypoint of the repository. A Nix file of core MUST NOT import the repository entrypoint or refer to a path in api or panel (principle I). A boundary check finds a file that does. A source file of core outside specs/ and docs/md/ MUST NOT name api or panel in its text, comments included: not a source file, a node or an option namespace of these leaves, and not a leaf itself in prose. A comment states the contract of core to any caller. A second boundary check finds a file that does. It is case-sensitive, thus a review holds the upper-case form, which also names the API of other products. A VM test of core MUST deploy from a tree without the repository entrypoint. A deployment reads the boundary of the operator tree: an entrypoint at the root of the tree gives it, and in a tree with no such entrypoint the entrypoint of core gives it. root-path and operator-root-path name the directory that holds core, not an entrypoint: core reads npins and nix in it, and a relative caller starts from it. Windmill runs the development shell of a deployment program from the flake in DEPLOYMENT_ROOT, thus a tree that a front-end deploys holds a flake at its root.
  • FR-009: A NixOS test MUST run each machine in a container (systemd-nspawn), and MUST run a machine as a VM (QEMU) only when it needs a feature that a container cannot give. Such a VM MUST state this feature where the test declares it. A guard check finds a VM that does not.

Key Entities

  • Check: a derivation with a name, a description and a runner route.
  • Tester: the shared harness of the NixOS tests.
  • Docs site: the published documentation of core.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The format check, the docs build and the unit tests pass.
  • SC-002: No check lacks a runner route.

Assumptions

  • The heavy checks run on the CI runners, not on the machine of a developer.