Check and test a change

This page tells how to check a change without a deploy, how to write a NixOS test, and how to test a feature branch or an operator branch.

Check a change without a deploy

You can check a change at five levels. Only the last one needs access to the hosting provider.

1. Build the NixOS systems of the nodes. This needs no infrastructure and changes nothing:

   nix build ./core#nixosConfigurations.operator-mastodon.config.system.build.toplevel
   

   Each node has a nixosConfigurations.<group>-<machine> attribute. The operator-* nodes use the sample operator configuration of core/setups/groups/operator.nix.

2. Deploy into a VM test. The tests deploy the applications with OpenTofu, and then check them:

   • checks.<system>.apps-tf deploys to NixOS VMs.
   • checks.<system>.apps-tf-incus deploys to Incus containers in a VM.
   • checks.<system>.apps-api deploys through the API, as the panel does.

   These tests need much memory. CI runs them on each pull request.

3. Read the scripts of a deploy. Each deploy verb is a shell script. Build it, and do not run it, to see what it would run:

   cat "$(nix build --impure --no-link --print-out-paths .#hosting-provider-tf-incus-all)"/bin/*
   

   The script calls other scripts by their store paths. You can read these with cat too. The first script only adds the dry run of the next step, and then runs the deploy script.

4. Evaluate a deploy, and do not run it. With DRY_RUN=1, a deploy evaluates the NixOS system of each node that it deploys, prints the derivation of each system and the command that it would run, and stops:

   DRY_RUN=1 nix run --impure .#hosting-provider-tf-incus-all
   

   It connects to no host and runs no tofu. It evaluates each node from the same configuration.nix as the deploy, with the same environment, such as DEPLOYMENT. Thus it can find errors that only your configuration causes, which step 1, with the sample configuration, does not find. It fails if a node does not evaluate. All the deploy attributes of the flake accept DRY_RUN=1. A verb that deploys no nodes, such as -plan, only prints its command.

5. Make a plan. The OpenTofu plan verb shows what a deploy would create, change and destroy, but changes nothing:

   nix run --impure .#hosting-provider-tf-incus-all-plan
   

   A plan reads the state, the addresses document and the coordinates of the hosting provider, so first prepare your shell as in Deploy from a workstation. To make a plan for an operator deployment, give its configuration as JSON in DEPLOYMENT, and use .#operator-tf-incus-all-plan. This is the configuration that the panel sends.

Write a NixOS test

A NixOS test runs each machine in a container (systemd-nspawn): declare the machine in containers. Use nodes, which runs the machine as a VM (QEMU), only when the machine needs a feature that a container cannot give. Write the phrase "QEMU rather than nspawn" and this feature in a comment directly above nodes. The check core/testers/test-machine-kind-test.nix finds a nodes attribute without this comment.

A container shares the kernel of the build host and runs in the Nix build sandbox. Thus some steps that work on a real machine fail in a container:

• The seccomp filter of the sandbox refuses setuid files and setxattr, /proc/sys is read-only, and the paths of the store have the owner 65534. core/testers/nspawn-sandbox-compat.nix removes the steps that fail for these reasons.
• The test driver runs its commands in a cgroup out of the cgroup namespace of the container. The kernel does not let a process move from there to a different cgroup (ENOENT). Thus a program that moves processes, such as podman, fails when the driver runs it. Run it from a unit of the container: a service, or systemd-run --wait --pipe.
• The container has no cgroup controllers, thus a program cannot set limits such as pids.max in it.

Test a feature branch

To deploy a branch or a pull request from CI, start the CD workflow by hand in "Actions", on main. Set ref to the name of the branch, or to refs/pull/<n>/head for pull request <n>. This also works for a pull request from a fork. Only a user with write access to the repository can run a workflow.

The run checks out the commit of ref, and runs its code with the secrets of the job. The workflow file stays that of main. verb: plan does a dry-run.

The run deploys to the same machines as main. Its deploy stays until the next deploy. A push to main deploys only when it changes the files of the machine. Thus, to restore main, start the workflow again with an empty ref. Runs of one workflow on main wait for each other, thus a run with ref does not overlap a deploy of main.

To test on machines that are not those of main, deploy from a workstation (see above) to a hosting provider that is for tests. Use coordinates of its own: a different state backend (TF_HTTP_ADDRESS), a different zone (DNS_DOMAIN), and a different Incus host, or a different Incus project (TF_VAR_project). The hypervisor declares only the project that core/effects/common/incus-project.nix names, thus a different project must exist on the host before the deploy. Then a test does not change the deploy of main.

To set up a hosting provider for tests, follow the bootstrap.

Test an operator branch

The operator code deploys the applications of a panel user, such as Mastodon. To test a change to that code, deploy it from the panel:

1. Push the branch to the main repository. The API accepts a source only from a repository that it trusts (trustedSourceUrls), and in that repository, only a branch. A pull request from a fork is not a branch, thus the API refuses it.
2. In the panel, open the form of a deployment, then open "Advanced".
3. In "Configuration source", type {"type": "git", "url": "https://git.fediversity.eu/fediversity/fediversity", "ref": "<branch>"}. The form then loads again, with the options of that branch.
4. Fill in the form, and deploy.

The ref floats: a deploy after a new push to the branch uses the new commit. To pin a commit, add "rev": "<commit id>". The branch must contain that commit.

To remove the test, click "Delete" on the deployment in the list of deployments. This destroys its resources.

To test a new application in the same way, first do the steps of Add an application.