Add an application

This chapter tells a maintainer how to add an application that operators can deploy from the panel, such as Mastodon, PeerTube or Pixelfed. The maintainer guide tells how to check, deploy and test the change after you add it.

Use Pixelfed as an example: core/components/operator/pixelfed/ has each part that this chapter names. In this chapter, <app> is the name of the new application, in lowercase, for example pixelfed.

What you do not change

The data model makes most of the application from one component. From components.<app>, the code makes these items, and you do not edit them:

The operator option applications.<app>, from the module of the component.
The form of the panel and the JSON schema of the API, through core/applications-type.nix.
The catalog of the API, with the title, description, home page and icon, through core/applications-catalog.nix.
The NixOS system nixosConfigurations.operator-<app> and the deploy attributes operator-tf-<app> and operator-tf-incus-<app>, from the node in the roster (see Add the node).
The DNS names, from the domain and subdomain options.

The panel and the API have no code for each application. The panel keeps the configuration of a deployment as JSON, thus a new application needs no database migration.

The data model page tells what each field of a component does.

Write the component

If upstream publishes a JSON schema of its configuration, pin it with npins, as pixelfed-config-schema in npins/sources.json. If the nixpkgs package needs a fix, add an overlay in nix/overlays/<app>.nix, and add it to nix/overlays/default.nix.

Make core/components/operator/<app>/default.nix. It sets components.<app>:

components.<app> = {
  inherit name;
  package = pkgs.<app>;
  artwork = ./artwork.nix;
  optional = true;
  module = import ../../nixosService.nix { ... };
  implementation = cfg: lib.optionalAttrs cfg.enable { ... };
};

description and homepage come from package.meta if you do not set them.

artwork is the path of a file in this repository that gives the logo of the application. Use the logo that the upstream project supplies:

# core/components/operator/<app>/artwork.nix
{ pkgs, ... }:
{
  icon = "${pkgs.<app>.src}/<path to the logo>.svg";
  # Optional: a variant for dark backgrounds.
  # iconDark = "${pkgs.<app>.src}/<path to the dark logo>.svg";
}

module declares the options that the operator sets. core/components/nixosService.nix makes it from sharedOptions.nix, which gives enable, subdomain and domain, and from the upstream services.<app> options that you name in expose. Name only the options that are the decision of the operator. If an option in expose does not exist, the evaluation fails.

implementation changes the settings of the operator into requests for resources. Each attribute is a named group with one request, of a resource in core/resources/:

Need	Request
Single sign-on	`"<app>-oidc".oidc`
Email	`"<app>-smtp".smtp`
Cache	`"<app>-cache".redis`
Database	`"<app>-db".postgresql`
Object storage and a backup	`"<app>-bucket".garage`, with the helpers of `core/components/operator/app-resources.nix`
TLS certificate	`"tls".ssl.domains = [ cfg.domain ];`
Open ports	`"web".firewall.publicTCPPorts = [ 80 443 ];`
Workload identity	`"spiffe".spiffe.enable = true;`
Generated secrets	`"vars-backend".generateFiles.instances.<app>-app`
The other NixOS settings	`"<name>".nixos-module.module`

A request can have a nixos-configuration function. It gets the result of the resource, for example the OIDC client secret, and sets the NixOS options of the application from it. The hosting resources page lists the options of each request.

Add ./<app> to the imports of core/components/operator/default.nix.

Add the node

Each application runs on its own node. Add the node to the roster in core/setups/groups/operator.nix, in machines.published:

{
  pixelfed = 54;
  peertube = 55;
  mastodon = 56;
  <app> = 57;
}

The name of the node must be the name of the component. The number gives the documentation addresses 192.0.2.<n> and 2001:db8::<n>, which a deploy replaces. Use a number that no other node uses.

If you do not add the node, the application is in the form, but a deploy does not deploy it, and no check fails.

Then add the application at this location:

core/default-configuration.json: add "<app>": { "enable": false }, so that the default configuration has each application off.

Audit the options

core/components/operator/operator-facing-options.csv has one row for each setting of each application. Each row tells if the setting is for the operator or for Fediversity, and if it can make the node run code of the choice of whoever sets it. Add a row for each services.<app> option, and for each key of the upstream configuration. A row can use a trailing * for a group of keys. Use this audit to select the options in expose, and the settings that the form shows.

Then add the version that you audited to the list in the operator configuration page.

The check operator-options-audit fails if an option of the form has no row with the audience operator, or if such a row has a code-execution value other than no. It does not find a missing row for a setting outside the form. An option that only the component declares has no row: add it to COMPONENT_OPTIONS in core/lib/operator_options_audit.py. A readOnly property has no setter, thus the check skips it.

Add the checks

Make core/components/operator/<app>/test.nix. It calls mkAppTest from core/components/operator/testing/lib.nix, which deploys the application on one node, with an SSO provider, a mail relay, a directory and object storage beside it. Add a test script that logs in and uses the application.
In core/checks/default.nix, add the test as a check, and give it a description:
```
test-<app>-service = runNixOSTest {
  imports = [ ../components/operator/<app>/test.nix ];
  _module.args.sources = sources;
};
```
The description becomes the meta.description of the check, which the flake attributes page shows. No check finds a missing description.
In core/checks/ci.nix, tell CI where to run the check, and on which changes. Copy the entry of test-pixelfed-service. If a check has no entry, the evaluation fails.
In .forgejo/workflows/nix-flake-check.sh, add a case for operator-<app> beside operator-pixelfed. Without it, CI builds the node on each change of the repository.
Make the CI workflows again:
```
bash .forgejo/workflows/nix-flake-check.sh
```
This adds checks-test-<app>-service.yaml, and a job for the node in nixosConfigurations-operator.yaml. CI fails if the workflows are not up to date.

The apps-tf, apps-tf-incus and apps-api tests deploy only Pixelfed, as a sample of all applications. You do not change them.

Update the specifications and the documentation

No check finds these, thus look at each one:

Add core/specs/008-components/operator/<app>.md, as pixelfed.md.
In the components specification, add the application to FR-004 and to the index of components.
In the operator configuration page and in the concepts page, add the application to the lists of applications.

Check the result

Do the steps of Check a change without a deploy in the maintainer guide:

nix build ./core#nixosConfigurations.operator-<app>.config.system.build.toplevel
nix build ./core#checks.x86_64-linux.test-<app>-service

Then test the operator branch from the panel: the form shows the new application from the schema of your branch.

What makes an application easy to add

This section is for the developers of an application and for Nix packagers. It tells what makes an application easy to add with the steps above. If an application does not have an item, the component must do that work, or the operator must do it by hand.

The package

Fediversity has no special requirement for a package. The guidance of NGI Forge is sufficient:

Set meta.description and meta.homepage: the catalog of the API uses them. Keep the logo of the project in its source: the artwork of the component uses it. For example, the package of Pixelfed in nixpkgs sets meta, and the artwork of the pixelfed component takes the icon from pixelfed.src.

# The package of an application. `meta` gives the catalog of the API the
# description and the home page of the application.
{
  lib,
  stdenv,
  fetchFromGitHub,
}:
stdenv.mkDerivation (finalAttrs: {
  pname = "example";
  version = "1.2.0";

  src = fetchFromGitHub {
    owner = "example";
    repo = "example";
    rev = "v${finalAttrs.version}";
    hash = lib.fakeHash;
  };

  meta = {
    description = "Photo sharing for the fediverse";
    homepage = "https://example.org";
    license = lib.licenses.agpl3Plus;
    mainProgram = "example";
  };
})

The service

The component uses the options of the upstream NixOS module, services.<app>, and names the options of the operator in expose. The component adds what the module does not have, for example the connection to the hosting provider, the secrets and the backups. These items in the NixOS module make the component smaller:

A settings option that accepts each key of the upstream configuration.
A path to a file for each secret, so that the secret does not go into the Nix store.

For example, the NixOS module of lldap has a free-form settings option and the setting ldap_user_pass_file. A minimal module with these two items:

# The NixOS module of an application. `settings` accepts each key of the
# configuration file, and the secret comes from a file at run time, so that
# it does not go into the Nix store.
{
  config,
  lib,
  pkgs,
  ...
}:
let
  cfg = config.services.example;
  format = pkgs.formats.json { };
in
{
  options.services.example = {
    enable = lib.mkEnableOption "Example";

    package = lib.mkOption {
      type = lib.types.package;
      description = "The package of Example.";
    };

    settings = lib.mkOption {
      type = lib.types.submodule {
        freeformType = format.type;
        options.port = lib.mkOption {
          type = lib.types.port;
          default = 8080;
          description = "The port of the web server.";
        };
      };
      default = { };
      description = "The configuration file of Example. Each key of the file is valid here.";
    };

    secretKeyFile = lib.mkOption {
      type = lib.types.str;
      description = "The absolute path of a file that holds the secret key.";
    };
  };

  config = lib.mkIf cfg.enable {
    systemd.services.example = {
      wantedBy = [ "multi-user.target" ];
      environment.EXAMPLE_SECRET_KEY_FILE = "%d/secret-key";
      serviceConfig = {
        ExecStart = "${lib.getExe cfg.package} --config ${format.generate "example.json" cfg.settings}";
        LoadCredential = [ "secret-key:${cfg.secretKeyFile}" ];
        DynamicUser = true;
      };
    };
  };
}

Contracts

A contract connects the software that needs a service, such as mail or a database, to the software that gives it. If the NixOS module of the application asks for a service through a contract, the hosting provider can give the service without code for each application. If the module does not, the component connects the result of the contract to the options of the module, as Pixelfed does in smtp.nix. The contract reference lists the contract types, with an example for each type.

This part of the module of the example above asks for mail, and gives the result to the configuration and the unit:

# The part of the NixOS module of an application that asks for mail through
# the `smtp` contract. The module does not name a mail server: the provider
# that the host selects answers the request.
{ config, lib, ... }:
let
  cfg = config.services.example;
  smtp = config.contracts.smtp.results.example.default;
in
{
  options.services.example.mail.sender = lib.mkOption {
    type = lib.types.str;
    description = "The sender address of the mail that Example sends.";
  };

  config = lib.mkIf cfg.enable {
    contracts.smtp.want.example.default.request.sender = cfg.mail.sender;

    services.example.settings.smtp = {
      inherit (smtp) host port username;
      from = smtp.sender;
      password_file = "%d/smtp-password";
    };
    systemd.services.example.serviceConfig.LoadCredential = [
      "smtp-password:${smtp.passwordFile}"
    ];
  };
}

A JSON schema of the configuration

If the application publishes a JSON schema of its configuration, the form of the panel and the JSON schema of the API get the type, the default and the description of each setting from it. The audit of the options can then find each key of the configuration. PeerTube publishes a schema. For Mastodon, Pixelfed and lldap, the pin comes from a branch that adds a schema, which we sent upstream as a pull request. Without a schema, the form shows only the options that the NixOS module declares, and no key of a free-form settings option.

Make the schema from the code that reads the configuration, for example with a command such as config:schema of Pixelfed, and put it in the source of each release at a fixed path. Then the pin of the schema can have the same version as the package. For example, PeerTube puts config/config-schema.json in its source, and pixelfed/pixelfed#6555 adds the command to Pixelfed.

A schema for the example above. The build of this page checks that the settings of the example are valid for it:

{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "title": "The configuration file of Example",
  "type": "object",
  "properties": {
    "port": {
      "type": "integer",
      "minimum": 1,
      "maximum": 65535,
      "default": 8080,
      "description": "The port of the web server."
    },
    "registration": {
      "enum": ["open", "approval", "closed"],
      "default": "approval",
      "description": "Who can make an account."
    },
    "smtp": {
      "type": "object",
      "description": "The server that sends mail.",
      "properties": {
        "host": { "type": "string" },
        "port": { "type": "integer" },
        "username": { "type": "string" },
        "from": { "type": "string", "format": "email" },
        "password_file": { "type": "string" }
      }
    }
  }
}

An OpenTofu provider

The NixOS module sets the configuration files of an application. It does not set the state that the application keeps at run time, such as the users, groups, OAuth clients and other objects that an administrator makes through the API. If the application has an OpenTofu (or Terraform) provider, the declarative-runtime library can pair the provider with the NixOS module. It adds options that declare this state, and a unit that applies the state after the service starts. Fediversity uses it for Incus and OpenBao, in core/components/hosting-provider/incus/runtime/module.nix and core/components/hosting-provider/openbao/runtime/module.nix. The library has pairings for more services, for example Forgejo, Keycloak, Garage, lldap and PostgreSQL.

For example, with the lldap pairing, this configuration declares a user, a group and a membership:

# The state of lldap at run time: its users, groups and memberships. The
# lldap pairing of declarative-runtime applies it through the `tasansga/lldap`
# OpenTofu provider after `lldap.service` starts.
{
  services.lldap = {
    enable = true;
    settings = {
      ldap_base_dn = "dc=example,dc=org";
      ldap_user_pass_file = "/run/secrets/lldap-admin-password";
      force_ldap_user_pass_reset = "always";
    };
    runtime = {
      enable = true;
      user.alice = {
        email = "alice@example.org";
        display_name = "Alice";
        passwordFile = "/run/secrets/lldap-alice-password";
      };
      group.editors = { };
      member.alice_editors = {
        user_id = "alice";
        group_id = "editors";
      };
    };
  };
}

If the application has an API but no provider, a provider for it makes its state at run time declarative. Without a provider and an API, an administrator must set this state by hand.