Technical design, as one document

This page holds the technical design and each of its parts. Print the page from the browser to get one file.

Technical design

This page is the entry of the technical design of Fediversity. It gives the scope, the list of the parts, and links to the page that has the details of each subject. The other pages of this site, the site of the panel and the site of the API stay the source for their subjects. This page does not repeat them.

To get the technical design as one file, open the one-document page and print it from the browser.

Scope

Fediversity lets a hosting provider offer applications to its customers. These we call operator: a person or an organization that has a deployment with applications on its own domain. The glossary defines the terms and the actors.

The technical design covers five functions:

1. Install applications.
2. Configure applications.
3. Update applications.
4. Manage users.
5. Switch an operator to and from a different hosting provider.

The repository has three components, each with its own flake and its own documentation site:

api and panel use core. core does not use api or panel.

Parts of the technical design

Each page in core/docs/md/technical-design/ is one part of the technical design. The build fails when this list does not link a part.

• Access model: the persons and the components that have access to the state of the system, and the limits of that access.
• Architecture: the system context, the containers of an installation, and the components of each deployment.
• Capability flows: the run-time flow of each capability, with the code paths, the stores that change and the failure paths.
• Configuration data flow: how an operator configuration becomes a deployment.
• Contracts (RFC 189): how the contracts connect the software that needs a service to the software that gives it.
• Credentials register: where each credential is, what makes it, how long it lives and how it changes.
• Data subject requests: which party answers a request about personal data, and which mechanism the code gives for it.
• Deployment topology: the nodes of a deployment, where they run, the networks between them and the trust boundaries.
• The five functions: how the system installs, configures and updates applications, manages users, and switches an operator to a different hosting provider.
• Metering and accounting: what the system measures about a deployment, and where a hosting provider reads it.
• Logs, traces and audit records: each channel that records what the software and its users do, with its personal data, store and retention.
• Requirements of hosting providers and design review: the requirements of the hosting providers, the place in the design that answers each one, and the review of the design.
• Quality attributes: the target of each quality attribute, what the code does, and the test that shows it.
• State lifecycle: what happens to the state when the code changes, or when a service gets a new version.
• State register: each store of state, its owner, its personal data, its retention and its protection.
• Traceability of the capabilities: the requirements, the code, the tests and the docs pages of each capability.

Where to find the details

Access model

This part gives the persons and the components that have access to the state of the system, and the limits of that access. The tables that have the label "generated" come from the code at each build of the documentation (core/docs/md/technical-design/access-model.nix). The threats and the controls are in the threat model.

Actors

Permissions of each role

In the panel

The panel has three levels of access.

• A visitor without an account can read the public pages.
• An operator can read and change only the deployments for which Deployment.operator is the account of the operator (panel/src/panel/models.py). The views filter on this field.
• A superuser has the access of an operator. A superuser can also list, claim and assign domains for all operators, and can read the audit history of each account (reads_all_history and _AUTHZ_SUPERUSER_PROCEDURES in panel/src/panel/views.py).

The admin site of Django (/admin/) shows only the User model (panel/src/panel/admin.py). The views of the panel do not use is_staff. The directory groups fediversity-staff and fediversity-admins give no permission in the panel.

In the api

The api does not keep accounts and does not decide on access. For each request, the api sends the name of the procedure, the scope and the headers to the authorization callback of the panel (authorize in api/src/fediversity_api/authz.py, AuthzCallback in panel/src/panel/views.py). The callback accepts a request only with the callback token of the api (AUTHZ_CALLBACK_TOKEN), when that token is set. The panel identifies the requester from a bearer token that the panel signed, which stays valid for API_TOKEN_TTL seconds (panel/src/panel/tokens.py).

The callback denies a procedure that is not in AUTHZ_PROCEDURE_RULES. Without a callback URL, the api starts only when API_AUTHZ_ALLOW_ALL is set, and then it permits each request.

Directory groups of the hosting provider

The groups are in the lldap of the hosting provider. The section Manage users and groups of the hosting-provider guide tells how to give a group to a user.

When ADMIN_SOURCES is set, the firewall of each node accepts the administrative ports (port 22 by default) only from those addresses (core/resources/firewall/default.nix). An operator gets no shell access to the nodes of its deployment. The persons who can read each log, trace and audit channel are in Logs, traces and audit records.

Application roles

Each deployment of an operator has its own lldap, with the same group names. The lldap of a deployment has the groups application-admins and application-moderators (core/resources/ldap/application-role-groups.nix), and the groups of each application. Each application gives a role to the members of a group at log-in. The Grafana of a deployment accepts the members of fediversity-admins in the lldap of that deployment (core/components/operator/grafana/oidc.nix).

Generated from the role-groups.nix file of each application in core/components/operator/:

Access of each component to each store

The tables in this section are generated from the configuration of the nodes, with the groups of core/testers/groups.nix. That evaluation knows no deployment. Thus <own deployment> shows where a name has the namespace of the deployment of the node, and <each deployment> shows a path that applies to all deployments.

Secrets in the KV of OpenBao

Each node logs in to OpenBao with its SPIRE certificate and gets the policy host-<name>. The entries of the hosting provider are below secret/data/vars/. The entries of a deployment are below secret/data/vars/operator-deployments/<namespace>/. Each entry has a public half and a secret half. The credentials register gives the lifetime and the rotation of each credential. The table gives the name of the entry when a policy has the same capabilities on the two halves, and <half>/<entry> when it has them on one half. The table gives a grant on secret/metadata/ only when the policy has no grant on the data of the same entry.

The policy fediversity-admins in this table is that of the evaluation, where ADMIN_UI_WRITE is not set.

Other paths in OpenBao

These paths give certificates (pki), database passwords (database, postgresql), directory passwords (ldap, ldap-operator) and child tokens.

PostgreSQL

The database engine of OpenBao owns the password of each role and changes it at the interval in the table. A node reads the password of its role from OpenBao (core/setups/wiring/postgresql-roster.nix). The database and the role of a deployment have the namespace of the deployment as a suffix, and the OpenTofu graph of the deployment creates and drops them (core/effects/tf/incus/operator/postgresql.tf).

The postgresql node also has these group roles:

Two actors have access to all databases of the node: the front-end deployer, which reads the password of the postgres superuser (the entry secret/postgresql-admin in the KV table), and a member of fediversity-admins through pgweb.

Buckets in Garage

Each row is one ensureAccess entry of a key that a node requests. owner lets the key change the configuration of the bucket.

How the system keeps the tenants apart

• Nodes. Each deployment has its own nodes for each application and for valkey, authelia, smtp, lldap and grafana (core/setups/groups/operator.nix). The users, the sessions and the cache of a deployment are thus not in a store that a different deployment uses.
• Network. All nodes are on one Incus bridge. Each deployment gets a /28 of its own, and an Incus network ACL on each NIC drops the traffic to and from the other deployments (core/effects/tf/incus/operator/bridge-isolation.tf). The Incus host declares the ACL of each /28 (core/components/hosting-provider/incus-bridge-acls.nix), thus the certificate of a deploy runner is restricted to the Incus project of the deployments and cannot change an ACL. Each NIC has MAC and IP address filtering (core/effects/tf/incus/instance.tf). A second ACL and the firewall of each node of the hosting provider permit only the ports that a deployment uses (core/components/hosting-provider/operator-reach.nix). The second ACL holds only the IPv4 addresses of the nodes of the hosting provider. The bridge has no IPv6 subnet (ipv6.address = none in core/components/hosting-provider/incus.nix), thus Incus drops all IPv6 on eth0. A node of a deployment sends IPv6 only on its public eth1, which has no NAT, thus the firewall of the destination node sees the real source address and applies the port limit.
• Secrets. The policy of a node of a deployment names the KV paths with a template that OpenBao replaces with the namespace from the certificate of the node. Thus a node reads only the entries of its own deployment (core/components/hosting-provider/openbao/spire-auth.nix).
• Databases. All deployments use the PostgreSQL node of the hosting provider. Each deployment has its own database and role for each application, and a node can read only the password of the role that has its own namespace.
• Buckets. All deployments use the Garage cluster of the hosting provider. Each bucket and key has the namespace of the deployment as a prefix (tenant in core/resources/garage/default.nix), and a key has access only to the buckets in the table above.
• Metrics. The query path for the metrics of an application adds the deployment_id label to each query (core/components/hosting-provider/metrics-query.nix).
• Panel and api. An operator gets only the deployments that it owns (see In the panel).

These components are common to all tenants, and the separation there is weaker:

• The front-end deployer is one identity for all deployments. Its OpenBao policy can write the entries of each deployment that the KV table gives, and it can read the password of the postgres superuser. The state backend has one user operator for the states of all deployments (core/components/hosting-provider/terraform-backend/front.nix).
• The Grafana of the hosting provider has one organization, thus fediversity-staff and fediversity-admins can read the telemetry of all deployments.
• A member of fediversity-admins can read the secrets, the databases and the containers of all deployments.

Architecture

This part shows the system context of Fediversity and the containers of one installation of a hosting provider. It also shows the components of each deployment, with the services that each component requests and provides.

System context

flowchart TB
  operator["<b>Operator</b><br/>[Person]<br/>Has a deployment with applications"]
  user["<b>Application user</b><br/>[Person]<br/>Uses an application of an operator"]
  provider["<b>Hosting provider</b><br/>[Person]<br/>Runs the infrastructure and manages the accounts"]
  maintainer["<b>Maintainer</b><br/>[Person]<br/>Changes and releases the code"]

  fediversity["<b>Fediversity</b><br/>[Software system]<br/>Installs, configures and updates applications, manages users, moves deployments between hosting providers"]

  registrar["<b>Domain registrar</b><br/>[External system]<br/>OpenProvider, INWX or Porkbun"]
  dns["<b>DNS service</b><br/>[External system]<br/>Authoritative DNS that is not in the deployment, for example deSEC or PowerDNS"]
  acme["<b>ACME certificate authority</b><br/>[External system]<br/>Issues TLS certificates"]
  relay["<b>Mail relay</b><br/>[External system]<br/>SMTP submission service of a third party"]
  upstream["<b>Upstream sources</b><br/>[External system]<br/>nixpkgs and the other pinned sources"]
  other["<b>Other hosting provider</b><br/>[Software system]<br/>A different installation of Fediversity"]

  operator -->|"configures and deploys, HTTPS"| fediversity
  user -->|"uses the applications, HTTPS"| fediversity
  provider -->|"configures, deploys and monitors"| fediversity
  maintainer -->|"releases code and pins"| fediversity
  fediversity -->|"registers and transfers domains"| registrar
  fediversity -->|"publishes DNS records"| dns
  fediversity -->|"requests certificates"| acme
  fediversity -->|"sends mail"| relay
  fediversity -->|"gets packages and modules"| upstream
  fediversity -->|"exports and imports a deployment"| other

  classDef external stroke-dasharray: 5 5
  class registrar,dns,acme,relay,upstream,other external

Each external system is a selection of the hosting provider, not a fixed part:

• Registrars: core/effects/tf/registrars.nix has one entry for each registrar, and the steps that each registrar can do.
• DNS: the dns resource has a provider for a BIND node in the deployment (core/resources/dns/self-hosted.nix) and a provider for a service that the hosting provider already has (core/resources/dns/external.nix). The core/effects/octodns effect publishes the records. Choose where the DNS zones are gives the selection.
• Certificates: the ssl resource has an ACME provider, an OpenBao PKI provider and a self-signed provider (core/resources/ssl/).
• Mail: the smtp resource has a provider that runs a mail server in the deployment and a provider for an external relay (core/resources/smtp/).
• Upstream sources: npins/sources.json pins nixpkgs and each other source. The SBOM page tells how to find the software of a node.

The threat model gives the trust boundaries between these parties, and data protection gives the parties that get personal data.

Containers

The diagram shows the parts of one installation of a hosting provider. The architecture page of the panel has the container and component views of panel, and the site of the API has those of api.

flowchart TB
  browser["<b>Browser of the operator</b><br/>[Person and browser]"]

  subgraph frontend["Front end"]
    panel["<b>panel</b><br/>[Container: Django]<br/>Accounts, ownership of deployments, configuration form"]
    panelDb[("<b>Panel database</b><br/>[Container: PostgreSQL or SQLite]")]
  end

  subgraph apiSide["API"]
    api["<b>api</b><br/>[Container: FastAPI]<br/>Validates, queues builds, runs deployments"]
    apiDb[("<b>API state store</b><br/>[Container: PostgreSQL or SQLite]")]
    centrifugo["<b>Centrifugo</b><br/>[Container]<br/>Sends progress events"]
    jobs["<b>Job runner</b><br/>[Container: subprocess or Windmill]<br/>Runs a build or a deployment"]
  end

  subgraph coreSide["core"]
    programs["<b>Deployment programs</b><br/>[Container: Nix and OpenTofu]<br/>Change a configuration into machines, records and secrets"]
    schema["<b>Schema and catalog</b><br/>[Container: Nix]<br/>JSON Schema of the options, list of applications"]
  end

  subgraph providerSide["Deployment of the hosting provider"]
    services["<b>Shared services</b><br/>[Containers]<br/>See the diagrams below"]
    incus["<b>Incus hypervisor</b><br/>[Container]<br/>Runs the machines of the deployments"]
  end

  subgraph operatorSide["Deployment of an operator"]
    apps["<b>Applications and their services</b><br/>[Containers]<br/>See the diagrams below"]
  end

  browser -->|"HTTPS"| panel
  browser -->|"HTTPS, bearer token"| api
  centrifugo -->|"WebSocket"| browser
  panel --> panelDb
  api -->|"authorization callback, HTTPS"| panel
  api --> apiDb
  api -->|"publishes events"| centrifugo
  api -->|"reads"| schema
  api -->|"starts a job"| jobs
  jobs -->|"runs"| programs
  programs -->|"creates and changes machines"| incus
  programs -->|"reads and writes state, secrets and addresses"| services
  incus -->|"runs"| apps
  apps -->|"use"| services

Notes for the diagram:

• api runs a deployment program as a nix subprocess, directly or in a Windmill job (api/src/fediversity_api/_runner.py, api/src/fediversity_api/deployment.py).
• api and panel are also components of the data model (api/nix/component/, panel/nix/component/), thus the deployment programs of core can deploy them the same as each other service.
• Configuration data flow shows how a configuration becomes a deployment in core.
• Flake attributes lists the deployment programs.

Components of a deployment of an operator

The build makes the two diagrams and the two tables below from the code. They show the components of the operator group and of the hosting-provider group of core/setups/groups, each with its default settings. A node shows the resources that the component provides (wiring.provides in core/components/wiringProperties.nix). An arrow goes from a component to the component that answers its request, and its label gives the resources. A request goes to a component of the same group. If the group has no such component, the request goes to the component of the hosting-provider group.

Two requests have a name that is different from the name of the backend that answers them: a generateFiles request goes to varsBackend, and a garage request goes to objectStore. A request without an arrow has no service component that provides it. Examples are nixos-module and firewall, which change the machine itself, and ssl, which has three providers. Hosting resources describes each resource.

flowchart LR
  subgraph applications["Applications that the operator selects"]
    applications_mastodon["<b>mastodon<br/>peertube<br/>pixelfed</b>"]
  end
  subgraph own["Service components of the operator group"]
    own_authelia["<b>authelia</b><br/>provides: oidc"]
    own_grafana["<b>grafana</b>"]
    own_lldap["<b>lldap</b><br/>provides: ldap"]
    own_smtp["<b>smtp</b><br/>provides: smtp"]
    own_valkey["<b>valkey</b><br/>provides: redis"]
  end
  subgraph central["Service components of the hosting-provider group"]
    central_garage["<b>garage</b><br/>provides: objectStore"]
    central_openbao["<b>openbao</b><br/>provides: varsBackend, pki"]
    central_postgresql["<b>postgresql</b><br/>provides: postgresql"]
    central_spire["<b>spire</b><br/>provides: spiffe"]
  end
  applications_mastodon -->|"garage"| central_garage
  applications_mastodon -->|"generateFiles"| central_openbao
  applications_mastodon -->|"postgresql"| central_postgresql
  applications_mastodon -->|"spiffe"| central_spire
  applications_mastodon -->|"oidc"| own_authelia
  applications_mastodon -->|"ldap"| own_lldap
  applications_mastodon -->|"smtp"| own_smtp
  applications_mastodon -->|"redis"| own_valkey
  own_authelia -->|"garage"| central_garage
  own_authelia -->|"generateFiles"| central_openbao
  own_authelia -->|"postgresql"| central_postgresql
  own_authelia -->|"spiffe"| central_spire
  own_authelia -->|"redis"| own_valkey
  own_grafana -->|"generateFiles"| central_openbao
  own_grafana -->|"spiffe"| central_spire
  own_grafana -->|"oidc"| own_authelia
  own_lldap -->|"garage"| central_garage
  own_lldap -->|"generateFiles"| central_openbao
  own_lldap -->|"postgresql"| central_postgresql
  own_lldap -->|"spiffe"| central_spire
  own_lldap -->|"oidc"| own_authelia
  own_smtp -->|"spiffe"| central_spire
  own_valkey -->|"spiffe"| central_spire
  own_valkey -->|"oidc"| own_authelia

Components of the deployment of the hosting provider

flowchart LR
  subgraph own["Service components of the hosting-provider group"]
    own_authelia["<b>authelia</b><br/>provides: oidc"]
    own_bind["<b>bind</b>"]
    own_garage["<b>garage</b><br/>provides: objectStore"]
    own_incus["<b>incus</b><br/>provides: instances"]
    own_lldap["<b>lldap</b><br/>provides: ldap"]
    own_netbox["<b>netbox</b><br/>provides: netbox"]
    own_openbao["<b>openbao</b><br/>provides: varsBackend, pki"]
    own_otel_collector["<b>otel-collector</b><br/>provides: otel"]
    own_overview["<b>overview</b><br/>provides: uiIndex"]
    own_postgresql["<b>postgresql</b><br/>provides: postgresql"]
    own_smtp["<b>smtp</b><br/>provides: smtp"]
    own_spire["<b>spire</b><br/>provides: spiffe"]
    own_telemetry["<b>telemetry</b><br/>provides: otelBackend"]
    own_terraform_backend["<b>terraform-backend</b><br/>provides: tfState"]
    own_valkey["<b>valkey</b><br/>provides: redis"]
    own_windmill["<b>windmill</b>"]
  end
  own_authelia -->|"garage"| own_garage
  own_authelia -->|"generateFiles"| own_openbao
  own_authelia -->|"postgresql"| own_postgresql
  own_authelia -->|"spiffe"| own_spire
  own_authelia -->|"redis"| own_valkey
  own_bind -->|"garage"| own_garage
  own_bind -->|"generateFiles"| own_openbao
  own_bind -->|"spiffe"| own_spire
  own_garage -->|"oidc"| own_authelia
  own_garage -->|"spiffe"| own_spire
  own_incus -->|"oidc"| own_authelia
  own_incus -->|"generateFiles"| own_openbao
  own_incus -->|"spiffe"| own_spire
  own_lldap -->|"oidc"| own_authelia
  own_lldap -->|"garage"| own_garage
  own_lldap -->|"generateFiles"| own_openbao
  own_lldap -->|"postgresql"| own_postgresql
  own_lldap -->|"spiffe"| own_spire
  own_netbox -->|"oidc"| own_authelia
  own_netbox -->|"garage"| own_garage
  own_netbox -->|"generateFiles"| own_openbao
  own_netbox -->|"otel"| own_otel_collector
  own_netbox -->|"postgresql"| own_postgresql
  own_netbox -->|"smtp"| own_smtp
  own_netbox -->|"spiffe"| own_spire
  own_openbao -->|"oidc"| own_authelia
  own_openbao -->|"garage"| own_garage
  own_openbao -->|"spiffe"| own_spire
  own_otel_collector -->|"generateFiles"| own_openbao
  own_otel_collector -->|"spiffe"| own_spire
  own_overview -->|"spiffe"| own_spire
  own_postgresql -->|"oidc"| own_authelia
  own_postgresql -->|"garage"| own_garage
  own_postgresql -->|"generateFiles"| own_openbao
  own_postgresql -->|"spiffe"| own_spire
  own_smtp -->|"spiffe"| own_spire
  own_telemetry -->|"oidc"| own_authelia
  own_telemetry -->|"garage"| own_garage
  own_telemetry -->|"generateFiles"| own_openbao
  own_telemetry -->|"smtp"| own_smtp
  own_telemetry -->|"spiffe"| own_spire
  own_terraform_backend -->|"garage"| own_garage
  own_terraform_backend -->|"generateFiles"| own_openbao
  own_terraform_backend -->|"spiffe"| own_spire
  own_valkey -->|"oidc"| own_authelia
  own_valkey -->|"spiffe"| own_spire
  own_windmill -->|"oidc"| own_authelia
  own_windmill -->|"garage"| own_garage
  own_windmill -->|"netbox"| own_netbox
  own_windmill -->|"generateFiles"| own_openbao
  own_windmill -->|"otel"| own_otel_collector
  own_windmill -->|"postgresql"| own_postgresql
  own_windmill -->|"smtp"| own_smtp
  own_windmill -->|"spiffe"| own_spire

api and panel are not in this table, because their component definitions are outside core.

Capability flows

This part gives the run-time flow of each capability: install, configure and update an application, manage users, and switch to and from a different hosting provider. Each flow has a sequence diagram, the code paths, the stores that change, the failure paths, and the work that is planned.

The flows have these participants:

• The operator is the customer of the hosting provider. The operator uses a browser.
• panel is the Django front-end (panel/). It keeps the accounts, the stored configurations and the permissions.
• api is the FastAPI service (api/). It validates a configuration and starts a deploy.
• core is the Nix and OpenTofu code (core/). It makes the schema, and it deploys.

The design of the API gives the contract of each call. The architecture of the panel gives the design of the panel.

Calls that each flow uses

Each call from the browser to api uses the same permission check:

1. The browser gets a bearer token from panel at api/token/ (views.ApiToken, panel/src/panel/tokens.py). panel signs the token with its SECRET_KEY. The token is valid for API_TOKEN_TTL.
2. The browser sends the call to api with the token.
3. api does not examine the token. It sends the procedure name, the scope and the headers to the callback of panel at api/authz/ (api/src/fediversity_api/authz.py, views.AuthzCallback).
4. panel answers allow, and for an allowed call the subject and the deployments that the caller can use. panel denies a procedure that has no rule in AUTHZ_PROCEDURE_RULES.
5. If panel denies the call, api answers authz_denied (HTTP 403).

The diagrams show this check as one step, "check the permission".

Install an application

The operator fills in the form of a deployment, enables an application, and deploys. The same call does the first install and each later change.

sequenceDiagram
    actor O as Operator
    participant P as panel
    participant A as api
    participant C as core
    participant I as Provider infrastructure
    participant N as Node
    O->>P: save the form
    P->>P: store the configuration
    O->>A: POST /deployments/submit/{id}
    A->>P: check the permission
    A->>C: build the schema of the module source
    A->>A: migrate renamed options, validate
    A->>C: evaluate the names of the deployment
    A->>A: claim the domain, store the build as queued
    A-->>O: build_id, status queued, position
    A->>C: run tf-run.sh
    C->>I: NetBox address, Incus instance and volume
    C->>I: bucket, database, secrets in OpenBao
    C->>I: hold the zone, register the domain, publish the records
    C->>N: switch to the NixOS configuration
    N->>N: order certificates with ACME, start the services
    C->>N: read the state of the units
    C-->>A: exit code and outputs
    A-->>O: build events through Centrifugo

Code paths:

• panel: views.DeploymentForm shows only the deployments of the operator. static/deployment_submit.js saves the form in panel, then calls api.
• api: the route deployment_submit (routes/deployments.py) calls procedures.deployment_submit. In sequence, this function:
  1. refuses a module source that is not on the trusted-source allowlist (source_allowlist.py),
  2. builds the schema and the rename table of the module source (schema_build.py),
  3. applies the renames and validates the configuration with jsonschema,
  4. takes the secrets of the operator out of the configuration (operator_secrets.py),
  5. refuses a full queue,
  6. evaluates the names of the deployment and claims the apex of the domain (apex_claims.py),
  7. makes the DNS token of the zone (zone_tokens.py), writes the secrets of the operator to OpenBao, and stores the deployment (state.upsert_deployment),
  8. starts the build (deployment.start_deployment).
• The build runs as a subprocess of api, or as a Windmill job when windmill_base_url is set (deployment.py). The runner starts nix develop .#<attribute> --impure --command tf-run.sh (_runner.py).
• core: tf-run.sh (core/effects/common/lib.nix, core/effects/tf/common/bootstrap-apply.sh) applies the OpenTofu graph of core/effects/tf/incus/operator/, then gives the outputs as JSON. The state of the graph is in the HTTP state backend.
• DNS: dns-lane-zone.tf and registrar.tf hold the zone, register the domain and wait until the parent zone delegates. dns-lane-records.tf publishes the names of each node and waits until the names resolve. All of this occurs before the switch of the node, because the certificate order of the node needs the records.
• TLS: core/resources/ssl/acme.nix sets security.acme. A node orders the certificate of an ordinary name with the HTTP challenge, and the certificate of a wildcard name with the DNS-01 challenge.
• After the switch, null_resource.deploy_verdict reads the units of each node. A node with a unit that continues to restart fails the deploy.

Stores that change:

Failure paths:

• A configuration that fails the schema gives invalid_configuration (HTTP 400). Nothing is stored.
• A full queue gives queue_full (HTTP 409). A domain that a different deployment holds gives conflict (HTTP 409).
• A schema build that passes its time limit gives build_timeout (HTTP 504).
• A deploy that fails sets the build to failed. api publishes the end of the log. The resources that the apply made stay in the state, thus the next submit continues from them.
• When api starts again, reconcile_orphaned_builds (deployment.py) marks the builds that lost their process.

Planned:

• Builds of one deployment that run in parallel (issue 242 of fediversity/fediversity).
• The configuration of the deploy method in the panel (issues 723 and 1071).

Configure an application

The form of an application comes from the options of its NixOS module. No form is written by hand.

sequenceDiagram
    actor O as Operator
    participant F as Form in the browser
    participant P as panel
    participant A as api
    participant C as core
    O->>P: open the form of the deployment
    P-->>F: stored, deployed and default module source
    F->>A: GET /schema and /schema/migrations
    A->>C: build provider-configuration-schema and configuration-renames
    C-->>A: JSON Schema and rename table
    A-->>F: JSON Schema and rename table
    F->>F: apply the renames, show the form
    O->>F: change a value
    F->>F: validate with AJV
    O->>F: deploy
    F->>P: save the configuration
    F->>A: POST /deployments/submit/{id}
    A->>A: apply the renames, validate with jsonschema
    A->>C: run tf-run.sh

Code paths:

• Schema: core/lib/json-schema.nix makes a JSON Schema from the options of frontendConfiguration (core/lib/data-model.nix). Add an application and the data model give the options.
• api: schema_build.py builds the outputs provider-configuration-schema and configuration-renames of the module source. A build has a time limit, and api limits the number of builds that run together. api keeps the result for each source in memory. The routes are in routes/schema.py.
• Form: panel/forms-island-adapter/panel-adapter.ts gets the schema and the rename table, validates each change with AJV, and shows the difference between the schema of the stored source and the schema of the source that is deployed.
• Validation occurs two times: in the browser with AJV, and in api with jsonschema (procedures._checked_submission). The check in api is the authority.
• Apply: the apply is the submit of the install flow. A changed option changes the NixOS configuration of a node, and the deploy switches the node to it.

Stores that change: the stored configuration in panel, then the stores of the install flow.

Failure paths:

• A module source that is not on the allowlist is refused before a build.
• A configuration that fails the schema gives invalid_configuration.
• POST /deployments/preview/{id} does the checks of the submit and shows the nodes that a submit deploys. It deploys nothing.

Planned:

• Resource versions, which let api find a difference between the environments of the provider and the pinned data model before a deploy (issue 634).
• To follow the new revisions of a fork of the module source (issue 3053).

Update an application

An update is a submit with a newer module source. The module source selects the revision of the application modules.

sequenceDiagram
    actor O as Operator
    participant P as panel
    participant A as api
    participant C as core
    participant N as Node
    participant S as Object store
    alt policy manual
        P-->>O: notice that a newer revision exists
        O->>A: POST /deployments/submit/{id} with the update applied
    else policy automatic or window
        A->>A: the scheduler submits the stored configuration again
    end
    A->>A: resolve the null source to the revision of the provider
    A->>A: apply the renames, validate
    A->>C: run tf-run.sh
    C->>N: switch to the new NixOS configuration
    N->>N: guard reads the recorded version
    alt downgrade or skipped release series
        N-->>C: the guard fails, the service does not start
    else upgrade
        N->>N: stop the service, dump the database
        N->>S: restic snapshot of the dump
        N->>N: start the service, run the migrations after the deployment
        N->>N: record the new version
    end
    C->>N: read the state of the units

Code paths:

• Policy: api/src/fediversity_api/update_policy.py has the policies manual, automatic and window. The operator sets the policy with PUT /deployments/update_policy/{id} (views.DeploymentUpdatePolicy, static/update_policy.js). static/update_notice.js shows that a newer revision exists. The form has a check box beside the deploy button: when it is off, the submit has apply_update set to false (static/deployment_submit.js) and api keeps the deployed revision.
• Scheduler: automatic_updates.run_scheduler submits the stored configuration with a null module source. It tries each revision of the provider one time for each deployment (state.claim_update_attempt), and writes the audit action automatic_update. It does not update a deployment that is suspended or not active. A deployment whose deployed module source is not the source of the provider is due too (run_due_updates), thus the scheduler moves a deployment from a fork to the source of the provider.
• Version boundary: version_boundary.py selects, for the layers core/resources and core/effects, the newer of the pinned revision and the revision of the provider.
• Guards: each application has a systemd.stateMigrations entry. core/components/operator/mastodon/migrations.nix records the version, refuses a downgrade or a skipped minor release series, dumps the database before an upgrade, and runs the migrations after the services are up again. The guards of PeerTube and Pixelfed refuse a downgrade and dump the database before an upgrade. core/setups/nixos/postgresql.nix guards the major version of PostgreSQL and uses pg_upgrade.
• Backup before the upgrade: core/components/shared/database-upgrade-backup.nix sends the dump to the restic repository <server>-upgrade in the backup bucket of the application. The repository keeps the snapshots of the 2 newest upgrades. Without an object store, the dump stays on the node.

Stores that change: the stores of the install flow, the database of the application (schema migrations), the version file of each guard on the node, and the restic repository of the upgrade backups.

Failure paths and rollback:

• A guard that fails keeps the service down. deploy_verdict then fails the deploy.
• The deploy method k8s sets a pod that does not become ready back to the previous image (kubectl rollout undo in core/effects/k8s/effect.nix) and fails the deploy. The state on the volume of the pod stays as it is. No other code path rolls back a deploy automatically.
• A guard refuses a downgrade, thus a submit of the older revision does not go back after a migration of the data.
• The way back is the dump from before the upgrade. The hosting provider guide gives the restore procedures.
• A submit with apply_update set to false uses the module source of the last build that was deployed.

Planned:

• Application versions that do not depend on the revision of the provider (issue 304).
• Resource versions (issue 634).
• An automatic rollback of a failed update, data included (issue 3052).
• A guide for the operator through the options that an update adds or changes (issue 3054).

Manage users

There are two levels of users: the account of the operator in panel, and the users of each deployment. Each deployment has its own directory (lldap at users.<domain>) and its own identity provider (Authelia at auth.<domain>).

sequenceDiagram
    actor O as Operator
    actor U as User of an application
    participant P as panel
    participant IdP as Identity provider of the hosting provider
    participant L as lldap of the deployment
    participant Au as Authelia of the deployment
    participant K as OpenBao KV
    participant App as Application
    O->>P: log in
    P->>IdP: OpenID Connect authorization
    IdP-->>P: identity of the operator
    O->>L: add a user, put the user in a group
    U->>App: log in
    App->>Au: OpenID Connect authorization
    Au->>L: check the password and the groups
    Au-->>App: identity of the user
    L->>K: timer writes the members of the role groups
    App->>K: timer reads the members
    App->>App: set the role of each member

Code paths:

• Log-in to panel: panel/src/panel/oidc.py uses OpenID Connect when OIDC_ISSUER, OIDC_CLIENT_ID and OIDC_CLIENT_SECRET_FILE are all set (settings.py). If not, panel uses the log-in of Django. A wallet log-in (panel.wallet.WalletAuth) opens only an account that linked the wallet.
• Directory: core/components/shared/lldap and core/components/shared/authelia. core/resources/ldap/lldap-provider.nix makes the first admin user.
• Single sign-on: core/resources/oidc/authelia-oidc-provider.nix makes an OpenID Connect client for each application. allowedGroups becomes an Authelia policy that allows only the members of those groups. The application reads its client from the oidc contract (for example core/components/operator/mastodon/oidc.nix).
• Groups to roles: core/resources/ldap/application-role-groups.nix and the role-groups.nix of each application declare the role groups, for example mastodon-owners, mastodon-admins and mastodon-moderators. core/resources/ldap/publish-roles.nix writes the members of the role groups to the OpenBao KV on a timer, only when they change. The ldap-roles.nix of each application reads the members on a timer and sets the roles in the application.
• Legal roles gives the role of each group in each application. The hosting provider guide and the user guide of the panel give the procedure.

Stores that change: the database of lldap, the members entry in the OpenBao KV, and the accounts and roles in the database of each application. The account of the operator is in the database of panel.

Failure paths:

• A role sync that fails does not fail a deploy. The timer tries again at its next interval.
• A role change is not immediate: the publish and the sync each run on a timer.
• Without the OpenBao backend for secrets, the role sync does nothing.
• A user who is not in an allowed group cannot log in to the application.

Planned: api has no route, and panel has no view, that changes the users of a deployment. The operator uses the web interface of lldap. Issue 3055 asks for the decision if the panel must manage users.

Switch hosting provider

A switch has two parts: the export at the source provider, and the import at the target provider. The migration string moves between them. The operator carries the string.

sequenceDiagram
    actor O as Operator
    participant SA as api of the source
    participant SN as Nodes of the source
    participant S3 as Object store of the source
    participant R as Registrar
    participant TA as api of the target
    participant TC as core of the target
    participant TN as Nodes of the target
    O->>SA: GET /migration-string/{id}
    SA->>R: read the auth code, unlock the domain, remove the DNSSEC keys
    SA->>SN: stop the application services
    SN->>S3: restic snapshot of each application
    SA->>SA: mark the deployment as exported, submit suspend
    SA-->>O: migration string
    O->>TA: POST /migration-import/{id} with the string
    TA->>TA: check the module source, remove the provider settings of the source
    TA->>TC: dry evaluation
    TA->>TA: write the snapshot pointers and the auth code to OpenBao
    TA->>TC: run tf-run.sh
    TC->>R: transfer the domain with the auth code
    TC->>TC: hold the zone, wait for the delegation, publish the records
    TC->>TN: switch to the NixOS configuration
    TN->>S3: restore each application from its snapshot
    TC->>TC: delete the pointers and the auth code
    SA->>TN: zone transfer, copy the records to the old zone

Code paths:

• Panel: static/migration_export.js calls GET /migration-string/{id} and shows the string. static/migration_import.js makes a deployment and sends the string to POST /migration-import/{id}.
• Domain: migration.migration_export (api/src/fediversity_api/migration.py) first calls release_domain, which reads the auth code at the registrar, unlocks the domain and removes its DNSSEC keys. This occurs before the snapshot, thus a registrar that fails stops no service.
• Export: migration_export then runs core/effects/migration/run.sh. This script stops the services of each node with migration-quiesce (core/components/shared/migration-quiesce.nix), then makes a snapshot of each backup unit. Then api marks the deployment as exported and submits the action suspend.
• Import: migration.migration_import parses the string, removes the settings that belong to the source provider, and changes the configuration to the shape of the target. import_check.py evaluates the result with DRY_RUN=1 before api stores it. Then the import is the submit of the install flow.
• Restore: migration_kv.py writes the snapshot pointers and the auth code to the OpenBao KV of the deployment. The OpenTofu graph gets only the snapshot references. core/effects/tf/incus/operator/restore-migration.sh reads the pointers, restores each unit, and deletes the pointers after the restore.
• Domain transfer: registrar.tf gives the auth code to openprovider_domain.operator, then null_resource.drop_domain_transfer deletes the code from the KV.
• DNS change: the deploy at the target holds the zone on the nameservers of the target (dns-lane-zone.tf). await_delegation waits until the parent zone delegates to them. Then the deploy publishes the records of the nodes.
• Zone follow: when the source holds the zone at deSEC, zone_follow.py at the source copies the records of the target to the old zone until the old delegation is out of the caches of resolvers. It asks the nameservers of the target for a zone transfer that it signs with the transfer_key of the migration string.

The hosting provider guide gives the procedure of the export and of the restore. The contract templates give the switching clauses.

Stores that change:

Trust between the two providers

• The two providers have no direct connection for the import, and no common account. The operator carries the migration string. The string is a bearer capability: its holder can read all data of the deployment.
• The target does not trust the configuration of the source. It validates the configuration against the schema, and accepts only a module source that is on its trusted-source allowlist.
• The target removes the provider settings of the source from the string (operator_settings, RUNTIME_SETTING_KEYS).
• The target reads the snapshots directly from the object store of the source, with the credentials of the string.
• The target allows a zone transfer from the addresses in transfer_sources, to the holder of the transfer_key.
• The string carries no registrar login.

Secrets that move

The migration string carries:

• for each backup unit, the S3 endpoint, bucket, access key and secret key of the snapshot repository, the password of the repository, and the snapshot reference,
• the keys that decrypt the data of the applications,
• the auth code of the domain, when the source has the domain in its registrar account,
• the TSIG key of the zone follow.

api does not keep the string, and does not write it to a log or to an audit event. At the target, the pointers and the auth code are in the OpenBao KV of the deployment, and not in the OpenTofu state or in the Windmill job. The deploy deletes them after their use. The threat model and data protection give the related risks and duties.

Downtime

The applications are down from the export until the target restores them. The export stops the services before the snapshot, thus no write is lost. The time has these parts: the snapshot, the transport of the string by the operator, the deploy at the target, the restore, and the DNS change. The code sets no limit on the full time, and the tree has no measured value. The limits that the code does set are:

• the dry evaluation of the import: 1800 seconds, then build_timeout,
• the cache time that the zone follow uses: at most 604800 seconds (TTL_CEILING in zone_follow.py).

To roll back, resume the source deployment: its data stays, and its services start again. After a domain transfer, the resume of the source fails.

Different versions at the two providers

• The migration string must name its module source. An import without a module source is refused, because the revision of the target can be different from the revision of the source.
• The target builds the schema of that module source, and applies its rename table to the configuration.
• The target deploys with the version boundary of the update flow: the pinned revision for the application modules, and the newer revision for core/resources and core/effects.
• The dry evaluation refuses a configuration that the pinned revision cannot deploy with the method and the environments of the target. This occurs before api stores the deployment.
• A source node that runs an older revision does not give the keys of its applications. The export continues, and the string has no keys for that node.

Failure paths:

• If a snapshot or the suspend fails, the export starts the services again and the deployment is not exported. If a snapshot fails, the export also locks the domain again.
• If api cannot read the auth code, the export fails and does not change the source.
• A second export of a suspended deployment gives conflict.
• A string in an old form, or with a module source that is not on the allowlist, gives invalid_configuration.
• If the restore fails, the deploy fails. The pointers stay in the KV, thus a new deploy restores again.
• A domain transfer that the registry refuses shows as transfer_failed in GET /domains.

Planned:

• Resource versions, which let the target reconcile a difference in the environments of the two providers (issue 634).
• The domain calls need the registrar command of api, which exists for openprovider. A second registrar is planned (issue 1908).

Configuration data flow

This page tells how an operator configuration becomes a deployment. The data model and hosting resources pages are the option references for the same parts. The concepts page defines the terms.

From a configuration to a deployment

flowchart
  subgraph configuration
    application-config
  end
  subgraph resource[resource module]
    request
    policy
  end
  subgraph application[application module]
    application-options{{application-options}} --> application-config
    application-config --> config-mapping
    request{{request}} --> config-mapping
    config-mapping(config-mapping)
  end
  subgraph fediversity[fediversity setup]
    config-mapping -->|required resources| resource-mapping(resource-mapping)
    policy{{policy}} --> provider-config -->|available resources| resource-mapping -->|configuration| deployment{deployment}
  end
  maintainer((maintainer)) -->|curates| resource
  contributor((developer)) -->|curates| application
  operator((operator)) -->|enters| configuration
  hosting-provider((hosting provider)) -->|maintains| fediversity

In the diagram, a circle is an actor, a box with angled sides is a type, a rectangle is a value, a box with round corners is a function, and a diamond is a state. An arrow points to the part that uses the other part. An application module is a component that the operator can select.

1. The operator selects the applications and sets their options in the panel. The panel form comes from frontendConfiguration, which has only the operator-facing components. The result is a configuration.
2. Each enabled component maps its options to the resources that it needs. The implementation of a component, a config-mapping, returns one request for each resource, keyed by an instance name. For example, Forgejo requests a TLS certificate as "tls".ssl, its secrets as "age".secrets, and outbound mail as "forgejo-smtp".smtp.
3. The hosting provider declares an environment. Its resources set the policy of each resource: which provider gives the resource, and with which limits.
4. The implementation of the environment, a resource-mapping, collects the requests of all components. For each resource, the process function of the policy turns the requests into NixOS modules or OpenTofu resources.
5. The result is a deployment for an effect, for example a set of Incus containers that OpenTofu creates and NixOS configures.

environments.<env>.deployment does all of these steps for one deployment.

Resources

A resource is the interface between a component and a hosting provider. The component sets the request, and the hosting provider sets the policy. The request/policy resource pattern on the data model page tells more, and the hosting resources page lists the request options of each resource.

A resource is not a contract of RFC 189. A resource usually uses one or more such contracts to give what a request asks for. The Contracts (RFC 189) page tells how.

Where the code is

Further reading

• The configuration data flow of the Fediversity architecture.
• The data model review of 2025-07-03.

Contracts (RFC 189)

This page tells how the contracts of RFC 189 connect the software that needs a service to the software that gives it. A contract is the interface between a consumer module and a provider module of a service, for example smtp, postgresql, ssl or oidc.

A contract is not a part of the Fediversity data model. The data model has resources, the interface between a component and a hosting provider. A resource usually uses one or more contracts to give what a request asks for. The configuration data flow page tells how a configuration becomes a deployment through the resources.

For example, Forgejo asks the smtp contract for outbound mail with a sender address. The mail relay that provides the contract answers with a host, a port, a user name and a password file:

A contract has these parts, under config.contracts.<type>:

• want.<consumer>.<instance>.request -- what a consumer asks for.
• providers.<name> -- the providers that can answer the request.
• defaultProvider -- the provider that answers when the consumer selects none. A want that no provider answers fails the evaluation.
• providerRequests.<provider> -- the requests that a provider reads.
• results.<consumer>.<instance> -- what the consumer reads back, for example a host, a port and a password file.

Each contract type defines the shape of its request and its result. The consumer and the provider agree only on these shapes, so a hosting provider can replace a provider without a change to the consumer. The contract reference lists the request and the result of each contract type, with its providers and an example.

Contracts across nodes

Each node of a deployment is a separate NixOS evaluation, so the nodes do not share config.contracts. When the consumer and the provider are on different nodes, crossNodeContract and crossNodeWant in core/lib/lib.nix evaluate the contract once, and give the result to each node. A contract that only gives coordinates, such as the address of a DNS server, does not need this. What a cross-node contract costs tells when to use one.

Where the code is

The contracts guide of the vendored contracts tree tells how to write a consumer, a provider and a new contract type.

Further reading

• RFC 189, the contracts RFC, and its implementation in Self Host Blocks.
• nixpkgs pull request 506343, which adds default providers, providers outside NixOS, contracts across nodes, and introspection to make a form.
• Talks on contracts: NixCon NA 2024 and NixCon 2024.

Credentials register

This part lists the credentials of the system: where each one is, what makes it, how long it lives, how it changes, and what a change interrupts. The page gives names and locations only. The components that can read each credential are in the access model.

The word "Open" in a cell means: the code and the documentation give no procedure. The page gives a period or a procedure only when the code or a document holds it.

How the register is made

The tables that have the label "generated" come from the code at each build of the documentation (core/docs/md/technical-design/credentials.nix). Each generated table has two inputs:

• The names, which the build reads from the code: the realised configuration of the nodes with the groups of core/testers/groups.nix, the file core/resources/secrets/secrets.nix, and the workflows in .forgejo/workflows/.
• The facts that the code does not hold (the procedure and what it interrupts), which a person writes in credentials.nix.

The build compares the two inputs. The check core-docs fails when a name has no facts, or when facts have no name. The other tables are written by hand in this file.

The specification of the secret rotation gives the requirements for rotation. Its table in "Key Entities" adds, for each secret, the backups that depend on the secret and how the consumers get a new value. This page is the source for the list of the credentials and for the state of their rotation.

Credentials that OpenBao rotates

OpenBao makes these credentials and changes them on a schedule. No person and no deploy writes them. A consumer reads the current value through its OpenBao agent. The lifetime of a value is the rotation period. The code does not state what a rotation interrupts: the column "Consumer refresh" of the specification gives how each consumer gets the new value.

Generated:

The rotation period of a Valkey namespace is not an option. It is 24 hours in core/components/hosting-provider/openbao/valkey-engine.nix. The engine rotates the user app of a namespace. It connects as the user default, and it does not rotate that user: the password of default is the generated secret valkey-<namespace> in the next section. The LDAP engine rotates only the accounts in the table. It connects with the management account of lldap, and it does not rotate that account (the instances ldap-bind-* in the next section).

Generated secrets in the KV of OpenBao

A node declares each secret that it needs as a generateFiles request. These facts apply to each row of the table:

• Where: the KV path is <prefix>/secret/<instance>/<file> in the mount secret. The prefix is vars for the hosting provider and vars/operator-deployments/<namespace> for a deployment.
• Maker: the script of the request makes the value at the first deploy of the node (nix/contracts/nixos/modules/services/security/vars-openbao.nix).
• Lifetime: the code sets no end. A deploy keeps a value that is in KV.
• Change: a deploy with VARS_FORCE_UPLOAD=<instance> or VARS_FORCE_UPLOAD=<instance>/<file>, or the command upload-vars-openbao-force on the node, makes a new value and writes it. This is a write, not a rotation procedure: it does not tell what the consumers do with the new value. Thus a row without a procedure for its consumers has "Open".
• For some keys OpenBao keeps all versions, because a backup from before a change needs the old key. The column "KV keeps all versions" shows them.

The table shows the nodes of the hosting provider group and of one operator group of the testers. A file that is not secret (for example a public key) is not in the table.

Generated:

The change of an application encryption key has its own work item (fediversity/fediversity issue 2433).

Other secrets in the KV of OpenBao

No generateFiles request of a node of core makes these entries. The api node and the panel node are not in the setups that the build of this page reads, thus their rows are written by hand.

Credentials of OpenBao

Generated, the names of the scoped tokens:

database-init, deployer-kv-prune, kv-key-metadata, ldap-engine, oidc-auth, operator-deployer-token, pki-init, pki-leaf, snapshot, spire-auth, spire-cert-auth, ssh-certificates, valkey-engine

Credentials of SPIRE

Each node logs in to OpenBao with an identity that SPIRE gives. Without an upstream root, the SPIRE server signs its own CAs, and each CA lives for about one day. With the upstream root, the root lives 10 years and each CA lives 24 hours. The private key of the root is the forge secret SPIRE_ROOT_KEY in the section about the forge. The specification of spire gives the custody and the rotation of the root in "The upstream root".

Keys of persons

Secrets of the development infrastructure

The files core/resources/secrets/<name>.age hold the secrets of the forge and its runners. Each file is encrypted to the recipients that core/resources/secrets/secrets.nix gives. A person makes each value, and the code sets no end for it. A change of the recipients is a rekey (core/resources/secrets/README.md, "Rekeying"): it needs a person who is a recipient now, and it does not change the value.

Generated:

Secrets of the forge

The CD jobs and the other workflows read these secrets from the Forgejo Actions secrets of the repository. A person with the rights to set secrets on the forge writes each one. The code sets no end for a value, apart from the incus client certificate: the command in notes/cd-credentials.md makes it for 365 days. .forgejo/workflows/cd-env.example.sh gives the same names for a deploy from a workstation. The forge variable TF_BACKEND_HTPASSWD holds one line for each user of the state backend (cd, operator, and workstation-<person>), with the hash of the password only. The rotation of the password of a workstation-<person> user is Open.

Generated:

The workflows also read these names as secrets. They give an address, a name or a selector, not a credential. Generated:

ADMIN_SOURCES, BIND2_IPV4, BIND2_IPV6, BIND_IPV4, BIND_IPV6, CLUSTER_SOURCES, DNS_APEX_NAMESERVERS, DNS_DOMAIN, DNS_PROVIDER, DNS_SECONDARIES_BY_ADDRESS, DNS_SERVER_URL, NETBOX_CLUSTER_NAME, NETBOX_CUSTOM_FIELDS, NETBOX_PREFIX_SELECTOR, NETBOX_RANGE_SELECTOR, NETBOX_SERVER_URL, NETBOX_TAGS, NETBOX_TENANT, OPERATOR_DNS_PROVIDER, OPERATOR_DNS_SERVER_URL, OPERATOR_NETBOX_RANGE_SELECTOR

Signing key of the bridge cache

The incus host signs the paths that its binary cache serves to its containers.

• Where: /var/lib/incus-bridge-cache/ on the host. The key does not go out of the host.
• Maker: the host.
• Lifetime: fediversity.incus.bridgeCacheKey.rotationDays, 90 days by default.
• Rotation: automatic. The timer incus-bridge-cache-key-rotate runs one time each day. The containers trust the previous key for fediversity.incus.bridgeCacheKey.overlapDays, 30 days by default.
• What a rotation interrupts: nix-serve restarts.

The procedure after a compromise is in Signing key of the bridge cache.

Data subject requests

A person can ask to see, to correct, to erase or to get a copy of the personal data that a deployment or a hosting provider holds about them. This page tells, for each store that holds personal data, which party answers such a request, and which mechanism the code in this repository gives for it. Where the code gives no mechanism, the page says "no mechanism found".

This page is not legal advice. It does not tell if a request is valid, which data is in its scope, or how much time the answer can take. It tells only what the code can do.

The stores and their personal data are in Data protection. The part state-register has the full list of the state that the system stores. This page does not repeat these lists.

Who answers a request

The party that controls the data answers the request. Data protection gives the roles:

• The operator is the controller of the data of its deployment: the accounts and the content of its applications, and the users of the lldap and of the Authelia of its group.
• The hosting provider is the controller of its own data: the accounts of the panel, the users of the lldap of the hosting-provider group, the audit records, and the accounts of Grafana, NetBox and Windmill.
• The hosting provider is the processor of the operator for the data of a deployment. For some stores of a deployment, only the hosting provider has a tool, for example a command on a node or a backup. Then the operator answers the request, and the hosting provider does the step.

flowchart TD
  request["Request of a person"] --> which{"Which data?"}
  which -->|"Data of a deployment"| operator["Operator answers"]
  which -->|"Data of the hosting provider"| provider["Hosting provider answers"]
  operator --> app["Functions of the application"]
  operator --> lldap["lldap of the group"]
  operator --> config["Configuration in the panel, for example deleteData"]
  operator -->|"asks for a step on a node, a log or a backup"| provider
  provider --> panel["Panel account page and admin pages"]
  provider --> api["Routes of api/"]
  provider --> node["Commands on a node, logs, backups"]

Mechanisms that the code gives

The panel and api/ have no route that finds, changes, erases or exports the data of one user of an application. The functions of the application do this work. This repository does not contain the code of the applications, thus this page does not describe these functions. Find them in the documentation of the application version that the deployment runs.

Data of a deployment

The operator answers a request about this data.

After a restore from a backup that is older than an erasure, the erasure must be done again. The operation "Record an erasure" of an application keeps the time and the reference of each erasure on the node, and a restore from an older snapshot lists them until "Confirm the erasures". See Do the erasures again after a restore.

The export of a deployment for a migration gives all the data of the deployment to a different hosting provider, and stops its applications. It is not an export for one person. See Export a deployment for a migration.

Data of the hosting provider

The hosting provider answers a request about this data. The persons are the staff of the operators and the staff of the hosting provider.

The removal of a panel account first erases the domain contact of the account. A contact that a domain of the registrar account names stays, and the panel writes a warning to its log. The removal does not remove the audit events of its actor or the user in lldap. Each of these is a different store with its own row in this table.

Where no mechanism exists

The code gives no mechanism for these requests:

• An export of the data of one person, for all stores but the accounts of the applications. The export of a deployment is for the deployment as a whole (#3058).
• Access, rectification or erasure for one person in the logs, the telemetry, the jobs of Windmill and the audit records. Only the retention removes this data. The account history of the panel gives access to the audit events of one actor (#3062).
• Access, rectification or erasure in the database of Authelia and in the caches of the applications (#3061).
• Erasure of one record from a restic backup before it expires. The procedure removes whole snapshots (#3060).
• A page where a panel account removes itself, or changes its user name or email address (#3056).
• One command or one page that finds all the data of one person in all stores. A party that answers a request goes to each store (#3057).

#2194 tracks the compliance needs of customers in the EU, and these gaps belong to it.

Deployment topology

This page shows which nodes a deployment has, where each node runs, and which networks connect the nodes. It also shows where the trust boundaries of the threat model are in that topology. The glossary defines the terms.

The build makes the diagrams and the tables of this page from the code: the group definitions (core/setups/groups), the components (core/components), the constants of the bridge (core/components/hosting-provider/incus-bridge.nix) and the attribute families of core (core/render/catalog.nix). The tables show the groups with the example apex example.org and with no settings from the environment.

Groups and nodes

A group is a set of machines that one deployment makes or changes. core/setups/groups/default.nix defines four groups:

• hosting-provider has the shared services of a hosting provider.
• operator has the applications of one operator and the services that only that operator uses. Each deployment of an operator is one instance of this group.
• hypervisor has the host that runs the Incus daemon. It is a group of its own because it hosts the instances that the deployment of the hosting-provider group makes.
• dev has the forge and the runners of the continuous integration.

Each group has two rosters of machines, published and internal. In the operator group, the published roster has the nodes of the applications and the internal roster has the nodes of the services that the applications use. Each node runs one component. In the tables, the column "Application" tells if an operator can enable or disable the component, and the column "Provides" gives the resources that the component supplies to other components (wiring.provides in core/components/wiringProperties.nix).

Group dev

Group hosting-provider

Group hypervisor

Group operator

Notes for the tables:

• The nodes api and panel are not in the tables. The leaves add them to the published roster of the hosting-provider group (api/default.nix, panel/default.nix).
• The hosting-provider group has the node bind2 only when the environment gives its address (core/setups/groups/hosting-provider.nix).
• The operator group has no database node and no object store node. Its applications use the postgresql and garage nodes of the hosting provider (core/setups/groups/operator.nix).
• The machine incus of the hypervisor group and the machines dev2, dev3, dev4 and fediversity-ci of the dev group each have their own NixOS configuration in core/setups/hosts/.

Deployment methods

A deployment method decides how the nodes of a group come to exist and how they get their configuration. Flake attributes lists all methods and their commands. The table shows the methods that each group has.

The methods give three variants of the topology:

• Machines that exist. The methods ssh and tf deploy to machines that are there before the deployment. The methods provision nothing (core/setups/ssh-hosts.nix, core/setups/tf-hosts.nix).
• Incus instances. The methods tf-incus and tf-incus-operator make one Incus instance for each node on the hypervisor, and then deploy to it (core/setups/tf-incus-hosts.nix, core/setups/tf-incus-operator-hosts.nix).
• Kubernetes pods. The methods k8s and k8s-operator run each node as a pod in a cluster (core/effects/k8s/effect.nix).

The method node-effects deploys nothing.

Incus instances

flowchart TB
  internet(["Internet"])
  operatorPerson(["Operator"])
  admins(["Contributors and administrators"])
  upstream(["Upstream sources and binary caches"])
  backupTarget(["External backup target"])

  subgraph dev["Group dev: machines that exist"]
    dev_dev2["dev2<br/>component forgejo-runner"]
    dev_dev3["dev3<br/>component forgejo-runner"]
    dev_dev4["dev4<br/>component forgejo-runner"]
    dev_fediversity_ci["fediversity-ci<br/>component forgejo-runner"]
    dev_forgejo["forgejo"]
  end

  subgraph hypervisor["Group hypervisor: machine incus"]
    daemon["Incus daemon<br/>API on port 8443"]
    gateway["Bridge address 10.0.100.1<br/>dnsmasq and nix-serve"]
    public["Public segment<br/>bridge pubbr0, second interface eth1 of a node"]
    subgraph bridge["Bridge incusbr0: 10.0.96.0/20"]
      subgraph provider["Group hosting-provider: 10.0.100.2-10.0.100.254"]
        hosting_provider_authelia["authelia"]
        hosting_provider_bind["bind"]
        hosting_provider_garage["garage"]
        hosting_provider_lldap["lldap"]
        hosting_provider_netbox["netbox"]
        hosting_provider_openbao["openbao"]
        hosting_provider_otel_collector["otel-collector"]
        hosting_provider_overview["overview"]
        hosting_provider_postgresql["postgresql"]
        hosting_provider_smtp["smtp"]
        hosting_provider_spire["spire"]
        hosting_provider_telemetry["telemetry"]
        hosting_provider_terraform_backend["terraform-backend"]
        hosting_provider_valkey["valkey"]
        hosting_provider_windmill["windmill"]
        leaves["api and panel<br/>nodes that the leaves add"]
      end
      subgraph operator["Group operator: one /28 of 10.0.104.0/21 for each deployment"]
        subgraph operatorPublished["Roster published"]
          operator_mastodon["mastodon"]
          operator_peertube["peertube"]
          operator_pixelfed["pixelfed"]
        end
        subgraph operatorInternal["Roster internal"]
          operator_authelia["authelia"]
          operator_grafana["grafana"]
          operator_lldap["lldap"]
          operator_smtp["smtp"]
          operator_valkey["valkey"]
        end
      end
    end
  end

  internet -->|"boundary 1"| public
  public --> provider
  public --> operatorPublished
  internet -->|"boundary 2: ports 22 and 8443"| daemon
  operatorPerson -->|"boundary 3"| leaves
  operator -->|"boundary 4: ports of the first table below"| provider
  operator -->|"ports of the second table below"| gateway
  provider --> gateway
  dev -->|"boundary 6"| daemon
  admins -->|"boundary 7"| hypervisor
  upstream -->|"boundary 8"| gateway
  daemon -->|"makes the instances"| bridge
  operator -->|"boundary 5"| hosting_provider_openbao
  hosting_provider_garage -->|"boundary 9"| backupTarget

The diagram shows the hosting-provider group, one deployment of the operator group, the hypervisor and the dev group. The numbers are the numbers of the trust boundaries in the threat model.

Instances

• One Incus daemon on the hypervisor runs the instances of the hosting provider and of all operator deployments (core/setups/groups/operator.nix).
• The name of an instance of an operator deployment is <namespace>-<node>. The deployment of the hosting provider has an empty namespace (core/setups/tf-incus-hosts.nix).
• An instance is a container by default. The setting instanceType accepts container and virtual-machine (core/effects/common/effect-meta.nix).
• A node that uses the /nix/store of the hypervisor must be a container. A deployment can use virtual-machine only if none of its nodes uses that store (core/effects/tf/incus/instance.tf).
• A container gets a software TPM 2.0 by default (enableTpm in core/effects/common/effect-meta.nix). The SPIRE agent of the node uses it to attest.
• The deployment program does not use SSH into a container. It connects to the Incus API on port 8443 of the hypervisor and uses incus exec (core/setups/tf-incus-hosts.nix).
• The nodes of the hosting-provider group that start deployments have a client credential for the Incus API (deployRunnerApps in core/setups/hosts-common.nix).

Network segments

All instances are on one bridge of the hypervisor, with IPv4 only. The bridge is the interface eth0 of each instance.

• The dnsmasq service of the bridge gives each node of the hosting provider a lease and a name (core/components/hosting-provider/incus-bridge.nix).
• Each operator deployment gets a /28 from NetBox. An Incus network ACL on each interface of such a node drops the traffic to and from the other operator deployments (core/components/hosting-provider/incus-bridge.nix).
• A node of the hosting provider accepts internal traffic from the bridge subnet. A node of an operator deployment accepts internal traffic from the provider prefix and from the addresses of its own deployment (internalSources in core/setups/hosts-common.nix).
• The public segment is a second interface, eth1, of an instance. With the uplink mode nat, which is the default of the Incus methods, an instance has no such interface (core/setups/wiring/public-uplink.nix). On the hypervisor, the bridge pubbr0 on a VLAN interface is the parent of the public interfaces (core/setups/hosts/incus.nix).
• A port of a node is open to the internal sources only, until the component that serves the port publishes it (core/setups/wiring/node-selection.nix, core/resources/firewall).

A node of an operator deployment can connect to these ports of the nodes of the hosting provider, by the resource that the node of the hosting provider supplies (core/components/hosting-provider/operator-reach.nix):

The nodes on the bridge can connect to these ports of the hypervisor (core/components/hosting-provider/hypervisor-reach.nix, core/components/hosting-provider/operator-reach.nix):

Kubernetes pods

flowchart TB
  internet(["Internet"])
  registry["Image registry"]

  subgraph cluster["Kubernetes cluster"]
    balancer["Load balancer or shared Gateway"]
    subgraph namespace["Namespace of one operator deployment: default-deny NetworkPolicy"]
    subgraph pod_authelia["Node authelia"]
      pod_authelia_deployment["Deployment authelia<br/>one pod"]
      pod_authelia_state[("PersistentVolumeClaim<br/>authelia-state")]
      pod_authelia_deployment --- pod_authelia_state
    end
    subgraph pod_grafana["Node grafana"]
      pod_grafana_deployment["Deployment grafana<br/>one pod"]
      pod_grafana_state[("PersistentVolumeClaim<br/>grafana-state")]
      pod_grafana_deployment --- pod_grafana_state
    end
    subgraph pod_lldap["Node lldap"]
      pod_lldap_deployment["Deployment lldap<br/>one pod"]
      pod_lldap_state[("PersistentVolumeClaim<br/>lldap-state")]
      pod_lldap_deployment --- pod_lldap_state
    end
    subgraph pod_smtp["Node smtp"]
      pod_smtp_deployment["Deployment smtp<br/>one pod"]
      pod_smtp_state[("PersistentVolumeClaim<br/>smtp-state")]
      pod_smtp_deployment --- pod_smtp_state
    end
    subgraph pod_valkey["Node valkey"]
      pod_valkey_deployment["Deployment valkey<br/>one pod"]
      pod_valkey_state[("PersistentVolumeClaim<br/>valkey-state")]
      pod_valkey_deployment --- pod_valkey_state
    end
    subgraph pod_mastodon["Node mastodon"]
      pod_mastodon_deployment["Deployment mastodon<br/>one pod"]
      pod_mastodon_state[("PersistentVolumeClaim<br/>mastodon-state")]
      pod_mastodon_deployment --- pod_mastodon_state
    end
    subgraph pod_peertube["Node peertube"]
      pod_peertube_deployment["Deployment peertube<br/>one pod"]
      pod_peertube_state[("PersistentVolumeClaim<br/>peertube-state")]
      pod_peertube_deployment --- pod_peertube_state
    end
    subgraph pod_pixelfed["Node pixelfed"]
      pod_pixelfed_deployment["Deployment pixelfed<br/>one pod"]
      pod_pixelfed_state[("PersistentVolumeClaim<br/>pixelfed-state")]
      pod_pixelfed_deployment --- pod_pixelfed_state
    end
    end
  end

  subgraph provider["Nodes of the hosting provider, not in the cluster"]
    openbao["openbao"]
    spire["spire"]
  end

  internet -->|"boundary 1"| balancer
  balancer -->|"Service of a node"| namespace
  registry -->|"image of a node"| namespace
  namespace -->|"boundary 5"| openbao
  namespace -->|"attestation of the SPIRE agents"| spire

The diagram shows one deployment of the operator group with the method k8s-operator. core/effects/k8s/effect.nix makes these objects:

• Each node is one Deployment with one pod. The pod runs an image of the NixOS configuration of the node (core/setups/nixos/k8s-pod.nix).
• Each node has the PersistentVolumeClaim <node>-state on /var/lib. The other paths of the pod start again from the image.
• Each role of a node is one more Deployment, <node>-<role>. A role with a service runs the image of that service as an unprivileged user.
• A NetworkPolicy lets no connection in to the pods of the namespace. One more NetworkPolicy for each node lets in what the firewall resource of the node permits.
• A node with ports for sources outside of the cluster has a Service of the type LoadBalancer.
• The method k8s-operator appends the namespace of the deployment to the name of the Kubernetes namespace. It publishes the names of a node at the address of its Service. It can also connect the nodes to a shared Gateway with a TLSRoute and an HTTPRoute.
• The OpenBao and SPIRE servers stay outside of the cluster. A pod has no TPM, thus the SPIRE agent of a pod attests with the token of its service account.

The deployment does not make the cluster, the image registry, the RuntimeClass, the storage class and the load balancer. They must be there before the deployment.

Trust boundaries in the topology

The five functions

This part tells how the system does each of the five functions of the scope.

Install applications

The operator selects applications in the form of panel. core makes the form data: a JSON Schema of the options of each application, and the catalog of the applications. api gives them to the browser (/schema, /applications). When the operator deploys, the browser sends the configuration to api (/deployments/submit/{id}). api asks panel if this account can change this deployment, validates the configuration, puts a build in the queue, and then runs the deployment program of core. The deployment program makes the machines on the Incus hypervisor, the DNS records, the certificates and the secrets, and it installs the NixOS configuration of each machine.

Details: Make a deployment in the user guide of the panel, the design of the API, and Add an application for the application side.

Configure applications

The options of an application are the options of its component in the data model of core (Operator configuration). panel shows them as a form. A change of the configuration is a new submit: api validates the new configuration and deploys it. core also gives a table of option renames, and api uses the table to move a stored configuration to the current options (/schema/migrations).

Details: the schema and migration sources of the API, and the data model.

Update applications

The version of an application comes from the revision of the code that a deployment uses, and that revision pins nixpkgs. Thus an update is a deploy with a newer revision. Each deployment has an update policy in api (/deployments/update_policy/{id}): manual, automatic or window. With manual, the operator starts the deploy. With the other two policies, api starts the deploy when it finds a new default source, immediately or in a maintenance window.

Details: the update policy of the API.

Manage users

There are two sets of users:

• The accounts of the operators are in panel. An operator logs in with the single sign-on of the hosting provider or with a local account. The hosting provider manages the accounts in the administration pages of panel. panel keeps which account owns which deployment, and api asks panel before it does an action.
• The users of the applications are in the deployment of the operator. Each deployment has its own lldap directory and its own Authelia single sign-on. core connects each application to them through the ldap and oidc resources. The operator adds users in lldap, and the groups in lldap give the roles in the applications.

Details: Manage the users of your deployment for the operator, Manage users and groups for the hosting provider, and the architecture page of the panel for the accounts.

Switch an operator to and from a different hosting provider

A move has a source and a target, and each is an installation of Fediversity.

• At the source, api exports the deployment (/migration-string/{id}). The core/effects/migration effect stops the services of each application and makes a snapshot of its data. The result is a migration string. It holds a pointer to the snapshot of each application, and the transfer code of the domain when the registrar account of the source holds the domain.
• At the target, api imports the string (/migration-import/{id}) and deploys the configuration. The core/effects/migration-restore effect restores each application from the snapshot of the source. The target can transfer the domain to its own registrar account.

The configuration of an operator can name its own module source. Thus the operator can move away from a hosting provider that uses a different revision of the code.

Details: Export a deployment for a migration and Restore the apps of a migration in the hosting-provider guide, the export of a migration in the design of the API, and the switching and exit clauses of the contract templates.

Metering and accounting

This part tells what the system measures about a deployment of an operator, where each number comes from, and where a hosting provider reads it. It also tells what the system does not measure.

The system gives the quantities. It does not calculate a price and it does not make an invoice. The only amounts of money in the system are the costs that the registrar gives for a domain.

The sources

The system has three sources, and each one has its own store.

flowchart LR
  subgraph api["api/"]
    builds["build table"]
    events["domain_event table"]
    usage["GET /usage"]
    builds --> usage
    events --> usage
  end
  deploy["Deploy job"] -- "result and domain.json" --> builds
  deploy --> events
  snapshot["Snapshot of the registrar account"] -- "renew, release" --> events

  subgraph telemetry["Telemetry of the hosting provider"]
    collector["otel-collector"]
    mimir["Mimir"]
    grafana["Grafana"]
    collector --> mimir --> grafana
  end
  storage["PostgreSQL and garage nodes"] -- "size of each database and bucket" --> collector
  hypervisor["Incus hypervisor"] -- "use of each instance" --> collector
  apps["Application nodes"] -- "application metrics" --> collector

  provider["Hosting provider"]
  usage --> provider
  grafana --> provider

The two sources of api/ and the telemetry have no link in the code. api/ does not read Mimir, and the telemetry does not read the state store of api/. The deployment id is the common key: it is deployment_id in the result of GET /usage and it is the label deployment_id of the metrics.

Usage periods

GET /usage (usage_list, api/src/fediversity_api/routes/usage.py) gives the usage periods. usage_list of api/src/fediversity_api/procedures.py calculates them at each call from the builds that succeeded (list_succeeded_builds of api/src/fediversity_api/state.py). api/ does not store a period.

• Each build that succeeded and is not a destroy starts a period when it finishes. The next build of the deployment that succeeds ends the period. A destroy also ends it. The last period of a deployment that is not destroyed has end set to null.
• A build that failed or was canceled changes no period.
• A period has the deployment_id, the build_id, the start, the end, the configuration and the module_source of its build, and the flag suspended. A suspended deployment stops its instances and keeps its volumes.
• A period has the domain that its configuration serves the applications below, and the registered_domain that the deployment holds at the registrar account of the hosting provider in that period, if any.
• api/ keeps the builds of a deployment after its destroy. Thus the periods of a destroyed deployment stay available.

The params since and until select the periods that overlap that time. api/ does not cut a period to these times. The param id selects deployments, and a caller can give it more than one time.

A period follows the builds and not the instances. It tells which configuration a deployment ran and when. It does not tell how much storage or compute the deployment used. The telemetry gives these quantities.

Domain events

The field domain_events of the same result tells what the registrar account charged for the domains of the deployments. api/src/fediversity_api/domain_events.py records the events in the domain_event table.

An event has its domain, its deployment_id, its cost in minor units of its currency, and the expiration_date of the registration after the event. The cost is null when the registrar charged nothing or when api/ does not know the charge.

api/ records renew and release only when registrarSnapshot of the NixOS module of api/ (api/nix/configuration.nix) points to the snapshot of the registrar account. The unit runs when the snapshot changes.

The events stay after the destroy of their deployment. The page api/docs/design/usage.md has the full rules.

Who can read the usage

GET /usage has the authorization of all routes of api/: api/ asks the authorization callback of the front-end, and the callback gives the deployments that the caller can see (api/src/fediversity_api/authz.py). api/ then limits the result to these deployments.

When the panel is the front-end, the rule of usage_list is that the caller owns the deployments (AUTHZ_PROCEDURE_RULES in panel/src/panel/views.py):

• The callback gives the deployments of one account, also for a superuser. Thus one call gives the usage of one operator.
• The callback includes the deployments that the operator deleted. The panel keeps the row of a deleted deployment for this purpose, until the removal of the account of the operator (panel/src/panel/models.py).
• A backend of the hosting provider can call api/ with the service token of api/ and the header X-On-Behalf-Of with the user name of an account. The callback then decides as for that account.

The panel has no page that shows the usage. No view of the panel calls usage_list; the panel only answers the authorization callback for it. A hosting provider reads GET /usage with its own tool.

Resource use in the telemetry

The telemetry gives the quantities that the usage periods do not have. Each metric below has the label deployment_id, and the metrics are in Mimir.

The label deployment_id comes from a name that the deployment cannot change after the fact:

• For a database, a bucket and an instance, a relabel rule takes it from the name of the database, the bucket or the instance (core/components/hosting-provider/deployment-relabel.nix). A database, a bucket or an instance of the hosting provider has no deployment_id.
• For the host metrics of a node of an operator deployment, the telemetry node sets it from the path of the request, which opens only with the credential of that deployment (core/components/hosting-provider/deployment-ingest.nix).
• For the application metrics, it comes from the KV path of the endpoint entry (core/components/hosting-provider/app-metrics-discovery.py). A deployment can write only below its own id. A deployment_id label that an application gives becomes exported_deployment_id.

The hosting provider reads these metrics in Grafana or with a query on Mimir. The dashboard "Deployment storage" (core/components/hosting-provider/dashboards/deployment-storage.json) shows the size of the databases and the buckets of each deployment. No dashboard shows the usage periods or the domain events.

Mimir keeps the metrics for services.fediversity-telemetry.retentionDays (core/components/hosting-provider/telemetry.nix). After that time, the resource use of a past period is not available. The usage periods and the domain events have no retention: api/ does not remove a build or a domain event.

Know the use of the deployments for billing in the hosting provider guide gives the procedure and example queries.

What the system does not do

• It has no prices, no tariffs and no invoices. A domain event has the cost that the registrar gave, and no other amount is in the system.
• It does not add the resource use to the usage periods. The hosting provider joins the two on the deployment id.
• It does not aggregate the metrics for a billing period. The hosting provider does this with a query.

Logs, traces and audit records

This part lists each channel that records what the software and its users do: the audit events, the logs, the traces and the job records. For each channel it gives the personal data, the purpose of the data, the store, the persons who can read it, the retention, the control that keeps credentials out of it, and the test that shows the control.

The page describes what the code does. It does not say what the law requires or permits. Where a legal question applies, the page says "Open". The hosting provider decides it, see the hosting provider guide.

The stores themselves (location, owner, encryption, backup, erasure) are in the state register. The roles and the directory groups are in the access model.

The tables that have the label "generated" come from the code at each build of the documentation (core/docs/md/technical-design/observability-records.nix).

Words on this page

• "None" means that the code was examined and has no such control, test or limit.
• "Open" means that no source in the tree gives the fact, or that the question is a legal one.
• "A person with root on the node" is a person who can open a shell as root on that node. The access model says who that is.

How a record goes from a node to the stores

flowchart TD
  units["Units of a node"]
  journal["Journal of the node"]
  nodeLogs["Node log shipper (otel-node-logs)"]
  auditShipper["Audit shipper (fediversity-audit-shipper)"]
  collector["otel-collector"]
  lokiDefault["Loki, default tenant"]
  lokiAudit["Loki, audit tenant"]
  tempo["Tempo"]
  grafana["Grafana"]
  apiTable["audit_event table of api/"]
  api["api/"]

  units --> journal
  api --> apiTable
  api --> journal
  journal --> nodeLogs
  journal --> auditShipper
  nodeLogs --> lokiDefault
  auditShipper --> lokiAudit
  collector --> lokiDefault
  collector --> tempo
  lokiDefault --> grafana
  lokiAudit --> grafana
  tempo --> grafana

The nginx of the telemetry node sets the tenant of each request (X-Scope-OrgID), thus a client cannot select a tenant (core/components/hosting-provider/telemetry.nix).

Retention

The hosting provider sets the retention periods. Each period below is the default of a setting, and the hosting provider can change it in its configuration. The reason for a period, and the law that applies to it, are not in the code: Open.

Generated:

Limits that are not a setting of this software:

• The nginx access log of a node other than the forge node: the logrotate rules of nixpkgs keep 26 weeks of files (the comment in core/setups/nixos/nginx.nix).
• The rows of the build table of api/: None. No job removes old rows. logRetentionDays removes only the log of a row.
• The disk queue of the audit shipper: None. The module sets no limit of size or of age.
• The logs of the CI jobs and the change log of NetBox: Open.

Channels

Audit events of the api

api/ records each action on a deployment as one audit event (api/src/fediversity_api/audit.py).

• Store: the audit_event table of the database of api/, and one JSON line on the logger fediversity_api.audit. The line goes to the journal, and then to Loki, see Audit lines in Loki.
• Readers: an operator reads its own events through the panel (recent_actions, GET /audit-events). A superuser of the panel reads the events of each account (reads_all_history). A person with access to the database reads the table.
• Retention: services.fediversity-api.auditRetentionDays. api/ removes the older events when it starts and each time that it records an event (prune_audit_events in api/src/fediversity_api/state.py). The events of a deleted deployment stay until they are older than the retention.
• Control that keeps credentials out: an event holds only the fields of the table below. For a request that is not valid, reason gives the location of the error and not the value. For an import, the event holds only the result. api/ cuts a long reason of the authorization callback, and cuts request_id to 128 characters.
• Test: api/src/fediversity_api/tests/test_audit.py: test_an_invalid_request_records_where_without_the_value, test_an_invalid_configuration_records_where_without_the_value, test_an_import_records_only_its_result, test_an_unknown_action_stores_nothing_of_the_caller, test_a_long_reason_of_the_callback_is_cut. The retention has the tests test_a_new_event_removes_the_events_older_than_the_retention, test_the_retention_is_a_setting and test_zero_keeps_all_events.
• Protection from changes: each event has a hash of the event and of the event before it. fediversity-api-audit-verify examines the chain (verify_audit_chain). The tests are test_a_changed_event_is_found, test_a_removed_event_is_found and test_a_prune_keeps_the_chain_correct.

The fields of an event, with the purpose of each. Generated:

An audit event does not hold the client address. To connect an event to a client address, a person must find the journal line or the access log line of the same request, and the access log holds only the network of the client.

api/ records the first refused call without a login in each interval of 300 seconds, and then one event with the number of the other calls. Thus an anonymous caller cannot fill the table.

Audit lines in Loki

The audit shipper of a node (core/components/hosting-provider/audit-shipper.nix) reads the journal of the unit of a component, keeps the lines that match the pattern of the component, and sends them to the audit tenant of Loki. Generated:

• Personal data and purpose: the same fields as the audit event.
• Store: the audit tenant of Loki. Before the shipper sends a line, the line is in its disk queue in /var/lib/fediversity-audit-shipper. The shipper does not stop its retries, thus a line stays in the queue while the telemetry node does not accept it.
• Readers: each account that can log in to the Grafana of the hosting provider, thus the members of fediversity-admins and fediversity-staff. Grafana has one organization and the datasource Loki audit has no permission of its own, thus a Viewer can query the tenant. A person with root on the node of api/ reads the queue.
• Retention: services.fediversity-telemetry.auditRetentionDays. The tenant refuses the deletion API of Loki, thus only the retention removes a line. The setting of api/ does not apply to these lines.
• Control that keeps credentials out: the control of the audit event. The shipper adds none.
• Test: None for the content of a shipped line.

The deploy credentials can change or delete the tenant, see the hosting provider guide.

Journal of each node

• Personal data: what each unit writes. sshd, Authelia and the nginx error log write full client IP addresses and user names. maddy writes email addresses. The applications write what their log level gives.
• Purpose: to find the cause of a failure on a node (the comment in core/setups/nixos/common.nix). A purpose for each field: Open.
• Store: the journal of the node. A node of the hosting provider sends the lines at the level of services.fediversity-node-logs.priority and higher to the default tenant of Loki (core/components/hosting-provider/node-logs.nix). A node of an operator deployment sends the lines at level warning and higher, and the telemetry node sets the label deployment_id (core/components/hosting-provider/deployment-ingest.nix).
• Readers: a person with root on the node reads the journal. Each account that can log in to the Grafana of the hosting provider reads the copy in Loki, for all deployments.
• Retention: see Retention. The lines of a deleted deployment stay in Loki until the retention removes them.
• Control that keeps credentials out: each unit can write a credential, because a request line or an error message of an application can hold one. Thus the control is in the pipeline and not in each unit. The node log shipper replaces the text that has the shape of a credential with [masked] before a line goes to Loki (core/components/hosting-provider/node-logs-masks.nix). The shapes are: the query string of a request line or of a URL, the user and the password of a URL, the value of an Authorization header, a bearer token, the value of a key whose name contains password, passwd, secret, token, api_key or credential, a JSON Web Token, and a token of OpenBao. A credential that has none of these shapes goes to Loki. The shipper does not read a message that is not UTF-8. The journal of the node keeps the full line: None for the journal.
• Test: the check node-logs-masks (core/components/hosting-provider/node-logs-masks-test.nix) sends an example of each shape through the masks. The check test-monitoring-storage (core/components/hosting-provider/monitoring-test.py, "the node log shipper masks a credential") writes values to the journal of a node and reads Loki. core/setups/hypervisor-config-test.nix examines the retention settings of the journal.

Audit device of OpenBao

OpenBao writes each request and each response as one JSON line to its standard output (core/components/hosting-provider/openbao/openbao.nix).

• Personal data: the client IP address, the path and the operation of the request, and the display name and the entity of the token, for example the name that an OIDC login gives to an administrator.
• Purpose: Open. No source in the tree gives the purpose of a field.
• Store: the journal of openbao.service, and then the default tenant of Loki. These lines do not go to the audit tenant.
• Readers: the readers of the journal and of the default tenant, see Journal of each node.
• Retention: that of the journal and of the default tenant. The lines do not have the retention of the audit tenant.
• Control that keeps credentials out: OpenBao writes secret values and tokens as HMAC-SHA256 hashes, and elide_list_responses removes the keys of a list response.
• Test: the check integrations-openbao-pki (core/checks/integrations/openbao-pki/nixosTest.nix, "The audit device writes each request to the journal with hashed values").

nginx access log

Each node that runs nginx writes one line for each request to /var/log/nginx/access.log (core/setups/nixos/nginx.nix).

• Personal data: the network of the client (the address with its last part removed: /24 for IPv4, /48 for IPv6), the method, the path and the protocol of the request, the referrer without its query string, the user agent, and the user name of HTTP basic authentication. On the vhosts with auth_basic (telemetry ingest, metrics query, audit path) this user name identifies the credential of one deployment, not a person. On other vhosts it is the text that the client sends.
• Purpose: Open. No source in the tree gives the purpose of a field.
• Store: files on the node. The forge node does not record the calls of the CI runners (core/components/dev/forgejo/forgejo.nix).
• Readers: a person with root on the node. The node log shipper reads the journal and not this file, thus the lines do not go to Loki.
• Retention: see Retention.
• Control that keeps credentials out: the log format holds no query string, of the request or of the referrer, and no password of HTTP basic authentication. A credential in a path or in the user agent goes to the file.
• Test: the check nginx-access-log (core/setups/nixos/nginx-access-log-test.nix) sends requests with a query string, a referrer and a user name with a password, and examines the lines. It examines the mask of an IPv4 address. For the mask of an IPv6 address: None.

Request logs of the api and the panel

api/ and the panel run on uvicorn with its access log on (api/src/fediversity_api/http_main.py, panel/nix/configuration.nix). Each request gives one line with the address of the peer, the method, the path with its query string, and the status.

• Personal data: the query string can hold an identity, for example the actor in GET /audit-events and the account in the history page of the panel.
• Purpose: Open.
• Store: the journal of the unit, and then the default tenant of Loki, because the lines are at level info.
• Readers and retention: those of the journal, see Journal of each node.
• Control that keeps credentials out: in Loki, the masks of the node log shipper, which replace the query string of a request line, see Journal of each node. In the journal of the node: None.
• Test: the check node-logs-masks has a request line of uvicorn as an example.

Jobs of Windmill

Windmill keeps the arguments, the result and the log of each completed job.

• Personal data: a deploy job holds the configuration of the deployment, which can hold email addresses and user names, and the deploy log.
• Purpose: the job runs the deploy, and api/ reads the log of the job to show it to the operator. A purpose for a completed job: Open.
• Store: the database of Windmill, see the state register.
• Readers: the administrators of Windmill. An operator reads the log of the builds of its own deployments through api/ (build_logs in api/src/fediversity_api/procedures.py).
• Retention: services.fediversity-windmill.jobRetentionDays.
• Control that keeps credentials out, deploy job: api/ removes the secret keys from the deploy_env that it sends to the job. Nothing removes a value from the text of the deploy log: None.
• Control that keeps credentials out, node-effect job (f/fediversity/node_effect): the arguments and the result of this job can hold bearer credentials, and Windmill keeps them in clear. api/ deletes the job when it is completed, with a success or a failure, before the result goes back to the caller. When api/ cannot delete a job, it writes the ID of the job to its log, and a person must delete the job (_run_effect_job in api/src/fediversity_api/node_effect.py). Until then, the credentials stay in the job.
• Test: api/src/fediversity_api/tests/test_deploy_env.py (SecretKeysStayOffTheWindmillPayload) for the deploy job. api/src/fediversity_api/tests/test_migration.py for the node-effect job: test_windmill_route_runs_the_job_and_deletes_it, test_a_failed_job_reports_its_error_and_is_deleted, test_a_timed_out_job_fails_and_is_deleted, test_a_job_that_cannot_be_deleted_is_logged_by_id.

Build logs and operation output in the api

• Personal data: the actor of each run of an operation (operation_run table). The text of a build log and of an operation output can hold what the deploy tools and the operation write.
• Purpose: api/ shows the result of a build or of an operation to the operator.
• Store: the build and operation_run tables of the database of api/. With Windmill, api/ reads the build log from the job. Without Windmill, api/ keeps the last 64 KiB of the output of a failed build in the build table.
• Readers: the operator of the deployment, through api/. A person with access to the database.
• Retention: api/ keeps the last runs of each deployment, see Retention, and deletes the runs after the deployment is destroyed. With services.fediversity-api.logRetentionDays, api/ also removes the log of a build and each run of an operation that finished before that number of days (prune_logs in api/src/fediversity_api/state.py). It does this when it starts, and each time that it stores a build log or a run. The default is 30 days. 0 keeps all logs and runs. The rows of the build table have no limit, because the usage records come from them: None.
• Control that keeps credentials out: api/ does not keep the output of an operation that has sensitive set. For the text of a build log, see the deploy job in Jobs of Windmill: the same runner writes the two logs.
• Test: api/src/fediversity_api/tests/test_operations.py: test_the_output_of_a_sensitive_operation_is_not_kept, test_the_deployed_revision_decides_if_the_output_stays. api/src/fediversity_api/tests/test_log_retention.py for the retention. For the build log: api/src/fediversity_api/tests/test_log_credentials.py (FailedBuildLogHoldsNoSecret).

Traces

• Sources: Grafana sends its own spans when tracingEndpoint is set. NetBox sends the spans of its requests, its HTTP calls and its database queries when fediversity.otel.endpoints.netbox is set (core/components/hosting-provider/netbox/otel.nix). Windmill has the settings but its open-source build sends no spans (core/components/hosting-provider/windmill-otel.nix).
• Personal data: a span can hold a request path. The attributes that the instrumentation libraries set: Open.
• Purpose: Open.
• Store: Tempo.
• Readers: each account that can log in to the Grafana of the hosting provider.
• Retention: services.fediversity-telemetry.retentionDays.
• Control that keeps credentials out: None. The collector has only the batch processor (core/components/hosting-provider/otel-collector.nix).
• Test: None.

Debug exporter of the collector

When services.fediversity-otel-collector.backends is not set, the collector has only the debug exporter with verbosity = "detailed". It then writes each record that it receives (logs, metrics and spans) to its own journal. The readers, the retention and the missing controls are those of the journal.

Records of Grafana and NetBox

• Grafana keeps its accounts and its sessions in its own database, see the state register.
• NetBox keeps a change log with the user of each change. No file in the tree sets its retention or its readers: Open.

Logs of the CI jobs

The forge keeps the log of each CI job. The workflows in .forgejo/workflows/ give secrets to the CD jobs as environment variables.

• Personal data: Open.
• Store, readers and retention: Open. No file in the tree sets them.
• Control that keeps credentials out: Open. No file in the tree sets a mask.
• Test: None.

Fields that serve to trace abuse

To trace abuse, a person must know who did a thing, from where and when. These fields give that:

• The audit events of api/: the fields with "Yes" in the last column of the table of the fields. They give the account, the action, the deployment and the time. They do not give a client address.
• The journal: the full client IP address and the user name in the lines of sshd and Authelia, with the time.
• The audit device of OpenBao: the client IP address and the display name of the token, with the path and the time.
• The nginx access log: only the network of the client. It cannot identify one address.

If these fields are sufficient for a duty of the hosting provider to trace abuse, and how long the hosting provider must or can keep them: Open.

Open items

Requirements of hosting providers and design review

This page lists the requirements that the sources of the project record for hosting providers. For each requirement, it gives the source and the place in the technical design that answers it. It also gives the record of the review of the design.

The specifications of core, of api and of panel hold the requirements that the project wrote for the code. They are not statements of hosting providers, thus this page does not repeat them. The part traceability connects them to the design.

Sources

The repository of the code holds no statement that a hosting provider made. The sources below are outside of this repository.

The front-end interview is the only source that records the words of a hosting provider. It has these limits:

• It has the statements of one person of one hosting provider.
• Its subject is the front end of the panel. It does not cover the infrastructure, the metering or the move between hosting providers.
• The person is also a member of the project. The record does not tell which statements the person made as a hosting provider.
• Some statements are ideas for a later version. The tables below mark them.

No source records requirements from other hosting providers.

Requirements from the front-end interview

The column "Answer" tells if the design answers the requirement: "Yes", "In part" or "No". "No" means that the code and the pages of the design show no function for the requirement.

Users and access

Use of the panel

Applications and releases

Requirements that the project wrote for hosting providers and operators

These statements are from the project. No source records that a hosting provider gave them.

Design review

No source records a review of this technical design. The repository of the code, the meta repository and the white-paper repository hold no list of reviewers, no date of a review and no result.

A code review of a pull request on the forge is not a review of the design by hosting providers, thus this page does not count it.

Quality attributes

This page gives, for each quality attribute of the full system, the target, what the code does, how a test shows it, and what the code does not do. A target is a value that a document or the code of this repository states. Where the repository states no target, this page says so. This page does not set a target.

The checks that this page names are attributes of the flake of core. To run one, use nix build ./core#checks.x86_64-linux.<name>. The page Flake attributes describes each check.

Summary

The specifications in core/specs/, api/specs/ and panel/specs/ have success criteria for each feature. No success criterion gives a time, a percentage or a quantity for an attribute of this page.

Availability

Target. The repository states no availability target: no uptime percentage and no maximum time that a service can be down. The contract templates have a row "Availability and backup" that the hosting provider fills in (Contract templates).

What the code does.

• Each stateful service runs on one node. The section Know the limits of a single node of the hosting provider guide lists the services. For example, garage has replication_factor = 1 (core/resources/garage/server.nix), and the stores of the telemetry node have replication_factor = 1 (core/components/hosting-provider/telemetry.nix).
• Grafana alert rules tell the hosting provider about a failure (core/components/hosting-provider/telemetry/alert-rules.nix). unit-failed and unit-stuck find a unit that failed or that does not finish its start. node-absent finds a node that sent host metrics before and sends none now. ingest-silent finds that no node sends metrics. disk-filling, disk-filling-trend and disk-full find a disk that gets full. cert-expiring, cert-expiring-critical and renewal-failed find a certificate that is not renewed. The section Monitor the deployments of the guide tells how to use them.
• A node counts its boots (core/setups/nixos/boot.nix). The boot loader marks a new generation as good only after the check boot-check-gateway gets an answer from the default gateway. A generation that fails the check reboots, and after the configured number of tries the boot loader starts the last good generation.
• With the deploy method k8s, a pod role has a replicas option (core/resources/pod-roles/default.nix). When the pod of a new deploy does not become ready, the Deployment continues to run the previous image (core/specs/009-deployment-methods/k8s.md, SC-003).

How it is tested. core/components/hosting-provider/telemetry/alert-rules-test.nix and core/components/hosting-provider/monitoring-test.py test the alert rules. The check integrations-boot-counting tests the boot counting. The check deployment-k8s-hosts runs the role lldap with two replicas. No test measures availability.

What it does not do. There is no replica, no standby and no automatic failover for a stateful service. When a node fails, its services are down until the hosting provider repairs the node or restores it from a backup. The alert rules report a failure; they do not repair it.

Recovery point and recovery time

Target. The hosting provider guide gives the recovery point objective (RPO) of each backup as 24 hours plus the duration of the backup run. This value is a result of the default schedule. Each failed run adds 24 hours. The section Know how much data a restore can lose has the table of the backups, their schedule options and their restore points. The repository states no recovery time objective (RTO).

What the code does.

• Each off-site backup has a timer with OnCalendar = "daily" and Persistent = true as a default (mkOffsiteBackup in core/components/operator/app-resources.nix). The hosting provider can change the schedule with the options that the table of the guide names.
• backupRetention in core/components/operator/app-resources.nix gives how long the backups keep their data: 30 days for the backups of an application, and 7 daily, 4 weekly and 6 monthly snapshots for the backups that restore the infrastructure.
• The media backup copies the buckets of an application (core/resources/garage/media-backup.nix). The external copy copies the backup data to a second location (core/resources/garage/external-copy.nix). garage makes snapshots of its metadata (core/resources/garage/metadata-snapshots.nix).
• The alert rule backup-failed fires when a backup run fails. The rule backup-stale fires when the last good run of a backup is more than 36 hours old.
• The restore drill (core/components/shared/restore-drill.nix, option fediversity.restoreDrill.enable) restores the newest database backup into a scratch database each day. It records the duration in the metric fediversity_restore_drill_duration_seconds and the time of the last good run in fediversity_restore_drill_last_success_timestamp_seconds. The alert rules restore-drill-failed and restore-drill-stale use these.

How it is tested.

The nix-unit tests in core/setups/backup-retention-test.nix test the retention. No test measures the age of the newest backup at the time of a failure; the rule backup-stale monitors it on a live installation.

What it does not do. No backup is continuous, thus a restore loses the changes after the newest good run. The duration that the restore drill records is the restore time of one database only. It does not include the time to make a new node, to restore files and media, or to find the failure. No test measures the time of a full restore. After a restore, the hosting provider must do the erasures of personal data again; the part data-subject-requests covers this subject.

Downtime of an upgrade

Target. The repository states no target for the downtime of an upgrade.

What the code does.

• A deploy builds the new configuration of a node and activates it with switch-to-configuration switch (core/effects/common/deploy.sh). systemd restarts the units that changed. Thus a changed service is down for the time of its restart.
• A service with state uses systemd.stateMigrations. The unit <name>-migrate-pre.service makes a backup and runs the steps for the version range before the service starts. A failed step restores the backup and keeps the service stopped. A downgrade and a skipped release series are refused with exit status 78. Examples are core/components/hosting-provider/openbao/migrations.nix and core/components/hosting-provider/netbox/migrations.nix.
• Before an upgrade of an application, mkUpgradeSnapshot in core/components/operator/app-resources.nix makes a backup while the application is stopped. When the backup cannot start, it tries again each 30 seconds. After UPGRADE_BACKUP_TIMEOUT seconds (default 600) it refuses the upgrade with exit status 78. The application then stays stopped with its data at the old version.
• The API applies updates as the update policy of the deployment tells it: manual, automatic or window (api/src/fediversity_api/update_policy.py, api/src/fediversity_api/automatic_updates.py). With window, an operator selects the period of the day in which an update can start. A window is at least 15 minutes and at most 24 hours.

How it is tested. The check integrations-state-migrations tests the upgrade steps, the refusals and the restore after a failed step. The checks integrations-postgresql-major and integrations-postgres-upgrade-dump test an upgrade of PostgreSQL to a new major version. A check integrations-<service>-series for each service with state, for example integrations-openbao-series and integrations-garage-series, tests an upgrade between release series. api/src/fediversity_api/tests/test_update_policy.py tests the update policy. These tests show that an upgrade keeps the data. No test measures the downtime.

What it does not do. There is no rolling upgrade and no second instance that takes the requests during an upgrade of a node. The downtime of a service is the time of its restart plus the time of its upgrade backup and its migration steps. That time depends on the quantity of data. A refused or failed upgrade keeps the service stopped until a person corrects the cause.

Downtime of a move between providers

Target. Concepts states: "Migrations involve service unavailability in the order of a day (due to DNS propagation) but no data loss." The repository states no more exact value.

What the code does.

• On the source, migration-quiesce stop stops the services of the applications, and the export makes the data available to the target (core/components/shared/migration-quiesce.nix, core/effects/migration/run.sh). The source deployment is suspended.
• On the target, the import restores the data and starts the applications (core/effects/migration-restore/).
• The applications are down from the export until the target has restored them and the DNS records point to the target. The sections Export a deployment for a migration and Restore the apps of a migration of the guide give the procedure. To go back, the hosting provider resumes the source deployment.
• The DNS time depends on the TTL of the records. The record of a node and the NS records of the apex have a TTL of 300 seconds (nodeTtl and apexNsTtl in core/setups/groups/hosting-provider.nix). The default TTL of a zone from NetBox is 86400 seconds (core/effects/octodns/netbox-zone.py). deSEC does not accept a TTL below 3600 seconds (core/effects/octodns/octodns_fediversity_desec.py).

How it is tested. The checks integrations-migration-restore, integrations-migration-restore-boot-wait and deployment-migration-quiesce test the export, the restore and the stop of the services. The check deployment-registrar-transfer tests the transfer of a domain. api/src/fediversity_api/tests/test_migration.py, test_migration_restore.py and test_migrate.py in the same directory test the API side. These tests show that the data arrives. No test measures the downtime, and no test shows the value "in the order of a day".

What it does not do. A move is not live: the applications do not run on the source and on the target at the same time. The downtime includes the copy of the data, which depends on the quantity of data and on the network between the two hosting providers. The code does not measure it.

Number of deployments for each provider

Target. Concepts states: "We target small- to medium-scale hosting providers with 20+ physical machines, able to scale to at least 1,000 operators." The repository states no other quantity.

What the code does.

• Each operator deployment gets a /28 prefix from the NetBox prefix for operator deployments (core/effects/tf/incus/operator/bridge-isolation.tf). Thus the size of that parent prefix, which the hosting provider selects, limits the number of deployments.
• The operator deployments share the PostgreSQL node, the garage node and the other nodes of the hosting provider (Contract templates gives the isolation). The capacity of these single nodes limits the number of deployments.
• The API builds at most schema_build_concurrency schemas at the same time, with a default of 2 (api/src/fediversity_api/schema_build.py). It runs at most one deploy for each deployment at a time, because the deploy job uses the deployment ID as its concurrency key (api/src/fediversity_api/deployment.py).
• The alert rules disk-filling, disk-filling-trend and disk-full tell the hosting provider when a disk gets full.

How it is tested. No test. No check makes a large number of deployments, and the repository has no load test.

What it does not do. The code does not spread the load of one stateful service across nodes, and it has no limit that refuses a new deployment when the capacity is used. The value of 1,000 operators is a design goal that no measurement supports.

Resources of the smallest installation

Target. The repository states no minimum for memory, processors or disk, and no test finds one.

What the code does. The groups in core/setups/groups/ define the nodes of an installation. The table shows the nodes of each group. A published node has a public name; an internal node does not.

• The nodes of the group hosting-provider exist once for each hosting provider. The front end adds the nodes of the API and the panel; they are not in this table.
• The nodes of the group operator exist for each operator deployment. A deployment has the nodes of the applications that the operator enables and the internal nodes that support them.
• The root disk of an Incus instance has a default size of 50GiB (instanceRoot.size in core/components/hosting-provider/incus.nix), which Incus applies only when the storage pool can hold a quota.
• The code of core sets no memory limit and no processor limit on an instance.

The part deployment-topology describes the nodes and where they run.

How it is tested. No test. The NixOS tests run each node in a virtual machine or a container with the resources of the test, which are not a statement about a production installation.

What it does not do. The code does not check that a host has sufficient resources before a deploy.

Related attributes

• Security. The threat model gives the assets, the trust boundaries, the threats and controls and the accepted risks and open gaps. This page does not repeat them. The part access-model covers who can do what.
• Observability. The nodes push metrics, logs and traces to the telemetry node of the hosting provider (core/components/hosting-provider/telemetry.nix). The section Monitor the deployments of the guide covers the dashboards and the alert rules.
• Data protection. The page Data protection covers the retention and the erasure of personal data.

State lifecycle

This page tells what happens to the state of a running system when the code of the project changes, and when a service gets a new version. State is the data that a deploy does not make again: databases, buckets, secrets, the OpenTofu state, and the records of api/ and panel/.

The rule for each type of change

The sections below give the details and the files.

Two types of OpenTofu state

The code uses OpenTofu in two layers:

• The deployment graphs are in core/effects/tf/. Their state is in the terraform-backend service of the hosting provider.
• The on-node configurations are generated from Nix and applied by a unit on the node, for example core/resources/postgresql/provision.nix, core/resources/ldap/provision.nix, core/resources/garage/provision.nix and core/components/hosting-provider/openbao/runtime/module.nix. Their state is a terraform.tfstate file on the node.

In the two layers, the state is what connects the code to the objects. Thus a change that must move or adopt an object is a block in the OpenTofu configuration, or code written for that change.

An option is renamed

The panel and api/ store the configuration of a deployment as JSON. When an option of the operator configuration gets a new path, a stored configuration still has the old path.

core/lib/renames-for-source.nix makes a table of { from; to; } entries from the option tree: each rename that is declared with lib.mkRenamedOptionModule or lib.mkAliasOptionModule, and each rename of a key in a pinned upstream JSON Schema, which a settings-renames.json file beside the component declares. core/renames-log.json keeps the entries of earlier revisions, each with the since version that added it. api/package.nix puts the table in the package as renames.json, and its build fails for a rename that the log does not have.

core/renames-log.json has 0 entries.

Before api/ validates a stored configuration, _migrate_for_validation in api/src/fediversity_api/procedures.py moves each value from its old path to its new path (api/src/fediversity_api/migrate.py). A deployment row records the version of the log that its configuration agrees with (renames_version in api/src/fediversity_api/state.py), thus only the newer entries apply. A rename that cannot apply is reported as unmigrated, and the value stays where it is.

This mechanism changes only JSON. If the renamed option is part of the name of a deployed object, the next apply sees a different object: see the next sections.

A stored deployment also records the source of its modules. api/ stores a pin of its own revision for a deployment without a source, thus a rebuild of api/ does not change the schema of a stored deployment until its next deploy. See "Default-source baseline freeze" in api/docs/design/schema-and-sources.md.

A value changes

For most values of the operator configuration, and for the environment variables of api/ and panel/, the deploy writes the value into a configuration file or into the environment of a unit. When the file or the environment changes, the unit starts again. Thus the next deploy applies the new value.

Some applications keep a value in their own state, and that value wins over the deployed value. The code prevents this for these values:

• Pixelfed writes a setting into its config_cache table at the first read, and from then on the row wins. core/components/operator/pixelfed/node.nix sets ENABLE_CONFIG_CACHE = false, thus the value of the .env file applies.
• PeerTube reads local-production.json in its configuration directory over the deployed configuration. Its web edit of the configuration writes that file. core/components/operator/peertube/node.nix turns off the web edit, and the preStart of peertube.service removes the file.
• The secrets of the operator are in OpenBao KV (api/src/fediversity_api/operator_secrets.py). When the rename table renames the field of a secret, a submit copies the value from the old name to the new name. When the operator clears a secret, the next deploy deletes its KV entry with all its versions (core/effects/tf/operator-provision/kv-clear.tf).

These values apply only to new objects, or stay with the application, by design:

The docs build generates this table and the list above from core/docs/md/technical-design/state-lifecycle.nix. The build fails when an option of a row is not in core/components/operator/operator-facing-options.csv, or when a file that a row names no longer has the code that the row describes.

A resource gets a new address

When the address of a resource changes and nothing else is done, OpenTofu destroys the object of the old address and creates an object at the new address. For an object that holds data, the new object is empty.

A moved block prevents this: it tells OpenTofu that the object of the old address is the object of the new address. The tree has these:

• core/effects/tf/operator-provision/garage.tf has the only moved block in a .tf file, for a terraform_data resource with a new name.
• core/lib/tf-labels.nix (movedBlocks) generates the moved blocks of the on-node configurations, from the labels <prefix>_<key> that the states of deployed nodes have, to the labels of the current code.
• core/resources/garage/provision.nix changes its state file with jq before the apply, because a moved block between two providers needs a provider that reads the old type.
• core/components/hosting-provider/incus.nix repairs its state file in the preStart of the unit, before the moved blocks apply.

The limits:

• A moved block and a removed block accept only static addresses. The comment before garage_bucket.bucket in core/effects/tf/operator-provision/garage.tf gives the result: a resource whose for_each keys change with each deployment cannot get such a block.
• No .tf file in the tree has a removed block.
• A moved block does not rename the object itself. A database, a bucket or a node with a new name is a new object.

An object exists, but the state does not have it

An import block adopts the object, so that the apply does not create it again:

• The on-node configurations of PostgreSQL, lldap and garage adopt an object that exists and is not in the state, for example a database that ensureDatabases made.
• core/effects/tf/incus/operator/registrar.tf adopts a domain claim in OpenBao, a domain that is in the registrar account already, and a name server group.

notes/forward-recovery.md gives the rules for an apply that failed: there is no rollback, and the same apply runs again.

A resource is dropped

When a resource leaves the configuration, the next apply destroys its object. This is the default of OpenTofu in the two layers:

• In an on-node configuration, an object that leaves the configuration is dropped. For PostgreSQL, a grant is revoked, and a database is dropped with its data (core/resources/postgresql/provision.nix).
• In a deployment graph, the destroy-time provisioners of a resource release the objects that are not in the state, for example DNS records.

Some resources have a different rule:

The docs build generates this table. It fails when a .nix or .tf file below core/ has prevent_destroy = true or destroy = false and no row names the file, and when a file of a row no longer has that code.

Gaps:

• An object that is in no state has no deletion path. The on-node configuration of PostgreSQL says so for its own objects: an object that is in neither the configuration nor the state is not touched, thus a name that left the configuration before the first apply stays.
• Only the node of the state backend has a guard against a replacement. For each other node, a change that replaces the node gives a new node, and the data on the old root disk is gone unless a volume or a backup holds it.

An object outside OpenTofu is dropped

A script or a unit makes some objects, not an OpenTofu resource. Such code follows a drop only when it lists the objects that exist, compares them with the objects that the code states, and removes the others. These do:

The docs build generates this table. It fails when a file of a row no longer has the code that removes the object. The list of backups with a sweep comes from the calls of mkRetiredRepository. The build does not find a new removal that has no row.

Gaps:

• When the rename table renames the field of a secret of the operator, the KV entry of the old name stays.
• When a role leaves core/components/operator/mastodon/role-groups.nix, the holders keep it on Mastodon (core/components/operator/mastodon/ldap-roles.rb). No path of the configuration gets to this.
• The sweep of retired restic repositories serves only the backups in the table above. Other restic repositories of a backup that stops stay, for example the repository of the dump before a PostgreSQL upgrade.

An application is disabled

The option deleteData of an application is in core/components/sharedOptions.nix. Its default is false.

• When enable becomes false and deleteData is false, the data stays: the database, the buckets, the generated secrets and the cache. A new enable uses the data again.
• When enable is false and deleteData is true, the next deployment deletes the data. purgedApplications in core/setups/wiring/node-selection.nix finds these applications. core/setups/wiring/supporting-requests.nix leaves their requests out, thus the operator graph deletes their buckets, databases and keys, and the valkey node deletes their cache. core/effects/tf/operator-provision/openbao.tf deletes their KV directories (purged_kv_dirs, from core/setups/wiring/purged-kv.nix).
• A *-backups bucket stays for the grace period, as at a destroy.

The operator documentation of the option is in the operator configuration, and its role for erasure is in Data protection.

A deployment is destroyed

A destroy of a deployment runs tofu destroy on the operator graph. It removes the nodes. core/effects/tf/operator-provision/postgresql.tf drops the databases and the login roles of the deployment on the shared PostgreSQL node: their names have the namespace of the deployment as a suffix, and this graph is the only actor that runs again when the deployment ends. terraform_data.after_instances in core/effects/tf/operator-provision/openbao.tf deletes what remains below the KV prefix of a namespaced deployment after the instances are gone. terraform_data.retire_bucket empties and deletes each bucket that is not a backup bucket.

The destroy keeps this state:

The check core/checks/integrations/destroy-reports-left-behind covers the last row. The operator steps for a retained bucket are in Know what a destroy keeps. An export for a move to a different hosting provider must be complete before the destroy: no key of the operator can read a retained bucket.

A destroy of all nodes of the hosting provider also destroys the node that keeps the OpenTofu state. Thus the run first moves the state to a local file (core/checks/integrations/destroy-detaches-state).

A service gets a new version

Each service that keeps state has a systemd.stateMigrations.<name> entry, in most cases in a file migrations.nix beside the service. The module is nix/contracts/nixos/modules/system/state-migrations.nix. An entry gives the version of the service in the current code. A unit <name>-migrate-pre runs before the service, compares that version with the version in a record file on the node, and decides:

flowchart TD
  start["The unit `<name>-migrate-pre` starts"] --> record{"Is there a record of the version?"}
  record -- "no" --> fresh["Record the version. Run `onFreshInstall` if the entry has one and the state is new."]
  record -- "yes, the same version" --> none["Do nothing"]
  record -- "yes, a later version than the code" --> down{"What is the value of `onDowngrade`?"}
  down -- "refuse" --> stop["Exit with status 78. The service stays stopped."]
  down -- "warn or ignore" --> older["Record the older version. Run nothing."]
  record -- "yes, an earlier version than the code" --> skip{"Is the step larger than `maxSkip` permits?"}
  skip -- "yes" --> stop
  skip -- "no" --> backup["Run `backup`, then `steps` and `onUpgrade`"]
  backup --> service["The service starts"]
  service --> post["The unit `<name>-migrate-post` runs `onUpgradePost` and records the version"]

The rules of the module:

• The backup. backup runs one time for each upgrade, before each other command of the upgrade.
• The downgrade. onDowngrade is refuse by default. Exit status 78 tells systemd not to start the unit again. With warn or ignore, the unit records the older version and runs nothing: these are services that also read the state of a later version.
• The largest step. maxSkip is not set by default, and then each step to a later version is permitted. With major, a step across more than one major release is refused. With minor, a step across more than one minor release is also refused.
• The restore. When the entry has a restore command, the unit records an upgrade that is not complete. A deploy of the version from before the upgrade then runs restore. A deploy of a third version is refused. Without restore, the way back is a manual restore of the backup.
• No record. When freshInstallTest is null and the node has no record, the unit records the version and does not migrate. Thus the first deploy of an entry onto state that exists does not make a backup.

What an entry does not do:

• It moves no object and changes no OpenTofu state.
• In most entries, it runs no schema command: the service migrates its own schema when it starts. The column "Upgrade command" of the table shows the entries that do run a command. For example, core/components/operator/mastodon/migrations.nix runs the database migrations of Mastodon, and the post-deployment migrations only after the services are up again.
• No entry has a command for one specific version: the list steps is empty in each entry.
• An entry whose version is a release series, for example 2.5, sees no change at a patch release. Thus a patch release makes no backup.

These files below core/components and core/resources define an entry. core/setups/nixos/postgresql.nix and core/setups/nixos/mysql.nix define the entries of PostgreSQL and MariaDB.

• core/components/dev/forgejo/migrations.nix
• core/components/hosting-provider/audit-shipper-migrations.nix
• core/components/hosting-provider/bind-migrations.nix
• core/components/hosting-provider/netbox/migrations.nix
• core/components/hosting-provider/openbao/migrations.nix
• core/components/hosting-provider/spire/migrations.nix
• core/components/hosting-provider/telemetry/migrations.nix
• core/components/hosting-provider/terraform-backend/migrations.nix
• core/components/hosting-provider/windmill/migrations.nix
• core/components/operator/mastodon/migrations.nix
• core/components/operator/peertube/migrations.nix
• core/components/operator/pixelfed/migrations.nix
• core/components/shared/authelia/migrations.nix
• core/components/shared/lldap/migrations.nix
• core/components/shared/valkey/migrations.nix
• core/resources/garage/migrations.nix
• core/resources/smtp/maddy-migrations.nix

The table shows the entries of the nodes of core with the default applications and the configuration of the tests. The docs build generates it from config.systemd.stateMigrations of each node of the groups hosting-provider and operator of core/testers/groups.nix. The column "Backup before the upgrade" is written by hand in core/docs/md/technical-design/state-lifecycle.nix, and the build fails when it is not in agreement with the backup script of the entry. A row is one entry, or the entries with the same name and the same values on more than one node. The version of each entry is the version of the package, or its release series (garage, for example), or its major version (PostgreSQL). An entry with a restic snapshot makes it on a node that has the offsite backup for that service. Without the offsite backup, the entry makes a local copy or a dump, or no backup.

These nodes of that configuration have no entry: hosting-provider/otel-collector, operator/grafana.

The table does not read the overview tag, the hypervisor and the development hosts. These entries are not on a node of that configuration, thus the table does not show them:

• fediversity-audit-shipper of core/components/hosting-provider/audit-shipper-migrations.nix. It refuses a downgrade and makes a local copy of its state directory.
• forgejo of core/components/dev/forgejo/migrations.nix.
• mysql of core/setups/nixos/mysql.nix, for a node that runs MariaDB.

Gaps:

• The postgresql entry itself does not limit the step. The guard of the PostgreSQL module, below, permits only a step of one major version.

PostgreSQL has one more rule. Each major version keeps its cluster in a directory of its own, thus a new major would start with an empty cluster. core/setups/nixos/postgresql.nix sets the major in the code. When the major goes up by one, its guard runs pg_upgrade --copy from the old cluster. For each other change of the major, it keeps PostgreSQL stopped with exit status 78. MariaDB has a rule of its own in core/setups/nixos/mysql.nix: the module runs mariadb-upgrade after each start of the server, because an upgrade is complete only after that command.

The maintainer steps for an update, and the reasons for each service, are in Update nixpkgs. Each entry has a check in core/checks/integrations/, for example garage-series and openbao-series.

The schema of the api or the panel changes

The two components keep their records in a PostgreSQL database of the hosting provider. Each applies its own migrations, and each has a systemd.stateMigrations entry that only records the version, refuses a downgrade, and makes the backup.

• api/ applies the Alembic revisions when it opens its store (_migrate in api/src/fediversity_api/state.py). A PostgreSQL advisory lock keeps a second process out of the same migration. api/ refuses a database at a revision that its release does not know. The revisions only go forward.
• panel/ runs manage migrate in the preStart of panel.service (panel/nix/configuration.nix), each time the package changes. Django does not refuse a database that later migrations changed, thus the version record is the only guard against a downgrade. The panel has no migrations that go back.
• The version of each entry is the number of the last migration file. Thus the version changes only when the schema changes. For the panel, the migrations of the Django apps from nixpkgs are not in the version.
• The backup comes from mkUpgradeBackupFor in core/components/operator/app-resources.nix: on a node with the offsite backup of core/components/shared/postgres-backup.nix, a dump of the database before the upgrade. Without that backup, the entry makes no backup.
• The largest step is not limited: the entries do not set maxSkip. Alembic and Django apply each migration between the two versions in sequence.

The two entries are not in the table of the section before, because the nodes of api/ and panel/ are not nodes of core.

An effect fails

An effect can fail, or a person or a timeout can stop it, at any point. No effect goes back to the state from before the run. Each effect is recovered forward: run it again. For a deployment graph, notes/forward-recovery.md gives the rules and tells how to read a failure.

The docs build generates this table. It fails when a file of a row no longer has the code that does what the row says. The build does not find a new retry or a new effect that has no row.

Gaps:

• No effect tries a failed run again by itself. The retries in the table are inside one run.
• A restore of a migration and some operations of an application do not say that a second run is safe.

What this page does not cover

• The list of each place where state is kept is in the part state-register.
• The backups, their retention and their restore are in the hosting provider guide.
• A move of an operator between hosting providers is in the part capability-flows.

State register

This part lists each store of state of the software: where it is, which component owns it, whose personal data it holds, how long it keeps the data, how it is protected, and how to erase data from it. It gives the facts for the record of processing that the data protection page assigns to the operator and to the hosting provider. It is not legal advice, and it gives no legal classification.

How the register is made

The tables on this page are generated at the documentation build from two inputs:

• The declaration core/docs/md/technical-design/state-register-stores.nix. It gives, for each store, the purpose, the personal data, the retention, the encryption, the backup and the procedure to erase, with the files that these statements come from.
• The code. The build reads the databases, the buckets, the valkey namespaces and the persistentPaths that the components request, the tables that the migrations of api/ create (api/src/fediversity_api/state_migrations/versions/), and the models of the panel (panel/src/panel/models.py). The components are those of an example deployment with each application enabled.

The build compares the two inputs. The check core-docs fails when:

• an item in the code is in no store of the declaration,
• the declaration names an item or a source file that does not exist,
• a component owns no store and is not in the list of components without a store.

The check sees only the items that the code declares as data. It does not see a state directory that a module sets on a node, for example /var/lib/maddy. The declaration gives these paths by hand.

"Open" in a table means that no source in the tree gives the fact. The hosting provider finds it in its setup or in the upstream documentation, see the unknowns of the data protection page.

Which store holds which

Some stores are inside other stores. A statement about the outer store, for example its encryption, also applies to the stores inside it.

flowchart TD
  pools["Incus storage pools on the hypervisor"]
  nodes["Paths on the nodes"]
  postgresql["PostgreSQL server"]
  garage["Garage"]
  databases["Databases of the applications and the services"]
  buckets["Media and telemetry buckets"]
  backups["Backup buckets"]
  pools --> nodes
  nodes --> postgresql
  nodes --> garage
  postgresql --> databases
  garage --> buckets
  garage --> backups
  databases -- "backup" --> backups
  nodes -- "backup" --> backups

Location and owner

The column "Declared in the code" gives the items that the build found in the code for the store. The databases of the panel and of api/ and their backup buckets are not in the example deployment, thus the build does not check them.