Capability flows

This part gives the run-time flow of each capability: install, configure and update an application, manage users, and switch to and from a different hosting provider. Each flow has a sequence diagram, the code paths, the stores that change, the failure paths, and the work that is planned.

The flows have these participants:

• The operator is the customer of the hosting provider. The operator uses a browser.
• panel is the Django front-end (panel/). It keeps the accounts, the stored configurations and the permissions.
• api is the FastAPI service (api/). It validates a configuration and starts a deploy.
• core is the Nix and OpenTofu code (core/). It makes the schema, and it deploys.

The design of the API gives the contract of each call. The architecture of the panel gives the design of the panel.

Calls that each flow uses

Each call from the browser to api uses the same permission check:

1. The browser gets a bearer token from panel at api/token/ (views.ApiToken, panel/src/panel/tokens.py). panel signs the token with its SECRET_KEY. The token is valid for API_TOKEN_TTL.
2. The browser sends the call to api with the token.
3. api does not examine the token. It sends the procedure name, the scope and the headers to the callback of panel at api/authz/ (api/src/fediversity_api/authz.py, views.AuthzCallback).
4. panel answers allow, and for an allowed call the subject and the deployments that the caller can use. panel denies a procedure that has no rule in AUTHZ_PROCEDURE_RULES.
5. If panel denies the call, api answers authz_denied (HTTP 403).

The diagrams show this check as one step, "check the permission".

Install an application

The operator fills in the form of a deployment, enables an application, and deploys. The same call does the first install and each later change.

sequenceDiagram
    actor O as Operator
    participant P as panel
    participant A as api
    participant C as core
    participant I as Provider infrastructure
    participant N as Node
    O->>P: save the form
    P->>P: store the configuration
    O->>A: POST /deployments/submit/{id}
    A->>P: check the permission
    A->>C: build the schema of the module source
    A->>A: migrate renamed options, validate
    A->>C: evaluate the names of the deployment
    A->>A: claim the domain, store the build as queued
    A-->>O: build_id, status queued, position
    A->>C: run tf-run.sh
    C->>I: NetBox address, Incus instance and volume
    C->>I: bucket, database, secrets in OpenBao
    C->>I: hold the zone, register the domain, publish the records
    C->>N: switch to the NixOS configuration
    N->>N: order certificates with ACME, start the services
    C->>N: read the state of the units
    C-->>A: exit code and outputs
    A-->>O: build events through Centrifugo

Code paths:

• panel: views.DeploymentForm shows only the deployments of the operator. static/deployment_submit.js saves the form in panel, then calls api.
• api: the route deployment_submit (routes/deployments.py) calls procedures.deployment_submit. In sequence, this function:
  1. refuses a module source that is not on the trusted-source allowlist (source_allowlist.py),
  2. builds the schema and the rename table of the module source (schema_build.py),
  3. applies the renames and validates the configuration with jsonschema,
  4. takes the secrets of the operator out of the configuration (operator_secrets.py),
  5. refuses a full queue,
  6. evaluates the names of the deployment and claims the apex of the domain (apex_claims.py),
  7. makes the DNS token of the zone (zone_tokens.py), writes the secrets of the operator to OpenBao, and stores the deployment (state.upsert_deployment),
  8. starts the build (deployment.start_deployment).
• The build runs as a subprocess of api, or as a Windmill job when windmill_base_url is set (deployment.py). The runner starts nix develop .#<attribute> --impure --command tf-run.sh (_runner.py).
• core: tf-run.sh (core/effects/common/lib.nix, core/effects/tf/common/bootstrap-apply.sh) applies the OpenTofu graph of core/effects/tf/incus/operator/, then gives the outputs as JSON. The state of the graph is in the HTTP state backend.
• DNS: dns-lane-zone.tf and registrar.tf hold the zone, register the domain and wait until the parent zone delegates. dns-lane-records.tf publishes the names of each node and waits until the names resolve. All of this occurs before the switch of the node, because the certificate order of the node needs the records.
• TLS: core/resources/ssl/acme.nix sets security.acme. A node orders the certificate of an ordinary name with the HTTP challenge, and the certificate of a wildcard name with the DNS-01 challenge.
• After the switch, null_resource.deploy_verdict reads the units of each node. A node with a unit that continues to restart fails the deploy.

Stores that change:

Failure paths:

• A configuration that fails the schema gives invalid_configuration (HTTP 400). Nothing is stored.
• A full queue gives queue_full (HTTP 409). A domain that a different deployment holds gives conflict (HTTP 409).
• A schema build that passes its time limit gives build_timeout (HTTP 504).
• A deploy that fails sets the build to failed. api publishes the end of the log. The resources that the apply made stay in the state, thus the next submit continues from them.
• When api starts again, reconcile_orphaned_builds (deployment.py) marks the builds that lost their process.

Planned:

• Builds of one deployment that run in parallel (issue 242 of fediversity/fediversity).
• The configuration of the deploy method in the panel (issues 723 and 1071).

Configure an application

The form of an application comes from the options of its NixOS module. No form is written by hand.

sequenceDiagram
    actor O as Operator
    participant F as Form in the browser
    participant P as panel
    participant A as api
    participant C as core
    O->>P: open the form of the deployment
    P-->>F: stored, deployed and default module source
    F->>A: GET /schema and /schema/migrations
    A->>C: build provider-configuration-schema and configuration-renames
    C-->>A: JSON Schema and rename table
    A-->>F: JSON Schema and rename table
    F->>F: apply the renames, show the form
    O->>F: change a value
    F->>F: validate with AJV
    O->>F: deploy
    F->>P: save the configuration
    F->>A: POST /deployments/submit/{id}
    A->>A: apply the renames, validate with jsonschema
    A->>C: run tf-run.sh

Code paths:

• Schema: core/lib/json-schema.nix makes a JSON Schema from the options of frontendConfiguration (core/lib/data-model.nix). Add an application and the data model give the options.
• api: schema_build.py builds the outputs provider-configuration-schema and configuration-renames of the module source. A build has a time limit, and api limits the number of builds that run together. api keeps the result for each source in memory. The routes are in routes/schema.py.
• Form: panel/forms-island-adapter/panel-adapter.ts gets the schema and the rename table, validates each change with AJV, and shows the difference between the schema of the stored source and the schema of the source that is deployed.
• Validation occurs two times: in the browser with AJV, and in api with jsonschema (procedures._checked_submission). The check in api is the authority.
• Apply: the apply is the submit of the install flow. A changed option changes the NixOS configuration of a node, and the deploy switches the node to it.

Stores that change: the stored configuration in panel, then the stores of the install flow.

Failure paths:

• A module source that is not on the allowlist is refused before a build.
• A configuration that fails the schema gives invalid_configuration.
• POST /deployments/preview/{id} does the checks of the submit and shows the nodes that a submit deploys. It deploys nothing.

Planned:

• Resource versions, which let api find a difference between the environments of the provider and the pinned data model before a deploy (issue 634).
• To follow the new revisions of a fork of the module source (issue 3053).

Update an application

An update is a submit with a newer module source. The module source selects the revision of the application modules.

sequenceDiagram
    actor O as Operator
    participant P as panel
    participant A as api
    participant C as core
    participant N as Node
    participant S as Object store
    alt policy manual
        P-->>O: notice that a newer revision exists
        O->>A: POST /deployments/submit/{id} with the update applied
    else policy automatic or window
        A->>A: the scheduler submits the stored configuration again
    end
    A->>A: resolve the null source to the revision of the provider
    A->>A: apply the renames, validate
    A->>C: run tf-run.sh
    C->>N: switch to the new NixOS configuration
    N->>N: guard reads the recorded version
    alt downgrade or skipped release series
        N-->>C: the guard fails, the service does not start
    else upgrade
        N->>N: stop the service, dump the database
        N->>S: restic snapshot of the dump
        N->>N: start the service, run the migrations after the deployment
        N->>N: record the new version
    end
    C->>N: read the state of the units

Code paths:

• Policy: api/src/fediversity_api/update_policy.py has the policies manual, automatic and window. The operator sets the policy with PUT /deployments/update_policy/{id} (views.DeploymentUpdatePolicy, static/update_policy.js). static/update_notice.js shows that a newer revision exists. The form has a check box beside the deploy button: when it is off, the submit has apply_update set to false (static/deployment_submit.js) and api keeps the deployed revision.
• Scheduler: automatic_updates.run_scheduler submits the stored configuration with a null module source. It tries each revision of the provider one time for each deployment (state.claim_update_attempt), and writes the audit action automatic_update. It does not update a deployment that is suspended or not active. A deployment whose deployed module source is not the source of the provider is due too (run_due_updates), thus the scheduler moves a deployment from a fork to the source of the provider.
• Version boundary: version_boundary.py selects, for the layers core/resources and core/effects, the newer of the pinned revision and the revision of the provider.
• Guards: each application has a systemd.stateMigrations entry. core/components/operator/mastodon/migrations.nix records the version, refuses a downgrade or a skipped minor release series, dumps the database before an upgrade, and runs the migrations after the services are up again. The guards of PeerTube and Pixelfed refuse a downgrade and dump the database before an upgrade. core/setups/nixos/postgresql.nix guards the major version of PostgreSQL and uses pg_upgrade.
• Backup before the upgrade: core/components/shared/database-upgrade-backup.nix sends the dump to the restic repository <server>-upgrade in the backup bucket of the application. The repository keeps the snapshots of the 2 newest upgrades. Without an object store, the dump stays on the node.

Stores that change: the stores of the install flow, the database of the application (schema migrations), the version file of each guard on the node, and the restic repository of the upgrade backups.

Failure paths and rollback:

• A guard that fails keeps the service down. deploy_verdict then fails the deploy.
• The deploy method k8s sets a pod that does not become ready back to the previous image (kubectl rollout undo in core/effects/k8s/effect.nix) and fails the deploy. The state on the volume of the pod stays as it is. No other code path rolls back a deploy automatically.
• A guard refuses a downgrade, thus a submit of the older revision does not go back after a migration of the data.
• The way back is the dump from before the upgrade. The hosting provider guide gives the restore procedures.
• A submit with apply_update set to false uses the module source of the last build that was deployed.

Planned:

• Application versions that do not depend on the revision of the provider (issue 304).
• Resource versions (issue 634).
• An automatic rollback of a failed update, data included (issue 3052).
• A guide for the operator through the options that an update adds or changes (issue 3054).

Manage users

There are two levels of users: the account of the operator in panel, and the users of each deployment. Each deployment has its own directory (lldap at users.<domain>) and its own identity provider (Authelia at auth.<domain>).

sequenceDiagram
    actor O as Operator
    actor U as User of an application
    participant P as panel
    participant IdP as Identity provider of the hosting provider
    participant L as lldap of the deployment
    participant Au as Authelia of the deployment
    participant K as OpenBao KV
    participant App as Application
    O->>P: log in
    P->>IdP: OpenID Connect authorization
    IdP-->>P: identity of the operator
    O->>L: add a user, put the user in a group
    U->>App: log in
    App->>Au: OpenID Connect authorization
    Au->>L: check the password and the groups
    Au-->>App: identity of the user
    L->>K: timer writes the members of the role groups
    App->>K: timer reads the members
    App->>App: set the role of each member

Code paths:

• Log-in to panel: panel/src/panel/oidc.py uses OpenID Connect when OIDC_ISSUER, OIDC_CLIENT_ID and OIDC_CLIENT_SECRET_FILE are all set (settings.py). If not, panel uses the log-in of Django. A wallet log-in (panel.wallet.WalletAuth) opens only an account that linked the wallet.
• Directory: core/components/shared/lldap and core/components/shared/authelia. core/resources/ldap/lldap-provider.nix makes the first admin user.
• Single sign-on: core/resources/oidc/authelia-oidc-provider.nix makes an OpenID Connect client for each application. allowedGroups becomes an Authelia policy that allows only the members of those groups. The application reads its client from the oidc contract (for example core/components/operator/mastodon/oidc.nix).
• Groups to roles: core/resources/ldap/application-role-groups.nix and the role-groups.nix of each application declare the role groups, for example mastodon-owners, mastodon-admins and mastodon-moderators. core/resources/ldap/publish-roles.nix writes the members of the role groups to the OpenBao KV on a timer, only when they change. The ldap-roles.nix of each application reads the members on a timer and sets the roles in the application.
• Legal roles gives the role of each group in each application. The hosting provider guide and the user guide of the panel give the procedure.

Stores that change: the database of lldap, the members entry in the OpenBao KV, and the accounts and roles in the database of each application. The account of the operator is in the database of panel.

Failure paths:

• A role sync that fails does not fail a deploy. The timer tries again at its next interval.
• A role change is not immediate: the publish and the sync each run on a timer.
• Without the OpenBao backend for secrets, the role sync does nothing.
• A user who is not in an allowed group cannot log in to the application.

Planned: api has no route, and panel has no view, that changes the users of a deployment. The operator uses the web interface of lldap. Issue 3055 asks for the decision if the panel must manage users.

Switch hosting provider

A switch has two parts: the export at the source provider, and the import at the target provider. The migration string moves between them. The operator carries the string.

sequenceDiagram
    actor O as Operator
    participant SA as api of the source
    participant SN as Nodes of the source
    participant S3 as Object store of the source
    participant R as Registrar
    participant TA as api of the target
    participant TC as core of the target
    participant TN as Nodes of the target
    O->>SA: GET /migration-string/{id}
    SA->>R: read the auth code, unlock the domain, remove the DNSSEC keys
    SA->>SN: stop the application services
    SN->>S3: restic snapshot of each application
    SA->>SA: mark the deployment as exported, submit suspend
    SA-->>O: migration string
    O->>TA: POST /migration-import/{id} with the string
    TA->>TA: check the module source, remove the provider settings of the source
    TA->>TC: dry evaluation
    TA->>TA: write the snapshot pointers and the auth code to OpenBao
    TA->>TC: run tf-run.sh
    TC->>R: transfer the domain with the auth code
    TC->>TC: hold the zone, wait for the delegation, publish the records
    TC->>TN: switch to the NixOS configuration
    TN->>S3: restore each application from its snapshot
    TC->>TC: delete the pointers and the auth code
    SA->>TN: zone transfer, copy the records to the old zone

Code paths:

• Panel: static/migration_export.js calls GET /migration-string/{id} and shows the string. static/migration_import.js makes a deployment and sends the string to POST /migration-import/{id}.
• Domain: migration.migration_export (api/src/fediversity_api/migration.py) first calls release_domain, which reads the auth code at the registrar, unlocks the domain and removes its DNSSEC keys. This occurs before the snapshot, thus a registrar that fails stops no service.
• Export: migration_export then runs core/effects/migration/run.sh. This script stops the services of each node with migration-quiesce (core/components/shared/migration-quiesce.nix), then makes a snapshot of each backup unit. Then api marks the deployment as exported and submits the action suspend.
• Import: migration.migration_import parses the string, removes the settings that belong to the source provider, and changes the configuration to the shape of the target. import_check.py evaluates the result with DRY_RUN=1 before api stores it. Then the import is the submit of the install flow.
• Restore: migration_kv.py writes the snapshot pointers and the auth code to the OpenBao KV of the deployment. The OpenTofu graph gets only the snapshot references. core/effects/tf/incus/operator/restore-migration.sh reads the pointers, restores each unit, and deletes the pointers after the restore.
• Domain transfer: registrar.tf gives the auth code to openprovider_domain.operator, then null_resource.drop_domain_transfer deletes the code from the KV.
• DNS change: the deploy at the target holds the zone on the nameservers of the target (dns-lane-zone.tf). await_delegation waits until the parent zone delegates to them. Then the deploy publishes the records of the nodes.
• Zone follow: when the source holds the zone at deSEC, zone_follow.py at the source copies the records of the target to the old zone until the old delegation is out of the caches of resolvers. It asks the nameservers of the target for a zone transfer that it signs with the transfer_key of the migration string.

The hosting provider guide gives the procedure of the export and of the restore. The contract templates give the switching clauses.

Stores that change:

Trust between the two providers

• The two providers have no direct connection for the import, and no common account. The operator carries the migration string. The string is a bearer capability: its holder can read all data of the deployment.
• The target does not trust the configuration of the source. It validates the configuration against the schema, and accepts only a module source that is on its trusted-source allowlist.
• The target removes the provider settings of the source from the string (operator_settings, RUNTIME_SETTING_KEYS).
• The target reads the snapshots directly from the object store of the source, with the credentials of the string.
• The target allows a zone transfer from the addresses in transfer_sources, to the holder of the transfer_key.
• The string carries no registrar login.

Secrets that move

The migration string carries:

• for each backup unit, the S3 endpoint, bucket, access key and secret key of the snapshot repository, the password of the repository, and the snapshot reference,
• the keys that decrypt the data of the applications,
• the auth code of the domain, when the source has the domain in its registrar account,
• the TSIG key of the zone follow.

api does not keep the string, and does not write it to a log or to an audit event. At the target, the pointers and the auth code are in the OpenBao KV of the deployment, and not in the OpenTofu state or in the Windmill job. The deploy deletes them after their use. The threat model and data protection give the related risks and duties.

Downtime

The applications are down from the export until the target restores them. The export stops the services before the snapshot, thus no write is lost. The time has these parts: the snapshot, the transport of the string by the operator, the deploy at the target, the restore, and the DNS change. The code sets no limit on the full time, and the tree has no measured value. The limits that the code does set are:

• the dry evaluation of the import: 1800 seconds, then build_timeout,
• the cache time that the zone follow uses: at most 604800 seconds (TTL_CEILING in zone_follow.py).

To roll back, resume the source deployment: its data stays, and its services start again. After a domain transfer, the resume of the source fails.

Different versions at the two providers

• The migration string must name its module source. An import without a module source is refused, because the revision of the target can be different from the revision of the source.
• The target builds the schema of that module source, and applies its rename table to the configuration.
• The target deploys with the version boundary of the update flow: the pinned revision for the application modules, and the newer revision for core/resources and core/effects.
• The dry evaluation refuses a configuration that the pinned revision cannot deploy with the method and the environments of the target. This occurs before api stores the deployment.
• A source node that runs an older revision does not give the keys of its applications. The export continues, and the string has no keys for that node.

Failure paths:

• If a snapshot or the suspend fails, the export starts the services again and the deployment is not exported. If a snapshot fails, the export also locks the domain again.
• If api cannot read the auth code, the export fails and does not change the source.
• A second export of a suspended deployment gives conflict.
• A string in an old form, or with a module source that is not on the allowlist, gives invalid_configuration.
• If the restore fails, the deploy fails. The pointers stay in the KV, thus a new deploy restores again.
• A domain transfer that the registry refuses shows as transfer_failed in GET /domains.

Planned:

• Resource versions, which let the target reconcile a difference in the environments of the two providers (issue 634).
• The domain calls need the registrar command of api, which exists for openprovider. A second registrar is planned (issue 1908).