Traceability of the capabilities

The deliverable is a technical design for a hosting panel with five capabilities: to install applications, to configure them, to update them, to do user management, and to switch customers to and from other hosting providers. This part connects each capability to the requirements in the specifications, to the code, to the tests and to the docs pages. It also shows where the code has no requirement.

How to read the tables

The specifications are in the directories core/specs, api/specs and panel/specs. A specification gives its requirements as numbered items, for example FR-001. The overview of the specifications gives the specifications of core.

A capability has more than one function. Each row of the tables is one function. The list of functions is made by hand in core/docs/md/technical-design/traceability.nix. The build of the docs stops when a row gives a path that is not in the repository, or a requirement that is not in the spec file. The build does not examine whether a test proves a requirement.

The build calculates the status of a row with these rules, in this sequence:

  1. no spec: the row has no requirement.
  2. planned: each requirement of the row has a marker (planned, #<number>) in its spec file. The status gives the issues.
  3. partial, planned: some requirements of the row have that marker.
  4. partial: the row has a gap, or the row has no test, or the status of a cited spec file is not Implemented. A spec file with the status Partially implemented does not say which of its requirements are open, thus each row that cites it is partial.
  5. built: all other rows.

A requirement that the code does not have gets the marker (planned, #<number>), where the number is the issue for the work. At this revision, no requirement in the three directories has the marker.

A spec file that has the section Known Deviations gives there the differences between the code and the design, and the issue of each difference when one exists. The table of the specification files gives these issues. The build does not read the state of an issue, thus an issue in the table can be closed.

Requirements, tests and status

Capability Function Spec and requirement Test Status
Install The catalog of the applications that an operator can select core/specs/003-applications-catalog/spec.md FR-001, FR-002, FR-003, FR-004, FR-005; api/specs/003-application-catalog/spec.md FR-001, FR-002, FR-003, FR-004 core/checks/applications-catalog.nix, api/src/fediversity_api/tests/test_http_application_list.py built
Install The operator submits a deployment, and a job builds and deploys it panel/specs/002-deployment-lifecycle/spec.md FR-001, FR-002; api/specs/001-deployments-and-builds/spec.md FR-001, FR-003, FR-011; api/specs/005-job-execution/spec.md FR-001, FR-002, FR-003, FR-004, FR-005; core/specs/004-deployment-programs/spec.md FR-001 panel/src/panel/tests/test_deployment_form.py, panel/checks/js-tests/deployment-submit.test.js, panel/checks/apps-panel, api/src/fediversity_api/tests/test_deployment.py, api/src/fediversity_api/tests/test_http_deployment.py, api/checks/apps-api, core/checks/deployment/tf-incus-operator built
Install The components of the applications Mastodon, PeerTube and Pixelfed core/specs/008-components/operator/mastodon.md FR-001, FR-002; core/specs/008-components/operator/peertube.md FR-001, FR-002; core/specs/008-components/operator/pixelfed.md FR-001, FR-002 core/checks/apps, core/components/operator/mastodon/test-script.py, core/components/operator/peertube/test-script.py, core/components/operator/pixelfed/test-script.py built
Install The domain of a deployment: the claim, the DNS records and the certificates api/specs/001-deployments-and-builds/spec.md FR-004; core/specs/009-deployment-methods/tf-incus-operator.md FR-018, FR-019; core/specs/007-resources/ssl.md FR-001, FR-003, FR-006; panel/specs/002-deployment-lifecycle/spec.md FR-003, FR-013, FR-015, FR-019 api/src/fediversity_api/tests/test_apex_claims.py, api/src/fediversity_api/tests/test_domains.py, panel/src/panel/tests/test_domain_page.py, core/checks/deployment/domain-claim-probe built
Install panel shows the addresses of the deployed instances api/specs/001-deployments-and-builds/spec.md FR-004, FR-006; panel/specs/004-instances-overview/spec.md FR-001, FR-003 api/src/fediversity_api/tests/test_instances.py, panel/src/panel/tests/test_instance_list.py built
Configure The form of the configuration comes from the options of the application modules core/specs/002-configuration-schema/spec.md FR-001, FR-003, FR-008, FR-010; api/specs/002-schema-and-sources/spec.md FR-001, FR-006, FR-007; panel/specs/003-configuration-form/spec.md FR-001, FR-002 core/lib/json-schema-test.nix, core/lib/application-schemas-test.nix, api/src/fediversity_api/tests/test_http_schema.py, panel/src/panel/tests/test_schema.py built
Configure A stored configuration follows the renames of options core/specs/002-configuration-schema/spec.md FR-005; api/specs/002-schema-and-sources/spec.md FR-008, FR-009, FR-010; panel/specs/003-configuration-form/spec.md FR-003 core/lib/renames-for-source-test.nix, api/src/fediversity_api/tests/test_migrate.py, panel/checks/apps-panel built
Configure The operator gives secrets that the configuration does not keep core/specs/002-configuration-schema/spec.md FR-011; panel/specs/003-configuration-form/spec.md FR-009 core/lib/operator-secrets-test.nix, api/src/fediversity_api/tests/test_operator_secrets.py, panel/src/panel/tests/test_deployment_secrets.py built
Configure A new deployment starts from the defaults that the modules declare core/specs/002-configuration-schema/spec.md FR-003; panel/specs/003-configuration-form/spec.md FR-007 api/src/fediversity_api/tests/test_config_schema.py, panel/src/panel/tests/test_deployment_form.py built
Update An update policy for each deployment api/specs/001-deployments-and-builds/spec.md FR-014, FR-015, FR-016; panel/specs/002-deployment-lifecycle/spec.md FR-009, FR-010, FR-011, FR-012 api/src/fediversity_api/tests/test_update_policy.py, panel/src/panel/tests/test_update_policy.py, panel/checks/js-tests/update-policy.test.js built
Update The source of the modules that a deployment builds from api/specs/002-schema-and-sources/spec.md FR-004, FR-005, FR-011 api/src/fediversity_api/tests/test_source_allowlist.py, api/src/fediversity_api/tests/test_self_pin_warm.py built
User management An operator account and its login in panel panel/specs/001-accounts-and-ownership/spec.md FR-001, FR-002 panel/src/panel/tests/test_login.py, panel/src/panel/tests/test_oidc.py built
User management A deployment has one owner, and api authorizes each call for that owner panel/specs/001-accounts-and-ownership/spec.md FR-003, FR-004, FR-006; panel/specs/005-api-authorization/spec.md FR-001, FR-002, FR-003; api/specs/004-authorization/spec.md FR-001, FR-002 panel/src/panel/tests/test_account_api_access.py, api/src/fediversity_api/tests/test_authz_callback.py, api/src/fediversity_api/tests/test_authz_required.py built
User management The removal of an operator account panel/specs/001-accounts-and-ownership/spec.md FR-007 panel/src/panel/tests/test_account_removal.py built
User management The hosting provider manages the operator accounts in the administration pages of panel -- -- no spec
User management A directory of the users and groups of a deployment core/specs/008-components/shared/lldap.md FR-001, FR-003, FR-006; core/specs/007-resources/ldap.md FR-002, FR-008, FR-009 core/checks/integrations/lldap-series, core/checks/integrations/lldap-provision, core/resources/ldap/provider-test.nix built
User management Single sign-on of the applications core/specs/008-components/shared/authelia.md FR-001, FR-006; core/specs/007-resources/oidc.md FR-006, FR-013; core/specs/008-components/operator/mastodon.md FR-004; core/specs/008-components/operator/peertube.md FR-004; core/specs/008-components/operator/pixelfed.md FR-006 core/checks/integrations/authelia-series, core/components/operator/mastodon/oidc-test.nix, core/resources/oidc/policy-test.nix built
User management The groups of the directory give the roles in an application -- core/resources/ldap/provider-test.nix, core/components/operator/mastodon/test-script.py, core/components/operator/peertube/test-script.py, core/components/operator/pixelfed/test-script.py no spec
Switch The backups of the databases and the media of the applications, and the operations that start a backup or a restore api/specs/001-deployments-and-builds/spec.md FR-018; core/specs/007-resources/garage.md FR-020, FR-022, FR-023, FR-027; core/specs/008-components/operator/mastodon.md FR-013; core/specs/008-components/operator/peertube.md FR-012; core/specs/008-components/operator/pixelfed.md FR-014 core/checks/integrations/media-backup, core/checks/integrations/app-database-restore, core/checks/deployment/app-operations, api/src/fediversity_api/tests/test_operations.py built
Switch The export of a deployment to a different hosting provider -- api/src/fediversity_api/tests/test_migration.py, api/checks/migration-kv-test.nix, core/checks/deployment/migration-quiesce, panel/checks/js-tests/migration-export.test.js no spec
Switch The import of a deployment from a different hosting provider, and the restore of its data core/specs/009-deployment-methods/tf-incus-operator.md FR-031, FR-032; panel/specs/002-deployment-lifecycle/spec.md FR-025; api/specs/004-authorization/spec.md FR-014 core/checks/integrations/migration-restore, api/src/fediversity_api/tests/test_migration_restore.py, api/src/fediversity_api/tests/test_import_check.py, panel/checks/js-tests/migration-import.test.js partial
Switch The transfer of a domain to and from the registrar of the hosting provider core/specs/010-effects/registrar-providers.md FR-011, FR-012, FR-028, FR-029, FR-031; core/specs/009-deployment-methods/tf-incus-operator.md FR-033; panel/specs/002-deployment-lifecycle/spec.md FR-020, FR-021 core/checks/deployment/registrar-transfer, api/src/fediversity_api/tests/test_domain_transfer_in.py, api/src/fediversity_api/tests/test_domain_take_out.py, api/src/fediversity_api/tests/test_domain_handover.py partial

Code and docs

A path that is a directory contains the code or the test.

Capability Function Code Docs
Install The catalog of the applications that an operator can select core/applications-catalog.nix, api/src/fediversity_api/catalog.py, api/src/fediversity_api/routes/applications.py Add an application
Install The operator submits a deployment, and a job builds and deploys it panel/src/panel/views.py, api/src/fediversity_api/routes/deployments.py, api/src/fediversity_api/deployment.py, api/src/fediversity_api/_runner.py Make a deployment
Install The components of the applications Mastodon, PeerTube and Pixelfed core/components/operator/mastodon, core/components/operator/peertube, core/components/operator/pixelfed Add an application
Install The domain of a deployment: the claim, the DNS records and the certificates api/src/fediversity_api/domain_claims.py, api/src/fediversity_api/apex_claims.py, api/src/fediversity_api/routes/domains.py, core/resources/dns, core/resources/ssl --
Install panel shows the addresses of the deployed instances api/src/fediversity_api/instances.py, panel/src/panel/views.py, panel/src/panel/templates/instance_list.html --
Configure The form of the configuration comes from the options of the application modules core/lib/json-schema.nix, core/lib/schema-for-source.nix, api/src/fediversity_api/schema.py, api/src/fediversity_api/routes/schema.py Operator configuration, Schema and migration sources
Configure A stored configuration follows the renames of options core/lib/renames-for-source.nix, api/src/fediversity_api/renames.py, api/src/fediversity_api/migrate.py Schema and migration sources
Configure The operator gives secrets that the configuration does not keep core/lib/operator-secrets.nix, api/src/fediversity_api/operator_secrets.py, panel/src/panel/secrets.py --
Configure A new deployment starts from the defaults that the modules declare core/lib/json-schema.nix, core/lib/schema-for-source.nix, panel/src/panel/views.py --
Update An update policy for each deployment api/src/fediversity_api/update_policy.py, api/src/fediversity_api/automatic_updates.py Update policy
Update The source of the modules that a deployment builds from api/src/fediversity_api/self_source.py, api/src/fediversity_api/source_allowlist.py Schema and migration sources
User management An operator account and its login in panel panel/src/panel/oidc.py Architecture of panel
User management A deployment has one owner, and api authorizes each call for that owner panel/src/panel/models.py, panel/src/panel/tokens.py, api/src/fediversity_api/authz.py, api/src/fediversity_api/callback.py Data model
User management The removal of an operator account panel/src/panel/models.py, panel/src/panel/admin.py --
User management The hosting provider manages the operator accounts in the administration pages of panel panel/src/panel/admin.py --
User management A directory of the users and groups of a deployment core/components/shared/lldap, core/resources/ldap/lldap-provider.nix, core/resources/ldap/provision.nix Manage the users of your deployment, Manage users and groups
User management Single sign-on of the applications core/components/shared/authelia, core/resources/oidc/authelia-oidc-provider.nix --
User management The groups of the directory give the roles in an application core/resources/ldap/application-role-groups.nix, core/resources/ldap/roles-consumer.nix, core/resources/ldap/publish-roles.nix --
Switch The backups of the databases and the media of the applications, and the operations that start a backup or a restore core/components/shared/postgres-backup.nix, core/components/shared/operations/backup.nix, api/src/fediversity_api/operations.py, core/resources/garage/media-backup.nix, core/resources/garage/media-backup.sh, core/resources/garage/restic-s3.nix --
Switch The export of a deployment to a different hosting provider api/src/fediversity_api/routes/migration.py, api/src/fediversity_api/migration.py, core/effects/migration/effect.nix, core/effects/migration/run.sh, core/components/shared/migration-quiesce.nix Export of a migration, Export a deployment for a migration, Switching and exit clauses
Switch The import of a deployment from a different hosting provider, and the restore of its data api/src/fediversity_api/routes/migration.py, api/src/fediversity_api/import_check.py, api/src/fediversity_api/migration_kv.py, core/effects/migration-restore/effect.nix, core/effects/migration-restore/run.sh Restore the apps of a migration
Switch The transfer of a domain to and from the registrar of the hosting provider core/effects/tf/registrars.nix, api/src/fediversity_api/registrar.py, api/src/fediversity_api/routes/domains.py --

Gaps

These functions are in the code, but the specifications do not fully cover them. A row with the status no spec has no numbered requirement at all.

Capability Function Gap
User management The hosting provider manages the operator accounts in the administration pages of panel No file in panel/specs has a requirement for the administration pages.
User management The groups of the directory give the roles in an application core/specs/007-resources/ldap.md has no requirement for the roles.
Switch The export of a deployment to a different hosting provider The design is prose in api/docs/design/migrations.md. No spec file has a numbered requirement for the export.
Switch The import of a deployment from a different hosting provider, and the restore of its data No file in api/specs has a numbered requirement for the import procedure.

Results of the grant

The grant agreement states the results of each capability in the section "Planned features", and two groups of milestones that help: "Front-end" and "Security". The text is in the amendment agreement/Amendment - AMD-101136078-2-1.pdf in the meta repository of the project. This table quotes each result and gives the functions of this page that cover it. The connection between a result and a function is made by hand in core/docs/md/technical-design/traceability.nix. The build stops when the table names a function that is not a row of this page.

no row means that no function of this page covers the result. It does not mean that the code does not have the result. The milestones are not capabilities, thus they have no row.

One function covers only a part of its result: panel shows the addresses of the applications on a page, and the code sends no message to the operator when a deploy is done.

Planned feature Result that the grant states Functions of this page
Install applications "Operators will be able to easily select desired applications to install (i.e. deploy), ideally in bulk." The catalog of the applications that an operator can select; The operator submits a deployment, and a job builds and deploys it
Install applications "Selected applications will be deployed to the operator's own domain." The domain of a deployment: the claim, the DNS records and the certificates
Install applications "The operator will be notified how to access their applications, once these are ready to use." panel shows the addresses of the deployed instances
Install applications "Operators indicate to be satisfied with the implemented solution." no row
User management "Operators have access to an administrator account on their deployed applications." The groups of the directory give the roles in an application
User management "Operators can create users accounts for their deployed applications from a central place, e.g. using LDAP." A directory of the users and groups of a deployment
User management "Log-in experience unified across applications using single sign-on (SSO)." Single sign-on of the applications
User management "Operators indicate to be satisfied with the implemented solution." no row
Switch hosting provider "Operators can back up their application data." The backups of the databases and the media of the applications, and the operations that start a backup or a restore
Switch hosting provider "Operators can trivially move deployments to a different hosting provider." The export of a deployment to a different hosting provider; The import of a deployment from a different hosting provider, and the restore of its data; The transfer of a domain to and from the registrar of the hosting provider
Switch hosting provider "Migrations gracefully handle Fediversity version differences." The import of a deployment from a different hosting provider, and the restore of its data; A stored configuration follows the renames of options
Update applications "Operators can set when to update their applications" An update policy for each deployment
Update applications "Operators are guided through new options on updates." no row (issue 3054)
Update applications "Updating deployment options automated." A stored configuration follows the renames of options
Configure applications "Configuration uses a batteries-included approach." A new deployment starts from the defaults that the modules declare
Configure applications "Relevant configuration options are available for operators to tweak, presented using a user-friendly web interface, that further aid the user in case updates change the available options." The form of the configuration comes from the options of the application modules; A stored configuration follows the renames of options
Front-end (milestone) "API specification published" no row
Front-end (milestone) "API available" no row
Front-end (milestone) "UX prototype finished" no row
Front-end (milestone) "web host application implements API" no row
Security (milestone) "Credentials handled" no row
Security (milestone) "TLS certificates handled" no row
Security (milestone) "Hypervisor users segregated" no row
Security (milestone) "Inter-node access authenticated" no row

The specification files

This table lists each file in the three directories that has a minimum of one numbered requirement. The column Planned markers counts the issues that the markers (planned, #<number>) of the requirements name. The column Issues of the known deviations gives the issues that the section Known Deviations of the file names. The last column gives the capabilities whose rows cite the file. A file with -- in that column covers a subject that is not one of the five capabilities, for example the data model or the rotation of secrets.

Spec file Title Status Requirements Planned markers Issues of the known deviations Capabilities that cite it
core/specs/001-data-model/spec.md Data model and extension seam Implemented 10 0 -- --
core/specs/002-configuration-schema/spec.md Configuration model and JSON Schema Implemented 13 0 issue 2132 Configure
core/specs/003-applications-catalog/spec.md Applications catalog Implemented 5 0 -- Install
core/specs/004-deployment-programs/spec.md Stateless deployment programs Implemented 9 0 -- Install
core/specs/005-setups-and-wiring/spec.md Setups and wiring Implemented 14 0 -- --
core/specs/006-testers-and-docs/spec.md Test harness, checks and documentation Implemented 9 0 -- --
core/specs/007-resources/dns.md Resource dns Implemented 8 0 -- --
core/specs/007-resources/firewall.md Resource firewall Implemented 11 0 -- --
core/specs/007-resources/garage.md Resource garage Implemented 38 0 -- Switch
core/specs/007-resources/generateFiles.md Resource generateFiles Implemented 12 0 -- --
core/specs/007-resources/ldap.md Resource ldap Implemented 18 0 -- User management
core/specs/007-resources/netbox.md Resource netbox Implemented 9 0 -- --
core/specs/007-resources/network.md Resource network Implemented 11 0 issue 2131 --
core/specs/007-resources/nixos-module.md Resource nixos-module Implemented 4 0 -- --
core/specs/007-resources/oidc.md Resource oidc Implemented 17 0 -- User management
core/specs/007-resources/otel.md Resource otel Implemented 6 0 -- --
core/specs/007-resources/pod-roles.md Resource pod-roles Implemented 15 0 -- --
core/specs/007-resources/postgresql.md Resource postgresql Implemented 19 0 -- --
core/specs/007-resources/redis.md Resource redis Implemented 11 0 -- --
core/specs/007-resources/secrets.md Resource secrets Implemented 9 0 -- --
core/specs/007-resources/smtp.md Resource smtp Implemented 14 0 -- --
core/specs/007-resources/spec.md Resource family Implemented 13 0 -- --
core/specs/007-resources/spiffe.md Resource spiffe Implemented 10 0 -- --
core/specs/007-resources/ssl.md Resource ssl Implemented 10 0 -- Install
core/specs/008-components/dev/forgejo-runner.md Component forgejo-runner Implemented 12 0 -- --
core/specs/008-components/dev/forgejo.md Component forgejo Implemented 17 0 issue 29 --
core/specs/008-components/dev/git-pages.md Component git-pages Implemented 8 0 issue 2126 --
core/specs/008-components/hosting-provider/bind.md Component bind Implemented 13 0 -- --
core/specs/008-components/hosting-provider/garage.md Component garage Implemented 4 0 -- --
core/specs/008-components/hosting-provider/incus.md Component incus Implemented 26 0 -- --
core/specs/008-components/hosting-provider/netbox.md Component netbox Implemented 12 0 -- --
core/specs/008-components/hosting-provider/openbao.md Component openbao Implemented 38 0 issue 2270 --
core/specs/008-components/hosting-provider/overview.md Component overview Implemented 3 0 -- --
core/specs/008-components/hosting-provider/postgresql.md Component postgresql Implemented 5 0 -- --
core/specs/008-components/hosting-provider/spire.md Component spire Implemented 20 0 -- --
core/specs/008-components/hosting-provider/telemetry.md Components telemetry and otel-collector Implemented 26 0 -- --
core/specs/008-components/hosting-provider/terraform-backend.md Component terraform-backend Implemented 15 0 -- --
core/specs/008-components/hosting-provider/windmill.md Component windmill Implemented 15 0 -- --
core/specs/008-components/operator/grafana.md Component grafana Implemented 7 0 -- --
core/specs/008-components/operator/mastodon.md Component mastodon Implemented 16 0 -- Install, User management, Switch
core/specs/008-components/operator/peertube.md Component peertube Implemented 13 0 -- Install, User management, Switch
core/specs/008-components/operator/pixelfed.md Component pixelfed Implemented 18 0 -- Install, User management, Switch
core/specs/008-components/shared/authelia.md Component authelia Implemented 12 0 -- User management
core/specs/008-components/shared/lldap.md Component lldap Implemented 10 0 -- User management
core/specs/008-components/shared/smtp.md Component smtp Implemented 4 0 -- --
core/specs/008-components/shared/valkey.md Component valkey Implemented 9 0 -- --
core/specs/008-components/spec.md Components Implemented 21 0 -- --
core/specs/009-deployment-methods/k8s.md Deployment method k8s Implemented 37 0 -- --
core/specs/009-deployment-methods/spec.md Method family deployment-methods Implemented 26 0 -- --
core/specs/009-deployment-methods/ssh.md Deployment method ssh Implemented 17 0 -- --
core/specs/009-deployment-methods/tf-incus-operator.md Deployment method tf-incus-operator Implemented 37 0 -- Install, Switch
core/specs/009-deployment-methods/tf-incus.md Deployment method tf-incus Implemented 23 0 -- --
core/specs/009-deployment-methods/tf.md Deployment method tf Implemented 11 0 issue 2220 --
core/specs/010-effects/image.md Effect tf-incus-image Implemented 15 0 issue 2131 --
core/specs/010-effects/incus-bootstrap.md Effect dev-tf-incus-bootstrap Implemented 9 0 -- --
core/specs/010-effects/netbox-seed.md Effect netbox-seed Implemented 12 0 -- --
core/specs/010-effects/octodns.md Effect octodns-zone Implemented 16 0 -- --
core/specs/010-effects/registrar-providers.md Registrar providers of the operator path Partially implemented 38 0 issue 1908 Switch
core/specs/010-effects/registrars.md Effects registrar-* Implemented 15 0 issue 2131 --
core/specs/010-effects/spec.md Effect family effects Implemented 14 0 issue 2129 --
core/specs/011-secret-rotation/spec.md Secret rotation Partially implemented 12 0 -- --
core/specs/012-license-report/spec.md License report Implemented, with the decisions in License decisions (core/docs/md/compliance/license-decisions.md) 11 0 -- --
api/specs/001-deployments-and-builds/spec.md Deployments and builds Implemented 19 0 -- Install, Update, Switch
api/specs/002-schema-and-sources/spec.md Schema, migrations and module sources Implemented 11 0 -- Configure, Update
api/specs/003-application-catalog/spec.md Application catalog Implemented 4 0 -- Install
api/specs/004-authorization/spec.md Authorization of callers Implemented 14 0 -- User management, Switch
api/specs/005-job-execution/spec.md Execution of deployment jobs Implemented 5 0 -- Install
api/specs/006-service-deployment/spec.md Deployment of the api service Implemented 5 0 issue 1940 --
panel/specs/001-accounts-and-ownership/spec.md Accounts and ownership Implemented 8 0 -- User management
panel/specs/002-deployment-lifecycle/spec.md Deployment lifecycle Implemented 26 0 -- Install, Update, Switch
panel/specs/003-configuration-form/spec.md Configuration form Implemented 9 0 -- Configure
panel/specs/004-instances-overview/spec.md Instances overview Implemented 3 0 -- Install
panel/specs/005-api-authorization/spec.md Authorization of calls to api Implemented 7 0 -- User management
panel/specs/006-service-deployment/spec.md Service deployment Implemented 7 0 -- --