Quality attributes

This page gives, for each quality attribute of the full system, the target, what the code does, how a test shows it, and what the code does not do. A target is a value that a document or the code of this repository states. Where the repository states no target, this page says so. This page does not set a target.

The checks that this page names are attributes of the flake of core. To run one, use nix build ./core#checks.x86_64-linux.<name>. The page Flake attributes describes each check.

Summary

The specifications in core/specs/, api/specs/ and panel/specs/ have success criteria for each feature. No success criterion gives a time, a percentage or a quantity for an attribute of this page.

Availability

Target. The repository states no availability target: no uptime percentage and no maximum time that a service can be down. The contract templates have a row "Availability and backup" that the hosting provider fills in (Contract templates).

What the code does.

• Each stateful service runs on one node. The section Know the limits of a single node of the hosting provider guide lists the services. For example, garage has replication_factor = 1 (core/resources/garage/server.nix), and the stores of the telemetry node have replication_factor = 1 (core/components/hosting-provider/telemetry.nix).
• Grafana alert rules tell the hosting provider about a failure (core/components/hosting-provider/telemetry/alert-rules.nix). unit-failed and unit-stuck find a unit that failed or that does not finish its start. node-absent finds a node that sent host metrics before and sends none now. ingest-silent finds that no node sends metrics. disk-filling, disk-filling-trend and disk-full find a disk that gets full. cert-expiring, cert-expiring-critical and renewal-failed find a certificate that is not renewed. The section Monitor the deployments of the guide tells how to use them.
• A node counts its boots (core/setups/nixos/boot.nix). The boot loader marks a new generation as good only after the check boot-check-gateway gets an answer from the default gateway. A generation that fails the check reboots, and after the configured number of tries the boot loader starts the last good generation.
• With the deploy method k8s, a pod role has a replicas option (core/resources/pod-roles/default.nix). When the pod of a new deploy does not become ready, the Deployment continues to run the previous image (core/specs/009-deployment-methods/k8s.md, SC-003).

How it is tested. core/components/hosting-provider/telemetry/alert-rules-test.nix and core/components/hosting-provider/monitoring-test.py test the alert rules. The check integrations-boot-counting tests the boot counting. The check deployment-k8s-hosts runs the role lldap with two replicas. No test measures availability.

What it does not do. There is no replica, no standby and no automatic failover for a stateful service. When a node fails, its services are down until the hosting provider repairs the node or restores it from a backup. The alert rules report a failure; they do not repair it.

Recovery point and recovery time

Target. The hosting provider guide gives the recovery point objective (RPO) of each backup as 24 hours plus the duration of the backup run. This value is a result of the default schedule. Each failed run adds 24 hours. The section Know how much data a restore can lose has the table of the backups, their schedule options and their restore points. The repository states no recovery time objective (RTO).

What the code does.

• Each off-site backup has a timer with OnCalendar = "daily" and Persistent = true as a default (mkOffsiteBackup in core/components/operator/app-resources.nix). The hosting provider can change the schedule with the options that the table of the guide names.
• backupRetention in core/components/operator/app-resources.nix gives how long the backups keep their data: 30 days for the backups of an application, and 7 daily, 4 weekly and 6 monthly snapshots for the backups that restore the infrastructure.
• The media backup copies the buckets of an application (core/resources/garage/media-backup.nix). The external copy copies the backup data to a second location (core/resources/garage/external-copy.nix). garage makes snapshots of its metadata (core/resources/garage/metadata-snapshots.nix).
• The alert rule backup-failed fires when a backup run fails. The rule backup-stale fires when the last good run of a backup is more than 36 hours old.
• The restore drill (core/components/shared/restore-drill.nix, option fediversity.restoreDrill.enable) restores the newest database backup into a scratch database each day. It records the duration in the metric fediversity_restore_drill_duration_seconds and the time of the last good run in fediversity_restore_drill_last_success_timestamp_seconds. The alert rules restore-drill-failed and restore-drill-stale use these.

How it is tested.

The nix-unit tests in core/setups/backup-retention-test.nix test the retention. No test measures the age of the newest backup at the time of a failure; the rule backup-stale monitors it on a live installation.

What it does not do. No backup is continuous, thus a restore loses the changes after the newest good run. The duration that the restore drill records is the restore time of one database only. It does not include the time to make a new node, to restore files and media, or to find the failure. No test measures the time of a full restore. After a restore, the hosting provider must do the erasures of personal data again; the part data-subject-requests covers this subject.

Downtime of an upgrade

Target. The repository states no target for the downtime of an upgrade.

What the code does.

• A deploy builds the new configuration of a node and activates it with switch-to-configuration switch (core/effects/common/deploy.sh). systemd restarts the units that changed. Thus a changed service is down for the time of its restart.
• A service with state uses systemd.stateMigrations. The unit <name>-migrate-pre.service makes a backup and runs the steps for the version range before the service starts. A failed step restores the backup and keeps the service stopped. A downgrade and a skipped release series are refused with exit status 78. Examples are core/components/hosting-provider/openbao/migrations.nix and core/components/hosting-provider/netbox/migrations.nix.
• Before an upgrade of an application, mkUpgradeSnapshot in core/components/operator/app-resources.nix makes a backup while the application is stopped. When the backup cannot start, it tries again each 30 seconds. After UPGRADE_BACKUP_TIMEOUT seconds (default 600) it refuses the upgrade with exit status 78. The application then stays stopped with its data at the old version.
• The API applies updates as the update policy of the deployment tells it: manual, automatic or window (api/src/fediversity_api/update_policy.py, api/src/fediversity_api/automatic_updates.py). With window, an operator selects the period of the day in which an update can start. A window is at least 15 minutes and at most 24 hours.

How it is tested. The check integrations-state-migrations tests the upgrade steps, the refusals and the restore after a failed step. The checks integrations-postgresql-major and integrations-postgres-upgrade-dump test an upgrade of PostgreSQL to a new major version. A check integrations-<service>-series for each service with state, for example integrations-openbao-series and integrations-garage-series, tests an upgrade between release series. api/src/fediversity_api/tests/test_update_policy.py tests the update policy. These tests show that an upgrade keeps the data. No test measures the downtime.

What it does not do. There is no rolling upgrade and no second instance that takes the requests during an upgrade of a node. The downtime of a service is the time of its restart plus the time of its upgrade backup and its migration steps. That time depends on the quantity of data. A refused or failed upgrade keeps the service stopped until a person corrects the cause.

Downtime of a move between providers

Target. Concepts states: "Migrations involve service unavailability in the order of a day (due to DNS propagation) but no data loss." The repository states no more exact value.

What the code does.

• On the source, migration-quiesce stop stops the services of the applications, and the export makes the data available to the target (core/components/shared/migration-quiesce.nix, core/effects/migration/run.sh). The source deployment is suspended.
• On the target, the import restores the data and starts the applications (core/effects/migration-restore/).
• The applications are down from the export until the target has restored them and the DNS records point to the target. The sections Export a deployment for a migration and Restore the apps of a migration of the guide give the procedure. To go back, the hosting provider resumes the source deployment.
• The DNS time depends on the TTL of the records. The record of a node and the NS records of the apex have a TTL of 300 seconds (nodeTtl and apexNsTtl in core/setups/groups/hosting-provider.nix). The default TTL of a zone from NetBox is 86400 seconds (core/effects/octodns/netbox-zone.py). deSEC does not accept a TTL below 3600 seconds (core/effects/octodns/octodns_fediversity_desec.py).

How it is tested. The checks integrations-migration-restore, integrations-migration-restore-boot-wait and deployment-migration-quiesce test the export, the restore and the stop of the services. The check deployment-registrar-transfer tests the transfer of a domain. api/src/fediversity_api/tests/test_migration.py, test_migration_restore.py and test_migrate.py in the same directory test the API side. These tests show that the data arrives. No test measures the downtime, and no test shows the value "in the order of a day".

What it does not do. A move is not live: the applications do not run on the source and on the target at the same time. The downtime includes the copy of the data, which depends on the quantity of data and on the network between the two hosting providers. The code does not measure it.

Number of deployments for each provider

Target. Concepts states: "We target small- to medium-scale hosting providers with 20+ physical machines, able to scale to at least 1,000 operators." The repository states no other quantity.

What the code does.

• Each operator deployment gets a /28 prefix from the NetBox prefix for operator deployments (core/effects/tf/incus/operator/bridge-isolation.tf). Thus the size of that parent prefix, which the hosting provider selects, limits the number of deployments.
• The operator deployments share the PostgreSQL node, the garage node and the other nodes of the hosting provider (Contract templates gives the isolation). The capacity of these single nodes limits the number of deployments.
• The API builds at most schema_build_concurrency schemas at the same time, with a default of 2 (api/src/fediversity_api/schema_build.py). It runs at most one deploy for each deployment at a time, because the deploy job uses the deployment ID as its concurrency key (api/src/fediversity_api/deployment.py).
• The alert rules disk-filling, disk-filling-trend and disk-full tell the hosting provider when a disk gets full.

How it is tested. No test. No check makes a large number of deployments, and the repository has no load test.

What it does not do. The code does not spread the load of one stateful service across nodes, and it has no limit that refuses a new deployment when the capacity is used. The value of 1,000 operators is a design goal that no measurement supports.

Resources of the smallest installation

Target. The repository states no minimum for memory, processors or disk, and no test finds one.

What the code does. The groups in core/setups/groups/ define the nodes of an installation. The table shows the nodes of each group. A published node has a public name; an internal node does not.

• The nodes of the group hosting-provider exist once for each hosting provider. The front end adds the nodes of the API and the panel; they are not in this table.
• The nodes of the group operator exist for each operator deployment. A deployment has the nodes of the applications that the operator enables and the internal nodes that support them.
• The root disk of an Incus instance has a default size of 50GiB (instanceRoot.size in core/components/hosting-provider/incus.nix), which Incus applies only when the storage pool can hold a quota.
• The code of core sets no memory limit and no processor limit on an instance.

The part deployment-topology describes the nodes and where they run.

How it is tested. No test. The NixOS tests run each node in a virtual machine or a container with the resources of the test, which are not a statement about a production installation.

What it does not do. The code does not check that a host has sufficient resources before a deploy.

Related attributes

• Security. The threat model gives the assets, the trust boundaries, the threats and controls and the accepted risks and open gaps. This page does not repeat them. The part access-model covers who can do what.
• Observability. The nodes push metrics, logs and traces to the telemetry node of the hosting provider (core/components/hosting-provider/telemetry.nix). The section Monitor the deployments of the guide covers the dashboards and the alert rules.
• Data protection. The page Data protection covers the retention and the erasure of personal data.