Metering and accounting

This part tells what the system measures about a deployment of an operator, where each number comes from, and where a hosting provider reads it. It also tells what the system does not measure.

The system gives the quantities. It does not calculate a price and it does not make an invoice. The only amounts of money in the system are the costs that the registrar gives for a domain.

The sources

The system has three sources, and each one has its own store.

Source	What it gives	Store	Where to read it
The builds of `api/`	The periods in which a deployment ran one configuration	The `build` table of the state store of `api/`	`GET /usage`, field `periods`
The domain events of `api/`	What the registrar account of the hosting provider charged for the domain of a deployment	The `domain_event` table of the same state store	`GET /usage`, field `domain_events`
The telemetry of the hosting provider	The storage, the compute and the network traffic of a deployment, and figures of its applications	Mimir, on the `telemetry` node	Grafana, or a query on Mimir

flowchart LR
  subgraph api["api/"]
    builds["build table"]
    events["domain_event table"]
    usage["GET /usage"]
    builds --> usage
    events --> usage
  end
  deploy["Deploy job"] -- "result and domain.json" --> builds
  deploy --> events
  snapshot["Snapshot of the registrar account"] -- "renew, release" --> events

  subgraph telemetry["Telemetry of the hosting provider"]
    collector["otel-collector"]
    mimir["Mimir"]
    grafana["Grafana"]
    collector --> mimir --> grafana
  end
  storage["PostgreSQL and garage nodes"] -- "size of each database and bucket" --> collector
  hypervisor["Incus hypervisor"] -- "use of each instance" --> collector
  apps["Application nodes"] -- "application metrics" --> collector

  provider["Hosting provider"]
  usage --> provider
  grafana --> provider

The two sources of api/ and the telemetry have no link in the code. api/ does not read Mimir, and the telemetry does not read the state store of api/. The deployment id is the common key: it is deployment_id in the result of GET /usage and it is the label deployment_id of the metrics.

Usage periods

GET /usage (usage_list, api/src/fediversity_api/routes/usage.py) gives the usage periods. usage_list of api/src/fediversity_api/procedures.py calculates them at each call from the builds that succeeded (list_succeeded_builds of api/src/fediversity_api/state.py). api/ does not store a period.

Each build that succeeded and is not a destroy starts a period when it finishes. The next build of the deployment that succeeds ends the period. A destroy also ends it. The last period of a deployment that is not destroyed has end set to null.
A build that failed or was canceled changes no period.
A period has the deployment_id, the build_id, the start, the end, the configuration and the module_source of its build, and the flag suspended. A suspended deployment stops its instances and keeps its volumes.
A period has the domain that its configuration serves the applications below, and the registered_domain that the deployment holds at the registrar account of the hosting provider in that period, if any.
api/ keeps the builds of a deployment after its destroy. Thus the periods of a destroyed deployment stay available.

The params since and until select the periods that overlap that time. api/ does not cut a period to these times. The param id selects deployments, and a caller can give it more than one time.

A period follows the builds and not the instances. It tells which configuration a deployment ran and when. It does not tell how much storage or compute the deployment used. The telemetry gives these quantities.

Domain events

The field domain_events of the same result tells what the registrar account charged for the domains of the deployments. api/src/fediversity_api/domain_events.py records the events in the domain_event table.

Kind	What records it	Cost
`register`, `transfer`	The end of a deploy that registered the domain or transferred it into the account	The cost that the registrar gave for the order
`adopt`	The end of a deploy that took a domain that the account held already	None
`renew`	The unit `fediversity-api-domain-renewals`, when the expiry in the snapshot of the account moved forward	The renewal price of one year, for each year
`release`	The same unit, when the domain left the account	None

An event has its domain, its deployment_id, its cost in minor units of its currency, and the expiration_date of the registration after the event. The cost is null when the registrar charged nothing or when api/ does not know the charge.

api/ records renew and release only when registrarSnapshot of the NixOS module of api/ (api/nix/configuration.nix) points to the snapshot of the registrar account. The unit runs when the snapshot changes.

The events stay after the destroy of their deployment. The page api/docs/design/usage.md has the full rules.

Who can read the usage

GET /usage has the authorization of all routes of api/: api/ asks the authorization callback of the front-end, and the callback gives the deployments that the caller can see (api/src/fediversity_api/authz.py). api/ then limits the result to these deployments.

When the panel is the front-end, the rule of usage_list is that the caller owns the deployments (AUTHZ_PROCEDURE_RULES in panel/src/panel/views.py):

The callback gives the deployments of one account, also for a superuser. Thus one call gives the usage of one operator.
The callback includes the deployments that the operator deleted. The panel keeps the row of a deleted deployment for this purpose, until the removal of the account of the operator (panel/src/panel/models.py).
A backend of the hosting provider can call api/ with the service token of api/ and the header X-On-Behalf-Of with the user name of an account. The callback then decides as for that account.

The panel has no page that shows the usage. No view of the panel calls usage_list; the panel only answers the authorization callback for it. A hosting provider reads GET /usage with its own tool.

Resource use in the telemetry

The telemetry gives the quantities that the usage periods do not have. Each metric below has the label deployment_id, and the metrics are in Mimir.

Quantity	Metrics	Source
Size of each PostgreSQL database	`pg_database_size_bytes`	The PostgreSQL exporter on the node that runs PostgreSQL (`core/components/hosting-provider/node-metrics.nix`), each 30 seconds
Size of each garage bucket	`garage_bucket_bytes`, `garage_bucket_objects`, `garage_bucket_unfinished_multipart_upload_bytes`	`core/components/hosting-provider/garage-usage-exporter.py`, which asks the admin API of garage at each scrape, each 60 seconds
Use of each instance	The `incus_` metrics of the hypervisor, for example `incus_cpu_seconds_total` and `incus_filesystem_size_bytes`, which the dashboard `core/components/operator/grafana/dashboards/resources.json` shows	The scrape `incus` of the otel-collector (`core/setups/hosts-common.nix`), each 30 seconds
Host metrics of each node	The `system_` metrics: CPU, memory, load, disk, filesystem, network and paging	The unit `otel-node-metrics` of the node (`core/components/hosting-provider/node-metrics.nix`), each 30 seconds
Figures of an application	The `nodeinfo_` gauges, for example `nodeinfo_users_total`, and the Prometheus endpoints that an application gives	The scrape job `app-metrics` of the otel-collector (`core/components/hosting-provider/app-metrics-discovery.nix`)

The label deployment_id comes from a name that the deployment cannot change after the fact:

For a database, a bucket and an instance, a relabel rule takes it from the name of the database, the bucket or the instance (core/components/hosting-provider/deployment-relabel.nix). A database, a bucket or an instance of the hosting provider has no deployment_id.
For the host metrics of a node of an operator deployment, the telemetry node sets it from the path of the request, which opens only with the credential of that deployment (core/components/hosting-provider/deployment-ingest.nix).
For the application metrics, it comes from the KV path of the endpoint entry (core/components/hosting-provider/app-metrics-discovery.py). A deployment can write only below its own id. A deployment_id label that an application gives becomes exported_deployment_id.

The hosting provider reads these metrics in Grafana or with a query on Mimir. The dashboard "Deployment storage" (core/components/hosting-provider/dashboards/deployment-storage.json) shows the size of the databases and the buckets of each deployment. No dashboard shows the usage periods or the domain events.

Mimir keeps the metrics for services.fediversity-telemetry.retentionDays (core/components/hosting-provider/telemetry.nix). After that time, the resource use of a past period is not available. The usage periods and the domain events have no retention: api/ does not remove a build or a domain event.

Know the use of the deployments for billing in the hosting provider guide gives the procedure and example queries.

What the system does not do

It has no prices, no tariffs and no invoices. A domain event has the cost that the registrar gave, and no other amount is in the system.
It does not add the resource use to the usage periods. The hosting provider joins the two on the deployment id.
It does not aggregate the metrics for a billing period. The hosting provider does this with a query.