Fediversity

Monitoring and billing

This page tells how to monitor the deployments of the operators, and how to know the use of each deployment for billing. On this page, <domain> is the domain of your hosting provider deployment.

Monitor the deployments

Grafana is at https://monitoring.<domain>/. Log in with "Sign in with Fediversity SSO". Members of fediversity-admins get the Admin role. Members of fediversity-staff get the Viewer role. Authelia refuses other users.

The "Fediversity" folder has these dashboards:

  • "Infrastructure overview": the state of all nodes, and the expiry of the domains of the registrar account.
  • "Node detail": the metrics of one node.
  • "Deployment storage": the size of the databases and the buckets of each deployment, as in Know the use of the deployments for billing.
  • "Logs and traces": the logs and the traces of the nodes. Select a service (the systemd unit) and a node, and type a text in "Search" to find log lines. The panel "Recent log lines" shows the lines at all levels. The panel "Recent error log lines" shows only the errors.

To browse the logs without a query, open "Drilldown" > "Logs" in the menu of Grafana.

Grafana sends alerts by email to the addresses in services.fediversity-telemetry.alertEmails.

Know the use of the deployments for billing

GET /usage of api/ gives the periods in which each deployment ran one configuration, with the configuration, the module source and whether the deployment was suspended. A period starts when a build of the deployment succeeds, and ends when the next build that succeeds, a destroy included, finishes. Give since and until in ISO 8601 to get the periods that overlap a billing period:

GET /usage?since=2026-01-01T00:00:00Z&until=2026-02-01T00:00:00Z

Each period also has its domain, and its registered_domain when the deployment holds that domain at your registrar account in that period, as in Assign a domain to a deployment. This is the case for a domain that is not below your own domain. Each build records this value at its submit, thus a later change of your domain does not change the past periods. The registrar keeps a domain after the destroy of its deployment or a change of its domain, with the renewal that it has. Thus the release of that domain comes when it leaves the account, not at the destroy or the change.

The result also has domain_events: what your registrar account charged for the domains of these deployments. A deploy that registers, transfers or adopts a domain records an event with the cost of the order. An adopted domain was in the account already, thus it has no cost. A deploy that fails after the order also records the event. A renewal records an event with the renewal price of one year times the years of the renewal, and a domain that leaves the account records a release. The cost is in minor units of the currency, as 999 for 9.99 EUR. api/ records the renewals and the releases from the snapshot of services.fediversity-registrar-domains, thus set services.fediversity-api.registrarSnapshot to its snapshotFile. The api/ component of core does this when the deploy credentials of the hosting provider have the OPENPROVIDER_ login. The since and until params limit the events to the time when api/ recorded them.

The result has the whole periods, thus cut them to the billing period yourself. Add one or more id params to get the periods of those deployments only. The authorization callback of the front-end decides which deployments the caller can see. api/ keeps the builds after a destroy, thus the periods of a destroyed deployment stay available. The panel also keeps a deleted deployment for the usage until you remove the account of its operator. Thus get the usage of an operator before you remove its account.

The storage of a deployment is in Mimir, and the "Deployment storage" dashboard of Grafana shows it. These metrics have the label deployment_id:

  • pg_database_size_bytes: the size of each PostgreSQL database, with the label datname.
  • garage_bucket_bytes, garage_bucket_objects and garage_bucket_unfinished_multipart_upload_bytes: the size of each garage bucket, with the label bucket.

The nodes get these sizes every 30 seconds (databases) and every 60 seconds (buckets). The deployment_id comes from the name of the database or the bucket, thus a database or a bucket of the provider has no deployment_id. To bill the storage of a month, query Mimir, for example the mean bytes of each deployment:

sum by (deployment_id) (avg_over_time({__name__=~"pg_database_size_bytes|garage_bucket_bytes", deployment_id!=""}[30d]))

The incus host gives the use of each instance, and these metrics also have the label deployment_id, with the label name for the instance:

  • incus_cpu_seconds_total: the CPU time, with the labels cpu and mode.
  • incus_memory_MemTotal_bytes and incus_memory_MemAvailable_bytes: the memory of the instance and the memory that it does not use.
  • incus_network_receive_bytes_total and incus_network_transmit_bytes_total: the network traffic, with the label device.
  • incus_disk_read_bytes_total and incus_disk_written_bytes_total: the disk traffic, with the label device.

The collector gets these metrics every 30 seconds. An operator deployment names each instance <deployment id>-<node>, thus the deployment_id comes from the name of the instance, and an instance of the provider has no deployment_id. For example, the CPU hours and the transmitted bytes of each deployment in a month:

sum by (deployment_id) (increase(incus_cpu_seconds_total{deployment_id!=""}[30d])) / 3600
sum by (deployment_id) (increase(incus_network_transmit_bytes_total{deployment_id!=""}[30d]))

Mimir keeps the metrics for services.fediversity-telemetry.retentionDays, thus bill before that time ends.

Know the use of the applications

A node of an operator deployment can give the metrics of its application, for example the number of its users. The module of the application sets fediversity.metrics (core/components/shared/metrics/endpoints.nix):

  • fediversity.metrics.endpoints.<name> gives a Prometheus endpoint of the application on one port of the node. nginx gives only GET of the metrics path on that port, and gives 404 for all other paths.
  • fediversity.metrics.nodeinfo.url gives the NodeInfo document of the application as the endpoint nodeinfo. The node changes the document to the gauges nodeinfo_up, nodeinfo_info (labels software and version), nodeinfo_users_total, nodeinfo_users_active_month, nodeinfo_users_active_halfyear, nodeinfo_local_posts and nodeinfo_local_comments.

These applications give metrics:

  • PeerTube gives the NodeInfo figures, and its OpenTelemetry metrics as the endpoint peertube.
  • Mastodon gives the NodeInfo figures, and the metrics of puma and sidekiq as the endpoint mastodon.
  • Pixelfed gives the NodeInfo figures. The operator cannot turn off its NodeInfo document.

The unit metrics-publish-endpoints of the node writes each endpoint to KV at <deployment vars>/public/metrics/<host label>.<name>, with the host, the port and the path. It also deletes the old entries of its host label.

The otel-collector node reads these entries every services.fediversity-app-metrics-discovery.interval and scrapes them in the job app-metrics. The OpenBao role of the otel-collector node gets read access to the public/metrics entries of all operator deployments, and no other access below the deployments. The deployment_id label of a metric comes from the KV path of its entry. The discovery refuses an entry whose host is not a name of its deployment (<deployment id>-<label>.<apex>), or whose host label is not the label of the entry. A refusal shows in the journal of the unit app-metrics-discovery. Thus a deployment cannot make the provider scrape the node of another deployment, or a host that is not a node. A label deployment_id that the application gives becomes exported_deployment_id.

For example, the users of each deployment at the end of a month:

max by (deployment_id) (last_over_time(nodeinfo_users_total[1d]))

Know these limits:

  • The scrapes use HTTP on the internal network, without TLS.
  • The firewall of a node accepts the scrape only from an internal source, such as the bridge. The resolver of the otel-collector must thus give the bridge address of the node, as the bridge DNS does for its own domain. When the name gives the routed address, the scrape comes from the routed address of the otel-collector, and the firewall drops it.
  • A host can be an external name that starts with <deployment id>-. The discovery does not check that the name resolves to a node.
  • When a node removes all its endpoints, its publisher does not run, and its old entries stay until a destroy of the deployment.

Know the metrics of the components

The unit otel-node-metrics of a node scrapes the Prometheus endpoints of the components of the node on loopback, and pushes them with the host metrics. The job label is the name of the scrape job (services.fediversity-node-metrics.scrapeJobs):

  • openbao: the metrics of OpenBao, for example vault_core_unsealed. OpenBao serves them on a listener on 127.0.0.1:8254 without a token. This listener serves only /v1/sys/metrics. On the public names, the metrics need a token.