Fediversity

Feature Specification: Components telemetry and otel-collector

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code of core/components/hosting-provider/telemetry/, core/components/hosting-provider/otel-collector/ and the node shippers at 093e68b6f, pull request #1690.

User Scenarios & Testing (mandatory)

User Story 1 - Collect the signals of the fleet (Priority: P1)

The otel-collector component provides otel. In a group with an otel provider, each node ships its host metrics and its journal over OTLP to the collector, and the collector node ships to itself. The collector exports traces, metrics and logs to their stores on the node that provides otelBackend.

Why this priority: without it, the dashboards and alerts have no data.

Independent Test: one VM test posts a trace and finds it in the collector debug output. Another posts each signal and queries it back from its store.

Acceptance Scenarios:

  1. Given a node whose roster name is an IPv4 literal, When the setups evaluate, Then the shippers identify it by that name, not a generic hostname.

User Story 2 - Store the signals in Garage (Priority: P1)

The telemetry node runs Loki, Tempo and Mimir on loopback, each with its own Garage bucket. The ingest name requires a password and routes each OTLP path to its store.

Why this priority: the stores keep the history for the dashboards and alerts.

Independent Test: the storage VM test checks the ingest gate and queries each signal back.

Acceptance Scenarios:

  1. Given new S3 keys, When they render, Then the stores restart with them.
  2. Given a log request on the ingest name that selects the audit tenant, When it reaches Loki, Then the lines go to the journal tenant, not to the audit tenant.
  3. Given a delete request to Loki, When it names the audit tenant, Then Loki refuses it. The same request for the journal tenant succeeds.
  4. Given the default audit retention of 365 days, When Loki runs, Then the audit tenant drops lines older than that. An audit retention of 0 keeps all the lines of the audit tenant.
  5. Given a request on the audit path of a source, When it has no credential, the credential of the collector, or the credential of a different source, Then the ingest name refuses it. The credential of a source is also refused on the log path of the collector.
  6. Given a unit that writes lines with and without the audit prefix, When the audit shipper of the node reads the journal, Then the audit tenant has the lines with the prefix, without the prefix, under the name of the source, and the audit shipper sends nothing to the journal tenant. The node log shipper is a different shipper. It sends the lines without the prefix to the journal tenant, under the name of the unit, and it does not send the lines with the prefix. The audit shipper does not send the lines without the prefix.
  7. Given a line that the unit wrote before the audit shipper started, When the shipper starts, Then the line arrives in the audit tenant.
  8. Given a request with the credential of one source whose line and headers name a different source, When it arrives on the audit path of the first source, Then the audit tenant has the line under the name of the first source only.

User Story 3 - Read dashboards and get alerts (Priority: P2)

An administrator logs in to Grafana with OIDC and sees the provisioned datasources, dashboards and alert rules. A firing rule sends mail to the alert addresses.

Why this priority: the alerts report failed backups, expiring certificates and full disks.

Independent Test: the storage VM test checks the provisioning and drives the failing-backup rule to alerting. No test runs the OIDC login.

Acceptance Scenarios:

  1. Given a deployment email with several addresses, When the setups evaluate, Then each is an alert address.

Edge Cases

  • An empty ingest credential or SMTP password stops the collector or Grafana from start, and it retries without a limit.
  • Without a store backend, the collector exports only to its debug output.
  • A new Grafana admin password or secret key restarts Grafana. Before Grafana starts, the node sets the new password on the admin account and encrypts the data keys again with the new key.
  • A node exports certificate metrics only when it holds ACME certificates.
  • Grafana is the only span source, because the windmill exporters are stubs.
  • The backup reads the Grafana database in one transaction while Grafana writes. The restore stops Grafana, replaces the database, removes its journal files, and starts it. It needs the Grafana secret key, which is in the KV store, not in the backup.

Requirements (mandatory)

Functional Requirements

  • FR-001: The otel-collector component MUST provide otel and request a SPIFFE identity.
  • FR-002: The collector MUST listen on loopback, except on the otel provider, where it MUST receive OTLP over gRPC and HTTP on the internal network with the ports open.
  • FR-003: The collector MUST send each signal only to its own store, with the ingest credential and the trust of the deployment CA.
  • FR-004: The collector MUST scrape each configured target, including Incus on an Incus deployment.
  • FR-005: The metrics shipper MUST collect host, service and certificate metrics at a regular interval. The log shipper MUST read the journal from info up, keyed on the unit, with the priority as severity. Both MUST identify the node by its roster name, run without a fixed user, and serve local diagnostics. Only the log shipper MUST read the journal.
  • FR-006: The telemetry component MUST provide otelBackend, use the bootstrap certificate audience, and serve an ingest name and a UI name under the apex with certificates.
  • FR-007: The component MUST generate the ingest credential, the Grafana secret key and admin password, and the S3 keys. One bucket per store MUST exist with the tenant prefix, owned by one key of the component.
  • FR-008: Grafana MUST log in through OIDC with the groups of the user, giving the admin role to the administrator group and the viewer role to others, and MUST provision the datasources, with one datasource for each tenant of Loki, the dashboards and the alert rules.
  • FR-009: The ingest name MUST require the ingest password and MUST NOT limit the body size. It MUST send the log lines to the journal tenant of Loki, whatever tenant the client selects. The node MUST open only HTTPS, and Grafana and the stores MUST listen only on loopback.
  • FR-010: The stores MUST keep data for the retention period, 90 days by default, and MUST NOT accept a retention without a limit. The audit tenant of Loki has its own retention (FR-016).
  • FR-011: The setups MUST point the collector at the otelBackend provider with its CA, give that provider the tracing endpoint and the alert addresses, and point app tracing at the gRPC receiver of the group.
  • FR-012: A group with no otel provider that asks for central telemetry MUST ship to the otel provider of the central group (the one group whose roster provides the varsBackend), as hypervisor does. The nodes of an operator deployment ship as FR-024 tells. Other groups ship nothing.
  • FR-013: The alert addresses MUST have no default. They MUST come from the deployment email, else from the group data. Without either, the evaluation MUST stop with an error that names the admin email setting.
  • FR-014: Grafana MUST send alert mail through its SMTP account.
  • FR-015: The component MUST stream a backup of the Grafana database as the Grafana user to a bucket of garage.
  • FR-016: Loki MUST require a tenant and have two tenants: the journal tenant, with the journal lines and the limits of the stores, and the audit tenant, with the audit lines. The audit tenant MUST refuse the deletion API of Loki and MUST keep its lines for the audit retention, 365 days by default, the same as api, where 0 keeps all lines. The OpenBao audit log can use the same tenant.
  • FR-017: For each audit source, the ingest name MUST serve an audit path that only the credential of that source opens, in place of the ingest credential, and MUST send the name of the source with the request, in place of a value of the client. The component MUST generate one credential for each audit source, and only the node of that source and the node that generates it MUST read it.
  • FR-018: When the component of a node marks a unit as an audit source and the group has an otelBackend provider, the setups MUST run an audit shipper on that node. The shipper MUST read the journal of the unit from the start, keep its place and its send queue on the disk, keep only the lines that start with the audit prefix and remove the prefix, name the source on each line, and send them to the audit path of the source with its credential and the trust of the deployment CA. It MUST NOT go through the collector, and it MUST refuse to start with an empty credential. The node log shipper of that node MUST NOT send these lines, and MUST send the other lines of the unit.
  • FR-019: The audit property of a component MUST be the one property that makes a unit an audit source, thus the OpenBao audit log can use the same shipper, path and tenant with a prefix that only its audit entries have.
  • FR-020: When there is an audit source, the telemetry node MUST run an audit-ingest collector on loopback. It MUST set the source of each line from the source that the ingest name sends, in place of the source that the line names, and send the lines to the audit tenant of Loki without a queue and without a retry. Thus the source of an audit line is the source whose credential sent it, and an error of Loki goes back to the shipper, which keeps the line in its queue.
  • FR-021: For each operator deployment with a credential in KV, the telemetry node MUST serve /metrics-query/<deployment id> on the ingest name. It MUST check the credential of that deployment before a query goes to Mimir, and MUST limit the requests of each deployment before that check.
  • FR-022: Each query on that path MUST go through prom-label-proxy, which adds the deployment_id of the path to each selector and gives an error for a query with a different matcher on that label. The path MUST NOT give the writes, the rules or the admin paths of Mimir.
  • FR-023: The node MUST accept a credential from KV only as the user grafana and a bcrypt hash of cost 5, so that a deployment cannot make the checks slow for the others.
  • FR-024: When a group has no otel provider and a node of the group generates the node telemetry credential, each node of an operator deployment of the group MUST ship its host metrics and its journal lines at level warning and higher to /otlp/deployment/<deployment id> on the ingest name, with that credential and the trust of the deployment CA. The lines at level info stay on the node, because they hold most of the personal data of the applications.
  • FR-025: For each operator deployment with a node telemetry credential in KV, the telemetry node MUST serve that path for logs and metrics only, and only the credential of that deployment MUST open it, in place of the ingest credential. The node MUST accept that credential only as the user node and a bcrypt hash of cost 5, and MUST limit the requests of each deployment before the check.
  • FR-026: A collector on loopback MUST set deployment_id on each resource and each data point from the path, in place of a value of the node, and MUST put the deployment id in front of service.instance.id. It MUST send to Loki and Mimir without a queue and without a retry, thus an error goes back to the shipper.

Key Entities

  • Ingest credential: the ingest password and the matching header of the collector.
  • Node telemetry credential: one for each operator deployment. The Grafana node of the deployment generates it and publishes its hash to KV. All nodes of the deployment send the matching header.
  • Audit ingest credential: one for each audit source. Its password opens the audit path of the source, and the shipper of the source sends the matching header.
  • Loki tenants: the journal tenant holds the journal of the fleet for the retention period. The audit tenant holds the audit lines for the audit retention, and refuses the deletion API. Both are in the Loki bucket.
  • Alert rules: failed mail, failed and stuck units, failed and stale backups, failed renewals, certificates near expiry (warning and critical), filling and full disks, absent nodes and silent ingest. At each start, Grafana deletes the rules that the code no longer provisions (telemetry/retired-alert-rules.nix).
  • Dashboards: an infrastructure overview, logs and traces, and a node detail.
  • Offsite backup: the Grafana database: users, preferences, service accounts, silences, annotations and unprovisioned dashboards.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The VM tests of the collector, the OTel wiring and the storage pass.
  • SC-002: The unit test of the hypervisor setup passes, so the hypervisor ships its telemetry.
  • SC-003: The unit test of the vars reload passes.

Assumptions

  • The garage node serves the buckets (see garage), and Authelia serves the OIDC client (see authelia).
  • The app nodes reach the collector over the internal network. The receivers have no auth.

Known Deviations

None.