Logs, traces and audit records

This part lists each channel that records what the software and its users do: the audit events, the logs, the traces and the job records. For each channel it gives the personal data, the purpose of the data, the store, the persons who can read it, the retention, the control that keeps credentials out of it, and the test that shows the control.

The page describes what the code does. It does not say what the law requires or permits. Where a legal question applies, the page says "Open". The hosting provider decides it, see the hosting provider guide.

The stores themselves (location, owner, encryption, backup, erasure) are in the state register. The roles and the directory groups are in the access model.

The tables that have the label "generated" come from the code at each build of the documentation (core/docs/md/technical-design/observability-records.nix).

Words on this page

• "None" means that the code was examined and has no such control, test or limit.
• "Open" means that no source in the tree gives the fact, or that the question is a legal one.
• "A person with root on the node" is a person who can open a shell as root on that node. The access model says who that is.

How a record goes from a node to the stores

flowchart TD
  units["Units of a node"]
  journal["Journal of the node"]
  nodeLogs["Node log shipper (otel-node-logs)"]
  auditShipper["Audit shipper (fediversity-audit-shipper)"]
  collector["otel-collector"]
  lokiDefault["Loki, default tenant"]
  lokiAudit["Loki, audit tenant"]
  tempo["Tempo"]
  grafana["Grafana"]
  apiTable["audit_event table of api/"]
  api["api/"]

  units --> journal
  api --> apiTable
  api --> journal
  journal --> nodeLogs
  journal --> auditShipper
  nodeLogs --> lokiDefault
  auditShipper --> lokiAudit
  collector --> lokiDefault
  collector --> tempo
  lokiDefault --> grafana
  lokiAudit --> grafana
  tempo --> grafana

The nginx of the telemetry node sets the tenant of each request (X-Scope-OrgID), thus a client cannot select a tenant (core/components/hosting-provider/telemetry.nix).

Retention

The hosting provider sets the retention periods. Each period below is the default of a setting, and the hosting provider can change it in its configuration. The reason for a period, and the law that applies to it, are not in the code: Open.

Generated:

Limits that are not a setting of this software:

• The nginx access log of a node other than the forge node: the logrotate rules of nixpkgs keep 26 weeks of files (the comment in core/setups/nixos/nginx.nix).
• The rows of the build table of api/: None. No job removes old rows. logRetentionDays removes only the log of a row.
• The disk queue of the audit shipper: None. The module sets no limit of size or of age.
• The logs of the CI jobs and the change log of NetBox: Open.

Channels

Audit events of the api

api/ records each action on a deployment as one audit event (api/src/fediversity_api/audit.py).

• Store: the audit_event table of the database of api/, and one JSON line on the logger fediversity_api.audit. The line goes to the journal, and then to Loki, see Audit lines in Loki.
• Readers: an operator reads its own events through the panel (recent_actions, GET /audit-events). A superuser of the panel reads the events of each account (reads_all_history). A person with access to the database reads the table.
• Retention: services.fediversity-api.auditRetentionDays. api/ removes the older events when it starts and each time that it records an event (prune_audit_events in api/src/fediversity_api/state.py). The events of a deleted deployment stay until they are older than the retention.
• Control that keeps credentials out: an event holds only the fields of the table below. For a request that is not valid, reason gives the location of the error and not the value. For an import, the event holds only the result. api/ cuts a long reason of the authorization callback, and cuts request_id to 128 characters.
• Test: api/src/fediversity_api/tests/test_audit.py: test_an_invalid_request_records_where_without_the_value, test_an_invalid_configuration_records_where_without_the_value, test_an_import_records_only_its_result, test_an_unknown_action_stores_nothing_of_the_caller, test_a_long_reason_of_the_callback_is_cut. The retention has the tests test_a_new_event_removes_the_events_older_than_the_retention, test_the_retention_is_a_setting and test_zero_keeps_all_events.
• Protection from changes: each event has a hash of the event and of the event before it. fediversity-api-audit-verify examines the chain (verify_audit_chain). The tests are test_a_changed_event_is_found, test_a_removed_event_is_found and test_a_prune_keeps_the_chain_correct.

The fields of an event, with the purpose of each. Generated:

An audit event does not hold the client address. To connect an event to a client address, a person must find the journal line or the access log line of the same request, and the access log holds only the network of the client.

api/ records the first refused call without a login in each interval of 300 seconds, and then one event with the number of the other calls. Thus an anonymous caller cannot fill the table.

Audit lines in Loki

The audit shipper of a node (core/components/hosting-provider/audit-shipper.nix) reads the journal of the unit of a component, keeps the lines that match the pattern of the component, and sends them to the audit tenant of Loki. Generated:

• Personal data and purpose: the same fields as the audit event.
• Store: the audit tenant of Loki. Before the shipper sends a line, the line is in its disk queue in /var/lib/fediversity-audit-shipper. The shipper does not stop its retries, thus a line stays in the queue while the telemetry node does not accept it.
• Readers: each account that can log in to the Grafana of the hosting provider, thus the members of fediversity-admins and fediversity-staff. Grafana has one organization and the datasource Loki audit has no permission of its own, thus a Viewer can query the tenant. A person with root on the node of api/ reads the queue.
• Retention: services.fediversity-telemetry.auditRetentionDays. The tenant refuses the deletion API of Loki, thus only the retention removes a line. The setting of api/ does not apply to these lines.
• Control that keeps credentials out: the control of the audit event. The shipper adds none.
• Test: None for the content of a shipped line.

The deploy credentials can change or delete the tenant, see the hosting provider guide.

Journal of each node

• Personal data: what each unit writes. sshd, Authelia and the nginx error log write full client IP addresses and user names. maddy writes email addresses. The applications write what their log level gives.
• Purpose: to find the cause of a failure on a node (the comment in core/setups/nixos/common.nix). A purpose for each field: Open.
• Store: the journal of the node. A node of the hosting provider sends the lines at the level of services.fediversity-node-logs.priority and higher to the default tenant of Loki (core/components/hosting-provider/node-logs.nix). A node of an operator deployment sends the lines at level warning and higher, and the telemetry node sets the label deployment_id (core/components/hosting-provider/deployment-ingest.nix).
• Readers: a person with root on the node reads the journal. Each account that can log in to the Grafana of the hosting provider reads the copy in Loki, for all deployments.
• Retention: see Retention. The lines of a deleted deployment stay in Loki until the retention removes them.
• Control that keeps credentials out: each unit can write a credential, because a request line or an error message of an application can hold one. Thus the control is in the pipeline and not in each unit. The node log shipper replaces the text that has the shape of a credential with [masked] before a line goes to Loki (core/components/hosting-provider/node-logs-masks.nix). The shapes are: the query string of a request line or of a URL, the user and the password of a URL, the value of an Authorization header, a bearer token, the value of a key whose name contains password, passwd, secret, token, api_key or credential, a JSON Web Token, and a token of OpenBao. A credential that has none of these shapes goes to Loki. The shipper does not read a message that is not UTF-8. The journal of the node keeps the full line: None for the journal.
• Test: the check node-logs-masks (core/components/hosting-provider/node-logs-masks-test.nix) sends an example of each shape through the masks. The check test-monitoring-storage (core/components/hosting-provider/monitoring-test.py, "the node log shipper masks a credential") writes values to the journal of a node and reads Loki. core/setups/hypervisor-config-test.nix examines the retention settings of the journal.

Audit device of OpenBao

OpenBao writes each request and each response as one JSON line to its standard output (core/components/hosting-provider/openbao/openbao.nix).

• Personal data: the client IP address, the path and the operation of the request, and the display name and the entity of the token, for example the name that an OIDC login gives to an administrator.
• Purpose: Open. No source in the tree gives the purpose of a field.
• Store: the journal of openbao.service, and then the default tenant of Loki. These lines do not go to the audit tenant.
• Readers: the readers of the journal and of the default tenant, see Journal of each node.
• Retention: that of the journal and of the default tenant. The lines do not have the retention of the audit tenant.
• Control that keeps credentials out: OpenBao writes secret values and tokens as HMAC-SHA256 hashes, and elide_list_responses removes the keys of a list response.
• Test: the check integrations-openbao-pki (core/checks/integrations/openbao-pki/nixosTest.nix, "The audit device writes each request to the journal with hashed values").

nginx access log

Each node that runs nginx writes one line for each request to /var/log/nginx/access.log (core/setups/nixos/nginx.nix).

• Personal data: the network of the client (the address with its last part removed: /24 for IPv4, /48 for IPv6), the method, the path and the protocol of the request, the referrer without its query string, the user agent, and the user name of HTTP basic authentication. On the vhosts with auth_basic (telemetry ingest, metrics query, audit path) this user name identifies the credential of one deployment, not a person. On other vhosts it is the text that the client sends.
• Purpose: Open. No source in the tree gives the purpose of a field.
• Store: files on the node. The forge node does not record the calls of the CI runners (core/components/dev/forgejo/forgejo.nix).
• Readers: a person with root on the node. The node log shipper reads the journal and not this file, thus the lines do not go to Loki.
• Retention: see Retention.
• Control that keeps credentials out: the log format holds no query string, of the request or of the referrer, and no password of HTTP basic authentication. A credential in a path or in the user agent goes to the file.
• Test: the check nginx-access-log (core/setups/nixos/nginx-access-log-test.nix) sends requests with a query string, a referrer and a user name with a password, and examines the lines. It examines the mask of an IPv4 address. For the mask of an IPv6 address: None.

Request logs of the api and the panel

api/ and the panel run on uvicorn with its access log on (api/src/fediversity_api/http_main.py, panel/nix/configuration.nix). Each request gives one line with the address of the peer, the method, the path with its query string, and the status.

• Personal data: the query string can hold an identity, for example the actor in GET /audit-events and the account in the history page of the panel.
• Purpose: Open.
• Store: the journal of the unit, and then the default tenant of Loki, because the lines are at level info.
• Readers and retention: those of the journal, see Journal of each node.
• Control that keeps credentials out: in Loki, the masks of the node log shipper, which replace the query string of a request line, see Journal of each node. In the journal of the node: None.
• Test: the check node-logs-masks has a request line of uvicorn as an example.

Jobs of Windmill

Windmill keeps the arguments, the result and the log of each completed job.

• Personal data: a deploy job holds the configuration of the deployment, which can hold email addresses and user names, and the deploy log.
• Purpose: the job runs the deploy, and api/ reads the log of the job to show it to the operator. A purpose for a completed job: Open.
• Store: the database of Windmill, see the state register.
• Readers: the administrators of Windmill. An operator reads the log of the builds of its own deployments through api/ (build_logs in api/src/fediversity_api/procedures.py).
• Retention: services.fediversity-windmill.jobRetentionDays.
• Control that keeps credentials out, deploy job: api/ removes the secret keys from the deploy_env that it sends to the job. Nothing removes a value from the text of the deploy log: None.
• Control that keeps credentials out, node-effect job (f/fediversity/node_effect): the arguments and the result of this job can hold bearer credentials, and Windmill keeps them in clear. api/ deletes the job when it is completed, with a success or a failure, before the result goes back to the caller. When api/ cannot delete a job, it writes the ID of the job to its log, and a person must delete the job (_run_effect_job in api/src/fediversity_api/node_effect.py). Until then, the credentials stay in the job.
• Test: api/src/fediversity_api/tests/test_deploy_env.py (SecretKeysStayOffTheWindmillPayload) for the deploy job. api/src/fediversity_api/tests/test_migration.py for the node-effect job: test_windmill_route_runs_the_job_and_deletes_it, test_a_failed_job_reports_its_error_and_is_deleted, test_a_timed_out_job_fails_and_is_deleted, test_a_job_that_cannot_be_deleted_is_logged_by_id.

Build logs and operation output in the api

• Personal data: the actor of each run of an operation (operation_run table). The text of a build log and of an operation output can hold what the deploy tools and the operation write.
• Purpose: api/ shows the result of a build or of an operation to the operator.
• Store: the build and operation_run tables of the database of api/. With Windmill, api/ reads the build log from the job. Without Windmill, api/ keeps the last 64 KiB of the output of a failed build in the build table.
• Readers: the operator of the deployment, through api/. A person with access to the database.
• Retention: api/ keeps the last runs of each deployment, see Retention, and deletes the runs after the deployment is destroyed. With services.fediversity-api.logRetentionDays, api/ also removes the log of a build and each run of an operation that finished before that number of days (prune_logs in api/src/fediversity_api/state.py). It does this when it starts, and each time that it stores a build log or a run. The default is 30 days. 0 keeps all logs and runs. The rows of the build table have no limit, because the usage records come from them: None.
• Control that keeps credentials out: api/ does not keep the output of an operation that has sensitive set. For the text of a build log, see the deploy job in Jobs of Windmill: the same runner writes the two logs.
• Test: api/src/fediversity_api/tests/test_operations.py: test_the_output_of_a_sensitive_operation_is_not_kept, test_the_deployed_revision_decides_if_the_output_stays. api/src/fediversity_api/tests/test_log_retention.py for the retention. For the build log: api/src/fediversity_api/tests/test_log_credentials.py (FailedBuildLogHoldsNoSecret).

Traces

• Sources: Grafana sends its own spans when tracingEndpoint is set. NetBox sends the spans of its requests, its HTTP calls and its database queries when fediversity.otel.endpoints.netbox is set (core/components/hosting-provider/netbox/otel.nix). Windmill has the settings but its open-source build sends no spans (core/components/hosting-provider/windmill-otel.nix).
• Personal data: a span can hold a request path. The attributes that the instrumentation libraries set: Open.
• Purpose: Open.
• Store: Tempo.
• Readers: each account that can log in to the Grafana of the hosting provider.
• Retention: services.fediversity-telemetry.retentionDays.
• Control that keeps credentials out: None. The collector has only the batch processor (core/components/hosting-provider/otel-collector.nix).
• Test: None.

Debug exporter of the collector

When services.fediversity-otel-collector.backends is not set, the collector has only the debug exporter with verbosity = "detailed". It then writes each record that it receives (logs, metrics and spans) to its own journal. The readers, the retention and the missing controls are those of the journal.

Records of Grafana and NetBox

• Grafana keeps its accounts and its sessions in its own database, see the state register.
• NetBox keeps a change log with the user of each change. No file in the tree sets its retention or its readers: Open.

Logs of the CI jobs

The forge keeps the log of each CI job. The workflows in .forgejo/workflows/ give secrets to the CD jobs as environment variables.

• Personal data: Open.
• Store, readers and retention: Open. No file in the tree sets them.
• Control that keeps credentials out: Open. No file in the tree sets a mask.
• Test: None.

Fields that serve to trace abuse

To trace abuse, a person must know who did a thing, from where and when. These fields give that:

• The audit events of api/: the fields with "Yes" in the last column of the table of the fields. They give the account, the action, the deployment and the time. They do not give a client address.
• The journal: the full client IP address and the user name in the lines of sshd and Authelia, with the time.
• The audit device of OpenBao: the client IP address and the display name of the token, with the path and the time.
• The nginx access log: only the network of the client. It cannot identify one address.

If these fields are sufficient for a duty of the hosting provider to trace abuse, and how long the hosting provider must or can keep them: Open.

Open items