Logs and personal data

This page tells which personal data the logs keep and for how long, and how to choose the store and the retention of the audit logs.

Know which personal data the logs keep

The logs and the telemetry keep personal data for a limited time:

Store	Retention	Personal data
The journal of each node	`30day`, and at most `512M`	Client IP addresses and user names (`sshd`, the nginx error log, Authelia), email addresses (`maddy`), and what the applications write to their log.
Loki (logs)	`services.fediversity-telemetry.retentionDays`, 90 days by default	A copy of the journal of each node of the hosting provider at level `info` and higher. From each node of an operator deployment, the lines at level `warning` and higher, with the label `deployment_id`.
Tempo (traces)	The same as Loki	The spans of Grafana. A span can contain a request path.
Mimir (metrics)	The same as Loki	None known. The labels identify nodes, units, deployments, databases and buckets.
The nginx access log of each node	26 rotated files, one for each period of `weekly`	The network of the client (the first three octets of an IPv4 address, the first three groups of an IPv6 address), the method, the path and the protocol of the request without its query string, the referrer without its query string, the user agent, and the user name of HTTP basic authentication, which on the vhosts with `auth_basic` identifies the credential of one deployment. The log holds no password.
The `audit_event` table of `api/`	`services.fediversity-api.auditRetentionDays`, 365 days by default. `api/` removes the older events when it starts and each time that it records an event. `0` keeps all events.	The actor of each action on a deployment: with the panel, the OpenID Connect `sub` and issuer of the account, or the primary key of an account without single sign-on. The event also holds the action, the deployment, the result, the time, and the `X-Request-ID` that the nginx of `api/` sets for the request. The table keeps the events of a deleted deployment until they are older than the retention.
The build logs and the runs of the operations in `api/`	`services.fediversity-api.logRetentionDays`, 30 days by default. `api/` removes the log of a build and each run of an operation that finished before that number of days, when it starts and each time that it stores a build log or a run. `api/` also keeps only the last 100 runs of each deployment. The rows of the `build` table stay. `0` keeps all logs and runs.	The output of a deploy or of an operation, which can hold what the tools write, and the actor of each run of an operation.
The jobs of Windmill	`services.fediversity-windmill.jobRetentionDays`, 30 days by default, which is also the most that the open-source build of Windmill keeps	The arguments, the result and the log of each job. A deploy job holds the configuration of the deployment, and the configuration can hold email addresses and user names. `api/` removes the secret keys from the deploy job. The arguments and the result of a node-effect job can hold bearer credentials in clear, thus `api/` deletes that job when it is completed. When `api/` cannot delete it, `api/` writes the ID of the job to its log and tries again when it starts and each 10 minutes.

The journal keeps the full client IP address. The nginx access log does not, see core/setups/nixos/nginx.nix.

The nodes of an operator deployment send their journal and their host metrics to the path of the deployment on the telemetry node (core/components/hosting-provider/deployment-ingest.nix). They send only the journal lines at level warning and higher: these show the failures, and the lines at level info hold most of the personal data of the applications, for example the request logs. The telemetry node sets the label deployment_id from the path of the request, and replaces the value that the node sends. Thus to find the lines of one deployment, select on deployment_id, for example {deployment_id="<namespace>"} in Loki. The node sets the other labels, for example service_name. The Grafana node of the deployment reads the host metrics of its nodes through its read path. These lines stay in Loki after a deployment is deleted, until the retention removes them.

api/ also writes each audit event as one JSON line on the fediversity_api.audit logger. These lines go to the journal of the node. The audit shipper of the node (fediversity-audit-shipper) sends them to the tenant audit of Loki, which keeps them for services.fediversity-telemetry.auditRetentionDays (365 days by default). The auditRetentionDays of api/ does not apply to these lines. The node log shipper does not send them to the tenant fake, as a copy there would have no reason: it has a shorter retention, and an operator can delete it.

OpenBao writes each request and each response as one JSON line to the journal of its node (openbao.service), so the retention of the journal and of Loki applies to these lines. A line holds the client IP address, the path and the operation of the request, and the display name and the entity of the token, for example the name that an OIDC login gives to an administrator. Secret values and tokens are HMAC-SHA256 hashes. To find the hash of a known value, use bao write sys/audit-hash/journal input=<value>.

Choose the store and the retention of the audit logs

The audit logs are the audit events of api/ and the audit lines of OpenBao. The hosting provider chooses where it keeps them, and for how long.

The deploy credentials can change or delete the tenant audit of Loki. To protect the audit logs from these credentials, keep them in a store that the deployment does not control, for example an S3 store with Object Lock, or a Loki that CD does not deploy. The hosting provider operates this store, or a party that the contract names as a sub-processor. The audit shipper has an endpoint option, but the software does not yet have a setting for a store outside the deployment. The audit lines of OpenBao do not go to the tenant audit. They stay in the journal and in the tenant fake.

EU law does not give a number of days. These rules apply:

GDPR Article 5(1)(e): keep personal data no longer than the purpose needs. Thus set a period, and write down the reason.
Implementing Regulation (EU) 2024/2690, point 3.2.5 of the annex, for the entities in its scope, for example a provider of cloud computing services or a managed service provider: keep and back up the logs for a predefined period, and protect them from unauthorized access and changes.
National guidance can give a number. In France, the CNIL recommendation on logging gives six months to one year, and permits more when the purpose needs it. Check the guidance of your national authority.

The suggested period is 365 days for the audit logs of api/ and of OpenBao. For api/, this is the default of auditRetentionDays, for the audit table and for the tenant audit. The audit lines of OpenBao do not have this period yet: they stay for at most 30 days in the journal and 90 days in the tenant fake. A shorter period can lose the evidence of an attack that stays unknown for months. The suggested legal basis is the legitimate interest of Article 6(1)(f) of the GDPR, with its recital 49 on network and information security, or the legal obligation of Article 6(1)(c) for an entity in the scope of the regulation above. This text is a suggestion, not legal advice.

Object Lock stops the erasure of a record before the end of the lock. Set the lock to the retention period, not to a longer period, and ask a lawyer if Article 17(3) of the GDPR permits the lock for your purpose.

Each field in these tables goes to the audit store. A field with no reason should not go to the store.

The fields of an audit event of api/ are in the table of Audit events of the api. For domain_contact_put and domain_contact_delete, deployment_id is the ID of the contact of the operator.

Fields of an audit line of OpenBao, from the upstream audit format:

Field	Personal data	Reason
`auth.display_name`, `auth.entity_id`, `auth.metadata`	Yes, for an administrator. An OIDC login gives its `sub`. For a service, it is the name of a node.	To tell who read or changed a secret.
`request.remote_address`	Yes, for an administrator. The full IP address.	To tell where an access came from, and to see a stolen credential that is used from a different address.
`request.remote_port`	Yes, together with the address.	With the address and the time, to find one client behind a shared address. OpenBao has no option to omit this field.
`request.path`, `request.operation`, `time`, `type`	Yes, together with the name. A path can contain a deployment or a user name.	To tell which secret an actor used.
`response.data` and `request.data`	The values are HMAC-SHA256 hashes. The keys are not, and a key can be a name.	To show that a value was read, with no copy of the value.

The audit lines of OpenBao go to the tenant fake of Loki with the other lines of the journal, as they have no audit shipper yet. Thus an operator can delete them.

The record of the newest hash of the chain (auditHead.publishCommand) holds no fields of an event, thus no personal data. The hash of an event is over its fields, thus removing a field from new events does not break the chain.