Fediversity

Feature Specification: Resource garage

Created: 2026-10-07

Status: Implemented

Input: Reconstructed from the code of core/resources/garage/ and its wiring in core/setups/ and core/components/ at 093e68b6f, pull requests #1184, #1238, #1246, #1266, #1361 and #1818, and issues #527, #1205 and #1206.

User Scenarios & Testing (mandatory)

The garage resource gives a component S3 buckets and keys on a garage cluster, and backs up the media buckets of the Fediverse applications. The shared rules are in the resource family spec.

User Story 1 - Give an application buckets and a key on a shared garage (Priority: P1)

The Fediverse applications keep their media in S3. A component requests buckets and keys, with grants per bucket. Every physical name carries the tenant prefix, so deployments on one cluster do not collide. Under the tf provisioning, the operator OpenTofu graph creates the buckets and keys on the shared garage node of the hosting provider and publishes each pair to OpenBao KV, where the consumer node reads it. The graph reads the requests in a plain-data shape.

Independent Test: a VM test of the operator graph, and unit tests of the provisioning shape, the KV grants and the key minters.

Acceptance Scenarios:

  1. Given a tenant and a bucket, When the deployment evaluates, Then the node gets the prefixed name in the s3 contract result of its logical bucket.
  2. Given the requests of a group, When the KV stems derive, Then the deployer may write each stem and the app node may read its own.
  3. Given a key that expects a pair which no part of the group mints, When a deploy target other than the tf-incus operator graph evaluates, Then the evaluation fails and names the key.
  4. Given the operator graph mints a pair, When it publishes the pair to KV, Then the state of the graph holds no copy of the KV values, and a new key of garage makes the graph write the pair again.
  5. Given a key of the operator graph, When an apply runs in a new month (UTC), or after the operator increments garageKeyGeneration, Then the apply makes a new key with the same grants, writes the new pair to KV, waits for the drain time while garage accepts the old pair and the new pair, and then deletes the old key and its grants.

User Story 2 - Provision on the group's own garage node (Priority: P1)

The operator graph mints only the keys of the operator groups. A group with its own garage node, for example for the telemetry buckets or the cache-backup bucket of each valkey namespace, provisions on that node. A reconciler holds the buckets, keys and grants in a state on the node, and the node mints each own key that expects no pair from elsewhere.

Independent Test: a unit test of the key minters, and a VM test of the reconciler.

Acceptance Scenarios:

  1. Given garage is not healthy, When the reconciler starts, Then it waits for a bounded time, fails, and retries later.
  2. Given a key whose pair is not there yet, When the reconciler runs, Then it applies everything else, and a periodic job applies the key soon after the pair arrives. That job succeeds without action while garage is down.
  3. Given an object in garage but not in the state, When the reconciler runs, Then it adopts the object and keeps its id.
  4. Given an object that leaves the configuration, When the reconciler runs, Then it deletes it, and empties a bucket first.
  5. Given a key in the state whose pair files went missing, When the reconciler runs, Then the key stays with its id.
  6. Given a state from an older layout, When the reconciler runs, Then it moves the state and changes no bucket.
  7. Given a key in the state whose pair files change to a new ID and secret, When the reconciler runs, Then garage refuses the old pair, the new pair has the same grants, and a backup from before the change restores with the new pair.

User Story 3 - Back up the media buckets of an application (Priority: P2)

The buckets hold the uploads of the end users, and nothing else holds a copy. The media backup is on by default for mastodon, pixelfed and peertube. Each day it copies the media buckets into the backup bucket of the application and encrypts the name and content of each object. A replaced or deleted object moves to the history of that run. An operator command restores the buckets, lists the history and erases one object.

Three measures remove the media that the application or an erasure request deletes:

  • The history expires after the application retention, the same period as the database backups.
  • For an erasure request (GDPR Art. 17), the operator removes the object from the backup and all of its history at once.
  • After a media restore without the database, the operator runs the orphan-media cleanup of the application.

Independent Test: a VM test of the media backup, and the single-app VM tests of the three applications.

Acceptance Scenarios:

  1. Given an enabled backup, When it runs twice with no change, Then the target holds one encrypted object per source object that no exclusion matches, and the second run transfers nothing.
  2. Given a credential that arrives empty, When the backup starts, Then it fails and names the credential.
  3. Given a deleted or overwritten object, When the operator restores, Then the object of the last backup is back with its metadata. At a point in the history, each object is the version of that backup, and a later object stays.
  4. Given an erasure, When the operator restores, Then the object does not come back.
  5. Given a deleted bucket that the apply did not make again, When the operator restores, Then the restore fails before it copies, names the bucket and tells the operator to run the apply.

User Story 6 - Keep the backups when the site is lost (Priority: P2)

All backups are in the garage of the group, which is on one site. A group that sets a target outside the site gets a daily copy of each backup bucket there. A person restores from that copy with a key from outside the site.

Why this priority: Without it, a loss of the site also loses the backups.

Independent Test: The check integrations-backup-external-copy copies buckets to a second garage, and restores a restic repository from it with only the key of the target.

Acceptance Scenarios:

  1. Given a group with a target, When the run ends, Then the target holds each populated *-backups bucket and no other bucket.
  2. Given a copied bucket, When an object or the whole bucket goes away in the source, Then the next run moves it from current/ to history/, and it stays there for historyDays.
  3. Given a target that is down, When the unit runs, Then the unit fails, and the alerts of the backups fire.
  4. Given a source where most copied buckets have no objects, When the unit runs, Then the unit fails, and the target does not change.

User Story 4 - Run a co-located garage for a single-app test (Priority: P2)

The application tests run without a hosting provider. Garage then runs on the app node with no tenant, a local domain and the in-nix provisioning. The app mints its own pair, and the node sets the layout, creates the buckets, applies the CORS rules and imports the key. After the import, the node deletes each other key with the same name, thus a new pair in the key files replaces the old key.

Independent Test: a contract test of s3, and the single-app VM tests.

User Story 5 - Read the earlier name of the website flag (Priority: P3)

A stored configuration keeps its meaning: the bucket type maps the earlier name of the website flag to the current one.

Independent Test: a unit test of the forward compatibility.

User Story 7 - Serve the media of a bucket below the domain of the application (Priority: P3)

An application can give the URLs of its media below its own domain. A bucket front makes a location of an nginx virtual host on the node send each request to the website endpoint of the bucket, at the same path.

Why this priority: Only pixelfed uses it. The other applications give the website domain of the bucket.

Independent Test: the unit tests of the garage consumer, and the VM test of pixelfed, which reads a posted image through its own domain.

Acceptance Scenarios:

  1. Given a bucket front and an s3 contract result for its bucket, When the node evaluates, Then the location of the virtual host proxies to the website URL of the bucket with the same path.
  2. Given a bucket front without a result for its bucket, When the node evaluates, Then it sets no location.

Edge Cases

  • CORS rules apply under each route. The admin API cannot set them, so each route sets them over S3 with a temporary key that is not shared or kept. A bucket without CORS rules loses its rules.
  • The node assigns the single-node layout only once, on a fresh cluster.
  • The S3 API has no TLS. The website endpoint uses HTTPS. The wildcard names need a DNS-01 certificate.
  • After a media restore, the next backup uploads all restored objects again, because a restored object is newer on the server. A restore is rare, and the other runs then need no request per object.
  • A restore does not make a bucket, because the key of the application may not, and a bucket that the graph did not make gets a new id that the next apply cannot adopt. The operator apply makes it again with its settings.
  • A destroy, or an apply that drops a bucket, empties and deletes the bucket, except a backup bucket. A backup bucket stays for a grace period under a dated retention alias, with a retention record (see tf-incus-operator). No key of the deployment can read it, so a restore then needs a new key from the hosting provider.
  • A retained bucket with another alias is in use again: the sweep deletes only its record. A bucket with only retention aliases and no record is deleted when the grace period after the date in its alias ends.
  • A live bucket with the name of a record means a new deployment of the app, and the sweep keeps the secrets of the record.

Requirements (mandatory)

Functional Requirements

  • FR-001: The request MUST accept whether it is enabled, the buckets and the keys. It MUST hold only data.
  • FR-002: A bucket MUST accept a website flag, also under its earlier name, and CORS rules (headers, methods, origins).
  • FR-003: A key MUST accept optional pair files, a KV stem name (the key name by default) and per-bucket grants for read, write and owner.
  • FR-004: The policy MUST accept whether it is enabled, the domain, the tenant (none by default), the provisioning route (in-nix or tf, tf by default), and the domains and ports of the S3 API, the website endpoint and the cluster RPC. The S3 and website domains MUST default to names below the domain.
  • FR-005: The resource MUST prefix bucket, key and grant names with a non-empty tenant, and MUST NOT prefix the KV stem.
  • FR-006: For each bucket, the consumer side MUST state the want and the resolved s3 instance under the logical bucket name, with the physical name, the endpoint, the port, the region, the S3 and website domains and the website URL. A component MUST read its buckets in a module that sets nothing when its bucket has no result.
  • FR-007: Under the in-nix provisioning, the consumer MUST mint a pair for each key with pair files, through generateFiles.
  • FR-008: Every pair MUST publish to the KV stem of its key, with the access key public and the secret key secret, whoever mints it.
  • FR-009: The provisioning shape MUST drop the key file paths, MUST keep the website flag, and MUST give the CORS rules only when they are enabled.
  • FR-010: The KV stems MUST cover all keys of the requests, without duplicates and without regard to the minter.
  • FR-011: The garage node MUST mint a pair in the garage key format for each own key without pair files, and MUST leave a key with pair files alone.
  • FR-012: The garage node MUST keep a single copy of each object, and MUST receive its admin token from a generated secret as a credential.
  • FR-013: The garage node MUST open HTTP, HTTPS, the S3 API, the cluster RPC and the admin API.
  • FR-014: The garage node MUST proxy the wildcard names of the website endpoint and the S3 API to the local garage.
  • FR-015: The cache-backup request MUST give each valkey namespace one bucket and one key with read, write and owner on it.
  • FR-016: The resource MUST request certificates for the S3 and website domains and their wildcards.
  • FR-017: The in-nix provisioning MUST fail when a step fails, MUST import a key only when garage does not have it, and MUST fail when a key file stays missing or empty.
  • FR-018: The provisioning MUST allow the website of a website bucket only.
  • FR-019: The evaluation MUST fail when a grant gives neither read, write nor owner, and name the key and the bucket.
  • FR-020: The backup key of mastodon, pixelfed and peertube MUST have read on each media bucket of its application.
  • FR-021: Each of these applications MUST mint the encryption keys of its media backup, also when the media backup is off.
  • FR-022: The media backup MUST be off unless the application wiring turns it on. The wiring MUST turn it on by default, with the history retention equal to the application retention. When it is off, the node MUST load no credential for it.
  • FR-023: The scheduled backup MUST load only the backup key and the encryption keys. Only the operator command MUST load the application key, which may write the source buckets.
  • FR-024: Each component that requests a backup bucket MUST mint its own restic password, and each valkey node one per namespace when its backup is on. The restic jobs MUST receive it as a credential. Only the backups of openbao and terraform-backend MUST use a fixed password.
  • FR-025: A restic job with a minted password MUST open its repository only with it, MUST NOT try the old password that all backups shared, and no Nix store file of its node MUST hold that old password for it.
  • FR-026: Under the tf provisioning, the deploy targets of a deployment without the tf-incus operator graph MUST fail evaluation when a key with pair files has no minter in the group.
  • FR-027: All backups of mastodon, pixelfed and peertube, the valkey backups, and the database backups of lldap, authelia, the panel and the API MUST keep their data for the same application retention of 30 days. restic MUST prune them by age, so that no snapshot older than this period before the newest one stays.
  • FR-028: When its vars go to OpenBao, the garage node MUST sweep the retained buckets daily. A record expires after the grace period (7 days by default) from its creation, or at its own earlier expiry. For each expired record, the sweep MUST delete the bucket, then each KV path of the record in its namespace with all versions, then the record.
  • FR-029: The sweep MUST read the grace period at each run, so a change applies to buckets that are already retained.
  • FR-030: The OpenBao policy of a sweeping garage node MUST grant read, list and delete on the metadata of the records and the retained KV stems of each deployment, and MUST NOT grant a read of their data.
  • FR-031: A bucket request MUST accept the KV stems that a retained backup bucket keeps. Each backup bucket of mastodon, pixelfed and peertube MUST keep the restic password and the media-backup encryption keys.
  • FR-032: The operator graph MUST publish a pair with write-only arguments, so its own state holds no copy of the KV values. It MUST write the pair again when the ID of the key changes. The secret key stays in the state of the garage key, because the garage provider keeps it there.
  • FR-033: The garage server MUST make snapshots of its metadata database at an interval, in a directory outside its metadata directory, and keep the two newest snapshots.
  • FR-034: A group MAY set an external copy target. The garage node of a group with a target MUST copy each populated *-backups bucket there daily, with no restic password and with no decryption, and it MUST keep what a run replaces or removes for historyDays.
  • FR-035: The copy MUST mint its source key at each run and delete it at the end. Only the garage node MUST be able to read the key of the target.
  • FR-036: A run MUST NOT change the target when more than half of the copied buckets have no objects in the source. A failed run MUST NOT remove old history.
  • FR-037: When the pair files of a key of the reconciler change to a new ID and secret, the next run of the reconciler MUST delete the key of the old ID, and MUST import the new pair with the same name and grants. A person starts that run, as the hosting provider guide tells.
  • FR-038: A bucket front MUST accept the logical name of a bucket, the name of an nginx virtual host and a path that starts and ends with /. When the bucket has an s3 contract result, the location of the path MUST proxy to the website URL of the bucket followed by the path. Without a result, the front MUST set nothing.

Key Entities

  • Tenant: the prefix of all physical names on a shared cluster: the deployment namespace, or the group when there is none.
  • KV stem: the OpenBao vars instance of one key pair, named after the key.
  • Media backup: the encrypted copy of the media buckets, with a current copy and a history per run.
  • Bucket front: an nginx location on the node that proxies a path of the domain of the application to the website endpoint of a bucket.
  • Retention record: the KV entry that keeps a retained backup bucket and its KV stems until the sweep.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: The unit tests of the garage consumer, forward compatibility, plain-data requests, KV stems, key minters, operator and valkey configurations, backup retention and app vars grants pass, and the panel API frontend check passes.
  • SC-002: The VM tests of the operator graph and the media backup, the s3 contract test and the single-app VM tests pass.
  • SC-003: No bucket that a provisioner creates is left without a key, and every published stem has a write grant for its publisher and a read grant for its consumer.
  • SC-004: The VM test of the metadata snapshots passes: garage makes the snapshots and keeps the two newest, and a snapshot replaces a lost metadata database while the buckets stay.

Assumptions

  • Garage has no row in the routing table of setups and wiring. The provisioning route decides.
  • The node that holds the buckets is the one whose component provides the object store.
  • The cache-backup keys feed the backups of the valkey component.
  • The admin port is not a setting. The resource, its web UI and the operator TF effect read it from one place, and the effect passes it to its graph, which cannot read the settings of a node in another group.
  • Each group has one garage node with replication_factor = 1, and the backups of the group are in its buckets. Thus the loss of that node loses the buckets and their backups. The metadata snapshots are on the same node, and they hold no data blocks: they are only for a corrupted metadata database.

Known Deviations

  • The media backup does not encrypt the metadata of an object, for example its content type.
  • The garage node holds a key that can delete in the external target, thus the target needs versioning or object lock at its provider to resist a compromised node.