Fediversity

Feature Specification: Deployment method tf-incus-operator

Created: 2026-10-07 Status: Implemented Input: Reconstructed from the code of core/effects/tf/incus/operator/ and core/setups/ at 14504f80d, PRs #1856, #1869.

The tf-incus-operator method deploys one operator deployment of the hosting provider. It is the tf-incus method (tf-incus.md) plus a tenancy per deployment on the shared nodes of the provider, in the same OpenTofu graph. Shared method behavior is in spec.md. The program grammar, the verbs and the state rules are in 004. The registrar lane is in registrars.md. The zone publish of the provider is in octodns.md.

This file describes only what the operator method adds to tf-incus or changes in it.

User Scenarios & Testing (mandatory)

User Story 1 - Mint a deployment's tenancy on the shared nodes (Priority: P1)

An operator deploys a group of applications. The graph creates the garage buckets and keys, the postgres databases and roles, and the OpenBao entries that the applications consume. A destroy removes them, except the backups, which stay for a grace period.

Why this priority: Without the tenancy, no application node can start. The shared nodes converge once for all deployments, so they cannot create or drop a name of one deployment themselves.

Independent Test: A VM test of the garage and OpenBao part. It mints a backup bucket and key on a shared garage node and publishes the pair into OpenBao KV over TLS. A second apply drops the buckets of an application. The test checks that the other bucket is deleted, that the backup bucket stays under a retained name with its object, its backup password and a retention record, and that the deployer cannot delete these. Then it runs the sweep of the garage node within and after the grace period, and checks that the bucket stays and then goes with its secrets. At last it destroys, and checks that the backup bucket stays under a retained name and that the other buckets and the KV of the deployment are gone. An integration test applies the postgres part against a real postgres that accepts only TLS and a dev OpenBao with the database engine. It checks that each request gets a login role, a database with the correct owner and a static role, that the credential of the engine logs in to the database, that the state holds no password, and that a drop of a request and a destroy drop only the objects of the deployment.

Acceptance Scenarios:

  1. Given a garage request with a bucket and a key, When the operator applies, Then the graph creates the bucket, the key and the permission, and writes the key pair under the KV prefix of the deployment.
  2. Given a postgres request, When the operator applies, Then the graph creates the login role, the database and an OpenBao static role on the named database mount.
  3. Given a tenancy with objects in its buckets, When the operator destroys, or applies requests without one of its buckets, Then the graph keeps each such backup bucket under a retained alias, writes its retention record, and empties and deletes each other such bucket.
  4. Given a namespaced deployment whose nodes wrote entries under its KV prefix, When the operator destroys, Then the graph deletes each entry under that prefix after the instances are gone, and keeps the entries of other deployments and of the shared prefix.
  5. Given a bucket with CORS rules, When the operator applies, Then a preflight request to the bucket from an allowed origin gets the CORS headers.
  6. Given a retained backup bucket and its retention record, When the deployer tries to read or delete the retained secrets of the bucket, or to delete or change the record, Then OpenBao refuses it.

User Story 2 - Publish the names the nodes serve (Priority: P1)

An operator deploys on a domain. The graph writes an address record for each name that a node serves before that node switches, so that ACME validation resolves.

Why this priority: A node that orders a certificate for a name that does not resolve fails its activation.

Independent Test: An integration test runs the publisher over its dynamic update lane against a primary and a secondary of the provider. A unit test of what the setup does with the uplink and the record lane. An integration test checks that the retire on the deSEC lane gets its credential from the OpenBao of the last apply.

Acceptance Scenarios:

  1. Given a record lane and a node with published names and a NetBox address, When the operator applies, Then the publisher writes the records, reads them back and waits until the names resolve, before the node switches.
  2. Given a reserved name, When the publisher runs, Then it refuses the name.
  3. Given a published record, When the operator destroys, Then the publisher deletes the record only while it still holds the address of this deployment.
  4. Given an own zone and a node that provides smtp, When that node has switched, Then the publisher writes the SPF, DKIM and DMARC records of the zone.
  5. Given a published record that the publisher cannot release, When the operator destroys, Then the destroy completes, names the record and fails with a distinct status.
  6. Given a published record on the deSEC lane and an OpenBao that moved after the record was written, When the operator applies and then destroys, Then the apply writes no record again, and the publisher gets the credential for the retire from the OpenBao of that apply.

User Story 3 - Hold and register an operator's own domain (Priority: P2)

An operator submits a domain that is not under the apex of the provider. On the bind lane, the graph adds the domain to the catalog of the provider. When the provider enables domain registration, the graph also registers the domain, or adopts it, at the registrar account of the provider. When the provider also delegates zones to deSEC, deSEC holds the domain in place of the named of the provider: the api creates the zone at deSEC and a token that can write only that zone, and the graph delegates the domain to the deSEC nameservers and gives the registry the DS records of the keys that deSEC signs the zone with.

Why this priority: Without the zone and the delegation, no record of the deployment resolves on the public internet.

Independent Test: VM tests of the claim probe, the registry handle probe and the zone key probe. Unit tests of the projection, of the grant on the claim ledger and of the coordinates that the curation of the provider fixes. An integration test applies the registration on the bind lane against a fake registrar and a dev OpenBao: a quote above the maximum cost fails the apply and orders nothing, a quote at the maximum cost writes the claim and registers the domain for one period, and a destroy removes the claim and keeps the domain. Then, against a registrar API where a different account holds the premium domain, with the auth code of a migration import in the deployment KV prefix: an apply writes the claim, sends one transfer order with the code at the transfer quote, and fails while the transfer is not complete, and the state and the output of the apply do not hold the code, a second apply fails with no order, and when the transfer is complete, an apply adopts the domain with no order. On the desec lane, against a fake deSEC API with the keys of the zone: an apply holds the zone before it reads the keys, creates the shared nameserver group once, and sends one order under that group with only the key-signing key, a second apply changes nothing, a destroy keeps the group and the domain, and an apply with claim_held_domain adopts both with no create and no order. A plan test with mocked providers: without domain_registration the graph plans no registrar resource; a free name gets the claim of the deployment at <prefix>/public/domain-claims/<domain> under the shared prefix, and the registration; a held name that no deployment claims, a free name that another deployment claims and a premium name fail the plan; a name that is held outside the account gets only the wait for the delegation. No plan test adopts a held name, because tofu test cannot import.

Acceptance Scenarios:

  1. Given the bind provider and a zone, When the operator applies, Then the publisher adds a catalog member and waits until every nameserver answers for the zone.
  2. Given a free domain that no deployment claims, When the operator applies, Then the graph writes a claim that fails if a claim exists, and registers the domain for one period.
  3. Given a domain that the account holds and a claim that names this deployment, When the operator applies, Then the graph adopts the claim and the domain in place of a new registration.
  4. Given a domain that the account holds with no claim for this deployment, When the operator plans without the flag to claim a held domain, Then the plan fails.
  5. Given a domain that the account holds and no deployment claims, When the operator applies with the flag to claim a held domain, as the api does on the target of a migration, Then the graph writes a claim that fails if a claim exists, and adopts the domain.
  6. Given a domain that a different registrar account holds, no claim, and the auth code of the domain in the deployment KV prefix, When the operator applies, Then the graph writes a claim that fails if a claim exists, transfers the domain, and fails until the transfer is complete.
  7. Given a registered domain, When the operator destroys, Then the graph keeps the registration and removes the catalog member.

User Story 4 - Give each node a SPIFFE identity (Priority: P2)

The graph registers the TPM endorsement key of each container on the spire node of the provider after the switch, and retires it on destroy.

Why this priority: An operator node logs in to the OpenBao of the provider with its SVID. Without a registration, the node waits forever for its agent.

Independent Test: A unit test that each operator node carries the agent and its key reader. integrations-operator-subnet-key applies the graph with the real registration scripts against a stub incus. It checks the registration and the trust bundle after the switch, a second read of the key of a redeployed node, and the retirement of each name at a destroy.

Acceptance Scenarios:

  1. Given a spire node on the group or on the hosting provider roster, When the setup evaluates, Then the SPIRE node is that node, reached over the incus transport.
  2. Given no spire node on either roster and no explicit SPIRE node, When the setup evaluates, Then the evaluation fails and names the missing spire node.

User Story 5 - Isolate the deployment on the shared bridge (Priority: P1)

The nodes of all deployments on an Incus host share its managed bridge. The graph gives this deployment a small subnet of its own from the NetBox prefix of the operator deployments, gives each node a static address in it, and puts on the bridge interface of each node the Incus network ACL that the Incus host declares for that subnet. The ACL drops the traffic between the subnet and the rest of that prefix, in the two directions. It also drops the traffic to the ports of the hosting provider that the deployment does not use: on the provider prefix, it drops each port that no resource of the provider uses, and on the bridge address of each provider node, it drops each port that the resources of that node do not use.

Why this priority: The nodes of all operator deployments are in one prefix of the bridge. The ACL drops the traffic between the deployments on the bridge. The firewall gate of each operator node is a second layer: it trusts only the provider prefix of the bridge and the bridge addresses of its own deployment. Thus without the ACL, the gate still drops the traffic from the nodes of other operator deployments to the ports that the node does not publish.

Independent Test: An integration test uses the ACLs that the Incus host declares for two subnets on a real Incus bridge, and sends ICMP and TCP between filtered containers, also with a rule for one provider node. Plan tests with mocked providers check the rules of the ports, and which provider nodes get a rule. A plan test checks that each node gets the bridge addresses of all nodes of its deployment as its peers. A unit test checks that the ports of each provider node are the ports of the resources that the node supplies, and a unit test checks that no source of the gate of a realised operator node covers a node of another operator deployment. No test applies the NetBox allocation of the subnet, and no VM test sends between two operator nodes without the ACL. integrations-operator-subnet-key applies the graph with the real NetBox provider against a stub of NetBox. It checks the /28 and the addresses that the graph takes, the static address and the ACLs of each NIC, and the release at a destroy.

Acceptance Scenarios:

  1. Given two deployments A and B, When a node of A sends to a node of B, or B to A, Then the ACL drops it.
  2. Given a deployment, When its node sends to a node of the same deployment, to a port of a node of the hosting provider that the resources of that node use, to the bridge gateway or out of the bridge, Then the ACL passes it, and the bridge resolver answers its DNS.
  3. Given a node of the hosting provider, When it sends to a node of an operator deployment, Then the ACL passes it.
  4. Given a deployment, When its node sends to a port of a node of the hosting provider that the resources of that node do not use, Then the ACL drops it, also when a different node of the provider uses that port.
  5. Given a deployment with nodes and no NetBox, When the graph plans, Then it fails, and puts no node on the bridge without isolation.
  6. Given two deployments A and B and no ACL, When a node of B sends to a port of a node of A that A does not publish, Then the firewall gate of the node of A drops it.

User Story 6 - Restore the apps of a migration import (Priority: P2)

On the target of a migration import, the graph restores each app from the snapshot of the source after the deploy.

Why this priority: Without a restore, the target starts with empty apps, and the data of the source does not move.

Independent Test: A unit test holds that the deployer policy grants the path of the pointers, and that the graph builds that path, with the same mount and prefix. It also holds that the policy grants on the LDAP mounts only the rotation of the bind role of authelia, and that the graph rotates the role that the policy names. A test of the api holds the same for the api. An integration test holds that the pointer of an app gives the public HTTPS endpoint of the object store of the source, and that a node that gets to the source only through that endpoint restores from the pointer. integrations-migration-restore applies the root of the graph with a migration import, with a stub incus client and mocks of the other providers: the graph reads the pointer from KV, restores the files on the target node, releases the certificate hold and restarts the instance, and a second apply does not restore again.

Acceptance Scenarios:

  1. Given no apps to restore, When the operator applies, Then the graph has no restore step.
  2. Given a snapshot reference for each backup unit of each app, and the pointers in the deployment KV prefix with the same keys, When the operator applies a deployment that is not suspended, Then the graph restores each backup unit of each app after the deploy verdict, restarts the instance of each app once, and replaces the pointers with a mark of the restore.
  3. Given apps to restore that include the LDAP directory, When the LDAP directory restores, Then the graph rotates the bind role of authelia in the LDAP mount of the deployment, thus authelia binds with a new password and not with the password of the source. A failed rotation is a warning, and the LDAP engine rotates the role within 2 minutes.
  4. Given pointers whose apps, backup units or snapshot references are not those that the graph gets, a backup unit that the target does not have, or an app that does not restore, When the operator applies, Then the apply fails and the pointers stay.
  5. Given apps to restore and a suspended deployment, When the operator applies, Then the graph does not restore.

Edge Cases

  • The graph registers nothing and waits for no delegation when the zone is the apex, is under the apex, or the provider is neither bind nor desec.
  • When the provider puts its apex at deSEC and delegates zones to deSEC, a domain under the apex writes its names into the apex zone at deSEC. A domain that is the apex gets no record lane and no token, and its nodes get certificates from the private CA.
  • A premium domain fails the plan, so the graph does not spend more than the maximum cost. A transfer of a premium domain does not fail the plan: the maximum cost bounds its price.
  • The claim probe fails the plan on any OpenBao answer other than found or not found. The handle probe fails the plan on any registrar failure.
  • Each wait (for a name to resolve, for the zone hold, for the delegation) is bounded, and a timeout fails the apply. A catalog member that the provider refuses makes the zone hold time out. A failed DKIM read publishes no mail record and fails the apply.
  • Garage refuses to delete a non-empty bucket. The destroy mints a temporary admin key to empty each bucket. A temporary key that it cannot delete is reported, not raised.
  • OpenTofu never deletes a bucket itself. A retirement step per bucket runs when the bucket leaves the graph (FR-009).
  • A bucket that the retirement does not find by its name is retired already, so a retry after a failed destroy continues with the other buckets.
  • A retry that finds the retained alias of a bucket accepts a refusal of its record write, because the policy refuses a change to an existing record. If the first run did not write the record, the bucket has no record, and the garage node deletes it by the date in its alias.
  • Without an OpenBao address, the retirement writes no record.
  • The garage admin API cannot set CORS rules, and garage accepts them only from a key that owns the bucket. The apply mints a temporary key to set the rules over S3. A failure to set them fails the apply, and the next apply sets them again.
  • Under the bridged uplink, the plan requires the uplink parent to be a bridge.
  • A node of an earlier deployment without isolation keeps its old DHCP address until it renews its lease, but the interface filter accepts only the new static address at once. The operator restarts such nodes after the first apply with isolation.
  • The selector of the bridge prefix must match exactly one IPv4 prefix, else the plan fails. This prefix must be the prefix from which the firewall gate of each node of the hosting provider limits the operator nodes, else the plan fails too, because the gate would not limit the nodes of the deployment. When all subnets are in use, NetBox refuses the allocation and the apply fails.
  • When the selector selects a different prefix, the next apply replaces the subnet, the bridge address of each node, and the ACL. Unlike the public addresses, the bridge addresses follow a change of their pool, because the ACL is computed from the subnet and the nodes must be in it. The operator restarts the nodes after such an apply.
  • The isolation is IPv4 only. The managed bridge has no IPv6 subnet, so the interface filters drop all IPv6 from the bridge interface.
  • The restore does not write the snapshot credentials to the state or the plan: the graph gets only the snapshot references, and the restore step reads the pointers from KV at apply time. A change of the snapshot references replaces the restore.

Requirements (mandatory)

Functional Requirements

  • FR-001: The method MUST run one OpenTofu graph through the shared OpenTofu runner, and MUST share the instance and NetBox parts with tf-incus.
  • FR-002: The graph MUST read the coordinates of the hosting provider from the state of the provider, with authentication.
  • FR-003: The graph MUST address OpenBao, garage and postgres at the bridge address of the provider node first, then its public address, then an address that the operator gives.
  • FR-004: The graph MUST derive the NetBox URL from the provider state, and MUST read the NetBox token from the provider KV when the operator does not give it.
  • FR-005: For each garage request, the graph MUST create the buckets, the keys and the bucket permissions, and MUST write the access key to the public part and the secret key to the secret part of the deployment KV prefix.
  • FR-006: The graph MUST create the garage names exactly as it gets them. The setup MUST hand them in with the namespace already applied.
  • FR-007: For each postgres request, the graph MUST create a login role, a database and an OpenBao static role with its rotation period on the named mount and connection, and MUST drop them on destroy.
  • FR-008: The graph MUST read the postgres admin password ephemerally from the provider KV, and MUST connect with TLS.
  • FR-009: A destroy, and an apply that drops a bucket from the requests, MUST retire each bucket that it removes. It MUST keep each backup bucket with its objects, MUST move it to a global retained alias that carries the retention date and a prefix of the bucket id, and MUST empty and delete each other bucket.
  • FR-010: Before it removes the old alias of a retained bucket, the retirement MUST write the retention record of the bucket. When the write fails, it MUST remove the retained alias again and fail.
  • FR-011: The deployer policy MUST deny all access to the retained secrets under each deployment prefix, and MUST grant only the creation of retention records.
  • FR-012: For a namespaced deployment, the destroy MUST delete each entry under the deployment KV prefix, with all its versions, after every instance is gone. It MUST NOT delete under any other prefix, and MUST NOT delete the retained secrets of a bucket.
  • FR-013: For each bucket, the apply MUST set the union of the CORS rules of its requests, MUST remove the CORS rules of a bucket that has none, and MUST set them again when they change or the bucket is created again.
  • FR-014: When the record lane is set, the graph MUST publish the names of each node before that node switches, and MUST retire them on destroy. On the desec provider, the retire MUST get the credential from the OpenBao of the last apply.
  • FR-015: The publisher MUST find the zone by a walk up the labels of the served name, and MUST refuse reserved names.
  • FR-016: When the mail records are enabled, the graph MUST publish them after the application records and after the switch of the node that provides smtp.
  • FR-017: On the bind provider with a zone, the graph MUST hold the zone in the catalog before it publishes, and MUST release it on destroy. On the desec provider with a zone, the api MUST create the zone at deSEC before the deploy, the graph MUST make sure that the zone exists before it publishes, and MUST keep it on destroy: the delegation stays, and a different deSEC account can create a deleted domain.
  • FR-018: With a registration, the graph MUST run the steps in this order: hold the zone, claim the domain, register the domain, wait for the delegation, publish the application records.
  • FR-019: The graph MUST register a domain only when it is free and no deployment claims it, and MUST adopt a held domain only when the claim names this namespace, or when the flag to claim a held domain is set and no deployment claims it.
  • FR-020: The registration MUST use one period, the nameserver group and the maximum cost of the provider. It MUST renew at the expiry only when the provider turns the renewal on. On destroy, it MUST NOT delete the domain, and MUST turn its renewal off. With deSEC zones, the registration MUST use the nameserver group of deSEC in place of that of the provider, and MUST give the key-signing keys of the zone at deSEC as DNSSEC keys. A deploy that cannot read these keys MUST fail before the registration. The graph MUST create the deSEC nameserver group when the registrar account does not have it, MUST adopt it when it does, and MUST NOT delete it on destroy.
  • FR-021: When a SPIRE node is known, the graph MUST register the key of each container after its switch, MUST skip the SPIRE node itself, and MUST retire the registration on destroy. When the addresses document of the hosting provider gives a new spire_generation, the next apply MUST register the key of each container again, also with no change of the node.
  • FR-022: The setup MUST fail evaluation when no spire node is published and no SPIRE node is given.
  • FR-023: The graph MUST run the verdict-only pass for each node after the key registration and the deploy.
  • FR-024: The deployer credentials (the DNS token, the OpenBao token, the garage admin token, the registrar credentials and the Incus client configuration) MUST come from the environment.
  • FR-025: The publisher MUST NOT put a token on a command line. With deSEC zones, the publisher and the key read MUST read the deSEC token of the deployment from KV with the OpenBao token. The graph MUST NOT take a deSEC token as a variable.
  • FR-026: When the deployment has nodes, the graph MUST allocate one subnet from the NetBox prefix that the bridge prefix selector selects, and one address in it per node. When NetBox is not available, the plan MUST fail. When the bridge prefix is not the prefix from which the firewall gate of each node of the hosting provider limits the operator nodes, the plan MUST fail.
  • FR-027: The graph MUST name the Incus network ACL that the Incus host declares for the deployment subnet, which drops egress to and ingress from each sibling subnet on the path from the bridge prefix to the deployment subnet. The graph MUST NOT create or change an ACL of the isolation, thus the certificate of the deploy needs no right to change ACLs. The ACL of the hosting provider MUST also drop egress to each port of the provider prefix that no resource in operator-reach.nix uses, egress to each port of the bridge address of a provider node that the resources of that node do not use, and egress to each port of a public IPv4 address in the addresses document of the hosting provider that is not an operator port or a published port of the node. It MUST drop nothing else.
  • FR-028: The bridge interface of each node MUST have its NetBox address as a static address, MUST carry the ACL, and MUST allow by default the traffic that the ACL does not name. The instance key user.fediversity.peers of each node MUST hold the bridge addresses of all nodes of the deployment. The firewall gate of each node MUST trust these addresses and the provider prefix of the bridge, and no other part of the bridge subnet.
  • FR-029: The method MUST route the postgresql and redis resources of the operator group to the providers of the central group (the group whose roster provides the secret store).
  • FR-030: A failed destroy step for records, zones or SPIRE registrations MUST NOT stop the destroy, so that a destroy of a half-built deployment, or of one whose DNS primary is gone, completes. The step MUST record the object that it left, one entry per record, zone or SPIRE registration where it can name it. Each create step for these objects MUST record the object that it wrote in a second list. At the end of the run, the wrapper MUST remove from the first list each object that is in the second list, because a replacement wrote it again and it is in use. It MUST then print the objects that remain, and MUST fail with a distinct status when the run otherwise succeeded. This applies to each verb, also to an apply, where a replacement or a node removal runs these steps. A retire of a name that no zone holds is not a failure.
  • FR-031: When there are apps to restore, the namespace is set and the deployment is not suspended, the graph MUST restore the apps after the deploy verdict and the run state. The restore MUST read the pointers from the deployment KV prefix, MUST fail when their apps, backup units or snapshot references are not those that the graph gets or an app does not restore, and MUST replace the pointers with a mark of the restore after all apps restore, with no version of the entry that keeps the pointers. When the entry holds the mark of the snapshot references that the graph gets, the restore MUST end with success and MUST NOT restore again. When the LDAP directory restores and the deployment has an LDAP mount, the restore MUST rotate the bind role of authelia in that mount, also when another app does not restore; a failed rotation MUST NOT fail the apply. The deployer policy MUST grant that rotation and nothing else on the LDAP mounts.
  • FR-032: The graph MUST NOT give a snapshot credential to OpenTofu. Only the snapshot references go into the state.
  • FR-033: When the deployment KV prefix has the auth code of the domain, a different account holds the domain and no deployment claims it, the graph MUST write the claim and then transfer the domain to the registrar account, with the nameserver group, the contacts and the renewal of a registration, for at most the maximum cost. While the transfer is not complete (REQ or PEN), the apply MUST fail before the wait for the delegation. A later apply MUST NOT transfer again. The auth code MUST NOT reach the plan or the state: the graph MUST read it with an ephemeral read and give it to a write-only attribute. A domain with the status DEL MUST NOT count as held by the account.
  • FR-034: When the provider delegates zones to deSEC, its apex is at deSEC and the domain is under the apex, the api MUST make a deSEC token of the deployment that can write only the A and AAAA records of the names that the deployment publishes, in the apex zone. Before the deploy, the api MUST add the policies of new names to the token, and MUST NOT remove a policy. The apex MUST be in the deSEC account of the token that makes the tokens. The api MUST refuse a deploy that publishes a name outside its domain, and MUST NOT make a token for a domain that is the apex.
  • FR-035: Each deSEC token of a deployment MUST work only from the networks of domainRegistration.zones.allowedSubnets, or from each network when the list is empty. The api MUST set these networks on a token that it makes, on a token that it keeps before a deploy, and on each token of the provider each hour.
  • FR-036: Each hour, the api MUST delete each deSEC token of the provider that is older than one hour and whose deployment does not use it: the deployment does not exist or is idle, and no zone follows it. A failed delete MUST NOT stop the deletes of the other tokens.
  • FR-037: When a submit gives a deSEC token of the operator for the zone of its domain, and the domain is not the apex or under it, the api MUST read the zone with that token before the deploy, and MUST refuse the submit when deSEC refuses the token or its account does not hold the zone. The api MUST then give that token to the deploy, MUST NOT make, change or delete a deSEC token for the zone, MUST delete the deSEC tokens of the provider for the deployment, and MUST NOT show the token. The api MUST keep the token for a later deploy of the same zone.

Key Entities

  • Tenancy request: The garage requests (buckets, keys, access, CORS) and the postgres requests (mount, connection, rotation, databases and roles). The setup collects them from every node of the deployment.
  • Deployment KV prefix: The KV prefix of one operator deployment, under the provider prefix. Without a namespace, it is the provider prefix.
  • Record lane: The DNS provider (powerdns or bind), the zone, the TTL, the reserved names and the optional mail part. With deSEC zones, the publisher writes to deSEC on the desec provider in place of bind, also into the apex zone for a domain under a deSEC apex.
  • Domain claim: A KV entry per zone that names the deployment that owns the domain. It tells a registration from an adoption.
  • Domain transfer: A KV entry in the deployment KV prefix with the domain and its auth code, from the migration string of the source or from domain_transfer_in. The api writes it, and the destroy deletes it.
  • Registration lane: The apex, the nameservers, the maximum cost, the owner handle and the optional deSEC zones, which the curation of the provider sets. The deSEC zones part has the provider, its address, its nameservers and its nameserver group.
  • Bridge prefix: The NetBox prefix of all operator deployments on one Incus bridge. Each deployment holds one subnet in it.
  • Migration restore: A map from each app to the snapshot reference of each of its backup units, and a KV entry in the deployment KV prefix with the state pointer of each backup unit. The api writes both for the target of a migration import.
  • SPIRE node: The spire node of the hosting provider, reached over the incus or the SSH transport.
  • Retained secrets: The vars instances of a backup bucket that stay with the bucket, such as its backup password and the keys of its offsite backup, each with its secret/ and its public/ directory. A component declares them with its garage request. Where the setup adds the group to the name of such an instance, as for ldap-key-<group> and <group>-authelia-oidc, the component gives the default name and the setup replaces it with the qualified name.
  • Retention record: A metadata-only KV entry per retained bucket. It names the bucket, its alias, the namespace, its retained secrets and an optional expiry. The garage node deletes the bucket and these secrets when the grace period ends (garage).

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After an apply, each requested key pair is in KV under the deployment prefix.
  • SC-002: After an apply, each published name resolves to the NetBox address of its node before that node switches.
  • SC-003: After a destroy, no key, database, role or KV entry of the deployment remains on the shared nodes, except its backup buckets under retained names, their retained secrets and their retention records, until the grace period ends.
  • SC-004: A deployment without a spire node fails at evaluation, not at apply.
  • SC-005: A held domain without a matching claim fails at plan, before the graph spends money, unless the flag to claim a held domain is set and no deployment claims it.
  • SC-006: The nodes of two deployments do not reach each other, and they reach their own deployment, the nodes of the provider, the gateway and its DNS.

Assumptions

  • The hosting provider is deployed first, and its state exposes the public and the bridge address of each node.
  • The shared postgres node converges once for all deployments and never drops a database. This is why the graph creates and drops the objects of each deployment.
  • The valkey tenancy is declarative on the valkey and OpenBao nodes. This graph does not provision it.
  • Each database and role name carries the namespace of the deployment before it reaches the graph.
  • The last writer wins on a published name.
  • The hosting provider seeds the bridge prefix in NetBox. Incus hosts that share one NetBox need one prefix and one selector each.
  • The hosting provider owns the ACL with the limits on its ports (core/effects/tf/incus/hosts/operator-limits.tf), and the NIC of each operator node names that ACL after its isolation ACL. An apply of the hosting provider changes the limits on each operator node at once. The operator deployments do not apply again.
  • A node of the hosting provider with no address yet, on the apply that creates it, has only the rule of the provider prefix until the next apply of the hosting provider.
  • The ACL limits each public IPv4 address of the hosting provider to the operator ports and the published ports of the node. A node with no published ports keeps only its operator ports there. A public address of a provider node that the hosting provider does not report stays reachable, and only the firewall gate limits the public IPv6 addresses. The ACL cannot tell these addresses from the internet.
  • The plan of a deployment with a bridge fails on a document with no operator_limits_acl output. Apply the hosting provider, then apply the deployment again.
  • Incus refuses to delete an ACL that a NIC uses. Destroy the operator deployments before the hosting provider.
  • The graph reads the addresses document of the hosting provider. The http apply of the hosting provider writes it. After a new deploy of the hosting provider, until that apply writes the document, an operator deploy fails on a 404. It succeeds when it runs again after that apply.
  • All operator deployments share one deployer identity, the token fediversity-operator-deployer. The deployer writes the list of retained secrets in the retention record, and the garage node deletes what an expired record names. A record with crafted entries can thus make the garage node delete the retained secrets of a different deployment, also of a deployment that is not destroyed, and its expires_at can skip the grace period. Such a record can delete no other entry: the grants of the garage node cover only the retained stems. The method accepts this, and trusts the holder of the deployer token for all deployments, for these reasons:
    • The garage admin token of the deployer can do more: it can delete each bucket (see Known Deviations).
    • A token for each run, with a policy on the records of one deployment, limits only a token of one run that leaks. The node that makes these tokens can make one for each deployment.
    • An identity that the nodes of a deployment attest isolates the deployments. But a deployment from before such an identity has none, thus it needs a migration, and the garage admin token stays.

Known Deviations

  • The garage admin token of the deployer can delete a retained bucket. A garage admin token has a scope of admin API functions, not of buckets, and the deployer needs DeleteBucket for its own buckets. The policy keeps only the secrets and the record out of reach of the deployer.
  • A deployment that was applied before the retirement step existed has no retirement step per bucket. The first apply with it makes one for each bucket in the requests. A bucket that this same apply drops gets none, and the apply forgets it in garage with its objects. A bucket without a record keeps no retained secrets either: the sweep deletes only the bucket.
  • A domain resource that a transfer made before the code was write-only keeps the auth code in its state until the destroy of the deployment.
  • While the KV of the deployment has the auth code, a transfer that the registry refuses (FAI) fails the apply with no order. The graph does not send the transfer again: domain_transfer_in of the api does. Without that entry, the graph adopts the domain, and the wait for the delegation times out.
  • No test runs a transfer against the real registrar API. The graph and the api take the status DEL for a domain that the account deleted or that a transfer moved to a different account.

Open Points

  • The erasure of a deployment before its grace period ends has no command. The retention record has an expiry field for it: an erasure sets it to the current time, and the next sweep deletes the bucket and its secrets.