Fediversity

Feature Specification: Deployment method k8s

Created: 2026-10-31 Status: Implemented Input: Issues #2632, #2630 and #2628, the effect core/effects/k8s/ and the node setup core/setups/nixos/k8s-pod.nix.

The k8s method has two forms. k8s-hosts deploys the nodes of a group. k8s-operator-hosts deploys the nodes of an operator deployment, and also runs the DNS lane and the registrar lane of the deployment (FR-019 to FR-022).

The k8s method runs each node of a group as the one pod of a Kubernetes Deployment, in a cluster of the hosting provider. The method does not switch a closure on a node. It builds an image of the node configuration, pushes it to a registry, and rolls the Deployment out to that image. If the new pod does not become ready, the method sets the Deployment back to the previous image.

The graph is an OpenTofu graph, with its state on the same HTTP state backend as the other OpenTofu methods (tf.md). The verbs and the state rules are in 004. The plans of the nodes, the submission and the key sets are in spec.md.

User Scenarios & Testing (mandatory)

User Story 1 - Deploy a node to a cluster as an image (Priority: P1)

An operator runs the k8s program for a group, with the kubeconfig of the cluster and the address of a registry. The program pushes the image of each node and applies its Deployment. The node then runs in the cluster, with systemd as PID 1 of the pod.

Why this priority: This is the only purpose of the method.

Independent Test: A VM test runs k3s and a registry in one VM and the state backend and the deployer in a container. It deploys one node of a group, checks that the pod becomes ready and that the Deployment names the image by its digest, checks that the pod keeps its address on eth0, checks that a pod in the namespace of the target can connect to the node and that a pod in a different namespace cannot, checks that /var/lib of the node is on its volume and that the machine ID is the container_uuid of the pod, and then checks that a plan shows no changes.

Acceptance Scenarios:

  1. Given a cluster that has the RuntimeClass of the method and a registry that it can pull from, When the operator runs the program, Then the image of each node goes to <registry>/<namespace>/<node>, and the Deployment of the node runs that image by its digest.
  2. Given a node whose configuration did not change, When the operator applies again, Then the push, the Deployment and the rollout are not replaced, and the pod is not restarted.
  3. Given a node whose configuration changed, When the operator applies, Then the program pushes a new image and the Deployment replaces the pod.
  4. Given a deployed node, When a pod in a different namespace connects to a port of the node that the node does not publish, Then the connection fails. The pod can connect to a published port. A pod in the namespace of the target can connect to each port that the node serves, and cannot connect to a port where the node listens but that it does not serve.

User Story 2 - Keep the previous image when a node does not become ready (Priority: P1)

The operator deploys a configuration with which systemd in the pod does not get to running. The rollout does not complete in its time limit. The program sets the Deployment back to the previous image and fails.

Why this priority: The method has no boot entry to revert. Without the rollback, a configuration that does not boot leaves the node down.

Independent Test: The same VM test deploys a configuration with a unit that always fails, after a good deploy. It checks that the program fails, that the output tells of the rollback, and that the Deployment runs the previous image again. The failed configuration also has a new version of a state migration. The test checks that the new pod has the file that the test wrote in /var/lib before the failed deploy, the same machine ID, and the version of the previous image in the record of the migration, and that the pod removed the copy of the state. It checks that the role of the node has 0 replicas after that deploy, and that the output shows the stop of the role before the change of the main Deployment. It then deploys the good configuration again, and checks that the role has its ready pods. Last, it deploys a configuration with a failed unit in a role, and checks that the program fails, that the node keeps the new image, that the role has 0 replicas, and that the next apply starts the role again.

Acceptance Scenarios:

  1. Given a Deployment that runs a good image, When the operator deploys a configuration whose pod does not become ready, Then the program sets the Deployment back to the good image and fails.
  2. Given a first deploy whose pod does not become ready, When the rollout times out, Then the program fails, and there is no previous image to set back.
  3. Given a failed rollout, When the operator applies again with the same configuration, Then the program tries the rollout again.
  4. Given a node with state in /var/lib, When a rollout or a rollback replaces its pod, Then the new pod has the same /var/lib, the same machine ID and the same host keys of sshd.

User Story 3 - Remove the deployment of a group (Priority: P2)

An operator runs destroy for a group.

Why this priority: The operator must be able to remove the nodes of a group from the cluster.

Independent Test: none.

Acceptance Scenarios:

  1. Given a deployed group, When the operator runs destroy, Then the program removes the namespace of the target with all that is in it, which includes the claims of the volumes of the state. The images stay in the registry.

Edge Cases

  • The cluster does not have the RuntimeClass of the method: the plan fails, before the program pushes an image.
  • The registry is not reachable from the deployer: the push fails, and the program changes nothing in the cluster.
  • The cluster cannot pull the image: the pod does not become ready, and the program sets the Deployment back as for a failed boot.
  • A unit of the node failed, thus systemd is degraded: the pod is not ready.
  • The configuration of a node declares eth0 in networking.interfaces: systemd-networkd does not manage eth0, thus the address and the routes of the pod stay.
  • A rollout failed and the program set the Deployment back: the state holds the new image, and the cluster runs the previous one, until the next apply.
  • The rollout of a node with roles failed: the main pod runs the previous image, and the roles have no pods, until the next apply.
  • The rollout of a role failed: the main pod runs the new image, and that role has no pods, until the next apply.
  • The cluster has no volume for the claim of a node: the pod does not get scheduled, does not become ready, and the program sets the Deployment back as for a failed boot.
  • A node is removed from the group, or gets a different name: the graph removes its claim, and the state of the node is lost when the StorageClass deletes the volume.
  • The spire node is not reachable at plan time: the plan fails, also the plan of a destroy, because the graph reads the trust bundle from the spire node.
  • The spire node is made again, with a new machine ID: the next plan replaces the hand-off, and the apply gives the new node the kubeconfig and the drop-files again.
  • The SPIRE server rotates its CA: the next plan changes the ConfigMap spire-agent, and the apply gives the pods the new bundle.
  • The cluster is made again, with a new UID of the namespace kube-system: the next plan replaces the hand-off. The apply retires the names of the old UID, registers the names of the new UID, and makes new pods. The agents have no identity until the new pods attest.
  • The roster of the group has a spiffe provider: the evaluation fails, because that spire node would be a pod of the same deployment.
  • No roster has a spiffe provider: the method registers nothing, and the unprovided assertion of the spiffe resource refuses a node that needs an identity.
  • A submission is given: the graph drops the nodes that the submission turns off, and an apply with a different submission fails (spec.md).

Requirements (mandatory)

Functional Requirements

  • FR-001: The method MUST run its OpenTofu graph through the shared OpenTofu runner of spec.md, with OpenTofu, its providers, skopeo and kubectl from the Nix store, and MUST keep its state on the HTTP state backend of tf.md.
  • FR-002: For each node, the graph MUST build the image of the node configuration on the deployer and push it to <registry>/<namespace>/<node>, with a tag from the node configuration. The push MUST be replaced when, and only when, the tag changes.
  • FR-003: The Deployment of a node MUST name the image by the digest that the registry gives for the pushed tag, so that a later change of the tag does not change the pod.
  • FR-004: The main Deployment <node> of a node MUST hold one pod, MUST replace the pod and not run two pods at a time, and MUST give the pod the RuntimeClass of the method, its own user namespace, and the settings that systemd needs as PID 1 (see core/checks/integrations/k3s-oci/nixosTest.nix).
  • FR-004a: For each role of a node (the pod-roles resource), the method MUST apply a Deployment <node>-<role> from the image of the node, with the number of replicas of the role and the settings of FR-004 and FR-005. The method MUST give PID 1 of these pods the argument fediversity.role=<role>. The method MUST apply this Deployment after the rollout of the main Deployment, and MUST wait for its rollout as FR-006a states. The method MUST refuse a role whose name is not a DNS label, and a name of a Deployment that is too long or that is the name of a different Deployment.
  • FR-004b: Each unit of a role MUST run only in the pods of that role. Each unit of mainUnits, each unit that orders a certificate and the metrics shipper of the node (otel-node-metrics) MUST run only in the main pod, because all pods of a node have the same host name. Each unit of sharedUnits and each unit in the list of the units of NixOS and of the platform in k8s-pod.nix MUST run in all pods. When a role runs the image of the node, a unit with no class MUST stop the evaluation, as pod roles FR-014 states. A skipped unit MUST NOT fail the units that require it. Without roles, the image MUST NOT change.
  • FR-004c: When a unit of a role fails and systemd does not restart it, the container MUST stop, thus kubernetes restarts only that container.
  • FR-004d: For a role with a service, the Deployment <node>-<role> of FR-004a MUST run the image of that modular service (system.build.serviceImages), from the repository <image of the node>-<role>. The method MUST push this image. The image MUST have the variables of fediversity.podRoles.services.<name>.environment. The container MUST run as a user that is not root, with no capabilities and a read-only root file system. The unit of the service MUST NOT run in a pod from the image of the node.
  • FR-004e: For a role with service and secrets, the Deployment MUST also have a native sidecar (restartPolicy: Always in initContainers), from the image system.build.serviceSecretImages.<role> in the repository <image of the node>-<role>/secrets. The sidecar MUST attest with k8s_psat under the identity of the node, and MUST write each secret to a file in /run/secrets, decoded when the secret has an encoding, with only the value of the variable when the secret has a variable, and with the text of the template when the secret has a template. After a new value of a secret with restart, the sidecar MUST stop the service, and kubernetes then starts the service again. Only a pod with such a secret MUST share its process namespace, and only its sidecar MUST have the capability KILL. When the node verifies OpenBao with services.vars-openbao.caCertFile, the sidecar MUST verify OpenBao with the CA from fediversity.openbao-pki.caFetchUrl. The startup probe of the sidecar MUST pass only when all files exist, thus the service MUST NOT start before it has its secrets. The service MUST mount only the volume of these files, read-only, with no access to the token of the pod or to the SVID. The sidecar MUST write a new value of a secret to the file. A new image of the sidecar MUST give a rollout, as a new image of the service does (FR-006).
  • FR-004f: For a role with ports, the method MUST apply a Service <node>-<role> of the type ClusterIP, with these ports, and a NetworkPolicy <node>-<role>. The NetworkPolicy MUST let in, on these ports only, the pods of the namespace and the gate sources of the node, which are the sources that reach the ports that the node serves. A role with no ports MUST have neither.
  • FR-004h: For a role with metrics, the method MUST apply a headless Service <node>-<role>-metrics (clusterIP: None), with the metrics port of the role, and a NetworkPolicy <node>-<role>-metrics that lets in only the main pod of the node, on that port only. The main pod MUST get the name of the node in the variable FEDIVERSITY_NODE. When the node has the metrics shipper, the shipper in the main pod MUST scrape each address of that Service, and MUST ship the series with the label role set to the role, and the label pod set to the address of the pod. The evaluation of the effect MUST fail when <node>-<role>-metrics has more than 63 characters.
  • FR-004i: A role MUST NOT receive connections from outside the cluster. The admin access of the operator to a pod of a role is kubectl exec.
  • FR-004g: For a role with companion, each pod MUST also have a second container companion from the image of the service, with the command /bin/companion, the volume of FR-004e as read-only at /run/secrets, and an empty directory at /var/lib/companion. This container MUST have the restrictions of the container of the service, and MUST NOT have a probe, because its work must not stop a rollout.
  • FR-005: The pod MUST be ready only when systemd in the pod is running. FR-005 does not apply to a role with a service, because its image has no systemd.
  • FR-006: The apply of the Deployment MUST NOT wait for the rollout. A separate step MUST wait for the rollout, for a time that the operator can set. If the rollout does not complete, that step MUST set the Deployment back to the previous image and MUST fail. FR-006 applies to the main Deployment of a node. FR-006a applies to the Deployment of a role.
  • FR-006a: Before the method applies a new image of a node to the main Deployment, it MUST set the replicas of the Deployment of each role without a service to 0, and MUST wait until the pods of these roles are gone. For a role with a service, the method MUST do this before it applies a new image of that service, and before the main Deployment. After the rollout of the main Deployment, the method MUST set the replicas of each role back to their number, and a separate step MUST wait for the rollout of the role, for the time of FR-006. If the rollout of the main Deployment does not complete, the roles MUST stay at 0 replicas until the next apply. If the rollout of a role does not complete, that step MUST set the replicas of that role to 0 and MUST fail. It MUST NOT set the role back to the previous image, because the main pod then has the new image and its migrations are done. The main Deployment MUST stay on the new image, and the next apply MUST start the role again.
  • FR-006b: When the new image of a node has a systemd.stateMigrations entry with a record in /var/lib and a version that is not the recorded version, the main pod MUST copy /var/lib to a directory of the same volume before a unit of the node starts. When the rollout does not complete and the previous image starts, that pod MUST put the copy back in /var/lib before a unit of the node starts. After a successful rollout of the main Deployment, the method MUST remove the copy.
  • FR-007: The method MUST create the namespace of the target, and a destroy MUST remove it. The namespace of the all target is K8S_NAMESPACE, fediversity when unset. The namespace of a per-app target is that name with -<app> appended, thus two targets do not hold the same Deployment.
  • FR-008: The method MUST read the address of the registry from K8S_REGISTRY, and MUST stop with an error that names it when it is unset. The method MUST read the kubeconfig of the cluster from the file that KUBE_CONFIG_PATH names, and the credentials of the registry, if any, from the file that REGISTRY_AUTH_FILE names. These are per-run values and MUST NOT be in the node plans. The hosting provider keeps the kubeconfig as a secret. The k8s profile of .forgejo/workflows/cd-env.sh writes the secrets KUBE_CONFIG and REGISTRY_AUTH to these files.
  • FR-009: The effect MUST publish its node names, as FR-016 of spec.md states.
  • FR-010: The method MUST apply NetworkPolicies in the namespace of the target, before the Deployments: one that refuses all connections to the pods of the namespace, and one for each node, from the settings of the firewall resource of the node. The pods of the namespace and the sources of the gate (internalSources, adminSources) MUST reach only the ports that the node serves on eth0. All sources MUST reach the ports that the node publishes. operatorSources MUST reach the operator ports and the public ports.
  • FR-011: The node MUST keep the address and the routes that the network plugin of the cluster puts on eth0. The gate of the firewall resource on the node MUST NOT filter eth0.
  • FR-011a: The node MUST keep the /etc/resolv.conf that the cluster gives the pod, thus its resolver is the DNS of the cluster.
  • FR-012: For each node, the method MUST apply a PersistentVolumeClaim <node>-state, of a StorageClass and a size that the operator can set, and the main pod MUST mount it on /var/lib. The claim MUST stay when the pod is replaced.
  • FR-013: The machine ID of a node MUST stay the same when its main pod is replaced, and MUST be different for each node. The method MUST give it to systemd as container_uuid, from a random UUID in the state of the graph.
  • FR-014: The host keys of sshd on the node MUST be in /var/lib.
  • FR-015: When the method has the coordinates of a SPIRE server node, the SPIRE agent of each node MUST attest with k8s_psat, with a projected token of the service account node-<node> of the namespace, of the audience spire-server. Only root on the node MUST be able to read the token. The agent MUST read a copy of it, which a root service makes again within the life of the token.
  • FR-016: Before the Deployments, the method MUST give the spire node a kubeconfig of the cluster with a token of the service account spire-server of the namespace, which can review tokens, and a drop-file for each node (spire, FR-017). The token MUST go from kubectl to the spire node through a pipe, and MUST NOT be in an argument, in the state, or on stdout. The method MUST give the pods the trust bundle of the server in the ConfigMap spire-agent, from a read at plan time, and MUST NOT make a change at each apply. The name of the cluster on the server MUST be <cluster UID>-<namespace>, where <cluster UID> is the UID of the namespace kube-system, from a read of the target cluster at plan time. Two clusters with the same namespace and node names MUST be able to use one SPIRE server. A change of the name MUST make new pods. A destroy MUST retire the names of its cluster entry and remove its kubeconfig on the spire node.
  • FR-017: The setup MUST give the method the coordinates of the spire node of the central group: its FQDN, and the user, the key, the host key and the ssh options of the SSH_USER, SSH_PRIVATE_KEY_FILE and SSH_NODES layers, as for a node of the ssh method. The setup MUST read the address of the API server for the spire node from K8S_SPIRE_API_SERVER, and MUST use the address of the kubeconfig when it is unset. The path of the key MUST NOT be in the input or in the triggers of a resource, because each deploy stages the key to a different path. A destroy MUST use SSH_PRIVATE_KEY_FILE. When the roster of the group has a spiffe provider, the evaluation MUST fail with an error that names the group.
  • FR-018: For each node whose firewall settings have published ports, admin ports or operator ports, the method MUST apply a Service <node> of the type LoadBalancer, with these ports and the policy externalTrafficPolicy: Local. The Service MUST NOT have a port that the settings do not name. Under k8s-hosts, the method MUST NOT wait for the address of the load balancer.
  • FR-019: Under k8s-operator-hosts, before the rollout of a node with published names, the method MUST wait for the load balancer to give the Service of the node an address, for a time that the operator can set, and MUST stop with an error when no address comes. The method MUST publish the published names of the node at that address, in the zone of the deployment, with the DNS lane of the operator graph of tf-incus-operator-hosts. A node with published names thus MUST have a Service.
  • FR-019a: Under k8s-operator-hosts, when the setup gives the method a NetBox, the method MUST allocate the address of the Service of each node with published names from the pool of the hosting provider in that NetBox, and MUST ask the load balancer for that address. The method MUST stop with an error when the load balancer gives the Service a different address. A destroy MUST release the address in NetBox. Without a NetBox, the Service gets the address that the load balancer selects.
  • FR-019b: Under k8s-operator-hosts, when the setup gives the method a Gateway of the cluster as <namespace>/<name> (K8S_GATEWAY), a node with published names whose ports for clients are only published TCP ports 80 and 443, with port 443 among them, MUST share the address of that Gateway. An admin port that is not also a public port is not a port for clients. For such a node, the method MUST apply a Service of the type ClusterIP in place of the Service of FR-018, a TLSRoute for port 443, and an HTTPRoute for port 80 when port 80 is a port for clients of the node. Each route MUST have the published names of the node as its hostnames, and MUST refer to the Gateway. The node terminates TLS itself, thus the Gateway MUST NOT terminate it. The method MUST stop with an error in the plan when the Gateway does not exist, has no IP address, has no listener for the protocol TLS on port 443 with the mode Passthrough, or has no listener for the protocol HTTP on port 80. A destroy MUST NOT need the Gateway. Before the rollout, the method MUST wait for the Gateway to accept the routes, for the time of FR-019, and MUST stop with an error when it does not. The method MUST publish the published names at the address of the Gateway, and MUST NOT allocate a NetBox address for the node. Any other node MUST keep its Service of FR-018.
  • FR-020: Under k8s-operator-hosts, the method MUST hold the zone of the deployment, and MUST publish the mail records of the zone after the rollout of the relay node. A destroy MUST retire the records of the published names, and on bind MUST release the zone, as the incus operator graph does.
  • FR-020a: Under k8s-operator-hosts, when the curation of the hosting provider gives the method a domainRegistration, the method MUST register the domain of the deployment at the registrar account of the hosting provider, with the registrar lane of the operator graph of tf-incus-operator-hosts (registrar.tf) and the same options. A domain at or under the apex of the hosting provider MUST NOT be registered. A destroy MUST turn the renewal of the domain off. Without a domainRegistration, the method MUST register no domain.
  • FR-021: Under k8s-operator-hosts, the namespace of a target MUST be the namespace of FR-007 with -<namespace> appended, where <namespace> is the id of the deployment that the API gives. With no id, nothing is appended. The method MUST stop with an error when the result is not a DNS label of at most 63 characters.
  • FR-022: Under k8s-operator-hosts, a suspend MUST set the replicas of each Deployment to 0 and retire the records of the published names. The zone and the mail records MUST stay. A resume MUST set the replicas of each Deployment back to their number and publish the records again.

Key Entities

  • Role: A part of a node that runs in a Deployment of its own, from the image of the node (pod-roles).
  • Node image: A docker archive with the rootfs of the node configuration as its one layer, and /sbin/init as its command.
  • Image reference: <registry>/<namespace>/<node>@<digest>.
  • Deployment: The Kubernetes Deployment of one node, in the namespace of the target.
  • State claim: The PersistentVolumeClaim <node>-state of one node. Its volume holds /var/lib of the node.
  • Cluster coordinates: The kubeconfig of the cluster and the address of the registry.
  • SPIRE coordinates: The address of the spire node, the user, the key, the host key and the ssh options to reach it, and the address of the API server of the cluster as the spire node reaches it.
  • Node NetworkPolicy: <node> in the namespace of the target. It selects the pod of the node by the label app.kubernetes.io/name. The ports that the node serves come from the label eu.fediversity.firewall.served of the image.
  • Node Service: <node> in the namespace of the target, of the type LoadBalancer, or of the type ClusterIP for a node behind the Gateway (FR-019b). A change of the type replaces the Service. It selects the pod of the node by the label app.kubernetes.io/name. Each port has the name <protocol>-<port>, for example tcp-443.
  • Role Service: <node>-<role> in the namespace of the target, of the type ClusterIP, for a role with ports (FR-004f). It selects the pods of the role by the label app.kubernetes.io/name.
  • Role NetworkPolicy: <node>-<role> in the namespace of the target, for a role with ports (FR-004f).
  • Role metrics Service: <node>-<role>-metrics in the namespace of the target, headless, for a role with metrics (FR-004h), with a NetworkPolicy of the same name.
  • Node routes: The TLSRoute and the HTTPRoute <node> in the namespace of the target (FR-019b), which send the published names of the node to its Service. The Gateway that they refer to is of the cluster, not of the method.
  • Node service account: node-<node>, the identity in the token of the pod of one node. The SPIRE node alias is spiffe://fediversity/k8s/<cluster UID>-<namespace>/<namespace>/node-<node>, and the host entry is spiffe://fediversity/host/<cluster UID>-<namespace>-<node>.

Success Criteria (mandatory)

Measurable Outcomes

  • SC-001: After an apply, every node runs as a ready pod, from the image of its configuration.
  • SC-002: After an apply, a plan shows no changes.
  • SC-003: After a deploy whose pod does not become ready, the Deployment runs the previous image, and the program failed.
  • SC-004: After a pod of a node is replaced, the node has the same /var/lib and the same machine ID.
  • SC-005: With the SPIRE coordinates set, the agent of each node has the X.509 SVID of its host entry, and the server lists the agent with the attestor k8s_psat.
  • SC-005a: With a role, each unit runs only in the pods that its list permits. Each pod has its own machine ID and its own agent, and reads the secrets of the node. A failed unit of a role restarts only one container.
  • SC-005b: While the main pod of a node gets a new image, no pod of a role of that node runs. After the apply, each role has its number of ready pods on the new image. After a failed rollout of the main pod or of a role, the role has 0 replicas, and the next apply that completes starts it again.
  • SC-006: A source outside of the cluster connects to the published ports of a node at the Service of the node, and only an operator source connects to the operator ports.

Assumptions

  • The hosting provider runs the cluster. The cluster has the RuntimeClass of the method, with a handler that gives systemd a writable cgroup file system.
  • The kubeconfig of the deployer can read the namespace kube-system.
  • The deployer and the cluster reach the registry at the same address.
  • The deployer can build every image, or substitute it.
  • One operator applies a given state key at a time. The backend lock enforces this.
  • The cluster gives a volume for each claim of the StorageClass of the method. The file system of the volume supports idmapped mounts, because each pod has its own user namespace.
  • The volume of a node has room for its state and for one copy of it, during a rollout that migrates the state. cp --reflink=auto shares the blocks when the file system of the volume supports this.
  • The OpenBao and SPIRE servers stay outside of the cluster. The deployer reaches the spire node over ssh, and the spire node reaches the API server of the cluster.
  • All groups that deploy with this method share the spire node of the central group. A group of this method has no spire node of its own, by design: the agents of all its nodes attest to that one server, and each node has its own host entry (spiffe://fediversity/host/<cluster UID>-<namespace>-<node>), thus a shared server gives no node the identity of a different node. A SPIRE server in the cluster would need the token reviews of its own service account, the drop-files through the API server in place of ssh, a stable Service name for the agents, a NodePort or LoadBalancer for agents outside of the cluster, the export of its trust bundle from the pod, and its data on a claim with the CA key in a KMS plugin.
  • Only a modular service gets an image of its own (FR-004d), by design. A role without a service runs the image of the node, with systemd as PID 1 (FR-004a), because its units come from NixOS modules. These modules need systemd for their credentials, their tmpfiles rules, their OpenBao agents and spire-agent, and a modular service has only process.argv as a part that runs without systemd. All pods of a node thus hold the same closure, and a new image of the node replaces the pods of each role without a service (FR-006a). Such a role still gets its own restart and its own number of replicas. To give a part of a node an image of its own, give it a modular service.
  • The network of the cluster routes to the addresses of the nodes of the hosting provider. CLUSTER_SOURCES of the hosting provider has the addresses from which the pods connect to these nodes, which are the addresses of the cluster nodes when the cluster translates the pod addresses. A pod reaches the same ports of a node as an operator deployment does, and a change of CLUSTER_SOURCES takes effect at the next deploy of the hosting provider.
  • The DNS of the cluster resolves the names outside of the cluster that a node connects to, such as the name of the OpenBao node. A node has no other resolver.
  • A load balancer of the cluster gives the Service of a node an address. With a NetBox, the load balancer gives the address of the annotation metallb.io/loadBalancerIPs (the operator can set a different annotation), or of spec.loadBalancerIP when the operator sets no annotation. Without one, the Service has no address, and sources outside of the cluster connect at its node ports. Under k8s-operator-hosts, a node with published names needs that address.
  • With K8S_GATEWAY, the cluster has that Gateway, with a listener for the protocol TLS on port 443 with the mode Passthrough and a listener for the protocol HTTP on port 80. The allowedRoutes of the listeners admit the namespace of the deployment, and the Gateway has an IP address that clients reach. The cluster has the TLSRoute CRD, and the deployer can get and list the Gateway and write routes.
  • The network plugin of the cluster enforces NetworkPolicy. The cluster accepts a NetworkPolicy also when nothing enforces it. With such a plugin, all pods of the cluster can connect to all ports of the nodes.
  • Under k8s-operator-hosts, the method does the DNS lane, the registrar lane, the NetBox address of the Service, and what an operator deployment has in the central services: the S3 buckets and key pairs on Garage, the databases and login roles on PostgreSQL, the static roles of the OpenBao database engine, and the entries in the KV of OpenBao. It shares this with tf-incus-operator-hosts (core/effects/tf/operator-provision/, core/effects/tf/incus/operator/registrar.tf). A run needs TF_VAR_openbao_token and TF_VAR_garage_admin_token, and the addresses document at hpStateAddress needs the addresses of openbao, garage and postgresql (or TF_VAR_openbao_host, TF_VAR_garage_host and TF_VAR_postgresql_host). A run that registers a domain needs OPENPROVIDER_USERNAME and OPENPROVIDER_PASSWORD. A pod has no node address: the address that the method gets from NetBox is the address of the Service (FR-019a).

Known Deviations

  • Without a NetBox, the address of the Service of a node can change when the cluster gives the Service a new address. The records then point to the old address until the next apply.

  • Behind a Gateway, a node sees the address of the Gateway as the source of a connection, not the address of the client. The rule with no from of the NetworkPolicy of a published port lets the Gateway connect.

  • A node with a port for clients other than the published TCP ports 80 and 443 stays on its own address, also with a Gateway. A Gateway has no route for such a port with the source gate of FR-018, thus such a node shares no address.

  • Behind a Gateway, the Service of a node has its admin ports, but the type ClusterIP keeps them in the cluster, thus ssh from outside of the cluster does not reach the node. Use kubectl exec.

  • A role with a service and no secrets has no volumes and no identity of SPIRE. A role with secrets (FR-004e) has the identity of the node in its sidecar, thus it can read all secrets of the node. A role cannot have less access than its node.

  • One sidecar does the work of an init container and of a sidecar, with one attestation of the pod. The service restarts on a new value only for a secret with restart. For each other secret, it must read the file again. A pod with a secret with restart shares its process namespace, thus the service can see the processes of the sidecar, but not their environment or their files.

  • The sidecar gets the CA of OpenBao over plain HTTP, as the unit openbao-pki-ca-fetch of a node does. It checks for a new CA each 15 minutes. It restarts when the CA changes, and has the old secrets until it has logged in again. A node with services.vars-openbao.caCertFile and an empty fediversity.openbao-pki.caFetchUrl fails the build of the image of the sidecar.

  • Only lldap, authelia, valkey, the audit collector (fediversity-audit-ingest) and prom-label-proxy have a modular service. Only deployment-k8s-hosts runs lldap and authelia in a role with a service. No check runs valkey in a role: the pod of a role has no volume for state, thus a valkey instance in a role loses its data and the password of its app user when the pod restarts. No check runs the audit collector in a role: the collector does not identify its client, and no rule limits the connections to its pod to the nginx that checks the credential of a source. No check runs prom-label-proxy in a role, for the same cause: the proxy reads the deployment from a header, and no rule limits the connections to its pod to the nginx that checks the credential of a deployment. No node of a deployment has such a role: the module of authelia has the variables of its image and the rules for a role, in core/resources/oidc/authelia-role.nix, and the check sets the role and its secrets by hand.

  • A pod of a role has no volume at /var/lib and no container_uuid. It makes new host keys and a new machine ID each time it starts, and a unit of a role loses what it writes to the disk. All pods of a node have the same host name.

  • The units of a role and the units of the main pod get the same closure, the same firewall and the same secrets. A role cannot have less access than its node.

  • The method does not run migration-quiesce stop: the stop of the pods of a role (FR-006a) takes its place. The roles of a node thus have no pods from the start of the rollout of the main pod until their own rollout, also when the new image has no migration. The units that run after a migration in the main pod start before the pods of a role start. A role with a service stops only for a new image of that service. deployment-k8s-hosts proves the stop for a role without a service only by the result of a failed rollout and by the sequence of the steps in the output of the apply.

  • FailureAction=exit takes effect only when systemd stops the restarts of a unit. A unit with Restart=always that fails more slowly than the start limit restarts inside its container, and kubernetes does not see the failures.

  • The method does not run the deploy body of spec.md. The readiness of the pod takes the place of the verdict. After a failed rollout, the program shows the failed units of the pod, or the log of the pod if the container exited.

  • After a rollback, the state holds the image that failed, and the cluster runs the previous image. A plan does not show this difference.

  • Only /var/lib stays when the pod is replaced. A node that keeps state in a different path, such as /etc, loses it on each rollout. The persistentPaths of a component are under /var/lib.

  • The service account spire-server can read all kubernetes nodes of the cluster, because the k8s_psat server reads the node of the pod.

  • A group that deploys with this method cannot have its own OpenBao node. OpenBao seals its unseal key to the TPM, and a pod has no TPM. The evaluation of the node fails with an assertion.

  • A node of this method cannot have an .age secret of the secrets resource. Such a secret is encrypted to a committed host key (core/keys/systems), and a pod makes its host keys on its first boot. The evaluation of the node fails with an assertion. The generateFiles resource does not use the host key.

  • The agent attests the node, not the workloads of the node by their pods. Workloads attest with the unix and systemd attestors of the agent.

  • The gate of the firewall resource does not filter the pod network. The NetworkPolicy of the node is the gate of that network, with these differences:

    • A NetworkPolicy has no match for ICMP. The rules of the policy are for TCP and UDP only, and the network plugin decides about ICMP.
    • The rules of a NetworkPolicy only add. An operator source that is also a source of the gate, or a pod of the namespace, reaches all ports that the node serves. The gate limits such a source to the operator ports and the public ports.
    • A rule on a source address (ipBlock) applies only when the cluster keeps the source address of a connection from outside of the cluster. The policy Local of the Service keeps it at the node ports. When a load balancer in front of the node ports changes it (SNAT), only the rule on the published ports applies to that connection.
    • The Service has no port ranges, and the ports that the node serves are not in the settings, thus not in the Service. A gate source outside of the cluster reaches those ports only at the address of the pod, when the network of the cluster routes it.
    • The interfaces of the node other than eth0 do not get a policy. bridgeInterfaces and trustedInterfaces do not apply.
    • When the image of a node changes, the ports that the node serves are unknown until the push, thus a plan does not show them.