Feature Specification: Resource `pod-roles`

Created: 2026-11-07

Status: Implemented

Input: Issue #2627, Option A: the same image of the node, with one Deployment for each part of the node.

User Scenarios & Testing (mandatory)

The pod-roles resource tells which parts of a node run as pods of their own, when the deployment method runs the node as a pod. It wraps no contract. Only the k8s method acts on it. On other methods, a node runs all its units as before. The shared rules are in the resource family spec.

User Story 1 - Scale a part of a node on its own (Priority: P1)

An operator wants kubernetes to restart and to scale the workers of an application, and not the entire node. The application requests a role with a number of replicas, and its node module tells which units run in that role.

Why this priority: This is the first step for kubernetes to manage the services of a node apart from each other.

Independent Test: The unit test resources-pod-roles and the VM test deployment-k8s-hosts.

Acceptance Scenarios:

Given an application that requests the role sidekiq with 2 replicas, When the method deploys the node, Then the node has a Deployment <node>-sidekiq with 2 pods from the image of the node.
Given a unit of the role, When the pods start, Then the unit is active only in the pods of the role.
Given a unit of mainUnits, When the pods start, Then the unit is active only in the main pod.
Given a unit of sharedUnits, When the pods start, Then the unit is active in all pods.
Given an application with no request, When the node evaluates, Then the image of the node does not change.
Given a request for a role with a service, When the method deploys the node, Then the pods of the role run the image of that modular service, with no systemd, and no pod from the image of the node starts the unit of the service.
Given a request for a role with a service and secrets, When the method deploys the node, Then the service starts after each secret is a file in /run/secrets, and the service cannot read the identity of the pod.
Given a request for a role with ports, When the method deploys the node, Then the pods of the role receive connections on these ports, at a Service of the role in the cluster, from the sources that reach the ports that the node serves.
Given a secret of a role with the encoding base64, When the pods start, Then the file in /run/secrets holds the decoded value.
Given a modular service with variables in fediversity.podRoles.services, When the pods of its role start, Then the service has these variables.
Given a secret of a role with a variable, whose value is a line of an environment file, When the pods start, Then the file in /run/secrets holds only the value of the variable, with no line end.
Given a secret of a role with a template, When the pods start, Then the file in /run/secrets holds the template, with the values from the response of OpenBao.
Given a secret of a role with restart, When the sidecar writes a new value of the secret, Then the container of the service restarts in the same pod. A new value of a secret with no restart, or a restart of the sidecar, does not restart it.
Given a request for a role with a service, secrets and companion, When the pods start, Then each pod also has a second container from the image of the service, which runs /bin/companion, reads the secrets of the role, and connects to the service on the loopback address of the pod.
Given a role that runs lldap and a reader of the ldap contract in a different pod of the node, When the reader connects with the coordinates of the contract, Then it reaches the Service of the role by its name over LDAPS, it verifies the certificate of the role with the CA file that the contract returns, and it binds with the password in the file that the contract returns.
Given a secret of a role with parameters and a template, for an engine of OpenBao that issues a value at each read, When the pods start, Then the sidecar sends the parameters in its request, and the file in /run/secrets holds the template, with the values from the response.
Given a role that runs the image of the node and an enabled unit that is in no list, When the node is evaluated, Then the evaluation stops with the name of the unit.

User Story 2 - Find a wrong name before the deploy (Priority: P2)

A condition on the name of a unit makes a unit with that name, thus a wrong name does not fail by itself. The real unit then runs in all pods, and it can write the same data from more than one pod.

Why this priority: The fault gives no error, and it can damage data.

Independent Test: The evaluation of a node on k8s-pod.nix, and the unit test resources-pod-roles for mastodon.

Acceptance Scenarios:

Given a role or a main list with a name of a unit that does not exist, When the node evaluates, Then an assertion names the unit.
Given a unit in two lists, When the node evaluates, Then an assertion refuses the node.
Given a role with units and no request, or a request and no units, When the node evaluates, Then an assertion names the role.
Given a role with a service that is not a modular service of the node, or with other units, When the node evaluates, Then an assertion names the role.
Given a role with secrets and no service, or a secret whose name is not the name of a file, When the node evaluates, Then an assertion names the role or the secret.
Given a role with the image of the node and a port or a metrics port that the firewall of the node does not open, When the node evaluates, Then an assertion names the role.
Given variables in fediversity.podRoles.services for a modular service that does not exist, When the node evaluates, Then an assertion names the service.
Given a secret with a template and also a field, an encoding or a variable, When the node evaluates, Then an assertion names the secret.
Given a role with companion and no secrets, or with no program in fediversity.podRoles.services for its service, When the node evaluates, Then an assertion names the role.
Given a role that runs lldap with no companion, with a loopback address in fediversity.lldap.host, with a file in /run/secrets in fediversity.lldap.readerBindPasswordFile, or with no fediversity.lldap.tls.enable or fediversity.lldap.tls.caFile, When the node evaluates, Then an assertion refuses the node.
Given a role that runs lldap on a node with lldap-publish-roles, When the role does not answer, Then the unit waits for its restart with no failed state, the main pod becomes ready, and the unit publishes the members when the role answers.
Given a secret with parameters and no template, When the node evaluates, Then an assertion names the secret.

Edge Cases

A unit of a role requires a unit of a different role. The condition skips the required unit, and a skipped unit does not fail the units that require it. The two units do not connect through local files.
The node module of an application sets units on an environment that is not k8s. The options do not change the units there. Only the assertions on the names apply.

Requirements (mandatory)

Functional Requirements

FR-001: The resource MUST take a request with the roles of a node. Each role has a name and a number of replicas, with the default 1.
FR-002: A request MUST hold only data, because the method reads the roles from the plan.
FR-003: The policy MUST give the replicas to the option fediversity.podRoles.roles of the node, and MUST merge the roles of all requests of the node.
FR-004: The node module of an application MUST state the units of each role and the units of the main pod, because the names of the units depend on the configuration of the node.
FR-005: The node MUST run in each pod only the units that its role permits, as k8s FR-004b states.
FR-006: A unit that must run one time, such as a unit that writes shared data or that sends a certificate order, MUST be in a role or in mainUnits.
FR-007: units, mainUnits and sharedUnits MUST NOT read config.systemd.services, because the names in these lists are names of systemd.services in k8s-pod.nix.
FR-008: A node with a request MUST fail its evaluation on each method when a list names a unit that does not exist, or a unit in two lists. A test of the application thus finds a changed name without k8s-pod.nix.
FR-009: A request MUST be able to give a role with a service a set of secrets. Each has the name of a file in /run/secrets, the full path of a secret in the KV v2 API of OpenBao, a field, the encoding of that field, and optionally the variable to read when the field is a line of an environment file. In place of the field, the encoding and the variable, a secret MUST be able to have a template, for a secret of a different engine of OpenBao, and with the template the parameters of its request to that engine. A secret MUST be able to restart the service when its value changes, for a service that reads the file only when it starts. A role with secrets and no service MUST fail its evaluation, because a pod from the image of the node gets its secrets from its own units.
FR-010: A request MUST be able to give a role a set of ports, each with a number and a protocol. A role with the image of the node and a port that networking.firewall of the node does not open MUST fail its evaluation, because the firewall in the pod would drop the connections.
FR-010a: A request MUST be able to give a role a metrics port and path, at which its pods serve metrics in the Prometheus format. The default is no metrics. A role with the image of the node and a metrics port that networking.firewall of the node does not open MUST fail its evaluation, as FR-010 states. The main pod of the node scrapes these metrics, as k8s FR-004h states.
FR-011: The node module of an application MUST be able to give the image of a modular service environment variables, for settings that apply only to the image. The image holds their values, thus a value MUST NOT be a secret.
FR-012: A request MUST be able to give a role with a service and secrets a second container (companion), for work that must connect to the service on the loopback address of the pod. The node module of the application MUST give the program in fediversity.podRoles.services. The plan holds only data, thus the program MUST be at a fixed path in the image of the service.
FR-013: The roles MUST change the units of a node only under the k8s method, by design. On each other method, the node MUST run each unit of each list, because one machine runs the entire node and has no pods to split it into. Only the assertions of FR-008 MUST apply there.
FR-014: With k8s-pod.nix, when a role runs the image of the node, each enabled unit of the node MUST be in a role, in mainUnits, in sharedUnits, or in the list of the units of NixOS and of the platform in k8s-pod.nix. Otherwise the evaluation MUST stop with the names of the units. A unit of sharedUnits MUST NOT also be in a role or in mainUnits. A new unit thus has no pod until a person gives it a class.

Key Entities

Role: A part of a node that runs in pods of its own, from the image of the node or of a modular service. Its ports are the ports at which its pods receive connections.
Main pod: The pod of the Deployment <node>, which holds /var/lib and has the identity of the node.

Success Criteria (mandatory)

Measurable Outcomes

SC-001: deployment-k8s-hosts shows each unit active only in the pods that its list permits.
SC-002: A wrong name of a unit stops the evaluation of the node, on each method.
SC-003: deployment-k8s-hosts runs the image of lldap from the real provider as a role with 2 replicas. A login through the Service of the role and a login at each pod succeed, a replica accepts a token from the other replica, the secrets of the role come from its sidecar, and its data is in a Postgres out of the pod, with a credential from the database engine of OpenBao. A new credential restarts the service of each replica in the same pod, and a login at each pod then succeeds. The second container of the role makes the user authelia-bind, and Postgres holds its one state of OpenTofu.
SC-004: In deployment-k8s-hosts, a reader in the main pod finds the user authelia-bind with the coordinates of the ldap contract, at the name of the Service of the role, over LDAPS with a certificate from the PKI engine of OpenBao. The plain LDAP port is not on the Service. The reader binds with the file that the contract returns, which the OpenBao agent of the pod writes from the KV entry that the sidecar of the role also reads. lldap-publish-roles in the main pod writes the members of a role group to OpenBao, and a new member in the directory gets to the KV entry.
SC-005: deployment-k8s-hosts runs the image of authelia from the real provider as a role with 1 replica. The nginx of the main pod serves the portal and connects to the Service of the role over HTTPS, with a certificate from the PKI engine of OpenBao that nginx verifies. A login of the user admin through the portal succeeds: the role binds to the Service of the role lldap over LDAPS, and stores the login in a Postgres out of the pod, with a credential from the database engine of OpenBao. A new credential restarts the service in the same pod, and a login then succeeds.

Assumptions

A unit of a role connects to other units over the network, and not through files in /var/lib or /run.
Each pod from the image of the node gets its secrets from OpenBao, as the main pod does. A pod with the image of a service gets them from its sidecar.

Known Deviations

No k3s test splits a real application into roles. deployment-k8s-hosts splits probe units and the service images of lldap and authelia, and the tests of an application with a role prove only the names of its units. Needs: a check that runs mastodon with the role sidekiq on k3s, with Postgres and Redis on Services, its keys from OpenBao, and its media on Garage.
lldap runs as a role only in deployment-k8s-hosts. The node module gives the name of the Service of the role in fediversity.lldap.host, because a node does not know the name of its Deployment.
The HTTP port of lldap has no TLS and stays on the Service of the role in deployment-k8s-hosts. No client of the check sends a password to it through the Service, except the check itself. The plain LDAP port still listens in the pod, because LDAP and LDAPS have the same ldap_host, and it is not on the Service.
The PKI engine issues a new certificate at each read, thus a restart of the sidecar writes a new value. A secret with parameters and restart thus also restarts the service when its sidecar restarts.
deployment-k8s-hosts adds the domain of its namespace to allowedDomains of the PKI role internal, thus each consumer of OpenBao in the check can issue a certificate for that domain. A role out of the check needs a PKI role of its own for the name of its Service.
The file of the bind password of a reader has its own option, fediversity.lldap.readerBindPasswordFile, because the sidecar writes bindPasswordFile only in the pods of the role. The node module gives the option, and renders the file from the KV entry of the bind password.
The module of authelia gives its image the variables of its secrets, with each file in /run/secrets under the base name of its path in the unit. A node with a role that runs authelia adds the two variables that make the role serve HTTPS from /run/secrets/tls.pem, and the nginx of the main pod connects to fediversity.authelia.host over HTTPS and verifies the certificate against fediversity.authelia.caFile. The evaluation stops for such a node when host is a loopback address, when caFile is null, when the storage is a local file, when smtp is null, or when forwardAuth.enable is true. SC-005 runs authelia as a role.
systemd.stateMigrations.lldap stops the unit lldap, thus it does not stop the pods of a role. A node with this migration must not run lldap as a role.
The second container of the role lldap does the work of lldap-provision, and the main pod does not start this unit. The unit would try again without end, and the main pod must be ready before the role starts.
The state of OpenTofu of the second container of lldap is in the database of lldap, in the schema lldap_provision, with the values of the objects. It is not in the pod, because a new image makes a new pod, and OpenTofu deletes an object that leaves the configuration only when the state has it.
A container of a pod cannot stop with no restart, thus the second container sleeps after an apply. It applies again only when it restarts.
A new value of a secret with restart also stops the second container, because it has the same user as the service. It then applies again.
Each sidecar reads a secret on its own, thus a new value of a secret with restart restarts the service in all replicas at about the same time.
Each process of lldap loads the list of revoked tokens only when it starts. A token that a logout revokes at one replica thus stays valid at the other replicas until they restart or the token expires, at most one day later.
On an empty database, the replicas of lldap make the schema and the admin user at the same time. A replica that fails on this exits, and kubernetes restarts it in the same pod.

Feature Specification: Resource pod-roles