Deployment topology

This page shows which nodes a deployment has, where each node runs, and which networks connect the nodes. It also shows where the trust boundaries of the threat model are in that topology. The glossary defines the terms.

The build makes the diagrams and the tables of this page from the code: the group definitions (core/setups/groups), the components (core/components), the constants of the bridge (core/components/hosting-provider/incus-bridge.nix) and the attribute families of core (core/render/catalog.nix). The tables show the groups with the example apex example.org and with no settings from the environment.

Groups and nodes

A group is a set of machines that one deployment makes or changes. core/setups/groups/default.nix defines four groups:

• hosting-provider has the shared services of a hosting provider.
• operator has the applications of one operator and the services that only that operator uses. Each deployment of an operator is one instance of this group.
• hypervisor has the host that runs the Incus daemon. It is a group of its own because it hosts the instances that the deployment of the hosting-provider group makes.
• dev has the forge and the runners of the continuous integration.

Each group has two rosters of machines, published and internal. In the operator group, the published roster has the nodes of the applications and the internal roster has the nodes of the services that the applications use. Each node runs one component. In the tables, the column "Application" tells if an operator can enable or disable the component, and the column "Provides" gives the resources that the component supplies to other components (wiring.provides in core/components/wiringProperties.nix).

Group dev

Group hosting-provider

Group hypervisor

Group operator

Notes for the tables:

• The nodes api and panel are not in the tables. The leaves add them to the published roster of the hosting-provider group (api/default.nix, panel/default.nix).
• The hosting-provider group has the node bind2 only when the environment gives its address (core/setups/groups/hosting-provider.nix).
• The operator group has no database node and no object store node. Its applications use the postgresql and garage nodes of the hosting provider (core/setups/groups/operator.nix).
• The machine incus of the hypervisor group and the machines dev2, dev3, dev4 and fediversity-ci of the dev group each have their own NixOS configuration in core/setups/hosts/.

Deployment methods

A deployment method decides how the nodes of a group come to exist and how they get their configuration. Flake attributes lists all methods and their commands. The table shows the methods that each group has.

The methods give three variants of the topology:

• Machines that exist. The methods ssh and tf deploy to machines that are there before the deployment. The methods provision nothing (core/setups/ssh-hosts.nix, core/setups/tf-hosts.nix).
• Incus instances. The methods tf-incus and tf-incus-operator make one Incus instance for each node on the hypervisor, and then deploy to it (core/setups/tf-incus-hosts.nix, core/setups/tf-incus-operator-hosts.nix).
• Kubernetes pods. The methods k8s and k8s-operator run each node as a pod in a cluster (core/effects/k8s/effect.nix).

The method node-effects deploys nothing.

Incus instances

flowchart TB
  internet(["Internet"])
  operatorPerson(["Operator"])
  admins(["Contributors and administrators"])
  upstream(["Upstream sources and binary caches"])
  backupTarget(["External backup target"])

  subgraph dev["Group dev: machines that exist"]
    dev_dev2["dev2<br/>component forgejo-runner"]
    dev_dev3["dev3<br/>component forgejo-runner"]
    dev_dev4["dev4<br/>component forgejo-runner"]
    dev_fediversity_ci["fediversity-ci<br/>component forgejo-runner"]
    dev_forgejo["forgejo"]
  end

  subgraph hypervisor["Group hypervisor: machine incus"]
    daemon["Incus daemon<br/>API on port 8443"]
    gateway["Bridge address 10.0.100.1<br/>dnsmasq and nix-serve"]
    public["Public segment<br/>bridge pubbr0, second interface eth1 of a node"]
    subgraph bridge["Bridge incusbr0: 10.0.96.0/20"]
      subgraph provider["Group hosting-provider: 10.0.100.2-10.0.100.254"]
        hosting_provider_authelia["authelia"]
        hosting_provider_bind["bind"]
        hosting_provider_garage["garage"]
        hosting_provider_lldap["lldap"]
        hosting_provider_netbox["netbox"]
        hosting_provider_openbao["openbao"]
        hosting_provider_otel_collector["otel-collector"]
        hosting_provider_overview["overview"]
        hosting_provider_postgresql["postgresql"]
        hosting_provider_smtp["smtp"]
        hosting_provider_spire["spire"]
        hosting_provider_telemetry["telemetry"]
        hosting_provider_terraform_backend["terraform-backend"]
        hosting_provider_valkey["valkey"]
        hosting_provider_windmill["windmill"]
        leaves["api and panel<br/>nodes that the leaves add"]
      end
      subgraph operator["Group operator: one /28 of 10.0.104.0/21 for each deployment"]
        subgraph operatorPublished["Roster published"]
          operator_mastodon["mastodon"]
          operator_peertube["peertube"]
          operator_pixelfed["pixelfed"]
        end
        subgraph operatorInternal["Roster internal"]
          operator_authelia["authelia"]
          operator_grafana["grafana"]
          operator_lldap["lldap"]
          operator_smtp["smtp"]
          operator_valkey["valkey"]
        end
      end
    end
  end

  internet -->|"boundary 1"| public
  public --> provider
  public --> operatorPublished
  internet -->|"boundary 2: ports 22 and 8443"| daemon
  operatorPerson -->|"boundary 3"| leaves
  operator -->|"boundary 4: ports of the first table below"| provider
  operator -->|"ports of the second table below"| gateway
  provider --> gateway
  dev -->|"boundary 6"| daemon
  admins -->|"boundary 7"| hypervisor
  upstream -->|"boundary 8"| gateway
  daemon -->|"makes the instances"| bridge
  operator -->|"boundary 5"| hosting_provider_openbao
  hosting_provider_garage -->|"boundary 9"| backupTarget

The diagram shows the hosting-provider group, one deployment of the operator group, the hypervisor and the dev group. The numbers are the numbers of the trust boundaries in the threat model.

Instances

• One Incus daemon on the hypervisor runs the instances of the hosting provider and of all operator deployments (core/setups/groups/operator.nix).
• The name of an instance of an operator deployment is <namespace>-<node>. The deployment of the hosting provider has an empty namespace (core/setups/tf-incus-hosts.nix).
• An instance is a container by default. The setting instanceType accepts container and virtual-machine (core/effects/common/effect-meta.nix).
• A node that uses the /nix/store of the hypervisor must be a container. A deployment can use virtual-machine only if none of its nodes uses that store (core/effects/tf/incus/instance.tf).
• A container gets a software TPM 2.0 by default (enableTpm in core/effects/common/effect-meta.nix). The SPIRE agent of the node uses it to attest.
• The deployment program does not use SSH into a container. It connects to the Incus API on port 8443 of the hypervisor and uses incus exec (core/setups/tf-incus-hosts.nix).
• The nodes of the hosting-provider group that start deployments have a client credential for the Incus API (deployRunnerApps in core/setups/hosts-common.nix).

Network segments

All instances are on one bridge of the hypervisor, with IPv4 only. The bridge is the interface eth0 of each instance.

• The dnsmasq service of the bridge gives each node of the hosting provider a lease and a name (core/components/hosting-provider/incus-bridge.nix).
• Each operator deployment gets a /28 from NetBox. An Incus network ACL on each interface of such a node drops the traffic to and from the other operator deployments (core/components/hosting-provider/incus-bridge.nix).
• A node of the hosting provider accepts internal traffic from the bridge subnet. A node of an operator deployment accepts internal traffic from the provider prefix and from the addresses of its own deployment (internalSources in core/setups/hosts-common.nix).
• The public segment is a second interface, eth1, of an instance. With the uplink mode nat, which is the default of the Incus methods, an instance has no such interface (core/setups/wiring/public-uplink.nix). On the hypervisor, the bridge pubbr0 on a VLAN interface is the parent of the public interfaces (core/setups/hosts/incus.nix).
• A port of a node is open to the internal sources only, until the component that serves the port publishes it (core/setups/wiring/node-selection.nix, core/resources/firewall).

A node of an operator deployment can connect to these ports of the nodes of the hosting provider, by the resource that the node of the hosting provider supplies (core/components/hosting-provider/operator-reach.nix):

The nodes on the bridge can connect to these ports of the hypervisor (core/components/hosting-provider/hypervisor-reach.nix, core/components/hosting-provider/operator-reach.nix):

Kubernetes pods

flowchart TB
  internet(["Internet"])
  registry["Image registry"]

  subgraph cluster["Kubernetes cluster"]
    balancer["Load balancer or shared Gateway"]
    subgraph namespace["Namespace of one operator deployment: default-deny NetworkPolicy"]
    subgraph pod_authelia["Node authelia"]
      pod_authelia_deployment["Deployment authelia<br/>one pod"]
      pod_authelia_state[("PersistentVolumeClaim<br/>authelia-state")]
      pod_authelia_deployment --- pod_authelia_state
    end
    subgraph pod_grafana["Node grafana"]
      pod_grafana_deployment["Deployment grafana<br/>one pod"]
      pod_grafana_state[("PersistentVolumeClaim<br/>grafana-state")]
      pod_grafana_deployment --- pod_grafana_state
    end
    subgraph pod_lldap["Node lldap"]
      pod_lldap_deployment["Deployment lldap<br/>one pod"]
      pod_lldap_state[("PersistentVolumeClaim<br/>lldap-state")]
      pod_lldap_deployment --- pod_lldap_state
    end
    subgraph pod_smtp["Node smtp"]
      pod_smtp_deployment["Deployment smtp<br/>one pod"]
      pod_smtp_state[("PersistentVolumeClaim<br/>smtp-state")]
      pod_smtp_deployment --- pod_smtp_state
    end
    subgraph pod_valkey["Node valkey"]
      pod_valkey_deployment["Deployment valkey<br/>one pod"]
      pod_valkey_state[("PersistentVolumeClaim<br/>valkey-state")]
      pod_valkey_deployment --- pod_valkey_state
    end
    subgraph pod_mastodon["Node mastodon"]
      pod_mastodon_deployment["Deployment mastodon<br/>one pod"]
      pod_mastodon_state[("PersistentVolumeClaim<br/>mastodon-state")]
      pod_mastodon_deployment --- pod_mastodon_state
    end
    subgraph pod_peertube["Node peertube"]
      pod_peertube_deployment["Deployment peertube<br/>one pod"]
      pod_peertube_state[("PersistentVolumeClaim<br/>peertube-state")]
      pod_peertube_deployment --- pod_peertube_state
    end
    subgraph pod_pixelfed["Node pixelfed"]
      pod_pixelfed_deployment["Deployment pixelfed<br/>one pod"]
      pod_pixelfed_state[("PersistentVolumeClaim<br/>pixelfed-state")]
      pod_pixelfed_deployment --- pod_pixelfed_state
    end
    end
  end

  subgraph provider["Nodes of the hosting provider, not in the cluster"]
    openbao["openbao"]
    spire["spire"]
  end

  internet -->|"boundary 1"| balancer
  balancer -->|"Service of a node"| namespace
  registry -->|"image of a node"| namespace
  namespace -->|"boundary 5"| openbao
  namespace -->|"attestation of the SPIRE agents"| spire

The diagram shows one deployment of the operator group with the method k8s-operator. core/effects/k8s/effect.nix makes these objects:

• Each node is one Deployment with one pod. The pod runs an image of the NixOS configuration of the node (core/setups/nixos/k8s-pod.nix).
• Each node has the PersistentVolumeClaim <node>-state on /var/lib. The other paths of the pod start again from the image.
• Each role of a node is one more Deployment, <node>-<role>. A role with a service runs the image of that service as an unprivileged user.
• A NetworkPolicy lets no connection in to the pods of the namespace. One more NetworkPolicy for each node lets in what the firewall resource of the node permits.
• A node with ports for sources outside of the cluster has a Service of the type LoadBalancer.
• The method k8s-operator appends the namespace of the deployment to the name of the Kubernetes namespace. It publishes the names of a node at the address of its Service. It can also connect the nodes to a shared Gateway with a TLSRoute and an HTTPRoute.
• The OpenBao and SPIRE servers stay outside of the cluster. A pod has no TPM, thus the SPIRE agent of a pod attests with the token of its service account.

The deployment does not make the cluster, the image registry, the RuntimeClass, the storage class and the load balancer. They must be there before the deployment.

Trust boundaries in the topology