A deploy from a workstation: the output of the nix run command.
A deploy that the panel started: the panel shows the end of the log. The
full log is in Windmill, at https://jobs.<apex>/, in the fediversity
workspace. The jobs of the f/fediversity/deploy and f/fediversity/destroy
scripts are the deploys and the removals.
A different request to the panel or the API: only a deploy and a removal
start a Windmill job. The services handle all other requests in their own
process, for example the list of deployments, the forms and the build logs.
These services write their requests and errors to the journal of their
container: the unit fediversity-api on the node api, and the unit panel
on the node panel.
Service in a container:ssh admin@<incus> 'sudo incus exec <node> -- journalctl -xeu <service>'
All containers in one place: each node sends its journal, at priority
info and higher, to Loki. In Grafana at https://monitoring.<apex>/, open
"Explore", select Loki, and find the records of a unit with a query such as
{service_name="fediversity-api.service", service_instance_id="api"}. The
records of a unit include the lines that systemd writes about the unit, as
journalctl -u shows them. The dashboard "Logs and traces" shows the same
records without a query.