Worker runs out of disk space
We have workers that fail with ENOSPACE because they hit limits on /tmp. Signing workers always have /tmp on tmpfs, but other workers can have /tmp on tmpfs just because it's a common default configuration... we should stop relying on /tmp to run the task except when we specifically want that (in case of signing workers to not leak key material to disk).
We should use subdirectories of the worker's home directory for instance /var/lib/debusine/worker/ for external workers, and /var/lib/debusine/signing/ for signing workers. A local admin could then install symlinks to move those directories to /srv where they have more diskspace (in case /var is a separate partition).
Maybe we should have some configuration setting to let the admin point to custom directories (in /etc/debusine/{worker,signing}/config.ini).
We configure the debusine-signing.service
unit with
TemporaryFileSystem=/tmp:mode=01777 /var/tmp:mode=01777
For these mounts, systemd
applies
src/basic/mountpoint-util.h:#define NESTED_TMPFS_LIMITS ",size=10%,nr_inodes=400k"
Freexian's signing workers operate with 32GB of RAM, which should be plenty, but taking 10% of that results in a 3.2GB /tmp, which happens to be too small for Linux. It is reasonable to run a signing worker at less RAM and Linux is a central use case.
This problem can be worked around using an override
[Service]
TemporaryFileSystem=
TemporaryFileSystem=/tmp:mode=01777,size=50% /var/tmp:mode=01777,size=50%
Is adding such a size argument appropriate for the default service configuration?