Skip to content

Add a testcase and a fix for testbed deadlock when running low on free space

Florent 'Skia' Jacquet requested to merge ubuntu-ci-team/autopkgtest:skia/fix_deadlock into master

When running low on free space, there is a situation where the testbed will deadlock. This happens (at least) when lib/adtlog.py:report() fails to write its report on disk, for some reasons (I suspect something with the atexit code in runner/autopkgtest, but nothing sure).

The testcase triggers exactly that situation with precise timing and disk space management. The other commit fixes the bug, but still has a TODO because I don't really know what would be the best behavior. That needs to be discussed in this MR.

Here is the command to run the test. I run it in a podman container with --cap-add SYS_ADMIN, because it needs mount. That's also something that may be discussed to see if some fuse filesystem would be better suited:

./tests/autopkgtest NullRunnerRoot -k no_space_left_on_device

I'm open to any suggestion and discussion here. We have the problem in production in the Ubuntu infrastructure, but didn't have yet the time to run this code for broader tests, since it's all fresh from this week.

This work originated from this Ubuntu MP, this bug, and this one too.

Merge request reports