Re-implement test-crossbuild-arm64 job so it is less likely to fail, and can be enabled by default for all packages
The test-crossbuild-arm64 was disabled by default in 112bee83.
Before we re-enable it by default for all packages, we need to have a more reliable job to avoid having large numbers of failures that make people just totally ignore the job, rendering it useless.
This issue tries to list some of the requirements and their potential solutions.
Version skew
Currently, when new toolchain versions are uploaded to Debian unstable they may migrate from unstable to testing at different times and cause version skew in a way that makes crossbuilds always fail.
@josch has been working on some remedies
This changed: https://lists.debian.org/debian-release/2024/10/msg00346.html There is just this change to autopkgtest which I'm waiting for before sbuild will block migration of toolchain packages to sbuild if they break cross building: ci-team/autopkgtest!490 (merged)
Additionally, we could make the crossbuild job run on top of Debian testing instead of unstable to ensure the platform is more stable.
Alignment with crossqa.debian.net
The job should also have review from @helmut as he is the main maintainer of https://crossqa.debian.net/ to ensure that the Salsa CI job is testing things in a way that accurately predicts if crossqa.debian.net test will actually pass.
More helpful error messages
When crossbuilds fail, the error message and last lines in the test-crossbuild-arm64 are mostly very cryptic for an average maintainer unfamiliar with how crossbuilds works. The error message should be augmented with a small primer on what the expectation from the maintainer is and maybe some link to Debian policy or developers' reference on how to debug and fix crossbuild issues.
Global temporary flag to disable crossbuilds
Both in cross-build and in reproducible build jobs we might have issues in the build dependencies lower down in the stack that break jobs for thousands of packages. We should explore if we could implement some global 'killswitch' that would turn off the tests if we know they are mostly going to fail and having hundreds of packagers look in vain at the same failures they can't fix would be wasting everybody's time.
See also
See also comments in #374 (closed).