test-crossbuild-arm64: Depends: libgcc-s1:arm64 but it is not installable

When running my pipeline, I am getting an error on the test-crossbuild-arm64.

https://salsa.debian.org/debian/sstp-client/-/jobs/6088878

It is not obvious how to fix this. The application being packaged is not a C++ application, and only C + GNU autotools is needed to compile it. It should compile and run fine on arm64 as far as I am concerned (runs on raspberry pi).

Anyone able to shed some light to why this is failing. I can of course disable this adding some CI overrides, but I find it strange it fails on a C++ dependency.

Setting up g++-x86-64-linux-gnu (4:14.1.0-2) ... Setting up g++-14 (14.2.0-1) ... Setting up gcc (4:14.1.0-2) ... Setting up g++ (4:14.1.0-2) ... update-alternatives: using /usr/bin/g++ to provide /usr/bin/c++ (c++) in auto mode Setting up build-essential (12.10) ... Processing triggers for libc-bin (2.39-6) ... $ test -z "${HOST_ARCH}" || eatmydata apt-get satisfy --no-install-recommends -y
Reading package lists... Building dependency tree... Reading state information... Some packages could not be installed. This may mean that you have requested an impossible situation or if you are using the unstable distribution that some required packages have not yet been created or been moved out of Incoming. The following information may help to resolve the situation: The following packages have unmet dependencies: libc6:arm64 : Depends: libgcc-s1:arm64 but it is not installable satisfy:command-line : Depends: libstdc++-dev:arm64 E: Unable to correct problems, you have held broken packages. Uploading artifacts for failed job 00:01 Uploading artifacts... salsa.env: found 1 matching artifact files and directories Uploading artifacts as "dotenv" to coordinator... 201 Created id=6088878 responseStatus=201 Created token=glcbt-64 Cleaning up project directory and file based variables 00:01 ERROR: Job failed: exit code 1

This suspect this issue is not specific to Salsa CI. This is a result of how the cross-builds in Debian work, and how in Debian unstable they are in constant flux of new libraries entering and getting rebuilt.

Just try again in a couple of days and it should work.

Overall status of cross-builds health in Debian can be monitored at https://crossqa.debian.net/ and for your package at https://crossqa.debian.net/src/sstp-client specifically.

This came up again on mailing list in https://lists.debian.org/debian-devel/2024/09/msg00358.html.

We should improve the error message in this job, and also the README to explain why the job is allowed to fail so people don't waste time investigating cross-build failures in vain.

Would it make more sense to disable cross-builds in the default template until this is fixed? Even in "warn only mode", it's still confusing.

Thanks!

There are at least 3 individual issue here:

• should the cross-builds job in the default template be disabled until the versions of linux and friends from experimental are uploaded to unstable? That's the issue from https://lists.debian.org/debian-devel/2024/09/msg00358.html
• what can be done about multi-arch:same version skews (that's what happened in the log of OP) -- the log does not indicate that it is the issue raised in https://lists.debian.org/debian-devel/2024/09/msg00358.html
• what can be done to improve the error message for this job so that people don't waste time investigating cross-build failures?

My suggestion is that until point 1 above is solved (currently experimental packages are uploaded to unstable), the default setting should be disabled, as in - the test currently doesn't give any useful info. Once that is done, it can be re-enabled, even if point 2 still happens - I don't know how common that is, but I assume it won't hit all packages, so at that point the test starts to be useful.

I'm not against it but note that bug #1065416 makes many, but not all packages FTCBFS. For example right now, according to dose3, 39% of all source packages in unstable have unsatisfiable cross build dependencies.

Please do consider disabling this test in the default config; I've been slowly turning on CI for my packages and most of them end up failing this step. Which means they all show "Warning" rather than ok for CI, and I have to explicitly check if it's just this test that's failing, or something else.

Flaky tests should either be fixed, or disabled.

Maybe salsa-ci can be a bit more clever with the defaults? There are many packages for which cross-building should work, like packages that just build some C/C++ library using cmake, meson or autotools or rust crates. There are also classes of packages where, even if cross-building is broken, it is usually relatively easy to fix it, like for Perl and Python packages. And then there is a class of packages where, right now, cross-building is known to not work, or where it's very hard to make it work like go, haskell, ocaml or R. Maybe salsa-ci can have some heuristic which for example detects the latter package category and, by default, doesn't enable the cross-build job?

I agree that having flaky tests is counterproductive, and thus filed !570 (merged). If you think we should merge it, please +1 or comment it.

We can then later re-enable it with additional "smartness" as @josch suggested, or enable enable it only manually for toolchain packages or something else. Anyway having this constantly fail for months and months is eroding "respect for pipeline greenness" and giving opportunities for more failures to creep in.

Also, MRs to improve the error message when cross-builds fail are welcome, and if anyone can come up with some nice automation to handle the toolchain skew issues when it undergoes transitions to avoid exploding the amount of CI noise for packages that can't do anything about the skew anyway.

mentioned in commit debian/mt-st@78473f64

mentioned in merge request !570 (merged)

mentioned in commit 5889ded0

closed with commit 59b0186a

mentioned in issue #407

The crossbuild job is disabled by default now. If you have more ideas on how to create a better test-crossbuild-arm64, please participate in brainstorming-planning in #407.