Draft: renable support for crossbuilding

The "Check_crossarchitectures" exits the program if it is not building on the same architecture which made it impossible to build a arm64 build on a amd64 host. This removes the check from:

• bootstrap_debootstrap
• binary_grub-efi
• binary_grub_cfg

Also a check if the qemu static binary exists is added.

added 1 commit

• 051dfcd3 - renable support for crossbuilding

Compare with previous version

It seems that we only partial crossbilding between architectures, e.g. amd64 on i386. I have successfully build a basic arm64 image on a amd64 host and it it seems that nothing is broken. Briefly looking at the code I don't see why we cannot relax that requirement and remove the checks completely.

Could you try to do a reproducible build: one on the amd64 host and one on a arm64 host?

Use test/rebuild.sh with a command line option similar to: rebuild.sh standard sid 20221211T215942Z. On the amd64-host you'll need to add the options to do cross-building around line 280

After that, run diffoscope on both outputs to verify that crossbuilding results in an identical image compared to native building. That will be the proof that everything is working fine.

Could you try to do a reproducible build: one on the amd64 host and one on a arm64 host?

Good idea! Once I've setup a build environment on the arm64 device I'll try that.

You could use the live-image that you generated on amd64 to boot on the arm64 device (and thus proving that it runs). Then you'll only need to install live-build and fetch the rebuild.sh script. Then you should be able generate an image on the arm64 device.

You might want to set up a proxy to reduce the network traffic, see https://wiki.debian.org/ReproducibleInstalls/LiveImages

The rebuild.sh script needs a small modification to work. Because the image has no -amd64 suffix of course when it is build for arm.

See: https://salsa.debian.org/live-team/live-build/-/blob/master/test/rebuild.sh#L322

So far, I've focussed only on the amd64 builds. Feel free to add $ARCHITECTURE or similar to the rebuild.sh script

I'll do that once I got the reproducible build setup working.

I hope that I can either this or next week do some more intensive testing again. Maybe for now adding arm64 to the list of architectures that are allowed to crossbuild from amd64 is a simpler way: https://salsa.debian.org/live-team/live-build/-/blob/master/functions/architectures.sh#L51

Have you been able to start your cross-built image on the arm64 device? And can you from there (even though it might be horribly slow) build a native arm64 image?

Yes, I've build native arm64 images on that device and also booted some cross-build images. Fortunately it is a M1 device, so building is quite fast. I'll try to build an exact equal image on both architectures tomorrow.

To get the reproducible build script to work with snapshots I needed to change the mirror to https://snapshot.debian.org because the other one does not include arm64.

I still currently have an issue with differing timestamps of files, but I need to investigate why that is happening.

Small update regarding building arm64 from a amd64 host: we currently run a live-build Version where we just patched it to allow arm64 in https://salsa.debian.org/live-team/live-build/-/blob/master/functions/architectures.sh#L51 and set --bootstrap-qemu-arch arm64 --bootstrap-qemu-static /usr/bin/qemu-aarch64-static. This works fine (with an around 2x slowdown in build time).

@TS Since the snapshot server is down, you can try to do a second build within the same DAK run (every 6 hours) to test for reproducibility, or (if the packages for arm64 are suitable) you can try to use Bookworm, which will only change on the next point release.

test/rebuild.sh --configuration standard --debian-version bookworm --debian-version-number 12.1.0 --timestamp archive --installer-origin archive

I ran it and the ISOs are unfortunately not the same. There are some differences:

• the rebuild script hashes are different because of the config changes required for crossbuilding
• /usr/bin/qemu-aarch64-static is part of the crossbuild ISO (this looks like a bug)
• initrds are different (not sure why yet)
• md5sum.txt and sha256sum.txt are sorted differently
• some directories are marked as different by diffoscope (to see the actual difference I probably need to let it run with a greater depth)

The /etc/mdadm/mdadm.conf has an additional comment in the non crossbuild, causing the difference in the initrds.

This PR is not forgotten, but currently don't have the time to check the root causes of those mismatches. We build ISOs with this change now for a couple of months and it works fine.

Are reproducibility issues considered a blocker for merge?

Depends I would say. If you are building the same live build config and it was reproducible before, it still should be reproducible after the change.

For this MR specifically: while it works, there are some issues found by rebuilding using qemu and native. I'm fine with the images not being 100% identical, but we should at least fully know why they are not.

Now that the MR is open for quite some time (over a year), I'd propose to split the work into several parts:

1. Enable cross-building properly (which will effectively be this MR) (not yet tested from my side)
2. Raise the bar for the quality and make the build reproducible in steps:
   1. Reproducible when cross-building on a x86-64 host
   2. Reproducible when natively building on an arm64 host
   3. Reproducible between both versions

Sounds like a plan I'll update this MR around FOSDEM and then lets merge it.

With the following invocation, I did not need to patch any file in live-build, in order to build an arm64 image on an amd64 host (although it is rather slow):

lb config --bootstrap-qemu-arch arm64 --bootstrap-qemu-static /usr/bin/qemu-aarch64-static --architecture=arm64 --distribution=sid

Can you verify whether --architecture=arm64 works for you as well, and that identical/similar images are built?

@rclobus-guest I switched our CI and it seems to work fine. I'll test a bit more next week and also compare it against a native build again with diffoscope.

So likely this MR is not required, instead we should just document somewhere how to cross-build.

I'm preparing updates to live-manual.

For some reason it now fails with one of our included third party repositories. Currently investigating is that is actually related to this change or just a timing issue.

Seems to be unrelated.

mentioned in merge request !346 (merged)