Skip to content
Snippets Groups Projects

Feature request: please support building on ARM

    • Closed Issue created by Federico Ceratto @federico

    ARM is becoming more relevant due to SBCs, servers and PinePhone/Book/Tab. Could you consider supporting armhf and arm64 in CI either through emulation or native buildbots? Thanks!

    To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information

    Child items ...

    • Activity

    • Otto Kekäläinen
      Otto Kekäläinen @otto ·
      Owner

      Here is a post on using Qemu-binfmt for ARM builds specifically on Gitlab-CI: https://community.arm.com/developer/research/b/articles/posts/continuous-cross-architecture-integration-with-gitlab

      Older Raspberry Pi use armhf and newer Raspberry Pi 3 and 4 use arm64 (also called AArch64). Arm64 is probably would make most sense in a effort/benefit point of view.

    • Federico Ceratto
      Federico Ceratto @federico ·
      Author

      FWIW I can provide a native ARM VM.

    • David Heidelberg
      David Heidelberg @dh ·

      @federico do you have yml which provides or extends salsa-ci.yml with pipeline "Build ARM64" / .build-package-arm64?

      Otherwise I'm thinking about trying to do it with Qemu or something as Otto suggested.

      EDIT: Also very relevant due to Librem 5, OnePlus 6/6T and Pocophone F1 phones. Not mentioning fact that ARM64 architecture seems to be lighter and better designed than older amd64.

      Edited by David Heidelberg
    • Otto Kekäläinen
      Otto Kekäläinen @otto ·
      Owner

      One option would be to add some ARM machines as shared Gitlab-CI runners. Do you @federico want to offer your machine? Contact the Salsa admins.

      If those shared runners are tagged say 'arm64' then all jobs that are tagged the same would only run on those builders.

      Another alternative is use emulation, but that will be much slower. A good post about the new Docker buildx tool that creates multi-platform Docker images can be found at https://www.auroria.io/docker-multi-architecture-builds-with-gitlab-runner/

      Seems native Docker buildx support was merged into Gitlab 11 months ago in https://gitlab.com/gitlab-org/gitlab-runner/-/merge_requests/2206

      I can locally also build emulated Docker container with these steps:

      sudo apt install qemu-user-static qemu-system-aarch64
cp /usr/bin/qemu-aarch64-static .
echo '{"experimental": true}' | sudo tee /etc/docker/daemon.json
sudo systemctl restart docker
docker pull --platform aarch64 debian:bullseye
echo '
FROM --platform=linux/aarch64 debian:bullseye
COPY ./qemu-aarch64-static /usr/bin/qemu-aarch64-static
RUN uname -a
' | tee Dockerfile

docker build .
Sending build context to Docker daemon  19.74MB
Step 1/3 : FROM --platform=linux/aarch64 debian:bullseye
 ---> 454fddc9ff45
Step 2/3 : COPY ./qemu-aarch64-static /usr/bin/qemu-aarch64-static
 ---> 2c1f66d151e7
Step 3/3 : RUN uname -a
 ---> [Warning] The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
 ---> Running in aa7837ec1ee5
Linux aa7837ec1ee5 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 aarch64 GNU/Linux
Removing intermediate container aa7837ec1ee5
 ---> e003c4b59006
Successfully built e003c4b59006

docker run -it e003c4b59006 bash
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
root@8c52794d8852:/# uname -a
Linux 8c52794d8852 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 aarch64 GNU/Linux
      • Otto Kekäläinen
        Otto Kekäläinen @otto ·
        Owner

        I am however not able to run a aarch64 directly like this:

        docker run -it aarch64/debian bash
Unable to find image 'aarch64/debian:latest' locally
latest: Pulling from aarch64/debian
7189468c6fc4: Pull complete 
Digest: sha256:903b5655da8689b6cdc7465ea118f674d4f1820a52109aeb2623b1c8e6225420
Status: Downloaded newer image for aarch64/debian:latest
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
standard_init_linux.go:219: exec user process caused: no such file or directory

        I must prepare the emulated container locally myself.

      • David Heidelberg
        David Heidelberg @dh ·

        docker run -it aarch64/debian bash missing :bullsyeye? (just guessing, since you used it previously and in this output is :latest)

        Edited by David Heidelberg
      • Otto Kekäläinen
        Otto Kekäläinen @otto ·
        Owner

        No, it does not matter. As said, I am unable to run a aarch64 image directly. Docker does not run any emulation (not at least the version I have). I have to make a image myself that is amd64 but runs aarch64 on the inside.

        The commands are there, you can play around with them yourself and repeat the same steps with any variation you want. Good way to learn how they work.

      • Please register or sign in to reply
    • David Heidelberg
      David Heidelberg @dh ·

      @otto @federico do you hang on some Matrix/IRC channels? I'm on #Debian:matrix.org and #Mobian:matrix.org

      Edited by David Heidelberg
    • Iñaki Malerba
      Iñaki Malerba @ina ·
      Owner

      Hello!

      I'm sorry I havent had the time to participate in this discussion, thanks for all the input.

      We've had offers for arm hardware several times, but for us to start providing arm support we need these runners to be configured as shared so everyone can access it (as users you can only set them up for the projects you have access to), which needs to be done by salsa admins. This means they need to maintain them, and it's understandable they have some requirements for that.

      @santiago asked them again about this a few days ago:

      <santiago> waldi, formorer, Ganneff, dear salsa admins: what would be the requirements for new shared runners (of archs other thant x86)?
<santiago> it would be great to have pipeline jobs that could run on other archs, and before looking for any, it would be great to have your input about the requirements/what would be needed.
[..]
<waldi> arm can be handled via aws after some internal amazon issues are resolved
[..]
<santiago> it would be great to have pipeline jobs that could run on other archs, and before looking for any, it would be great to have your input about the requirements/what would be needed.
<waldi> SPI maintaining the contract with the vendor. some sponsorship to allow usage. supported by docker machine to provide ephemeral vm. currently google and aws fulfill that
    • Sebastian Spaeth
      Sebastian Spaeth @spaetz ·

      while shared arm runners would be terrific, just providing arm64 CI images would be terrific. We can use a project qemu-user runner but the stock debian arm images are not suitable for the salsa CI jobs.

    • Otto Kekäläinen
      Otto Kekäläinen @otto ·
      Owner

      I found another trick from https://medium.com/codex/building-arm-docker-images-with-gitlab-runner-6195e8326094 and now I can run native arm images directly without having to build any locally first:

      $ docker run --rm --privileged hypriot/qemu-register
Unable to find image 'hypriot/qemu-register:latest' locally
latest: Pulling from hypriot/qemu-register
bdbbaa22dec6: Pull complete 
fdf885a1ac26: Pull complete 
a3e80b5d80b4: Pull complete 
e459efb85f24: Pull complete 
7cdc40468629: Pull complete 
84a0c12617bc: Pull complete 
Digest: sha256:4fea65076a099128763969a83f0b629b172002f6fb95245c0e272471e819f987
Status: Downloaded newer image for hypriot/qemu-register:latest
---
Installed interpreter binaries:
-rwxr-xr-x    1 root     root       6292296 Feb  6  2020 /qemu-aarch64
-rwxr-xr-x    1 root     root       5702696 Feb  6  2020 /qemu-arm
-rwxr-xr-x    1 root     root       5984808 Feb  6  2020 /qemu-ppc64le
-rwxr-xr-x    1 root     root       5115848 Feb  6  2020 /qemu-riscv64
---
Registered interpreter=qemu-aarch64
enabled
interpreter /qemu-aarch64
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200b700
mask ffffffffffffff00fffffffffffffffffeffffff
---
Registered interpreter=qemu-arm
enabled
interpreter /qemu-arm
flags: OCF
offset 0
magic 7f454c4601010100000000000000000002002800
mask ffffffffffffff00fffffffffffffffffeffffff
---
Registered interpreter=qemu-ppc64le
enabled
interpreter /qemu-ppc64le
flags: OCF
offset 0
magic 7f454c4602010100000000000000000002001500
mask ffffffffffffff00fffffffffffffffffeffff00
---
Registered interpreter=qemu-riscv64
enabled
interpreter /qemu-riscv64
flags: OCF
offset 0
magic 7f454c460201010000000000000000000200f300
mask ffffffffffffff00fffffffffffffffffeffffff
---


$ docker run -it aarch64/debian bash
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
root@465e567b2f07:/# uname -a
Linux 465e567b2f07 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 aarch64 GNU/Linux
root@465e567b2f07:/# exit


$ docker run -it armhf/debian bash
Unable to find image 'armhf/debian:latest' locally
latest: Pulling from armhf/debian
c8e66f55425f: Pull complete 
Digest: sha256:ae705bf6614c467bad86891b8e441c9e0c987389b6e4d34c7b1371c0f841249f
Status: Downloaded newer image for armhf/debian:latest
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64) and no specific platform was requested
root@6d8aad63d732:/# uname -a
Linux 6d8aad63d732 4.15.0-143-generic #147-Ubuntu SMP Wed Apr 14 16:10:11 UTC 2021 armv7l GNU/Linux
    • IOhannes zmölnig
      IOhannes zmölnig @umlaeute ·
      Contributor

      i created a related MR that sets up a cross-building job (!293 (merged)).

      the default job does not create any artifacts (but that could of course be added easily, either in the central pipeline, or by enhancing your project's pipeline).

      currently the crossbuild test is setup to build for arm64, but that could be changed to whatever architecture you want using a single variable.

    • Santiago R.R. mentioned in merge request !344 (merged)

      mentioned in merge request !344 (merged)

    • Santiago R.R. closed with merge request !344 (merged)

      closed with merge request !344 (merged)

    • Santiago R.R. closed with commit 793977ec

      closed with commit 793977ec

    • Santiago R.R. mentioned in commit 2875c272

      mentioned in commit 2875c272

    Please register or sign in to reply