Start building Ubuntu images so one could use salsa-ci.yml with proper RELEASE on Ubuntu packaging branches?

Based on !457 (merged) seems Salsa-CI might want to recommend devs to reference the files in https://salsa.debian.org/salsa-ci-team/pipeline/-/tree/master/recipes which currently has one for Debian and one for Kali. If this is the structure, then supporting Ubuntu is just a matter of cloning the kali.yml to a ubuntu.yml and doing some small modifications + testing.

Related, I noticed that in https://salsa.debian.org/debian/devscripts/-/blob/main/.gitlab-ci.yml they use special image 'ubuntu-devel' for testing Ubuntu compatibility/QA.

In !400 (closed) there is an attempt to adapt Salsa-CI to run for a new distribution (Kali). Perhaps same approach could be used for Ubuntu support as well.

Just as an example: I tried to use RELEASE: noble on otto/mariadb-server@f89f65f3 to maintain the Ubuntu stable release packaging on branch ubunt/24.04-noble but it does not work (failing on base:noble not found).

As you mention, we have some kali images definition in https://salsa.debian.org/salsa-ci-team/pipeline/-/blob/master/.images-kali.yml, but they are actually built in their fork: https://gitlab.com/kalilinux/tools/kali-ci-pipeline/-/blob/master/.gitlab-ci.yml?ref_type=heads#L27

It is more a question of who is using and willing to maintain those images in the Salsa CI Container Registry.

In other words, it would be great to have a commitment, from the person that proposes an MR to include Ubuntu's (or any other vendors) images, to maintain them during their life cycle, and handle their removal when they are no longer supported.

And more thoughts:

I have had in my plate since some quite months the switch to sbuild. I have a couple of WIP MRs already, and this is in my background. I am removing the "Accepting MRs" tag, while that is not ready. In other words: please hold on before adding new images until the switch to sbuild is done.
I had considered this issue for testing how projects build on Ubuntu images. But what about autopkgtests and other tests? If you require/want autopkgtest, "you" would need to maintain the autopkgtest image and lxc.tars related to Ubuntu.

Thanks for your thoughts. I propose we continue to keep this issue open and see how many +1 in support we get, and how many maintainers state they share the need to package and test for Ubuntu, and how many DDs announce willingness to contribute.

If there is in say 6 months more than 10 people who state that they package for both Debian and Ubuntu and care about CI and quality testing, then we know it is worthwhile, otherwise we can close the issue.

added Accepting MRs label

removed Accepting MRs label

Greetings! I'm just one Ubuntu developer here, but this email was sent to the ubuntu-devel-discuss mailing list at Ubuntu. (I also use my Ubuntu email address, etc. in my uploads to Debian as a DM just for awareness). Additionally, my words are my own, NOT that of the Ubuntu project, so please read this as MY OPINION and not Ubuntu's opinion.

My only real concern here is, I really wonder here whether this actually is an Ubuntu issue but instead a Debian Salsa / Team issue here in Debian.

Ubuntu has no sway in how Debian IT does their things, nor what is enabled or disabled on the Salsa-CI side of things. Additionally, GitLab CI (the underlying software behind Salsa) does have the capacity to run Ubuntu images. It's up to the CI maintainers here, however, to enable Ubuntu as an option.

I'm not sure that there's anything that Ubuntu has to do to enable this, nor does this explicitly need Ubuntu's backing as it's then up to the Debian Salsa / IT team to handle support for Ubuntu images and definitions in Salsa-CI. It is my opinion that this requires wholly Debian team support, not Ubuntu's explicit support.

Direct ping of @otto who went to ubuntu-devel-discuss on this, so they see my opinion on the radar.

There is nothing in particular Ubuntu has to do, and nobody is claiming so.

Making this happen is mostly just a matter of how many people who upload to Ubuntu wants this.

..and reading your reply several times I don't actually see you mentioning what is your opinion. Thus checking would you have any use or need of this feature? Have you ever wished that your Ubuntu branches on Salsa hosted packaging source repositories would have a CI testing relevant pre-upload quality?

My workflows have no use of Salsa-CI for Ubuntu, because whenever I develop for Debian I also simultaneously as a force of habit test in Ubuntu as well if a package being revised is being targeted for existing Ubuntu releases under Ubuntu's SRU policy (which MOST do not). None of my code or packages I handle have any requirements to work on Ubuntu. If there's changes necessary to fix something for Ubuntu, it gets distropatched via Quilt patches downstream, or it's an Upstream Origin Problem that also affects Debian.

My opinion is "I have no need for this extra level of tooling in Debian, and it's up to Debian CI team whether they want to support this as an option." Basically, in all the things I touch and watch, if it works in Debian it works in Ubuntu (because mostly server or CLI packages and utilities).

For reference, the Ubuntu SRU documentation about SRU testing requirements: https://canonical-sru-docs.readthedocs-hosted.com/en/latest/reference/requirements/ (drafted largely by @rbasak). It highlights that builds must work and autopkgtest must pass. This is rather obvious, but the takeaway is that it does not require for the package to have testing to ensure that it is Lintian clean, have no BLHC issues or builds in a reproducible way etc.

mentioned in merge request mariadb-team/mariadb-server!82 (merged)

Only 4 people so far, seems the overlap of Debian and Ubuntu maintainers who also use Salsa-CI isn't that big..

mentioned in merge request mariadb-team/mariadb-server!83 (merged)

This is interesting when we merge the apt salsa-ci stuff because lots of apt release branches are building for Ubuntu releases.

mentioned in merge request apt-team/apt!348

This has now 6 . We could do a time-boxed test of providing Ubuntu images for Salsa-CI for 3 months and see how it goes and how much of maintenance burden it creates.

IMHO, the maintenance "burden" would appear when a new release is published or another reaches end-of-support. So I don't think a three months time boxed test would be enough.

Questions:

What versions would you be planing to support, LTS-only, or any release?
Would you also create autopkgtest images?

mentioned in merge request !518 (merged)

Hi, I am working on this issue and created a draft MR for it.

It currently builds base, lintian, blhc and generic_tests for all Ubuntu current supported releases and only for amd64.

I intend to later add armhl, arm64, and i386. It seems from Ubuntu Autopkgtest service statistics that ppc64el and s390x should also be supported, but I expect that we don't have runners for them yet for Salsa CI.

I am also thinking about how should I deal with autopkgtest job. My current thoughts are that it should mimic Ubuntu Autopkgtest service.

I intend to name the job that mimics Ubuntu Autopkgtest service ubuntu-autopkgtest to distinguish it from the original one that mimics debci.

I also need to test the draft by using it in a package.

Any suggestions or thoughts are welcomed.

Hi @otto!

Regarding autopkgtest job, Do you think that I should try to mimic Ubuntu Autopkgtest's service? or just mimic debci on an Ubuntu image?

To my knowledge those two are simply different user interfaces to the same underlying system. The autopkgtest runs whatever test code is in the package in debian/tests directory. I am not aware of anything special in this regard and I assume that for the Salsa CI pipeline all you need to do is just run it on the Ubuntu release that matches the debian/changelog marked target. So for example when running Salsa CI in mariadb-team/mariadb-server!82 (merged) it should have ran autopkgtest on in Ubuntu 24.04 "Noble" images.

Just passing by, and to complete this: yes, autopkgtest.ubuntu.com is "just" a wrapper on autopkgtest, mostly managing the queue and scheduling the jobs. The infrastructure has however a few quirks, and one of them is that is has a proxy, making some tests to struggle (look at rust-reqwest for an example). There is also the big_packages mechanism, for when a test gets killed by running out of memory, or disk space, and this makes the infra to provide a bigger VM only for the packages/arch listed in there. Lastly, all architectures are running on real VMs, except for armhf tests that are running in LXD containers on a bunch of static arm64 hosts.

I'm not sure whether or not you want to mimic those specific things (I'd vote for no, because it adds complexity, and not really worth it), but just know that they can lead to a different test result from Debian's infra when uploading the package in Ubuntu and running the tests in autopkgtest.u.c.

@hyask, @otto: thanks for your feedback, I will take it into consideration.

You asked about i386, so just reiterating what I said on a call: The i386 architecture on Ubuntu is only "partial", so not all packages are expected to work. For the sake of Salsa CI for Ubuntu, it is better for us to keep a minimal scope and avoid things that are unlikely to fully work, thus you can skip the i386 architecture.

mentioned in commit ahmedsiam/salsa-pipeline@47b8116d

mentioned in commit ahmedsiam/salsa-pipeline@37846000

Hi! I am involved in two packages that would greatly benefit from this:

src:autopkgtest. We do already run some tests on Ubuntu, see https://salsa.debian.org/ci-team/autopkgtest/-/blob/3813168463b38edd3d067e3b469b6782ebf762e5/.gitlab-ci.yml#L40, but this is by far not as nice and complete as having the full salsa pipeline Ubuntu-aware. For autopkgtest in Ubuntu we (Ubuntu autopkgtest people) already work on Salsa to leverage CI, but we have to drop the Salsa pipeline, see for example this sad commit: ubuntu-ci-team/autopkgtest@ece8d3fe.
src:isc-kea. Packaging work is mainly driven by Ubuntu people at the moment, see https://salsa.debian.org/debian/isc-kea/-/commits/debian/unstable/?ref_type=HEADS. We actively try to keep isc-kea in Ubuntu a sync, and do all the work in Debian when possible. However, we don't really have CI for Ubuntu, which is a pity.

I could easily bring more examples, and I think support for Ubuntu would make life of many developers easier, so big +1 from me on this proposal.

One related thing I would like to have is: the possibility to run both the Debian and Ubuntu tests on some branch, even if the d/changelog mentions something Debian-specific (unstable/experimental/UNRELEASED). This would require adding a fallback to $(ubuntu-distro-info --latest) in that case. And would also require somehow namespacing jobs (e.g. the blhc job on Ubuntu becomes ubuntu/blhc or something like that).

I am not sure of how difficult this would be, but if we want to cover the case where a Debian packaging branch also targets Ubuntu (as in the case of autopkgtest, isc-kea), we need something like that I believe.

I had a better look at the pipeline definitions, and I looks like this is not an easy thing to implement.

Hi @paride!

I didn't notice this comment and the one before it for unknown reason until now.

Regarding the ability to run both the Debian and Ubuntu tests in the same pipeline, this can be done by creating two yaml pipeline files, one for Ubuntu and one for Debian and trigger them by the main pipeline.

Here is an example: https://salsa.debian.org/ahmedsiam/hit-here/-/pipelines/715952

Thanks for the pointer @ahmedsiam, I'll definitely try that.

This would require adding a fallback to $(ubuntu-distro-info --latest) in that case.

I guess from this line that you also want the pipeline to automatically test for the devel version unless otherwise specified.

This will be the case if UNRELEASED is specified in the changelog. (I didn't implement that yet)

I am not sure what should be the behavior if a debian release is specified in the changelog and the RELEASE variable isn't specified, Maybe it should fail or also default to the Ubuntu devel release.

My initial thought is that it should fail. For example in mariadb-team/galera-4!22, if I have a Ubuntu pipeline chosen and the debian/changelog entry is about unstable, but an old version (that synced to went from Debian to Ubuntu 10 months ago) and if I had forgotten to define RELEASE: noble, it should fail.

If it silently builds for latest unstable I feel it would mislead me to fix newer things that don't actually apply for Ubuntu Noble.

mentioned in commit ahmedsiam/salsa-pipeline@33e2f57b

mentioned in commit ahmedsiam/salsa-pipeline@1f323a11

mentioned in commit ahmedsiam/salsa-pipeline@ce638430

mentioned in commit ahmedsiam/salsa-pipeline@da3d3b45

mentioned in commit ahmedsiam/salsa-pipeline@911cd2d6

mentioned in commit ahmedsiam/salsa-pipeline@0cd29f5e

mentioned in merge request ahmedsiam/salsa-pipeline!3 (merged)

mentioned in commit ahmedsiam/salsa-pipeline@8c49b30b

mentioned in commit ahmedsiam/salsa-pipeline@b3c0b79e

Hi!

The Ubuntu pipeline port I am working on now works well for noble amd64.

If you want to use it, include ubuntu recipe from my master repo and assign noble for RELEASE variable in salsa-ci.yml file like this:

---
include:
  - https://salsa.debian.org/ahmedsiam/salsa-pipeline/-/raw/master/recipes/ubuntu.yml
variables:
  RELEASE: noble

Let me know if you had any errors or have suggestions.

The pipeline doesn't work well for focal and jammy yet.

Hi @ahmedsiam, I tested your branch here: https://salsa.debian.org/ubuntu-ci-team/autopkgtest/-/pipelines/715317. Worked fine apart the lintian job, which failed because of bad-distribution-in-changes-file. This is because d/changelog in that branch targets oracular (current devel release), and lintian in Noble doesn't know about it.

I on my hand tested this at mariadb-team/galera-4!22 and seems to work well!

Hi @paride, Thanks for testing it.

I will add support for Oracular. I didn't do that before because I thought there are no container images for it, but this wasn't true.

I guess it should work fine like Noble.

mentioned in commit ubuntu-ci-team/autopkgtest@2f26e7b9

mentioned in commit ahmedsiam/salsa-pipeline@747915ae

Should I add support for Ubuntu backports releases?

@otto: Do you have any thoughts in this regard?

mentioned in commit ahmedsiam/salsa-pipeline@866c0846

I guess I'm going to be the asshole after all ...

But how about waiting with such things until there is (significant) overcapacity on Salsa (CI runners)?
Or is Ubuntu going to provide the runners so that the pipeline runs for Debian don't get any slower then they already do?

We are doing this work for ourselves. Discussing whether some entity representing Ubuntu will provide computing capacity out of scope of the development work in this CI pipeline issue.

The question of pipeline runs being slow is however an important topic. I just think we should discuss it in a separate Salsa CI issue about potential optimizations of running the pipeline faster, and in a salsa.debian.org instance issue on if or how Salsa admins want to receive computing capacity donations, and if there even is a dashboard/stats page of what the runner count and utilization currently is.

And as long as the ubuntu-related build jobs are not enabled by default, the impact on CPU consumption caused by !518 (merged) should be minimal.

The question of pipeline runs being slow is however an important topic.
potential optimizations of running the pipeline faster

The problem is about the current (IMO very) limited capacity of Salsa (CI runners), which isn't fixable by optimizations.
Yesterday I started a pipeline manually as after 5 minutes after my (git) push, the pipeline still wasn't created, so I assumed it wouldn't get created at all. Such things don't happen all the time, but it's not exactly rare either.

I don't know what the current state/plans are, but I do know there have been talks about increasing capacity. Until that is realized, I think that the limited capacity should be reserved for ('pure') Debian work.

Ref: https://lists.debian.org/debian-kernel/2024/05/msg00205.html

And as long as the ubuntu-related build jobs are not enabled by default

That would help.

I think we should discuss it in a separate Salsa CI issue about potential optimizations of running the pipeline faster, and in a salsa.debian.org instance issue on if or how Salsa admins want to receive computing capacity donations, and if there even is a dashboard/stats page of what the runner count and utilization currently is.

Anecdotal stories of what happened on a git push one day does not help much. Having actual hard facts and documentation about current runner fleet, a utilization dashboard, and documentation on how to donate more capacity to expand it is needed first.

Anecdotal stories of what happened on a git push one day does not help much.

one day? Really? sigh
o/

Otto Kekäläinen commented on a discussion: #327 (comment 515709)

I think we should discuss it in a separate Salsa CI issue about potential optimizations of running the pipeline faster, and in a salsa.debian.org instance issue on if or how Salsa admins want to receive computing capacity donations, and if there even is a dashboard/stats page of what the runner count and utilization currently is.

FTR, a discussion started on DC24 about the resources available for the shared runner. I would like to propose to the salsa admins to upgrade the machine used to run the shared runner, but we need to figure out first what are the available sponsored resources. As soon as I get some news, I will keep the interested people and teams informed.

...

Em 14 de agosto de 2024 21:11:31 GMT-03:00, "Otto Kekäläinen (@otto)" gitlab@salsa.debian.org escreveu:

There are plenty of hardware sponsors around. The challenge is human time. For example Salsa admins have yet to have time to document what is the process to donate/join the runner pool for those who want to offer hardware.

I'm withdrawing my opposition.

@diederik, Can I know why did you withdraw your opposition?

It was a tongue-in-cheek remark ;-) (seeing as Salsa has been unusable for days now)

...

On Wed Sep 4, 2024 at 1:06 PM CEST, Ahmed Siam (@ahmedsiam) wrote:

@diederik, Can I know why did you withdraw your opposition?

mentioned in commit ahmedsiam/salsa-pipeline@c914965c

mentioned in commit ahmedsiam/salsa-pipeline@3370c158

mentioned in commit ahmedsiam/salsa-pipeline@0b08ce73

mentioned in commit ahmedsiam/salsa-pipeline@f5ab1d77

mentioned in commit ahmedsiam/salsa-pipeline@df8334b1

mentioned in commit ahmedsiam/salsa-pipeline@1f1f7983

Now the pipeline works well for Oracular, Noble, Jammy and Focal. (amd64 only but arm builds are expected to work fine once the pipeline merged to salsa-ci)

You can use it through including ubuntu recipe from my master repo like this:

---
include:
  - https://salsa.debian.org/ahmedsiam/salsa-pipeline/-/raw/master/recipes/ubuntu.yml

It will detect the release through the changelog. If UNRELEASED is specified, the RELEASE will be the ubuntu developement release (oracular). If a debian release is specified, the pipeline will fail and ask you to specify a supported Ubuntu release through the pipeline's yaml file or the changelog.

If you tried the pipeline, it will be nice to share your pipeline run.

Hmm, maybe I should add the fact that it will use the ubuntu development release if UNRELEASED is specified in the error message.

The pipeline should use whatever is defined in the RELEASE variable. In other words, a RELEASE variable, if present, should override the debian/changelog entry info. If no RELEASE is defined, it should read the debian/changelog file and use the latest entry. If the latest entry is UNRELEASED, it should read the second latest entry. Does this make sense?

The confusing part is probably that the automatic reading of debian/changelog is broken and/or having RELEASE variable honored is broken. At least all the backport builds are currently failing because the RELEASE variable contents is not honored in those builds (see #348 (closed) and #349 (closed)). This needs to be fixed independently of the Ubuntu support work.

The Ubuntu images should not change this logic. The only new thing here is that if an Ubuntu release name is encountered, the pipeline can actually run and not fail on all container images missing.

The logic Otto suggests makes sense, however I think it would be useful to have a special RELEASE value that causes the Ubuntu devel release to be used (better: the latest release to be used, as in ubuntu-distro-info --latest). I'm proposing a special RELEASE value, but any other mechanism to achieve the same would do.

@otto do you have suggestions on this?

Otto wrote:

If the latest entry is UNRELEASED, it should read the second latest entry. Does this make sense?

This makes sense.

The confusing part is probably that the automatic reading of debian/changelog is broken and/or having RELEASE variable honored is broken. At least all the backport builds are currently failing because the RELEASE variable contents is not honored in those builds (see !348 (merged) and !349 (merged)). This needs to be fixed independently of the Ubuntu support work.

MRs are mentioned while the intention is to mention issues #348 (closed) and #349 (closed). Solving this issue is in my todo list. I will work on solving it in the next week.

Paride wrote:

The logic Otto suggests makes sense, however I think it would be useful to have a special RELEASE value that causes the Ubuntu devel release to be used (better: the latest release to be used, as in ubuntu-distro-info --latest). I'm proposing a special RELEASE value, but any other mechanism to achieve the same would do.

The solution in my mind is to add a variable called UBUNTU_DEV_RELEASE that contains the current development release so if pipeline users wanted to achieve what is mentioned, they will assign UBUNTU_DEV_RELEASE to RELEASE.

The variable's value will be hardcoded and changed every time a new development release get created.