Salsa Continuous Integration (CI) – Quality Assurance for Debian packaging

TL;DR

This Salsa Continuous Integration (CI) pipeline increases the quality of Debian packages by automatically testing every commit on a Debian package.

To activate Salsa CI, open settings at https://salsa.debian.org/%{project_path}/-/settings/ci_cd, and in General pipelines set the project CI/CD configuration file to debian/salsa-ci.yml, and on the Debian packaging git branch create the file containing:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

Table of contents

• Benefits – Why use Salsa CI?
• Activate Salsa CI
  • Enable feature 'CI/CD' in Settings
  • Include Salsa CI in an existing GitLab CI pipeline
• Customize Salsa CI
  • Select which jobs run in the CI pipeline
  • Allow a job to fail
  • Disable the pipeline on certain branches
  • Allow pipeline to run when git tags are pushed
  • Setting variables on pipeline creation
  • Run only selected jobs
• Customize builds and build dependencies
  • Extend the job timeout
  • Decrease build timeout to leave margin for cache upload
  • Disabling building on i386
  • Enable building on ARM and RISC-V
  • Add more architectures or CI runners
  • Testing build of arch=any and arch=all packages
  • Testing build profiles
  • Enable cross-builds
  • Enable building packages twice in a row
  • Enable generation of dbgsym packages
  • Build with non-free dependencies
  • Add private repositories to the builds
  • Add extra arguments to dpkg-buildpackage
  • Adding extra arguments to gbp-buildpackage
  • Git attributes
  • Customize reprotest
• Lintian, Autopkgtests, Piuparts and other quality assurance CI jobs
  • Customize Lintian
  • Make autopkgtest more strict
  • Add extra arguments to autopkgtest
  • Avoid autopkgtest failures on systemd masked tmp
  • Run a pre-install / post-install script in piuparts
  • Using automatically built apt repository
  • Enable wrap-and-sort job
  • Debian release bump
• Distribution and release selection
  • Customise what Debian release to use
  • Experimental: Ubuntu support
• Testing Salsa CI (and GitLab CI in general) pipeline definitions locally
  • Testing definitiion file for correctness
  • Running the pipeline locally
• General Debian packaging support and resources
• General Salsa information
• Support for Salsa CI use
• Contributing

Benefits – Why use Salsa CI?

Salsa CI provides anyone working on Debian packaging with instant feedback about the package quality. The Salsa CI pipeline mimics various Debian quality assurance tools that run for every new Debian package revision uploaded to the Debian archive, but instead of requiring a new upload, Salsa CI can run on every commit pushed to Salsa, making the feedback cycle significantly faster.

While the official Debian QA tools will flag issues with a package, any package already uploaded to the Debian archives will continue to exist in a broken state, potentially causing havoc, until a fixed version is uploaded. Ensuring every package fully passed Salsa CI before upload to Debian archives will help prevent archive-wide effects of broken packages.

In addition to catching machine detectable regressions, Salsa CI can also be used to validate that new fixes are correct and that previously detected issues are no longer present.

Salsa CI currently offers:

• Building the package using git-buildpackage with multiple build profiles and architectures
• Static analysis for Debian packages using Lintian
• Functional testing using Autopkgtest
• Package installation and removal testing using Piuparts
• Conflicting file path check against packages that are not declared with Breaks and Replaces
• Checking proper use of secure build flags etc using Buildd Log Scanner
• Build reproducibility testing using Reprotest

Activate Salsa CI

Enable feature 'CI/CD' in Settings

To use the Salsa Pipeline, the first thing to do is to enable the project's Pipeline. Go to Settings (General), expand Visibility, project features, permissions, and in Repository, enable CI/CD. This makes the CI/CD settings and menu available. Then, change the project's setting to make it point to the pipeline's config file. This can be done on Settings ⇾ CI/CD, expand General Pipelines and update the field CI/CD configuration file.

It is recommended to set this value to debian/salsa-ci.yml and create the file containing at least the following lines:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

⚠️ Note: The pipeline is not run automatically after configuring it. You can either trigger it by running the pipeline manually or pushing something.

Activating Salsa CI without using the salsa-ci.yml file not recommended

While technically possible, it is not recommended to activate Salsa CI by simply setting recipes/debian.yml@salsa-ci-team/pipeline as the CI config path. This refers directly to a file kept in the salsa-ci-team/pipeline repository without creating a debian/salsa-ci.yml. This may initially save some effort, but in the long run it ends up causing more work as the CI will run indiscriminately on all branches no matter if they are intended for Debian packaging or not, and collaborators will not be able to see why and how the CI is triggered.

Having an explicit debian/salsa-ci.yml on each branch, potentially augmented with branch-specific additions, is easier to reason about and causes less maintenance work in the long-term.

Alternatively use command-line tools salsa or glab

If you wish to activate Salsa CI directly from the command-line, set up the salsa tool and run this for all your projects:

salsa update_projects $NAMESPACE/$PROJECT --jobs yes --ci-config-path debian/salsa-ci.yml
cd $PROJECT/debian
curl -LO https://salsa.debian.org/salsa-ci-team/pipeline/-/raw/master/recipes/salsa-ci.yml
git commit -m "Enable Salsa CI using default template" salsa-ci.yml
git push

Alternatively, use the more generic command-line tool glab which works for any GitLab instance.

Include Salsa CI in an existing GitLab CI pipeline

If your project already has a GitLab pipeline, you can add Salsa CI in the .gitlab-ci.yml file as an additional child pipeline with:

stage: package
trigger:
  include: debian/salsa-ci.yml
  strategy: depend
rules:
  - if: $CI_COMMIT_TAG != null
    when: never

Don't run Salsa CI on every commit

If you don't want to run Salsa CI on every commit, you can add custom rules to trigger the Salsa CI pipeline only manually or for example when a scheduled pipeline run takes place. The typical use case is when maintaining native Debian packages where the program code is tested on every git commit, and the commits are frequent, while the packaging needs to be tested only infrequently.

stage: package
trigger:
  include: debian/salsa-ci.yml
  strategy: depend
    # run if user manually presses the "play" icon on pipeline
    - if: $CI_PIPELINE_SOURCE == "web"
      when: manual
    # run during for scheduled pipeline runs
    - if: $CI_PIPELINE_SOURCE == "schedule"
      when: always
    # never allow other pipeline sources to trigger running this job
    - when: never

Customize Salsa CI

Salsa CI is designed to run as-is for the vast majority of Debian packages. In most cases Salsa CI should be used without any customizations. However, for packages that need it, Salsa CI offers many ways to customize how it runs and what it runs.

The easiest way to grasp the possibilities is to browse various examples of Salsa CI being used on prominent Debian packages, from simple to complex:

• https://salsa.debian.org/sudo-team/sudo/-/blob/master/debian/salsa-ci.yml
• https://salsa.debian.org/mariadb-team/mariadb-server/-/blob/debian/latest/debian/salsa-ci.yml
• https://salsa.debian.org/kernel-team/linux/-/blob/master/debian/salsa-ci.yml

Salsa CI offers a wide range of variables that can be used to easily turn on/off certain features or customize how they behave.

It is even possible to only include the salsa-ci.yml template for jobs definitions without adding any jobs to the pipeline from pipeline-jobs.yml.

For an example of a very customized CI pipeline, one can have a look at the Debian's Linux kernel salsa-ci.yml, which starts like this:

include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

# jobs are defined below ...

Select which jobs run in the CI pipeline

There are several ways to skip a certain job. The recommended way is to set to 1 (or "yes" or "true") the SALSA_CI_DISABLE_* variables that have been created for this purpose.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

# This sample disables all default tests, only disable those that you
# don't want
variables:
  SALSA_CI_DISABLE_APTLY: 1
  SALSA_CI_DISABLE_AUTOPKGTEST: 1
  SALSA_CI_DISABLE_BLHC: 1
  SALSA_CI_DISABLE_LINTIAN: 1
  SALSA_CI_DISABLE_PIUPARTS: 1
  SALSA_CI_DISABLE_REPROTEST: 1
  SALSA_CI_DISABLE_MISSING_BREAKS: 1
  SALSA_CI_DISABLE_BUILD_PACKAGE_ALL: 1
  SALSA_CI_DISABLE_BUILD_PACKAGE_ANY: 1
  SALSA_CI_DISABLE_BUILD_PACKAGE_I386: 1

Alternatively, it is possible to disable all tests, and then enable only the ones you want, which is good when testing a specific test:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

# This sample disables all default tests, and then enables just autopkgtest
variables:
  SALSA_CI_DISABLE_ALL_TESTS: 1
  SALSA_CI_ENABLE_AUTOPKGTEST: 1

Allow a job to fail

Without completely disabling a job, you can allow it to fail without failing the whole pipeline. That way, if the job fails, the pipeline will pass and show an orange warning telling you something went wrong.

For example, even though reproducible builds are important, reprotest's behaviour can sometimes be a slightly flaky and fail randomly on packages that aren't totally reproducible (yet!). In such case, you can allow reprotest to fail by adding this variable in your salsa-ci.yml manifest:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

reprotest:
  allow_failure: true

Disable the pipeline on certain branches

It is possible to configure the pipeline to skip branches you don't want CI to run on. The SALSA_CI_IGNORED_BRANCHES variable can be set to a regex that will be compared against the ref name and will decide if a pipeline is created.

By default, pipelines are only created for branches that contain a debian/ folder.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_IGNORED_BRANCHES: 'some-branch|another-ref'

Allow pipeline to run when git tags are pushed

By default, the pipeline is run only for commits, tags are ignored. To run the pipeline against tags as well, export the SALSA_CI_ENABLE_PIPELINE_ON_TAGS variable and set it to one of "1", "yes" or "true", like in the following example:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_ENABLE_PIPELINE_ON_TAGS: 1

Setting variables on pipeline creation

You can set these and other similar variables when launching a new pipeline in different ways:

• Using the web interface under CI/CD, Run Pipeline, and setting the desired variables.
• Using the GitLab API. For example, check the script salsa_drive_build.py, in particular the function launch_pipelines.
• Setting them as part of a pipeline-triggered build.

Skip pipeline temporarily when running a git push

There may be reasons to skip the whole pipeline for a single commit, for example when updating a README in a large package with a long-running CI.

You can achieve this in two ways:

You can insert [ci skip] or [skip ci], using any capitalization, in the commit message. With this marker, GitLab will not run the pipeline when the commit is pushed.

Alternatively, you can skip CI on all commits on a single git push option by passing a git option:

git push -o ci.skip

See also https://salsa.debian.org/help/ci/pipelines/index.md#skip-a-pipeline

Run only selected jobs

If you want to use the definitions provided by the Salsa CI Team, but want to explicitly define which jobs to run, you might want to declare your YAML as follows:

---
include: https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

extract-source:
    extends: .provisioning-extract-source

variables:
  RELEASE: 'experimental'

build:
  extends: .build-package

test-build-any:
  extends: .test-build-package-any

test-build-all:
  extends: .test-build-package-all

test-crossbuild-arm64:
  extends: .test-crossbuild-package-arm64

reprotest:
  extends: .test-reprotest

lintian:
  extends: .test-lintian

autopkgtest:
  extends: .test-autopkgtest

blhc:
  extends: .test-blhc

piuparts:
  extends: .test-piuparts

aptly:
  extends: .publish-aptly

On the previous example, the package is built on Debian experimental and checked through on all tests currently provided. You can choose to run only some of the jobs by deleting any of the definitions above.

As new changes are expected to happen from time to time, we firmly recommend NOT to do define all jobs manually. Most of the time it is better to simply select which jobs run in the CI pipeline.

Customize builds and build dependencies

Extend the job timeout

To prevent runaway jobs from reserving CI runners, jobs have a default timeout that is typically 1h or 3h. There can also be maximum default value defined in the GitLab instance configuration.

If a CI job has a valid reason to run beyond the default limits, a custom value can be defined with the timeout keyword for individual jobs. However, the primary way to address slow CI jobs should by optimizing the build to run faster, for example by ensuring that the build cache works. For C/C++ programs ensure the build is compatible with ccache.

build:
  extends: .build-package
  timeout: 3h

Decrease build timeout to leave margin for cache upload

When a job runs longer than allowed and times out, the job will stop immediately without entering the after_script phase, and without saving the cache and without saving the artifacts.

To prevent this, the build phase of the build job and the build phase of the reprotest job have a timeout of 2.75h (typically the runner's timeout is 3h). This permits also saving the cache of ccache. That way, on the next run, there is more chance to finish the job since it can use ccache's cache.

You can set the SALSA_CI_BUILD_TIMEOUT_ARGS variable to override this. The arguments can be any valid argument used by the timeout command . For example, you may set:

variables:
  SALSA_CI_BUILD_TIMEOUT_ARGS: "0.75h"

Disabling building on i386

The build i386 job builds packages against the 32-bit x86 architecture. If for any reason you need to skip this job, set the SALSA_CI_DISABLE_BUILD_PACKAGE_I386 in the variables' block to 1, 'yes' or 'true'. i.e;

variables:
  SALSA_CI_DISABLE_BUILD_PACKAGE_I386: 1

Enable building on ARM and RISC-V

Salsa CI includes builds jobs for armel, armhf, arm64 and riscv64. They are disabled by default, but can be enabled if a project has a GitLab runner on arm64 hardware available and tagged arm64, or on RISC-V tagged riscv64.

If you know you have such a runner available, you can activate ARM or RISC-V build respectively by setting the related variables to anything different than 1, 'yes' or 'true':

variables:
  SALSA_CI_DISABLE_BUILD_PACKAGE_ARM64: 0
  SALSA_CI_DISABLE_BUILD_PACKAGE_ARMEL: 0
  SALSA_CI_DISABLE_BUILD_PACKAGE_ARMHF: 0
  SALSA_CI_DISABLE_BUILD_PACKAGE_RISCV64: 0

All Debian Developers can enable salsaci-arm64-runner-01.debian.net or salsaci riscv64 runner 01 in any project.

Contributors who fork the project might not have runners tagged arm64 or riscv64 in their project. The example below illustrates how to limit the special build jobs to the debian project name space with extra rules:

build arm64:
  extends: .build-package-arm64
  rules:
    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"

build armel:
  extends: .build-package-armel
  rules:
    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"

build armhf:
  extends: .build-package-armhf
  rules:
    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"

build riscv64:
  extends: .build-package-riscv64
  rules:
    - if: $CI_PROJECT_ROOT_NAMESPACE  == "debian"

Add more architectures or CI runners

If you want to add more architectures or for other reasons have your own runners, see RUNNERS.md.

Testing build of arch=any and arch=all packages

If your package contains binary packages for all or any, you may want to test if those can be built in isolation from the full build normally done.

This verifies the Debian buildds can build your package correctly when building for other architectures that the one you uploaded in or in case a binNMU is needed or you want to do source-only uploads.

The default pipeline-jobs.yml does this automatically based on the contents of debian/control, but if you manually define the jobs to run, you also need to include the test-build-any and test-build-all jobs manually as well:

test-build-any:
  extends: .test-build-package-any

test-build-all:
  extends: .test-build-package-all

.test-build-package-any runs dpkg-buildpackage with the option --build=any and will only build arch-specific packages.

.test-build-package-all does the opposite and runs dpkg-buildpackage with the option --build=all building only arch-indep packages.

Testing build profiles

As its name suggests, the test-build-profiles job makes it possible to easily test specific build profiles. This job is disabled by default, so you need to enable it manually and set the profile(s) in the BUILD_PROFILES variable as a comma-separated list. For example:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_ENABLE_BUILD_PACKAGE_PROFILES: 1
  BUILD_PROFILES: nocheck,nodoc

Also, it is also possible to run several jobs in parallel to test different profiles independently:

include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_ENABLE_BUILD_PACKAGE_PROFILES: 1

test-build-profiles:
  extends: .test-build-package-profiles
  parallel:
    matrix:
      - BUILD_PROFILES: nocheck
      - BUILD_PROFILES: nodoc

Enable cross-builds

The job test-crossbuild-arm64 can be used to check whether it is possible to cross-build the package. To enable this check, either run your pipeline manually with SALSA_CI_DISABLE_CROSSBUILD_ARM64 set to anything different than 1, 'yes' or 'true' or by adding the following to your debian/salsaci.yml:

variables:
  SALSA_CI_DISABLE_CROSSBUILD_ARM64: 0

Enable building packages twice in a row

The job test-build-twice can be used to check whether it is possible to run dpkg-buildpackage twice in a row. To enable this check, either run your pipeline manually with SALSA_CI_DISABLE_BUILD_PACKAGE_TWICE set to anything different than 1, 'yes' or 'true' or by adding the following to your debian/salsaci.yml:

variables:
  SALSA_CI_DISABLE_BUILD_PACKAGE_TWICE: 0

Enable generation of dbgsym packages

To reduce the size of the artifacts produced by the build jobs, auto-generation of dbgsym packages is disabled by default. This behaviour can be controlled by the SALSA_CI_DISABLE_BUILD_DBGSYM. Set it to anything different than 1, 'yes' or 'true', to generate those packages.

variables:
  SALSA_CI_DISABLE_BUILD_DBGSYM: 0

Build with non-free dependencies

By default, only main repositories are used. If your package has dependencies or build-dependencies in the contrib or non-free components (archive areas), set SALSA_CI_COMPONENTS to indicate this:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
    RELEASE: 'stretch'
    SALSA_CI_COMPONENTS: 'main contrib non-free'

This is currently used for piuparts, but is likely to be used for other stages in future.

It is possible to use the SALSA_CI_EXTRA_REPOSITORY support to add a suitable apt source to the build environment and allow builds to access build-dependencies from contrib and non-free. You will need permission to modify the Salsa Settings for the project.

The CI/CD settings are at a URL like:

https://salsa.debian.org/%{project_path}/-/settings/ci_cd Expand the section on Variables and add a File type variable:

Key: SALSA_CI_EXTRA_REPOSITORY

Value: deb [signed-by=/usr/share/keyrings/debian-archive-keyring.gpg] https://deb.debian.org/debian/ sid contrib non-free

The apt source should reference sid or unstable.

Many contrib and non-free packages only build on amd64, so the 32-bit x86 build (build i386) should be disabled. (refer to the Disabling building on i386 Section).

Add private repositories to the builds

The variables SALSA_CI_EXTRA_REPOSITORY and SALSA_CI_EXTRA_REPOSITORY_KEY can be used to add private apt repositories to the sources.list, to be used by the build and tests, and (optionally) the signing key for the repositories in armor format. Alternatively, the single variable SALSA_CI_EXTRA_REPOSITORY_SOURCES can be used to add an extra repository (with its corresponding signing key) in deb822-style format (which is conveniently rendered by the aptly job). These variables are of type file, which eases the multiline handling, but have the disadvantage that their content can't be set on the salsa-ci.yml file - but they can be added to the repository as files and have their filenames then set in the salsa-ci.yml file:

---
include: https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml

variables:
  SALSA_CI_EXTRA_REPOSITORY: debian/ci/extra_repository.list
  SALSA_CI_EXTRA_REPOSITORY_KEY: debian/ci/extra_repository.asc
  # or
  SALSA_CI_EXTRA_REPOSITORY_SOURCES: debian/ci/extra_repository.sources

See also Using automatically built apt repository

Add extra arguments to dpkg-buildpackage

Sometimes it is desirable to add direct options to the dpkg-buildpackage that is run for the package building.

You can do this using the SALSA_CI_DPKG_BUILDPACKAGE_ARGS variable.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_DPKG_BUILDPACKAGE_ARGS: --your-option

Adding extra arguments to gbp-buildpackage

Sometimes it is desirable to add direct options to the gbp buildpackage command.

You can do this using the SALSA_CI_GBP_BUILDPACKAGE_ARGS variable.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_GBP_BUILDPACKAGE_ARGS: --your-option

Git attributes

Some upstream projects ship a .gitattributes file to set up special attributes to specific paths. To properly handle those path attributes, the Salsa CI's pipeline relies on gbp setup-gitattributes, that is call after fetching all the required branches from the repository. See gitattributes(5) and gbp-setup-gitattributes(1), and #322. If gbp setup-gitattributes is causing trouble (such as staging changes or encoding inconsistencies), the gbp setup-gitattributes call can be disabled setting the SALSA_CI_DISABLE_GBP_SETUP_GITATTRIBUTES variable to 1, 'yes' or 'true':

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/salsa-ci.yml
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/pipeline-jobs.yml

variables:
  SALSA_CI_DISABLE_GBP_SETUP_GITATTRIBUTES: 1

Customize reprotest

The reprotest job runs with the time (see below) and build_path variations disabled by default. To save memory usage, it also runs with diffoscope disabled. This behaviour can be modified as explained here below.

Add extra arguments to reprotest

Sometimes it is desirable to disable (or enable) some reprotest validations because the reproducibility issue comes inherently from the programming language being used, and not from the code being packaged.

You can customize the variations tested by reprotest by adding extra parameters in the SALSA_CI_REPROTEST_ARGS variable. For example, some compilers embed the build path in the generated binaries. As mentioned above, the build_path variation is disabled by default, but if you still want to test it, you can enable it as shown below. Please refer to reprotest(1) to find more information about the variations available.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_REPROTEST_ARGS: --vary=+build_path

Run reprotest for custom build artifacts

By default reprotest will check reproducibility of any binary packages built, using the *.deb shell glob. This should be fine for most packages, but some can generate files that don't match this pattern. For example, various packages create "udeb"s to be used by the installer and the debian-installer package generates a tarball of images.

You can customize the build artifacts tested by reprotest by setting the SALSA_CI_REPROTEST_ARTIFACT_PATTERN variable.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_REPROTEST_ARTIFACT_PATTERN: '*.deb *.udeb *-images_*.tar.gz'

Run reprotest with diffoscope

Reprotest stage can be run with diffoscope, which is an useful tool that helps identifying reproducibility issues. Large projects will not pass on low resources runners as the ones available right now.

To enable diffoscope, setting SALSA_CI_REPROTEST_ENABLE_DIFFOSCOPE to 1 (or 'yes' or 'true') is needed.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_REPROTEST_ENABLE_DIFFOSCOPE: 1

Break up the reprotest job into the different variations

By default, reprotest applies all the known variations together (see the full list at reprotest(1)). One way to debug a failing reprotest job and find out what variations are producing unreproducibility issues is to run the variations independently.

If you want to run multiple reprotest jobs, one for each variation, set the SALSA_CI_ENABLE_ATOMIC_REPROTEST variable to 1, 'yes' or 'true':

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_ENABLE_ATOMIC_REPROTEST: 1

You can also set the SALSA_CI_ENABLE_ATOMIC_REPROTEST variable when triggering the pipeline, without the need of creating a specific commit.

Lintian, Autopkgtests, Piuparts and other quality assurance CI jobs

Customize Lintian

The Lintian job can be customized to ignore certain tags.

To ignore a tag, add it to the setting SALSA_CI_LINTIAN_SUPPRESS_TAGS.

By default, the Lintian jobs fail either if a Lintian run-time error occurs or if Lintian finds a tag of the error category.

To also fail the job on findings of the category warning, set SALSA_CI_LINTIAN_FAIL_WARNING to 1 (or "yes" or "true").

To make Lintian shows overridden tags, set SALSA_CI_LINTIAN_SHOW_OVERRIDES to 1 (or "yes" or "true").

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_LINTIAN_FAIL_WARNING: 1
  SALSA_CI_LINTIAN_SUPPRESS_TAGS: 'orig-tarball-missing-upstream-signature'

It is possible to add any other lintian argument using SALSA_CI_LINTIAN_ARGS. Those arguments will be appended after the arguments generated by the pipeline.

Make autopkgtest more strict

By default, the autopkgtest job will succeed if autopkgtest exits with status 0, 2 or 8. If you would like the autopkgtest job to only succeed if all tests pass and fail otherwise, you can restrict success to exit status 0 by writing:

variables:
  SALSA_CI_AUTOPKGTEST_ALLOWED_EXIT_STATUS: '0'

To allow multiple exit codes, separate them by comma.

Add extra arguments to autopkgtest

Sometimes it is desirable to add arguments to autopkgtest.

You can do this by setting the arguments in the SALSA_CI_AUTOPKGTEST_ARGS variable.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_AUTOPKGTEST_ARGS: '--debug'

Note that autopkgtest can access the repository in the current directory, making it possible for --setup-commands to read commands from a file. For example:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_AUTOPKGTEST_ARGS: '--setup-commands=ci/pin-django-from-backports.sh'

Avoid autopkgtest failures on systemd masked tmp

If an autopkgtest fail with:

$> systemctl restart myservice
Failed to restart myservice.service: Unit tmp.mount is masked.

The error is probably due to bug #1078157.

You could workaround by adding at the top of the autopkgtest:

SERVICE=myservice
mkdir -p /run/systemd/system/$SERVICE.service.d
tee /run/systemd/system/$SERVICE.service.d/disabletmp.conf << "EOF"
[Service]
PrivateTmp=no
EOF
systemctl daemon-reload

Run a pre-install / post-install script in piuparts

Sometimes it is desirable to execute a pre-install or post-install scripts in piuparts.

You can do this using the SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT and SALSA_CI_PIUPARTS_POST_INSTALL_SCRIPT variables, that may point to the path of scripts in the project repository. This could be used, for instance, to pin dependencies. For example, if you had a ci/pin-django-from-backports.sh script:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  SALSA_CI_PIUPARTS_PRE_INSTALL_SCRIPT: 'ci/pin-django-from-backports.sh'

The ci/pin-django-from-backports.sh:

#!/bin/sh

set -e

# Ensure that python3-django from bullseye-backports will be
# installed as needed (bullseye has too old version)
cat >/etc/apt/preferences.d/99django-backports <<EOT
Package: python3-django
Pin: release a=bullseye-backports
Pin-Priority: 900
EOT

Using automatically built apt repository

The Aptly task runs in the publish stage and will save published apt repository files as its artifacts, so downstream CI tasks may access built binary/source packages directly through artifacts url via apt. This is currently disabled by default. To enable it, set the SALSA_CI_DISABLE_APTLY variable of the repository whose artifacts you want to use to anything other than 1, 'yes' or 'true'. (check example below)

To specify repository signing key, export the gpg key/passphrase as CI / CD Variables SALSA_CI_APTLY_GPG_KEY and SALSA_CI_APTLY_GPG_PASSPHRASE. Otherwise, an automatically generated one will be used.

For example, to let package src:pkgA of team ${TEAM} and project ${PROJECT} setup an aptly repository and let package src:pkgB use the repository, enable the aptly job by adding the following line to the debian/salsa-ci.yml of src:pkgA:

variables:
  SALSA_CI_DISABLE_APTLY: 0

The next time the pipeline of src:pkgA is run, a new job called aptly will be part of the "Publish" stage of the pipeline. Click on the job to obtain the job number which will be needed in the debian/salsa-ci.yml file of src:pkgB:

In the debian/salsa-ci.yml file of src:pkgB add the following lines after the variables section

before_script:
  - echo "deb [trusted=yes] https://salsa.debian.org/%{CI_PROJECT_PATH_SLUG}/-/jobs/${CI_JOB_ID}/artifacts/raw/aptly unstable main" | tee /etc/apt/sources.list.d/pkga.list
  - apt-get update

Replace %{CI_PROJECT_PATH_SLUG} with complete path to project, and ${CI_JOB_ID} with the aptly job number of src:pkgA pipeline. Now you can use the binary packages produced by src:pkgA in src:pkgB.

Note: When you make changes to src:pkgA, src:pkgB will continue using the old repository that the job number points to. If you want src:pkgB to use the updated binary packages, you have to retrieve the job number of the aptly job from src:pkgA and update the ${CI_JOB_ID} of src:pkgB.

See also howto add private repositories to the builds.

Enable wrap-and-sort job

The job wrap-and-sort can be used to check if files in the debian/ folder are wrapped properly using wrap-and-sort(1). To enable this check, either run your pipeline manually with SALSA_CI_DISABLE_WRAP_AND_SORT set to anything different than 1, 'yes' or 'true' or by adding the following to your debian/salsa-ci.yml:

variables:
  SALSA_CI_DISABLE_WRAP_AND_SORT: 0

You can configure the parameters passed to wrap-and-sort using the SALSA_CI_WRAP_AND_SORT_ARGS variable like this:

variables:
  SALSA_CI_DISABLE_WRAP_AND_SORT: 0
  SALSA_CI_WRAP_AND_SORT_ARGS: '-asbkt'

The style to use is a subjective decision. The default behaviour may change over time #895570. We suggest to consider -a --wrap-always, -s --short-indent, -b --sort-binary-packages, -k --keep-first, and -t --trailing-comma as they encourage a minimal, consistent and deterministic style.

Debian release bump

By default, the build job will increase the release number using the +salsaci suffix. To disable this behavior set the SALSA_CI_DISABLE_VERSION_BUMP to 1, 'yes' or 'true'.

Distribution and release selection

Customise what Debian release to use

By default, everything will run based on the target release in debian/changelog. If the latest entry is UNRELEASED, then the target release from the second entry will be used.

The target release can be customized by setting the RELEASE variable explicitly.

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/debian.yml

variables:
  RELEASE: 'bookworm'

The following releases are available:

• stretch
• buster
• bullseye
• bookworm
• bookworm-backports
• stable
• testing
• unstable
• experimental

Experimental: Ubuntu support

⚠️ The Ubuntu support is experimental and exists to collect feedback and potential contributions from the wider Ubuntu community. It may be removed if the benefits are not clearly larger than the cost of maintaining it.

Test Ubuntu packages by using recipe debian/salsa-ci.yml:

---
include:
  - https://salsa.debian.org/salsa-ci-team/pipeline/raw/master/recipes/ubuntu.yml

Additionally you must ensure a Ubuntu release is defined in the latest debian/changelog entry or as a RELEASE variable.

Currently autopkgtest is the only test job enabled by default (see 1 and 2). Other test jobs can be enabled using SALSA_CI_DISABLE_* variables.

Testing Salsa CI (and GitLab CI in general) pipeline definitions locally

Testing definitiion file for correctness

The salsa-ci.yml file contents can be pasted into the embedded editor on the project page Build > Pipeline editor. In addition to the interactive feedback, the tabs Visualize, Validate and Full configuration can help in understanding how the pipeline definition will be interpreted by the GitLab CI runner. The Validate functionality is also available in the glab command-line client:

$ glab ci lint debian/salsa-ci.yml
Validating...
debian/salsa-ci.yml is invalid.
1 (<unknown>): could not find expected ':' while scanning a simple key at line 7 column 5

# After fix
$ glab ci lint debian/salsa-ci.yml
Validating...
✓ CI/CD YAML is valid!

As the pipeline definition file is plain YAML, the generic yamllint debian/ is also useful to detect syntax issues.

Running the pipeline locally

There are various efforts to provide tooling to allow individual developers to run the full pipeline or individual jobs locally for debugging purposes, with the most notable being gitlab-ci-local. These are still under development and usefulness will vary depending on pipeline complexity.

General Debian packaging support and resources

• The Debian Policy: packages must conform to it.
• The Developers Reference: details best packaging practices.
• The Guide for Debian Maintainers: puts a bit of the two above in practice.

General Salsa information

The GitLab instance salsa.debian.org is maintained by the Debian Salsa admin team, which is separate from the Salsa CI team.

Support for Salsa CI use

The Salsa CI developers and users can be found at:

• Mailing list: debian-salsa-ci_at_alioth-lists.debian.net
• Matrix: #salsaci:matrix.debian.social
• IRC: #salsaci @ OFTC (webchat)
• Issue tracker: salsa-ci-team/pipeline/issues

Contributing

The success of this project comes from meaningful contributions that are made by interested contributors like you. If you want to contribute to this project, follow the detailed guidelines in the CONTRIBUTING file