Last year, I wrote a .gitlab-ci.yml file to automate building the website on gitlab CI. Unfortunately, it severely overloaded the salsa server on every commit; for this reason, automatically building the website was disabled in November of that year.
This merge request tries to remedy the situation, by taking several steps:
- The original CI system used a (1.5G) custom docker image. This needed to be downloaded from salsa for every build. Several dozens of GCS instances downloading that much data simultaneously caused bandwidth issues on salsa. To remedy, this system uses the "debian:stable" image instead, which is downloaded from the docker hub rather than salsa, and then installs the necessary software using apt-get. While that takes somewhat longer to run, it is assumed that the time used is less of a problem than the bandwidth used by the other solution.
- The original CI system used a script to determine whether a job (that had already been started) needed to do anything useful, and would do a fast exit if not. Unfortunately, by that time the job had already been created, and necessary resources (docker image...) downloaded, which was the majority of the issues. To remedy, this system uses the new "only:changes" key, which did not yet exist a year ago, which instructs gitlab CI to not even create a job if no changes were made in relevant places.
There are a few downsides:
- The previous system allowed to force-run jobs in one or more languages by adding a line like
force languages: dutch german frenchto the commit message, or by setting a variable in the CI configuration. Unfortunately, the new system does not allow that, since it would require more flexibility than the
only:changesconfiguration allows for. The
rules:feature that is currently in development by GitLab would allow for that, but that has not yet been released; an example configuration is added, but not activated (and can't be until the
rules:feature is deployed on salsa).
- The new system will build all languages for a tag (which are not used by the webwml repository, so should not be a problem) or for the first commit on a new branch -- where the first commit in a fork counts as a new branch. This might be problematic, and should be evaluated.