Increase default job timeout back to 2h for salsa-runner.debian.net

Previously my MariaDB builds have been running perfectly on Salsa:

This week it seems that the default runner timeout has decreased to 1h, which is too little, and all MariaDB builds fail:

Could you please consider revergint that change and make 2h the default timeout again. Thank you!

Background

MariaDB is a big and complex package. Thanks to the CI features of Salsa, we have have been able to decrease the number of packaging related bugs and catch a lot of upstream regressions pre-upload, before anybody had to suffer. We absolutely love Salsa and the CI facilities in particular. We've spend hundreds of ours in developing a nice CI suite and it has proven to be very successful. It would be a real shame if we, or anybody forking and contributing the packaging, would not be able to run the CI anymore.

changed the description

https://salsa.debian.org/mariadb-team/mariadb-10.3/-/jobs/459101

https://salsa.debian.org/mariadb-team/mariadb-10.3/-/jobs/435944

I understand that big builds consume Salsa resources, but I would argue that the cost/benefit of using CI in MariaDB packaging for Debian is good and should be allowed.

Seems Gitlab 10.7 introduced a possibility to override the timeout per runner: https://salsa.debian.org/help/user/project/pipelines/settings.html#timeout-overriding-on-runner-level

Maybe the upgrade 10.7 lead to this situation? According to https://salsa.debian.org/help/ci/runners/README.html#setting-maximum-job-timeout-for-a-runner the default runner timeout is however 2h.

I have 2h defined in https://salsa.debian.org/mariadb-team/mariadb-10.3/-/settings/ci_cd as the general timeout.

I am about to package MariaDB 10.4 in the coming days. It would be super important to have a working CI. Can somebody please help out here to get back the 2h timeout limit?

More on the background:

• https://debconf19.debconf.org/talks/63-how-mariadb-packaging-uses-salsa-ci-to-ensure-smooth-upgrades-and-avoid-regressions/
• https://www.slideshare.net/ottokekalainen/how-mariadb-packaging-uses-salsaci-to-ensure-smooth-upgrades-and-avoid-regressions

I am also affected with a weekly build of a sample database for distro-tracker. The job was working fine (but it was taking about 3 hours) and the limits have been reduced without any prior announce. I have no alternate solutions... @waldi was rather unhelpful in #alioth suggesting that we should run our own runner if we want to have jobs taking longer. He said that the QA team has quantz and that we could install a GitLab runner on quantz.

https://salsa.debian.org/qa/distro-tracker/pipelines

And quantz is very much not suitable to run any kind of CI system on it, thank you.

Custom runners are not the solution because that works only for committed existing developers. All aspiring new contributors would be put off as they could not even get the CI to pass for the baseline. New aspiring contributors are the most valuable thing to invest in and we should certainly not have any structural things that make the new contributor experience worse than it needs to be.

So is this a question of computing resources? How much do we have now and how big of a donation do you want to have to be able to run a big enough public worker for Salsa?

This issue has been open for 2 weeks now.. Any updates from Salsa Admins? (@formorer @ganneff @waldi)

Otto Kekäläinen commented:

This issue has been open for 2 weeks now.. Any updates from Salsa Admins? (@formorer @ganneff @waldi) and those 2 weeks were vacation weeks.

We will have a meeting on monday.

Alex

Otto Kekäläinen commented:

This issue has been open for 2 weeks now.. Any updates from Salsa Admins? (@formorer @ganneff @waldi) we had that meeting and I changed the timeout to 3 hours. Please ensure not to create too many jobs using that timeout.

Alex

mentioned in merge request mariadb-team/mariadb-10.3!26 (merged)

Thank you @formorer! The 2h limit that was before is enough for MariaDB, no need to have 3h. The build passed after 1h36m, and now when the ccache is freshly primed, all new builds pass in 30m. Thanks to this change I was able to review and merge a MR from a contributor just now. Thanks!

closed