Skip to content

QA workflow can detect and highlight regressions

User story

As a member of a security team, when I prepare an update, I really care about avoiding regressions in the package. But if the package was in a bad shape earlier, I don't necessarily want to deal with those problems.

As such, the QA workflow should be enhanced to be able to provide an earlier version of the package, and when any of the QA task fails, then it should run the same QA task on the current version. If that task also fails, then it's not a regression and can be safely ignored.

  • The QA workflow should have a custom overview showing each QA test for the submitted package and the reference package, and the resulting conclusion.
  • The status of the QA workflow should be influenced by the fail_on setting, i.e. it can be "success" if we have failures but no regression when fail_on is set to regression.

Notes

The detailed design is in the blueprint from !1810 (merged)

Edited by Raphaël Hertzog
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information