QA workflow can detect and highlight regressions
User story
As a member of a security team, when I prepare an update, I really care about avoiding regressions in the package. But if the package was in a bad shape earlier, I don't necessarily want to deal with those problems.
As such, the QA workflow should be enhanced to be able to provide an earlier version of the package, and when any of the QA task fails, then it should run the same QA task on the current version. If that task also fails, then it's not a regression and can be safely ignored.
- The QA workflow should have a custom overview showing each QA test for the submitted package and the reference package, and the resulting conclusion.
- The status of the QA workflow should be influenced by the
fail_on
setting, i.e. it can be "success" if we have failures but no regression whenfail_on
is set toregression
.
Notes
The detailed design is in the blueprint from !1810 (merged)
Edited by Raphaël Hertzog