Slow .json test_html_regression_875281 makes the whole suite 3 times slower
The test_html_regression_875281
in tests/test_presenters.py
takes more than 1min, about 70% of the time to run the entire test suite (357 passed, 78 skipped) on my system. In other words, excluding this single test makes testing almost 3 times faster on my system with one process. With pytest-3 --numprocesses=4
, excluding this single test makes running the whole suite between 5 and 6 times faster.
This of course also consumes 3 times more valuable CI resources and delays the CI feedback loop significantly.
Much worse, it's really not clear what this test achieves.
- The second element of its
pair=(debian-bug-875281.diff.json',)
argument is empty, does this mean it's comparingdebian-bug-875281.diff.json
with nothing?! That's not even possible on the command line and could explain why it's taking so long (all the time is spent in diffoscope main() method) - As of HEAD a93aa33d, the comment
# this test fails when you git revert -Xtheirs ccd926f
is wrong. The test keeps passing even after this revert.
Workaround: time pytest-3 -n 4 -k 'not '875281'
Edited by Chris Lamb