Skip to content

Slow .json test_html_regression_875281 makes the whole suite 3 times slower

The test_html_regression_875281 in tests/test_presenters.py takes more than 1min, about 70% of the time to run the entire test suite (357 passed, 78 skipped) on my system. In other words, excluding this single test makes testing almost 3 times faster on my system with one process. With pytest-3 --numprocesses=4, excluding this single test makes running the whole suite between 5 and 6 times faster.

This of course also consumes 3 times more valuable CI resources and delays the CI feedback loop significantly.

Much worse, it's really not clear what this test achieves.

  • The second element of its pair=(debian-bug-875281.diff.json',) argument is empty, does this mean it's comparing debian-bug-875281.diff.json with nothing?! That's not even possible on the command line and could explain why it's taking so long (all the time is spent in diffoscope main() method)
  • As of HEAD a93aa33d, the comment # this test fails when you git revert -Xtheirs ccd926f is wrong. The test keeps passing even after this revert.

Workaround: time pytest-3 -n 4 -k 'not '875281'

https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=875281

Edited by Chris Lamb
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information