@lamby would it possible to change this @skip to something like:

@skip if black_version < 19.10b0 AND os.getenv('CI_JOB_ID')

Happy to submit the merge request if something like it is possible.

I happen to have a black version 19.3 that produces the exact same output than 19.10b0 on every merge request I just submitted recently - too bad I had to run it manually after submitting to CI.

Even if I had been less lucky and found some actual 19.3-19.10 differences, these differences would have been much smaller and much more convenient than the ones CI reported to me when the test suite didn't run black at all.

Not really, as we also run this test in many other environments (Debian's autopkgtest, buildds, local builds, etc).

Understood, but I assumed these other environments either had the latest black version required, or didn't mind a minor black difference test failure now and then.

I just realized what is the core issue with test_source.py: code inspection should be independent from real tests. See !37 (comment 130310)

Understood, but I assumed these other environments either had the latest black version required, or didn't mind a minor black difference test failure now and then.

at least within debian, the testsuite is also run, for example, when building backports, where the version of black may and often is older than required, and forcing a newer version in the test-dependencies would only make my life much harder (as it would probably for you).