Partially revert 7bf04a62

This is caused by a bug in `file`:
    https://bugs.astron.com/view.php?id=256
Fixed by:
    https://github.com/file/file/commit/749e1ecfc3d333e5ec8b1a2e639da7ccb2498c34



Closes: reproducible-builds/diffoscope#251
Signed-off-by: Mattia Rizzolo <mattia@debian.org>