Fix a regression introduced in diffoscope version 207 where diffoscope would crash if one directory contained a directory that wasn't in the other. Thanks to Alderico Gallo for the testcase. (Closes: reproducible-builds/diffoscope#310)