Prefer to use magic.Magic over the magic.open compatibility interface. (Closes: reproducible-builds/diffoscope#236)

See <https://github.com/ahupp/python-magic/blob/master/COMPAT.md> for more info.

Explicitly remove our top-level temporary directory. (Closes: reproducible-builds/diffoscope#234, Debian:#981123)

We cannot rely on __del__ being called determinstically, if at all, and
lets remove our top-level directory. Thanks to Dimitrios Apostolou and
Mattia Rizzolo.

This essentially reverts commit c645d403.

Gbp-Dch: ignore

Save sys.argv in our top-level temporary directory, in case it helps debug errant temporary directories.

Ref: reproducible-builds/diffoscope#234

Clarify in a comment that __del__ is not always called, so temporary directories are not neccessary removed the *moment* they go out of scope.

At least in CPython...

Ref: reproducible-builds/diffoscope#234

Conrad Ratschan authored Jan 27, 2021 and Chris Lamb committed Jan 28, 2021

Use manual magic comparison for FIT images (device tree blob) to work
around a bug in libmagic that results in device tree blob files not
correctly being identified when the structure section is larger than 1MB.

libmagic bug report: https://bugs.astron.com/view.php?id=232



Signed-off-by: Chris Lamb <lamby@debian.org>

Increase fuzzy matching threshold to ensure that we show more differences. (Closes: reproducible-builds/diffoscope#232)

The two files in this bug report had a score of 74, but another file had a
score of 106.

Gbp-Dch: ignore

Gbp-Dch: ignore

Collapse --acl and --xattr into --extended-filesystem-attributes to cover all of these extended attributes, defaulting the new option to false (ie. to not check these very expensive external calls).

Dimitrios Apostolou authored Jan 08, 2021 and Chris Lamb committed Jan 18, 2021

This offers a significant speedup to directory comparisons
by skipping invocation of utilities getfacl and lsattr.

Dimitrios Apostolou authored Jan 08, 2021 and Chris Lamb committed Jan 18, 2021

We compare (some fields at least) using python's os.lstat().
Only if we find differences, we call external stat command
so that we can generate a diff using its output.

This fixes some dpkg-gencontrol warnings:
dpkg-gencontrol: warning: package diffoscope: substitution variable ${diffoscope-minimal:Recommends} unused, but is defined
dpkg-gencontrol: warning: package diffoscope-minimal: substitution variable ${diffoscope:Recommends} unused, but is defined

Signed-off-by: Mattia Rizzolo <mattia@debian.org>

Ignore the DeprecationWarning regarding the `imp` module deprecation, as it comes from a 3rd party library

Signed-off-by: Mattia Rizzolo <mattia@debian.org>

This saves a DeprecationWrning in jenkins.

Signed-off-by: Mattia Rizzolo <mattia@debian.org>

Signed-off-by: Mattia Rizzolo <mattia@debian.org>

Gbp-Dch: ignore

Gbp-Dch: ignore

Commands with short outputs, like `stat`, `getfacl` and `lsattr`, now store
all their output in memory buffers and we run an in-memory comparison first.
We proceed to call `diff` only if the buffers are not identical.

* If the two sizes are different, files are different
* Else read the first SMALL_FILE_THRESHOLD bytes (64K)
  * If the first bytes are different, files are different
  * Else first bytes are identical, and if file size is
    <= SMALL_FILE_THRESHOLD then files are identical

Only if all these checks fail, spawn external command `cmp -s` to do the
comparison.

Conrad Ratschan authored Nov 28, 2020 and Chris Lamb committed Jan 05, 2021

Add comparator for U-Boot Flattened Image Tree files. Add required tools
and tests.

Dimitrios Apostolou authored Jan 04, 2021 and Chris Lamb committed Jan 05, 2021

Previously the temporary directories where all left in place until the
end of the process, when a massive cleanup was being run. The cause
was the _DIRS variable that was holding references to the
TemporaryDirectory objects, so their destructor was not running.

We now remove the _DIRS list completely (replacing it with _BASEDIR for
storing only the parent temporary directory). As a result, the temporary
directories and their contents are being cleaned up automatically when
the class destructor runs.

This in turn requires modifications to preexisting code:

1. Code that was directly using `get_temporary_directory().name`
   stopped working; the object from `get_temporary_directory()`
   needs to be referenced directly to keep it alive.
2. `get_temporary_directory()` is now starting a `with` block
   (context manager) where it was straightforward.

Dimitrios Apostolou authored Jan 04, 2021 and Chris Lamb committed Jan 05, 2021

These files can be seen in thousands, named "fifo1" and "fifo2", under the
temporary directory while diffoscope still runs. Instead of cleaning up
massively in the end, we now clean up while we go.