To compare text files we need to know the encoding. So we need
to rely solely on libmagic and not match the extension.

In order to properly read text files, we need to get the encoding.
As they are identified by libmagic, we simply need to pass what appears
in the MIME type to the comparator function. We do so by capturing
part of the regex using named group, and passing the resulting dictionary
to the comparison function.

With a small context large blocks of changes that have a single
or a few equal lines inbetween will be split into multiple
blocks.  By having a larger context, they will stay in one block
and more lines can be trimmed.

Creating HTML diffs with vim gets very slow when the files
become larger and there are very large blocks that differ.
To speed this up, the individual differing blocks are
shrunk when they are longer than a configured number of lines.
The information about the blocks is obtained by first calling
'diff', which is extremely fast even on large files.

Closes: #764140

Helmut Grohne authored Jan 09, 2015 and Jérémy Bobbio committed Jan 09, 2015

Let's not call guess_mime_type for every comparator available. Doing it once
should be enough.

Using magic and doing smart comparisons is slower than comparing bytes
directly. Let's assume most files will actually be identical and start
comparing small files directly. If they match, we can skip the smart
bits entirely.

This lead to drastic run time improvements when comparing source tarballs
with many small files (e.g. linux-source).

Closes: #773573