Skip to content
Commits on Source (13)
# Contributing
The preferred way to report bugs about *diffoscope*, as well as suggest
fixes and requests for improvements, is to submit reports to the issue
tracker at:
https://salsa.debian.org/reproducible-builds/diffoscope/issues
You can also submit patches via *Merge Request* on Salsa, Debian's
Gitlab instance. Start by forking the
[`diffoscope` Git repository](https://salsa.debian.org/reproducible-builds/diffoscope)
(see [documentation](https://salsa.debian.org/help/gitlab-basics/fork-project.md)),
make your changes and commit them as you normally would. You can then push your
changes and submit a *merge request* via Salsa. See:
[Gitlab documentation](https://salsa.debian.org/help/gitlab-basics/add-merge-request.md)
about *Merge Requests*.
You can also submit bugs about Debian specific issues to the Debian bug
tracker.
## Testing
diffoscope's test suite relies on [pytest](https://docs.pytest.org/);
to run all tests use `pytest[-3]`, appending `-n 4` or similar to enable
running tests concurrently. For faster interactive development here's
an example of how to run a (much) smaller subset of tests:
$ pytest -v --exitfirst -k lib tests/comparators/test_elf.pyc [-pdb]
More options are available at `[pytest -h]`.
## Add a comparator
Diffoscope doesn't support a specific file type? Please contribute to
the project! Each file type is handled by a comparator, and writing a
new one is usually very easy. Here are the steps to add a new
comparator:
- Add the new comparator in `diffoscope/comparators/` (have a look at
the other comparators in the same directory to have an idea of what
to do)
- Declare the comparator File class in `ComparatorManager` in
`diffoscope/comparators/__init__.py`
- Add a test in `tests/comparators/`
- If required, update the `Build-Depends` list in `debian/control`
- If required, update the `EXTERNAL_TOOLS` list in
`diffoscope/external_tools.py`
## Uploading the package
When uploading diffoscope to the Debian archive, please take extra care
to make sure the uploaded source package is correct, that is it includes
the files tests/data/test(1o) which in some cases are removed by
dpkg-dev when building the package.
See [#834315](https://bugs.debian.org/834315) for an example FTBFS bug
caused by this. (See [#735377](https://bugs.debian.org/735377#44)
and followups to learn how this happened and how to prevent it)
Please also release a signed tarball:
$ VERSION=FIXME
$ git archive --format=tar --prefix=diffoscope-${VERSION}/ ${VERSION} | bzip2 -9 > diffoscope-${VERSION}.tar.bz2
$ gpg --detach-sig --armor --output=diffoscope-${VERSION}.tar.bz2.asc < diffoscope-${VERSION}.tar.bz2
And commit them to our LFS repository at:
https://salsa.debian.org/reproducible-builds/reproducible-lfs
After uploading, please also update the version on PyPI using:
$ python3 setup.py sdist upload --sign
Once the tracker.debian.org entry appears, consider tweeting the release
on `#reproducible-builds` with:
%twitter diffoscope $VERSION has been released. Check out the changelog here: $URL
Finally, update the Docker image using:
$ docker build -t registry.salsa.debian.org/reproducible-builds/diffoscope .
$ docker push registry.salsa.debian.org/reproducible-builds/diffoscope
Contributing
============
The preferred way to report bugs about diffoscope, as well as suggest fixes and
requests for improvements, is to submit reports to the issue tracker at
https://salsa.debian.org/reproducible-builds/diffoscope/issues
You can also submit patches via *merge request* to Salsa, Debian's Gitlab. Start
by forking the `diffoscope Git
repository <https://salsa.debian.org/reproducible-builds/diffoscope>`__
(see
`documentation <https://salsa.debian.org/help/gitlab-basics/fork-project.md>`__),
make your changes and commit them as you normally would. You can then push your
changes and submit a *merge request* via Salsa. See `Gitlab documentation
<https://salsa.debian.org/help/gitlab-basics/add-merge-request.md>`__ about
*merge requests*.
You can also submit bugs about Debian specific issues to the Debian bug tracker.
Testing
=======
diffoscope's test suite relies on `pytest <https://docs.pytest.org/>`__;
to run all tests use ``pytest[-3]``, appending ``-n 4`` or similar to enable
running tests concurrently. For faster interactive development here's an
example of how to run a (much) smaller subset of tests:
pytest -v --exitfirst -k lib tests/comparators/test_elf.pyc [ --pdb ]
More options in `pytest -h`.
Add a comparator
================
Diffoscope doesn't support a specific file type? Please contribute to the
project! Each file type is handled by a comparator, and writing a new one is
usually very easy.
Here are the steps to add a new comparator:
- Add the new comparator in ``diffoscope/comparators/`` (have a look at the
other comparators in the same directory to have an idea of what to do)
- Declare the comparator File class in ``ComparatorManager`` in
``diffoscope/comparators/__init__.py``
- Add a test in ``tests/comparators/``
- If required, update the ``Build-Depends`` list in ``debian/control``
- If required, update the ``EXTERNAL_TOOLS`` list in
``diffoscope/external_tools.py``
Uploading the package
=====================
When uploading diffoscope to the Debian archive, please take extra care to make
sure the uploaded source package is correct, that is it includes the files
tests/data/test(1|2).(a|o) which in some cases are removed by dpkg-dev when
building the package. See `#834315 <https://bugs.debian.org/834315>`__ for an example
FTBFS bug caused by this. (See `#735377
<https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=735377#44>`__ and followups
to learn how this happened and how to prevent it)
Please also release a signed tarball::
$ VERSION=FIXME
$ git archive --format=tar --prefix=diffoscope-${VERSION}/ ${VERSION} | bzip2 -9 > diffoscope-${VERSION}.tar.bz2
$ gpg --detach-sig --armor --output=diffoscope-${VERSION}.tar.bz2.asc < diffoscope-${VERSION}.tar.bz2
And commit them to our LFS repository at https://salsa.debian.org/reproducible-builds/reproducible-lfs
After uploading, please also update the version on PyPI using::
$ python3 setup.py sdist upload --sign
Once the tracker.debian.org entry appears, consider tweeting the release on
``#reproducible-builds`` with::
%twitter diffoscope $VERSION has been released. Check out the changelog here: $URL
Finally, update the Docker image using::
docker build -t registry.salsa.debian.org/reproducible-builds/diffoscope .
docker push registry.salsa.debian.org/reproducible-builds/diffoscope
......@@ -60,7 +60,7 @@ diffoscope/presenters/icon.py: favicon.png
echo '""".replace("\\n", "")'; \
) > $@
favicon.png: logo.svg
favicon.png: doc/logo.svg
inkscape -w 32 -h 32 -e $@ $<
override_dh_auto_clean:
......
......@@ -81,7 +81,7 @@ class AndroidBootImgContainer(Archive):
class AndroidBootImgFile(File):
DESCRIPTION = "Android boot images"
FILE_TYPE_RE = re.compile(r'^Android bootimg\b')
CONTAINER_CLASS = AndroidBootImgContainer
CONTAINER_CLASSES = [AndroidBootImgContainer]
def compare_details(self, other, source=None):
return [Difference.from_command(AbootimgInfo, self.path, other.path)]
......@@ -24,12 +24,11 @@ import subprocess
from diffoscope.tools import tool_required
from diffoscope.tempfiles import get_temporary_directory
from diffoscope.difference import Difference
from .utils.file import File
from .utils.archive import Archive
from .utils.compare import compare_files
from .zip import zipinfo_differences
from .zip import ZipContainer, zipinfo_differences
from .missing_file import MissingFile
logger = logging.getLogger(__name__)
......@@ -173,7 +172,7 @@ class ApkFile(File):
FILE_TYPE_HEADER_PREFIX = b"PK\x03\x04"
FILE_TYPE_RE = re.compile(r'^(Java|Zip) archive data.*\b')
FILE_EXTENSION_SUFFIX = '.apk'
CONTAINER_CLASS = ApkContainer
CONTAINER_CLASSES = [ApkContainer, ZipContainer]
def compare_details(self, other, source=None):
return zipinfo_differences(self, other)
......
......@@ -60,7 +60,7 @@ class ArSymbolTableDumper(Command):
class ArFile(File):
DESCRIPTION = "ar(1) archives"
CONTAINER_CLASS = ArContainer
CONTAINER_CLASSES = [ArContainer]
FILE_TYPE_RE = re.compile(r'\bar archive\b')
def compare_details(self, other, source=None):
......
......@@ -67,7 +67,7 @@ class BinwalkFileContainer(Archive):
class BinwalkFile(File):
FILE_TYPE_RE = re.compile(r'\bcpio archive\b')
CONTAINER_CLASS = BinwalkFileContainer
CONTAINER_CLASSES = [BinwalkFileContainer]
@classmethod
def recognizes(cls, file):
......
......@@ -55,5 +55,5 @@ class Bzip2Container(Archive):
class Bzip2File(File):
DESCRIPTION = "bzip2 archives"
CONTAINER_CLASS = Bzip2Container
CONTAINER_CLASSES = [Bzip2Container]
FILE_TYPE_RE = re.compile(r'^bzip2 compressed data\b')
......@@ -125,7 +125,7 @@ def is_header_valid(buf, size, offset=0):
class CbfsFile(File):
DESCRIPTION = "Coreboot CBFS filesystem images"
CONTAINER_CLASS = CbfsContainer
CONTAINER_CLASSES = [CbfsContainer]
@classmethod
def recognizes(cls, file):
......
......@@ -28,7 +28,7 @@ from .utils.libarchive import LibarchiveContainer, list_libarchive
class CpioFile(File):
DESCRIPTION = "cpio archives"
CONTAINER_CLASS = LibarchiveContainer
CONTAINER_CLASSES = [LibarchiveContainer]
FILE_TYPE_RE = re.compile(r'\bcpio archive\b')
def compare_details(self, other, source=None):
......
......@@ -111,7 +111,7 @@ class DebContainer(LibarchiveContainer):
class DebFile(File):
CONTAINER_CLASS = DebContainer
CONTAINER_CLASSES = [DebContainer]
FILE_TYPE_RE = re.compile(r'^Debian binary package')
@property
......@@ -233,7 +233,7 @@ class DebTarContainer(TarContainer):
class DebDataTarFile(File):
CONTAINER_CLASS = DebTarContainer
CONTAINER_CLASSES = [DebTarContainer]
@classmethod
def recognizes(cls, file):
......
......@@ -109,7 +109,7 @@ class DebControlContainer(Container):
class DebControlFile(File):
CONTAINER_CLASS = DebControlContainer
CONTAINER_CLASSES = [DebControlContainer]
@property
def deb822(self):
......@@ -261,7 +261,7 @@ class DotBuildinfoContainer(DebControlContainer):
class DotBuildinfoFile(DebControlFile):
DESCRIPTION = "Debian .buildinfo files"
CONTAINER_CLASS = DotBuildinfoContainer
CONTAINER_CLASSES = [DotBuildinfoContainer]
FILE_EXTENSION_SUFFIX = '.buildinfo'
FILE_TYPE_RE = re.compile(r'^(ASCII text|UTF-8 Unicode text)')
......
......@@ -60,4 +60,4 @@ class DexContainer(Archive):
class DexFile(File):
DESCRIPTION = "Dalvik .dex files"
FILE_TYPE_RE = re.compile(r'^Dalvik dex file .*\b')
CONTAINER_CLASS = DexContainer
CONTAINER_CLASSES = [DexContainer]
......@@ -621,7 +621,7 @@ class ElfContainer(Container):
class ElfFile(File):
DESCRIPTION = "ELF binaries"
CONTAINER_CLASS = ElfContainer
CONTAINER_CLASSES = [ElfContainer]
FILE_TYPE_RE = re.compile(r'^ELF ')
def compare_details(self, other, source=None):
......
......@@ -88,7 +88,7 @@ class FsImageContainer(Archive):
class FsImageFile(File):
DESCRIPTION = "ext2/ext3/ext4/btrfs/fat filesystems"
CONTAINER_CLASS = FsImageContainer
CONTAINER_CLASSES = [FsImageContainer]
FILE_TYPE_RE = re.compile(r'^(Linux.*filesystem data|BTRFS Filesystem).*')
@classmethod
......
......@@ -56,7 +56,7 @@ class GzipContainer(Archive):
class GzipFile(File):
DESCRIPTION = "Gzipped files"
CONTAINER_CLASS = GzipContainer
CONTAINER_CLASSES = [GzipContainer]
FILE_TYPE_RE = re.compile(r'^gzip compressed data\b')
# Work around file(1) Debian bug #876316
......
......@@ -76,7 +76,7 @@ class ISO9660Listing(Command):
class Iso9660File(File):
DESCRIPTION = "ISO 9660 CD images"
CONTAINER_CLASS = LibarchiveContainerWithFilelist
CONTAINER_CLASSES = [LibarchiveContainerWithFilelist]
FILE_TYPE_RE = re.compile(r'\bISO 9660\b')
@classmethod
......
......@@ -56,7 +56,7 @@ class Lz4Container(Archive):
class Lz4File(File):
DESCRIPTION = "LZ4 compressed files"
CONTAINER_CLASS = Lz4Container
CONTAINER_CLASSES = [Lz4Container]
FILE_TYPE_RE = re.compile(r'^LZ4 compressed data \([^\)]+\)$')
# Work around file(1) Debian bug #876316
......
......@@ -109,7 +109,7 @@ class RpmContainer(Archive):
class RpmFile(AbstractRpmFile):
DESCRIPTION = "RPM archives"
CONTAINER_CLASS = RpmContainer
CONTAINER_CLASSES = [RpmContainer]
def compare_details(self, other, source=None):
return [compare_rpm_headers(self.path, other.path)]
......@@ -60,7 +60,7 @@ class RustObjectContainer(Archive):
class RustObjectFile(File):
DESCRIPTION = "Rust object files (.deflate)"
CONTAINER_CLASS = RustObjectContainer
CONTAINER_CLASSES = [RustObjectContainer]
FILE_TYPE_HEADER_PREFIX = b'RUST_OBJECT\x01\x00\x00\x00'
FILE_EXTENSION_SUFFIX = '.deflate'
......