Diffoscope unable to compare large images
Hi All,
I am facing issues while trying to compare 2 disk images with diffoscope. Each image is around ~ 5.5 GB. I am using diffoscope 271. There are multiple partitions in the disk image. When I run diffoscope, I see that diffoscope gets stuck while executing the xxd command while comparing one of the partitions. After a while (around 1 hour), the process gets killed.
Attaching the debug logs here:
adithya@debian12:~/work/diffoscope/diffoscope$ bin/diffoscope --debug --html output.html ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 16:34:38 D: diffoscope.main: Starting diffoscope 271
2024-07-19 16:34:38 D: diffoscope.presenters.formats: Will generate the following presenter formats: html
2024-07-19 16:34:38 D: diffoscope.environ: Normalising locale, timezone, etc. PATH is /home/adithya/.pyenv/libexec:/home/adithya/.pyenv/plugins/python-build/bin:/home/adithya/.pyenv/plugins/pyenv-virtualenv/bin:/home/adithya/.pyenv/plugins/pyenv-update/bin:/home/adithya/.pyenv/plugins/pyenv-doctor/bin:/home/adithya/.pyenv/plugins/pyenv-virtualenv/shims:/home/adithya/.pyenv/shims:/home/adithya/.pyenv/bin:/usr/local/bin:/usr/bin:/bin:/usr/local/games:/usr/games:/sbin:/usr/sbin:/usr/local/sbin:/usr/lib/x86_64-linux-gnu
2024-07-19 11:04:38 D: diffoscope.main: Starting comparison
2024-07-19 11:04:38 D: diffoscope.comparators.decompile: radare2 not found, disabling decompiler
2024-07-19 11:04:40 D: diffoscope.comparators: Loaded 92 comparator classes
2024-07-19 11:04:40 D: diffoscope.comparators.utils.command: Executing 7z l {}
2024-07-19 11:04:40 D: diffoscope.comparators.utils.specialize: Using sevenz.SevenZFile for ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 11:04:40 D: diffoscope.comparators.utils.command: Executing 7z l {}
2024-07-19 11:04:40 D: diffoscope.comparators.utils.specialize: Using sevenz.SevenZFile for ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 11:04:40 D: diffoscope.comparators.utils.compare: Comparing ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic (SevenZFile) and ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic (SevenZFile)
2024-07-19 11:04:40 D: diffoscope.comparators.utils.file: has_same_content(../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic, ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic)
2024-07-19 11:04:40 D: diffoscope.comparators.utils.command: Executing 7z l {}
2024-07-19 11:04:40 D: diffoscope.comparators.utils.command: Executing 7z l {}
2024-07-19 11:04:40 D: diffoscope.tempfiles: Created top-level temporary directory: /tmp/diffoscope_62iusjvk_image2 (free space: 282.09 GiB)
2024-07-19 11:04:40 D: diffoscope.diff: Running diff -a -U7 /tmp/diffoscope_62iusjvk_image2/tmpdk687cxt/fifo1 /tmp/diffoscope_62iusjvk_image2/tmpdk687cxt/fifo2
2024-07-19 11:04:40 D: diffoscope.diff: diff -a -U7 /tmp/diffoscope_62iusjvk_image2/tmpdk687cxt/fifo1 /tmp/diffoscope_62iusjvk_image2/tmpdk687cxt/fifo2: returncode 0, parsed True
2024-07-19 11:04:40 D: diffoscope.comparators.utils.file: Instantiating a sevenz.SevenZContainer for ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 11:04:40 D: diffoscope.comparators.utils.command: Calling external command: 7z e /home/adithya/work/diffoscope/cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 11:05:51 D: diffoscope.comparators.utils.file: Instantiating a sevenz.SevenZContainer for ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 11:05:51 D: diffoscope.comparators.utils.command: Calling external command: 7z e /home/adithya/work/diffoscope/cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking systemb.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpv7wkpufb_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking BOOT1.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmp4fgxp7ys_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking home.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpcuv9wlf7_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking BOOT0.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmp8mct291b_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking efi.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmppyqnwh8y_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking systema.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpm4d1pkaj_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking var.img from ../cip-core-images-v1.3/image1/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpdhastsew_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking systemb.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpo8ikc134_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking BOOT1.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmp4n2o2kct_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking home.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmp8zac50o1_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking BOOT0.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpe9jwzuki_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking efi.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpvdpf4ko__SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking systema.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmpxuu_ze31_SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.archive: Unpacking var.img from ../cip-core-images-v1.3/image2/cip-core-image-security-cip-core-bookworm-qemu-amd64.wic to /tmp/diffoscope_62iusjvk_image2/tmp57ziaf2__SevenZContainer
2024-07-19 11:07:20 D: diffoscope.comparators.utils.compare: Comparing systemb.img (ArchiveMember) and systemb.img (ArchiveMember)
2024-07-19 11:07:20 D: diffoscope.comparators.utils.specialize: systemb.img not identified by any comparator. Magic says: data
2024-07-19 11:07:20 D: diffoscope.comparators.utils.specialize: systemb.img not identified by any comparator. Magic says: data
2024-07-19 11:07:20 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/systemb.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/systemb.img)
2024-07-19 11:07:20 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/systemb.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/systemb.img
2024-07-19 11:07:39 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:07:39 D: diffoscope.comparators.directory: Excluding directory metadata for paths (systemb.img, systemb.img)
2024-07-19 11:07:39 D: diffoscope.comparators.utils.compare: Comparing BOOT1.img (ArchiveMember) and BOOT1.img (ArchiveMember)
2024-07-19 11:07:39 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for BOOT1.img
2024-07-19 11:07:39 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for BOOT1.img
2024-07-19 11:07:39 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/BOOT1.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/BOOT1.img)
2024-07-19 11:07:39 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/BOOT1.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/BOOT1.img
2024-07-19 11:07:41 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:07:41 D: diffoscope.comparators.directory: Excluding directory metadata for paths (BOOT1.img, BOOT1.img)
2024-07-19 11:07:41 D: diffoscope.comparators.utils.compare: Comparing home.img (ArchiveMember) and home.img (ArchiveMember)
2024-07-19 11:07:41 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for home.img
2024-07-19 11:07:41 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for home.img
2024-07-19 11:07:41 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/home.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/home.img)
2024-07-19 11:07:41 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/home.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/home.img
2024-07-19 11:08:06 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:08:06 D: diffoscope.comparators.directory: Excluding directory metadata for paths (home.img, home.img)
2024-07-19 11:08:06 D: diffoscope.comparators.utils.compare: Comparing BOOT0.img (ArchiveMember) and BOOT0.img (ArchiveMember)
2024-07-19 11:08:06 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for BOOT0.img
2024-07-19 11:08:06 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for BOOT0.img
2024-07-19 11:08:06 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/BOOT0.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/BOOT0.img)
2024-07-19 11:08:06 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/BOOT0.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/BOOT0.img
2024-07-19 11:08:08 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:08:08 D: diffoscope.comparators.directory: Excluding directory metadata for paths (BOOT0.img, BOOT0.img)
2024-07-19 11:08:08 D: diffoscope.comparators.utils.compare: Comparing efi.img (ArchiveMember) and efi.img (ArchiveMember)
2024-07-19 11:08:08 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for efi.img
2024-07-19 11:08:08 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for efi.img
2024-07-19 11:08:08 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/efi.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/efi.img)
2024-07-19 11:08:08 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/efi.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/efi.img
2024-07-19 11:08:08 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:08:08 D: diffoscope.comparators.directory: Excluding directory metadata for paths (efi.img, efi.img)
2024-07-19 11:08:08 D: diffoscope.comparators.utils.compare: Comparing systema.img (ArchiveMember) and systema.img (ArchiveMember)
2024-07-19 11:08:08 D: diffoscope.comparators.utils.specialize: Using squashfs.SquashfsFile for systema.img
2024-07-19 11:08:08 D: diffoscope.comparators.utils.specialize: Using squashfs.SquashfsFile for systema.img
2024-07-19 11:08:08 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/systema.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/systema.img)
2024-07-19 11:08:08 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/systema.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/systema.img
2024-07-19 11:08:28 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:08:28 D: diffoscope.comparators.directory: Excluding directory metadata for paths (systema.img, systema.img)
2024-07-19 11:08:28 D: diffoscope.comparators.utils.compare: Comparing var.img (ArchiveMember) and var.img (ArchiveMember)
2024-07-19 11:08:28 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for var.img
2024-07-19 11:08:28 D: diffoscope.comparators.utils.specialize: Using fsimage.FsImageFile for var.img
2024-07-19 11:08:28 D: diffoscope.comparators.utils.file: has_same_content(/tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/var.img, /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/var.img)
2024-07-19 11:08:28 D: diffoscope.comparators.utils.file: Executing: cmp -s /tmp/diffoscope_62iusjvk_image2/tmpjz2tjq96_7z/var.img /tmp/diffoscope_62iusjvk_image2/tmphse72d6e_7z/var.img
2024-07-19 11:09:06 D: diffoscope.comparators.utils.compare: has_same_content_as returned True; skipping further comparisons
2024-07-19 11:09:06 D: diffoscope.comparators.directory: Excluding directory metadata for paths (var.img, var.img)
2024-07-19 11:09:06 D: diffoscope.comparators.utils.command: Executing xxd {}
Killed
Is this a known issue or am I missing something here.