Skip to content

Excessive memory use when comparing Linux Debian .dbg packages

This bug was originally reported by Ben Hutchings (@benh) in Debian bug #958271:

Package: diffoscope
Version: 141
Severity: normal

Comparing two versions of linux-image-5.5.0-trunk-amd64-dbg causes
diffoscope to use up about 13 GiB of VM, and then crash because it
still wants more:

$ TMPDIR=/var/tmp diffoscope --max-diff-block-lines-saved 1000 --text dummy.diffoscope linux-image-5.6.0-trunk-amd64-dbg_5.6.4-1~exp{1,2}_amd64.deb
Traceback (most recent call last):#######|  100%                             ETA:  0:00:00
  File "/usr/bin/diffoscope", line 36, in <module>
    main()
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 740, in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3/dist-packages/diffoscope/main.py", line 692, in run_diffoscope
    difference = compare_root_paths(path1, path2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 74, in compare_root_paths
    difference = compare_files(file1, file2)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 128, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 453, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 401, in _compare_using_details
    other.as_container, no_recurse=no_recurse
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 195, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 128, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 453, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 401, in _compare_using_details
    other.as_container, no_recurse=no_recurse
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 195, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 128, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 453, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 401, in _compare_using_details
    other.as_container, no_recurse=no_recurse
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/container.py", line 195, in compare_pair
    file1, file2, source=None, diff_content_only=no_recurse
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/compare.py", line 128, in compare_files
    return file1.compare(file2, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 453, in compare
    difference = self._compare_using_details(other, source)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/utils/file.py", line 371, in _compare_using_details
    details.extend(self.compare_details(other, source))
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/elf.py", line 629, in compare_details
    return _compare_elf_data(self.path, other.path)
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/elf.py", line 271, in _compare_elf_data
    for x in list(READELF_COMMANDS) + READELF_DEBUG_DUMP_COMMANDS
  File "/usr/lib/python3/dist-packages/diffoscope/comparators/elf.py", line 271, in <listcomp> 
    for x in list(READELF_COMMANDS) + READELF_DEBUG_DUMP_COMMANDS
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line 256, in from_command
    klass, path1, path2, *args, **kwargs
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line 288, in from_command_exc
    feeder1, feeder2, path1, path2, *args, **kwargs
  File "/usr/lib/python3/dist-packages/diffoscope/difference.py", line 213, in from_feeder
    unified_diff = diff(feeder1, feeder2)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 330, in diff
    return run_diff(fifo1_path, fifo2_path, fifo1.end_nl_q, fifo2.end_nl_q)
  File "/usr/lib/python3/dist-packages/diffoscope/tools.py", line 100, in tool_check
    return fn(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/diffoscope/diff.py", line 179, in run_diff
    p = subprocess.run(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
  File "/usr/lib/python3.7/subprocess.py", line 472, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib/python3.7/subprocess.py", line 775, in __init__
    restore_signals, start_new_session)
  File "/usr/lib/python3.7/subprocess.py", line 1453, in _execute_child
    restore_signals, start_new_session, preexec_fn)
OSError: [Errno 12] Cannot allocate memory
Edited by Chris Lamb
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information