test_libmix_differences broken on macOS
Hello and thank you for your work on diffoscope, it's an excellent tool!
I have been working on the packaging of diffoscope for Nix on macOS, and we are experiencing some test failures which we think might be upstream bugs.
You can get our full build log here: https://gist.github.com/lf-/2e7685146861f71795c66f45bd6d6cf7/raw/e5bc94b7588beb8c12bc92d6d882980d36dec0b2/gistfile0.txt
To reproduce exactly what I was experiencing but in a dev shell, first get access to an aarch64 Mac with Nix on it, then:
git clone https://github.com/lf-/nixpkgs -b jade/diffoscope-headrepro --depth 1
cd nixpkgs
nix --extra-experimental-features 'flakes nix-command' develop -f . diffoscope
(inside the resulting shell)
phases="${prePhases:-} unpackPhase patchPhase" genericBuild
phases="${preConfigurePhases:-} configurePhase ${preBuildPhases:-} buildPhase" genericBuild
(trigger the bugs, also python3 -m pytest)
phases=pytestCheckPhase genericBuildSummary of my diagnoses:
- test_trim_stderr_in_command - believed to be a sandbox related bug, not your fault
- test_item_rdb - fairly certain this is a bug in diffoscope test, possibly dependent on specific versions of R.
- test_libmix_differences - possibly us having a somehow oddball setup, but you should be aware.
test_libmix_differences fails since it gets some otool command instead of llvm-objdump. This might be due to the Nix tooling setup, but it seems to reveal a lack of resiliency somewhere along the chain.
bash-5.2$ python3 -m pytest -vv -k test_libmix_differences --pdb
====================================================================================================== test session starts ======================================================================================================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0 -- /nix/store/al81jk4wz596q8if684x6a9mqlv7vc3z-python3-3.11.7/bin/python3
cachedir: .pytest_cache
rootdir: /Users/jade/nixpkgs/source
configfile: pytest.ini
collected 722 items / 721 deselected / 1 selected
tests/comparators/test_elf.py::test_libmix_differences FAILED [100%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
libmix_differences = [<Difference file list -- file list []>, <Difference Mach-O.o -- Mach-O.o [<Difference x86_64 -- x86_64 [<Difference o...ection=.text {} -- objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {} []>]>]
@skip_unless_tools_exist("xxd")
@skip_unless_tools_exist("llvm-readobj", "llvm-objdump")
@skip_unless_tools_exist("readelf", "objdump")
@skip_unless_tool_is_at_least("readelf", readelf_version, "2.38.50")
@skip_if_binutils_does_not_support_x86()
def test_libmix_differences(libmix_differences):
assert len(libmix_differences) == 5
file_list, mach_o, x86_o, src_c, x_obj = libmix_differences
# Check order and basic identification
assert file_list.source1 == "file list"
x86_o = x86_o.details[0]
assert x86_o.source1.startswith("objdump ")
assert src_c.source1.endswith(".c")
mach_o = mach_o.details[0]
for diff in mach_o.details:
> assert diff.source1.startswith("llvm-objdump ")
E AssertionError: assert False
E + where False = <built-in method startswith of str object at 0x10b8f4150>('llvm-objdump ')
E + where <built-in method startswith of str object at 0x10b8f4150> = 'otool -arch x86_64 -v -V -s __TEXT __text {}'.startswith
E + where 'otool -arch x86_64 -v -V -s __TEXT __text {}' = <Difference otool -arch x86_64 -v -V -s __TEXT __text {} -- otool -arch x86_64 -v -V -s __TEXT __text {} []>.source1
tests/comparators/test_elf.py:191: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/jade/nixpkgs/source/tests/comparators/test_elf.py(191)test_libmix_differences()
-> assert diff.source1.startswith("llvm-objdump ")
(Pdb) p diff.source1
'otool -arch x86_64 -v -V -s __TEXT __text {}'
(Pdb) p libmix_differences
[<Difference file list -- file list []>, <Difference Mach-O.o -- Mach-O.o [<Difference x86_64 -- x86_64 [<Difference otool -arch x86_64 -v -V -s __TEXT __text {} -- otool -arch x86_64 -v -V -s __TEXT __text {} []>]>]>, <Difference return42_or_3_long_name.o -- return42_or_3_long_name.o [<Difference objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {} -- objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {} []>]>, <Difference return42_or_3_long_name.c -- return42_or_3_long_name.c []>, <Difference return42_or_3_long_name.obj -- return42_or_3_long_name.obj [<Difference objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {} -- objdump --line-numbers --disassemble --demangle --reloc --no-show-raw-insn --section=.text {} []>]>]
(Pdb) p mach_o
<Difference x86_64 -- x86_64 [<Difference otool -arch x86_64 -v -V -s __TEXT __text {} -- otool -arch x86_64 -v -V -s __TEXT __text {} []>]>
(Pdb)test_item_rdb appears to be that the underlying tool changed something about the code it is outputting. However, this does not seem to be changes in a "it broke" direction, so it appears strongly like a bug in the diffoscope test of not handling different versions of R (?). My naive interpretation is that the test is printing out internal attributes which were then changed in some version, and should not have been included in the first place.
I have attached a diff below between the "seen" and "expected" strings.
bash-5.2$ python3 -m pytest -vv -k test_item_rdb --pdb
======================================================== test session starts ========================================================
platform darwin -- Python 3.11.7, pytest-7.4.3, pluggy-1.3.0 -- /nix/store/al81jk4wz596q8if684x6a9mqlv7vc3z-python3-3.11.7/bin/python3
cachedir: .pytest_cache
rootdir: /Users/jade/nixpkgs/source
configfile: pytest.ini
collected 722 items / 721 deselected / 1 selected
tests/comparators/test_rdata.py::test_item_rdb FAILED [100%]
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> traceback >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
differences_rdb = [<Difference Rscript --vanilla - {} -- Rscript --vanilla - {} []>]
@skip_unless_tools_exist("Rscript")
@skip_unless_tool_is_at_least("Rscript", rscript_version, "4.2.0")
def test_item_rdb(differences_rdb):
assert differences_rdb[0].source1.startswith("Rscript")
> assert_diff(differences_rdb[0], "rdb_expected_diff")
tests/comparators/test_rdata.py:78:
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
difference = <Difference Rscript --vanilla - {} -- Rscript --vanilla - {} []>, filename = 'rdb_expected_diff'
def assert_diff(difference, filename):
# Assign seen and expected values to local variables to improve contextual
# information in failed tests.
seen = difference.unified_diff
expected = get_data(filename)
> assert seen == expected
E AssertionError
tests/utils/data.py:64: AssertionError
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> entering PDB >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>> PDB post_mortem (IO-capturing turned off) >>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>>
> /Users/jade/nixpkgs/source/tests/utils/data.py(64)assert_diff()
-> assert seen == expected$ diff -u seen.txt expected.txt
--- seen.txt 2024-01-17 21:25:19.101672454 +0000
+++ expected.txt 2024-01-17 21:25:34.717028832 +0000
@@ -10,25 +10,25 @@
+.__C__Biodetection (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("Biodetection", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__CD (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("CD", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__CountsBio (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("CountsBio", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__GCbias (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("GCbias", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__Output (S4) = new("classRepresentation", slots = list(results = structure("list", package = "methods"),
@@ -88,25 +88,25 @@
+ return(TRUE)
+ }
+ }, access = list(), className = structure("Output", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__PCA (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("PCA", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__Saturation (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("Saturation", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__lengthbias (S4) = new("classRepresentation", slots = list(dat = structure("list", package = "methods")),
+ contains = list(), virtual = FALSE, prototype = <S4 object of class NULL>,
+ validity = NULL, access = list(), className = structure("lengthbias", package = "NOISeq"),
-+ package = "NOISeq", subclasses = list(), versionKey = <pointer: 0x0>,
++ package = "NOISeq", subclasses = list(), versionKey = <pointer: (nil)>,
+ sealed = FALSE)
+
+.__C__myInfo (S4) = new("classRepresentation", slots = list(method = structure("character", package = "methods"),
@@ -133,7 +133,7 @@
+ what) <- slot(value, what)
+ from
+ }, simple = TRUE, by = character(0), dataPart = FALSE,
-+ distance = 1)), versionKey = <pointer: 0x0>, sealed = FALSE)
++ distance = 1)), versionKey = <pointer: (nil)>, sealed = FALSE)
.__NAMESPACE__. (environment) =
{Let me know if there is any specific thing you need to further diagnose these problems; I have access to the Nix community aarch64 Mac machine that I can run tests on.
Environment details:
- macOS 13.6.3 arm64
- diffoscope f1822463 (current HEAD as of this writing a few weeks ago)
- Nix 2.18.1 (although exact nix version is extremely unlikely to matter)