diffoscope fails on Fedora 35

Running diffoscope on a distribution with python-apt installed which is not Debian derivate gives the following exception:

[jelle@toolbox cockpit]$ diffoscope cockpit-bridge cockpit-bridge.old
Traceback (most recent call last):
  File "/usr/lib/python3.10/site-packages/diffoscope/main.py", line 751, in main
    sys.exit(run_diffoscope(parsed_args))
  File "/usr/lib/python3.10/site-packages/diffoscope/main.py", line 706, in run_diffoscope
    difference = compare_root_paths(path1, path2)
  File "/usr/lib/python3.10/site-packages/diffoscope/comparators/utils/compare.py", line 66, in compare_root_paths
    file1 = specialize(FilesystemFile(path1, container=container1))
  File "/usr/lib/python3.10/site-packages/diffoscope/comparators/utils/specialize.py", line 53, in specialize
    for cls in ComparatorManager().classes:
  File "/usr/lib/python3.10/site-packages/diffoscope/comparators/__init__.py", line 127, in __init__
    self.reload()
  File "/usr/lib/python3.10/site-packages/diffoscope/comparators/__init__.py", line 138, in reload
    mod = importlib.import_module(
  File "/usr/lib64/python3.10/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1006, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 688, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/usr/lib/python3.10/site-packages/diffoscope/comparators/debian.py", line 24, in <module>
    from debian.deb822 import Dsc, Deb822
  File "/usr/lib/python3.10/site-packages/debian/deb822.py", line 283, in <module>
    import debian.debian_support
  File "/usr/lib/python3.10/site-packages/debian/debian_support.py", line 46, in <module>
    apt_pkg.init()
apt_pkg.Error: W:Unable to read /etc/apt/apt.conf.d/ - DirectoryExists (2: No such file or directory), E:Unable to determine a suitable packaging system type

This should probably be handled by diffoscope as the upstream report isn't getting much attention it seems and other distributions might suffer the same issue.