PEP-440 invalid versions in Debian packages

Filing a ticket here, because it needs triage decision:

We got a very rough bug report on debian-project about PEP-440 invalid versions breaking the next pip version on Buster machines.

The background here is that setuptools since 66 has required PEP-440 valid versions for all packages installed on a system. Pip makes a noise about this since 23.3 in preparation for completely rejecting them in pip 24.

These haven't been common in PyPI in many years, but in Debian we've had some where Debian package versions got stuffed into Python module versions.

Of course on venerable Debian LTS releases, we don't care too much about what upstream projects are doing in their latest versions. But setuptools and pip are rather privileged here. The upstream of both strongly recommends users run the latest versions at all times. It's often required to upgrade setuptools, to build packages from source.

Usually it's fair to assume that all of this happens in a virtualenv, where it's isolated from system packages with weird versions, but there's no guarantees of that. Users can create --system-site-packages virtualenvs and can upgrade in /usr/local.

I did some research to see how badly Debian LTS/ELTS is affected:

Debian 10 LTS (buster):

Invalid Shredder 2.8.0.Maidenly.Moose
Invalid dcos SNAPSHOT
Invalid drslib 0.3.1p3
Invalid duecredit 0.6.4.debian1
Invalid epoptes 1.0.1_2
Invalid hy unknown
Invalid multicorn _VERSION_
Invalid nagstamon 3.2.1.debianbuster_sid
Invalid reportbug 7.5.3_deb10u1
Invalid requestbuilder devel
Invalid targetcli_fb 2.1.fb48
Invalid twms 0.06y
Invalid vmdb2 0.13.2_git

Debian 9 ELTS (stretch):

Invalid dcos SNAPSHOT
Invalid duecredit 0.6.0.debian1
Invalid lvm 2.02.168_2_._2016_11_30_
Invalid mpld3 0.3git
Invalid multicorn _VERSION_
Invalid openmolar 1.0.15_gd81f9e5
Invalid pyvmomi 5.5.0_2014.1.1
Invalid requestbuilder devel
Invalid targetcli_fb 2.1.fb43
Invalid uucp_lmtp 0.20130117_nmu1
Invalid woo 1.0_dfsg1_1_b4

Debian 8 ELTS (jessie):

Not a concern, setuptools >= 66 won't install on Python 3.4.

script I used to generate this

import subprocess
import pkg_resources
from pkg_resources._vendor.packaging.version import InvalidVersion, Version

output = subprocess.check_output(('apt-file', 'search', '.egg-info')).decode('utf-8')
pkgs = set()
broken = set()
for line in output.splitlines():
    container, _, eggpath = line.strip().partition(': ')
    if line.endswith('.egg-info'):
        eggname = eggpath.rsplit('/', 1)[1]
    else:
        eggpath = eggpath.rsplit('/', 1)[0]
        eggname = eggpath.rsplit('/', 1)[1]
    assert eggname.endswith('.egg-info')
    if not eggpath.startswith('/usr/lib/python3/dist-packages/'):
        continue
    name = eggname.rsplit('.', 1)[0]
    try:
        pkg, ver = name.split('-')
    except ValueError:
        if eggpath not in broken:
            broken.add(eggpath)
            print("Check manually", eggpath)
    else:
        pkgs.add((pkg, ver))
for pkg, ver in sorted(pkgs):
    try:
        x = Version(ver)
    except InvalidVersion:
        print("Invalid", pkg, ver)

Edited Oct 19, 2023 by Stefano Rivera