Signed-off-by: Holger Levsen <holger@layer-acht.org>

Should fix:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
    section.generate_output(output_directory, section_names, problem_list, web_host)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
    self.generate_html()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1587, in generate_html
    failures = dwke_get_failures(self._binary_db, self._problem_list)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1761, in dwke_get_failures
    add_cnt = make_kprs(logdict, kprdict, problem_list)
  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/dwke.py", line 252, in make_kprs
    logbody = lb.read()
  File "/usr/lib/python3.7/codecs.py", line 322, in decode
    (result, consumed) = self._buffer_decode(data, self.errors, final)
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xf1 in position 26184: invalid continuation byte

Move __pycache__ directory removal from debian/rules to the `clean`
Makefile target. The change should have no effect on the generated
packages; it just makes `make clean` more complete and debian/rules
simpler.

Instead of using a shell loop for passing the files to `rm`, also use
find's `-delete` option, avoiding any potential issues with weird
filenames.

Since the __pycache__ folders may contain .pyc files, make sure these
are all deleted before.

Piuparts does not have `--list` (without further arguments) or `--info`
options, hence running the autopkgtest currently fails. Replace the two
broken stubs by two working tests:

 1. run `piuparts --version`
 2. create a minimal dummy binary package and run piuparts on it

Using a single-letter package name for the dummy package ensures that
there is no package with the same name in the archive -- according to
policy 5.6.1, all official package names must be at least two characters
long.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 29, 2019 and Holger Levsen committed Dec 29, 2019

piuparts-report may encounter old md5cache files pickled by python2 in
text mode. Loading such files may fail if there are non-ascii
characters. The existing cache files involved in generating the reports
on piuparts.debian.org most likely use utf-8 encoding, so let's use that
instead of the default assumption 'ascii'.

Should fix:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
    section.generate_output(output_directory, section_names, problem_list, web_host)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
    self.generate_html()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1563, in generate_html
    self._md5cache['old'] = pickle.load(f)
UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 71: ordinal not in range(128)

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Jessica Clarke authored Dec 29, 2019 and Holger Levsen committed Dec 29, 2019

This regressed in 0b47bf88

, which
erroneously used the original trees rather than the modified copies.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

after confirming with diffoscope that the changes are sane

Signed-off-by: Holger Levsen <holger@layer-acht.org>

d/rules: set GOCACHE to a safely writable directory during build as the cache is mandatory with Go 1.12.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 28, 2019 and Holger Levsen committed Dec 28, 2019

Do not error out if the cache file is empty.

Fixes:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
    section.generate_output(output_directory, section_names, problem_list, web_host)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
    self.generate_html()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1563, in generate_html
    self._md5cache['old'] = pickle.load(f)
EOFError: Ran out of input

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 28, 2019 and Holger Levsen committed Dec 28, 2019

Fixes:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
    section.generate_output(output_directory, section_names, problem_list, web_host)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
    self.generate_html()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1600, in generate_html
    pickle.dump(self._md5cache['new'], f)
TypeError: write() argument must be str, not bytes

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 28, 2019 and Holger Levsen committed Dec 28, 2019

Fixes:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
    section.generate_output(output_directory, section_names, problem_list, web_host)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
    self.generate_html()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1580, in generate_html
    self.print_by_dir(self._output_directory, logs_by_dir)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 943, in print_by_dir
    desc_by_dir[vdir], vlist)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 930, in write_log_list_page
    "logrows": "".join(lines),
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 890, in _write_template_html
    write_template_html(filename, body, mapping, defer_if_unmodified=defer_if_unmodified, md5cache=self._md5cache)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 714, in write_template_html
    content_md5 = hashlib.md5(htmlpage.safe_substitute(mapping)).hexdigest()
TypeError: Unicode-objects must be encoded before hashing

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 28, 2019 and Holger Levsen committed Dec 28, 2019

In python3 dict().keys() returns a dict_keys() view and not a list as in
python2. Replace the now unavailable remove() by a set subtraction.

Fixes:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1911, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1851, in main
    section.generate_output(output_directory, section_names, problem_list, web_host)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1660, in generate_output
    self.generate_html()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1574, in generate_html
    self.cleanup_removed_packages(logs_by_dir)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1536, in cleanup_removed_packages
    vdirs.remove("reserved")
AttributeError: 'dict_keys' object has no attribute 'remove'

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 28, 2019 and Holger Levsen committed Dec 28, 2019

During the port to python3 piupartslib was changed to no longer
auto-import some of its submodules. Import the required submodules
directly where now needed.

Fixes:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1910, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 1849, in main
    section = Section(section_name, master_directory, doc_root, packagedb_cache=packagedb_cache)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 827, in __init__
    self._load_package_database(section, master_directory)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-report", line 859, in _load_package_database
    db = piupartslib.packagesdb.PackagesDB(prefix=sectiondir)
AttributeError: module 'piupartslib' has no attribute 'packagesdb'

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 27, 2019 and Holger Levsen committed Dec 28, 2019

No need to crash with a stacktrace when the slave is gone and the pipe
to it is broken. Just log this and exit 1.

Fixes:

Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 433, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 423, in main
    m = Master(sys.stdin, sys.stdout)
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 154, in __init__
    self._writeline("hello")
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-master-backend", line 111, in _writeline
    self._output.flush()
BrokenPipeError: [Errno 32] Broken pipe
Exception ignored in: <_io.TextIOWrapper name='<stdout>' mode='w' encoding='UTF-8'>
BrokenPipeError: [Errno 32] Broken pipe

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 26, 2019 and Holger Levsen committed Dec 27, 2019

When trying to decode a byte sequence into a string, it is possible that
the sequence ends in the middle of a multibyte character. In this case
the decoding fails.

This hack assumes that the input byte sequence is always encoded in
UTF-8, and that the original data stream never ends in the middle of a
multibyte character.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 26, 2019 and Holger Levsen committed Dec 27, 2019

It is better to avoid using names of builtin functions as variable
names.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 26, 2019 and Holger Levsen committed Dec 26, 2019

Fixes:

  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/packagesdb.py", line 260, in create
    if os.write(fd, contents) != len(contents):
TypeError: a bytes-like object is required, not 'str'

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 26, 2019 and Holger Levsen committed Dec 26, 2019

While one set of dict_keys() was already wrapped in a list() during the
initial porting work, the second one was still missing. Fixes:

  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/packagesdb.py", line 818, in get_best_package_state
    for state in self._good_states + list(self._propagate_waiting_state.keys()) + self._propagate_error_state.keys():
TypeError: can only concatenate list (not "dict_keys") to list

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Nis Martensen authored Dec 26, 2019 and Holger Levsen committed Dec 26, 2019

Arguments were accidentally introduced when porting the code to python3.
Remove them again. Fixes:

  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/dependencyparser.py", line 266, in _parse_version_dependency
    self._cursor.mynext(self._cursor)
TypeError: mynext() takes 1 positional argument but 2 were given

Signed-off-by: Holger Levsen <holger@layer-acht.org>

subprocess.Popen().stdout is a byte string in python3. We need to decode
it since we want a unicode string here.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

This reverts commit 030f1e97.

Signed-off-by: Holger Levsen <holger@layer-acht.org>

Mattia Rizzolo authored Sep 10, 2019 and Holger Levsen committed Sep 10, 2019

At this point it would probably be best to completely rewrite that thing
using StringIO and some other nicer way to do buffering, but this should
do the job for now.

should fix:

09:46:14 Running section testing2sid (precedence=2)
Traceback (most recent call last):
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-slave", line 1064, in <module>
    main()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-slave", line 1029, in main
    test_count += section.run(do_processing=(test_count == 0))
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-slave", line 575, in run
    processed = self._process()
  File "/srv/piuparts.debian.org/share/piuparts/piuparts-slave", line 667, in _process
    packagenames)
  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/packagesdb.py", line 183, in load_packages_urls
    self._read_file(stream, restrict_packages=restrict_packages)
  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/packagesdb.py", line 190, in _read_file
    headers = rfc822_like_header_parse(input)
  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/packagesdb.py", line 53, in rfc822_like_header_parse
    line = input.readline()
  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/__init__.py", line 60, in readline
    empty = not self._refill()
  File "/srv/piuparts.debian.org/lib/python3/dist-packages/piupartslib/__init__.py", line 52, in _refill
    self._buffer = self._buffer + chunk
TypeError: can only concatenate str (not "bytes") to str

Signed-off-by: Mattia Rizzolo <mattia@debian.org>
Signed-off-by: Holger Levsen <holger@layer-acht.org>

Mattia Rizzolo authored Sep 10, 2019 and Holger Levsen committed Sep 10, 2019

Signed-off-by: Mattia Rizzolo <mattia@debian.org>
Signed-off-by: Holger Levsen <holger@layer-acht.org>