Modify the vendor hook accordingly.

The process discards all mails which can be associated to a package
as those are automatic emails that we already receive through
dispatch@. The remaining mails are forwarded to team members with
the "contact" keyword. The headers X-Distro-Tracker-Team and
X-Distro-Tracker-Keyword are set appropriately.

Closes: #891504

Fix excuses behaviour

See merge request qa/distro-tracker!29

A package may pass the policy even though it's blocked by a dependency,
in such a case, excuses should state it instead of saying nothing.

Implement get_resource_content(..., force_update=True). This will
request to make new HTTP request without any If-Modified-Since or
If-None-Matches so that we always new fresh data in all cases. It
will also ignore the existence of non-expired data in the cache.

Current versions need more configuration to work out of the box
and it has been a while that it was no longer working due to
those missing bits.

Use this sample configuration to generate the sample database. And add a
quick-setup.sh script that puts this configuration in place and that
downloads the ready-to-use sample database.

Gitlab CI job for creating a sample distro-tracker database

Closes #13

See merge request qa/distro-tracker!27

Fixes: #13

The code tries to parse the localpart as "team+foo" when the email is
hosted in the domain dedicated to distro-tracker. But it failed badly
when the maintainer email did not contain any plus sign.

This commit fixes the following exception:

Traceback (most recent call last):
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/tests/tests_retrieve_data.py", line 1185, in test_with_unqualified_team_email
    self.run_task()
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/tests/tests_retrieve_data.py", line 1109, in run_task
    self.task.execute()
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/retrieve_data.py", line 1094, in execute
    self.add_package_to_maintainer_teams(package, maintainer)
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/retrieve_data.py", line 1060, in add_package_to_maintainer_teams
    service, slug = localpart.split('+', 1)
ValueError: not enough values to unpack (expected 2, got 1)

The preferred workflow has largely changed with the switch to
salsa.debian.org. Also document new expectations concerning git
commit notices and sorting of import statements.

We're now (also) using the bug tracker on salsa and the documentation
is now officially hosted on https://qa.pages.debian.net/distro-tracker/

It's not clear how this happened but it did happen once at least:

File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in run
  517.                     task.execute()
File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in wrapper
  544.             func(self)
File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in execute
  942.                 versions.value = self._extract_versions_for_package(package)
File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in _extract_versions_for_package
  919.             'default_pool_url': package_name.main_entry.directory_url,

Exception Type: AttributeError
Exception Value: 'NoneType' object has no attribute 'directory_url'

File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in run
  517.                     task.execute()
File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in wrapper
  544.             func(self)
File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in execute
  999.                 binaries.value = self._get_all_binaries(package)
File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in _get_all_binaries
  969.         repository = package.main_entry.repository

Exception Type: AttributeError
Exception Value: 'NoneType' object has no attribute 'repository'

Without this fix we would get the following exception:

Traceback (most recent call last):
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/dispatch.py", line 412, in handle_bounces
    bounce_email, user_email = verp.decode(sent_to_address)
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/utils/verp.py", line 108, in decode
    left_part, encodedrdomain = left_part.rsplit('=', 1)
ValueError: not enough values to unpack (expected 2, got 1)

Without this fix, it would trigger this exception:

Traceback (most recent call last):
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/tests/tests_control.py", line 570, in test_ensure_no_failure_with_utf8_message_id
    self.control_process()
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/tests/tests_control.py", line 62, in control_process
    control.process(self.message)
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/control/__init__.py", line 187, in process
    send_plain_text_warning(msg, logdata)
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/control/__init__.py", line 91, in send_plain_text_warning
    recipient_email=logdata['from'])
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/control/__init__.py", line 54, in send_response
    message_id = unfold_header(original_message.get('Message-ID', ''))
  File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/utils/email_messages.py", line 253, in unfold_header
    return re.sub(r'\r?\n(\s)', r'\1', header, 0, re.MULTILINE)
  File "/usr/lib/python3.5/re.py", line 182, in sub
    return _compile(pattern, flags).sub(repl, string, count)
TypeError: expected string or bytes-like object

Add title tag description on some graphics mouseover

See merge request qa/distro-tracker!25

BugsPanel: show bug statistics even when the bug count is zero

See merge request qa/distro-tracker!23

Closes: #758768

Closes: #814491

Closes: #787662

All the modified settings default to lists in the current version of
Django. Special care must be taken to ensure that we duplicate the list
objects in the various namespaces otherwise we get a nice mess with a
single shared configuration combining all the operations of all the
settings that are loaded but not necessarily imported in the main settings
namespace.

We add __all__ attributes to some of the core modules to better control
what settings are actually exported.

This also fixes the regression introduced in
dac264a2 where the debug_toolbar
application suddenly started to be required for tests too.

That way we don't trigger any RuntimeWarning due to the lack of the
timezone data when we store the result in a DateTimeField.

UpdateAutoRemovalsStatsTask: import source package version in the ActionItem

See merge request qa/distro-tracker!22

Closes: #893670

UpdateExcusesTask: ignores all packages with a "/" in their name

Closes #9

See merge request qa/distro-tracker!21

It was done in the html parsing task, but got forgotten in the
refactoring

Rename PackageExtractedInfo to PackageData

Closes #7

See merge request qa/distro-tracker!20

This makes the code far more readable and lighter. In general, the
django-canonical name is too heavy and not relevant enough to be kept

PackageExtractedInfo is awfully long and doesn't reflect properly what
it is, as some info are not extracted at all from the package. Also,
PackageData is shorter while being more accurate. This will lighten a
little the code.