...
 
Commits (13)
  • Raphaël Hertzog's avatar
    Do not fail on broken Message-ID fields containing UTF-8 characters · 74bc4412
    Raphaël Hertzog authored
    Without this fix, it would trigger this exception:
    
    Traceback (most recent call last):
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/tests/tests_control.py", line 570, in test_ensure_no_failure_with_utf8_message_id
        self.control_process()
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/tests/tests_control.py", line 62, in control_process
        control.process(self.message)
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/control/__init__.py", line 187, in process
        send_plain_text_warning(msg, logdata)
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/control/__init__.py", line 91, in send_plain_text_warning
        recipient_email=logdata['from'])
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/control/__init__.py", line 54, in send_response
        message_id = unfold_header(original_message.get('Message-ID', ''))
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/utils/email_messages.py", line 253, in unfold_header
        return re.sub(r'\r?\n(\s)', r'\1', header, 0, re.MULTILINE)
      File "/usr/lib/python3.5/re.py", line 182, in sub
        return _compile(pattern, flags).sub(repl, string, count)
    TypeError: expected string or bytes-like object
    74bc4412
  • Raphaël Hertzog's avatar
    mail: do not choke on emails sent to bounces@<domain> · 4fb6bc36
    Raphaël Hertzog authored
    Without this fix we would get the following exception:
    
    Traceback (most recent call last):
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/mail/dispatch.py", line 412, in handle_bounces
        bounce_email, user_email = verp.decode(sent_to_address)
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/utils/verp.py", line 108, in decode
        left_part, encodedrdomain = left_part.rsplit('=', 1)
    ValueError: not enough values to unpack (expected 2, got 1)
    4fb6bc36
  • Raphaël Hertzog's avatar
    core: handle corner-case where we no longer have any package entry · 9ab656fa
    Raphaël Hertzog authored
    It's not clear how this happened but it did happen once at least:
    
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in run
      517.                     task.execute()
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in wrapper
      544.             func(self)
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in execute
      942.                 versions.value = self._extract_versions_for_package(package)
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in _extract_versions_for_package
      919.             'default_pool_url': package_name.main_entry.directory_url,
    
    Exception Type: AttributeError
    Exception Value: 'NoneType' object has no attribute 'directory_url'
    
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in run
      517.                     task.execute()
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/tasks.py" in wrapper
      544.             func(self)
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in execute
      999.                 binaries.value = self._get_all_binaries(package)
    File "/srv/tracker.debian.org/distro-tracker/distro_tracker/core/retrieve_data.py" in _get_all_binaries
      969.         repository = package.main_entry.repository
    
    Exception Type: AttributeError
    Exception Value: 'NoneType' object has no attribute 'repository'
    9ab656fa
  • Raphaël Hertzog's avatar
    Update documentation and bugs URLs · f2520736
    Raphaël Hertzog authored
    We're now (also) using the bug tracker on salsa and the documentation
    is now officially hosted on https://qa.pages.debian.net/distro-tracker/
    f2520736
  • Raphaël Hertzog's avatar
    docs: update the Contributing page · e778f8b6
    Raphaël Hertzog authored
    The preferred workflow has largely changed with the switch to
    salsa.debian.org. Also document new expectations concerning git
    commit notices and sorting of import statements.
    e778f8b6
  • Raphaël Hertzog's avatar
    Do not choke on maintainer emails without a plus sign · bb654041
    Raphaël Hertzog authored
    The code tries to parse the localpart as "team+foo" when the email is
    hosted in the domain dedicated to distro-tracker. But it failed badly
    when the maintainer email did not contain any plus sign.
    
    This commit fixes the following exception:
    
    Traceback (most recent call last):
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/tests/tests_retrieve_data.py", line 1185, in test_with_unqualified_team_email
        self.run_task()
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/tests/tests_retrieve_data.py", line 1109, in run_task
        self.task.execute()
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/retrieve_data.py", line 1094, in execute
        self.add_package_to_maintainer_teams(package, maintainer)
      File "/home/rhertzog/deb/core/distro-tracker/distro_tracker/core/retrieve_data.py", line 1060, in add_package_to_maintainer_teams
        service, slug = localpart.split('+', 1)
    ValueError: not enough values to unpack (expected 2, got 1)
    bb654041
  • Raphaël Hertzog's avatar
    21b4ae96
  • Chirath R's avatar
    f5e1f5ce
  • Chirath R's avatar
    1d9f04a0
  • Raphaël Hertzog's avatar
    Merge branch 'sample_database' into 'master' · c68d506a
    Raphaël Hertzog authored
    Gitlab CI job for creating a sample distro-tracker database
    
    Closes #13
    
    See merge request !27
    c68d506a
  • Raphaël Hertzog's avatar
    Create a sample development configuration for easy hacking · 3eab4abf
    Raphaël Hertzog authored
    Use this sample configuration to generate the sample database. And add a
    quick-setup.sh script that puts this configuration in place and that
    downloads the ready-to-use sample database.
    3eab4abf
  • Raphaël Hertzog's avatar
    Fix configuration of the debug_toolbar Django application · cf73439b
    Raphaël Hertzog authored
    Current versions need more configuration to work out of the box
    and it has been a while that it was no longer working due to
    those missing bits.
    cf73439b
  • Raphaël Hertzog's avatar
    utils: new parameter to force HTTP request even with cached data · da243bc3
    Raphaël Hertzog authored
    Implement get_resource_content(..., force_update=True). This will
    request to make new HTTP request without any If-Modified-Since or
    If-None-Matches so that we always new fresh data in all cases. It
    will also ignore the existence of non-expired data in the cache.
    da243bc3
......@@ -4,7 +4,7 @@ before_script:
- apt-get update
# Runtime dependencies
- apt-get -y install python3-django/stretch-backports
- apt-get -y install python3-requests python3-django-jsonfield python3-django-captcha python3-debian python3-debianbts python3-apt python3-yaml python3-bs4 python3-pyinotify python3-gpg
- apt-get -y install python3-requests python3-django-jsonfield python3-django-captcha python3-debian python3-debianbts python3-apt python3-yaml python3-bs4 python3-pyinotify python3-gpg python3-django-debug-toolbar
# Test dependencies
- apt-get -y install python3-coverage python3-selenium chromium-driver xvfb python3-pip
# Dependencies to build docs
......@@ -14,6 +14,8 @@ flake8:
script:
- pip3 install --upgrade flake8 # Ensure we have the latest version
- python3 -m flake8 django_email_accounts/ functional_tests/ distro_tracker/
except:
- schedules
unit-tests:
script:
......@@ -26,6 +28,8 @@ unit-tests:
- .coverage
when: always
expire_in: 6h
except:
- schedules
functional-tests:
# We have to run the tests as user nobody as chromium will not run as root
......@@ -34,6 +38,8 @@ functional-tests:
- runuser -u nobody xvfb-run python3 ./manage.py test functional_tests
after_script:
- chown root data
except:
- schedules
pages:
stage: deploy
......@@ -49,3 +55,18 @@ pages:
- public
only:
- master
except:
- schedules
sample-database:
only:
- schedules
script:
- cp distro_tracker/project/settings/local.py.sample-debian-dev distro_tracker/project/settings/local.py
- python3 manage.py migrate
- python3 manage.py loaddata distro_tracker/core/fixtures/sample-database-repositories.xml
- python3 manage.py tracker_update_repositories
artifacts:
paths:
- data/distro-tracker.sqlite
expire_in: 1 weeks
......@@ -8,8 +8,8 @@ of packages).
## Documentation
The [documentation](https://tracker.debian.org/docs/) is updated once a
day and matches what is deployed on tracker.debian.org.
The [documentation](https://qa.pages.debian.net/distro-tracker/) always
matches what's in the git repository's master branch.
Otherwise you can generate the documentation yourself by doing `make html`
in the docs subdirectory of the distro-tracker git repository.
......@@ -19,7 +19,7 @@ in the docs subdirectory of the distro-tracker git repository.
### How to contribute
Have a look at the ["Contributing"
section](https://tracker.debian.org/docs/contributing.html) of the
section](https://qa.pages.debian.net/distro-tracker/contributing.html) of the
documentation.
### Contact information
......@@ -33,16 +33,20 @@ The lead developer is Raphaël Hertzog (buxy on IRC).
### Reporting bugs and vulnerabilities
We use the Debian bug tracker to manage bug reports (including security
issues, there's a "security" tag). If you have something to report, please
file a bug against the `tracker.debian.org` package. You can follow
[Debian's instructions](https://www.debian.org/Bugs/Reporting) to complete
this process. You should check the [list of open
We are using [GitLab's bug
tracker](https://salsa.debian.org/qa/distro-tracker/issues) to manage bug
reports. You should file new bugs there.
However we also use the Debian bug tracker with its `tracker.debian.org`
pseudo-package. You should thus check [its list of open
issues](https://bugs.debian.org/tracker.debian.org) before filing a new
bug to avoid duplicates.
bug to avoid duplicates. You can also have a look at all the [closed bug
reports](https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=1;package=tracker.debian.org)
too.
You can also have a look at all the [closed bug
reports](https://bugs.debian.org/cgi-bin/pkgreport.cgi?archive=1;package=tracker.debian.org) too.
Security issues should be reported to the bug tracker like other bugs.
If you believe that the issue is really sensitive, then you can
mail [Raphaël Hertzog](mailto:hertzog@debian.org) privately.
## Misc information
......
#!/bin/sh
set -e
if [ -e distro_tracker/project/settings/local.py ]; then
echo "ERROR: You already have a configuration file (distro_tracker/project/settings/local.py)" >&2
exit 1
fi
if [ -e data/distro-tracker.sqlite ]; then
echo "ERROR: You already have a database file (data/distro-tracker.sqlite)" >&2
exit 1
fi
if [ ! -e distro_tracker/project/settings/local.py.sample-debian-dev ]; then
echo "ERROR: are you at the root of the distro-tracker repository?"
echo "USAGE: ./bin/$0"
exit 1
fi
echo ">>> Ensuring we have the required packages"
packages="python3-django python3-requests python3-django-jsonfield python3-django-debug-toolbar python3-debian python3-debianbts python3-apt python3-gpg python3-yaml python3-bs4 python3-pyinotify python3-selenium chromium-driver"
if ! dpkg-query -W $packages >/dev/null; then
echo ">>> Installing the required packages with “sudo apt install”"
sudo apt install $packages
fi
version=$(dpkg-query -W -f'${Version}' python3-django)
if dpkg --compare-versions $version lt 1:1.11; then
echo "WARNING: you need python3-django >= 1:1.11"
echo "Trying to install it from stretch-backports"
sudo apt install python3-django/stretch-backports
fi
echo ">>> Installing a configuration file"
cp distro_tracker/project/settings/local.py.sample-debian-dev distro_tracker/project/settings/local.py
echo ">>> Downloading a pre-built sample database file"
# Note: when https://gitlab.com/gitlab-org/gitlab-ce/issues/45697 will be
# fixed, we should be able to use
# https://salsa.debian.org/qa/distro-tracker/-/jobs/artifacts/master/raw/data/distro-tracker.sqlite?job=sample-database
url=$(bin/sample-database-url)
if [ -n "$url" ]; then
wget "$url" -O data/distro-tracker.sqlite
else
echo "ERROR: unable to find sample database url (bin/sample-database-url returned nothing)"
exit 1
fi
#!/usr/bin/python3
import re
import sys
import requests
from bs4 import BeautifulSoup as soup
# NOTE: this script is required to work-around this bug:
# https://gitlab.com/gitlab-org/gitlab-ce/issues/45697
for i in range(1,5):
jobs_url = 'https://salsa.debian.org/qa/distro-tracker/-/jobs?page=%d' % i
r = requests.get(jobs_url)
r.raise_for_status()
s = soup(r.text, 'html.parser')
for td in s.find_all('td', string=re.compile('sample-database')):
if td.parent.find_all(class_='ci-success'):
download_url = td.parent.findChild('a', class_='btn-build')['href']
url = 'https://salsa.debian.org' + download_url.replace(
'/download', '/raw/data/distro-tracker.sqlite')
print(url)
sys.exit(0)
<?xml version="1.0" encoding="utf-8"?>
<django-objects version="1.0">
<object pk="1" model="core.repository">
<field type="CharField" name="name">Debian Unstable</field>
<field type="CharField" name="shorthand">unstable</field>
<field type="CharField" name="uri">http://deb.debian.org/debian</field>
<field type="CharField" name="public_uri">http://deb.debian.org/debian</field>
<field type="CharField" name="suite">unstable</field>
<field type="CharField" name="codename">sid</field>
<field type="TextField" name="components">main contrib non-free</field>
<field type="BooleanField" name="default">True</field>
<field type="BooleanField" name="optional">False</field>
<field type="BooleanField" name="binary">True</field>
<field type="BooleanField" name="source">True</field>
<field type="IntegerField" name="position">1</field>
<field to="core.architecture" name="architectures" rel="ManyToManyRel"><object pk="1"></object></field>
</object>
<object pk="2" model="core.repository">
<field type="CharField" name="name">Debian Experimental</field>
<field type="CharField" name="shorthand">exp</field>
<field type="CharField" name="uri">http://deb.debian.org/debian</field>
<field type="CharField" name="public_uri">http://deb.debian.org/debian</field>
<field type="CharField" name="suite">experimental</field>
<field type="CharField" name="codename">experimental</field>
<field type="TextField" name="components">main contrib non-free</field>
<field type="BooleanField" name="default">False</field>
<field type="BooleanField" name="optional">False</field>
<field type="BooleanField" name="binary">False</field>
<field type="BooleanField" name="source">True</field>
<field type="IntegerField" name="position">2</field>
<field to="core.architecture" name="architectures" rel="ManyToManyRel"><object pk="1"></object><object pk="5"></object></field>
</object>
</django-objects>
......@@ -938,9 +938,12 @@ class UpdateVersionInformation(PackageUpdateTask):
},
'version': entry.source_package.version,
})
default_pool_url = None
if package_name.main_entry:
default_pool_url = package_name.main_entry.directory_url
versions = {
'version_list': version_list,
'default_pool_url': package_name.main_entry.directory_url,
'default_pool_url': default_pool_url,
}
return versions
......@@ -990,6 +993,8 @@ class UpdateSourceToBinariesInformation(PackageUpdateTask):
Returns a list representing binary packages linked to the given
source package.
"""
if not package.main_entry:
return []
repository = package.main_entry.repository
return [
{
......@@ -1052,9 +1057,9 @@ class UpdateTeamPackagesTask(BaseTask):
team.packages.add(package)
if maintainer.email.endswith("@" + settings.DISTRO_TRACKER_FQDN):
localpart, _ = maintainer.email.split('@', 1)
service, slug = localpart.split('+', 1)
if service != 'team':
if not localpart.startswith("team+"):
return
service, slug = localpart.split('+', 1)
team = get_or_none(Team, slug=slug)
if team:
team.packages.add(package)
......
<div id="footer-title">{% if DISTRO_TRACKER_VENDOR_URL %}<a href="{{ DISTRO_TRACKER_VENDOR_URL }}">{{ DISTRO_TRACKER_VENDOR_NAME }}</a>{% else %}{{ DISTRO_TRACKER_VENDOR_NAME }}{% endif %} Package Tracker</div>
<div id="footer-copyright">
<a href="https://deb.li/DTAuthors">Copyright</a> 2013-2018 The Distro Tracker Developers
</div>
</div>
<div id="footer-repository">
<a href="https://salsa.debian.org/qa/distro-tracker/issues">Bugs</a><a href="https://salsa.debian.org/qa/distro-tracker">Git Repository</a>
<a href="https://qa.pages.debian.net/distro-tracker/">Documentation</a>
<a href="https://salsa.debian.org/qa/distro-tracker/issues">Bugs</a>
<a href="https://salsa.debian.org/qa/distro-tracker">Git Repository</a>
<a href="https://qa.pages.debian.net/distro-tracker/contributing.html">Contributing</a>
</div>
......@@ -1161,6 +1161,29 @@ class UpdateTeamPackagesTaskTests(TestCase):
self.assertEqual(1, self.team.packages.count())
self.assertEqual(self.package.name, self.team.packages.all()[0].name)
def test_with_unqualified_team_email(self):
"""
Non-regression test for a failure to handle team@ (instead of
team+<slug>@).
"""
team_email = "team@{}".format(settings.DISTRO_TRACKER_FQDN)
self.package = create_source_package({
'name': 'team-package',
'version': '1.0.0',
'maintainer': {
'name': 'Maintainer',
'email': team_email,
},
})
self.repository.add_source_package(self.package)
self.add_mock_events('new-source-package-version-in-repository', {
'name': self.package.name,
'version': self.package.version,
'repository': self.repository.name,
})
self.run_task()
def test_new_package_version_team_has_package(self):
"""
Tests that there is no change to a team when a new package version
......
......@@ -1247,9 +1247,27 @@ class HttpCacheTest(SimpleTestCase):
content = get_resource_content(self.url, mock_cache)
# The update request has been made and returned new data
mock_cache.update.assert_called_once_with(self.url)
mock_cache.update.assert_called_once_with(self.url, force=False)
self.assertEqual(content, self.response_content)
def test_get_resource_content_utility_function_force_update(self):
"""
Tests the :func:`distro_tracker.core.utils.http.get_resource_content`
utility function when the force_update keyword argument is passed.
"""
mock_cache = self.get_mock_of_http_cache()
# In this test, the cache is expired, and hence update has
# to be called.
mock_cache.is_expired.return_value = False
mock_cache.update.return_value = (None, True)
get_resource_content(self.url, mock_cache, force_update=True)
# The update request has been made with the force=True
mock_cache.update.assert_called_once_with(self.url, force=True)
def test_get_resource_content_with_only_arg_and_cache_expired(self):
"""
Tests the :func:`distro_tracker.core.utils.http.get_resource_content`
......@@ -1269,7 +1287,7 @@ class HttpCacheTest(SimpleTestCase):
self.assertEqual(content, self.response_content)
# The function updated the cache
mock_cache.update.assert_called_once_with(self.url)
mock_cache.update.assert_called_once_with(self.url, force=False)
def test_get_resource_content_with_only_arg_and_cache_not_expired(self):
"""
......@@ -1308,7 +1326,7 @@ class HttpCacheTest(SimpleTestCase):
# Nothing returned because the update request resulted in no new data
self.assertIsNone(content)
mock_cache.update.assert_called_once_with(self.url)
mock_cache.update.assert_called_once_with(self.url, force=False)
@mock.patch('distro_tracker.core.utils.http.get_resource_content')
def test_get_resource_text(self, mock_get_resource_content):
......
......@@ -250,4 +250,4 @@ def unfold_header(header):
"""
if header is None:
return None
return re.sub(r'\r?\n(\s)', r'\1', header, 0, re.MULTILINE)
return re.sub(r'\r?\n(\s)', r'\1', str(header), 0, re.MULTILINE)
......@@ -186,7 +186,7 @@ class HttpCache(object):
def get_resource_content(url, cache=None, compression="auto",
only_if_updated=False):
only_if_updated=False, force_update=False):
"""
A helper function which returns the content of the resource found at the
given URL.
......@@ -211,6 +211,9 @@ def get_resource_content(url, cache=None, compression="auto",
:param only_if_updated: if set to `True` returns None when no update is
done. Otherwise, returns the content in any case.
:type only_if_updated: bool
:param force_update: if set to `True` do a new HTTP request even if we
non-expired data in the cache.
:type force_update: bool
:returns: The bytes representation of the resource found at the given url
:rtype: bytes
......@@ -220,9 +223,9 @@ def get_resource_content(url, cache=None, compression="auto",
cache = HttpCache(cache_directory_path)
updated = False
if cache.is_expired(url):
if force_update or cache.is_expired(url):
try:
_, updated = cache.update(url)
_, updated = cache.update(url, force=force_update)
except IOError:
# Ignore network errors but log them
import logging
......
......@@ -408,7 +408,12 @@ def handle_bounces(sent_to_address, message):
bounced email was returned.
:type sent_to_address: string
"""
bounce_email, user_email = verp.decode(sent_to_address)
try:
bounce_email, user_email = verp.decode(sent_to_address)
except ValueError:
logger.warning('bounces :: no VERP data to extract from %s',
sent_to_address)
return
match = re.match(r'^bounces\+(\d{8})@' + DISTRO_TRACKER_FQDN, bounce_email)
if not match:
logger.warning('bounces :: invalid address %s', bounce_email)
......
Return-path: <dfa@d29g14r.com.ggse85.cn>
Received: from mailly.debian.org ([2001:41b8:202:deb:6564:a62:52c3:4b72])
from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=mailly.debian.org,EMAIL=hostmaster@mailly.debian.org (verified)
by ticharich.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.89)
(envelope-from <dfa@d29g14r.com.ggse85.cn>)
id 1f7Dkt-0006l1-Ez
for control@tracker.debian.org; Sat, 14 Apr 2018 05:29:43 +0000
Received: from quantz.debian.org ([2001:41c8:1000:21::21:28])
from C=NA,ST=NA,L=Ankh Morpork,O=Debian SMTP,OU=Debian SMTP CA,CN=quantz.debian.org,EMAIL=hostmaster@quantz.debian.org (verified)
by mailly.debian.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.89)
(envelope-from <dfa@d29g14r.com.ggse85.cn>)
id 1f7Dkt-00024d-2x
for control@tracker.debian.org; Sat, 14 Apr 2018 05:29:43 +0000
Received: from [117.48.213.121] (helo=mail.cnd550.cn)
by quantz.debian.org with esmtp (Exim 4.89)
(envelope-from <dfa@d29g14r.com.ggse85.cn>)
id 1f7Dks-0007S1-4r
for pts@qa.debian.org; Sat, 14 Apr 2018 05:29:42 +0000
Received: by mail.cnd550.cn for <pts@qa.debian.org>; Sat, 14 Apr 2018 13:29:38 +0800 (envelope-from <dfa@d29g14r.com.ggse85.cn>)
Message-Id: <PP04C7RC-0U7N-6AKB-MXXN-B07E4UFBU2VG@d[分钟]g[秒]r.com.ggse85.cn>
Mime-Version: 1.0
From: =?GBK?B?ts7J0LLF?= <dfa@d29g14r.com.ggse85.cn>
To: pts@qa.debian.org
Subject: =?GBK?B?IEZ3OtejxPq/7L/swNbA1iAgICAgIDIwMTjE6jTUwjE0yNUxMzoyOToxNCAg?=
Date: Sat, 14 Apr 2018 13:29:14 +0800
Content-Type: text/html; charset=GBK
Content-Transfer-Encoding: quoted-printable
Delivered-To: pts@qa.debian.org
Delivered-To: control@tracker.debian.org
<HTML>
15 29 4
4
2018=C4=EA4=D4=C214=C8=D5
4
4
29
<img hj_bfc@sjzu=2Eedu=2Ecn 14 pts@qa=2Edebian=2Eorg luxury=2Ecom =
15 src =3D "http://uewmbv=2Ecn/ha=D2=C0=B0=A2=CE=DE=D0=C4s 29"> e3?1=
4/jas29=2Ejpeg" >
ahslwht@126=2Ecom
......@@ -39,7 +39,8 @@ from distro_tracker.core.models import (
from distro_tracker.core.utils import (
distro_tracker_render_to_string,
extract_email_address_from_header,
get_or_none
get_or_none,
message_from_bytes
)
from distro_tracker.mail import control
from distro_tracker.mail.control.commands import UNIQUE_COMMANDS
......@@ -560,6 +561,13 @@ class ControlBotBasic(EmailControlTest):
self.control_process()
def test_ensure_no_failure_with_utf8_message_id(self):
"""Non-regression test for a failure with an UTF-8 message id"""
with open(self.get_test_data_path('utf8-message-id.txt'), 'rb') as f:
self.message = message_from_bytes(f.read())
self.control_process()
def test_end_processing_on_signature_delimiter(self):
"""
Tests that processing commands halts when the signature delimiter is
......
......@@ -641,6 +641,12 @@ Diagnostic-Code: smtp; 550-5.7.1 [....] Our system has detected
self.create_bounce_address('unknown-user@domain.com'),
self.message)
def test_bounce_handler_with_base_email_address(self):
'''Mail for bounces@domain should not generate errors'''
dispatch.handle_bounces(
'bounces@' + DISTRO_TRACKER_FQDN,
self.message)
class BounceStatsTest(TestCase):
"""
......
......@@ -14,6 +14,7 @@ __all__ = [
'DEBUG',
'EMAIL_BACKEND',
'INSTALLED_APPS',
'INTERNAL_IPS',
'MIDDLEWARE',
'SITE_URL',
'TEMPLATES',
......@@ -49,4 +50,8 @@ INSTALLED_APPS = defaults.INSTALLED_APPS.copy()
INSTALLED_APPS.append('debug_toolbar')
MIDDLEWARE = defaults.MIDDLEWARE.copy()
MIDDLEWARE.append('debug_toolbar.middleware.DebugToolbarMiddleware')
MIDDLEWARE.insert(0, 'debug_toolbar.middleware.DebugToolbarMiddleware')
INTERNAL_IPS = [
'127.0.0.1',
]
......@@ -12,9 +12,8 @@ from . imports defaults
from .selected import *
# If you want to run a development setup close to what's running on
# tracker.debian.org, then uncomment the next two lines.
# from .debian import *
# from .db_sqlite import DATABASES
# tracker.debian.org, use the local.py.sample-debian-dev file next to this
# one.
## Add your custom settings here
......@@ -44,6 +43,9 @@ INSTALLED_APPS.extend([
# Extract common files from the source package
# 'distro_tracker.extract_source_files',
# Features for derivatives
# 'distro_tracker.derivative',
# Must be added if you set DJANGO_EMAIL_ACCOUNTS_USE_CAPTCHA to True
# 'captcha',
])
"""Site-specific settings
Rename this file to local.py to have a development setup with a configuration
close to what's running on tracker.debian.org
"""
# Load the selected configuration (selected.py is a symlink to preferred config)
from .development import *
from .debian import *
from .db_sqlite import DATABASES
# Disable some production settings
ALLOWED_HOSTS = []
try:
# Drop distro_tracker.extract_source_files as it takes way too much time
# and network bandwith for running the task
INSTALLED_APPS.remove('distro_tracker.extract_source_files')
except ValueError:
pass
# Re-inject development settings lost by .debian import
INSTALLED_APPS.append('debug_toolbar')
MIDDLEWARE.insert(0, 'debug_toolbar.middleware.DebugToolbarMiddleware')
# === Add your custom settings here ===
# If you don't use the packaged version of Distro Tracker, put a random secret
# key here. DO NOT USE THE EXAMPLE KEY GIVEN BELOW.
# SECRET_KEY = 'etu2#5lv=!0(g9l31mw=cpwhioy!egg60lb5o3_67d83#(wu-u'
# DJANGO_EMAIL_ACCOUNTS_USE_CAPTCHA = True
# DISTRO_TRACKER_FQDN = "tracker.debian.org"
# DISTRO_TRACKER_VENDOR_NAME = "Debian"
# DISTRO_TRACKER_VENDOR_URL = "http://www.debian.org"
# DISTRO_TRACKER_VENDOR_RULES = "distro_tracker.vendor.debian.rules"
# DISTRO_TRACKER_CONTACT_EMAIL = 'owner@' + DISTRO_TRACKER_FQDN
# You can enable supplementary Django applications here, refer to the
# documation for details about what they do
INSTALLED_APPS.extend([
# Can be useful to create visualizations of the models
# 'django_extensions',
# Generate news when packages are uploaded/removed/migrated
# 'distro_tracker.auto_news',
# Extract common files from the source package
# 'distro_tracker.extract_source_files',
# Features for derivatives
# 'distro_tracker.derivative',
# Must be added if you set DJANGO_EMAIL_ACCOUNTS_USE_CAPTCHA to True
# 'captcha',
])
......@@ -263,9 +263,11 @@ if settings.DJANGO_EMAIL_ACCOUNTS_USE_CAPTCHA:
if settings.DEBUG:
import django.views.static
import debug_toolbar
urlpatterns = [
url(r'^media/(?P<path>.*)$', django.views.static.serve,
{'document_root': settings.MEDIA_ROOT}),
url(r'^static/(?P<path>.*)$', django.views.static.serve,
{'document_root': settings.STATIC_ROOT}),
url(r'^__debug__/', include(debug_toolbar.urls)),
] + urlpatterns
......@@ -8,7 +8,7 @@ but should you find some features that are lacking or some other problems,
then please file bug reports against the
<a href="https://bugs.debian.org/tracker.debian.org">tracker.debian.org</a>
pseudo-package. Also don't hesitate to help us by
<a href="https://tracker.debian.org/docs/contributing.html">contributing
<a href="https://qa.pages.debian.net/distro-tracker/contributing.html">contributing
patches</a> for those issues.</p>
Not convinced? Go back to the old <a
href="https://packages.qa.debian.org/{{ package }}">package tracking
......
......@@ -11,45 +11,78 @@ The recommended way to send feedback is to write to the Debian Quality
Assurance mailing list <debian-qa@lists.debian.org>. You can also reach us
using IRC on the #debian-qa channel at irc.debian.org.
You can also report bugs against the `tracker.debian.org pseudo-package
<https://bugs.debian.org/cgi-bin/pkgreport.cgi?pkg=tracker.debian.org>`_, to do so
please follow `the usual instructions
<https://www.debian.org/Bugs/Reporting>`_.
When you report a bug, ensure you include detailed steps to reproduce it
and any details that might be helpful in troubleshooting.
You can also `report bugs <https://salsa.debian.org/qa/distro-tracker/issues>`_
in GitLab's interface. When you report a bug, ensure you include detailed
steps to reproduce it and any details that might be helpful in
troubleshooting.
If you are proposing a feature, please explain in detail how it would work,
and keep the scope as narrow as possible, to make it easier to implement.
If you do not know where to start, we have `a list of tasks suitable for
newcomers
<https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=tracker.debian.org;tag=newcomer>`_,
mentors will review your changes with special care when you try to tackle
those.
If you do not know where to start, we have tasks suitable for
newcomers:
* `in Debian's bug tracker <https://bugs.debian.org/cgi-bin/pkgreport.cgi?dist=unstable;package=tracker.debian.org;tag=newcomer>`_
* `in GitLab's bug tracker <https://salsa.debian.org/qa/distro-tracker/issues?label_name%5B%5D=newcomer>`_
There are mentors willing to review your changes with special care when
you try to tackle those.
Please remember this is a volunteer-driven project, and that contributions are
welcome.
Please remember that this is a volunteer-driven project, and that
contributions are welcome.
Contribute
----------
Ready to contribute? Here's how to set up `distro-tracker` for local
Ready to contribute? Here's how to set up distro-tracker for local
development:
Usual workflow
~~~~~~~~~~~~~~
1. Clone distro-tracker locally::
1. Create a guest account on `Salsa <https://salsa.debian.org>`_ (a GitLab
instance run by the Debian project) by visiting this page:
https://signup.salsa.debian.org
Follow all the steps to confirm your email, fill your profile,
`setup your SSH keys
<https://salsa.debian.org/help/gitlab-basics/create-your-ssh-keys.md>`_.
You might want to have a look at `Salsa's
documentation <https://wiki.debian.org/Salsa/Doc>`_ and `GitLab's
documentation <https://salsa.debian.org/help>`_ if you have doubts
about something.
Note that Debian Developers can skip this step as they already have
an account on this service.
2. Visit the `project's page <https://salsa.debian.org/qa/distro-tracker>`_
and fork distro-tracker in your own account. See `GitLab's
help <https://salsa.debian.org/help/gitlab-basics/fork-project.md>`_.
3. Clone distro-tracker locally::
$ git clone git@salsa.debian.org:your-account-guest/distro-tracker.git
Note that ``your-account-guest`` should be replaced by your Salsa's username.
If you want to clone the project without creating any account then
use this command::
$ git clone https://salsa.debian.org/qa/distro-tracker.git
Note that you can also browse the sources at
https://salsa.debian.org/qa/distro-tracker
4. For a quick startup, run this command::
$ bin/quick-setup.sh
It will install required packages with apt, put a sample
configuration file in place and download a pre-built database file to
save you some setup time.
2. Follow the steps in the chapter :ref:`setting-up`.
If you have more time and want to learn more about the configuration
of distro tracker, follow the steps in the sections :ref:`setting-up`
and :ref:`repositories`.
3. Start a local test server::
5. Start a local test server::
$ ./manage.py runserver
[...]
......@@ -58,17 +91,13 @@ Usual workflow
Visit the URL returned to have access to the test website.
4. Configure the package repositories as explained in
:ref:`repositories`. With your test server, the URL of
the admin web interface is http://127.0.0.1:8000/admin/.
5. Switch to a new branch::
6. Switch to a new branch::
$ git checkout -b name-of-your-bugfix-or-feature
6. Develop your new feature, ideally following the rules of :ref:`tdd`.
7. Develop your new feature, ideally following the rules of :ref:`tdd`.
7. When you're done, check that all tests are succeeding in all
8. When you're done, check that all tests are succeeding in all
supported platforms::
$ tox
......@@ -76,8 +105,25 @@ Usual workflow
This basically runs “./manage.py test” with multiple versions
of Django and Python.
8. Push your changes on a public repository or send them by
email to the Debian Quality Assurance team::
9. Push your branch to your repository::
$ git push -u origin name-of-your-bugfix-or-feature
10. Submit us your work, ideally by opening a `merge
request <https://salsa.debian.org/qa/distro-tracker/merge_requests/>`_.
You can do this easily by visiting the distro-tracker
project fork hosted in your own account (either through the “Branches”
page, or through the “Merge requests” page). See `GitLab's
help <https://salsa.debian.org/help/gitlab-basics/add-merge-request.md>`_
if needed.
Make sure to address any issue identified by the continuous
integration system, the result of its “pipeline” can be directly
seen in the merge request (and in the commits pushed in your own
repository).
If you don't have any Salsa account, you can generate patches and
send them by email to the Debian Quality Assurance team::
$ git format-patch -o . origin/master
$ mutt debian-qa@lists.debian.org -a *.patch
......@@ -133,17 +179,46 @@ Conventions
<http://legacy.python.org/dev/peps/pep-0008/>`_ with a few exceptions.
2. Functions are documented using doctrings with `Sphinx markup
<http://sphinx-doc.org/contents.html>`_.
<http://sphinx-doc.org/en/master/>`_.
3. Imports are sorted in 3 groups separated by one empty line: first the
Python standard modules, then the third-party modules and finally
the project modules. Each group is further split in two between
``import foo`` statements and ``from foo import bar`` statements.
Within each group entries are alphabetically sorted. The isort
command used to implement the initial formatting was ``isort -m 3 -l
80 -rc django_email_accounts/ distro_tracker/``.
Git commit notices
~~~~~~~~~~~~~~~~~~
Please invest some time to write good commit notices. Just like your code,
you write it once but it will be read many times by different persons
looking to understand why you made the change. So make it pleasant to
read.
The first line is the “summary” (or title) and describes briefly what the
commit changes. It's followed by an empty line and a long description. The
long description can be as long as you want and should explain why you
implemented the change seen in the commit.
The long description can also be used to close bugs by putting some
pseudo-fields at the end of the description:
* for a GitLab issue, use ``Fixes: #XX`` (this is a standard GitLab
feature)
* for a Debian bug, use ``Closes: #XXXXXX`` (this is implemented by a
`webhook <https://salsa.debian.org/salsa/webhook>`_)
Write access to the git repository
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Debian developers in the "qa" group have write access to the
repository and should use the following command to checkout
a git repository where they can push changes::
$ git clone ssh://<yourdebianlogin>@git.debian.org/git/qa/distro-tracker.git
`Project (and Debian QA group) members
<https://salsa.debian.org/qa/distro-tracker/project_members>`_ have write
access to the main git repository. They can thus clone the repository
with this URL::
Anyone with commit access can use topic branches in the
“people/`debianlogin`/” hierarchy.
$ git clone git@salsa.debian.org:qa/distro-tracker.git
From there they can push their changes directly. They are however free to
use a fork and request review anyway when they prefer.