Skip to content
Commits on Source (4)
Hide whitespace changes
Inline Side-by-side
_blog/posts/165.md
View file @ 61f37e28
......@@ -34,76 +34,21 @@ Packages reviewed and fixed, and bugs filed
tests.reproducible-builds.org development
-----------------------------------------
There were a number of changes to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org/), including:
* Mattia Rizzolo (FIXME):
* [rblib.models: take the \_status namedtuple out of the Status enum](https://salsa.debian.org/qa/jenkins.debian.net/commit/f874effd)
* [Move from jessie-backports to stretch-backports for pbuilder in the arm64 hosts](https://salsa.debian.org/qa/jenkins.debian.net/commit/42decf0c)
* [Use APT pinning instead of using `apt-get install -t`.](https://salsa.debian.org/qa/jenkins.debian.net/commit/67fb58ea)
* [update\_jdn: stop installing lintian on our hosts, we really have no use for it](https://salsa.debian.org/qa/jenkins.debian.net/commit/6f915a55)
* [simplify apt's sources.list of the arm64 hosts, disable restricted and multiverse](https://salsa.debian.org/qa/jenkins.debian.net/commit/bf8f6144)
* [reproducible: common.sh: error out when `psql` fails](https://salsa.debian.org/qa/jenkins.debian.net/commit/3cb06dd7)
* [reproducible: html\_breakages: check for dbdtxt and deal with the new timeout status](https://salsa.debian.org/qa/jenkins.debian.net/commit/953c5276)
* [reproducible: html\_breakages: remove pbuilder\_dep\_fail(), we had the depwait state for a long time](https://salsa.debian.org/qa/jenkins.debian.net/commit/3030488f)
* [\_html\_dashboard: always include depwait and E404 packages in the index\_suite\_arch, even when they are 0](https://salsa.debian.org/qa/jenkins.debian.net/commit/84da4810)
* [\_html\_dashboard: also include timeout packages in index\_suite\_arch pages](https://salsa.debian.org/qa/jenkins.debian.net/commit/22a799aa)
* [\_html\_dashboard: fix NFU icon](https://salsa.debian.org/qa/jenkins.debian.net/commit/49799666)
* [dashboard.sh: fix typo](https://salsa.debian.org/qa/jenkins.debian.net/commit/056eac54)
* [reproducible: introduce a new "timeout" status to represent build timeouts](https://salsa.debian.org/qa/jenkins.debian.net/commit/4395f641)
* build.sh:
* [take the automatic artifacts notifications back to #-changes](https://salsa.debian.org/qa/jenkins.debian.net/commit/0b123b98)
* [fix SQL error](https://salsa.debian.org/qa/jenkins.debian.net/commit/e6af59dd)
* [do an UPSERT insted of manually checking for presence before UPDATE/INSERT](https://salsa.debian.org/qa/jenkins.debian.net/commit/34570a21)
* [reinstate saving the builds in stats\_build](https://salsa.debian.org/qa/jenkins.debian.net/commit/c8b887fe)
* [keep the diffoscope errors notification in #-changes for a while to see how many there are](https://salsa.debian.org/qa/jenkins.debian.net/commit/20f72bf3)
* [differentiate the irc message between diffoscope errors and timeouts](https://salsa.debian.org/qa/jenkins.debian.net/commit/bae726d3)
* [split notification handling to a different function, and have it called only when running in master mode](https://salsa.debian.org/qa/jenkins.debian.net/commit/480cf7a2)
* [apply the 120m timeout directly to diffoscope instead of the wrapping schroot](https://salsa.debian.org/qa/jenkins.debian.net/commit/a903dd7b)
* [log this in the rbuild](https://salsa.debian.org/qa/jenkins.debian.net/commit/548a9b5c)
* [aid debug by quoting the variables](https://salsa.debian.org/qa/jenkins.debian.net/commit/3d4e5fa8)
* [drop old case when DURATION could have been undefined](https://salsa.debian.org/qa/jenkins.debian.net/commit/bb4adfb6)
* [remove some query\_db fallbacks from the sqlite time](https://salsa.debian.org/qa/jenkins.debian.net/commit/bdaff5af)
* [silence the greps](https://salsa.debian.org/qa/jenkins.debian.net/commit/b5b250cd)
* [fix depwait case](https://salsa.debian.org/qa/jenkins.debian.net/commit/aadebb76)
* [drop spurious set -x](https://salsa.debian.org/qa/jenkins.debian.net/commit/3d2d5ab7)
* [fix ftbfs check after the last refactor](https://salsa.debian.org/qa/jenkins.debian.net/commit/bbb991a3)
* [drop pipe and console logging during first\_build(): those msgs are lost anyway](https://salsa.debian.org/qa/jenkins.debian.net/commit/69544673)
* [copy build artifacts locally before further ascertain the build status](https://salsa.debian.org/qa/jenkins.debian.net/commit/63ced24c)
* [also consider the case of NOTIFY=''](https://salsa.debian.org/qa/jenkins.debian.net/commit/601a37d8)
* [move the post-build steps into build\_rebuild() instead of the main script body](https://salsa.debian.org/qa/jenkins.debian.net/commit/e7cde572)
* [merge filter\_changes\_files() into call\_diffoscope\_on\_changes\_files()](https://salsa.debian.org/qa/jenkins.debian.net/commit/256e1d88)
* [turn list of elif into a case](https://salsa.debian.org/qa/jenkins.debian.net/commit/d012e181)
* [don't send IRC notification for NOTIFY=0](https://salsa.debian.org/qa/jenkins.debian.net/commit/53e059fd)
* [print out the duration at the end of the cleanup](https://salsa.debian.org/qa/jenkins.debian.net/commit/84dffbff)
* [change the way ftbfs are handled](https://salsa.debian.org/qa/jenkins.debian.net/commit/fda61fc7)
* [let the cleanup be implied by the trap](https://salsa.debian.org/qa/jenkins.debian.net/commit/c94aea7c)
* [drop handling of the "version mismatch between master and build node", that is not possible anymore](https://salsa.debian.org/qa/jenkins.debian.net/commit/0470f48c)
* [worker.sh: also include the timestamp in the error log](https://salsa.debian.org/qa/jenkins.debian.net/commit/4d3f3c33)
* [reproducible: html\_indexes: fixup: drop another status mangling that is now unneeded](https://salsa.debian.org/qa/jenkins.debian.net/commit/975ea2f7)
* [reproducible: \_common.sh: drop support for 'not\_for\_us' and 'not for us' status names](https://salsa.debian.org/qa/jenkins.debian.net/commit/a349fad1)
* [rblib: remove the now unused get\_status\_icon() helper function](https://salsa.debian.org/qa/jenkins.debian.net/commit/751d4d7b)
* [reproducible: \_html\_notes: use the new Status class instead of get\_status\_icon()](https://salsa.debian.org/qa/jenkins.debian.net/commit/6f71ed16)
* [reproducible: \_html\_notes: list packages in all status in the issues pages](https://salsa.debian.org/qa/jenkins.debian.net/commit/f6dfd246)
* [reproducible: maintenance: reschedule stale builds, instead of dropping them](https://salsa.debian.org/qa/jenkins.debian.net/commit/99cdf49a)
* [reproducible: \_html\_pkg\_sets: Use the new Status object instead of get\_status\_icon()](https://salsa.debian.org/qa/jenkins.debian.net/commit/dfffafb2)
* [reproducible: \_html\_live\_status: also here, use the new Status enum instead of get\_status\_icon()](https://salsa.debian.org/qa/jenkins.debian.net/commit/6a18da2e)
* [reproducible: \_html\_indexes: start to use the new Status enum instead of passing around plain strings](https://salsa.debian.org/qa/jenkins.debian.net/commit/92b6fa82)
* [rblib/models: introduce a Status enum](https://salsa.debian.org/qa/jenkins.debian.net/commit/187636a0)
* [reproducible: \_html\_packages: use the new Status class instead of get\_status\_icon()](https://salsa.debian.org/qa/jenkins.debian.net/commit/3d923465)
* [reproducible: \_html\_packages: flake8 fixes](https://salsa.debian.org/qa/jenkins.debian.net/commit/01aaa463)
* [reproducible: \_html\_indexes: print the fully-compiled query with all the binded parameters](https://salsa.debian.org/qa/jenkins.debian.net/commit/9ceca98a)
* [reproducible: \_db\_maintenance: fixup after a broken rebase](https://salsa.debian.org/qa/jenkins.debian.net/commit/9a9923e9)
* [reproducible: rename status '404' to 'E404' to make a future planned change easier](https://salsa.debian.org/qa/jenkins.debian.net/commit/ca3416d7)
* [reproducible: normalize 'unreproducible' to 'FTBR' in computer-oriented strings](https://salsa.debian.org/qa/jenkins.debian.net/commit/ccc94fcf)
* [reproducible: rename the inconsistent 'not for us' and 'not\_for\_us' to NFU](https://salsa.debian.org/qa/jenkins.debian.net/commit/09012516)
* [reproducible: \_db\_maintenance: automatically bump the value in rb\_schema without having to specify the query at each update](https://salsa.debian.org/qa/jenkins.debian.net/commit/01ac367e)
* [reproducible: \_db\_maintenance: run the update queries in a single transaction](https://salsa.debian.org/qa/jenkins.debian.net/commit/cb6b2f90)
* [reproducible: \_db\_maintenance: fix typo in function name](https://salsa.debian.org/qa/jenkins.debian.net/commit/26739d90)
* [rblib: get\_status\_icon(): fail when looking up an unknown status](https://salsa.debian.org/qa/jenkins.debian.net/commit/457b5456)
There were a large number of changes to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org/), including:
* Chris Lamb:
* [Correct "which build timed out" grammar](https://salsa.debian.org/qa/jenkins.debian.net/commit/30a24438).
* Mattia Rizzolo:
* [Introduce a new "timeout" status to represent build timeouts](https://salsa.debian.org/qa/jenkins.debian.net/commit/4395f641), requiring multiple changes including [appling the timeout directly to diffoscope](https://salsa.debian.org/qa/jenkins.debian.net/commit/a903dd7b), adding a [new icon to the HTML dashboard](https://salsa.debian.org/qa/jenkins.debian.net/commit/49799666), [differentiating the IRC messages](https://salsa.debian.org/qa/jenkins.debian.net/commit/bae726d3), [listing packages on the issues page](https://salsa.debian.org/qa/jenkins.debian.net/commit/f6dfd246) and mulitple other fixups.
* Don't install [Lintian][https://lintian.debian.org] on our hosts. ([commit](https://salsa.debian.org/qa/jenkins.debian.net/commit/6f915a55))
* Use [APT pinning](https://wiki.debian.org/AptPreferences) instead of [using `apt-get install -t`](https://salsa.debian.org/qa/jenkins.debian.net/commit/67fb58ea).
* [Error out when `psql` fails](https://salsa.debian.org/qa/jenkins.debian.net/commit/3cb06dd7).
* Move the "automatic artifacts" notifications [back to the `#debian-devel-changes` IRC channel](https://salsa.debian.org/qa/jenkins.debian.net/commit/0b123b98).
* Perform [SQL `UPSERT` operation](https://wiki.postgresql.org/wiki/UPSERT) instead of [manually checking for presence before UPDATE/INSERT](https://salsa.debian.org/qa/jenkins.debian.net/commit/34570a21).
* [Reschedule stale builds, instead of dropping them](https://salsa.debian.org/qa/jenkins.debian.net/commit/99cdf49a).
* [Rename the `404` status to `E404` to make a future change easier](https://salsa.debian.org/qa/jenkins.debian.net/commit/ca3416d7).
Misc.
-----
......
bin/generate-draft
View file @ 61f37e28
......@@ -43,7 +43,6 @@ def main(*args):
return 0
def log(msg, *args, **kwargs):
print("I: " + msg.format(*args, **kwargs), file=sys.stderr)
......@@ -114,7 +113,6 @@ def get_data(week, max_age=3600):
return data
def get_author(week_start, week_end):
return os.environ.get('DEBFULLNAME', 'FIXME')
......@@ -159,7 +157,8 @@ def bugs(week_start, week_end, extra="true"):
AND
{extra}
AND
CAST(arrival AS DATE) BETWEEN to_timestamp(@{week_start}) AND to_timestamp(@{week_end})
CAST(arrival AS DATE) BETWEEN
to_timestamp(@{week_start}) AND to_timestamp(@{week_end})
""".format(**{
'fields': ', '.join(fields),
'extra': extra,
......@@ -199,7 +198,8 @@ def get_uploads(week_start, week_end):
WHERE
source IN ({sources})
AND
CAST(date AS date) BETWEEN to_timestamp(@{week_start}) AND to_timestamp(@{week_end})
CAST(date AS date) BETWEEN
to_timestamp(@{week_start}) AND to_timestamp(@{week_end})
ORDER BY
date
""".format(**{
......@@ -250,7 +250,8 @@ def get_packages_yml(week_start, week_end):
def open_packages_yml(date):
return subprocess.Popen(
"git show $(git rev-list -n1 --until @{0} origin/master):packages.yml".format(date),
"git show $(git rev-list -n1 --until @{0} "
"origin/master):packages.yml".format(date),
shell=True,
cwd=sibling_repo_gitdir('reproducible-notes'),
stdout=subprocess.PIPE).stdout
......@@ -290,6 +291,7 @@ def commits(week_start, week_end, project, path='.'):
'--all',
'--',
path,
':(exclude,glob)_blog/posts/*.md',
), cwd=git_dir).decode('utf-8')
result = collections.defaultdict(list)
......