Skip to content
Commits on Source (13)
Show whitespace changes
Inline Side-by-side
_blog/posts/185.md
View file @ 4a084554
--- ---
layout: blog layout: blog
week: 185 week: 185
published: 2018-11-13 13:56:58
--- ---
Here's what happened in the [Reproducible Builds](https://reproducible-builds.org) effort between Sunday November 4 and Saturday November 10 2018: Here's what happened in the [Reproducible Builds](https://reproducible-builds.org) effort between Sunday November 4 and Saturday November 10 2018:
...@@ -9,31 +10,25 @@ Here's what happened in the [Reproducible Builds](https://reproducible-builds.or ...@@ -9,31 +10,25 @@ Here's what happened in the [Reproducible Builds](https://reproducible-builds.or
[![]({{ "/images/logos/rb_joins_sfc.png#center" | prepend: site.baseurl }})](https://reproducible-builds.org/news/2018/11/08/reproducible-builds-joins-software-freedom-concervancy/) [![]({{ "/images/logos/rb_joins_sfc.png#center" | prepend: site.baseurl }})](https://reproducible-builds.org/news/2018/11/08/reproducible-builds-joins-software-freedom-concervancy/)
[Conservancy](https://sfconservancy.org/about/) not-for-profit organization based in New York that helps promote, develop and defend free software projects, providing important services for its member projects. [Conservancy](https://sfconservancy.org/about/) is a not-for-profit organisation that helps promote, develop and defend free software projects. We can now can take directed donations and the Conservancy can also provide projects us with basic legal services. The Reproducible Builds project is delighted and honoured to be associated with Conservancy's outreach work and other work of the project and look forward to a long and mutually beneficial relationship.
We can now can take directed donations and the Software Freedom Conservancy can also provide projects with basic legal services. The Reproducible Builds project is delighted and honoured to be associated with the outreach work and other work of the Conservancy project and look forward to a long and mutually beneficial relationship. * The month-long session of students from the [Application Security](http://bulletin.engineering.nyu.edu/preview_course_nopop.php?catoid=9&coid=23997) course at [New York University](https://www.nyu.edu/), cataloguing, submitting and merging reproduciblity bugs concluded this week. This year, students made 55 tags and issues for Debian and [Arch Linux](https://www.archlinux.org/) packages and sent 18 pull requests upstream of which 4 have been merged.
* The month-long session of students from the [Application Security](http://bulletin.engineering.nyu.edu/preview_course_nopop.php?catoid=9&coid=23997) course at [New York University](https://www.nyu.edu/), cataloguing, submitting and merging reproduciblity bugs concluded this week. This year, students: * Richard Parkins [posted a detailed message to our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2018-November/001251.html) on the topic of algorithms used for comparing binary files in a way that makes the result easily consumable by humans. Most binary file comparators just compare bytes and thus do not semantically detect deletions or insertions. This is relevant to our work on [diffoscope](https://diffoscope.org/). He linked to some [example code on GitHub](https://github.com/rparkins999/bindiff).
- Made 55 tags and issues for Debian and [Arch Linux](https://www.archlinux.org/) packages * There was further discussion on Debian bug [#869184](https://bugs.debian.org/869184) which relates to `dpkg` generating source uploads that include architecture in the name of the `.buildinfo` file (eg. `_amd64.buildinfo`). This week, Salvatore Bonaccorso reported that the [Debian Security Team](https://wiki.debian.org/Teams/Security) were [hit by this issue again](https://bugs.debian.org/869184#60).
- Sent 18 pull requests upstream
- ... of which 4 have been merged
* Richard Parkins [posted a detailed message to our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2018-November/001251.html) on the topic of algorithms used for comparing binary files in a way that makes the result easily consumable by humans. Most binary file comparators just compare bytes which don't detect deletions or insertions and is thus relevant to our work on [diffoscope](https://diffoscope.org/). He linked to some [example code on GitHub](https://github.com/rparkins999/bindiff).
* There was further discussion on Debian bug [#869184](https://bugs.debian.org/869184) which relates to `dpkg` generating source uploads including the build architecture in the name of the `.buildinfo` file can cause problems (eg. `_amd64.buildinfo`). This week, Salvatore Bonaccorso reported that the [Debian Security Team](https://wiki.debian.org/Teams/Security) were [hit by this issue again](https://bugs.debian.org/869184#60).
* On Tuesday 6th November, Chris Lamb [hosted a seminar and a lengthy Q&A session](http://talks.cam.ac.uk/talk/index/114232) at the William Gates Building at the University of Cambridge on reproducible builds as part of the [Computer Laboratory NetOS Group](https://www.cl.cam.ac.uk/research/srg/netos/). * On Tuesday 6th November, Chris Lamb [hosted a seminar and a lengthy Q&A session](http://talks.cam.ac.uk/talk/index/114232) at the William Gates Building at the University of Cambridge on reproducible builds as part of the [Computer Laboratory NetOS Group](https://www.cl.cam.ac.uk/research/srg/netos/).
* [Simon McVittie](http://smcv.pseudorandom.co.uk/) kindly [provided a patch](https://bugs.debian.org/901473#33) to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](tests.reproducible-builds.org) to vary whether we apply the "merged `/usr`" directory scheme between builds. This is where the `/{bin,sbin,lib}/` directories are symbolic links to `/usr/{bin,sbin,lib}/`. It was subsequently merged by Holger Levsen and resulted in some variations in (at least) [quilt](https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/quilt.html) and [systemd](https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/systemd.html). * [Simon McVittie](http://smcv.pseudorandom.co.uk/) kindly [provided a patch](https://bugs.debian.org/901473#33) to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](tests.reproducible-builds.org) to vary whether we apply the "merged `/usr`" directory scheme between builds. This is where the `/{bin,sbin,lib}/` directories are symbolic links to `/usr/{bin,sbin,lib}/`. It was subsequently merged by Holger Levsen and resulted in some variations in (at least) [quilt](https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/quilt.html) and [systemd](https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/systemd.html).
* Chris Lamb updated `strip-nondeterminism` (our tool to post-process files to remove known non-deterministic output) to [catch invalid ZIP "local" field lengths](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/e5f5008); we were previously inherently blindly trusting the value supplied in the ZIP file ([#803503](https://bugs.debian.org/803503)). In addition, he applied a patch from Emmanuel Bourg to [update the Javadoc handler to handle OpenJDK 11](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/f745484) ([#913132](https://bugs.debian.org/913132)). He then subsequently uploaded version `0.044-1` [to Debian unstable](https://tracker.debian.org/news/1001570/accepted-strip-nondeterminism-0044-1-source-all-into-unstable/). * Chris Lamb updated `strip-nondeterminism` (our tool to post-process files to remove known non-deterministic output) to [catch invalid ZIP "local" field lengths](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/e5f5008) — we were previously inherently blindly trusting the value supplied in the ZIP file ([#803503](https://bugs.debian.org/803503)). In addition, he applied a patch from Emmanuel Bourg to [update the Javadoc handler to handle OpenJDK 11](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/f745484) ([#913132](https://bugs.debian.org/913132)). He then subsequently uploaded version `0.044-1` [to Debian unstable](https://tracker.debian.org/news/1001570/accepted-strip-nondeterminism-0044-1-source-all-into-unstable/).
* Agustin Henze announced in a mail to the [`debian-devel`](https://lists.debian.org/debian-devel/) mailing list that [the new Debian CI pipeline](https://lists.debian.org/debian-devel/2018/11/msg00183.html) includes support testing for reproducibility using `reprotest`. These tests are currently available on-demand and need to be set up individually. * Agustin Henze announced in a mail to the [`debian-devel` mailing list](https://lists.debian.org/debian-devel/) that [the new Debian CI pipeline](https://lists.debian.org/debian-devel/2018/11/msg00183.html) includes support testing for reproducibility using `reprotest`. These tests are currently available on-demand and need to be set up individually.
* 33 Debian package reviews were added, 14 were updated and 33 were removed in this week, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb also updated the [`dc_created_timestamp_in_javadoc`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/1b314210) issue and added a new [`cflags_recorded_in_in_ada_ali_files`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/f3c2f1be) toolchain issue. * 33 Debian package reviews were added, 14 were updated and 33 were removed in this week, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb also updated the [`dc_created_timestamp_in_javadoc`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/1b314210) issue and added a new [`cflags_recorded_in_in_ada_ali_files`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/f3c2f1be) toolchain issue.
* We have received more than 45 registrations for the [Reproducible Builds summit in Paris between 11th—13th December 2018](https://reproducible-builds.org/events/paris2018/) and thus are in the process of closing registrations. If you are interested in attending and are contributing to a project not yet represented, please do *get in touch*, registrations will close for real very soon! * We have received more than 45 registrations for the upcoming [Reproducible Builds summit in Paris](https://reproducible-builds.org/events/paris2018/) between 11th—13th December 2018 and thus are in the process of closing registrations. If you are interested in attending and are contributing to a project not yet represented, please do get in touch as registrations will close shortly.
* Our [report from last week](https://reproducible-builds.org/blog/posts/184/) was quoted in [LWN](https://lwn.net/)'s ["Distribution quotes of the week"](https://lwn.net/Articles/770530/). * Our [report from last week](https://reproducible-builds.org/blog/posts/184/) was quoted in [LWN](https://lwn.net/)'s ["Distribution quotes of the week"](https://lwn.net/Articles/770530/).
...@@ -107,13 +102,13 @@ In addition to that we had contributions from Deb Nicholson, Chris Lamb, Georg F ...@@ -107,13 +102,13 @@ In addition to that we had contributions from Deb Nicholson, Chris Lamb, Georg F
Test framework development Test framework development
-------------------------- --------------------------
There were a large number of updates to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](tests.reproducible-builds.org) by Holger Levsen this week, see below. The most important work was done behind the scenes, outside of git, which was a long debugging session to find out why jenkins java processes were suddenly eating all CPU while the machine had a load of 60-200. This involved (temporarily) removing all 1300 jobs, disabling plugins and other things, all didn't help. In the end it turned out that the underlying SSH/HDD performance was configured poorly, after this was fixed, java/jenkins immediately ran normal. There were a large number of updates to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](tests.reproducible-builds.org) by Holger Levsen this week (see below). The most important work was done behind the scenes outside of Git which was a long debugging session to find out why the Jenkins Java processes were suddenly consuming all of the system resources whilst the machine had a load of 60-200. This involved temporarily removing all 1,300 jobs, disabling plugins and other changes. In the end, it turned out that the underlying SSH/HDD performance was configured poorly and, after this was fixed, Jenkins returned to normal.
* [Debian GNU/Linux](https://www.debian.org/)-specific changes: * [Debian](https://www.debian.org/)-specific changes:
* Merge patch by Simon McVittie to [perform build2 in merged /usr environment for >= buster](https://salsa.debian.org/qa/jenkins.debian.net/commit/d04769a7) (Closes: [#901473](https://bugs.debian.org/901473). * Merge patch by Simon McVittie to apply the "merged `/usr`" directory scheme between builds ([#901473](https://bugs.debian.org/901473)). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d04769a7)]
* Document that we vary by installing the `usr-merge` package [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7902f640)] and add a link to the [corresponding Debian Wiki page](https://wiki.debian.org/UsrMerge]) [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fb44311e)]. * Document that we vary by installing the `usr-merge` package [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7902f640)] and add a link to the [corresponding Debian Wiki page](https://wiki.debian.org/UsrMerge]) [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fb44311e)].
* [Use `pbuilder` from the "backports" repositories everywhere](https://salsa.debian.org/qa/jenkins.debian.net/commit/2081b3a4), to achieve that also [force installation of `pbuilder` from bpo on ubuntu 16.04](https://salsa.debian.org/qa/jenkins.debian.net/commit/d28a62fb) * [Use `pbuilder` from the "backports" repositories everywhere](https://salsa.debian.org/qa/jenkins.debian.net/commit/2081b3a4), to achieve that also [force installation of `pbuilder` from backports on Ubuntu 16.04](https://salsa.debian.org/qa/jenkins.debian.net/commit/d28a62fb)
* Deal with flacky armhf boards. ([1](https://salsa.debian.org/qa/jenkins.debian.net/commit/6121cd22), [2](https://salsa.debian.org/qa/jenkins.debian.net/commit/099a8de5), [3](https://salsa.debian.org/qa/jenkins.debian.net/commit/2bc5747f), [4](https://salsa.debian.org/qa/jenkins.debian.net/commit/410d530b)) * Deal with flaky `armhf` boards. ([1](https://salsa.debian.org/qa/jenkins.debian.net/commit/6121cd22), [2](https://salsa.debian.org/qa/jenkins.debian.net/commit/099a8de5), [3](https://salsa.debian.org/qa/jenkins.debian.net/commit/2bc5747f), [4](https://salsa.debian.org/qa/jenkins.debian.net/commit/410d530b))
* Remove java and depends from all 49 build nodes manually. Also clean up cruft from the jessie2stretch upgrades on armhf nodes. * Remove java and depends from all 49 build nodes manually. Also clean up cruft from the jessie2stretch upgrades on armhf nodes.
* Misc/generic changes: * Misc/generic changes:
...@@ -121,7 +116,7 @@ There were a large number of updates to our [Jenkins](https://jenkins.io/)-based ...@@ -121,7 +116,7 @@ There were a large number of updates to our [Jenkins](https://jenkins.io/)-based
* [Increase heap size further and drop all other Java arguments](https://salsa.debian.org/qa/jenkins.debian.net/commit/005aab43). * [Increase heap size further and drop all other Java arguments](https://salsa.debian.org/qa/jenkins.debian.net/commit/005aab43).
* [Do not recover `schroot` sessions](https://salsa.debian.org/qa/jenkins.debian.net/commit/69cfa0c1). Thanks to Helmut Grohne. * [Do not recover `schroot` sessions](https://salsa.debian.org/qa/jenkins.debian.net/commit/69cfa0c1). Thanks to Helmut Grohne.
* [Run our health check less often](https://salsa.debian.org/qa/jenkins.debian.net/commit/246b3c25). * [Run our health check less often](https://salsa.debian.org/qa/jenkins.debian.net/commit/246b3c25).
* [Make jenkins `schroot` configuration the common/default.](https://salsa.debian.org/qa/jenkins.debian.net/commit/aba431d3). * [Make jenkins `schroot` configuration the common/default](https://salsa.debian.org/qa/jenkins.debian.net/commit/aba431d3).
* [Drop code related to volume groups](https://salsa.debian.org/qa/jenkins.debian.net/commit/8f0c9c43). * [Drop code related to volume groups](https://salsa.debian.org/qa/jenkins.debian.net/commit/8f0c9c43).
In addition, Mattia Rizzolo fixed an issue in the web-based package rescheduling tool by [encoding a string before passing to `subprocess.run`](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b1832b4) and to [fix the parsing of the "issue" selector option](https://salsa.debian.org/qa/jenkins.debian.net/commit/641cfb29). In addition, Mattia Rizzolo fixed an issue in the web-based package rescheduling tool by [encoding a string before passing to `subprocess.run`](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b1832b4) and to [fix the parsing of the "issue" selector option](https://salsa.debian.org/qa/jenkins.debian.net/commit/641cfb29).
......