185.md 14.4 KB
Newer Older
1
---
2
layout: new/blog
3
week: 185
4
published: 2018-11-13 13:56:58
5 6
---

Chris Lamb's avatar
Chris Lamb committed
7
Here's what happened in the [Reproducible Builds](https://reproducible-builds.org) effort between Sunday November 4 and Saturday November 10 2018:
8

Chris Lamb's avatar
Chris Lamb committed
9
* **We are excited to announce that the Reproducible Builds project [has joined the Software Freedom Conservancy](https://reproducible-builds.org/news/2018/11/08/reproducible-builds-joins-software-freedom-concervancy/)**!
10

11
   [![]({{ "/images/logos/rb_joins_sfc.png#center" | prepend: site.baseurl }})](https://reproducible-builds.org/news/2018/11/08/reproducible-builds-joins-software-freedom-concervancy/)
12

Chris Lamb's avatar
Chris Lamb committed
13
   [Conservancy](https://sfconservancy.org/about/) is a not-for-profit organisation that helps promote, develop and defend free software projects. We can now can take directed donations and the Conservancy can also provide projects us with basic legal services. The Reproducible Builds project is delighted and honoured to be associated with Conservancy's outreach work and other work of the project and look forward to a long and mutually beneficial relationship.
14

Chris Lamb's avatar
Chris Lamb committed
15
* The month-long session of students from the [Application Security](http://bulletin.engineering.nyu.edu/preview_course_nopop.php?catoid=9&coid=23997) course at [New York University](https://www.nyu.edu/), cataloguing, submitting and merging reproduciblity bugs concluded this week. This year, students made 55 tags and issues for Debian and [Arch Linux](https://www.archlinux.org/) packages and sent 18 pull requests upstream of which 4 have been merged.
16

Chris Lamb's avatar
Chris Lamb committed
17
* Richard Parkins [posted a detailed message to our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/2018-November/001251.html) on the topic of algorithms used for comparing binary files in a way that makes the result easily consumable by humans. Most binary file comparators just compare bytes and thus do not semantically detect deletions or insertions. This is relevant to our work on [diffoscope](https://diffoscope.org/). He linked to some [example code on GitHub](https://github.com/rparkins999/bindiff).
18

Chris Lamb's avatar
Chris Lamb committed
19
* There was further discussion on Debian bug [#869184](https://bugs.debian.org/869184) which relates to `dpkg` generating source uploads that include architecture in the name of the `.buildinfo` file (eg. `_amd64.buildinfo`). This week, Salvatore Bonaccorso reported that the [Debian Security Team](https://wiki.debian.org/Teams/Security) were [hit by this issue again](https://bugs.debian.org/869184#60).
Bernhard M. Wiedemann's avatar
Bernhard M. Wiedemann committed
20

Chris Lamb's avatar
Chris Lamb committed
21
* On Tuesday 6th November, Chris Lamb [hosted a seminar and a lengthy Q&A session](http://talks.cam.ac.uk/talk/index/114232) at the William Gates Building at the University of Cambridge on reproducible builds as part of the [Computer Laboratory NetOS Group](https://www.cl.cam.ac.uk/research/srg/netos/).
22

23
* [Simon McVittie](http://smcv.pseudorandom.co.uk/) kindly [provided a patch](https://bugs.debian.org/901473#33) to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org/) to vary whether we apply the "merged `/usr`" directory scheme between builds. This is where the `/{bin,sbin,lib}/` directories are symbolic links to `/usr/{bin,sbin,lib}/`. It was subsequently merged by Holger Levsen and resulted in some variations in (at least) [quilt](https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/quilt.html) and [systemd](https://tests.reproducible-builds.org/debian/rb-pkg/buster/amd64/diffoscope-results/systemd.html).
24

Chris Lamb's avatar
Chris Lamb committed
25
* Chris Lamb updated `strip-nondeterminism` (our tool to post-process files to remove known non-deterministic output) to [catch invalid ZIP "local" field lengths](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/e5f5008) — we were previously blindly trusting the value supplied in the ZIP file ([#803503](https://bugs.debian.org/803503)). In addition, he applied a patch from Emmanuel Bourg to [update the Javadoc handler to handle OpenJDK 11](https://salsa.debian.org/reproducible-builds/strip-nondeterminism/commit/f745484) ([#913132](https://bugs.debian.org/913132)). He then subsequently uploaded version `0.044-1` [to Debian unstable](https://tracker.debian.org/news/1001570/accepted-strip-nondeterminism-0044-1-source-all-into-unstable/).
Chris Lamb's avatar
Chris Lamb committed
26

Chris Lamb's avatar
Chris Lamb committed
27
* Agustin Henze announced in a mail to the [`debian-devel` mailing list](https://lists.debian.org/debian-devel/) that [the new Debian CI pipeline](https://lists.debian.org/debian-devel/2018/11/msg00183.html) includes support testing for reproducibility using `reprotest`. These tests are currently available on-demand and need to be set up individually.
Chris Lamb's avatar
Chris Lamb committed
28 29 30

* 33 Debian package reviews were added, 14 were updated and 33 were removed in this week, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). Chris Lamb also updated the [`dc_created_timestamp_in_javadoc`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/1b314210) issue and added a new [`cflags_recorded_in_in_ada_ali_files`](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/f3c2f1be) toolchain issue.

31
* We have received more than 45 registrations for the upcoming [Reproducible Builds summit in Paris](https://reproducible-builds.org/events/paris2018/) between 11th—13th December 2018 and thus are in the process of closing registrations. If you are interested in attending and are contributing to a project not yet represented, please do get in touch as registrations will close shortly.
Chris Lamb's avatar
Chris Lamb committed
32 33

* Our [report from last week](https://reproducible-builds.org/blog/posts/184/) was quoted in [LWN](https://lwn.net/)'s ["Distribution quotes of the week"](https://lwn.net/Articles/770530/).
Bernhard M. Wiedemann's avatar
Bernhard M. Wiedemann committed
34 35 36 37 38

Packages reviewed and fixed, and bugs filed
-------------------------------------------

* Bernhard M. Wiedemann:
Chris Lamb's avatar
Chris Lamb committed
39
    * [geany](https://github.com/geany/geany/pull/1989) (Don't use [inode numbers](https://en.wikipedia.org/wiki/Inode) in useless ways)
Bernhard M. Wiedemann's avatar
Bernhard M. Wiedemann committed
40 41
    * [python-qscintilla-qt5](https://build.opensuse.org/request/show/647086) (sort `readdir(2)`, [submitted upstream](https://www.riverbankcomputing.com/pipermail/qscintilla/2018-November/001349.html))
    * [pesign-obs-integration](https://bugzilla.opensuse.org/show_bug.cgi?id=1114605) (bug)
Bernhard M. Wiedemann's avatar
Bernhard M. Wiedemann committed
42
    * [grep](https://build.opensuse.org/request/show/647618) ([PGO](https://github.com/bmwiedemann/theunreproduciblepackage/tree/master/pgo)/parallelism)
43

Chris Lamb's avatar
Chris Lamb committed
44 45 46 47
* Chris Lamb:
    * [#912957](https://bugs.debian.org/912957) filed against [python-multipletau](https://tracker.debian.org/pkg/python-multipletau) ([https://github.com/FCS-analysis/multipletau/pull/16](merged upstream))
    * Chris's previously-authored patches for [GNU mtools](https://www.gnu.org/software/mtools/) to ensure the [Debian Installer](https://www.debian.org/devel/debian-installer/) images could become reproducible which were sent upstream last week ([1](http://lists.gnu.org/archive/html/info-mtools/2018-10/msg00003.html) & [2](http://lists.gnu.org/archive/html/info-mtools/2018-10/msg00004.html)) were merged and should appear in the [upcoming mtools 4.0.20 release](http://lists.gnu.org/archive/html/info-mtools/2018-11/msg00000.html).

Oskar Wirga's avatar
Oskar Wirga committed
48 49 50 51 52 53
* Oskar Wirga:
    * [python-octaviaclient](https://salsa.debian.org/openstack-team/clients/python-octaviaclient/merge_requests/1) (Set `PYTHONHASHSEED`)
    * [felix-osgi-obr](https://salsa.debian.org/java-team/felix-osgi-obr/merge_requests/1) (Timestamp in documentation)
    * [easyconf](https://salsa.debian.org/java-team/easyconf/merge_requests/1) (Ended up being a bug in [strip-nondeterminism](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913132))
    * [commons-daemon](https://salsa.debian.org/java-team/commons-daemon/merge_requests/1) (Ended up being a bug in [strip-nondeterminism](https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=913132))

54 55
* Snahil Singh:
    * [#913195](https://bugs.debian.org/913195): Please make netmrg reproducible.
Chris Lamb's avatar
Chris Lamb committed
56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77

diffoscope development
----------------------

[diffoscope](https://diffoscope.org/) is our in-depth "diff-on-steroids" utility which helps us diagnose reproducibility issues in packages. This week, version `105` was [uploaded to Debian unstable](https://tracker.debian.org/news/1001952/accepted-diffoscope-105-source-into-unstable/) by Mattia Rizzolo. It [included contributions already covered in previous weeks](https://salsa.debian.org/reproducible-builds/diffoscope/commits/105) as well as new ones from:

* Chris Lamb:
    * [Don't assume all files called ".a" are ELF binaries because we specified a `FILE_EXTENSION_SUFFIX`](https://salsa.debian.org/reproducible-builds/diffoscope/commit/cd4c642). This prevents a potential "Unrecognized archive format" traceback. ([#903446](https://bugs.debian.org/903446))
    * [Prevent errors when obtaining PDF metadata from files with multiple PDF metadata dictionary definition entries](https://salsa.debian.org/reproducible-builds/diffoscope/commit/9624319). ([#913315](https://bugs.debian.org/913315))
    * [Display the reason when cannot extract metadata from PDF files](https://salsa.debian.org/reproducible-builds/diffoscope/commit/cf3bc34).

* Daniel Shahaf:
    * [Fix test failures with upcoming `file(1)` 5.35](https://salsa.debian.org/reproducible-builds/diffoscope/commit/0dfb818). Thanks to Christoph Biedl for the heads-up in advance. ([#912756](https://bugs.debian.org/912756))

* Mattia Rizzolo:
    * [Update GPG key](https://salsa.debian.org/reproducible-builds/diffoscope/commit/44e2c29).
    * [Fix a number of `flake8` and other deprecation warnings](https://salsa.debian.org/reproducible-builds/diffoscope/commit/becf992).

* Will Thompson:
    * [Add a `--list-missing-tools` option](https://salsa.debian.org/reproducible-builds/diffoscope/commit/339a431).


78 79
Website updates
---------------
Chris Lamb's avatar
Chris Lamb committed
80

Holger Levsen's avatar
Holger Levsen committed
81
There were a large number of changes to [our website](https://reproducibile-builds) this week:
Chris Lamb's avatar
Chris Lamb committed
82 83 84 85 86 87 88 89 90 91 92 93 94 95 96

* Bernhard M. Wiedemann:
    * [Update the "CMake" info](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/11828b3) to our page about the [`SOURCE_DATE_EPOCH`](https://reproducible-builds.org/docs/source-date-epoch/) environment variable.

* Chris Lamb:
    * [Use our "contribute to Salsa" page in the footer](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/f944a44).
    * [Add Cambridge University seminar](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/cb1f822).

* Holger Levsen:
    * Add Huawei, Alpine Linux and Bazel to [Paris summit event page](https://reproducible-builds.org/events/paris2018/). ([1](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9559dc5), [2](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ad07633), [3](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/ddd1aea))

* Mattia Rizzolo:
    * [Prepend `site.baseurl` (not `site.url`) in our logo](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/cc6f579).
    * [Add a donation page](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/77b419d).

Snahil Singh's avatar
Snahil Singh committed
97

Holger Levsen's avatar
Holger Levsen committed
98 99
In addition to that we had contributions from Deb Nicholson, Chris Lamb, Georg Faerber, Holger Levsen and Mattia Rizzolo *et al.* on the [press release regarding joining the Software Freedom Conservancy](https://reproducible-builds.org/news/2018/11/08/reproducible-builds-joins-software-freedom-concervancy/):

Chris Lamb's avatar
Chris Lamb committed
100 101 102 103

Test framework development
--------------------------

104
There were a large number of updates to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org/) by Holger Levsen this week (see below). The most important work was done behind the scenes outside of Git which was a long debugging session to find out why the Jenkins Java processes were suddenly consuming all of the system resources whilst the machine had a load of 60-200. This involved temporarily removing all 1,300 jobs, disabling plugins and other changes. In the end, it turned out that the underlying SSH/HDD performance was configured poorly and, after this was fixed, Jenkins returned to normal.
Chris Lamb's avatar
Chris Lamb committed
105

Chris Lamb's avatar
Chris Lamb committed
106
* [Debian](https://www.debian.org/)-specific changes:
Chris Lamb's avatar
Chris Lamb committed
107
    * Merge patch by Simon McVittie to apply the "merged `/usr`" directory scheme between builds ([#901473](https://bugs.debian.org/901473)). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d04769a7)]
108
    * Document that we vary by installing the `usr-merge` package [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7902f640)] and add a link to the [corresponding Debian Wiki page](https://wiki.debian.org/UsrMerge]) [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/fb44311e)].
Chris Lamb's avatar
Chris Lamb committed
109
    * [Use `pbuilder` from the "backports" repositories everywhere](https://salsa.debian.org/qa/jenkins.debian.net/commit/2081b3a4), to achieve that also [force installation of `pbuilder` from backports on Ubuntu 16.04](https://salsa.debian.org/qa/jenkins.debian.net/commit/d28a62fb)
Chris Lamb's avatar
Chris Lamb committed
110
    * Deal with flaky `armhf` boards. ([1](https://salsa.debian.org/qa/jenkins.debian.net/commit/6121cd22), [2](https://salsa.debian.org/qa/jenkins.debian.net/commit/099a8de5), [3](https://salsa.debian.org/qa/jenkins.debian.net/commit/2bc5747f), [4](https://salsa.debian.org/qa/jenkins.debian.net/commit/410d530b))
111
    * Remove java and depends from all 49 build nodes manually. Also clean up cruft from the jessie2stretch upgrades on armhf nodes.
Chris Lamb's avatar
Chris Lamb committed
112 113 114 115

* Misc/generic changes:

    * [Increase heap size further and drop all other Java arguments](https://salsa.debian.org/qa/jenkins.debian.net/commit/005aab43).
Holger Levsen's avatar
Holger Levsen committed
116
    * [Do not recover `schroot` sessions](https://salsa.debian.org/qa/jenkins.debian.net/commit/69cfa0c1). Thanks to Helmut Grohne.
Chris Lamb's avatar
Chris Lamb committed
117
    * [Run our health check less often](https://salsa.debian.org/qa/jenkins.debian.net/commit/246b3c25).
Chris Lamb's avatar
Chris Lamb committed
118
    * [Make jenkins `schroot` configuration the common/default](https://salsa.debian.org/qa/jenkins.debian.net/commit/aba431d3).
Holger Levsen's avatar
Holger Levsen committed
119
    * [Drop code related to volume groups](https://salsa.debian.org/qa/jenkins.debian.net/commit/8f0c9c43).
Holger Levsen's avatar
Holger Levsen committed
120

Chris Lamb's avatar
Chris Lamb committed
121 122 123 124
In addition, Mattia Rizzolo fixed an issue in the web-based package rescheduling tool by [encoding a string before passing to `subprocess.run`](https://salsa.debian.org/qa/jenkins.debian.net/commit/5b1832b4) and to [fix the parsing of the "issue" selector option](https://salsa.debian.org/qa/jenkins.debian.net/commit/641cfb29).

---

Chris Lamb's avatar
Chris Lamb committed
125
This week's edition was written by Arnout Engelen, Bernhard M. Wiedemann, Chris Lamb, Holger Levsen, Oskar Wirga, Santiago Torres, Snahil Singh & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.