195.md 13.3 KB
Newer Older
1 2 3
---
layout: new/blog
week: 195
4
published: 2019-01-22 12:40:47
5 6
---

Chris Lamb's avatar
Chris Lamb committed
7
Here's what happened in the [Reproducible Builds](https://reproducible-builds.org) effort between Sunday January 13th and Saturday January 19th 2019:
8

Chris Lamb's avatar
Chris Lamb committed
9
* In the [Rust](https://www.rust-lang.org/) programming language community there was an interesting discussion on the [/r/rust](https://www.reddit.com/r/rust) subreddit around [the ripgrep utility becoming reproducible in Debian](https://www.reddit.com/r/rust/comments/afscgo/ripgrep_0100_is_reproducible_in_debian/). In addition, [Tony Arcieri](https://tonyarcieri.com/) opened a issue in the Rust's [Secure Code Working Group](https://twitter.com/rustsecurecode) enquiring about [reproducible builds tooling](https://github.com/rust-secure-code/wg/issues/28).
10

Chris Lamb's avatar
Chris Lamb committed
11
* Last week, Chris Lamb opened Debian bug [#919207](https://bugs.debian.org/919207) requesting that the [`squashfs-tools`](https://tracker.debian.org/pkg/squashfs-tools) package (which creates and manipulates read-only compressed file systems) applies a patch to remove non-deterministic data introduced by a "fragmentation deflator" thread. This was the final patch required for reproducible images for (at least) [Tails](https://tails.boum.org).
12

Chris Lamb's avatar
Chris Lamb committed
13
    Whilst Laszlo Boszormenyi applied the patch, he [subsequently reverted the change](https://bugs.debian.org/919207#15) as it was breaking [LZO](https://en.wikipedia.org/wiki/Lempel%E2%80%93Ziv%E2%80%93Oberhumer) compression. However, Chris subsequently [updated and fixed the issue](https://bugs.debian.org/919207#24) which was then uploaded in version [`1:4.3-11`](https://tracker.debian.org/news/1022501/accepted-squashfs-tools-143-11-source-into-unstable/).
14

heinrich5991's avatar
heinrich5991 committed
15
* As part of the [Debian Long Term Support (LTS)](https://wiki.debian.org/LTS) effort it was [noticed that an old package was failing to build beyond ~2015](https://lists.debian.org/debian-lts/2019/01/msg00033.html).
16

Mattia Rizzolo's avatar
Mattia Rizzolo committed
17
* Holger Levsen released and uploaded `disorderfs` (our [FUSE](https://github.com/libfuse/libfuse)-based filesystem that deliberately introduces non-determinism into filesystems) version `0.5.6-1` to Debian unstable [[...](https://tracker.debian.org/news/1021833/accepted-disorderfs-056-1-source-into-unstable/)] and Chris Lamb released/uploaded `strip-nondeterminism` (our tool that post-processes files to remove known non-deterministic output) version `1.1.0-1` to Debian unstable [[...](https://tracker.debian.org/news/1020523/accepted-strip-nondeterminism-110-1-source-all-into-unstable/)] too.
18

Chris Lamb's avatar
Chris Lamb committed
19
* Chris Lamb added 8 Debian package reviews but 12 were also updated and 14 were removed in this week, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html).
20

Chris Lamb's avatar
Chris Lamb committed
21 22 23 24
* There were a number of interesting discussions on [our mailing list](https://lists.reproducible-builds.org/pipermail/rb-general/) this week including:
    * Hervé Boutemy posted [a brief introduction to "reproducible-central"](https://lists.reproducible-builds.org/pipermail/rb-general/2019-January/001408.html) after a number of discussions and documentation regarding [Java Virtual Machine rebuilder attestations](https://reproducible-builds.org/docs/jvm/) and the [Apache Maven](https://maven.apache.org/) build tool.
    * Elio Qoshi from [Ura Design](https://ura.design/) asking [whether we would be interested](https://lists.reproducible-builds.org/pipermail/rb-general/2019-January/001412.html) in updating [our style guide]({{ "/style-epoch/" | prepend: site.baseurl }}).
    * Lastly, Eli Schwartz posted an update regarding [reproducible package archives](https://lists.reproducible-builds.org/pipermail/rb-general/2019-January/001402.html) in [Arch Linux](https://www.archlinux.org/).
25

Chris Lamb's avatar
Chris Lamb committed
26
## Packages reviewed and fixed, and bugs filed
27

Bernhard M. Wiedemann's avatar
Bernhard M. Wiedemann committed
28 29
* Bernhard M. Wiedemann:
    * [python-cmarkgfm](https://github.com/theacodes/cmarkgfm/pull/17) (merged, sort python glob)
Chris Lamb's avatar
Chris Lamb committed
30 31 32 33 34
* Chris Lamb:
    * [#919566](https://bugs.debian.org/919566) filed against [satpy](https://tracker.debian.org/pkg/satpy) ([forwarded upstream](https://github.com/pytroll/satpy/pull/579)).

## [diffoscope](https://diffoscope.org/) development

35
[diffoscope](https://diffoscope.org/) is our in-depth "diff-on-steroids" utility which helps us diagnose reproducibility issues in packages. There were a few updates this week including contributions from:
Chris Lamb's avatar
Chris Lamb committed
36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63

* Chris Lamb:
    * Fix inverted logic and invalid reference to `file` in the [FreePascal](https://www.freepascal.org/) comparator. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/248e9ba)]
    * Use [`str.format`](https://docs.python.org/3.4/library/stdtypes.html#str.format) over `+` for string concatenation. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/7bf992b)]
    * Re-enable [Gnumeric](http://www.gnumeric.org/) `Build-Depends`. [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/b696748)] [[...](https://salsa.debian.org/reproducible-builds/diffoscope/commit/12882be)]

* Jelle van der Waa:
    * Remove an unused `re` import in the [WebAssembly](https://webassembly.org/) comparator. ([MR: !18](https://salsa.debian.org/reproducible-builds/diffoscope/merge_requests/18))

Version `108` was then [uploaded to Debian unstable](https://tracker.debian.org/news/1020530/accepted-diffoscope-108-source-all-into-unstable/) by Chris Lamb and was subsequently [backported to the `stretch-backports` distribution](https://tracker.debian.org/news/1021754/accepted-diffoscope-108bpo91-source-all-into-stretch-backports/) by Mattia Rizzolo.

## Website development

There were a number of updates to the [reproducible-builds.org](https://reproducible-builds.org) project website this week, including:

* Hervé Boutemy:
    * Large number of changes to the [Java Virtual Machine]({{ "/docs/jvm/" | prepend: site.baseurl }}) page including adding the `build-root` property for multi-module builds [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/bfd1421)], adding instructions on [Apache Maven](https://maven.apache.org/) rebuild arguments [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/190ca8b)], amongst many others [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4a7e0fe)] [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/928c71f)] [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/12a8023)].
* Holger Levsen:
    * Drop [LEDE](https://en.wikipedia.org/wiki/LEDE) as the project has re-merged with [OpenWrt](https://openwrt.org/). [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4257259)]
* Peter Wu:
    * Mention `QT_RCC_SOURCE_DATE_OVERRIDE` and add some more CMake, RPATH and Qt notes on the [deterministic build systems]({{ "/docs/deterministic-build-systems/" | prepend: site.baseurl }}) page. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/36bca83)] [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/83d4700)] [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/77d689d)].
    * Document the use of `-fmacro-prefix-map` and `-ffile-prefix-map` on the [build path]({{ "/docs/build-path/" | prepend: site.baseurl }}) page. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/edca632)]
    * Fix some links and typos on the [contribute]({{ "/contribute/" | prepend: site.baseurl }}) page, some dead links to [Salsa](https://salsa.debian.org) and correct some link formatting issues. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/61fd247)] [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/56d3c75)] [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/821d532)]

## Test framework development

We operate a comprehensive [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](https://tests.reproducible-builds.org). This week:

Bernhard M. Wiedemann's avatar
Bernhard M. Wiedemann committed
64
* [Arch Linux](https://www.archlinux.org/) is the first project being built on nodes dedicated from [OSUOSL](https://osuosl.org/).
Chris Lamb's avatar
Chris Lamb committed
65 66 67

   Interestingly, these new nodes are running `4.19` Linux kernels from the `stretch-backports` distribution as [Qt](https://www.qt.io/) in Arch needs a newer kernel than the kernel in Debian stretch to build. As a result of this we are now seeing 1,736 builds of Arch packages in the last 24h, meaning our subset of packages are being fully rebuilt every 5 or 6 days.

68
* [F-Droid](https://f-droid.org/) became the second project to be tested on these new nodes after Holger Levsen increased the size of various partitions to accommodate the builds, as well as to provide a [Squid](http://www.squid-cache.org/) proxy for all our OSUOSL nodes.
Chris Lamb's avatar
Chris Lamb committed
69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91

The following more-specific changes were made:

* Eli Schwartz:
    * Import Arch Linux [GnuPG](https://www.gnupg.org/) keys before running [`makepkg`](https://wiki.archlinux.org/index.php/makepkg). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1b36569b)]
    * Perform a giant cleanup of trailing whitespaces. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1fa9f22b)]

* Holger Levsen:
    * [Arch Linux](https://www.archlinux.org/)-specific changes:
        * Adjust the rescheduling of packages which have been tested *X* days ago. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/14ddcf02)] [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/18a33594)] [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/90da004f)]
        * Adopt maintenance job to work with the new OSUOSL nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c8702012)]
        * Support [OpenSSH](https://www.openssh.com/) running on ports other than `22`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/71879c41)]
        * Fix the path to the Arch Linux `mirrorlist`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a6675634)]
    * [Debian](https://www.debian.org/)-specific changes:
        * Fix warning message to include the name of broken package sets [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/f281ec75)] and also show the total number of packages in a package set [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/47654780)].
        * Don't update [`pbuilder`](https://wiki.debian.org/PbuilderTricks) and Debian [`schroots`](https://wiki.debian.org/Schroot) on OSUOSL nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/38ed2e99)]
        * Clarify "stalled" status of the [LeMaker HiKey960](http://www.lemaker.org/product-hikeysecond-specification.html) boards. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/82b6130d)]
        * Document how to access [Codethink](https://www.codethink.co.uk/)'s `arm64` nodes. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/71b4ea09)]
    * [F-Droid](https://f-droid.org/)-specific changes:
        * Remove duplicate job definitions. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c2718ea4)]
    * Misc/generic changes:
        * Update the "job health page", adding a helpful footer. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0b414680)] [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/d56945e4)]
        * Use `time.osuosl.org` as the [NTP](https://en.wikipedia.org/wiki/Network_Time_Protocol) server for OSUOSL nodes, de.pool.ntp.org for the rest. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/060817f9)]
92 93
        * Warn if we detect the wrong [Maximum Transmission Unit (MTU))[https://en.wikipedia.org/wiki/Maximum_transmission_unit). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/2def2676)]
        * Drop another mention of [LEDE](https://en.wikipedia.org/wiki/LEDE). [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/5f457969)]
Chris Lamb's avatar
Chris Lamb committed
94 95 96 97 98 99 100 101
    * Node maintenance. ([[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/a5b35523)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e86711cc)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/34a9e2bb)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0133292e)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/dce5c729)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/6f3a20a9)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/20e2b9f6)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/99b5ae65)], [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/193fb3a3)], etc.)

* Mattia Rizzolo:
    * Fix a variable name in the "deploy Jenkins" script. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/20e39567)]
    * Fix a non-fatal syntax error in the "health check" script. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4bf29894)]
    * Node maintenance. ([[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/b1ab00ca)], etc.)

* Vagrant Cascadian:
102
    * Node maintenance. ([[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/49de7ae8)], [[...](https://bugs.debian.org/919851)], etc.)
Chris Lamb's avatar
Chris Lamb committed
103 104 105

---

Chris Lamb's avatar
Chris Lamb committed
106
This week's edition was written by [Bernhard M. Wiedemann](https://lizards.opensuse.org/author/bmwiedemann/), [Chris Lamb](https://chris-lamb.co.uk/), `heinrich5991`, [Holger Levsen](http://layer-acht.org/thinking/), Mattia Rizzolo, [Vagrant Cascadian](https://www.aikidev.net/) & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.