Skip to content

Commits on Source 8

Show whitespace changes
Inline Side-by-side
_blog/posts/191.md
View file @ 5e10be8c
......@@ -3,34 +3,94 @@ layout: new/blog
week: 191
---
* Dave Rosenthal wrote about securing the software supply chain: https://blog.dshr.org/2018/12/securing-software-supply-chain.html
Here's what happened in the [Reproducible Builds](https://reproducible-builds.org) effort between Sunday December 16 and Saturday December 22 2018:
* [FIXME](https://sfconservancy.org/blog/2018/dec/18/JoshT/)
* The [F-Droid](https://f-droid.org) project, an catalogue of free-software applications for the Android platform, have published a page on their website [describing their adoption and implementation of reproducible builds](https://f-droid.org/en/docs/Reproducible_Builds/).
* [FIXME](https://lists.apache.org/thread.html/ceb357513ff0403414b5fff7dbeb1ea43961e71f9e48425d6e3cea8f@%3Cgeneral.incubator.apache.org%3E)
* [Dave Rosenthal](https://blog.dshr.org/) wrote about [securing the software supply chain](https://blog.dshr.org/2018/12/securing-software-supply-chain.html) touching on reproducible builds, certificate transparency, etc. In addition, Avery ("apenwarr") Pennarun wrote a blog post entitled "[mtime comparison considered harmful](https://apenwarr.ca/log/20181113)".
* [FIXME](https://pca.st/6mqx#t=42m3s)
* Chris Lamb updated `strip-nondeterminism` (our tool to post-process files to remove known non-deterministic output):
* [FIXME](On January 9th 2019, Chris Lamb will speak at Rennes University, France on reproducible builds.)
* Remove `javaproperties` handler after Emmanuel Bourg's patch was released in `openjdk-11` version `11.0.1+13-3`. ([#914289](https://bugs.debian.org/914289))
* Drop `.ar` handler; `binutils` is reproducible. ([#781262](https://bugs.debian.org/781262), [#843811](https://bugs.debian.org/843811))
* Ignore encrypted `.zip` files as we can never normalise them. ([#852207](https://bugs.debian.org/852207))
* [mtimes](https://apenwarr.ca/log/20181113)
* As part of the [Software Freedom Conservancy](https://sfconservancy.org)'s fundraiser, Josh Triplett [referenced us in a short interview](https://sfconservancy.org/blog/2018/dec/18/JoshT/):
* Summit report from Nix developers zimbatm, Profpatsch and lewo: https://discourse.nixos.org/t/reproducible-builds-summit-report/1683/2
> Reproducible Builds represents one of those ideas where the goal seems obvious and yet the execution requires an incredible and pervasive effort across the industry, and the people working on it have done an amazing job.
* [under module authentication CT logs etc](https://blog.golang.org/modules2019)
* [Joachim Breitner](http://www.joachim-breitner.de/blog) wrote a blog post titled "[Thoughts on Bootsrapping GHC](http://www.joachim-breitner.de/blog/748-Thoughts_on_bootstrapping_GHC)", attempting to answer the question of "how can we build a whole operating system from just and only source code, using very little, or even no, binary seeds or auto-generated files."
* [FIXME](https://f-droid.org/en/docs/Reproducible_Builds/)
* A full and in-depth report about [our recent summit](https://reproducible-builds.org/events/paris2018/) is being prepared but in the meantime [there were further reports published](https://discourse.nixos.org/t/reproducible-builds-summit-report/1683/2) from [NixOS](https://nixos.org/) developers `zimbatm`, `Profpatsch` and `lewo`.
Packages reviewed and fixed, and bugs filed
-------------------------------------------
* Reproducible Builds were mentioned in [Episode 2](https://librelounge.org/episodes/episode-2-thanksgiving-npm-and-malware-in-free-software.html) of the [Libre Lounge](https://librelounge.org/) podcast in a more-general discussion about software supply chains around the recent [NPM event-stream attack](https://blog.bitpay.com/npm-package-vulnerability-copay/). ([Direct link](https://pca.st/6mqx#t=42m3s))
* Bernhard M. Wiedemann:
* Julian Hyde posted to the Apache "Incubator" mailing list discussing the [differences between their binary and source releases](https://lists.apache.org/thread.html/ceb357513ff0403414b5fff7dbeb1ea43961e71f9e48425d6e3cea8f@%3Cgeneral.incubator.apache.org%3E) and how they should correlate.
* After overhauling the [diffoscope.org](https://diffoscope.org) website last week, Chris Lamb tidied the spacing of the logo when viewing on a mobile device [[...](https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/3e4b549)] and ensured the ["Fork me on Salsa" ribbon](https://chris-lamb.co.uk/posts/salsa-ribbons) was hidden too [[...](https://salsa.debian.org/reproducible-builds/diffoscope-website/commit/b8d16b6)].
* There was further discussion on our mailing list on discussing Reproducible Builds through a [mathematical formalism perspective](https://lists.reproducible-builds.org/pipermail/rb-general/2018-December/001346.html).
* The blog for the [Go programming language](https://golang.org) posted their [plans for Go modules in 2019](https://blog.golang.org/modules2019#TOC_5) which include providing a "notary" service. As some background to the problem:
> Today, `go get` relies on connection-level authentication (HTTPS or SSH) to check that it is talking to the right server to download code. There is no additional check of the code itself, leaving open the possibility of man-in-the-middle attacks if the HTTPS or SSH mechanisms are compromised in some way. Decentralization means that the code for a build is fetched from many different servers, which means the build depends on many systems to serve correct code.
* 6 Debian package reviews were added, 10 were updated and 11 were removed in this week, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html).
* On January 9th 2019, [Chris Lamb](https://chris-lamb.co.uk/) will speak at [Université de Rennes](https://www.univ-rennes1.fr/), France on reproducible builds.
## Packages reviewed and fixed, and bugs filed
* [Bernhard M. Wiedemann](https://lizards.opensuse.org/author/bmwiedemann/):
* [hpx](https://build.opensuse.org/request/show/660040) (use [upstream patch](https://github.com/STEllAR-GROUP/hpx/pull/3585))
* [python-jupyter_imatlab_kernel](https://build.opensuse.org/request/show/660055) (do not let pip embed a random tmp path)
* [rust](https://github.com/rust-lang/rust/issues/57041) (nondeterministic asm / cmpq)
* [python-jupyter_imatlab_kernel](https://build.opensuse.org/request/show/660055) (do not let `pip` embed a random "temp" path)
* [rust](https://github.com/rust-lang/rust/issues/57041) (non-deterministic `asm` / `cmpq`)
* [FIXME](http://www.joachim-breitner.de/blog/748-Thoughts_on_bootstrapping_GHC)
* Chris Lamb:
* [#917101](https://bugs.debian.org/917101) filed against [python-sshoot](https://tracker.debian.org/pkg/python-sshoot).
* [#917102](https://bugs.debian.org/917102) filed against [node-nodedbi](https://tracker.debian.org/pkg/node-nodedbi).
* Jelle van der Waa:
* [wavemon](https://github.com/uoaerg/wavemon/pull/59) (date and time in binary)
## Test framework development
There were a number of updates to our [Jenkins](https://jenkins.io/)-based testing framework that powers [tests.reproducible-builds.org](tests.reproducible-builds.org) this week by Holger Levsen including:
* [Arch Linux](https://www.archlinux.org/)-specific changes:
* Extend `refresh_pattern`. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/cdc6bc14)]
* [Debian](https://www.debian.org/)-specific changes:
* Adopt to new "offline nodes" syntax. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/72f80401)]
* Fix a bug where offline nodes were not recognised as such. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8cf7c66f)]
* Limit `unrep_with_dbd_issues()` to Debian architectures for now. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/e4803e6b)]
* Misc/generic changes:
* Don't use existing hosts as example. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/3c660d2d)]
* Add link to database schema. (Thanks for Bernhard M. Wiedemann for pointing out that was missing.) [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/31ff1930)]
* Thank the [OSU Open Source Lab from Oregon State University (OSUOSL)](https://osuosl.org/) for hosting the new `amd64` nodes [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c7afe284)] as well as add the new nodes themselves ([[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/8ea537f4)] & [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/4160dbf6)]), perform the various networking configuration [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/710b804c)] and other various tweaks [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/01d0462f)].
* Various bits node maintenance. (eg. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/c6298df6)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/7b275c0a)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/1bdb6b3f)][[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/958278ae)])
In addition, Mattia Rizzolo updated the `reproducible_notes.py` script to only store notes for Debian packages in the database for now. [[...](https://salsa.debian.org/qa/jenkins.debian.net/commit/0faa4aaf)]
## [reproducible-builds.org](https://reproducible-builds.org) website development
Chris Lamb made a huge number of updates to our [reproducible-builds.org](https://reproducible-builds.org) project website this week:
* Apply some initial, easy styling improvements to our pages via a custom CSS stylesheet for easier merging. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/173c604)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6d8a037)]
* Move the [blog index page](https://reproducible-builds.org/blog/) and blog posts to the new style. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/1e3f5a2)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/94e53f2)]
* Migrate [news entries](https://reproducible-builds.org/news/) and the index to the new style. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/a0c586c)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b2ded56)]
* Put the [list of involved projects](https://reproducible-builds.org/who/) in a nice card grid. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/762d822)]
* Ensure we don't horizontally scroll due to oversized images in blog posts. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/6247c90)]
* Set a more informative site title. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/29c40a3)]
* Add a simple footer for the new style. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/5638819)]
* Don't space out [Markdown](https://en.wikipedia.org/wiki/Markdown)-generated bulleted lists so much. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/b423129)]
Holger Levsen also updated the pages for [our recent summit in Paris](https://reproducible-builds.org/events/paris2018/) to add links to the summit report [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/4a2c567)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/285ff3e)] and to credit other organisers and sponsors [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/34c9554)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/77eaf6c)]. He later added links to [Jelle van der Waa](https://vdwaa.nl) and [Bernhard M. Wiedemann](https://lizards.opensuse.org/author/bmwiedemann/) reports. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/9aac7b7)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/909b390)]
Finally, `heinrich5991` provided two grammatical fixups to our "[How to join the Salsa group](https://reproducible-builds.org/contribute/salsa/)" page. [[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/81c073e)][[...](https://salsa.debian.org/reproducible-builds/reproducible-website/commit/011f9b9)]
---
This week's edition was written by Bernhard M. Wiedemann, Chris Lamb, Jelle van der Waa, Holger Levsen & reviewed by a bunch of Reproducible Builds folks on IRC & the mailing lists.
_layouts/new/blog.html
View file @ 5e10be8c
......@@ -5,23 +5,17 @@ title: Weekly report #{{ page.week }}
<p>&larr; <a href="{{ "/blog/" | prepend: site.baseurl }}">View all weekly reports</a></p>
<h1>Reproducible Builds: Weekly report #{{ page.week }}</h1>
<h1><a href="{{ page.url }}">Reproducible Builds: Weekly report #{{ page.week }}</a></h1>
{% if page.published %}
<p class="text-muted">
Published: {{ page.published | date: "%b %-d, %Y" }}.
</p>
{% else %}
<p class="text-center">
<strong>
This is an unpublished draft post.
</strong>
</p>
<p>
<div class="col-12 alert alert-warning" role="alert">
<strong>This is an unpublished draft post.</strong>
<a href="https://salsa.debian.org/reproducible-builds/reproducible-website.git/tree/_blog/posts/{{ page.week }}.md">(source)</a>
</p>
</div>
{% endif %}
<hr>
......
assets/styles/custom.scss
View file @ 5e10be8c
......@@ -3,6 +3,10 @@ main {
margin-top: 2.5rem;
}
h1 a, h2 a, h3 a, h4 a, h5 a, h6 a {
color: inherit;
}
img {
max-width: 100%;
}
......@@ -12,6 +16,11 @@ main {
display: block;
}
blockquote {
font-style: italic;
margin-left: 2.5rem;
}
.projects .card-title a {
color: inherit;
}
......
bin/generate-draft.template
View file @ 5e10be8c
......@@ -9,8 +9,7 @@ Here's what happened in the [Reproducible Builds](https://reproducible-builds.or
* {{ packages_stats['added'] }} Debian package reviews were added, {{ packages_stats['updated'] }} were updated and {{ packages_stats['removed'] }} were removed in this week, adding to [our knowledge about identified issues](https://tests.reproducible-builds.org/debian/index_issues.html). FIXME issue types have been updated: {% for _, xs in issues_yml.items()|sort %}{% for x in xs %}[{{ x['title'] }}](https://salsa.debian.org/reproducible-builds/reproducible-notes/commit/{{ x['sha'] }}), {% endfor %}{% endfor %}
Packages reviewed and fixed, and bugs filed
-------------------------------------------
## Packages reviewed and fixed, and bugs filed
{% for x, ys in patches.items()|sort %}* {{ x }}:
{% for y in ys %} * [#{{ y['id'] }}](https://bugs.debian.org/{{ y['id'] }}) filed against [{{ y['source'] }}](https://tracker.debian.org/pkg/{{ y['source'] }}).
......@@ -21,14 +20,13 @@ In addition, build failure bugs were reported by:
* {{ k }} ({{ v|length }}){% endfor %}{% endif %}
{% for project in projects %}
{{ project }} development
------------{{ "-" * project|length }}
## {{ project }} development
{% for x in uploads[project] %}
{{ project }} version `{{ x['version'] }}` was [uploaded to Debian {{ x['distribution'] }}](https://tracker.debian.org/pkg/{{ project }}?FIXME) by {{ x['signed_by_name'] }}. It [included contributions already covered in previous weeks](https://salsa.debian.org/reproducible-builds/{{ project }}/commits/{% if project != 'diffoscope' %}debian/{% endif %}{{ x['version'] }}) as well as new ones from:
{% endfor %}
{% for x, ys in commits[project].items()|sort %}* {{ x }}:{% for y in ys %}
* [{{ y['title'] }}]({% if project == "jenkins.debian.net" %}https://salsa.debian.org/qa/jenkins.debian.net/commit/{{ y['sha'] }}{% else %}https://salsa.debian.org/reproducible-builds/{{ project }}/commit/{{ y['sha'] }}{% endif %}){% endfor %}
* {{ y['title'] }}. [[...]({% if project == "jenkins.debian.net" %}https://salsa.debian.org/qa/jenkins.debian.net/commit/{{ y['sha'] }}{% else %}https://salsa.debian.org/reproducible-builds/{{ project }}/commit/{{ y['sha'] }}{% endif %})]{% endfor %}
{% endfor %}
{% endfor %}
......
bin/migrate-blog-posts deleted 100755 → 0
View file @ 99a3d0c9
#!/usr/bin/env python3
import os
import re
import sys
import glob
import click
import shutil
from dateutil import parser
re_date = re.compile(r'^\[\[!meta date="([^\"]+)"\]\]$', re.MULTILINE)
re_meta = re.compile(r'^\[\[!(meta|tag) .*\]\]$', re.MULTILINE)
re_tabs = re.compile(r'\t')
re_links = re.compile(r'\[\[(?P<text>[^\|]+)\|(?P<url>[^\]]+)\]\]')
re_plugins = re.compile(r'\[\[!(?P<key>\S+)\s+(?P<value>[^\]]+)\]\]', re.MULTILINE)
re_filename = re.compile(r'/(?P<week>\d+).mdwn$')
@click.command()
@click.argument('original_blog_dir')
@click.argument('target_dir', default='_blog/posts')
def main(original_blog_dir, target_dir):
log("Removing and recreating {}".format(target_dir))
shutil.rmtree(target_dir, ignore_errors=True)
os.makedirs(target_dir)
for x in glob.glob(os.path.join(original_blog_dir, 'posts', '*.mdwn')):
m = re_filename.search(x)
if m is None:
log("Ignoring file {}".format(x))
continue
week = int(m.group('week'))
generate_week(x, os.path.join(target_dir, '{}.md'.format(week)), week)
def generate_week(src, dst, week):
with open(src, 'r') as f:
body = f.read()
m = re_date.search(body)
if m is None:
raise ValueError("Could not parse date date from {}".format(src))
date = m.group(1)
metadata = (
('layout', 'blog'),
('week', week),
('published', parser.parse(date).strftime('%Y-%m-%d %H:%m:%S')),
)
for x, y in (
# Strip leading meta
(re_meta, ''),
# We don't have plugins in Jekyll
(re_plugins, re_plugins_callback),
# We don't have [[text|target]] links in Jekyll. Last in list)
(re_links, re_links_callback),
# Replace tabs
(re_tabs, ' '),
):
body = x.sub(y, body)
with open(dst, 'w') as f:
print("---", file=f)
print('\n'.join('{}: {}'.format(x, y) for x, y in metadata), file=f)
print("---\n", file=f)
print(body.strip(), file=f)
log("Wrote {}".format(dst))
def log(*args, **kwargs):
print("I: {}".format(*args, **kwargs), file=sys.stderr)
def re_links_callback(m):
return '<a href="{}">{}</a>'.format(
m.group('url'),
m.group('text'),
)
def re_plugins_callback(m):
url, value = {
'bug': ('https://bugs.debian.org/{}', '#{}'),
'pkg': ('https://tracker.debian.org/pkg/{}', '{}'),
'patch': ('https://bugs.debian.org/{}', '#{}'),
'issue': ('https://tests.reproducible-builds.org/issues/unstable/{}_issue.html', '{}'),
'pkgset': ('https://tests.reproducible-builds.org/debian/unstable/amd64/pkg_set_{}.html', '{}'),
}[m.group('key')]
return '[{}]({})'.format(
value.format(m.group('value')),
url.format(m.group('value')),
)
if __name__ == '__main__':
main()