1. Almost everybody uses isolation for builds, but the details vary a lot (VMs, containers, VMs + containers, raw namespacing, userspace syscall substition, MACs).
Isolation matters for dependency detection, dependency minimization,but conversly dependency sharing, isolation of builds, security obvs., resource control.
2. Network might or might not be allowed. One option is to allow network access in some phases, but require a fixed hash of the downloaded artifact (Nix/Guix solution).
3. Bootstrapping is an extreme case of dependency minimization.
== Log ==
Gabor: everybody doing RB has some isolation in place.
Why are we doing is? unknown sources, foil autodetection, prevent spillover between builds, minimize dependencies that are used during build, share dependencies/state between builds and minimize, e.g. glibc version
separate hosts, VMs, namespaces / jails, +selinux optionally
VMs is not enough, because it provides isolation from host, but is not a minimal environment, easier and faster to do local builds
Justin: library os layer to abstract OS differences, maybe ready for next RBS
recompile software to substitute system calls with function calls into the library
gvisor is similar, the new thing is more portable
rr, dettrace
Jochen: Some packages want to load kernel modules during build… (bup)
nvidia drivers are needed after the build for some things
Gabor: should we substitute syscalls?
apparmor? e.g. for a database that is spun up for tests…
Debian packages have nocheck, but not always honoured, and the test dependencies leak into build results and increase the system surface that is accessed during package builds.
Some test stuff could/should be moved out of the build phase.
Debian: upstream tests in the build phase, then downstream integration tests, also piuparts
Gabor: should resource constraints be introduced into the build? Should this be enforced? This happens to some extent automatically with VMs and containers. Not really a question of BR.
Gabor: Should builds be allowed to access the network? Debian yes, Fdroid no. Proxies can be used. Nix and Guix have two types of derivations: one, hash of the result, fixed output delivaration, e.g. a downloaded tarball or repo checkout, two, a real build with type one as inputs, but no network access. Then the question of network access doesn't matter. Except when the source might go away at some point.
It is more economic to build a system which allows BR for a limited time, e.g. when stuff is available to download from the CDN.
For Nix/Guix the build is just anything that reproduces the output hash, allowing caching. Next step is to abstract hardware, so that builds can be done without access to specific hardware types.
For nix any user belonging to a group can trigger builds.
Jochen: bootstrapping? Use a shell script to bootstrap for a new architecture. Bootstrapping is an extreme case of dependency minimization. In Guix, 400 byte bootstrap binary + 25MB guile driver. Debian is interested in cross-arch boostrapping, because new arch every year or so, but the set of base packages changes every time, so hard to reuse the process. Maybe fewer new architectures after Risc-V.
Jochen: reproducibility of cross and native builds is good for trust.
Julien: trustless binary distribution #scale #infratructure
- get rid of trust / the opposite of martin said about changing who you trust over time
- Michael: having to pick someone to trust is a hard thing to do
it is hard to do so in a way that is not absolute or muddy
> wanting confidence without having to pick a point of failure
- Julien: I don't want to have users pick who to turst because it sounds complicated
Michael: tooling and infra to support multi party reproducibility verification at scale #scale
- Michael: we want to fund this
Holger: distributed verification as the second step after making 90+ % things of reproducible #fromPossibleToUsefulOrActionable
- explore difference between useful and actionable
- make everyone benefit by default (not just a process you seek out)
- get away from manual verification
Marc: RB creates resilience across a diverse set of build tools/diversity of build environments #security #notRelyOnASingleTool
- problem of monocultures?
Herve: as a user of a lib, I want ot make sure I can do a simple fix #constraintOnSolution
- make it sure it stays possible to
Martin: I want to be able to change who I trust over time
- Michael's reframe: want to be able to reason about my roots of trusti (and how they change over time) -> make tampering exponentially expensive for attackers
- reducing the number of roots of trust?
Julien: make roots of trust transparent / easy to users
Mark: I want to be able to trust a published artifact because someone I trust reproduced it
- Michael's reframe: I want to be able to choose who I trust
Michael: Reproducible build cost by sharing the load in a pool #cost #practicality
- everyone reproducing everything is not the solution
- contribute the pool of reproducibility / benefit from the pool of reproducibility
- pick whom I trust
Holger: if the russians, the cinese and the US agree, the build output is probably fine
- How to make it easy for developers and rebuilders to make the packages reproducible?
## Notes
- Taking lessons learned from one ecosystem and apply to others
- Java rebuilds / maven gradle and more
- Python and npm rebuilds
Some issues people have already encountered:
- Finding the right commit
- There might be multiple commits across multiple repositories (multirepo)
- There might be multiple projects in a single repo (monorepo)
- Finding the source material
- Intended fields for linking the source is not always used for the actual source
- Finding the build instructions
- attached signatures
- "poor deployment hygiene", i.e. publishing irrelevant files like DS store or vscode directories
- Split of internal and public repos, where you publish from an internal repo which may have slight differences
- unreproducible automation of build processes
- cache files
- finding the correct build tool in ecosystems that have multiple build tools available
- tooling that does not solve existing reproducibility problems, even if solutions are known, so the developers have to create their own implementations of these solutions
- Nondeterministic minification? Are they reproducible even if you know the exact build tool versions? Or are they generally non-deterinistic?
- Perhaps more general, optimization techniques for size or for performance, transformations like compilation, transpilation, ...
- On-the-fly patching java versions during the build process, apparently python sometimes does it as well.
- Sometimes build processes create local commits, use this as the commit sha from which the artifact was produced, and then publish it, but never push the respective commit into the public VCS.
- What trust do we put into a successfully rebuilt project? Do we assume the repository is the correct repo for future versions as well?
- Sometimes there is the same code in different locations, so it's hard to find what is the "real" source.
- Developers often don't care about reproducibility so they don't give rebuilders a lot of hints on how to build their package.
- Developers have no incentive to make reproducibility happen because they don't see any benefit.
- Should reproducibility be enforced?
- Perhaps automation, which may be a good way to facilitate more reproducible tooling, would be an incentive that makes things easier for developers.
-> what would be a good incentive for devs to want to use reproducible builds?
- big players might be able to make the process more reproducible, like people developing the build tools, so that developers don't have to care about it.
- Should perhaps sources have a metadata field pointing to the canonical location of the artifact?
- Sometimes for builds the very specific build version is important, but some other time it's fine to do it in some range of the build tool, and the less specific you can be the better you can scale up.
- How would you identify which version is acceptable?
- Perhaps the definition needs a bit of work so we can better differentiate between the environment, source and build instructions in order to make clear guidelines how to provide all of these three as a developer.
