TODO 26.9 KB
Newer Older
1 2
ToDo for jenkins.debian.net
===========================
3 4 5
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
6
:Status:           working, in progress
7 8 9 10
:lang:             en
:Doctype:          article
:Licence:	   GPLv2

11 12
== About jenkins.debian.net

13
See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian.net"] for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The links:https://jenkins.debian.net/userContent/contributing.html[the preferred form of contributions] are patches via pull requests.
14

Holger Levsen's avatar
Holger Levsen committed
15 16
== Fix user submitted bugs

17
* There are  link:https://bugs.debian.org/jenkins.debian.org[bugs filed against the pseudopackage 'jenkins.debian.org'] in the BTS which would be nice to be fixed rather sooner than later, as some people actually care to file bugs.
Holger Levsen's avatar
Holger Levsen committed
18

19 20
== meeting agenda for jenkins-qa meetings

21
* where: on #debian-qa on irc.oftc.net
Holger Levsen's avatar
Holger Levsen committed
22 23
* schedule: currently none
* next meeting: no idea
24 25 26

=== reoccuring agenda

27
* short intro, why are we here (aka: say hi)
28 29
* jenkins.d.n status
* jenkins.d.o migration next steps (see below for details)
30
* jenkins sprint in spring 2018? 
31 32 33 34 35 36 37 38
* AOB
* thanks to profitbricks for hosting

=== old meetings

* 2016-08-24: http://meetbot.debian.net/debian-qa/2016/debian-qa.2016-08-24-18.00.html
* 2016-09-28: http://meetbot.debian.net/debian-qa/2016/debian-qa.2016-09-28-19.02.html
* 2016-10: none
39
* 2016-11-23: http://meetbot.debian.net/debian-qa/2016/debian-qa.2016-11-23-18.06.html
40 41 42
* 2016-12: canceled because of 33C3
* 2017-01: none
* 2017-02-22: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-02-22-18.04.html
43
* 2017-03-22: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-03-22-18.01.html
44
* 2017-04-26: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-04-26-18.14.html
45
* 2017-08-21: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-08-23-18.30.html
46
* 2017-09-27: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-09-27-17.58.html
Holger Levsen's avatar
Holger Levsen committed
47

48 49
== General ToDo

Holger Levsen's avatar
Holger Levsen committed
50
* replace amd64 in scripts with $HOSTARCH
51
* extend /etc/rc.local to do cleanup of lockfiles
52
* explain in README how to write jobs, eg which pathes are on tmpfs
53
** EXECUTOR_NUMBER for X
54
* run all bash scripts with set -u and set -o pipefail: http://redsymbol.net/articles/unofficial-bash-strict-mode/
55
* teach bin/chroot-*.sh and bin/d-i_build.sh how to nicely deal with network problems… (as both reproducible_build.sh and schroot-create.sh do)
Holger Levsen's avatar
Holger Levsen committed
56
* use static IP for the nodes (h01ger)
57
* use vmdebootstrap where applicable
58
* this is outdated with salsa: add to all git post-receive hooks: `curl -s "https://jenkins.debian.net/git/notifyCommit?url=https://salsa.debian.org/installer-team/$(basename $PWD .git)"` which will trigger jenkins to pull (check) that git repo…
59 60
* salsa stuff:
** triggers for d-i repos are missing
Holger Levsen's avatar
Holger Levsen committed
61

62 63
=== new nodes at OSUOSL

64
* use those nodes:
65 66
** done: osuosl167: squid (only, atm)
** done: osuosl168: fdroid
67
*** underused. the old system had 2 cores and 8gb ram
68 69
** done: osuosl169: archlinux
** done: osuosl170: archlinux (future)
Holger Levsen's avatar
Holger Levsen committed
70
** done: osuosl171: openwrt, coreboot, netbsd
71
*** is constantly online (in jenkins) now, which is not how we like it and still doesnt solve the netbsd job probs
Holger Levsen's avatar
Holger Levsen committed
72
** done: osuosl172: openwrt, coreboot (future)
73
** osuosl173: debian rebuilder#1
74
** osuosl174: bwiedemann, opensuse tests
75
* add to README 
76
* add link to OSUOSL in THANKS and explain usage
77
* drop 'setup in progress' from r_html_nodes_info
78
* blog post when done
79 80 81 82 83 84 85 86 87 88 89 90 91 92 93
* setup netconsoles:
----
<guerby> | Ramereth, h01ger netconsole is about the ony way to diagnose this kind of issue in my experience
<guerby> | h01ger, setup rsyslogd on one of the gccserver then it's just one modprobe netconsole
<guerby> | h01ger, since the machine are on the same LAN
<guerby> (if not on the same LAN, modprobe netconsole netconsole=+@${IPSRC}/eth0,514@${IPRSYSLOG}/${GWMACADDR} )
<guerby> where GWMACADDR is 00:11:22... your gateway MAC
<guerby> saved the day lots of time here :)
<guerby> found this ext4 bug thanks to it, CVE : https://bugzilla.redhat.com/show_bug.cgi?id=1360968
<guerby> "Red Hat would like to thank Laurent Guerby for bringing this to our attention." and /me thanks netconsole :)
<guerby> https://bugzilla.kernel.org/show_bug.cgi?id=107301
<Ramereth> ya, netconsole is very nice to find odd kernel issues
<h01ger> | and you recommend to have that running all the time on all the nodes? or only when we suspect probs?
<Ramereth> we currently have it running all the time on all physical nodes
----
94

95 96
=== Debian rebuilder

97
* prototype only dealing with buster and amd64 (and arch all packages)
98 99 100 101 102 103 104 105 106
* run jenkins job on pb7 (=buildinfos.d.n)
* keep data in sqlite on pb7
** table shoud have these fields:
** src-pkg, binary-pkg, sha1 of .deb on ftp.d.o, sha1 of .buildinfo file (signed), sha1 of .buildinfo file (unsigned or signature stripped), signature type, name of .buildinfo file (from ftp.d.o)
*** and either architecture of .deb or filename of .deb (as that includes the architecture)
* export data as .json via https
* import data on jenkins
* schedule/trigger rebuilds on osuosl173, put data in sqlite on pb7

107 108 109 110 111
=== upgrades to buster

* only done on osuosl174 yet. 
** so we can drop hosts/osuosl-build174-amd64/etc/apt/sources.list once hosts/common/... has buster

112
==== proper backup
113

114
* postponed til we run on .debian.org
115
* this needs to be backed up:
116 117 118 119 120 121 122 123 124
* '/var/lib/jenkins/jobs' (the results - the configs are in .git)
* '/var/lib/munin'
* '/var/log'
* '/root/' (contains etckeeper.git)
* '/var/lib/jenkins/reproducible.db' (is backed up manually)
* '/srv/jenkins.debian.net-scm-sync.git' (is backed up manually)
* '/var/lib/jenkins/plugins/*.jpi' (can be derived from jdn-scm-sync.git)
* '/srv/jenkins.debian.net-scm-sync.git'
* '/etc/.git' and '/etc'
125

126 127
=== To be done once bugs are fixed

128 129
* link:https://bugs.debian.org/767100[#767100] work in progress in etc/munin/plugins/cpu
* link:https://bugs.debian.org/767018[#767018] work in progress in etc/munin/plugins/iostat_ios
130

131 132
=== jenkins-job-builder related

Holger Levsen's avatar
Holger Levsen committed
133
* investigate whether its possible nowadays to let it delete jobs which were removed.. nope. But there is a Makefile now which will find zombies…
134
* yaml should be refactored, lots of duplication in there. this seems to be helpful: http://en.wikipedia.org/wiki/YAML#References (pyyaml which jenkins-job-builder uses supports them)
Holger Levsen's avatar
Holger Levsen committed
135

Holger Levsen's avatar
Holger Levsen committed
136
=== debugging job runs should be made easy
137 138 139 140 141 142 143 144

----
 <      h01ger> | i think the jenkins-debug-job script should be a python script
 <      h01ger> | and j-j-b or another yaml parser can supply job configuration knowledge to that script
 <      h01ger> | \o/
 <      h01ger> | and that python script can also first determine whether the environment is as needed for the job, and if not, complain verbosely+helpfully and exit
----

Holger Levsen's avatar
Holger Levsen committed
145
== Improve existing tests
Holger Levsen's avatar
Holger Levsen committed
146

147
=== tests.reproducible-builds.org
148

149
==== General website
150

151 152 153
* install cbfstool in diffoscope schroots: (useful for openwrt+coreboot)
** 'git clone --recursive http://review.coreboot.org/p/coreboot.git ; cd coreboot/util/cbfstool ; make ; cp cbfstool $TARGET/usr/local/bin/'

154
* See https://wiki.debian.org/ReproducibleBuilds/TestsToDo for the tests.reproducible-builds.org related ToDo list.
155 156

=== Debian reproducible builds
157

Holger Levsen's avatar
Holger Levsen committed
158
* get rid off "set -x # # to debug diffoscoppe/schroot problems"
159
** add check if package to be build has been blacklisted since scheduling and abort
160
** on SIGTERM, also cleanup on remote build nodes there! (via ssh &)
Holger Levsen's avatar
Holger Levsen committed
161
** check rbuild logs for "DIFFOSCOPE='E: Failed to change to directory /tmp: Permission denied' and deal with those
162

163
* higher prio:
Holger Levsen's avatar
Holger Levsen committed
164
** reenable disorderfs setup, check that it *always* unmounts + cleans up nicely
165 166 167
** pkg pages
*** new table in pkg/test history page: schedule - if that package is currently scheduled
*** add link to pkg set(s) if pkg is member of some
168
** link pkg sets and issues, that is: at least show packages without issues on pkg set pages, maybe also some issues which need actions (like uninvestigated test failures)
Holger Levsen's avatar
Holger Levsen committed
169 170
** notes related:
*** #786396: classify issue by "toolchain" or "package" fix needed: show bugs which block a bug
171
*** new page with annoted packages without categorized issues (and probably without bugs as only note content too, else there are too many)
Holger Levsen's avatar
Holger Levsen committed
172 173 174 175
*** new page with packages that have notes with comments (which are often useful / contain solutions / low-hanging fruits for newcomers)
*** new page with notes that doesnt make sense: a.) packages which are reproducible but should not, packages that build but shouldn't, etc.
*** new page with packages which are reproducible on one arch and unreproducible on another arch (in the same suite, so unstable only atm)
*** new page with packages which ftbfs on one arch and build fine on another arch (in the same suite, so unstable only atm)
176
*** new page with packages which ftbfs in stretch but build fine on sid
Holger Levsen's avatar
Holger Levsen committed
177
*** new page with packages which are orphaned but have a reproducible usertagged patch
Holger Levsen's avatar
Holger Levsen committed
178
*** new page showing arch all packages which are cross-reproducible, and those which are not
179
** new pages: r.d.n/$maintainer-email redirecting to r.d.n/maintainers/unstable/${maintainer-email}.html, showing the unreproducible packages for that address. and a sunny "yay, thank you"-summary for those with only reproducible packages.
180
** new page: "open bugs with patches, sorted by maintainers" page and to the navigation, make those NMUable bugs visible
Holger Levsen's avatar
Holger Levsen committed
181
** improve ftbfs page: list packages without bugs and notes first
182
** bin/_html_indexes.py: bugs = get_bugs() # this variable should not be global, else merely importing _html_indexes always queries UDD
183
** once firefox 48 is available: revert 1b4dc1b3191e3623a0eeb7cacef80be1ab71d0a2 / grep for _js and remove it…
184 185

* lesser prio
186
** scheduler: check if there have been more than X failures or depwait in the last Y hours and if so unschedule all packages, disable scheduling and send a mail informing us.
187
** pkg sets related:
188
*** add new pkg set: torbrowser-build-depends
189
*** fix essential set: currently it only has the ones explicitly marked Essential:yes; they and their dependencies make up the full "essential closure set" (sometimes also called pseudo-essential)
Holger Levsen's avatar
Holger Levsen committed
190
*** replace bin/reproducible_installed_on_debian.org with a proper data provider from DSA, eg https://salsa.debian.org/dsa-team/mirror/debian.org/blob/master/debian/control
191
** a reproducible_log_grep_by_sql.(py|sh) would be nice, to only grep in packages with a certain status (build in the last X days)
192 193
** database issues
*** stats_build table should have package ids, not just src+suite+arch as primary key
194
*** move "untested" field in stats table too? (as in csv output...)
195
** blacklist script should tell if a package was already blacklisted. also proper options should be used...
196
** maintenance.sh: delete the history pages once a page has been removed from all suites+archs
197
** reproducible.debian.net rename: rgrep all the files…
198
** debbindiff2diffoscope rename: do s#dbd#ds#g and s#DBD#DS#g and rename dbd directories?
199
** diffoscope needs to be run on the target arch... (or rather: run on a 64bit architecture for 64bit architectures and on 32bit for 32 bit archs), this should probably be doable with a simple i386 chroot on the host (so using qemu-static to run it on armhf should not be needed, probably.)
200 201 202
** support for arbitrary (to be implemented) Debian-PPAs and external repos, by just giving a source URL
** once stabilized notification emails should go through the package tracker.  The 'build' keyword seems to be the better fit for this.  To do so just send the emails to dispatch@tracker.debian.org, setting "X-Distro-Tracker-Package: foo" and "X-Distro-Tracker-Keyword: build".  This way people wanting to subscribe to our notification don't need to ask us and can do that by themselves.
** repo-comparison: check for binaries without source
203
** issues: currently only state of amd64 is shown. it would be better to display packages as unreproducible if they are unreproducible on any architecture.
204
** include diffoscope run time in log
205 206 207 208 209 210 211 212 213
* for _service.sh
** enabling the service in update_jdn
** maintenance job might want to:
*** check for running builds using systemctl show & ps fax
*** check if at least one build is running on a build node
** use ExecStop to kill remote builds on shutdown
*** remove cleanup_nodes.sh once this works
*** add support for disabling archs and for shutdown+respan of workers
*** add support for starting/stopping workers for specific archs
214
* missing variations:
215
** 32/64 bit kernel variation on i386
216
** prebuilder does (user) group variation like this: https://salsa.debian.org/reproducible-builds/reproducible-misc/blob/master/prebuilder/pbuilderhooks/A02_user
Holger Levsen's avatar
Holger Levsen committed
217
** variation of $TERM and $COLUMN (and maybe $LINES), unset in the first run, set to "linux" and "77" (and maybe "42") in the 2nd run.
218
*** actually TERM is set to "linux" by default already, COLUMN is unset
219
** vary order of $PATH entries, see #844500
220
** have redundant $PATH entries (`/bin:/bin:…`)
Mattia Rizzolo's avatar
Mattia Rizzolo committed
221
** vary the length of the build paths (/build/first vs /build/second), only once the unreproducibilities caused by different build paths are solved
222
** vary the init system: sysv and systemd
223
** vary (with) usrmerge and without (needs debootstrap from jessie-backports) once #843073 is fixed (#810499 might be relevant as well)
224
** vary SSD/HDD on i386?
225
** vary temp dir variables such as TMP/TMPDIR/TEMP/TEMPDIR/MAGICK_TMPDIR
226
** maybe vary build with pbuilder and sbuild (but maybe only useful with different setup jobs only…)
Holger Levsen's avatar
Holger Levsen committed
227

228 229
==== reproducible Debian armhf

230 231 232
* rename all the nodes from $HOSTNAME to $HOSTNAME-armhf-rb ?
** we could get rid of the links in jenkins.d.n.git/hosts/
** we could simplefy .../hosts/*/etc/munin/munin-node.conf
233
* make systems send mail, use port 465
234

235 236
==== reproducible Debian arm64

237
* vary DEB_BUILD_OPTS? (NUM_CPU)
238

239 240 241
==== reproducible Debian installation

* see https://wiki.debian.org/ReproducibleInstalls
Holger Levsen's avatar
Holger Levsen committed
242
* run this as a new job
243

244 245 246
==== reproducible non-Debian tests, new host for 398 day variation is unused

* locations in the code which need to be changed
247 248
** RPM_BUILD_NODE=osuosl-build171-amd64
** grep for osuosl-build171-amd64, there's more
249 250 251 252 253 254
* IOW: these tests should use it:
** coreboot
** netbsd
** fedora
** (fdroid)

255 256 257 258 259 260 261 262 263 264 265 266 267
==== reproducible coreboot

* add more variations: domain+hostname, uid+gid, USER, UTS namespace
* build the docs?
* also build with payloads. x86 use seabios as default, arm boards dont have a default. grub is another payload. and these: bayou  coreinfo  external  filo  libpayload  nvramcui - and:
** CONFIG_PAYLOAD_NONE=y
** CONFIG_PAYLOAD_ELF is not set
** CONFIG_PAYLOAD_LINUX is not set
** CONFIG_PAYLOAD_SEABIOS is not set
** CONFIG_PAYLOAD_FILO is not set
** CONFIG_PAYLOAD_GRUB2 is not set
** CONFIG_PAYLOAD_TIANOCORE is not set
* libreboot ships images, verify those?
268
* explain status in plain english
269
* use disorderfs for 2nd build
270

271

272 273 274 275
==== reproducible OpenWrt

* add credit for logo/artwork
* explain status in plain english
276 277
* build path variation
* use disorderfs for 2nd build
278
* incorporate popular third-party ("external feeds") packages?
279 280 281
* html: build variations are wrong
* html: git commit output includes garbage
* html: css: add some space on the left side
282

283
==== reproducible NetBSD
284

285
* explain status in plain english
286
** explain MKREPRO is set to "yes"
287
** explain MKREPRO_TIMESTAMP set to $SOURCE_DATE_EPOCH
288
* use disorderfs for 2nd build
289

290
==== reproducible FreeBSD
291

292
* useful improvements:
293 294
** investigate how to use tmpfs on freebsd and build there. see mdmfs(8)
** find a way to be informed about updates and keep it updated - see 'freebsd-update cron' and 'pkg audit'.  The latter is run periodic(8) as part of the nightly root@ emails.
295
** modify PATH, uid, gid and USER too and host+domainname as well. The VM is only used for this, so we could change the host+domainname temporaily between builds too.
296
** add freebsd vm as node to jenkins and run the script directly there, saves lot of ssh hassle
297
** run diffoscope nativly
298 299 300 301 302 303 304 305

* random notes, to be moved to README
** we build freebsd 10.1 (=released) atm
** we build with sudo too
*** rather not change /usr/obj to be '~jenkins/obj' and build with WITH_INSTALL_AS_USER. also not build in /usr/src. if so, we need to define some variable so we can do so.... but we need a stable path anyway, so whats the point.
*** maybe build as user in /usr/src...
* first build world, later build ports (pkg info...)

306
* document how the freebsd build VM was set up:
307
** base 10.1 install following https://www.urbas.eu/freebsd-10-and-profitbricks/
308 309 310 311
** modified files:
*** /etc/rc.conf
*** /etc/resolv.conf
*** /boot/loader.conf.local
312
** pkg install screen git vim sudo denyhosts munin-node poudriere
313 314
*** configure /usr/local/etc/munin/munin-node.conf to allow jenkins to access it
*** configure /usr/local/etc/denyhosts.conf and /etc/hosts.allow and touch /etc/hosts.deniedssh
315 316
** adduser holger
** adduser jenkins (with bash as default shell)
317
** adduser mattia
318 319
** mkdir -p /srv/reproducible-results
** chown -R jenkins:jenkins /srv/
320

321
* system maintenance
322
** upgraded the VM to FreeBSD 11.0
323
*** done with: 'freebsd-update upgrade -r 10.2' as root in screen
324
*** and with:  'freebsd-update upgrade -r 10.3'
325
*** and with:  'freebsd-update upgrade -r 11.0'
326 327
*** and with:  'freebsd-update upgrade -r 11.1'
*** and with:  'freebsd-update upgrade -r 11.2' followed by 'pkg-static install pkg ; pkg upgrade'
328

329 330
* online disk resizing howto: https://www.freebsd.org/doc/handbook/disks-growing.html

331 332
==== reproducible Fedora

333 334
* make sure the pages meet https://fedoraproject.org/wiki/Design/Requirements
 and ask the web design team for help via filing a ticket as described there
335 336
* '/var/cache/mock/fedora-23-x86_64/' has three subdirs we need to handle (put on tmpfs, monitor size, clean sometimes): ccache, root_cache and  yum_cache
* '/var/lib/mock' should be put on /srv/workspace aka tmpfs
337
* dont hardcode 23 in reproducible_setup_mock.sh and …build_rpm.sh
338 339 340
* setup script:
** mock --clean just uninstalls the chroot but it'll still be rebuilt next time using cache.  you can delete the caches from /var/cache/mock/ or touch the mock config
** is /etc/yum/repos.d/fedora.repo really needed?
341
** hosts/osuosl-build171/etc/yum/repos.d/* is really not sooo good but works…
342
* build script
343
** cleanup mock cache between two builds: --scrub=all might be too much, but whats sensible (or is it --scrub=all?)?
344 345 346 347
** no variations introduced yet:
*** use '-j$NUM_CPU' and 'NEW_NUM_CPU=$(echo $NUM_CPU-1|bc)'
*** modify TZ, LANG, LC_ALL, umask
* other bits:
348 349
** use modified rpmbuild package from dhiru
** verify gpg signatures (via /etc/mock/)
350
** one day we will want to schedule all 17k source packages in fedora…
351
* build rawhide too (once fedora-23 builds nicely), releasever=rawhide
352 353

* more notes:
354 355
** https://fedoraproject.org/wiki/Using_Mock_to_test_package_builds
** http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_-_build_packages_in_memory/index.html
356
** manually create a fedora chroot using rpm, wget + yum: http://geek.co.il/2010/03/14/how-to-build-a-chroot-jail-environment-for-centos
357

358
==== reproducible Arch Linux
359

360 361
* setup_archlinux_schroot job:
** needs to be made idempotent (currently it removes the schroot at the beginning of the job, instead of creating it elsewhere and replacing it on success at the job end…)
Mattia Rizzolo's avatar
Mattia Rizzolo committed
362
** use schroot tarballs (gzipped), moves are atomic then
363
* only disable cert checking on the node running in the future
364
* compare the just built pkg.tar.xz with the one available on the arch mirrors. *then* one can truely say "X% of the Arch Linux packages are reproducible and could bit by bit be reproduced in the real world."
365 366
* maintenance job:
** check for archlinux schroot sessions which should not be there and delete them. complain if that fails.
367

368
* use db - see https://tests.reproducible-builds.org/reproducibledb.htm
369
** scheduler.sh:
370
*** use asp to update trunk packages?
371
** html:
372
*** leave all pkg.html files, delete them (much) later, then check all pkg.* files are gone
373
*** recreate them newly as index.html
374
**** link to https://www.archlinux.org/packages/$repo/x86_64/$pkgname/
375
**** provide links to debian pages if same package name exist
376 377 378
*** create new job to recreate all pkg pages
*** create json
** misc:
379
*** write into all the stats_ tables
380
*** make build.sh respect pacman exit code, see FIXME in _html_.sh
381
*** comparing versions (in scheduler) is probably needed (though it wuld be better if not): if its not empty it must be higher (due to repo constraints), but it can be even higher than in the repo, because we build trunk
382 383
*** debian uses a table removed_packages, should this be used here too?
*** debian uses a table manual_scheduler to limit the amount of packages someone can schedule per day...
384
*** should breakages job look for archlinux breakages? for which cases? a.) grep "was killed after running into timeout after 30m" pkg.html
385
** using notes.git should be next!
386

387
* rebuilding against the archlinux repos
388
** only needs to be done once using https://github.com/Foxboron/devtools-repro
389
** to not (pacman know about trunk and thus) build trunk, we need to configure asp.
390
** we'll keep building against repo+trunk as we do now (so that archlinux can also benefit from the QA effects)
391

392
* fix build.sh:
393
** build2.log doesnt get deleted if build1 fails
394
** -> rename build2.log to $version_build2.log (dont include package name...)
395

396 397 398
* things to be done before enabling more builders:
** build in /srv/workspace instead of /tmp (once this has been done reduce /tmp size back to 15G)

399
* build on OSUOSL nodes:
400
** update thanks & readme
401

402 403
==== reproducible fdroid

404
* reproducible_setup_fdroid_buildserver.sh:
405
** ./jenkins-build-makebuildserver
406
*** manually added the jenkins user to the vboxdrv group
407 408
*** this downloads a base debian image and all Android tarballs (SDK, NDK, Gradle...)
*** then enters the image, installs all debian packages and Android stuff from the cached tarballs
409 410
** cache is kept outside ('~/.cache/fdroidserver') but installed inside
** '~/.cache/fdroidserver' needs to be cleaned at some times…
411 412

* reproducible_build_fdroid_apk.sh
413 414 415 416 417 418 419
** 1st run ./fdroid build some.app:vercode --server
** 2nd run ./fdroid build some.app:vercode --server
*** eg: org.fdroid.fdroid:98006
*** or: "fdroid build -l org.fdroid.fdroid" to build the latest
** run diffopscope on the results

* also see https://f-droid.org/wiki/page/Build_Server_Setup
420

421 422 423 424 425 426 427 428 429 430
* diskspace needs:
----
$ du -hd1 | sort -h | tail -n 6
4.4G    ./android-sdk-linux_86
8.1G    ./fdroidserver
8.3G    ./.vagrant.d
71G     ./VirtualBox VMs
150G    ./fdroiddata
242G    .
----
431

432 433
==== reproducible qubes

434
* add qubes test on t.r-b.o
435 436 437 438 439 440 441 442 443
----
        git clone https://github.com/qubesos/qubes-builder
        make get-sources BUILDERCONF=scripts/travis-builder.conf COMPONENTS=installer-qubes-os
        export DIST_DOM0=fc23
        export USE_QUBES_REPO_VERSION=3.2
        export INSTALLER_KICKSTART=/tmp/qubes-installer/conf/travis-iso.ks

        make qubes iso BUILDERCONF=scripts/travis-builder.conf VERBOSE=0 COMPONENTS=installer-qubes-os
----
444
* depends:  apt install createrepo python-yaml
445 446
* once this iso is being tested, it will be interesting to build the Qubes templates as well, as those images (Qubes templates are images) will be copied on the installation iso. the above iso is a stripped down iso without templates… (and not the real thing)

447 448 449 450 451
==== reproducible guix

* there's no "apt-get install", because of non-FHS conformance, but see https://www.gnu.org/software/guix/download/
*  there's a privileged build daemon, which is needed to perform fully isolated builds, see https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html#Build-Environment-Setup
* it's a bit of work to set up, but all the steps are documented. the "binary installation" method being the easiest.
452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467 468 469 470 471 472 473 474 475 476 477 478 479 480 481 482
* Manolis wrote:
----
There are two ways to install guix, through prebuilt binaries or through
the source.

*Binary installation:

Go to
<http://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html>,
grab the tarball and follow the instructions there.

*Source instalation:

First make sure you have the dependencies mentioned at
<http://www.gnu.org/software/guix/manual/guix.html#Requirements> installed.

Then download Guix's source from
ftp://alpha.gnu.org/gnu/guix/guix-0.9.0.tar.gz and use the usual
./configure && make && make install

After you have Guix built, you need to create the build-users and have
the guix-daemon run as root, as described here
<https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html>.

Keep in mind that the guix-daemon must always run as root.

*Testing if everything works:

Now just run `guix package -i vim` as a non-root user. If it runs
correctly, Guix is ready for work.
----
483

484 485 486
==== reproducible...

* openembedded.org!
487
* Gentoo?
488

Holger Levsen's avatar
Holger Levsen committed
489 490
=== qa.debian.org*

Holger Levsen's avatar
Holger Levsen committed
491
* udd-versionskew: explain jobs in README
492
* udd-versionskew: also provide arch-relative version numbers in output too
Holger Levsen's avatar
Holger Levsen committed
493

Holger Levsen's avatar
Holger Levsen committed
494 495
=== d-i_manual*

496
* d-i_check_jobs.sh: check for removed manuals (but with existing jobs) missing
Holger Levsen's avatar
Holger Levsen committed
497 498 499 500 501
* svn:trunk/manual/po triggers the full build, should trigger language specific builds.
* svn:trunk/manual is all thats needed, not whole svn:trunk

=== d-i_build*

502
* d-i_check_jobs.sh: check for removed package (but with existing jobs) missing
Holger Levsen's avatar
Holger Levsen committed
503
* build packages using jenkins-debian-glue and not with the custom scripts used today?
Holger Levsen's avatar
Holger Levsen committed
504 505 506 507
* run scripts/digress/ ?

=== chroot-installation_*

Holger Levsen's avatar
Holger Levsen committed
508
* use schroot for chroot-installation, stop using plain chroot everywhere
509 510
** https://salsa.debian.org/dsa-team/mirror/dsa-puppet/tree/master/modules/schroot
** https://salsa.debian.org/dsa-team/mirror/dsa-puppet/tree/master/modules/porterbox/files/dd-schroot-cmd
Holger Levsen's avatar
Holger Levsen committed
511
** https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/build-wrapper
512
* add alternative tests with aptitude and possible apt
Holger Levsen's avatar
Holger Levsen committed
513
* split etc/schroot/default
Holger Levsen's avatar
Holger Levsen committed
514
* inform debian-devel@l.d.o or -qa@?
Holger Levsen's avatar
Holger Levsen committed
515
* warn about transitional packages installed (on non-upgrades only)
Holger Levsen's avatar
Holger Levsen committed
516
* install all the tasks "instead", thats rather easy nowadays as all task packages are called "task*".
Holger Levsen's avatar
Holger Levsen committed
517
** make sure this includes blends
Holger Levsen's avatar
Holger Levsen committed
518 519 520

=== g-i-installation_*

521
Development of these tests has stopped. In future we will use https://openqa.debian.net instead.
Holger Levsen's avatar
Holger Levsen committed
522

Holger Levsen's avatar
Holger Levsen committed
523

Holger Levsen's avatar
Holger Levsen committed
524
== Further ideas...
Holger Levsen's avatar
Holger Levsen committed
525 526


527 528 529 530 531 532
=== rebuild sid completly on demand

* nthykier wants to be able to rebuild all of sid to test how changes to eg lintian, debhelper, cdbs, gcc affect the archive:
* h01ger> | nthykier: so a.) rebuild everything from sid plus custom repo. b.) option to only rebuild a subset, like all rdepends or all packages build-depending on something
* h01ger> | and c.) only build once, not continously and d.) enable more cores+ram on demand to build faster

Holger Levsen's avatar
Holger Levsen committed
533 534
* have a job to trigger such a rebuild on AWS?

Holger Levsen's avatar
Holger Levsen committed
535
=== Test them all
536

537
* build packages from all team repos on alioth with jenkins-debian-glue on team request (eg, via a .txt file in a git.repo) for specific branches (which shall also be automated, eg. to be able to only have jessie+sid branches build, but not all other branches.)
538

Holger Levsen's avatar
Holger Levsen committed
539
== Debian Packaging related
540

Holger Levsen's avatar
Holger Levsen committed
541
This setup should come as a Debian source package...
542

Holger Levsen's avatar
Holger Levsen committed
543 544
* /usr/sbin/jenkins.debian.net-setup needs to be written
* what update-j.d.n.sh does, needs to be put elsewhere...
Holger Levsen's avatar
Holger Levsen committed
545 546 547 548
* debian/copyright is incorrect about some licenses:
** the profitbricks+debian+jenkins logos
** the preseeding files
** ./feature/ is gpl3
549

Holger Levsen's avatar
Holger Levsen committed
550
// vim: set filetype=asciidoc: