TODO 26.2 KB
Newer Older
1 2
ToDo for jenkins.debian.net
===========================
3 4 5
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
6
:Status:           working, in progress
7 8 9 10
:lang:             en
:Doctype:          article
:Licence:	   GPLv2

11 12
== About jenkins.debian.net

13
See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian.net"] for a general description of the setup. Below is the current TODO list, which is long and probably incomplete too. The links:https://jenkins.debian.net/userContent/contributing.html[the preferred form of contributions] are patches via pull requests.
14

Holger Levsen's avatar
Holger Levsen committed
15 16
== Fix user submitted bugs

17
* There are  link:https://bugs.debian.org/jenkins.debian.org[bugs filed against the pseudopackage 'jenkins.debian.org'] in the BTS which would be nice to be fixed rather sooner than later, as some people actually care to file bugs.
Holger Levsen's avatar
Holger Levsen committed
18

19 20
== meeting agenda for jenkins-qa meetings

21
* where: on #debian-qa on irc.oftc.net
Holger Levsen's avatar
Holger Levsen committed
22 23
* schedule: currently none
* next meeting: no idea
24 25 26

=== reoccuring agenda

27
* short intro, why are we here (aka: say hi)
28 29
* jenkins.d.n status
* jenkins.d.o migration next steps (see below for details)
30
* jenkins sprint in spring 2018? 
31 32 33 34 35 36 37 38
* AOB
* thanks to profitbricks for hosting

=== old meetings

* 2016-08-24: http://meetbot.debian.net/debian-qa/2016/debian-qa.2016-08-24-18.00.html
* 2016-09-28: http://meetbot.debian.net/debian-qa/2016/debian-qa.2016-09-28-19.02.html
* 2016-10: none
39
* 2016-11-23: http://meetbot.debian.net/debian-qa/2016/debian-qa.2016-11-23-18.06.html
40 41 42
* 2016-12: canceled because of 33C3
* 2017-01: none
* 2017-02-22: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-02-22-18.04.html
43
* 2017-03-22: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-03-22-18.01.html
44
* 2017-04-26: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-04-26-18.14.html
45
* 2017-08-21: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-08-23-18.30.html
46
* 2017-09-27: http://meetbot.debian.net/debian-qa/2017/debian-qa.2017-09-27-17.58.html
Holger Levsen's avatar
Holger Levsen committed
47

48 49
== General ToDo

Holger Levsen's avatar
Holger Levsen committed
50
* replace amd64 in scripts with $HOSTARCH
51
* extend /etc/rc.local to do cleanup of lockfiles
52
* explain in README how to write jobs, eg which pathes are on tmpfs
53
** EXECUTOR_NUMBER for X
54
* run all bash scripts with set -u and set -o pipefail: http://redsymbol.net/articles/unofficial-bash-strict-mode/
55
* teach bin/chroot-*.sh and bin/d-i_build.sh how to nicely deal with network problems… (as both reproducible_build.sh and schroot-create.sh do)
Holger Levsen's avatar
Holger Levsen committed
56
* use static IP for the nodes (h01ger)
57
* use vmdebootstrap where applicable
58
* this is outdated with salsa: add to all git post-receive hooks: `curl -s "https://jenkins.debian.net/git/notifyCommit?url=https://salsa.debian.org/installer-team/$(basename $PWD .git)"` which will trigger jenkins to pull (check) that git repo…
59 60
* salsa stuff:
** triggers for d-i repos are missing
Holger Levsen's avatar
Holger Levsen committed
61

62 63
=== new nodes at OSUOSL

64
* use those nodes,current plan:
65 66
** done: osuosl167: squid (only, atm)
** done: osuosl168: fdroid
67
*** underused. the old system had 2 cores and 8gb ram
68
*** shutdown pb7
69
*** inform hans, also about better specs
70 71
** done: osuosl169: archlinux
** done: osuosl170: archlinux (future)
Holger Levsen's avatar
Holger Levsen committed
72
** done: osuosl171: openwrt, coreboot, netbsd
73
*** is constantly online (in jenkins) now, which is not how we like it and still doesnt solve the netbsd job probs
Holger Levsen's avatar
Holger Levsen committed
74
** done: osuosl172: openwrt, coreboot (future)
75 76
** osuosl173: debian rebuilder#1
** osuosl174:
77
* add to README 
78
* add link to OSUOSL in THANKS and explain usage
79
* drop 'setup in progress' from r_html_nodes_info
80
* blog post when done
81 82 83 84 85 86 87 88 89 90 91 92 93 94 95
* setup netconsoles:
----
<guerby> | Ramereth, h01ger netconsole is about the ony way to diagnose this kind of issue in my experience
<guerby> | h01ger, setup rsyslogd on one of the gccserver then it's just one modprobe netconsole
<guerby> | h01ger, since the machine are on the same LAN
<guerby> (if not on the same LAN, modprobe netconsole netconsole=+@${IPSRC}/eth0,514@${IPRSYSLOG}/${GWMACADDR} )
<guerby> where GWMACADDR is 00:11:22... your gateway MAC
<guerby> saved the day lots of time here :)
<guerby> found this ext4 bug thanks to it, CVE : https://bugzilla.redhat.com/show_bug.cgi?id=1360968
<guerby> "Red Hat would like to thank Laurent Guerby for bringing this to our attention." and /me thanks netconsole :)
<guerby> https://bugzilla.kernel.org/show_bug.cgi?id=107301
<Ramereth> ya, netconsole is very nice to find odd kernel issues
<h01ger> | and you recommend to have that running all the time on all the nodes? or only when we suspect probs?
<Ramereth> we currently have it running all the time on all physical nodes
----
96

97
==== proper backup
98

99
* postponed til we run on .debian.org
100
* this needs to be backed up:
101 102 103 104 105 106 107 108 109
* '/var/lib/jenkins/jobs' (the results - the configs are in .git)
* '/var/lib/munin'
* '/var/log'
* '/root/' (contains etckeeper.git)
* '/var/lib/jenkins/reproducible.db' (is backed up manually)
* '/srv/jenkins.debian.net-scm-sync.git' (is backed up manually)
* '/var/lib/jenkins/plugins/*.jpi' (can be derived from jdn-scm-sync.git)
* '/srv/jenkins.debian.net-scm-sync.git'
* '/etc/.git' and '/etc'
110

111 112
=== To be done once bugs are fixed

113 114
* link:https://bugs.debian.org/767100[#767100] work in progress in etc/munin/plugins/cpu
* link:https://bugs.debian.org/767018[#767018] work in progress in etc/munin/plugins/iostat_ios
115

116 117
=== jenkins-job-builder related

Holger Levsen's avatar
Holger Levsen committed
118
* investigate whether its possible nowadays to let it delete jobs which were removed.. nope. But there is a Makefile now which will find zombies…
119
* yaml should be refactored, lots of duplication in there. this seems to be helpful: http://en.wikipedia.org/wiki/YAML#References (pyyaml which jenkins-job-builder uses supports them)
Holger Levsen's avatar
Holger Levsen committed
120

Holger Levsen's avatar
Holger Levsen committed
121
=== debugging job runs should be made easy
122 123 124 125 126 127 128 129

----
 <      h01ger> | i think the jenkins-debug-job script should be a python script
 <      h01ger> | and j-j-b or another yaml parser can supply job configuration knowledge to that script
 <      h01ger> | \o/
 <      h01ger> | and that python script can also first determine whether the environment is as needed for the job, and if not, complain verbosely+helpfully and exit
----

Holger Levsen's avatar
Holger Levsen committed
130
== Improve existing tests
Holger Levsen's avatar
Holger Levsen committed
131

132
=== tests.reproducible-builds.org
133

134
==== General website
135

136 137 138
* install cbfstool in diffoscope schroots: (useful for openwrt+coreboot)
** 'git clone --recursive http://review.coreboot.org/p/coreboot.git ; cd coreboot/util/cbfstool ; make ; cp cbfstool $TARGET/usr/local/bin/'

139
* See https://wiki.debian.org/ReproducibleBuilds/TestsToDo for the tests.reproducible-builds.org related ToDo list.
140 141

=== Debian reproducible builds
142

Holger Levsen's avatar
Holger Levsen committed
143
* get rid off "set -x # # to debug diffoscoppe/schroot problems"
144
** add check if package to be build has been blacklisted since scheduling and abort
145
** on SIGTERM, also cleanup on remote build nodes there! (via ssh &)
Holger Levsen's avatar
Holger Levsen committed
146
** check rbuild logs for "DIFFOSCOPE='E: Failed to change to directory /tmp: Permission denied' and deal with those
147

148
* higher prio:
Holger Levsen's avatar
Holger Levsen committed
149
** reenable disorderfs setup, check that it *always* unmounts + cleans up nicely
150 151 152
** pkg pages
*** new table in pkg/test history page: schedule - if that package is currently scheduled
*** add link to pkg set(s) if pkg is member of some
153
** link pkg sets and issues, that is: at least show packages without issues on pkg set pages, maybe also some issues which need actions (like uninvestigated test failures)
Holger Levsen's avatar
Holger Levsen committed
154 155
** notes related:
*** #786396: classify issue by "toolchain" or "package" fix needed: show bugs which block a bug
156
*** new page with annoted packages without categorized issues (and probably without bugs as only note content too, else there are too many)
Holger Levsen's avatar
Holger Levsen committed
157 158 159 160
*** new page with packages that have notes with comments (which are often useful / contain solutions / low-hanging fruits for newcomers)
*** new page with notes that doesnt make sense: a.) packages which are reproducible but should not, packages that build but shouldn't, etc.
*** new page with packages which are reproducible on one arch and unreproducible on another arch (in the same suite, so unstable only atm)
*** new page with packages which ftbfs on one arch and build fine on another arch (in the same suite, so unstable only atm)
161
*** new page with packages which ftbfs in stretch but build fine on sid
Holger Levsen's avatar
Holger Levsen committed
162
*** new page with packages which are orphaned but have a reproducible usertagged patch
Holger Levsen's avatar
Holger Levsen committed
163
*** new page showing arch all packages which are cross-reproducible, and those which are not
164
** new pages: r.d.n/$maintainer-email redirecting to r.d.n/maintainers/unstable/${maintainer-email}.html, showing the unreproducible packages for that address. and a sunny "yay, thank you"-summary for those with only reproducible packages.
165
** new page: "open bugs with patches, sorted by maintainers" page and to the navigation, make those NMUable bugs visible
Holger Levsen's avatar
Holger Levsen committed
166
** improve ftbfs page: list packages without bugs and notes first
167
** bin/_html_indexes.py: bugs = get_bugs() # this variable should not be global, else merely importing _html_indexes always queries UDD
168
** once firefox 48 is available: revert 1b4dc1b3191e3623a0eeb7cacef80be1ab71d0a2 / grep for _js and remove it…
169 170

* lesser prio
171
** scheduler: check if there have been more than X failures or depwait in the last Y hours and if so unschedule all packages, disable scheduling and send a mail informing us.
172
** pkg sets related:
173
*** add new pkg set: torbrowser-build-depends
174
*** fix essential set: currently it only has the ones explicitly marked Essential:yes; they and their dependencies make up the full "essential closure set" (sometimes also called pseudo-essential)
Holger Levsen's avatar
Holger Levsen committed
175
*** replace bin/reproducible_installed_on_debian.org with a proper data provider from DSA, eg https://salsa.debian.org/dsa-team/mirror/debian.org/blob/master/debian/control
176
** a reproducible_log_grep_by_sql.(py|sh) would be nice, to only grep in packages with a certain status (build in the last X days)
177 178
** database issues
*** stats_build table should have package ids, not just src+suite+arch as primary key
179
*** move "untested" field in stats table too? (as in csv output...)
180
** blacklist script should tell if a package was already blacklisted. also proper options should be used...
181
** maintenance.sh: delete the history pages once a page has been removed from all suites+archs
182
** reproducible.debian.net rename: rgrep all the files…
183
** debbindiff2diffoscope rename: do s#dbd#ds#g and s#DBD#DS#g and rename dbd directories?
184
** diffoscope needs to be run on the target arch... (or rather: run on a 64bit architecture for 64bit architectures and on 32bit for 32 bit archs), this should probably be doable with a simple i386 chroot on the host (so using qemu-static to run it on armhf should not be needed, probably.)
185 186 187
** support for arbitrary (to be implemented) Debian-PPAs and external repos, by just giving a source URL
** once stabilized notification emails should go through the package tracker.  The 'build' keyword seems to be the better fit for this.  To do so just send the emails to dispatch@tracker.debian.org, setting "X-Distro-Tracker-Package: foo" and "X-Distro-Tracker-Keyword: build".  This way people wanting to subscribe to our notification don't need to ask us and can do that by themselves.
** repo-comparison: check for binaries without source
188
** issues: currently only state of amd64 is shown. it would be better to display packages as unreproducible if they are unreproducible on any architecture.
189
** include diffoscope run time in log
190 191 192 193 194 195 196 197 198
* for _service.sh
** enabling the service in update_jdn
** maintenance job might want to:
*** check for running builds using systemctl show & ps fax
*** check if at least one build is running on a build node
** use ExecStop to kill remote builds on shutdown
*** remove cleanup_nodes.sh once this works
*** add support for disabling archs and for shutdown+respan of workers
*** add support for starting/stopping workers for specific archs
199
* missing variations:
200
** 32/64 bit kernel variation on i386
201
** prebuilder does (user) group variation like this: https://salsa.debian.org/reproducible-builds/reproducible-misc/blob/master/prebuilder/pbuilderhooks/A02_user
Holger Levsen's avatar
Holger Levsen committed
202
** variation of $TERM and $COLUMN (and maybe $LINES), unset in the first run, set to "linux" and "77" (and maybe "42") in the 2nd run.
203
*** actually TERM is set to "linux" by default already, COLUMN is unset
204
** vary order of $PATH entries, see #844500
205
** have redundant $PATH entries (`/bin:/bin:…`)
Mattia Rizzolo's avatar
Mattia Rizzolo committed
206
** vary the length of the build paths (/build/first vs /build/second), only once the unreproducibilities caused by different build paths are solved
207
** vary the init system: sysv and systemd
208
** vary (with) usrmerge and without (needs debootstrap from jessie-backports) once #843073 is fixed (#810499 might be relevant as well)
209
** vary SSD/HDD on i386?
210
** vary temp dir variables such as TMP/TMPDIR/TEMP/TEMPDIR/MAGICK_TMPDIR
211
** maybe vary build with pbuilder and sbuild (but maybe only useful with different setup jobs only…)
Holger Levsen's avatar
Holger Levsen committed
212

213 214
==== reproducible Debian armhf

215 216 217
* rename all the nodes from $HOSTNAME to $HOSTNAME-armhf-rb ?
** we could get rid of the links in jenkins.d.n.git/hosts/
** we could simplefy .../hosts/*/etc/munin/munin-node.conf
218
* make systems send mail, use port 465
219

220 221
==== reproducible Debian arm64

222
* vary DEB_BUILD_OPTS? (NUM_CPU)
223

224 225 226
==== reproducible Debian installation

* see https://wiki.debian.org/ReproducibleInstalls
Holger Levsen's avatar
Holger Levsen committed
227
* run this as a new job
228

229 230 231
==== reproducible non-Debian tests, new host for 398 day variation is unused

* locations in the code which need to be changed
232 233
** RPM_BUILD_NODE=osuosl-build171-amd64
** grep for osuosl-build171-amd64, there's more
234 235 236 237 238 239
* IOW: these tests should use it:
** coreboot
** netbsd
** fedora
** (fdroid)

240 241 242 243 244 245 246 247 248 249 250 251 252
==== reproducible coreboot

* add more variations: domain+hostname, uid+gid, USER, UTS namespace
* build the docs?
* also build with payloads. x86 use seabios as default, arm boards dont have a default. grub is another payload. and these: bayou  coreinfo  external  filo  libpayload  nvramcui - and:
** CONFIG_PAYLOAD_NONE=y
** CONFIG_PAYLOAD_ELF is not set
** CONFIG_PAYLOAD_LINUX is not set
** CONFIG_PAYLOAD_SEABIOS is not set
** CONFIG_PAYLOAD_FILO is not set
** CONFIG_PAYLOAD_GRUB2 is not set
** CONFIG_PAYLOAD_TIANOCORE is not set
* libreboot ships images, verify those?
253
* explain status in plain english
254
* use disorderfs for 2nd build
255

256

257 258 259 260
==== reproducible OpenWrt

* add credit for logo/artwork
* explain status in plain english
261 262
* build path variation
* use disorderfs for 2nd build
263
* incorporate popular third-party ("external feeds") packages?
264 265 266
* html: build variations are wrong
* html: git commit output includes garbage
* html: css: add some space on the left side
267

268
==== reproducible NetBSD
269

270
* explain status in plain english
271
** explain MKREPRO is set to "yes"
272
** explain MKREPRO_TIMESTAMP set to $SOURCE_DATE_EPOCH
273
* use disorderfs for 2nd build
274

275
==== reproducible FreeBSD
276

277
* useful improvements:
278 279
** investigate how to use tmpfs on freebsd and build there. see mdmfs(8)
** find a way to be informed about updates and keep it updated - see 'freebsd-update cron' and 'pkg audit'.  The latter is run periodic(8) as part of the nightly root@ emails.
280
** modify PATH, uid, gid and USER too and host+domainname as well. The VM is only used for this, so we could change the host+domainname temporaily between builds too.
281
** add freebsd vm as node to jenkins and run the script directly there, saves lot of ssh hassle
282
** run diffoscope nativly
283 284 285 286 287 288 289 290

* random notes, to be moved to README
** we build freebsd 10.1 (=released) atm
** we build with sudo too
*** rather not change /usr/obj to be '~jenkins/obj' and build with WITH_INSTALL_AS_USER. also not build in /usr/src. if so, we need to define some variable so we can do so.... but we need a stable path anyway, so whats the point.
*** maybe build as user in /usr/src...
* first build world, later build ports (pkg info...)

291
* document how the freebsd build VM was set up:
292
** base 10.1 install following https://www.urbas.eu/freebsd-10-and-profitbricks/
293 294 295 296
** modified files:
*** /etc/rc.conf
*** /etc/resolv.conf
*** /boot/loader.conf.local
297
** pkg install screen git vim sudo denyhosts munin-node poudriere
298 299
*** configure /usr/local/etc/munin/munin-node.conf to allow jenkins to access it
*** configure /usr/local/etc/denyhosts.conf and /etc/hosts.allow and touch /etc/hosts.deniedssh
300 301
** adduser holger
** adduser jenkins (with bash as default shell)
302
** adduser mattia
303 304
** mkdir -p /srv/reproducible-results
** chown -R jenkins:jenkins /srv/
305

306
* system maintenance
307
** upgraded the VM to FreeBSD 11.0
308
*** done with: 'freebsd-update upgrade -r 10.2' as root in screen
309
*** and with:  'freebsd-update upgrade -r 10.3'
310
*** and with:  'freebsd-update upgrade -r 11.0'
311 312
*** and with:  'freebsd-update upgrade -r 11.1'
*** and with:  'freebsd-update upgrade -r 11.2' followed by 'pkg-static install pkg ; pkg upgrade'
313

314 315
* online disk resizing howto: https://www.freebsd.org/doc/handbook/disks-growing.html

316 317
==== reproducible Fedora

318 319
* make sure the pages meet https://fedoraproject.org/wiki/Design/Requirements
 and ask the web design team for help via filing a ticket as described there
320 321
* '/var/cache/mock/fedora-23-x86_64/' has three subdirs we need to handle (put on tmpfs, monitor size, clean sometimes): ccache, root_cache and  yum_cache
* '/var/lib/mock' should be put on /srv/workspace aka tmpfs
322
* dont hardcode 23 in reproducible_setup_mock.sh and …build_rpm.sh
323 324 325
* setup script:
** mock --clean just uninstalls the chroot but it'll still be rebuilt next time using cache.  you can delete the caches from /var/cache/mock/ or touch the mock config
** is /etc/yum/repos.d/fedora.repo really needed?
326
** hosts/osuosl-build171/etc/yum/repos.d/* is really not sooo good but works…
327
* build script
328
** cleanup mock cache between two builds: --scrub=all might be too much, but whats sensible (or is it --scrub=all?)?
329 330 331 332
** no variations introduced yet:
*** use '-j$NUM_CPU' and 'NEW_NUM_CPU=$(echo $NUM_CPU-1|bc)'
*** modify TZ, LANG, LC_ALL, umask
* other bits:
333 334
** use modified rpmbuild package from dhiru
** verify gpg signatures (via /etc/mock/)
335
** one day we will want to schedule all 17k source packages in fedora…
336
* build rawhide too (once fedora-23 builds nicely), releasever=rawhide
337 338

* more notes:
339 340
** https://fedoraproject.org/wiki/Using_Mock_to_test_package_builds
** http://miroslav.suchy.cz/blog/archives/2015/05/28/increase_mock_performance_-_build_packages_in_memory/index.html
341
** manually create a fedora chroot using rpm, wget + yum: http://geek.co.il/2010/03/14/how-to-build-a-chroot-jail-environment-for-centos
342

343
==== reproducible Arch Linux
344

345 346
* setup_archlinux_schroot job:
** needs to be made idempotent (currently it removes the schroot at the beginning of the job, instead of creating it elsewhere and replacing it on success at the job end…)
Mattia Rizzolo's avatar
Mattia Rizzolo committed
347
** use schroot tarballs (gzipped), moves are atomic then
348
* only disable cert checking on the node running in the future
349
* compare the just built pkg.tar.xz with the one available on the arch mirrors. *then* one can truely say "X% of the Arch Linux packages are reproducible and could bit by bit be reproduced in the real world."
350 351
* maintenance job:
** check for archlinux schroot sessions which should not be there and delete them. complain if that fails.
352

353
* use db - see https://tests.reproducible-builds.org/reproducibledb.htm
354
** scheduler.sh:
355
*** use asp to update trunk packages?
356
** html:
357
*** leave all pkg.html files, delete them (much) later, then check all pkg.* files are gone
358
*** recreate them newly as index.html
359
**** link to https://www.archlinux.org/packages/$repo/x86_64/$pkgname/
360
**** provide links to debian pages if same package name exist
361 362 363
*** create new job to recreate all pkg pages
*** create json
** misc:
364
*** write into all the stats_ tables
365
*** make build.sh respect pacman exit code, see FIXME in _html_.sh
366
*** comparing versions (in scheduler) is probably needed (though it wuld be better if not): if its not empty it must be higher (due to repo constraints), but it can be even higher than in the repo, because we build trunk
367 368
*** debian uses a table removed_packages, should this be used here too?
*** debian uses a table manual_scheduler to limit the amount of packages someone can schedule per day...
369
*** should breakages job look for archlinux breakages? for which cases? a.) grep "was killed after running into timeout after 30m" pkg.html
370
** using notes.git should be next!
371

372
* rebuilding against the archlinux repos
373
** only needs to be done once using https://github.com/Foxboron/devtools-repro
374
** to not (pacman know about trunk and thus) build trunk, we need to configure asp.
375
** we'll keep building against repo+trunk as we do now (so that archlinux can also benefit from the QA effects)
376

377
* fix build.sh:
378
** build2.log doesnt get deleted if build1 fails
379
** -> rename build2.log to $version_build2.log (dont include package name...)
380

381 382 383
* things to be done before enabling more builders:
** build in /srv/workspace instead of /tmp (once this has been done reduce /tmp size back to 15G)

384
* build on OSUOSL nodes:
385
** update thanks & readme
386

387 388
==== reproducible fdroid

389
* reproducible_setup_fdroid_buildserver.sh:
390
** ./jenkins-build-makebuildserver
391
*** manually added the jenkins user to the vboxdrv group
392 393
*** this downloads a base debian image and all Android tarballs (SDK, NDK, Gradle...)
*** then enters the image, installs all debian packages and Android stuff from the cached tarballs
394 395
** cache is kept outside ('~/.cache/fdroidserver') but installed inside
** '~/.cache/fdroidserver' needs to be cleaned at some times…
396 397

* reproducible_build_fdroid_apk.sh
398 399 400 401 402 403 404
** 1st run ./fdroid build some.app:vercode --server
** 2nd run ./fdroid build some.app:vercode --server
*** eg: org.fdroid.fdroid:98006
*** or: "fdroid build -l org.fdroid.fdroid" to build the latest
** run diffopscope on the results

* also see https://f-droid.org/wiki/page/Build_Server_Setup
405

406 407 408 409 410 411 412 413 414 415
* diskspace needs:
----
$ du -hd1 | sort -h | tail -n 6
4.4G    ./android-sdk-linux_86
8.1G    ./fdroidserver
8.3G    ./.vagrant.d
71G     ./VirtualBox VMs
150G    ./fdroiddata
242G    .
----
416

417 418
==== reproducible qubes

419
* add qubes test on t.r-b.o
420 421 422 423 424 425 426 427 428
----
        git clone https://github.com/qubesos/qubes-builder
        make get-sources BUILDERCONF=scripts/travis-builder.conf COMPONENTS=installer-qubes-os
        export DIST_DOM0=fc23
        export USE_QUBES_REPO_VERSION=3.2
        export INSTALLER_KICKSTART=/tmp/qubes-installer/conf/travis-iso.ks

        make qubes iso BUILDERCONF=scripts/travis-builder.conf VERBOSE=0 COMPONENTS=installer-qubes-os
----
429
* depends:  apt install createrepo python-yaml
430 431
* once this iso is being tested, it will be interesting to build the Qubes templates as well, as those images (Qubes templates are images) will be copied on the installation iso. the above iso is a stripped down iso without templates… (and not the real thing)

432 433 434 435 436
==== reproducible guix

* there's no "apt-get install", because of non-FHS conformance, but see https://www.gnu.org/software/guix/download/
*  there's a privileged build daemon, which is needed to perform fully isolated builds, see https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html#Build-Environment-Setup
* it's a bit of work to set up, but all the steps are documented. the "binary installation" method being the easiest.
437 438 439 440 441 442 443 444 445 446 447 448 449 450 451 452 453 454 455 456 457 458 459 460 461 462 463 464 465 466 467
* Manolis wrote:
----
There are two ways to install guix, through prebuilt binaries or through
the source.

*Binary installation:

Go to
<http://www.gnu.org/software/guix/manual/html_node/Binary-Installation.html>,
grab the tarball and follow the instructions there.

*Source instalation:

First make sure you have the dependencies mentioned at
<http://www.gnu.org/software/guix/manual/guix.html#Requirements> installed.

Then download Guix's source from
ftp://alpha.gnu.org/gnu/guix/guix-0.9.0.tar.gz and use the usual
./configure && make && make install

After you have Guix built, you need to create the build-users and have
the guix-daemon run as root, as described here
<https://www.gnu.org/software/guix/manual/html_node/Build-Environment-Setup.html>.

Keep in mind that the guix-daemon must always run as root.

*Testing if everything works:

Now just run `guix package -i vim` as a non-root user. If it runs
correctly, Guix is ready for work.
----
468

469 470 471
==== reproducible...

* openembedded.org!
472
* Gentoo?
473

Holger Levsen's avatar
Holger Levsen committed
474 475
=== qa.debian.org*

Holger Levsen's avatar
Holger Levsen committed
476
* udd-versionskew: explain jobs in README
477
* udd-versionskew: also provide arch-relative version numbers in output too
Holger Levsen's avatar
Holger Levsen committed
478

Holger Levsen's avatar
Holger Levsen committed
479 480
=== d-i_manual*

481
* d-i_check_jobs.sh: check for removed manuals (but with existing jobs) missing
Holger Levsen's avatar
Holger Levsen committed
482 483 484 485 486
* svn:trunk/manual/po triggers the full build, should trigger language specific builds.
* svn:trunk/manual is all thats needed, not whole svn:trunk

=== d-i_build*

487
* d-i_check_jobs.sh: check for removed package (but with existing jobs) missing
Holger Levsen's avatar
Holger Levsen committed
488
* build packages using jenkins-debian-glue and not with the custom scripts used today?
Holger Levsen's avatar
Holger Levsen committed
489 490 491 492
* run scripts/digress/ ?

=== chroot-installation_*

Holger Levsen's avatar
Holger Levsen committed
493
* use schroot for chroot-installation, stop using plain chroot everywhere
494 495
** https://salsa.debian.org/dsa-team/mirror/dsa-puppet/tree/master/modules/schroot
** https://salsa.debian.org/dsa-team/mirror/dsa-puppet/tree/master/modules/porterbox/files/dd-schroot-cmd
Holger Levsen's avatar
Holger Levsen committed
496
** https://gitweb.torproject.org/project/jenkins/tools.git/tree/slaves/linux/build-wrapper
497
* add alternative tests with aptitude and possible apt
Holger Levsen's avatar
Holger Levsen committed
498
* split etc/schroot/default
Holger Levsen's avatar
Holger Levsen committed
499
* inform debian-devel@l.d.o or -qa@?
Holger Levsen's avatar
Holger Levsen committed
500
* warn about transitional packages installed (on non-upgrades only)
Holger Levsen's avatar
Holger Levsen committed
501
* install all the tasks "instead", thats rather easy nowadays as all task packages are called "task*".
Holger Levsen's avatar
Holger Levsen committed
502
** make sure this includes blends
Holger Levsen's avatar
Holger Levsen committed
503 504 505

=== g-i-installation_*

506
Development of these tests has stopped. In future we will use https://openqa.debian.net instead.
Holger Levsen's avatar
Holger Levsen committed
507

Holger Levsen's avatar
Holger Levsen committed
508

Holger Levsen's avatar
Holger Levsen committed
509
== Further ideas...
Holger Levsen's avatar
Holger Levsen committed
510 511


512 513 514 515 516 517
=== rebuild sid completly on demand

* nthykier wants to be able to rebuild all of sid to test how changes to eg lintian, debhelper, cdbs, gcc affect the archive:
* h01ger> | nthykier: so a.) rebuild everything from sid plus custom repo. b.) option to only rebuild a subset, like all rdepends or all packages build-depending on something
* h01ger> | and c.) only build once, not continously and d.) enable more cores+ram on demand to build faster

Holger Levsen's avatar
Holger Levsen committed
518 519
* have a job to trigger such a rebuild on AWS?

Holger Levsen's avatar
Holger Levsen committed
520
=== Test them all
521

522
* build packages from all team repos on alioth with jenkins-debian-glue on team request (eg, via a .txt file in a git.repo) for specific branches (which shall also be automated, eg. to be able to only have jessie+sid branches build, but not all other branches.)
523

Holger Levsen's avatar
Holger Levsen committed
524
== Debian Packaging related
525

Holger Levsen's avatar
Holger Levsen committed
526
This setup should come as a Debian source package...
527

Holger Levsen's avatar
Holger Levsen committed
528 529
* /usr/sbin/jenkins.debian.net-setup needs to be written
* what update-j.d.n.sh does, needs to be put elsewhere...
Holger Levsen's avatar
Holger Levsen committed
530 531 532 533
* debian/copyright is incorrect about some licenses:
** the profitbricks+debian+jenkins logos
** the preseeding files
** ./feature/ is gpl3
534

Holger Levsen's avatar
Holger Levsen committed
535
// vim: set filetype=asciidoc: