README 16.7 KB
Newer Older
1 2 3 4 5
About jenkins.debian.net
========================
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
6
:Status:           working, in progress
7 8
:lang:             en
:Doctype:          article
Mattia Rizzolo's avatar
Mattia Rizzolo committed
9
:License:          GPLv2
10 11 12

== About jenkins.debian.net

13
https://jenkins.debian.net is a tool for automated quality monitoring of Debian. It is *work in progress* despite being in existence since October 15th 2012.
Holger Levsen's avatar
Holger Levsen committed
14

15
Get the source by running `git clone https://salsa.debian.org/qa/jenkins.debian.net`. It's all in there, no (relevant) manual setup has been done besides what's in this git repository. (The irrelevant bits are some very simple configuration files containing passwords.)
Holger Levsen's avatar
Holger Levsen committed
16

17
The (virtualized) hardware is sponsored since October 2012 by http://www.profitbricks.co.uk - currently it's using more than hundred cores and almost 300 GB memory, thanks a lot!
Holger Levsen's avatar
Holger Levsen committed
18

19
Some stats are available using link:https://jenkins.debian.net/munin/jenkins-month.html[munin-plugins for jenkins].
20

21
Three persons have shell access (incl. root) to the machine: link:mailto:holger@layer-acht.org[Holger Levsen], link:mailto:helmutg@debian.org[Helmut Grohne] and link:mailto:mattia@debian.org[Mattia Rizzolo]. All of them have also access to the web intereface, where tasks like stopping and scheduling job runs can be done, also they have the rights to edit the jenkins scripts (i.e. what jenkins executes) directly, though this is limited to cases like firefighting (IOW deploying changes via the git repository are the norm). The deploying of changes is still limited to people with root powers.
Mattia Rizzolo's avatar
Mattia Rizzolo committed
22

23
== Getting involved
24

25
jenkins.debian.net is a QA resource for the whole Debian project. Please contact us (via #debian-qa on IRC or via the debian-qa mailinglist) If you / your project is interested to run tests in this setup!
26 27

If you notice some jobs has problems and you want to find out why, read <<debug,debug certain jobs>> to learn how to do debug jobs locally.
28

29 30
include::CONTRIBUTING[]

31 32
== Notifications

Philip Hands's avatar
Philip Hands committed
33
There are two types of notifications being used: email and IRC. At the end of each builds console log it says to where notifications have been sent. An address of the form 'jenkins-foo' means an IRC notification has been sent to the #foo IRC channel.
34

35 36
All job result notifications should be sent to https://lists.alioth.debian.org/mailman/listinfo/qa-jenkins-scm and optionally to other recipients as well.

37 38
== Jobs being run

39
There are over 1600 jobs being run currently. If you can think of ways to improve the usefulness of certain jobs, please do give feedback!
40

41
=== g-i-installation jobs
Holger Levsen's avatar
Holger Levsen committed
42

43 44
These jobs are deprecated, in future we will use https://openqa.debian.net instead.

45 46
Installation tests with g-i, the graphical version of d-i, the debian-installer.

Holger Levsen's avatar
Holger Levsen committed
47 48
* 'g-i-installation_debian_sid_daily-rescue'
** boot of rescue system with daily build sid image
49

50
* 'g-i-installation_debian_sid_daily-lxde' and '-xfce' and '-kfreebsd' and '-hurd'
51
** sid installation of Xfce/LXDE desktop with daily build sid image
Holger Levsen's avatar
Holger Levsen committed
52

Holger Levsen's avatar
Holger Levsen committed
53 54 55 56 57
* 'g-i-installation_debian_jessie_lxde','-xfce','-kde' and '-gnome' and '-kfreebsd'
** jessie installation of Xfce/LXDE/KDE desktop and kfreebsd install with weekly build jessie image

* 'g-i-installation_debian_wheezy_lxde','-xfce','-kde' and '-gnome' and '-kfreebsd'
** wheezy installation of Xfce/LXDE/KDE desktop and kfreebsd install with wheezy release image
58 59 60

=== debian-installer jobs

Holger Levsen's avatar
Holger Levsen committed
61
* 'd_i_build_$source_package'
62 63
** there is one job for each git repo referred to in http://anonscm.debian.org/viewvc/d-i/trunk/.mrconfig?view=co
** each job pdebuilds the master branch of its git repo on every git push in a sid environment. (If the architecture(s) specified in debian/control are not amd64,all or any the build exits cleanly.)
64
** while these jobs are triggered on commits, the SCM is only polled every 6min to see if there are new commits.
65

Holger Levsen's avatar
Holger Levsen committed
66
* 'd_i_manual'
67
** builds the full installation-guide package with pdebuild in sid on every commit to svn://anonscm.debian.org/svn/d-i/ matching suitable patterns.
68
** while this job is triggered on commits, the SCM is only polled every 15min to see if there are new commits.
69

Holger Levsen's avatar
Holger Levsen committed
70
* 'd_i_manual_$language_html'
71
** builds a language (on jessie) on every commit of svn/trunk/manual/$LANG with `make languages=$LANG architectures=amd64 formats=html`.
72
** while these jobs are triggered on commits, the SCM is only polled every 15min to see if there are new commits.
Holger Levsen's avatar
Holger Levsen committed
73
** on successful build, 'd_i_manual_$lang_pdf' is triggered.
74

Holger Levsen's avatar
Holger Levsen committed
75
* 'd_i_parse_build_logs' - parses logs from http://d-i.debian.org/daily-images/build-logs.html daily, to give them a bit more exposure.
76
** this job is run daily.
Holger Levsen's avatar
Holger Levsen committed
77

78
=== chroot-installation jobs
79

80 81
Installation tests inside chroot environments.

82
* 'chroot-installation_maintenance_$distro':
83
** make sure chroots have been cleaned up properly
84 85
** runs daily at 05:00 UTC and triggers the $distro specific bootstrap job on success
** wheezy is only triggered on the 4th day and 18th of each month (as it was released on the 4th)
86 87
* $distro-bootstrap jobs:
** just `debootstrap $distro` (install a base Debian distribution $distro)
88
** there is one job for *sid*, one for *wheezy* and one for *jessie*: 'chroot-installation_sid_bootstrap', 'chroot-installation_wheezy_bootstrap' and 'chroot-installation_jessie_bootstrap'
89 90 91 92
** on successful run of the bootstrap job, six $distro-install(+upgrade) jobs are triggered.

* $distro-install jobs (and $distro-install+upgrade jobs):
** `debootstrap $distro`, install a *$set_of_packages* (and upgrade to *$2nd_distro*)
93
** these $set_of_packages exist: 'gnome', 'kde', 'kde-full', 'lxde', 'lxqt', 'xfc', 'full_desktop' (all six desktops plus `vlc evince iceweasel chromium cups build-essential devscripts wine texlive-full asciidoc vim emacs` and (`libreoffice virt-manager mplayer` (stretch/sid) and 'develop'
94
*** install is done with `apt-get install`, except for 'develop' where `apt-get build-dep` is used to install the build dependencies of these packages.
Holger Levsen's avatar
Holger Levsen committed
95
** Then there are also all the corresponding upgrade jobs, eg 'chroot-installation_wheezy_install_gnome_upgrade_to_jessie'
Holger Levsen's avatar
Holger Levsen committed
96

97 98 99 100
=== Debian Edu related jobs

* All Debian Edu related jobs can be seen at these two URLs:
** https://jenkins.debian.net/view/edu_devel/ about Debian Edu Jessie
101
** https://jenkins.debian.net/view/edu_stable/ about Debian Edu Wheezy
102 103

* Then there are three types of jobs:
Holger Levsen's avatar
Holger Levsen committed
104
** 'g-i-installation_$(distro)_$(profile)':
105 106 107
*** tests installation of a profile with preseeding in the graphical installer,
*** screenshots and logs are preserved and a movie created,
*** testing clients against the main-server is planned too, for some time...
Holger Levsen's avatar
Holger Levsen committed
108
** 'chroot-installation_$(distro)_install_$(education-metapackage)':
Philip Hands's avatar
Philip Hands committed
109
*** tests apt installation of a metapackage in a specific distro.
Holger Levsen's avatar
Holger Levsen committed
110
* 'edu-packages_$(distro)_$(src-package)':
111
** builds one of the six debian-edu packages ('debian-edu', 'debian-edu-config', 'debian-edu-install', 'debian-edu-doc', 'debian-edu-artwork', 'debian-edu-archive-keyring' on every push to it's git master branch
112
** and whenever 'debian-edu-doc' is build, https://jenkins.debian.net/userContent/debian-edu-doc/ gets updated automatically afterwards too.
113

114 115 116
=== qa.debian.org related jobs

* There are jobs for lintian and for piuparts:
117
** they simply run a build and/or the tests of the master branch of their git repository on every commit against sid. If that succeeds, the same source will be built on stretch, then on jessie and - in the lintian case only - also for wheezy.
118 119 120 121 122
* There are also jobs related to link:https://udd.debian.org[UDD]:
** they check for multiarch version screws in various suites or issues with orphaned packages without the correct the relevant bug.
*** the UDD schema is available at https://udd.debian.org/schema/udd.html
* Last but not least, dpkg related jobs:
** they tests for trigger cycles using data from the archive and http://binarycontrol.debian.net
123 124 125

=== haskell jobs

126
* See https://wiki.debian.org/Haskell for more information about those jobs.
127 128 129

=== rebootstrap jobs

130
* See https://wiki.debian.org/HelmutGrohne/rebootstrap for more information about these jobs.
131

Holger Levsen's avatar
Holger Levsen committed
132
=== reproducible builds jobs
133

134
* See https://wiki.debian.org/ReproducibleBuilds to learn more about "Reproducible Builds" in Debian and beyond.
135

136
* Several jobs are being used to assemble the website https://tests.reproducible-builds.org which is actually a collection of static html and log files (and very few images) being served from this host. Besides the logfiles data is stored in a database which can be downloaded from https://tests.reproducible-builds.org/reproducible.sql.xz. (That copy is updated daily.)
137

Chris Lamb's avatar
Chris Lamb committed
138
* The (current) purpose of https://tests.reproducible-builds.org is to show the potential of reproducible builds for Debian - and six other projects currently. This is research, showing what could (and should) be done... check https://wiki.debian.org/ReproducibleBuilds for the real status of the project for Debian!
139

140
* For Debian, four suites, 'stretch', 'buster', 'unstable' and 'experimental', are tested on four architectures: 'amd64', 'i386', 'arm64' and 'armhf'. The tests are done using 'pbuilder' through several concurrent workers: 40 for 'amd64', 24 for 'i386', 32 for 'arm64' and 52 for 'armhf', which are each constantly testing packages and saving the results of these tests. There's a single link:https://salsa.debian.org/qa/jenkins.debian.net/blob/master/bin/reproducible_build_service.sh[systemd service] starting all of these link:https://salsa.debian.org/qa/jenkins.debian.net/blob/master/bin/reproducible_worker.sh[workers] which in turn launch the actual link:https://salsa.debian.org/qa/jenkins.debian.net/blob/master/bin/reproducible_build.sh[build script]. (So the actual builds and tests are happening outside the jenkins service.)
141 142
** To shutdown all the workers use:  `sudo systemctl stop reproducible_build@startup.service ; /srv/jenkins/bin/reproducible_cleanup_nodes.sh`
** To start all the workers use: `sudo systemctl start reproducible_build@startup.service`
143

144 145
* These builds on remote nodes run on very different hardware:
** for 'amd64' we are using four virtual machines, profitbricks-build(1+5+11+15)-amd64, which have 15 or 16 cores and 48gb ram each. These nodes are sponsored by link:https://jenkins.debian.net/userContent/thanks.html[Profitbricks].
146
** for 'i386' we are also using four virtual machines, profitbricks-build(2+6+12+16)-i386, which have 10 or 9 cores and 36gb ram each. pb2+12 run emulated AMD Opteron CPUs and pb6+16 Intel Xeon CPUs. These nodes are also sponsored by link:https://jenkins.debian.net/userContent/thanks.html[Profitbricks].
147
** for 'arm64' we are using eight "moonshot" sleds, codethink-sled9-15-arm64, which have 8 cores and 64gb ram each. These nodes are sponsored by link:https://jenkins.debian.net/userContent/thanks.html[Codethink].
148
** To test 'armhf' we are using 24 small boards hosted by vagrant@d.o:
149
*** six quad-cores (cbxi4a, cbxi4b, ff4a, jtx1a, jtx1b, jtx1c) with 4gb ram,
150
*** one hexa-core (ff64a) with 2gb ram,
151
*** three octo-cores (odxu4a, odxu4b and odxu4c) with 2gb ram,
152 153
*** twelve quad-cores (wbq0, cbxi4pro0, ff2a, ff2b, odu3a, opi2a, opi2b, opi2c, jtk1a, jtk1b, p64b and p64c) with 2gb ram, and
*** two dual-core (bbx15 and cb3a) with 2gb ram each.
154
* We would love to have more or more powerful ARM hardware in the future, if you can help, please talk to us!
Mattia Rizzolo's avatar
Mattia Rizzolo committed
155

156
* Packages to be build are scheduled in the database via a scheduler job, which runs every hour and if the queue is below a certain threshold schedules four types of packages:
157
** new untested packages (either uploaded to 'unstable' or 'experimental' or migrated to 'buster' or 'stretch'),
158 159 160
** new versions of existing packages, which were already tested - these are always scheduled, no matter how full the queue is
** old versions, already tested (at least two weeks ago)
** and also some old versions which failed to build (at least ten days ago), if no bug has been filed.
161

162
* Several other jobs exist to build the HTML pages and to create two JSON files which can be downloaded from https://tests.reproducible-builds.org/reproducible.json and https://tests.reproducible-builds.org/reproducible-tracker.json. The 1st one has all the data (except history) and the 2nd has all the data we consider relevant to bother maintainers with, that is, some ftbfs isses are excluded.
163

164
* Information from https://salsa.debian.org/reproducible-builds/reproducible-notes is incorporated on pushes to that git repo.
165

Philip Hands's avatar
Philip Hands committed
166
* There are suite specific jobs to create the pbuilder base.tgz's per suite, which have the reproducible apt repo added. Similarly there's another job per suite to create the schroots used by the builder jobs to download the packages sources to build.
167

168
* Then there are two more jobs to create sid and testing schroots to run diffoscope on the the two results. This is necessary since to investigate haskell binaries, diffoscope needs access to the same haskell compiler version as the investigated packages have been built with.
169

170
* For making sure things are considerably under control at any time, there is a maintenance job running every 3h, mostly doing cleanups.
171

172 173
* The jenkins job overview at https://jenkins.debian.net/view/reproducible/ probably makes it clearer how the job scheduling works in practice.

174
* If you are in the reproducible team you can reschedule packages by yourself:
175
** log into alioth.debian.org via ssh, in the team home (/home/groups/reproducible/) there is a reschedule.sh script you can call. Use the --help switch to get the online help.
176
** The team IRC channel will get a notification about the scheduling and optionally when the build finishes too.
177

178
* If you are not in the reproducible team or if you want to reschedule big sets of packages please ask for a manual rescheduling in the '#debian-reproducible' IRC channel on OFTC. Those with shell access to jenkins can bypass the limitations imposed to remote calls, which are limited to 500 schedulings per day, which should be plenty for normal usage.
179

Philip Hands's avatar
Philip Hands committed
180
* Blacklisting packages can be done similarly:
181 182

----
183
jenkins@jenkins:~$ /srv/jenkins/bin/reproducible_blacklist.sh $suite $package1
184 185
----

186
* We support sending automatic link:https://tests.reproducible-builds.org/index_notify.html[email notification] for status changes to maintainers. Enabling/disabling these notifications can be done by people with shell access to jenkins:
187 188 189 190 191 192 193 194 195 196

----
jenkins@jenkins:~$ /srv/jenkins/bin/reproducible_setup_notify.py -h
usage: reproducible_setup_notify.py [-h] [-o] [-p PACKAGES [PACKAGES ...]]
                                    [-m MAINTAINER]
  -h, --help            show this help message and exit
  -o, --deactivate      Deactivate the notifications
  -p PACKAGES [PACKAGES ...], --packages PACKAGES [PACKAGES ...]
                        list of packages for which activate notifications
  -m MAINTAINER, --maintainer MAINTAINER
197
                        email address of a maintainer
198 199
----

200
* Job configuration is at the usual location for 'jenkins.debian.net': there's a 'job-cfg/reproducible.yaml' defining all the jobs and lots of scripts in 'bin/reproducible_*.(sh|py)', plus a few config files like for 'sudo' or 'apache2'.
201

202
* Finally, there are also jobs testing the link:http://www.coreboot.org/[coreboot], link:https://openwrt.org/[OpenWrt], link:http://www.netbsd.org/[NetBSD] and https://www.freebsd.org/[FreeBSD] projects. The results of the tests can be seen respectively at https://tests.reproducible-builds.org/coreboot/, https://tests.reproducible-builds.org/openwrt/, https://tests.reproducible-builds.org/netbsd/ and https://tests.reproducible-builds.org/freebsd/.
Holger Levsen's avatar
Holger Levsen committed
203

204
=== jenkins.d.n jobs
205 206 207

These are jobs for making sure jenkins.debian.net is running smoothly.

208 209 210
[[debug]]
== Debugging certain jobs

Holger Levsen's avatar
Holger Levsen committed
211
To debug most jobs, a jenkins setup is actually not needed.
212

Holger Levsen's avatar
Holger Levsen committed
213
* In principle the shell commands from the various jobs should run on any Debian system just fine. Please use a test system though, as all your data might be eaten.
214

Holger Levsen's avatar
Holger Levsen committed
215
** A good first step is to use this git repo as a Debian source package, build it and then install the jenkins.d.n-debug package and all it's recommends on your test system. NOTE: this ain't as helpful as it used to be as many depends have only been added to 'update_jdn.sh' and not to 'debian/control'.
216

Holger Levsen's avatar
Holger Levsen committed
217 218
=== Feedback

219
We love to get feedback on this! Either by sending an email to debian-qa@lists.debian.org or by joining #debian-qa on irc.debian.org and expressing yourself there. The best way is to link:https://bugs.debian.org/jenkins.debian.org[report bugs], even better if accompanied by patches or pull requests. But really, all feedback is appreciated!
Holger Levsen's avatar
Holger Levsen committed
220 221 222

=== Setup

223
See link:https://jenkins.debian.net/userContent/setup.html[INSTALL].
Holger Levsen's avatar
Holger Levsen committed
224 225 226

=== ToDo

227
There is still a lot of work left, check the current link:https://jenkins.debian.net/userContent/todo.html[ToDo list].
Holger Levsen's avatar
Holger Levsen committed
228

229 230 231
=== Thanks

See link:https://jenkins.debian.net/userContent/thanks.html[THANKS].
Holger Levsen's avatar
Holger Levsen committed
232

233
== License
234

235
** GPLv2, see link:https://salsa.debian.org/qa/jenkins.debian.net/blob/master/LICENSE[LICENSE].
236

Holger Levsen's avatar
Holger Levsen committed
237
// vim: set filetype=asciidoc: