README

About jenkins.debian.net
========================
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
:Status:           working, in progress
:lang:             en
:Doctype:          article
:Licence:	   GPLv2

== About jenkins.debian.net

https://jenkins.debian.net is a tool for automated quality monitoring of Debian. It is *work in progress* despite being in existence since October 15th 2012.

Get the source by running `git clone git://git.debian.org/git/qa/jenkins.debian.net.git`. It's all in there, no (relevant) manual setup has been done besides what's in this git repository. (The irrelevant bits are some very simple configuration files containing passwords.)

The (virtualized) hardware is sponsored since October 2012 by http://www.profitbricks.co.uk - currently it's using 23 cores and 75 GB memory, thanks a lot!

Some stats are available using link:https://jenkins.debian.net/munin/jenkins-month.html[munin-plugins for jenkins].

=== Get involved

jenkins.debian.net is ment as QA ressource for the whole Debian project. Please contact us (via #debian-qa on IRC or via the debian-qa mailinglist) If you / your project is interested to run tests in this setup!

If you notice some jobs has problems and you want to find out why, read <<debug,debug certain jobs>> to learn how to do debug jobs locally.

== Notifications

There are two types of notifications being used: email and IRC. At the end of each builds console log it says to where notifcations has been send. An adress of the form 'jenkins-foo' means an IRC notification has been send to the #foo IRC channel.

All job result notifications should be sent to https://lists.alioth.debian.org/mailman/listinfo/qa-jenkins-scm and optionally to other recipients as well.

== Jobs being run

There are over 400 jobs being run currently. If you can think of ways to improve the usefulness of certain jobs, please do give feedback!

=== g-i-installation jobs

Installation tests with g-i, the graphical version of d-i, the debian-installer.

* 'g-i-installation_debian_sid_daily-rescue'
** boot of rescue system with daily build sid image

* 'g-i-installation_debian_sid_daily-lxde' and '-xfce' and '-kfreebsd' and '-hurd'
** sid installation of Xfce/LXDE desktop with daily build sid image

* 'g-i-installation_debian_jessie_lxde','-xfce','-kde' and '-gnome' and '-kfreebsd'
** jessie installation of Xfce/LXDE/KDE desktop and kfreebsd install with weekly build jessie image

* 'g-i-installation_debian_wheezy_lxde','-xfce','-kde' and '-gnome' and '-kfreebsd'
** wheezy installation of Xfce/LXDE/KDE desktop and kfreebsd install with wheezy release image

=== debian-installer jobs

* 'd_i_build_$source_package'
** there is one job for each git repo referred to in http://anonscm.debian.org/viewvc/d-i/trunk/.mrconfig?view=co
** each job pdebuilds the master branch of its git repo on every git push in a sid environment. (If the architecture(s) specified in debian/control are not amd64,all or any the build exits cleanly.)
** while these jobs are triggered on commits, the SCM is only polled every 6min to see if there are new commits.

* 'd_i_manual'
** builds the full installation-guide package with pdebuild in sid on every commit to svn://anonscm.debian.org/svn/d-i/ matching suitable patterns.
** while this job is triggered on commits, the SCM is only polled every 15min to see if there are new commits.

* 'd_i_manual_$language_html'
** builds a language (on wheezy) on every commit of svn/trunk/manual/$LANG with `make languages=$LANG architectures=amd64 formats=html`.
** while these jobs are triggered on commits, the SCM is only polled every 15min to see if there are new commits.
** on successful build, 'd_i_manual_$lang_pdf' is triggered.

* 'd_i_parse_build_logs' - parses logs from http://d-i.debian.org/daily-images/build-logs.html daily, to give them a bit more exposure.
** this job is run daily.

=== chroot-installation jobs

Installation tests inside chroot environments.

* 'chroot-installation_maintenance_$distro':
** make sure chroots have been cleaned up properly
** runs daily at 05:00 UTC and triggers the $distro specific bootstrap job on success
** wheezy is only triggered on the 4th day and 18th of each month (as it was released on the 4th)
** squeeze only on the 25th of each month
* $distro-bootstrap jobs:
** just `debootstrap $distro` (install a base Debian distribution $distro)
** there is one job for *sid*, one for *wheezy* and one for *jessie*: 'chroot-installation_sid_bootstrap', 'chroot-installation_wheezy_bootstrap' and 'chroot-installation_jessie_bootstrap'
* plus there is 'chroot-installation_squeeze_bootstrap_upgrade_to_wheezy', which bootstraps *squeeze* and upgrades that to *wheezy*
** on successful run of the bootstrap job, six $distro-install(+upgrade) jobs are triggered.

* FIXME: describe upgrade jobs here too

* $distro-install jobs (and $distro-install+upgrade jobs):
** `debootstrap $distro`, install a *$set_of_packages* (and upgrade to *$2nd_distro*)
** these $set_of_packages exist: 'gnome', 'kde', 'kde-full', 'lxde', 'xfc', 'full_desktop' (all five desktops plus `vlc evince iceweasel chromium cups build-essential devscripts wine texlive-full asciidoc vim emacs` and (`libreoffice virt-manager mplayer2` (jessie/sid) or `openoffice virtualbox mplayer` (squeeze/wheezy)) and 'develop'
*** install is done with `apt-get install`, except for 'develop' where `apt-get build-dep` is used to install the build dependencies of these packages.

=== Debian Edu related jobs

* All Debian Edu related jobs can be seen at these two URLs:
** https://jenkins.debian.net/view/edu_devel/ about Debian Edu Jessie
** https://jenkins.debian.net/view/edu_stable/ about Debian Edu Wheezy

* Then there are three types of jobs:
** 'g-i-installation_$(distro)_$(profile)':
*** tests installation of a profile with preseeding in the graphical installer,
*** screenshots and logs are preserved and a movie created,
*** testing clients against the main-server is planned too, for some time...
** 'chroot-installation_$(distro)_install_$(education-metapackage)':
*** tests apt installation of a metapackage in a specic distro.
* 'edu-packages_$(distro)_$(src-package)':
** builds one of the six debian-edu packages ('debian-edu', 'debian-edu-config', 'debian-edu-install', 'debian-edu-doc', 'debian-edu-artwork', 'debian-edu-archive-keyring' on every push to it's git master branch
** and whenever 'debian-edu-doc' is build, https://jenkins.debian.net/userContent/debian-edu-doc/ get's updated automatically afterwards too.

=== qa.debian.org related jobs

* There are jobs for lintian and for piuparts:
** they simply run a build and/or the tests of the master branch of their git repository on every commit against sid. If that succeeds, the same source will be built on jessie and then (only for lintian) on wheezy.
* There are also jobs related to link:https://udd.debian.org[UDD]:
** they check for multiarch version screws in various suites or issues with orphaned packages without the correct the relevant bug.
*** the UDD schema is available at https://udd.debian.org/schema/udd.html
* Last but not least, dpkg related jobs:
** they tests for trigger cycles using data from the archive and http://binarycontrol.debian.net

=== haskell jobs

* See https://wiki.debian.org/Haskell for more information about those jobs.

=== rebootstrap jobs

* See https://wiki.debian.org/HelmutGrohne/rebootstrap for more information about these jobs.

=== reproducible builds jobs

* See https://wiki.debian.org/ReproducibleBuilds to learn more about "Reproducible Builds" in Debian and beyond.

* Several jobs are being used to assemble the website https://reproducible.debian.net which is actually a collection of static html and log files (and very few images) being served from this host. Besides the logfiles data is stored in an SQLite database which can be downloaded from https://reproducible.debian.net/reproducible.db. (That copy is updated every four hours.)

* The (current) purpose of https://reproducible.debian.net is to show the prospects of reproducible builds for Debian. IOW: this is research, showing what could (and should) be done... check https://wiki.debian.org/ReproducibleBuilds for the real status of the project!

* Currently, three suites are tested on amd64: 'testing', 'unstable' and 'experimental'. The tests are done using 'pbuilder' using link:https://wiki.debian.org/ReproducibleBuilds/ExperimentalToolchain[our toolchain] through eight concurrent builder jobs which are each constantly testing packages and saving the results of these tests.

* Packages to be build are scheduled in the SQLite database via a scheduler job, which runs every hour and which schedules "new packages" to be tested if the queue is below a certain threshold. New packages" in this context means:
** truely new packages (either uploaded to 'unstable' or 'experimental' or migrated to 'testing'),
** new versions of existing packages.

* If the scheduling queue is low, already tested package will be scheduled for re-testing.

* Several other jobs exist to build the html pages and to create a JSON file which can be downloaded from https://reproducible.debian.net/reproducible.json.

* Information from https://anonscm.debian.org/cgit/reproducible/notes.git is incorporated on pushes to that git repo.

* There are suite specific jobs to create the pbuilder base.tgz's per suite, which have the reproducible apt repo added. Similarily there's another job per suite to create the schroots used by the builder jobs to download the packages sources to build.

* Then there are two more jobs to create sid and testing schroots to run debbindiff on the the two results. This is neccessary since to investigate haskell binaries, debbindiff needs access to the same haskell compiler version as the investigated packages have been built with.

* Finally, there is a maintenance job running every 4h, making sure things are considerably under control.

* The jenkins job overview at https://jenkins.debian.net/view/reproducible/ probably makes it clearer how the job scheduling works in practice.

* If you are in the reproducible team you can reschedule packages by yourself:
** log into alioth.debian.org, in the team home (/home/groups/reproducible/) there is a reschedule.sh script you can call. Use the --help switch to get the online help.
** The team IRC channel will get a notification both when the scheduling is done and once the build finishes.

* If you are not in the reproducible team or if you want to reschedule big sets of packages please ask for a manual rescheduling in the '#debian-reproducible' IRC channel on OFTC. Those with shell access to jenkins need to run this command:

----
jenkins@jenkins:~$ /srv/jenkins/bin/reproducible_schedule_on_demand.sh --$option $suite $package1 $package2
----

The script can be called with either one of the two options: with "--artifacts" the build job will preserve the produced packages and publish them under reproducible.debian.net, to allow easy+deeper investigation of reproducible issues. The URL is published in the build log and on IRC.
Alternativly it's possible to use "--notify" to make the build job notify the team IRC channel when a package finishes building.

* Blacklisting packages can be done similarily:

----
jenkins@jenkins:~$ /srv/jenkins/bin/reproducible_blacklist.sh $suite $package1
----

* Job configuration is at the usual location for 'jenkins.debian.net' - there's a 'job-cfg/reproducible.yaml' defining all the jobs and lots of scripts in 'bin/reproducible_*.(sh|py)', plus a few config files like for 'sudo' or 'apache2'.

=== dvswitch jobs

* There are three jobs currently:
** 'dvswitch_sid' and 'dvswitch_jessie' build the master branch of dvswitch against that environment.
** while 'dvswitch_against_libav_git' builds the master branch of dvswitch using libav also build from it's master branch. This is done in an sid environment.
** Except for 'dvswitch_jessie' which is triggered by a successful 'dvswitch_sid' run, the jobs are triggered by git commits.

=== jenkins.d.n jobs

These are jobs for making sure jenkins.debian.net is running smoothly.

[[debug]]
== Debugging certain jobs

*FIXME*: To debug certain jobs, a jenkins setup is actually not needed. Make this easy and describe here.

*This is work in progress too, but mostly just needs documentation!* In principle the shell commands from the various jobs should run on any Debian system just fine. Please use a test system though, as all your data might be eaten.

* Build this source as a debian package and install the jenkins.d.n-debug package and all it's recommends.

=== Feedback

I love to get feedback on this! Either by sending an email to debian-qa@lists.debian.org or by joining #debian-qa on irc.debian.org and expressing yourself there. The best way is to link:https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=jenkins;users=qa.debian.org%40packages.debian.org[report bugs], even better if accompanied by patches or pull requests. But really, all feedback is appreciated!

=== Setup

See link:https://jenkins.debian.net/userContent/setup.html[INSTALL].

=== ToDo

There is still a lot of work left, check the current link:https://jenkins.debian.net/userContent/todo.html[ToDo list].

=== Thanks

See link:https://jenkins.debian.net/userContent/thanks.html[THANKS].

== License

* everything except features and bin/libvirt_cucumber_tests:
** GPLv2, see link:http://anonscm.debian.org/cgit/qa/jenkins.debian.net.git/LICENSE[LICENSE].
* features and bin/libvirt_cucumber_tests:
** GPLv3+

// vim: set filetype=asciidoc: