INSTALL 8.02 KB
Newer Older
1 2 3 4 5
Setup of jenkins.debian.net
===========================
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
6
:Status:           working, in progress
7 8 9 10 11 12
:lang:             en
:Doctype:          article
:Licence:	   GPLv2

== About jenkins.debian.net

13
See link:https://jenkins.debian.net/userContent/about.html["about jenkins.debian.net"].
14 15 16 17 18

== Setup

=== Installed software that makes jenkins.debian.net run

19
link:https://jenkins.debian.net[jenkins.debian.net] runs Debian jessie.
20 21 22

=== Jenkins

Holger Levsen's avatar
Holger Levsen committed
23 24 25 26
Currently running the long term support .deb from jenkins-ci.org.

Jenkins is installed using 'deb http://pkg.jenkins-ci.org/debian-stable binary/' into '/etc/apt/sources.list.d/jenkins.list' and `apt-get update && apt-get install jenkins`.

27
We would prefer to run a jenkins package from jessie-backports.
28 29 30

==== Jenkins plugins

31 32
* Git plugin
* Subversion Plug-in
Holger Levsen's avatar
Holger Levsen committed
33

34
* Email Extension Plugin
Holger Levsen's avatar
Holger Levsen committed
35 36
* Log Parser Plugin
* HTML Publisher plugin
37
* Static Analysis Utilities
Holger Levsen's avatar
Holger Levsen committed
38 39 40

* Read-only configurations
* SCM Sync Configuration Plugin
41 42
* Dependency Graph Viewer Plugin
* View Job Filters
43
* Console Column Plugin
44
* Priority Sorter Plugin
45
* Throttle Concurrent Builds Plugin
46
* Build Blocker Plugin
47
* Naginator Plugin
48
* Next Executions
49
* Parameterized Trigger plugin
50
* Environment Injector Plugin
51 52
* build timeout plugin
* Build Authorization Token Root Plugin
53
* Embeddable Build Status Plugin
54
* Plot Plugin
55
* GitLab Plugin
Holger Levsen's avatar
Holger Levsen committed
56

57 58 59
* Sidebar Link
* Green Balls
* Compact Columns
60
* Console Tail Plugin
61
* Cron Column Plugin
62
* Simple Theme Plugin
63
* Live Screenshot
Holger Levsen's avatar
Holger Levsen committed
64
* Locale plugin
65

Holger Levsen's avatar
Holger Levsen committed
66
=== updating jenkins.debian.net with 'update_jdn.sh'
67

68
'update_jdn.sh' is a hackish script which will install all needed packages, copy their configuration in place, generate the documentation webpages and update all jenkins jobs if needed.
Holger Levsen's avatar
Holger Levsen committed
69

70
As user run:
Holger Levsen's avatar
Holger Levsen committed
71 72
----
cd
73
git clone https://salsa.debian.org/qa/jenkins.debian.net
Holger Levsen's avatar
Holger Levsen committed
74 75 76 77
cd jenkins.debian.net
./update_jdn.sh
----

Holger Levsen's avatar
Holger Levsen committed
78
'vncdotool' is the only software which is not installed by packages and is not covered by this script.
79

Holger Levsen's avatar
Holger Levsen committed
80
=== IRC notification configuration using kgb-client
Holger Levsen's avatar
Holger Levsen committed
81

Holger Levsen's avatar
Holger Levsen committed
82
The jenkins jobs are configured to send email to 'jenkins+$IRC_CHANNEL' (like 'jenkins+debian-qa'), this is parsed by a script ('/srv/jenkins/bin/email2irc.sh') run through 'procmail' which then in turn notifies 'kgb-client', which notfies 'kgb-server'(s) on the internet, which are the bots notifying the IRC channels.
83 84 85

The Jenkins EMail notification plugin is used as its state tracking is best (and the Jenkins IRC plugin is way too powerful).

86 87 88 89 90 91
=== Installing vncdotool

----
apt-get install python-twisted python-imaging
git clone https://github.com/sibson/vncdotool
cd vncdotool/
92
python setup.py install
93 94
----

95 96 97
=== adding new slaves to jenkins

Process to follow to add a new node to jenkins:
Mattia Rizzolo's avatar
Mattia Rizzolo committed
98

99 100 101 102 103 104 105
* Open link:[https://jenkins.debian.net/computer/new]
* Use the FQDN of the new node, and select "Permanent Agent"
* Configure it as follow:
  * '# of executors': the number of CPUs of the new node
  * 'Remote root directory': `/srv/jenkins/pseudo-hosts/$hostname`
  * 'Labels': `$arch $short_name` (with '$short_name' like ff64a if the hostname is ff64a-armhf-rb)
  * 'Usage': select "Only build jobs with label expressions matching this node"
106 107
  * 'Launch method': select "Launch agent via execution of command on the master"
     * 'Launch command': `/srv/jenkins/bin/start-slave.sh`
108 109 110
  * 'Availability': select "Take this agent online when in demand, and offline when idle"
    * 'In demand delay': 0 (so that builds will start right away)
    * 'Idle delay': 5 (this is an arbitrary amount of time)
111 112 113 114 115 116 117

The slave setup is done so that the slave.jar program doesn't get run on the remote nodes,
to avoid needing Java available in there.
Jobs wanting to run entirely on a remote host will just execute `jenkins_master_wrapper.sh`
which will do the needed ssh and execute `jenkins_node_wrapper.sh` on the remote host,
which will need to know what to execute after to actually start the wanted program.

118 119
=== munin

120
link://munin-monitoring[Munin] is installed, running with link:https://jenkins.debian.net/munin/[jenkins plugins] not yet published elsewhere.
121

Holger Levsen's avatar
Holger Levsen committed
122 123 124 125 126 127 128 129
=== Using jenkins-job-builder

'update_jdn.sh' takes care of this too, so this is just a quick run through. Make sure to also look at the upstream documentation at http://ci.openstack.org/jenkins-job-builder/

* run `cd /srv/jenkins/job-cfg ; jenkins-job-builder update .` to update jobs
* for testing: `cd /srv/jenkins/job-cfg ; rm test-output/ -r ; mkdir test-output ; jenkins-job-builder test . -o test-output/ ; ls test-output/`


130 131
=== Backup

Holger Levsen's avatar
Holger Levsen committed
132
Currently only the configuration is backed up, but neither the jenkins results nor system logs. But that's at least sufficient to re-setup the system if needed.
133 134 135 136 137

* The _etckeeper_ package is installed and automatically keeps '/etc' in '/etc/.git'.
* The _SCM Sync Configuration Plugin_ is installed in Jenkins, putting all Jenkins configuration into '/srv/jenkins.debian.net-scm-sync.git'.
** (This is not ideal yet, because it also puts the generated job configuration there...)

138 139
I've cloned these two git repos on my desktop and manually run `git pull`. This is certainly not ideal but good enough to re-setup the system anytime, which is something.

140

141
=== Setup of standalone tests.reproducible-builds.org for development
142

143 144
If you'd like to run tests.repoducible-builds.org without the rest of the jenkins set up, here are some necessary steps. Note that most of these steps are performed in the update_jdn.sh script, so if you have a test system it might be simpler to just run it.

145
Your clone of the jenkins git (https://salsa.debian.org/qa/jenkins.debian.net) will be referred to as $JENKINSGIT.
146 147 148 149 150

==== Dependencies, users, directories and files

Some dependencies:
----
151
sudo apt install python3 python3-psycopg2 python3-yaml sqlite3 python-rpy2 dctrl-tools
152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173
----

Create a user jenkins for testing. Create the following directories:
----
sudo mkdir -p /var/lib/jenkins/userContent/reproducible/ /srv/reproducible-results/ /srv/workspace /schroots /chroots
----

Create the following symlinks:
----
ln -s $JENKINSGIT/bin /srv/jenkins/bin/ 
ln -s $JENKINSGIT/userContent/reproducible/static /var/lib/jenkins/userContent/reproducible/static/
----

Additional, give user jenkins write access to the root $JENKINSGIT directory and following directory trees:

* /srv/reproducible-results/...
* /var/lib/jenkins/...
* /srv/workspace/...
* $JENKINSGIT/userContent/...
* $JENKINSGIT/bin/...

==== Database
174

175 176
Get a recent copy of the database from:
----
177
wget https://tests.reproducible-builds.org/reproducible.sql.xz
178 179
----

180
The database is kept using PostgreSQL and initialised by running /srv/jenkins/bin/reproducible_db_maintenance.py manually as the jenkins user.
181 182 183

==== Websites scripts

184
The $JENKINSGIT/bin/reproducible_html_*.py and $JENKINSGIT/bin/reproducible_html_*.sh scripts produce the debian pages and you probably want to run all of them at least once. Below are the requirements for some scripts.
185

186
All webpages will be served out of /var/lib/jenkins/userContent/reproducible and you need to point a webserver to this directory. Build the reproducible dashboard for debian with the $JENKINSGIT/bin/reproducible_html_dashboard.sh, and navigate to <host>/debian/reproducible.html in your browser.
187 188 189

*Requirements for $JENKINSGIT/bin/reproducible_html_dashboard.sh*

190
Before running, create the following directories and files:
191 192
----
sudo mkdir /srv/reproducible-results/node-information/
193
for i in i386 amd64 arm64 armhf ; do touch /srv/reproducible-results/node-information/dummy_$i ; done
194 195 196 197 198 199
----

And run the following script as user jenkins:
----
$JENKINSGIT/bin/reproducible_html_repository_comparison.sh
----
200

201 202 203 204 205
*Requirements for $JENKINSGIT/bin/reproducible_html_notes.py*

This script construct pages that contain notes about the ongoing status and causes of nondeterminism for a package. These pages can be accessed at <host>/debian/notes/<package>_note.html

First, clone the notes git repo:
206
git clone https://salsa.debian.org/reproducible-builds/reproducible-notes
207

208
Create symlinks for the the files *packages.yml* and *issues.yml* in your $JENKINSGIT/bin/ directory.
209 210 211

// vim: set filetype=asciidoc: