README.infrastructure

About the nodes used for jenkins.debian.net
===========================================
:Author:           Holger Levsen
:Authorinitials:   holger
:EMail:            holger@layer-acht.org
:Status:           working, in progress
:lang:             en
:Doctype:          article
:License:          GPLv2

== Scope

This README is about all the nodes used for jenkins.debian.net and tests.reproducible-builds.org.

All nodes run Debian stable, except for the amd64 moonshot nodes, which run Ubuntu 16.04.

Mattia and Holger have root access to all nodes, including jenkins. Additionally Vagrant has
root access to all armhf nodes.

=== jenkins / trbo shell monitor

Run `/srv/jenkins/bin/jenkins-shell-monitor.sh` as jenkins on jenkins to get an
overview of how jenkins.debian.net & tests.reproducible-builds.org (trbo) are
doing.

The shell monitor both includes information from 
https://tests.reproducible-builds.org/trbo.status.html as well as lacks some
other, so it is advised to monitor both the shell script as well as that URL
- and act on it.

=== amd64 and i386 nodes

amd64 and i386 nodes run as VMs in the IONOS cloud. Mattia and Holger have "physical"
access including remote console and BIOS access.

The VMs for jenkins.d.n and rb-mail1.r-b.o have a statically allocated IP addresses, as well
as PTR records for them.  IONOS asked us to inform them (support@cloud.ionos.com) in case
at some point we will decommission them and the reserved IPs won't be needed anymore, so they
can remove the PTR records and unblock the IP addresses.

==== nodes at osuosl.org

We now also have 8 nodes running at osuosl.org. Contact via #osuosl on freenode or mail to 
support@osuosl.org mentioning the machine name in the subject.

The nodes are used for these jobs:

* osuosl167: debian-janitor, mmdebstrap-jenkins jobs and openqa
* osuosl168: fdroid
* osuosl170: archlinux (future), alpine
* osuosl171: openwrt, coreboot, netbsd
** is constantly online (in jenkins) now, which is not how we like it and still doesnt solve the netbsd job probs
* osuosl172: openwrt, coreboot (future)
* osuosl173: debian/archlinux rebuilder#1 / live-builds
* osuosl174: bwiedemann: opensuse tests, vagrant: ocasional manual tests
* osuosl184: archlinux, alpine

Please note that the machines are called gccXXX at osuosl.org, though usually
they are just refered to by their numbers.

=== arm64 nodes

We have 8 arm64 nodes running on moonshot hardware kindly given access to us by Codethink.
Holger and Mattia have access to a 9th node acting as console server, reachable as
jumpserv.colo.codethink.co.uk.

The process of setting up 8 LeMaker HiKey960 boards is currently stalled, because the setup 
turned out to be more complicated than expected. These boards 
(http://www.96boards.org/product/hikey960/) were sponsored by Hewlett Packard Enterprise
(https://www.hpe.com/) and will be hosted by the students association
SOSETH (https://sos.ethz.ch/) at ETH Zurich (https://www.ethz.ch/).
Currently the are located with Axel Beckert.

=== armhf nodes

Vagrant hosts a fantastic zoo of over 23 armhf boards in his basement, to which besides him also
Mattia and Holger have access via a console server. (Technically some of these armhf nodes are also arm64 systems.)

There's a script to trigger the power on/off/cycle if needed, from hbi1 on Vagrant's local network:

  localhost$ ssh -p 2227 hbi1.dyn.aikidev.net
	# ssh key fingerprints:
	# 2048 f5:03:59:fa:a9:f6:97:b0:80:9c:0c:3c:dc:9e:38:59 /etc/ssh/ssh_host_rsa_key.pub (RSA)
	# 256 38:e8:16:bb:13:d1:5b:28:39:85:73:11:f2:1f:29:81 /etc/ssh/ssh_host_ecdsa_key.pub (ECDSA)
	# 256 16:cc:1f:0e:39:80:37:64:f7:23:dc:18:3f:58:d3:52 /etc/ssh/ssh_host_ed25519_key.pub (ED25519)
	# It's advertising an ipv6 address, but it's actually only ipv4 for the
	# foreseeable future and I just need to fix DNS.

  hbi1$ cd /srv/rb/remote-power ; ./remote-power NAME on|off|ccl
	# ccl means powercycling…

There are at least three machines, jtx1a, jtx1b and jtx1c that don't tend to
come up after power cycling, but sometimes even those work too, so it's
worth a try.

A small number of the names listed in cereal differ slightly from
the names used natively... partly due to bugs in cereal. They should be
fairly obvious (e.g. bpi0a vs. bpi0, hb0a vs. hb0).

The serial console access is still read-only, not actually interacting
with the console; think this is a limitation of the way "cereal"
interacts with screen. "cereal l" lists the available systems, and
"cereal f XXX" would connect to "XXX" and tail it's console logs.


There are now some virtual machine host servers running running armhf
nodes as virtual machines. Vagrant, Holger and Mattia have access to
the host server.

virt*a hosted on:

 $ ssh -p 2270 mst00-armhf-rb.debian.net cat /etc/ssh/ssh_host_ed25519_key.pub
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAID36orreKmE4taJMnxkVeRMUsCKtkD53hlxngUM6eA5e root@mst00t

virt*b hosted on:

 $ ssh -p 2271 mst01-armhf-rb.debian.net cat /etc/ssh/ssh_host_ed25519_key.pub
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIF55EBPXG2gMM7fveuFK8lcBwDE/PBsiuEEtgrQELT0h root@mst01

virt*c hosted on:

 $ ssh -p 2276 mst02-armhf-rb.debian.net cat /etc/ssh/ssh_host_ed25519_key.pub
 ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFJtD97OeFQzLc0GZ4HzKFp6oaddLwUX6GjgZhKfbE1r root@mst02

The virsh commands can be used to list, start and shutdown virtual
machines:

 $ sudo virsh list
 Id   Name      State
 -------------------------
  1    virt64a   running
  2    virt32a   running

 $ sudo virsh shutdown virt32a
 ...
 $ sudo virsh start virt32a
 ...

Alternately, you can connect using the virt-manager GUI. You need to
be in the "libvirt" group on the host server. Set up a connection for
each of the servers:

  File -> Add Connection
  Check "Connect to remote hostover SSH"
  Username $your_username
  Hostname mst00-armhf-rb.debian.net:2270
  Click "Connect"

The nodes should appear in a sidebar. You can click on them and
things.


There's a pad too, where we maintain notes about the status of the nodes. The URL
for this pad is available upon request.

// vim: set filetype=asciidoc: