Install ISO tester/builder VMs in Iguana

Zen Fu
2022-06-03T14:39:16Z

The first step to start using Iguana for building/testing is to install 2 fresh VMs in it and connect them to our current Jenkins:

- [x] Install ISO builder VM in Iguana.
- [x] Install ISO tester VM in Iguana.
- [x] Connect them to Jenk...

The first step to start using Iguana for building/testing is to install 2 fresh VMs in it and connect them to our current Jenkins:

Install ISO builder VM in Iguana.

Install ISO tester VM in Iguana.

Connect them to Jenkins (maybe already move away from JNLP-based agent-to-controller connection -- #17798 (closed))

Enable 1 first builder in production

Investigate "illegal reflective access operations"

This is tracked upstream ⇒ no further action needed from our side at this point.

Run test suite in Jenkins and record issues below in this task list

Various isotester → services firewalling trouble

reboot_node fails: https://jenkins.tails.boum.org/job/reboot_node/7475/console

I had the same problem on sib (workers running on 2 different machines). We could solve it the same way I did there: do ssh -v ${RESTART_NODE} instead of ssh -v ${RESTART_NODE}.lizard, and rename all nodes hosted on lizard to explicitly have the .lizard suffix. -- intrigeri

jenkins@jenkins.lizard can't SSH into isotester6.iguana: likely a missing firewall rule

Avoid adding .lizard to the end of Iguana's agents hostnames (see #17846 (comment 176630)).

Enable 1 first tester in production

Setup automatic SSH access from Iguana's users to *.iguana VMs.

Figure out how many isobuilders & isotesters we want to run on iguana (keeping in mind that for this first iteration, we bought half the RAM we actually want in this box)

I'm pretty sure that most of the time, the bottleneck is isotesters, so I propose we start by adding 1 more isotesters (on top of isotester6.iguana) and then see how it goes. -- intrigeri

We decided to add 1 more ISO tester and 1 more ISO builder, ending up with a total of 3 ISO testers (6✓, 7✓ and 8) and 2 ISO builders (5✓ and 6) in the new hardware. -- zen-fu

Set up the remaining isobuilders & isotesters

Update documentation on installing ISO testers/builders

Manually accepting ISO testers' SSH key is needed so jenkins.lizard can reboot them. → Add the SSH fingerprints to hieradata/common.yaml under tails::profile::ssh::sshkeys:.

We now need to setup SCSI devices because of #11788 (closed):

When installing the guest:

...
   --controller scsi,model=virtio-scsi,address=0:0:7.0,index=0 \
   --disk "/dev/iguana/${VI_GUEST}-system,size=${VI_SYSTEM_DISK_GB},bus=scsi,cache=directsync,io=native,discard=unmap" \
...

When editing the guest XML: Add indexed SCSI controllers and point disks to different controllers.

ISO testers should be excluded from Puppet Agent monitoring checks.

ISO testers: Add the generated /root/.ssh/id_rsa.pub to the role-jenkins-isotester GitLab user's SSH keys.

Create follow up issue to buy, install, and use the remaining RAM -- #17866 (closed)