The first step to start using Iguana for building/testing is to install 2 fresh VMs in it and connect them to our current Jenkins: - [x] Install ISO builder VM in Iguana. - [x] Install ISO tester VM in Iguana. - [x] Connect them to Jenkins (maybe already move away from JNLP-based agent-to-controller connection -- #17798) - [x] Enable 1 first builder in production - [x] Investigate ["illegal reflective access operations"](https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17846#note_174699) - This is [tracked upstream](https://issues.jenkins.io/browse/JENKINS-60891?jql=component%3D15893) ⇒ no further action needed from our side at this point. - [x] Run test suite in Jenkins and record issues below in this task list - [x] Various isotester → services firewalling trouble - [x] `reboot_node` fails: https://jenkins.tails.boum.org/job/reboot_node/7475/console - [x] I had the same problem on sib (workers running on 2 different machines). We could solve it the same way I did there: do `ssh -v ${RESTART_NODE}` instead of `ssh -v ${RESTART_NODE}.lizard`, and rename all nodes hosted on lizard to explicitly have the `.lizard` suffix. -- intrigeri - [x] `jenkins@jenkins.lizard` can't SSH into `isotester6.iguana`: likely a missing firewall rule - [x] Avoid adding `.lizard` to the end of Iguana's agents hostnames (see https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17846#note_176630). - [x] Enable 1 first tester in production - [x] Setup automatic SSH access from Iguana's users to `*.iguana` VMs. - [x] Figure out how many isobuilders & isotesters we want to run on iguana (keeping in mind that for this first iteration, we bought half the RAM we actually want in this box) - I'm pretty sure that most of the time, the bottleneck is isotesters, so I propose we start by adding 1 more isotesters (on top of isotester6.iguana) and then see how it goes. -- intrigeri - We decided to add 1 more ISO tester and 1 more ISO builder, ending up with a total of 3 ISO testers (6✓, 7✓ and 8) and 2 ISO builders (5✓ and 6) in the new hardware. -- zen-fu - [x] Set up the remaining isobuilders & isotesters - [x] Update documentation on installing ISO testers/builders - [x] ~~Manually accepting ISO testers' SSH key is needed so `jenkins.lizard` can reboot them.~~ → Add the SSH fingerprints to `hieradata/common.yaml` under `tails::profile::ssh::sshkeys:`. - [x] We now need to setup SCSI devices because of #11788: - [x] When installing the guest: ``` ... --controller scsi,model=virtio-scsi,address=0:0:7.0,index=0 \ --disk "/dev/iguana/${VI_GUEST}-system,size=${VI_SYSTEM_DISK_GB},bus=scsi,cache=directsync,io=native,discard=unmap" \ ... ``` - [x] When editing the guest XML: Add indexed SCSI controllers and point disks to different controllers. - [x] ISO testers should be excluded from Puppet Agent monitoring checks. - [x] ISO testers: Add the generated `/root/.ssh/id_rsa.pub` to the `role-jenkins-isotester` GitLab user's SSH keys. - [x] Create follow up issue to buy, install, and use the remaining RAM -- #17866