Install ISO tester/builder VMs in Iguana

The first step to start using Iguana for building/testing is to install 2 fresh VMs in it and connect them to our current Jenkins:

• Install ISO builder VM in Iguana.
• Install ISO tester VM in Iguana.
• Connect them to Jenkins (maybe already move away from JNLP-based agent-to-controller connection -- #17798 (closed))
• Enable 1 first builder in production
• Investigate "illegal reflective access operations"
  • This is tracked upstream ⇒ no further action needed from our side at this point.
• Run test suite in Jenkins and record issues below in this task list
• Various isotester → services firewalling trouble
• reboot_node fails: https://jenkins.tails.boum.org/job/reboot_node/7475/console
  • I had the same problem on sib (workers running on 2 different machines). We could solve it the same way I did there: do ssh -v ${RESTART_NODE} instead of ssh -v ${RESTART_NODE}.lizard, and rename all nodes hosted on lizard to explicitly have the .lizard suffix. -- intrigeri
  • jenkins@jenkins.lizard can't SSH into isotester6.iguana: likely a missing firewall rule
  • Avoid adding .lizard to the end of Iguana's agents hostnames (see #17846 (comment 176630)).
• Enable 1 first tester in production
• Setup automatic SSH access from Iguana's users to *.iguana VMs.
• Figure out how many isobuilders & isotesters we want to run on iguana (keeping in mind that for this first iteration, we bought half the RAM we actually want in this box)
  • I'm pretty sure that most of the time, the bottleneck is isotesters, so I propose we start by adding 1 more isotesters (on top of isotester6.iguana) and then see how it goes. -- intrigeri
  • We decided to add 1 more ISO tester and 1 more ISO builder, ending up with a total of 3 ISO testers (6✓, 7✓ and 8) and 2 ISO builders (5✓ and 6) in the new hardware. -- zen-fu
• Set up the remaining isobuilders & isotesters
• Update documentation on installing ISO testers/builders
  • Manually accepting ISO testers' SSH key is needed so jenkins.lizard can reboot them. → Add the SSH fingerprints to hieradata/common.yaml under tails::profile::ssh::sshkeys:.
  • We now need to setup SCSI devices because of #11788 (closed):
    • When installing the guest:

      ...
         --controller scsi,model=virtio-scsi,address=0:0:7.0,index=0 \
         --disk "/dev/iguana/${VI_GUEST}-system,size=${VI_SYSTEM_DISK_GB},bus=scsi,cache=directsync,io=native,discard=unmap" \
      ...

    • When editing the guest XML: Add indexed SCSI controllers and point disks to different controllers.
  • ISO testers should be excluded from Puppet Agent monitoring checks.
  • ISO testers: Add the generated /root/.ssh/id_rsa.pub to the role-jenkins-isotester GitLab user's SSH keys.
• Create follow up issue to buy, install, and use the remaining RAM -- #17866 (closed)