Migrate Jenkins to Dragon

intrigeri
2023-10-16T10:11:48Z

In our CI jobs, we do a lot of copying artifacts back and forth between Jenkins and its workers. The performance of all such operations (possibly happening concurrently) is bound by the speed of 1 single CPU core on the Jenkins orchestrator. lizard's CPU cores are individually slow, so it would help quite a bit if we migrated it to Dragon.

Preparation:

set up nginx locally on jenkins.lizard to serve jenkins.tails.boum.org and nightly.tails.boum.org

manually copy letsencrypt certificates from www to jenkins

coordinate downtime

prepare a base install jenkins.dragon VM on dragon

prepare firewalling on dragon to forward port 80/443 to jenkins.dragon

prepare static html on www to serve during downtime (a simple maintenance page)

Migration steps:

shut down both jenkins.lizard and jenkins.dragon

shut down all isotesters/builders/workers

let www serve the maintenance page

rsync all data except puppet keys/certs from jenkins.lizard to jenkins.dragon

replace jenkins.lizard with jenkins.dragon everywhere in hiera, puppet-tails, and jenkins-jobs

rename the node entry in manifests/nodes.pp from jenkins.lizard to jenkins.dragon

remove the NFS service from jenkins.dragon

fix networking on jenkins.dragon (both manually and in puppet)

spin up jenkins.dragon and fix whatever is broken / breaks during puppet run

spin up all isotesters/builders/workers

change DNS entries for jenkins.tails.boum.org and nightly.tails.boum.org

test/debug/fix jenkins

set up backups for jenkins.dragon

update documentation

Cleanup:

remove the NFS mount on www

remove the jenkins.tails.boum.org and nightly.tails.boum.org vhosts on www

remove the jenkins.lizard VM

remove the backup job for jenkins.lizard

remove monitoring for jenkins.lizard