Building many IUKs (v2) takes a while on Jenkins
Originally created by @intrigeri on #17434 (Redmine)
With the new “single SquashFS diff” upgrade scheme (tails#15281 (closed) and friends), during the release process, the RM has to build many IUKs on Jenkins:
- to ensure they match those the RM has built locally
- to publish them without having to upload gigabytes of data
For example, when we’ll release 4.20, we’ll be building more than 2 dozens IUKs.
The first iteration (tails#15287 (closed)) of the https://jenkins.tails.boum.org/job/build_IUKs/ job builds all the needed IUKs sequentially, on one single isobuilder, which takes about 15 minutes per IUK ⇒ more than 6 hours in my 4.20 example. That’s going to be pretty bad both wrt. time to remediation for security flaws and wrt. RMs’ nerves.
Now that we’ve learned about this problem, we should iterate and improve on this.
The obvious solution is to parallelize these builds.
Potential sources of inspiration:
- https://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.trigger-builds
- https://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.multijob
-
https://docs.openstack.org/infra/jenkins-job-builder/builders.html#builders.copyartifact
(if one main job triggers all the other builds, ideally it would
join all their artifacts so they’re available from a single place,
where
bin/copy-iuks-to-rsync-server-and-verifycan fetch them from)
Worst case, if implementing this on Jenkins is too feasible soon enough,
we can document in the release process how to split the set of IUKs to
build into several runs of build_IUKs. This would also require extra
work on bin/copy-iuks-to-rsync-server-and-verify.
Related issues
- Related to tails#17361 (closed)
- Related to tails#17435 (closed)
- Related to tails#17437 (closed)
- Related to tails#17658 (closed)