Allow building IUKs in parallel (on Jenkins)
Originally created by @CyrilBrulebois on #17658 (Redmine)
This is similar and related to #17657 (closed) except the way things are happening in Jenkins seems to warrant a slightly different approach.
At first, I wanted to just write a “RM-side” wrapper that would trigger
as many Jenkins jobs as there are items in the IUK_SOURCE_VERSIONS
list but I seem to have failed at submitting jobs using either Perl’s
LWP or Python’s Requests modules. This would have meant polling until
all jobs have finished, checking the status, and gathering all results
anyway.
That’s why I switched to a different direction: I had some experience
with some build jobs, triggering sub-jobs for different configurations
(e.g. building for amd64
or i386
). This is achieved with a
configuration matrix and a user-defined axis. I’ve demo’d this with the
kibi_build_IUKs
job, which takes the same parameters as the
build_IUKs
one, except for the SOURCE_VERSIONS
parameter that
becomes a SOURCE_VERSION
(singular) user-defined axis.
The biggest downside with this approach is that one has to first re-configure the job to update the list of source versions for that configuration matrix, before submitting the other parameters to trigger a build.
But that allows parallelism as all sub-jobs are triggered on various
machines (see the isobuilder1 || isobuilder2 || isobuilder3 || isobuilder4
value for label_exp
) when they are available.
Since I don’t know any better, I’ve had to tweak a few things for that to work:
- Using a temporary directory to avoid having overlong blablabla/main-job/blablabla/sub-job paths that could trigger some issues.
- Archiving the (single) generated IUK file.
- Setting up a
kibi_collect_IUKs
downstream job to gather all generated IUK files in one place, so thatcopy-iuks-to-rsync-server-and-verify
can be pointed at it and do its job as usual.
At this point, I’d like to take a step back and ask anonym &
intrigeri
what they think about the idea as a whole, and whether they would have
ideas to make that better.
A few things I have in mind right now:
- Keeping
build_IUKs
as it is would be a nice and cheap safeguard: if anything goes wrong, we can still run everything sequentially and tada.
- Naming: maybe
parallel_build_IUKs
? (Usingkibi_*
was a quick way to make sure no clashes would happen during the experimentation.)
- Collecting: is collecting all results in a separate job something
entirely crazy that would need fixing? If that looks fine enough,
maybe name it
parallel_collect_IUKs
?
- Usability: if we could keep the concept of a “user-defined axis” — for the parallelism/automated distribution of sub-jobs — but have that be a parameter rather than an hard-coded list that needs updating (through re-configuration), that would be much cleaner and a tad more practical (build submission in one go).
- Longterm: I think Jenkins jobs are defined somewhere so that one doesn’t have to rely on the configuration done through the web UI, but if that’s indeed the case, I didn’t find where that happens when I last looked.
I’ve used that successfully for 4.4.1, 4.5~rc1, and 4.5 (see job history in https://jenkins.tails.boum.org/job/kibi_collect_IUKs/); it didn’t feel too risky to try this proof-of-concept as the results have to match those we obtain locally, so any deviations (possibly due to this experimentation) were bound to be discovered.
Feature Branch: feature/17658-allow-building-iuks-in-parallel-on-jenkins
Related issues
- Related to sysadmin#17434 (closed)