Set up 2 more Jenkins workers on iguana

intrigeri
2022-12-13T12:18:06Z

We should be able to set up 2 more generic Jenkins workers on iguana, each with 16GB of RAM, thanks to:

The 15G of RAM that were already unallocated.

3×(27-16) = 27GB of RAM saved by dropping the tmpfs on /tmp/TailsToaster for the existing 3 isotesters (#17866 (comment 190567)).

With "generic worker" I mean a node that can both runs builds & tests.

And then we'll have 2 generic workers + 3 isotesters + 2 isobuilders, each with 16GB of RAM, for a total of 7×16 = 112GB, which leaves plenty enough on the host (and even some to run a GitLab runner or Jenkins orchestrator).

Once we do this, it may turn out that I/O becomes a bottleneck and causes test suite fragility. If that's the case, we should:

1. Temporarily shut down enough of these extra isotesters until things stabilize back.

2. Either buy another storage drive for iguana & spread the isotester*-TailsToaster LVs over the storage drives, or first test this setup on the new CI machine (#17886 (closed)).

It could also be that CPU becomes the bottleneck and makes individual test suite runs slower (traded against increased total throughput of our CI). If that's the case we should disable 1 of these isotesters and see if that is the sweet spot.

So perhaps the best approach is to first set up 1 generic Jenkins worker on iguana, see how it fares, and if we're happy add another one.