Copying artifacts between Jenkins and any of its builder nodes is capped at ~10 MiB/s

For details about the problem, impact on release management, and possible solutions, see the "[Tails-dev] Help appreciated to investigate some Jenkins performance issue" thread.

FTR, on my local Jenkins I'm seeing the same issue, and the bottleneck seems to be CPU usage of the Jenkins Java process on the Jenkins orchestrator:

• consider 2 bare metal machines, connected via a Gbit switch and cables: sib and ant01
• sib runs apt-cacher-ng in a VM (apt-proxy.sib) and Jenkins in another one jenkins.sib
• ant01 runs Jenkins builders (workerN.ant01)
• on a Jenkins builder such as worker1.ant01, APT maxes out the Gbit link when downloading packages from apt-proxy.sib
• when a Jenkins job running on workerN.ant01 downloads artifacts from the Jenkins orchestrator that's running on jenkins.sib, I never see transfer rates faster than 20MB/s

From this it seems we can conclude that the network setup itself works fine, and only transfers done via Jenkins are slow. Additionally, the CPU on sib is almost twice as fast as the CPU on jenkins.lizard, and interestingly jenkins.sib transfers artifacts almost twice as fast as jenkins.lizard.

Historical data:

• tails#17750 (closed)
• tails#17742 (closed)