reproducibly_build_Tails_ISO_stable Jenkins job always fails

Originally created by @intrigeri on #14924 (Redmine)

We’re quite unlucky: build_Tails_ISO_stable is scheduled daily, and with Jenkins’ hashing system it happens to run at 07:15 so reproducibly_build_Tails_ISO_stable starts around 08:00. We update the Debian security APT snapshot at 7:41 so in practice, reproducibly_build_Tails_ISO_stable always gets a different snapshot and thus always produces a different ISO.

This prevents us from easily noticing when an update or branch merge broke reproducibility. And possibly worse, it teaches us to ignore email notifications from Jenkins (this has been going on for a week and nobody noticed or if they did, then they kept it to themselves for some reason).

Due to the way our Jenkins jobs are generated, we have little flexibility here: we can’t simply reschedule build_Tails_ISO_stable (and only this one) to run at a different time. And if we start playing with the H symbol we’ll be lucky if we don’t break devel or testing.

The simplest option we have seems to slightly reschedule the APT snapshot update. Now, of course this will likely break “reproducibly” (sic) jobs for other branches, but until we have a better solution, not breaking stable/testing/devel gets higher priority by far. While implementing this I’ll need to be careful not to break testing or devel, but at least I can easily reason about it, while I don’t know how the Jenkins hash thing works exactly.

Long term, the only sane option is probably to teach our build system how to use snapshots specified on the command line (instead of “latest”), and to teach Jenkins to pass the correct parameters when trying to rebuild a given ISO image. Whoever feels responsible for this, please file a ticket about it.

Parent Task: tails#5630 (closed)

Related issues

• Related to #12633 (closed)
• Related to tails#15107 (closed)
• Blocked by tails#14923 (closed)
• Blocked by tails#14946 (closed)