autobuild_specs

title: Automated builds specification

This blueprint helps to keep track of the discussion on the mailing list, and is attached to tails#8655 (closed) to specify how to implement tails#6196 (closed) ("Build all active branches").

Some metrics about the number of branches merged per releases are available on the dedicated statistics page.

Our Jenkins now has the Global Build Stats plugin live, Tails core developers have access to the metrics.

Question to discuss
Scenarios
Future ideas
Statistics

Question to discuss

Which branches we want to build?

We already build the base branches (stable, testing, devel and experimental) + feature/jessie.

The questions raised is mostly concern the feature/* and bugfix/* branches (so topic branches), as well as the doc/* and test/* branches.

Given an ISO build takes around 30-45 minutes on lizard (worst case), given two builders lizard will be able to build something like 64-96 ISOs a day.

Developers can easily add a branch back to the automated builds whenever it has been removed from the list (for example after a release) by merging its base branch into it.

They also can at the moment manually trigger a build if they uploaded to a APT suite:

if anyone really want to trigger an immediate rebuild, they can do it manually in the Jenkins interface (people who have upload rights to our APT repo also have powerful-enough access to Jenkins to trigger builds);
the proposal says that all active branches are built daily, in addition to post-Git-push => worst case, the branch will be rebuilt within 24h;
if in a hurry, or for whatever reason, you can still force a rebuild by pushing a dummy commit (ugly, but it works).

Proposal1:

branches which are not merged into master, devel, stable and testing
but had new commits or Debian package uploaded since the previous release

How to build it

A topic branch may be lagging behind the base branch it's based upon. What we're interested in is generally not whether a (possibly outdated) topic branch builds fine, but whether it would build fine once merged into its base branch:

that's critical from a reviewer's perspective: what they have to evaluate is the result of the merge, not the state of a topic branch forked N weeks ago from its base branch, that possibly has diverged since then;
that's important from a developer's perspective: this is how they will notice if incompatible changes have landed into the base branch since last time they worked on their topic branch.

Hence, when building topic branch F, we need to build from branch F once merged into branch B. However, this merge must only be done locally, at least because Jenkins doesn't have push access to our Git repo.

Here, locally means: in Jenkins own temporary Git checkout.

The exact direction of the merge (B->F vs. F->B) should not matter given how Git merge works, if we got it clearly. We'll see.

This locally-merge-before-building process requires tails#8654 (closed) to be implemented, otherwise we can't easily merge branches locally without affecting the state of our production APT repo.

When to build it

Define the regularity we want to build topic branches, apart from being built on Git push or new Debian package upload.

Note that we will have to plug that in automatic tests when they will be deployed.

Proposal 1: A build a day.

Notifications

When to notify who? And how to notify them?

Proposal 1:

Email will be the main interface.

For base branches, notify the RM.
For topic branches, notify the author of the last commit.

Note that this proposal doesn't take into consideration how to notify when the branch is built because of a Debian package upload.

An alternative for topic branches we might want to use in the future is to notify all authors of the topic branch since it deviated from the base branch:

git log --pretty="format:%an <%ae>" ${BASE_BRANCH}.. | sort -u

Scenarios

In the following scenario:

topic branches are named branch F
base branches are named branch B
builds are ran on merges which don't raise a conflict. If the merge raises a conflict, then the topic branch's developer should take care of resolving it.

Scenario 1 : reviewer

As a reviewer
When I'm asked to review branch F into branch B
Then I need to know if branch F builds fine
  once locally merged into branch B (fresh results!)
So, if there is no such fresh build available
   Then I manually trigger a build of branch F
And if the build succeeded
  The resulting ISO must be made available to me
  The pkg list must be made available to me
  The Redmine ticket should be notified
Otherwise if it fails the developer who proposed the merge must be notified
  And the developer *needs* to see the build logs
  And the ticket should be reassigned to the branch submitter
  And Status should be set to "In Progress"

Bonus:

Notify the manual build requester of the build results. Depends on using Jenkins internal authentication system, and may be complicated if it doesn't attach email address info to each user (apparently the Email-ext plugin just builds the email address by concatenating login, @ and a fixed domain name -- this could be worked around with email aliases hosted somewhere on our infrastructure).
Make it easy for the reviewer to know whether the last build of branch F is current. Or, better (?), trigger rebuilds of all topic branches upon modifications (possibly == rebuild) on their base branch.

Scenario 2 : developer

As a developer who has the commit bit
When I'm working on branch F
Then I need to know if my branch builds after I've pushed and it
    is has been locally merged in base branch B
And I need to know if my branch build is broken by something else
   possibly weeks after my last commit (by e.g Debian changes,
   changes in branch B, ...)
And if the build succeeded
  The resulting ISO must be made available to me
  The pkg list must be made available to me
  The Redmine ticket should be notified
Otherwise if it fails I *need* to see the build logs
  And the developer who proposed the merge must be notified
  And the ticket should be reassigned to the branch submitter
  And Status should be set to "In Progress"

Scenario 3 : RM

As the current RM
When working the full dev release cycle
Then I need to know when a base branch fails to build
And when this happens the build logs must be made available to me.

Future ideas

This list other scenarios not part of the first deployement iteration, but we might want to consider it in the future.

Scenario 10

As a Tails developer working on branch B
When I upload a package to APT suite B
Then I want to know if it broke the build ASAP

(same responsiveness as when pushing to git) (acceptable workaround: being able to manually trigger a build.)

Scenario 11

As the current RM
When I push new tag T on branch B
Then I want the APT suite for tag T to be created
And I want the APT suite B to be copied into the APT suite T
And once this is done, I want a build from the checkout of tag T to be
  triggered
And I want the squashfs sort file to be generated, and the diff sent to me

Scenario 12

As a Tails developer
When the test suite is ran on the ISO build from my last commit
I want to watch TV and see the test video in HTML5 from Tor Browser

Scenario 13

As a Tails developer
When an ISO is build from my last commit
I want to access it through remote desktop (VNC/Spice/...) over Tor

Scenario 14

As a Tails developer
When I push a new commit or a new Debian package to a base branch
I want the affected feature branches to be rebuilt with that change

Statistics

As of 2015-02-02, there are 26 branches that would be automatically built as part of the next 1.3 release, following the for now defined criterias (above in this blueprint):

feature/7779-revisit-touchpad-settings
feature/6992-whisperback-email-address
bugfix/8714-tor-is-ready-robustness
bugfix/8680-git-without-polipo
feature/8719-mount-output-in-bug-reports
feature/6241-gnupg2
feature/8725-remove-vagrant-bootstrap-cache
bugfix/8715-build-system-independent-APT-sources
feature/7756-reintroduce-whisperback
bugfix/8699-only-create-timestamps-in-Jenkins
feature/8740-new-signing-key-phase-2
feature/8665-remove-adblock
bugfix/8756-repair-local-packages
feature/7530-docker_anonym
feature/7530-docker-with-apt-cacher-ng
feature/7963-background-color
feature/8491-live-additional-software-in-whisperback
feature/7530-docker
feature/linux-3.18
feature/torbrowser-alpha
bugfix/8747-update-tails-apt-repo-signing-key
feature/8726-use-homogenous-Debian-mirrors-at-build-time
feature/5525-sandbox-web-browser
feature/7752-keyringer
feature/6739-install-electrum
bugfix/quote-wrappers-arguments