title: Automated builds specification
This blueprint helps to keep track of the discussion on the mailing list, and is attached to tails#8655 (closed) to specify how to implement tails#6196 (closed) ("Build all active branches").
Some metrics about the number of branches merged per releases are available on the dedicated statistics page.
Our Jenkins now has the Global Build Stats plugin live, Tails core developers have access to the metrics.
Question to discuss
Which branches we want to build?
We already build the base branches (stable, testing, devel and experimental) + feature/jessie.
The questions raised is mostly concern the feature/* and bugfix/* branches (so topic branches), as well as the doc/* and test/* branches.
Given an ISO build takes around 30-45 minutes on lizard (worst case), given two builders lizard will be able to build something like 64-96 ISOs a day.
Developers can easily add a branch back to the automated builds whenever it has been removed from the list (for example after a release) by merging its base branch into it.
They also can at the moment manually trigger a build if they uploaded to a APT suite:
- if anyone really want to trigger an immediate rebuild, they can do it manually in the Jenkins interface (people who have upload rights to our APT repo also have powerful-enough access to Jenkins to trigger builds);
- the proposal says that all active branches are built daily, in addition to post-Git-push => worst case, the branch will be rebuilt within 24h;
- if in a hurry, or for whatever reason, you can still force a rebuild by pushing a dummy commit (ugly, but it works).
Proposal1:
- branches which are not merged into master, devel, stable and testing
- but had new commits or Debian package uploaded since the previous release
How to build it
A topic branch may be lagging behind the base branch it's based upon. What we're interested in is generally not whether a (possibly outdated) topic branch builds fine, but whether it would build fine once merged into its base branch:
- that's critical from a reviewer's perspective: what they have to evaluate is the result of the merge, not the state of a topic branch forked N weeks ago from its base branch, that possibly has diverged since then;
- that's important from a developer's perspective: this is how they will notice if incompatible changes have landed into the base branch since last time they worked on their topic branch.
Hence, when building topic branch F, we need to build from branch F once merged into branch B. However, this merge must only be done locally, at least because Jenkins doesn't have push access to our Git repo.
Here, locally means: in Jenkins own temporary Git checkout.
The exact direction of the merge (B->F vs. F->B) should not matter given how Git merge works, if we got it clearly. We'll see.
This locally-merge-before-building process requires tails#8654 (closed) to be implemented, otherwise we can't easily merge branches locally without affecting the state of our production APT repo.
When to build it
Define the regularity we want to build topic branches, apart from being built on Git push or new Debian package upload.
Note that we will have to plug that in automatic tests when they will be deployed.
Proposal 1: A build a day.
Notifications
When to notify who? And how to notify them?
Proposal 1:
Email will be the main interface.
- For base branches, notify the RM.
- For topic branches, notify the author of the last commit.
Note that this proposal doesn't take into consideration how to notify when the branch is built because of a Debian package upload.
An alternative for topic branches we might want to use in the future is to notify all authors of the topic branch since it deviated from the base branch:
git log --pretty="format:%an <%ae>" ${BASE_BRANCH}.. | sort -u
Scenarios
In the following scenario:
- topic branches are named branch F
- base branches are named branch B
- builds are ran on merges which don't raise a conflict. If the merge raises a conflict, then the topic branch's developer should take care of resolving it.
Scenario 1 : reviewer
As a reviewer
When I'm asked to review branch F into branch B
Then I need to know if branch F builds fine
once locally merged into branch B (fresh results!)
So, if there is no such fresh build available
Then I manually trigger a build of branch F
And if the build succeeded
The resulting ISO must be made available to me
The pkg list must be made available to me
The Redmine ticket should be notified
Otherwise if it fails the developer who proposed the merge must be notified
And the developer *needs* to see the build logs
And the ticket should be reassigned to the branch submitter
And Status should be set to "In Progress"
Bonus:
- Notify the manual build requester of the build results. Depends on
using Jenkins internal authentication system, and may be complicated
if it doesn't attach email address info to each user (apparently the
Email-ext plugin just builds the email address by concatenating
login,
@
and a fixed domain name -- this could be worked around with email aliases hosted somewhere on our infrastructure). - Make it easy for the reviewer to know whether the last build of branch F is current. Or, better (?), trigger rebuilds of all topic branches upon modifications (possibly == rebuild) on their base branch.
Scenario 2 : developer
As a developer who has the commit bit
When I'm working on branch F
Then I need to know if my branch builds after I've pushed and it
is has been locally merged in base branch B
And I need to know if my branch build is broken by something else
possibly weeks after my last commit (by e.g Debian changes,
changes in branch B, ...)
And if the build succeeded
The resulting ISO must be made available to me
The pkg list must be made available to me
The Redmine ticket should be notified
Otherwise if it fails I *need* to see the build logs
And the developer who proposed the merge must be notified
And the ticket should be reassigned to the branch submitter
And Status should be set to "In Progress"
Scenario 3 : RM
As the current RM
When working the full dev release cycle
Then I need to know when a base branch fails to build
And when this happens the build logs must be made available to me.
Future ideas
This list other scenarios not part of the first deployement iteration, but we might want to consider it in the future.
Scenario 10
As a Tails developer working on branch B
When I upload a package to APT suite B
Then I want to know if it broke the build ASAP
(same responsiveness as when pushing to git) (acceptable workaround: being able to manually trigger a build.)
Scenario 11
As the current RM
When I push new tag T on branch B
Then I want the APT suite for tag T to be created
And I want the APT suite B to be copied into the APT suite T
And once this is done, I want a build from the checkout of tag T to be
triggered
And I want the squashfs sort file to be generated, and the diff sent to me
Scenario 12
As a Tails developer
When the test suite is ran on the ISO build from my last commit
I want to watch TV and see the test video in HTML5 from Tor Browser
Scenario 13
As a Tails developer
When an ISO is build from my last commit
I want to access it through remote desktop (VNC/Spice/...) over Tor
Scenario 14
As a Tails developer
When I push a new commit or a new Debian package to a base branch
I want the affected feature branches to be rebuilt with that change
Statistics
As of 2015-02-02, there are 26 branches that would be automatically built as part of the next 1.3 release, following the for now defined criterias (above in this blueprint):
- feature/7779-revisit-touchpad-settings
- feature/6992-whisperback-email-address
- bugfix/8714-tor-is-ready-robustness
- bugfix/8680-git-without-polipo
- feature/8719-mount-output-in-bug-reports
- feature/6241-gnupg2
- feature/8725-remove-vagrant-bootstrap-cache
- bugfix/8715-build-system-independent-APT-sources
- feature/7756-reintroduce-whisperback
- bugfix/8699-only-create-timestamps-in-Jenkins
- feature/8740-new-signing-key-phase-2
- feature/8665-remove-adblock
- bugfix/8756-repair-local-packages
- feature/7530-docker_anonym
- feature/7530-docker-with-apt-cacher-ng
- feature/7963-background-color
- feature/8491-live-additional-software-in-whisperback
- feature/7530-docker
- feature/linux-3.18
- feature/torbrowser-alpha
- bugfix/8747-update-tails-apt-repo-signing-key
- feature/8726-use-homogenous-Debian-mirrors-at-build-time
- feature/5525-sandbox-web-browser
- feature/7752-keyringer
- feature/6739-install-electrum
- bugfix/quote-wrappers-arguments