title: Automated tests specification
Ticket: tails#8667 (closed)
- Things to keep in mind
- Future ideas
Running the full test suite on 1 isotester hosted on Lizard takes around 8 hours. We intend to run 4 isotesters, so at the moment we would be able to run 12 full test suites per day. (tails#6094 (closed) will save ~25% on the full test suite runtime, but we'll keep on adding more tests, so let's stick to the 8 hours estimate for now.)
We have 2 isobuilders on Lizard, that build a total of a bit less than 400 ISOs/month (that's an average of 13 ISOs/day). At the moment, we build from 6 ISOs to 30-40 ISOs a day, depending on the activity.
We usually build the stable, devel, experimental, feature/jessie (+ testing sometimes) and a bunch of other branches.
This numbers are expected to grow when the automated builds will be put in production. It's difficult to guess what would the maximum number of builds per day be, but the minimum should be expected to raise to in between 10/20 ISOs/day.
Things to keep in mind
There will probably be new hardware at the end of the year to deploy more isotesters. If a machine is dedicated to that usage, we can throw in faster CPUs and run the test suite on bare metal, which would speed up the test process. That's tails#9264 (closed).
So in this discussion, we have to think to a deployment that might have two iterations with different computational powers (and thus different amounts of tests/day possible), and the defined implementation should be modular enough to handle both of them without too many changes.
When to test the builds
It doesn't sound reasonable to run the full test suite on every ISO automatically built by Jenkins, at least in the first iteration.
We need to lessen the number of tests per day. Probably even in the second iteration, with the expected growth of the number of automated builds.
At first, we'll try to run the full test suite on all of them. If our infra can't cope with that, we have ideas written below to help in that. The one that everyone seem to agree on is to run the full test suite for base branches and features branches that are in ReadyForQA status in Redmine.
We might want to play with job priorities, the same way we'll do with the automated builds (see tails#9760 (closed)). Luckily, configuring that for the builds will probably also work by itself for their automated tests, as this jobs will be chained together.
For branch stable: Must test ISOs built daily and on every Git push; so that we're always ready to put out an emergency release; For the next scheduled release branch (either stable, testing, or devel): Must test ISOs built daily and on every Git push; so that we always know the state of the next release; For other branches: For branches with a `Needs Validation` ticket: Must test ISO built daily and on every Git push; Must test ISOs built on every Git push, with some rate-limiting if necessary;
How to run the tests
This section heavily depends on the discussion about the previous one.
To test a specific ISO, the automated test suite MUST have access to:
- the corresponding build artifacts;
- the commit ID of that build;
- if applicable, the commit at which the base branch was at, when it was merged into the branch being built;
- the ISO of the previous Tails release.
The automated test suite MUST be run in a clean environment, using a fresh --tmpdir for each run.
The automated test suite MUST be run on a freshly built ISO, corresponding to the commit it tests.
When tails#9515 (closed) and friends will be resolved and a first deployment of the automated test suite will clarify its need, we'll see if it should be able to accept a treshold of failures for some features before sending notifications. This could help if a scenario fails e.g. because of a network congestion.
Email will be the main interface. Use the same settings than for automated builds:
- For base branches, notify the RM.
- For topic branches, notify the author of the last commit.
When notifying Redmine, the use of the ReadyForQA custom field is not so semantically correct, and may overlap with the dev usage of it. We will probably have to add a new one for Jenkins to decide to test the branch if we choose to test only the feature branch that are ready to be reviewed.
What kind of result shall be kept
The test suite produces different kind of artifacts: logfiles, screen captures for failing steps, snapshots of the test VM, and also videos of the running test session.
We can keep the video captures in the build artifacts, now that tails#10001 (closed) is resolved.
- For green test suite run: keep the test logs (Jenkins natively do that).
- For red test suite run: keep the screenshots and video captures, the logs and the pcap files.
In tails#10155 (closed) we calculated that we can probably keep the video captures for a full release cycle. This will be refine is reality claims the contrary after an evaluation.
In the following scenario:
- topic branches are named branch F
- base branches are named branch B
We also assume that the following scenarios expectations only cover the test suite artifacts, the ISO one being covered in the automated builds specification.
Scenario 1 : reviewer
As a reviewer When I'm asked to review branch F into branch B Then I need to know if branch F passes the automated tests once locally merged into branch B and built (fresh results!) And if all the automated tests scenarios suceeded The resulting test logs must be made available to me The Redmine ticket should be notified Otherwise the developer who proposed the merge must be notified And the developer *needs* to see the test logs and screenshots And the ticket should be reassigned to the branch submitter And Status should be set to "In Progress"
Scenario 2 : developer
As a developer who has the commit bit When I'm working on branch F Then I need to know if my branch passes the automated tests suite once locally merged in base branch B And I need to know if my branch fails to pass the automated test suite because of an external change possibly weeks after my last commit (by e.g Debian changes, changes in branch B, ...) And if my branch passes the automated test suite The resulting test logs must be made available to me The Redmine ticket should be notified Otherwise I *need* to see the build logs and screenshots And I must be notified And the ticket should be reassigned to me if needed And Status should be set to "In Progress"
Scenario 3 : RM
As the current RM When working the full dev release cycle Then I need to know if a base branch does not pass the automated test suite And when this happens the build logs and screenshots must be made available to me.
Cutting the number of run per day
For feature branches, we could run the full test suite only on the daily builds or on "Needs Validation" ones, and either only the automated tests related to the branch on every git push, and/or a subset of the whole test suite.
We can also maybe find more ways to split the automated test suite in faster subsets of features depending on the context, define priorities for built ISO and/or tests.
The automated test suite could be able to run features in parallel for a single automated build ISO. This way, if more than one isotester are idle, it can use several of them to test an ISO faster.
The automated suite then should be able when there are more than one ISO queued for testing to fairly distribute the parallelizing of their features.
The automated test suite also should not allocate all the isotesters for one ISO when others are waiting to be tested.
Folowing is a list of other scenarios that we have also envisaged for the second iteration of the automated builds deployment, as they are really tightened.
As a Tails developer working on branch B When I upload a package to APT suite B Then I want to know if the automated test suite passes
(same responsiveness as when pushing to git) (acceptable workaround: being able to manually trigger a test suite.)
As the current RM When I push new tag T on branch B Then I want the APT suite for tag T to be created And I want the APT suite B to be copied into the APT suite T And once this is done, I want the automated test suite to be run on the ISO build from the checkout of tag T
As a Tails developer When the test suite is ran on the ISO build from my last commit I want to watch TV and see the test video in HTML5 from the Tor Browser
As a Tails developer When I push a new commit or a new Debian package to a base branch I want the affected feature branches ISO to be tested with that change