|
|
[[!tag archived]]
|
|
|
|
|
|
[[!meta title="Automated tests specification"]]
|
|
|
|
|
|
*Ticket*: [[!tails_ticket 8667]]
|
|
|
---
|
|
|
title: Automated tests specification
|
|
|
---
|
|
|
|
|
|
|
|
|
*Ticket*: tails/tails#8667
|
|
|
|
|
|
This blueprint helps to keep track of the discussions and decisions
|
|
|
about the specification of automated tests ran in Tails' Jenkins on
|
|
|
the [[automatically build ISOs|autobuild_specs]] ([[!tails_ticket 5288]]).
|
|
|
the [automatically build ISOs](automated_builds_and_tests/autobuild_specs) (tails/tails#5288).
|
|
|
|
|
|
|
|
|
[[_TOC_]]
|
|
|
|
|
|
[[!toc levels=3]]
|
|
|
|
|
|
# Facts
|
|
|
|
|
|
Running the full test suite on 1 isotester hosted on Lizard takes
|
|
|
around 8 hours.
|
|
|
We intend to run __4 isotesters__, so at the moment we would be able
|
|
|
to run __12 full test suites__ per day. ([[!tails_ticket 6094]] will
|
|
|
to run __12 full test suites__ per day. (tails/tails#6094 will
|
|
|
save ~25% on the full test suite runtime, but we'll keep on adding
|
|
|
more tests, so let's stick to the 8 hours estimate for now.)
|
|
|
|
... | ... | @@ -38,7 +43,7 @@ raise to in between __10/20 ISOs/day__. |
|
|
There will probably be new hardware at the end of the year to deploy
|
|
|
more isotesters. If a machine is dedicated to that usage, we can throw
|
|
|
in faster CPUs and run the test suite on bare metal, which would
|
|
|
speed up the test process. That's [[!tails_ticket 9264]].
|
|
|
speed up the test process. That's tails/tails#9264.
|
|
|
|
|
|
So in this discussion, we have to think to a deployment that might
|
|
|
have two iterations with different computational powers (and thus
|
... | ... | @@ -66,7 +71,7 @@ base branches and features branches that are in ReadyForQA status in |
|
|
Redmine.
|
|
|
|
|
|
We might want to play with job priorities, the same way we'll do with
|
|
|
the automated builds (see [[!tails_ticket 9760]]). Luckily, configuring
|
|
|
the automated builds (see tails/tails#9760). Luckily, configuring
|
|
|
that for the builds will probably also work by itself for their automated
|
|
|
tests, as this jobs will be chained together.
|
|
|
|
... | ... | @@ -104,7 +109,7 @@ fresh _--tmpdir_ for each run. |
|
|
The automated test suite MUST be run on a freshly built ISO,
|
|
|
corresponding to the commit it tests.
|
|
|
|
|
|
When [[!tails_ticket 9515]] and friends will be resolved and a first
|
|
|
When tails/tails#9515 and friends will be resolved and a first
|
|
|
deployment of the automated test suite will clarify its need, we'll see
|
|
|
if it should be able to accept a treshold of failures for some features
|
|
|
before sending notifications. This could help if a scenario fails
|
... | ... | @@ -131,7 +136,7 @@ captures for failing steps, snapshots of the test VM, and also videos of |
|
|
the running test session.
|
|
|
|
|
|
We can keep the video captures in the build artifacts, now that
|
|
|
[[!tails_ticket 10001]] is resolved.
|
|
|
tails/tails#10001 is resolved.
|
|
|
|
|
|
Decision:
|
|
|
|
... | ... | @@ -140,7 +145,7 @@ Decision: |
|
|
* For red test suite run: keep the screenshots and video captures, the
|
|
|
logs and the pcap files.
|
|
|
|
|
|
In [[!tails_ticket 10155]] we calculated that we can probably keep the
|
|
|
In tails/tails#10155 we calculated that we can probably keep the
|
|
|
video captures for a full release cycle. This will be refine is reality
|
|
|
claims the contrary after an evaluation.
|
|
|
|
... | ... | @@ -259,3 +264,4 @@ really tightened. |
|
|
As a Tails developer
|
|
|
When I push a new commit or a new Debian package to a base branch
|
|
|
I want the affected feature branches ISO to be tested with that change
|
|
|
|