Commit f1680f20 authored by intrigeri's avatar intrigeri
Browse files

Report findings about the usability our current CI

Closes #16959
parent 194ecaec
[[!meta title="CI usability issues"]]
Here, we collect CI usability issues.
This effort started with [[!tails_ticket 16959]], in order to have a better
understanding of how our current setup feels to its users.
This data will help us define our strategy for the future of our CI (e.g.
switching to GitLab CI, switching to Jenkins pipelines, or merely removing some
UX stumbling blocks without changing the big picture of our setup).
[[!toc levels=2]]
# Cumbersome navigation
* Finding the Jenkins jobs corresponding to a given branch or MR is cumbersome, because:
- Jenkins does not readily expose the full CI pipeline progress & outcome
⇒ developers and reviewers need to track/check the result of 3 different jobs.
([[!tails_gitlab tails/sysadmin/-/issues/17071]])
- CI results are not visible on GitLab MRs
([[!tails_gitlab tails/sysadmin/-/issues/17070]])
# Misleading output
* When a Cucumber _After_ hook fails, e.g. `@check_tor_leaks`, the corresponding
scenario is mistakenly listed as successful o the Jenkins _Cucumber reports_
page. ([[!tails_ticket 16150]])
* The _Revision: $COMMIT_ information, that's displayed on the page of
a `test_Tails_ISO_*` job run, may be incorrect. In order to determine what
commit the test suite was run from, look for `git reset --hard` in the console
output of the job run.
* The success/failure of the `keep_node_busy_during_cleanup` job does
not matter.
# Missing information
* When a Jenkins build aborts due to a timeout, no summary of the scenarios
that did run is generated. ([[!tails_ticket 17678]])
# Very long feedback loop
As of August 2020, a full CI pipeline takes almost 7 hours to run.
This problem is tracked on [[!tails_gitlab tails/sysadmin/-/issues/16960]] and
[[a dedicated blueprint|blueprint/hardware_for_automated_tests_take3]], which
focuses on the hard facts.
Here we focus on feelings and human perception.
## Release Managers
Several CI jobs are on the critical path of our release process, which forces
the RM to wait. In a situation that can already be stressful, this can be tough
on their patience.
## Developers
Some developers have adapted their workflow around this constraint. They feel
that making the CI loop twice shorter would not make a significant difference to
them: they would anyway come back to it the next day.
Another developer, who routinely use a replica of our CI setup that's twice
faster, instead feels that this allows them to iterate faster, complete work on
a given task earlier, and limit context-switching. With a 3.5 hours feedback
loop, it becomes possible _in a single day_ to:
1. do some work
2. wait for CI results
3. fix bugs found by this first iteration
4. send the second iteration to CI
5. check the CI results for the second iteration the next day
# Robustness problems
Note: this is not about robustness problems inherent to our test suite,
that don't depend much on _how_ we run the test suite.
* When GitLab is down or connectivity between lizard and GitLab is poor, many
Jenkins jobs — if not all — fail. ([[!tails_gitlab
tails/sysadmin/-/issues/17715]])
......@@ -52,16 +52,11 @@ to rebuild images first. To do so:
## Known usability issues
We collect a list of other known CI usability issues on [[!tails_ticket 16959]].
We collect a list of other known CI usability issues on [[a dedicated
blueprint|blueprint/CI_usability]].
Apart of that, be aware that:
* The _Revision: $COMMIT_ information, that's displayed on the page of
a `test_Tails_ISO_*` job run, may be incorrect. In order to determine what
commit the test suite was run from, look for `git reset --hard` in the console
output of the job run.
* The success/failure of the `keep_node_busy_during_cleanup` job does
not matter.
If something feels odd, misleading, or confusing, please check that page:
perhaps it explains the problem you're experiencing; else, please add it.
# For sysadmins
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment