Commit 2db259ca authored by Zen Fu's avatar Zen Fu
Browse files

Document GitLab and the Tails infrastructure (sysadmin#17733)

parent da32589f
[[!meta title="GitLab and the Tails infrastructure"]]
This page documents a proposal for how to fit the new GitLab repositories in the current Tails workflow from a Sysadmin perspective.
**This is currently under construction and should probably go into the [GitLab Blueprint](https://tails.boum.org/blueprint/GitLab/) once it's mature.**
[[_TOC_]]
# Before GitLab
- Tails Gitolite repository (`puppet-git.lizard:tails.git`) has [hooks](https://git.tails.boum.org/puppet-tails/tree/manifests/gitolite/hooks/tails.pp) to:
- Avoid receiving some references depending on their name ([pre-receive](https://git.tails.boum.org/puppet-tails/tree/templates/gitolite/hooks/tails-pre-receive.hook.erb)).
- Notify Jenkins of updates ([post-receive](https://git.tails.boum.org/puppet-tails/tree/templates/gitolite/hooks/tails-post-receive.erb)) - #51.
- Prevent Weblate from messing the repo ([update](https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/tails-weblate-update.hook)) - #55.
- Ping Ikiwiki so it generates the static website ([installed separatelly](https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_hooks.sh), [post-update](https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_ping-post-update.hook)).
- Some ["underlay" repositories](https://git.tails.boum.org/puppet-tails/tree/manifests/website.pp#n19) also have a [hook](https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_underlays-post-update.hook) to refresh the website.
- Jenkins master VM (`jenkins.lizard`) [has](https://git.tails.boum.org/puppet-tails/tree/manifests/jenkins/iso_jobs_generator.pp) a [cron job](https://git.tails.boum.org/puppet-tails/tree/files/jenkins/master/update_tails_iso_jobs) that:
- Pulls from the Tails Gitolite repository (`gitolite@puppet-git.lizard:tails.git`) - #52.
- Pushes to the Jenkins jobs repository (`gitolite@puppet-git.lizard:jenkins-jobs.git`) to create new jobs.
- Jenkins jobs repository (`puppet-git.lizard:jenkins-jobs.git`) has a [hook](https://git.tails.boum.org/puppet-tails/tree/templates/gitolite/hooks/jenkins_jobs-post-update.hook.erb) that:
- SSH's back into the Jenkins master VM.
- Runs [a script](https://git.tails.boum.org/puppet-tails/tree/files/jenkins/master/deploy_jenkins_jobs) to tell Jenkins about the new jobs it should run.
- Jenkins slave VMs:
- [Query redmine](https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/decide_if_reproduce) to determine whether to do reproducibility tests - #53.
- [Query redmine](https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/output_ISO_builds_and_tests_notifications) to decide where to send e-mail - #53.
- Redmine VM:
- Has a [cron job](https://git.tails.boum.org/puppet-tails/tree/manifests/redmine/reminder.pp#n48) that [sends an e-mail](https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/redmine-remind) to remind about stalled issues - #54.
- Ikiwiki, when pinged by `tails.git` or underlays, does the following:
- Pulls from `gitolite@puppet-git.lizard:tails`.
- Generates the website. Maybe commits changes to `.po` files.
- Pushes to `gitolite@puppet-git.lizard:tails` (which [avoids pinging again](https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_ping-post-update.hook#n7) to prevent endless loops.
- Weblate has a [cron job](https://git.tails.boum.org/puppet-tails/tree/files/weblate/scripts/cron.sh) to:
- Fetch from `gitolite@puppet-git.lizard:tails` and [merge](https://git.tails.boum.org/puppet-tails/tree/files/weblate/scripts/merge_canonical_changes.py) into integration repo.
- Fetch from Weblate repo and [merge](https://git.tails.boum.org/puppet-tails/tree/files/weblate/scripts/merge_weblate_changes.py) into integration repo.
- Push changes back to `gitolite@puppet-git.lizard:tails`.
# Considerations
- We will not have SSH access to the VM hosting our GitLab instance. That makes using [Server Hooks](https://docs.gitlab.com/ce/administration/server_hooks.html) impractical.
- GitLab has [Webhooks](https://docs.gitlab.com/ce/user/project/integrations/webhooks.html) that can POST to specified URLs when things happen in repositories, but it implies having to write and run our own endpoint to receive and process these requests.
- All parts that query Redmine have to be adapted to query GitLab instead.
- It makes sense to maintain the Tails Gitolite repository as a "local" (to the infrastructure) clone/mirror of the new GitLab repository.
- GitLab can [mirror](https://docs.gitlab.com/ce/user/project/repository/repository_mirroring.html) it's `tails.git` in Gitolite, (but the [API doc](https://docs.gitlab.com/ce/api/remote_mirrors.html) doesn't mention how to configure SSH keys programatically).
- Currently, both Ikiwiki and Weblate can commit and push changes to `tails.git`. Weblate merge strategy was designed to recover from divergences with its remote, while Ikiwiki has a very weak retry strategy that causes it to break in some situations. If they could both pull/push from/to GitLab directly in the new setup, Gitolite could be a plain mirror of GitLab. Given there's no practical way of refusing to update from Weblate in GitLab directly, we could maintain Weblate pushing to Gitolite, thus using Gitolite as a gatekeeper to proxy Weblate updates to GitLab.
- Ikiwiki can use GitLab as remote to avoid having to go through Gitolite.
- Pushes to Gitolite's `tails.git` trigger `ikiwiki.cgi?do=ping` post-update. This causes Ikiwiki to pull, eventually commit, and push back to canonical repo. If the canonical repo is updated between Ikiwiki's pull and push, then Ikiwiki will fail its push and reiterate. Thus, Ikiwiki is already not free from race conditions.
- We want prevent a privilege escalation in the Weblate VM to endanger the main repo, so we don't want to get rid of a third-party validation of updates. It is not a good idea to use the `tails.git` repo in Gitolite for that because then it'd have to push to GitLab, creating race conditions as it's also a mirror of that repository. We may use a new repository in `puppet-git.lizard` to itself
# After GitLab (proposal)
```
.-------------------Ping-----------------------------------.
v |
.----------. .----------.
| Ikiwiki |--------------Push-----. .-----Mirror----->| Gitolite |<--.
'----------' v | '----------' |
| .--------. | |
'-------------------Pull---->| GitLab |<--------. | |
'--------' | | |
^ ^ ^ Needs | |
.------------. | | | validation? Ping Pull
| Weblate | | | | | | |
| Gatekeeper |-------------Push------' | | .----------. | |
'------------' | | | Jenkins | | |
^ | | | Slaves | | |
| | | '----------' v |
Force Push Pull | ^ .----------.
| | | | | Jenkins |
| .----------. | | '---Start---| Master |
'------------| Weblate |--------' | jobs '----------'
'----------' | | ^
| | |
Get stalled Push SSH
issues info | |
| v |
.---------------. .------------------.
| gitlab-remind | | jenkins-jobs.git |
'---------------' '------------------'
|
v
Send e-mail
about stalled
issues
```
- GitLab mirrors (push) `gitlab.tails.boum.org/tails-team/tails.git` to Gitolite.
- Ikiwiki now uses GitLab as remote.
- Weblate now pulls from GitLab and pushes to a gatekeeper repo in another VM which itself has a hook to insistently rebase and push to GitLab.
- Gitolite still pings Ikiwiki and Jenkins on every update.
- Jenkins slaves now query GitLab.
- The stalled issues reminder now queries GitLab.
# Action plan (proposal)
- Query "Needs Validation" label from GitLab issues - #53.
- Create a reminder based on GitLab to send e-mail about stalled issues - #54.
- Mirror infrastructural repositories in GitLab - #49.
- Make Gitolite's `tails.git` a mirror of GitLab's `tails.git` - #57.
- Mirror underlay repositories in Gitolite so pushes to GitLab will trigger website refreshes - #50.
- Make Ikiwiki push/pull to/from GitLab - #61.
- Make Weblate pull from GitLab - #55.
- Make Weblate push to GitLab using a gatekeeper repo - #56.
If we go with the proposal above, the following wouldn't need to be done:
- #51, although it could be implemented as GitLab Webhooks (note: there's no auth layer to interact with Jenkins for now).
- #52, although it'd be very easy to change remote.
- #58, because it's now unneeded.
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment