Rename blueprints: .mdwn → .md authored by intrigeri's avatar intrigeri
Rationale:

 - Right now:

   - These pages' Markdown will be rendered when browsing our
     repository in the GitLab web interface

   - The .md extension is more common nowadays: GitHub, GitLab, and various
     other major players have settled on it. Let's get used to it.

 - If we decide to migrate our blueprints to GitLab (#18079),
   this will be a necessary first step.
This is about migrating our data and workflow from Redmine to GitLab,
which is tracked as [[!tails_ticket 15878]].
[[!toc levels=3]]
<a id="access-control"></a>
# Access control
## Objects
- _Canonical Git repo_: the authoritative `tails.git` repository
hosted on GitLab
- _Major branches_: `master`, `stable`, `testing`, `devel`,
and possibly `feature/bullseye`
- _Release tags_: a signed Git tag that identifies the source code
used to build a specific Tails release; currently all tags
in the authoritative `tails.git` repository are release tags;
the tag name is a version number, with '~' replaced by '-'.
- _Particularly sensitive data_: confidential data that specific teams
like Fundraising and Accounting need to handle, but that other
contributors generally don't need direct access to. This definitely
include issues; this might include Git repositories at some point.
Note that as of 2019-11-20, it is undefined:
- What subset of this data can go to a web-based issue tracker or not.<br/>
This is already a problem with Redmine.<br/>
Fixing this will require discussions between various stakeholders.
- What subset of this data could live in a cleartext Git
repository hosted here or there, as opposed to requiring
end-to-end encryption between members of these teams.
This is a hypothetical problem for now.
## Subjects
- An _admin_ can do anything that other roles can, and:
- can delete issues
- can edit team membership
- MUST comply with our "Level 3" security policy
- can view issues that contain particularly sensitive data
- can create new groups and projects that are not forks
- A _committer_:
- can push and force-push to any ref in the canonical Git repo,
including major branches and release tags;<br/>
incidentally, this ensures the following requirement is met:
- their branches are picked up by Jenkins; it follows that they
MUST comply with our "Infrastructure" security policy
- can merge MRs into major branches
- can modify issues metadata
- MAY be allowed to view confidential issues in the main GitLab
project; if it's the case, then particularly sensitive data MUST
live somewhere else with stricter access control
- can edit other users' comments
- MAY be allowed to add new team members
- MUST comply with our "Level 3" security policy
- A _regular, particularly trusted contributor_:
- can push and force-push to a subset of refs in the canonical Git repo;
this subset MUST NOT include any major branch nor release tag;<br/>
this is required to ensure the following requirement is met:
- their branches are picked up by Jenkins; it follows that they
MUST comply with our "Infrastructure" security policy
- can modify issues metadata
- MAY be allowed to view confidential issues in the main GitLab
project; if it's the case, then particularly sensitive data MUST
live somewhere else with stricter access control
- A _regular contributor_:
- can fork the Git repositories and push changes to their own fork
- can modify issues metadata
- MAY be allowed to view confidential issues in the main GitLab
project; if it's the case, then particularly sensitive data MUST
live somewhere else with stricter access control
- _Anybody with a GitLab account_ on the instance we use:
- can submit issues
- can submit MRs
## Implementation
### Relevant GitLab doc
- [Permissions](https://docs.gitlab.com/ee/user/permissions.html)
- [Authorization for Merge requests](https://docs.gitlab.com/ce/user/project/merge_requests/authorization_for_merge_requests.html)
- [Protected Branches](https://docs.gitlab.com/ce/user/project/protected_branches.html)
- [Groups](https://docs.gitlab.com/ee/user/group/) and [Subgroups](https://docs.gitlab.com/ee/user/group/subgroups/)
### Common settings
- pipelines: disabled
- wiki: disabled
### The parent "tails" group
- Visibility: Public
- The `root` user has "Owner" access to this group.
- Release managers have "Reporter" access to every project in this group
(after releasing a new Tails, they need to postpone issues,
including confidential ones, and manage group labels),
until [[!tails_ticket 17589]] is done.
- Nobody else is a member of this group.
- Allow users to request access: disabled
### Most public projects
The following applies to each of our public project that
has no dedicated section below:
- [Protected branch flow](https://docs.gitlab.com/ce/user/project/merge_requests/authorization_for_merge_requests.html#protected-branch-flow):
it's the most adequate for regular contributors, since it's the
least disruptive in terms of workflow and habits and requires no
work to adjust our Jenkins CI setup.
- We mark our major branches as "Protected".
- Committers get "Maintainer" access.
- Maintainers are allowed to merge and to push to
protected branches.
- The Git workflow remains unchanged for regular developers who are
not granted full commit access but have access to our CI:
they get "Developer" access, can
push a topic branch to the canonical Git repository and our CI will
pick it up. The only difference is that they are not restricted to
pushing to their own `$nickname/*` namespace, which makes things
simpler and has other advantages, e.g. they can use the `wip/`
prefix (so our Jenkins CI ignores the branch) and collaborate with
others on shared branches.
- Other contributors get access strictly lower than "Developer".
They push topic branches to their own fork of the repository and
create merge requests.
- Our current Jenkins CI jobs generation process remains unchanged.
(Technically, we could adjust it so it generates CI jobs for _any_
merge request (based on `refs/merge-requests/*/head`), but this
would give arbitrary code execution on our CI to _anyone_.
Our infrastructure is not currently equipped to cope with this.)
- issues: disabled except for the `tails/tails` project
- MRs: enabled
- Allow users to request access: enabled
### rm
- visibility: private
- issues: disabled
- MRs: disabled (gcrypt)
- repository: enabled
- Allow users to request access: disabled
- Nobody has access except RMs
### Public sysadmin projects
Projects:
- `bitcoin`
- `bitcoin_libunivalue` → rename to `libunivalue`
- public `puppet-*` (additional "Maintainer": role-lizard)
- `jenkins-jobs` (additional "Maintainer": role-lizard)
Properties:
- public
- [forking workflow](https://docs.gitlab.com/ce/user/project/merge_requests/authorization_for_merge_requests.html#forking-workflow)
- issues: disabled (tracked in tails/tails)
- MRs: enabled
- regular Tails contributors have "Reporter" access
### tails/accounting, tails/fundraising, tails/private
- visibility: private
- issues: enabled
- MRs: disabled
- repository: disabled (could be enabled later if these teams
want their git-remote-gcrypt repo migrated)
- Allow users to request access: disabled
- Maintainer access given to:
- accounting: accounting team
- fundraising: fundraising team
- private: tails@ members
### tails/sysadmin
- visibility: public
- issues: enabled
- MRs: disabled
- repository: disabled (contributors with "Guest" access would be allowed
to clone it)
- Allow users to request access: disabled
- regular contributors have "Guest" access level (so they can create new issues
but not see confidential ones)
- Maintainer access given to: sysadmin team
### test-suite-shared-secrets
- visibility: private
- issues: disabled
- MRs: enabled
- repository: enabled
- Allow users to request access: disabled
- Nobody has access except committers
### mirror-pool
- visibility: public
- issues: disabled
- MRs: enabled
- repository: enabled
- Allow users to request access: disabled
- mirrors team members have "Maintainer" access
- [forking workflow](https://docs.gitlab.com/ce/user/project/merge_requests/authorization_for_merge_requests.html#forking-workflow)
- regular Tails contributors have "Reporter" access
## Management
With Redmine + Gitolite, we centrally manage, in 1-2 single places, team
membership and the corresponding access level. For example, when someone gets
commit access or joins a team, we add them to the corresponding Gitolite group
and Redmine role. That's very convenient.
If we don't do anything special, to implement the access control specified
above, we would need to:
* During the migration to GitLab: give dozens of users "Reporter", "Maintainer"
or "Developer" access on dozens of project.
* Day-to-day operations: occasionally, give someone "Reporter", "Maintainer" or
"Developer" access on dozens of project.
I don't think we want to do any of this manually.
It seems we could maintain user groups via the GitLab web interface and [share
the relevant projects with these user
groups](https://docs.gitlab.com/ce/user/project/members/share_project_with_groups.html).
These groups would not contain any project themselves: their primary purpose is to
manage access to projects in batch.
And in passing, such user groups would allow us to mention a team with one
single `@xyz-team` or `@xyz-group`. For example, `@sysadmins-team`.
Another option would be to use
[subgroups](https://docs.gitlab.com/ce/user/group/subgroups/), and have for
example `tails/accounting-team/accounting`, but references to issues etc.
become pretty long, e.g. "tails/accounting-team/accounting#1234", compared to
"tails/accounting#1234" in the current implementation. Unless there's a way
around this problem, that's a usability deal breaker and we should not use
subgroups merely to simplify access level management.
# Wiki
It's out of scope for the first iteration but at some point, we might
want to migrate our blueprints to GitLab's
[wiki](https://docs.gitlab.com/ce/user/project/wiki/):
[[!tails_ticket 9174]]).
<a id="git"></a>
# Git repositories
Migration includes preserving access rights, which are documented
for each repository below.
## Must be migrated
From `git.tails.boum.org` aka. `git-tails.immerda.ch`:
- `bitcoin` (public; read-write: sysadmins)
- `bitcoin_libunivalue` (public; read-write: sysadmins)
- `chutney` (public; read-write: committers)
- `htp` (public; read-write: committers)
- `jenkins-tools` (public; read-write: committers)
- `liveusb-creator` → rename to `installer` (public; read-write: committers)
- `network-manager` (public; read-write: committers)
- `onioncircuits` (public; read-write: committers)
- `rm` (private; read-write: RMs)
- `tails-workarounds` → rename to `workarounds` (public; read-write: committers)
- `test-suite-shared-secrets` (private; read-write: committers)
- `torbrowser-launcher` (public; read-write: committers)
- `ux` (public; read-write: committers)
- `verification-extension` (public; read-write: committers)
- `whisperback` (public; read-write: committers)
From `puppet-git.lizard`:
- `etcher-binary` (public; read-write: committers)
- `mirror-pool` (public; read-write: mirrors team)
- `mirror-pool-dispatcher` (public; read-write: committers)
- `promotion-material` (public; read-write: committers)
Read-only mirrors to be migrated from `git.tails.boum.org` aka.
`git-tails.immerda.ch` (with a description that makes it clear
to people with commit access that it's a read-only mirror):
- public `puppet-*` (public; read-write: role-lizard)
- `jenkins-jobs` (public; read-write: role-lizard)
## Leave on immerda for now
- `download-and-verify-extension`: obsolete
- `iceweasel`: very big, unused since years
- `thunderbird`: very big, hopefully obsolete soon
- `uui-binary`: obsolete
<a id="interfaces"></a>
# Interfaces that will need adjustments
## Git repository → various mirrors
Our Git repository keeps a number of mirrors up-to-date:
- <https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_remotes.sh>
- <https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_hooks.sh>
In particular:
- repositories that must now be mirrored on our GitLab
(in addition to, or instead of, the current immerda remote):
- public `puppet-*`
- `jenkins-jobs`
## Git repositories → website
A refresh of our website is triggered by:
- pushing to the master branch of our Git repository
- pushing to a few "underlay" repositories
Current implementation:
- <https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_hooks.sh>
- <https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_ping-post-update.hook>
- <https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_underlays-post-update.hook>
- <https://git.tails.boum.org/puppet-tails/tree/templates/website/update-underlays.erb>
## Git repository → Jenkins
Pushing to our Git repository pings Jenkins so it can run jobs as needed:
<https://git.tails.boum.org/puppet-tails/tree/templates/gitolite/hooks/tails-post-receive.erb>
<a id="interfaces-jenkins-git"></a>
## Jenkins → Git
Jenkins jobs are generated on `jenkins.lizard`, from the list of
branches in our main Git repository. For details and pointers to the
corresponding code, see the corresponding
[[blueprint|blueprint/automated_builds_and_tests/jenkins]].
Here are the kinds of jobs relevant in this discussion:
- `check_PO_master` runs the `check_po` script on all PO files
- this script comes from a Git submodule referenced by
`tails.git`'s master branch (`tails::tester::check_po`)
- in a Jenkins isobuilder
- as a sudoer user
- `build_website_master` runs `./build-website`
- from `tails.git`'s master branch
- in a Jenkins isobuilder
- as a sudoer user
- `build_Tails_ISO_*` and `reproducibly_build_Tails_ISO_*` run
`rake build`
- from the corresponding `tails.git` branch
- in a Jenkins isobuilder
- as a sudoer user
- `test_Tails_ISO_*` run `./run_test_suite`
- from the corresponding `tails.git` branch
- in a Jenkins isotester
- as root via sudo
Wrt. who can push to which branch in `tails.git`, see the "Access control"
section.
## Jenkins → Redmine
Tails images build reproducibility is tested if the corresponding
ticket has "Needs Validation" status:
<https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/decide_if_reproduce>
(Currently disabled:) Email notifications are sent on job status
change if the ticket has "Needs Validation" status:
<https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/output_ISO_builds_and_tests_notifications>
## Ticket triaging → Redmine
Our Redmine email reminder sends email to users who have at least one
ticket assigned to them, that satisfies at least one of these criteria:
- "Stalled work-in-progress": status "In Progress", that were
not updated since more than 6 months
- "Reviews waiting for a long time": not been updated since 45 days
or more
Current implementation:
- <https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/redmine-remind>
- <https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/email_body>
- <https://git.tails.boum.org/puppet-tails/tree/manifests/redmine/reminder.pp>
With GitLab one can define policies (YAML) and have them applied automatically
to issues and merge requests:
- <https://about.gitlab.com/handbook/engineering/quality/triage-operations/#triage-automation>
- <https://gitlab.com/gitlab-org/quality/triage-ops>
- <https://gitlab.com/gitlab-org/gitlab-triage>
Potential future work: [[!tails_ticket 17589]].
## Translation platform → Git
A hook ensures that Weblate only pushes changes to our main Git
repository that meet certain conditions:
<https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/tails-weblate-update.hook>
# GitLab hosting options
See <https://salsa.debian.org/tails-team/gitlab-migration/wikis/hosting/comparison>.
# Resources
* Tor's [migration plan](https://nc.torproject.net/s/3MpFApQ7cwfrPZE)
from Trac to GitLab.
- Their instance is <https://dip.torproject.org/>.
- Converted issues: <https://dip.torproject.org/ahf-admin/legacy-20/issues/>
- Migration problems: <https://pad.riseup.net/p/gitlab-migration-problems>
- Timeline: none as of 2020-02-27
- Older meetings:
- [agenda and notes](https://pad.riseup.net/p/e-q1GP43W4gsY_tYUNxf)
- 2019-10-15 meeting
[logs](http://meetbot.debian.net/tor-meeting/2019/tor-meeting.2019-10-15-17.01.html)
- 2019-10-01 meeting
[logs](http://meetbot.debian.net/tor-meeting/2019/tor-meeting.2019-10-01-18.00.html)
* KDE migration to GitLab:
- <https://gitlab.com/gitlab-org/gitlab-foss/issues/57338/designs>
- <https://gitlab.com/gitlab-org/gitlab/issues/24900>
- Wrt. workflow (e.g. does everyone push to their own fork or are some people
allowed to push their topic branches to the main repo?), see:
- <https://marc.info/?t=155298413600004&r=1&w=2>
- <https://marc.info/?t=155091510600001&r=3&w=2>
- <https://marc.info/?l=kde-devel&m=155095845826395&w=2>
- <https://marc.info/?l=kde-devel&m=155094926223103&w=2>
* Project
[import/export](https://docs.gitlab.com/ee/user/project/settings/import_export.html):
successfully tested from Salsa to 0xacab (code, MRs, no issues).
Notes are authored by the user who performs the import,
but attributed in the comment itself to their original author.
Commits are authored by the original author.
Example: <https://0xacab.org/tails/test-import/>