This is about migrating our data and workflow from Redmine to GitLab, which is tracked as tails#15878 (closed).
- Access control
- Wiki
- Git repositories
- Interfaces that will need adjustments
- GitLab hosting options
- Resources
Access control
Objects
-
Canonical Git repo: the authoritative
tails.git
repository hosted on GitLab -
Major branches:
master
,stable
,testing
,devel
, and possiblyfeature/bullseye
-
Release tags: a signed Git tag that identifies the source code used to build a specific Tails release; currently all tags in the authoritative
tails.git
repository are release tags; the tag name is a version number, with '~' replaced by '-'. -
Particularly sensitive data: confidential data that specific teams like Fundraising and Accounting need to handle, but that other contributors generally don't need direct access to. This definitely include issues; this might include Git repositories at some point.
Note that as of 2019-11-20, it is undefined:
-
What subset of this data can go to a web-based issue tracker or not.
This is already a problem with Redmine.
Fixing this will require discussions between various stakeholders. -
What subset of this data could live in a cleartext Git repository hosted here or there, as opposed to requiring end-to-end encryption between members of these teams. This is a hypothetical problem for now.
-
Subjects
-
An admin can do anything that other roles can, and:
- can delete issues
- can edit team membership
- MUST comply with our "Level 3" security policy
- can view issues that contain particularly sensitive data
- can create new groups and projects that are not forks
-
A committer:
- can push and force-push to any ref in the canonical Git repo,
including major branches and release tags;
incidentally, this ensures the following requirement is met: - their branches are picked up by Jenkins; it follows that they MUST comply with our "Infrastructure" security policy
- can merge MRs into major branches
- can modify issues metadata
- MAY be allowed to view confidential issues in the main GitLab project; if it's the case, then particularly sensitive data MUST live somewhere else with stricter access control
- can edit other users' comments
- MAY be allowed to add new team members
- MUST comply with our "Level 3" security policy
- can push and force-push to any ref in the canonical Git repo,
including major branches and release tags;
-
A regular, particularly trusted contributor:
- can push and force-push to a subset of refs in the canonical Git repo;
this subset MUST NOT include any major branch nor release tag;
this is required to ensure the following requirement is met: - their branches are picked up by Jenkins; it follows that they MUST comply with our "Infrastructure" security policy
- can modify issues metadata
- MAY be allowed to view confidential issues in the main GitLab project; if it's the case, then particularly sensitive data MUST live somewhere else with stricter access control
- can push and force-push to a subset of refs in the canonical Git repo;
this subset MUST NOT include any major branch nor release tag;
-
A regular contributor:
- can fork the Git repositories and push changes to their own fork
- can modify issues metadata
- MAY be allowed to view confidential issues in the main GitLab project; if it's the case, then particularly sensitive data MUST live somewhere else with stricter access control
-
Anybody with a GitLab account on the instance we use:
- can submit issues
- can submit MRs
Implementation
Relevant GitLab doc
Common settings
- pipelines: disabled
- wiki: disabled
The parent "tails" group
- Visibility: Public
- The
root
user has "Owner" access to this group. - Release managers have "Reporter" access to every project in this group (after releasing a new Tails, they need to postpone issues, including confidential ones, and manage group labels), until tails#17589 (closed) is done.
- Nobody else is a member of this group.
- Allow users to request access: disabled
Most public projects
The following applies to each of our public project that has no dedicated section below:
- Protected branch flow: it's the most adequate for regular contributors, since it's the least disruptive in terms of workflow and habits and requires no work to adjust our Jenkins CI setup.
- We mark our major branches as "Protected".
- Committers get "Maintainer" access.
- Maintainers are allowed to merge and to push to protected branches.
- The Git workflow remains unchanged for regular developers who are
not granted full commit access but have access to our CI:
they get "Developer" access, can
push a topic branch to the canonical Git repository and our CI will
pick it up. The only difference is that they are not restricted to
pushing to their own
$nickname/*
namespace, which makes things simpler and has other advantages, e.g. they can use thewip/
prefix (so our Jenkins CI ignores the branch) and collaborate with others on shared branches. - Other contributors get access strictly lower than "Developer". They push topic branches to their own fork of the repository and create merge requests.
- Our current Jenkins CI jobs generation process remains unchanged.
(Technically, we could adjust it so it generates CI jobs for any
merge request (based on
refs/merge-requests/*/head
), but this would give arbitrary code execution on our CI to anyone. Our infrastructure is not currently equipped to cope with this.) - issues: disabled except for the
tails/tails
project - MRs: enabled
- Allow users to request access: enabled
rm
- visibility: private
- issues: disabled
- MRs: disabled (gcrypt)
- repository: enabled
- Allow users to request access: disabled
- Nobody has access except RMs
Public sysadmin projects
Projects:
bitcoin
-
bitcoin_libunivalue
→ rename tolibunivalue
- public
puppet-*
(additional "Maintainer": role-lizard) -
jenkins-jobs
(additional "Maintainer": role-lizard)
Properties:
- public
- forking workflow
- issues: disabled (tracked in tails/tails)
- MRs: enabled
- regular Tails contributors have "Reporter" access
tails/accounting, tails/fundraising, tails/private
- visibility: private
- issues: enabled
- MRs: disabled
- repository: disabled (could be enabled later if these teams want their git-remote-gcrypt repo migrated)
- Allow users to request access: disabled
- Maintainer access given to:
- accounting: accounting team
- fundraising: fundraising team
- private: tails@ members
tails/sysadmin
- visibility: public
- issues: enabled
- MRs: disabled
- repository: disabled (contributors with "Guest" access would be allowed to clone it)
- Allow users to request access: disabled
- regular contributors have "Guest" access level (so they can create new issues but not see confidential ones)
- Maintainer access given to: sysadmin team
test-suite-shared-secrets
- visibility: private
- issues: disabled
- MRs: enabled
- repository: enabled
- Allow users to request access: disabled
- Nobody has access except committers
mirror-pool
- visibility: public
- issues: disabled
- MRs: enabled
- repository: enabled
- Allow users to request access: disabled
- mirrors team members have "Maintainer" access
- forking workflow
- regular Tails contributors have "Reporter" access
Management
With Redmine + Gitolite, we centrally manage, in 1-2 single places, team membership and the corresponding access level. For example, when someone gets commit access or joins a team, we add them to the corresponding Gitolite group and Redmine role. That's very convenient.
If we don't do anything special, to implement the access control specified above, we would need to:
- During the migration to GitLab: give dozens of users "Reporter", "Maintainer" or "Developer" access on dozens of project.
- Day-to-day operations: occasionally, give someone "Reporter", "Maintainer" or "Developer" access on dozens of project.
I don't think we want to do any of this manually.
It seems we could maintain user groups via the GitLab web interface and share the relevant projects with these user groups. These groups would not contain any project themselves: their primary purpose is to manage access to projects in batch.
And in passing, such user groups would allow us to mention a team with one
single @xyz-team
or @xyz-group
. For example, @sysadmins-team
.
Another option would be to use
subgroups, and have for
example tails/accounting-team/accounting
, but references to issues etc.
become pretty long, e.g. "tails/accounting-team/accounting#1234", compared to
"tails/accounting#1234" in the current implementation. Unless there's a way
around this problem, that's a usability deal breaker and we should not use
subgroups merely to simplify access level management.
Wiki
It's out of scope for the first iteration but at some point, we might want to migrate our blueprints to GitLab's wiki: tails#9174 (closed)).
Git repositories
Migration includes preserving access rights, which are documented for each repository below.
Must be migrated
From git.tails.boum.org
aka. git-tails.immerda.ch
:
-
bitcoin
(public; read-write: sysadmins) -
bitcoin_libunivalue
(public; read-write: sysadmins) -
chutney
(public; read-write: committers) -
htp
(public; read-write: committers) -
jenkins-tools
(public; read-write: committers) -
liveusb-creator
→ rename toinstaller
(public; read-write: committers) -
network-manager
(public; read-write: committers) -
onioncircuits
(public; read-write: committers) -
rm
(private; read-write: RMs) -
tails-workarounds
→ rename toworkarounds
(public; read-write: committers) -
test-suite-shared-secrets
(private; read-write: committers) -
torbrowser-launcher
(public; read-write: committers) -
ux
(public; read-write: committers) -
verification-extension
(public; read-write: committers) -
whisperback
(public; read-write: committers)
From puppet-git.lizard
:
-
etcher-binary
(public; read-write: committers) -
mirror-pool
(public; read-write: mirrors team) -
mirror-pool-dispatcher
(public; read-write: committers) -
promotion-material
(public; read-write: committers)
Read-only mirrors to be migrated from git.tails.boum.org
aka.
git-tails.immerda.ch
(with a description that makes it clear
to people with commit access that it's a read-only mirror):
- public
puppet-*
(public; read-write: role-lizard) -
jenkins-jobs
(public; read-write: role-lizard)
Leave on immerda for now
-
download-and-verify-extension
: obsolete -
iceweasel
: very big, unused since years -
thunderbird
: very big, hopefully obsolete soon -
uui-binary
: obsolete
Interfaces that will need adjustments
Git repository → various mirrors
Our Git repository keeps a number of mirrors up-to-date:
- https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_remotes.sh
- https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_hooks.sh
In particular:
- repositories that must now be mirrored on our GitLab
(in addition to, or instead of, the current immerda remote):
- public
puppet-*
jenkins-jobs
- public
Git repositories → website
A refresh of our website is triggered by:
- pushing to the master branch of our Git repository
- pushing to a few "underlay" repositories
Current implementation:
- https://git.tails.boum.org/puppet-tails/tree/files/gitolite/install_hooks.sh
- https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_ping-post-update.hook
- https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/www_website_underlays-post-update.hook
- https://git.tails.boum.org/puppet-tails/tree/templates/website/update-underlays.erb
Git repository → Jenkins
Pushing to our Git repository pings Jenkins so it can run jobs as needed: https://git.tails.boum.org/puppet-tails/tree/templates/gitolite/hooks/tails-post-receive.erb
Jenkins → Git
Jenkins jobs are generated on jenkins.lizard
, from the list of
branches in our main Git repository. For details and pointers to the
corresponding code, see the corresponding
blueprint.
Here are the kinds of jobs relevant in this discussion:
-
check_PO_master
runs thecheck_po
script on all PO files- this script comes from a Git submodule referenced by
tails.git
's master branch (tails::tester::check_po
) - in a Jenkins isobuilder
- as a sudoer user
- this script comes from a Git submodule referenced by
-
build_website_master
runs./build-website
- from
tails.git
's master branch - in a Jenkins isobuilder
- as a sudoer user
- from
-
build_Tails_ISO_*
andreproducibly_build_Tails_ISO_*
runrake build
- from the corresponding
tails.git
branch - in a Jenkins isobuilder
- as a sudoer user
- from the corresponding
-
test_Tails_ISO_*
run./run_test_suite
- from the corresponding
tails.git
branch - in a Jenkins isotester
- as root via sudo
- from the corresponding
Wrt. who can push to which branch in tails.git
, see the "Access control"
section.
Jenkins → Redmine
Tails images build reproducibility is tested if the corresponding ticket has "Needs Validation" status: https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/decide_if_reproduce
(Currently disabled:) Email notifications are sent on job status change if the ticket has "Needs Validation" status: https://git.tails.boum.org/puppet-tails/tree/files/jenkins/slaves/isobuilders/output_ISO_builds_and_tests_notifications
Ticket triaging → Redmine
Our Redmine email reminder sends email to users who have at least one ticket assigned to them, that satisfies at least one of these criteria:
-
"Stalled work-in-progress": status "In Progress", that were not updated since more than 6 months
-
"Reviews waiting for a long time": not been updated since 45 days or more
Current implementation:
- https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/redmine-remind
- https://git.tails.boum.org/puppet-tails/tree/files/redmine/reminder/email_body
- https://git.tails.boum.org/puppet-tails/tree/manifests/redmine/reminder.pp
With GitLab one can define policies (YAML) and have them applied automatically to issues and merge requests:
- https://about.gitlab.com/handbook/engineering/quality/triage-operations/#triage-automation
- https://gitlab.com/gitlab-org/quality/triage-ops
- https://gitlab.com/gitlab-org/gitlab-triage
Potential future work: tails#17589 (closed).
Translation platform → Git
A hook ensures that Weblate only pushes changes to our main Git repository that meet certain conditions:
https://git.tails.boum.org/puppet-tails/tree/files/gitolite/hooks/tails-weblate-update.hook
GitLab hosting options
See https://salsa.debian.org/tails-team/gitlab-migration/wikis/hosting/comparison.
Resources
- Tor's migration plan
from Trac to GitLab.
- Their instance is https://dip.torproject.org/.
- Converted issues: https://dip.torproject.org/ahf-admin/legacy-20/issues/
- Migration problems: https://pad.riseup.net/p/gitlab-migration-problems
- Timeline: none as of 2020-02-27
- Older meetings:
- agenda and notes
- 2019-10-15 meeting logs
- 2019-10-01 meeting logs
- KDE migration to GitLab:
- https://gitlab.com/gitlab-org/gitlab-foss/issues/57338/designs
- https://gitlab.com/gitlab-org/gitlab/issues/24900
- Wrt. workflow (e.g. does everyone push to their own fork or are some people allowed to push their topic branches to the main repo?), see:
- Project import/export: successfully tested from Salsa to 0xacab (code, MRs, no issues). Notes are authored by the user who performs the import, but attributed in the comment itself to their original author. Commits are authored by the original author. Example: https://0xacab.org/tails/test-import/