translation_platform.mdwn 8.09 KB
Newer Older
1
[[!meta title="Translation platform"]]
127.0.0.1's avatar
127.0.0.1 committed
2
[[!toc levels=2]]
127.0.0.1's avatar
127.0.0.1 committed
3
4
5
6

Our (website) translation infrastructure has a pretty high barrier for
new translators, especially those who are not familiar with Git and/or
the command line.
7
8
9
10
Furthermore, the current process makes it hard to add new languages, as
often a team cannot be built easily over a long period of time and a web
interface could nevertheless help keep translations until a new person
arrives.
127.0.0.1's avatar
127.0.0.1 committed
11

12
13
14
15
Corresponding ticket: [[!tails_ticket 10034]]

Choosing a translation web platform
===================================
127.0.0.1's avatar
127.0.0.1 committed
16
17

MUST
18
----
sajolida's avatar
sajolida committed
19

127.0.0.1's avatar
127.0.0.1 committed
20
* provide a usable easy web interface
127.0.0.1's avatar
127.0.0.1 committed
21
* be usable from Tor Browser
sajolida's avatar
sajolida committed
22
* automatic pull from main Git repo
127.0.0.1's avatar
127.0.0.1 committed
23
* provide a common glossary for each language, easy to use and improve
24
25
26
* allow translators to view, in the correct order, all strings that
  come from the entire page being translated, both in English and in
  the target language
27
* make it easy to view the translations in context i.e. when translating an entire page, all strings to be translated should only come from this page. translators should be able to view the page in context.
intrigeri's avatar
intrigeri committed
28
* provide user roles (admin, reviewer, translator)
127.0.0.1's avatar
127.0.0.1 committed
29
30

SHOULD
31
------
sajolida's avatar
sajolida committed
32

127.0.0.1's avatar
127.0.0.1 committed
33
* be "privacy sensitive", i.e. be operated by a non-profit
sajolida's avatar
sajolida committed
34
35
36
37
38
* allow translators to push translations through Git (so that core
  developers only have to fetch reviewed translations from there)
* provide support for Git standard development branches (devel, stable,
  and testing) but we could also agree upon translation only master
  through this interface
127.0.0.1's avatar
127.0.0.1 committed
39
* provide checks for inconsistent translations
127.0.0.1's avatar
127.0.0.1 committed
40
* provide feature to write/read comments between translators
127.0.0.1's avatar
127.0.0.1 committed
41
42

MAY
43
---
sajolida's avatar
sajolida committed
44

45
46
47
48
49
* allow translating topic branches without confusing translators,
  causing duplicate/premature work, fragmentation or merge conflicts
  -- e.g. present only new or updated strings in topic branches;
  see <https://mailman.boum.org/pipermail/tails-l10n/2015-March/002102.html>
  for details
127.0.0.1's avatar
127.0.0.1 committed
50
* provide a feature to easily see what is new, what needs updating, what are translation priorities
127.0.0.1's avatar
127.0.0.1 committed
51
* provide possibility to set up new languages easily
127.0.0.1's avatar
127.0.0.1 committed
52
* send email notifications
53
54
  - to reviewers whenever new strings have been translated or updated
  - to translators whenever a resource is updated
127.0.0.1's avatar
127.0.0.1 committed
55
* respect authorship (different committers?)
127.0.0.1's avatar
127.0.0.1 committed
56
* provide statistics about the percentage of translated and fuzzy strings
57
58
59
60
61
* Letting translators report about problems in original strings, e.g.
  with a "Report a problem in the original English text" link, that
  e.g. results in an email being sent to -l10n@ or -support-private@.
  If we don't have that, then [[contribute/how/translate]] MUST
  document how to report issues in the original English text.
sajolida's avatar
sajolida committed
62

63
64
65
66
67
68
69
70
71
72
73
74
75
76
Setup of our translation platform & integration with our infrastructure
=======================================================================

Documentation
-------------

We have two documentations:
- this one, which presents the current status of our work and all public
  information
- translate-server.git which contains more technical information,
  cronjobs, configs, inner workings of the setup.

Translation web interface
-------------------------
sajolida's avatar
sajolida committed
77

78
79
80
We are testing a [Weblate instance](https://translate.tails.boum.org/)
to see if it fits our requirements. Read [[!tails_ticket 11759]] for
more information.
127.0.0.1's avatar
127.0.0.1 committed
81

82
83
There are several languages enabled and users can suggest translations
in several languages:
127.0.0.1's avatar
127.0.0.1 committed
84

85
86
<a href="http://translate.tails.boum.org/engage/tails/?utm_source=widget">
<img src="http://translate.tails.boum.org/widgets/tails/-/multi-red.svg" alt="Translation status" />
127.0.0.1's avatar
127.0.0.1 committed
87
88
</a>

127.0.0.1's avatar
127.0.0.1 committed
89
90
What we plan to do is:

91
92
93
[Schematics of the different Git repos, ikiwiki instances, and their relationships.](https://labs.riseup.net/code/attachments/download/1922/weblate.svg)

[Schematics of the infrastructure such as Git hooks.](https://labs.riseup.net/code/attachments/download/1928/weblate_hooks.svg)
127.0.0.1's avatar
127.0.0.1 committed
94

127.0.0.1's avatar
127.0.0.1 committed
95
96
97
Repository
----------

98
Currently the repository used by Weblate is built upon the Tails master
99
100
repo, but the changes generated in translate.lizard are not fed back
onto Tails master automatically.
127.0.0.1's avatar
127.0.0.1 committed
101

102
103
There are several languages enabled, some of them with few or not
translations.
127.0.0.1's avatar
127.0.0.1 committed
104
105
106
107
108

You can check out weblate-generated Tails repo with:

    git clone https://translate.tails.boum.org/git/tails/index/

109
110
111
112
113
114
115
116
117
118
This Tails repository has two main differences with other repos:

- The ikiwiki.setup file has been changed to build more languages.
  → Ideally, we would create ikiwiki-translationplatform.setup
- There is a .gitlab.yml file to trigger tests when pushed to gitlab CI
  enabled repositories. → We can either add this file to tails/master or
  document this only for people who use gitlab.
- There are new language files from currently non-active languages on
  the main website.

127.0.0.1's avatar
127.0.0.1 committed
119
Reviewing translation platform output
127.0.0.1's avatar
127.0.0.1 committed
120
121
-------------------------------------

122
123
For languages like fr, fa, de, and it - that are part of tails master repo -
you can get the files to review and submit to tails-l10n:
127.0.0.1's avatar
127.0.0.1 committed
124
125
126
127
128
129

    git remote add translations https://translate.tails.boum.org/git/tails/index/
    git checkout tails/master
    find . -name '*.fa.po' -exec git checkout translations/master --  {} \;
    git reset *

127.0.0.1's avatar
127.0.0.1 committed
130
And you will have all the changes to farsi (*.fa.po) to review. The same goes for the other languages.
127.0.0.1's avatar
127.0.0.1 committed
131

132
Staging website
127.0.0.1's avatar
127.0.0.1 committed
133
134
---------------

135
From this (or a modified clone of this) repository, a version of the
136
137
website with more languages will be built [[!tails_ticket 15077]] so
that users can see how the file they are translating looks.
127.0.0.1's avatar
127.0.0.1 committed
138

127.0.0.1's avatar
updates    
127.0.0.1 committed
139
140
Updating the repo
-----------------
127.0.0.1's avatar
127.0.0.1 committed
141

142
143
POT, then PO files for enabled languages are built and updated from the
*.mdwn files when [[building the wiki|contribute/build/]].
127.0.0.1's avatar
127.0.0.1 committed
144

145
146
147
This process is currently done outside of the Weblate instance, and
merged onto the Weblate repo by hand. As translate.lizard has so many
languages, if there are many changes this process can take a while.
127.0.0.1's avatar
127.0.0.1 committed
148
149

The changes generated [[while building the wiki|contribute/build/]]
150
151
152
153
154
can be fed back to Weblate by cherry picking.

On the long term, we will automate this process by
- pulling automatically from tails/master and applying a merge strategy
  for po [[!tails_ticket ]]
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173


Weblate installation and maintenance - a hybrid approach
--------------------------------------------------------

The Tails infrastructure uses Puppet to make it easier to enforce and replicate system configuration, and usually relies on Debian packages to ensure stability of the system. But the effort to maintain a stable system somehow conflicts with installing and maintaining Weblate, a Python web application, which requires using up-to-date versions of Weblate itself and of its dependencies.

Having that in mind, and taking into account that we already started using Docker to replicate the translation server environment to experiment with upgrading and running an up-to-date version of Weblate, it can be a good trade-off to use Puppet to provide an environment to run Docker, and to use a Docker container to actually run an up-to-date Weblate installation.

From the present state of the Docker image, which currently uses (slightly modified/updated) Puppet code to configure the environment and then sets up Weblate, the following steps could be taken to achieve a new service configuration as described above:

* Move the database to a separate Docker service.
* Remove all Puppet code from the Docker image: inherit from the simplest possible Docker image and setup a Weblate Docker image with all needed dependencies.
* Modify the Puppet code to account for setting up an environment that has Docker installed and that runs the Weblate Docker image.
* Set up persistence for the Weblate git repository and configuration.
* Set up persistence and backups for the database service.
* Update the Puppet code to run tmserver (if/when it's needed -- latest Weblate accounts for basic suggestions using its own database).

After that, we should have a clear separation between stable infrastructure maintenance using Debian+Puppet in one side and up-to-date Weblate application deployment using Docker in the other side. The Docker image would have to be constantly maintained to account for Weblate upgrades, but that should be easier cleaner than deploying Weblate directly in the server.