translation_platform.mdwn 25.4 KB
Newer Older
1
2
3
[[!meta title="Translation platform"]]
[[!toc levels=2]]

Ulrike Uhlig's avatar
Ulrike Uhlig committed
4
Until 2019, our (website) translation infrastructure relied on
5
6
7
8
translators [[being able to know how to use
Git|contribute/how/translate/with_Git]]. This was a pretty high entry
barrier for new translators, especially those who are not familiar with
Git or the command line.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
9

Ulrike Uhlig's avatar
Reorder    
Ulrike Uhlig committed
10
11
12
13
This is the technical design documentation of our setup.

We also provide a dedicated [[documentation for translators on how to use
Weblate|contribute/how/translate/with_Weblate]] to contribute
Ulrike Uhlig's avatar
Ulrike Uhlig committed
14
15
translations. (This link will work once [[!tails_ticket 11763]] is
fixed.)
Ulrike Uhlig's avatar
Reorder    
Ulrike Uhlig committed
16

17
18
19
20
21
22
23
Terms used in this document
===========================

- Canonical Git repository: the main Tails Git repository that our
  website relies on, in scripts often called "main repository" or "main
  Git"
- Production server: the server that hosts our website
Ulrike Uhlig's avatar
Ulrike Uhlig committed
24
- translate.lizard: the VM that hosts our Weblate web interface, the
Ulrike Uhlig's avatar
Ulrike Uhlig committed
25
  corresponding Git repositories, as well as the staging website.
26

27
[Corresponding tickets on Redmine](https://redmine.tails.boum.org/code/projects/tails/issues?query_id=321)
28
29
30
31

Setup of our translation platform & integration with our infrastructure
=======================================================================

Ulrike Uhlig's avatar
Ulrike Uhlig committed
32
We are using our own [Weblate instance](https://translate.tails.boum.org/).
33

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
34
35
36
37
38
39
40
41
42
43
Weblate uses a clone of the Tails main Git repository, to which
translations get committed, once they have been approved by a user with
reviewer status. Non-approved translations live on Weblate's database
only, until they get reviewed. A staging website allows translators to
preview non-reviewed translations in context.

Approved changes are automatically fed back into our canonical Git
repository. This presents a major challenge, indeed we need to ensure
that:

Ulrike Uhlig's avatar
Ulrike Uhlig committed
44
45
46
47
48
49
50
51
52
- no merge conflicts occur:

  - such conflicts often occur on PO file headers which prevents Weblate
    from automatically merging changes
  - many contributors work on the same code base using different tools
    (PO files can be edited by hand, using translation software such as
    POedit, or they are generated by ikiwiki itself, which results in
    different formatting)

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
53
- only PO files are committed.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
54
- the committed PO files comply with shared formatting standards.
Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
55
56
57
58
- no compromised code is introduced.

In order to integrate Weblate and the work done by translators into our
process, we have set up this scheme:
59

Ulrike Uhlig's avatar
Ulrike Uhlig committed
60
[[!img "lib/design/git_repository_details.svg" link="no"]]
61

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
62
63
Website and Weblate
-------------------
Ulrike Uhlig's avatar
Ulrike Uhlig committed
64

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
65
66
67
68
69
70
71
Our website uses ikiwiki and its PO plugin. It uses markdown files for
the English original language and carries a PO file for each translated
language. Thereby we distinguish languages that are activated on our
website from languages that have translations but are not yet activated
on the website because they do not [[cover enough of a portion of
our core pages|contribute/how/translate/team/new/]] to be considered
usable.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
72

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
73
74
75
76
77
78
79
We have defined [[a list of tier-1
languages|contribute/how/translate#tier-1-languages]], that we consider
to be of importance to our user base. No more languages shall be
activated in Weblate as our main Git repository carries reviewed, and
thus approved translations of all languages enabled on the Weblate
platform, while only part of them are active on the website.

80
Each PO file corresponds to a single component in Weblate, in order to
Ulrike Uhlig's avatar
Ulrike Uhlig committed
81
appear in the Weblate interface. For example, the component:
82
83
84
85
86

    wiki/src/support.*.po

relates to the files support.mdwn, support.es.po, support.de.po, support.pot,
etc.
87

88
89
Repositories
------------
90

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
91
92
93
94
95
96
The repository used by Weblate is cloned and updated from the Tails main
repository, and its master branch. Changes generated on Weblate's copy
of the Tails main Git repository, located on the VM which hosts the
Weblate platform, are fed back to the Tails main repository, into the
master branch, automatically. This happens through a number of scripts,
checks, and cronjobs that we'll describe below.
97

Ulrike Uhlig's avatar
Ulrike Uhlig committed
98
There are several languages enabled, some of them with few or no
Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
99
100
translations. As everything is fed back to the Tails canonical
repository, all files are available when cloning this repository:
101

102
103
    git clone https://git-tails.immerda.ch/tails

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
104
105
If needed, for exceptional means, Weblate's Git repository can be cloned
or added as a remote:
106
107
108

    git clone https://translate.tails.boum.org/git/tails/index/

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
109
At the server the repository is located in
110
111
112

    ~weblate/repositories/vcs/tails/index

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
113
114
115
116
Weblate can commit to its local repository at any time, whenever
translations get approved. Changes done in the canonical repository by
Tails contributors via Git and changes done in Weblate thus need to be
merged - in a safe place. This happens in an integration repository:
117
118
119

    ~weblate/repositories/integration

Ulrike Uhlig's avatar
Rewrite    
Ulrike Uhlig committed
120
121
On the VM (translate.lizard), a third repository is used for the staging
website:
122
123
124

    ~weblate/repositories/vcs/staging

Ulrike Uhlig's avatar
Ulrike Uhlig committed
125
126
Automatic merging and pushing
-----------------------------
127

Ulrike Uhlig's avatar
Ulrike Uhlig committed
128
The integration work between the different repositories is done by a
Sandro Knauß's avatar
Sandro Knauß committed
129
130
131
132
script which is executed on the VM hosting Weblate as [a cronjob every
5 minutes](https://git-tails.immerda.ch/puppet-tails/tree/manifests/weblate.pp). The script
[`cron.sh`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/cron.sh)
 has the following steps which we will explain:
133
134
135

  1. Canonical → Integration:
     Update the integration repository with changes made on the
Sandro Knauß's avatar
Sandro Knauß committed
136
137
138
139
140
141
142
143
144
145
146
147
     canonical repository (called "main" in the script).
  2. Trigger Weblate to commit pending approved translations ([`manage.py
     commit_pending`](https://docs.weblate.org/en/weblate-2.20/admin/management.html#commit-pending)).
     As Weblate tries to reduce commits (called lazy commits), we need to ask
     Weblate explicitly to commit every component, that wasn't touched for 24h and
     has outstanding changes.
  3. Weblate → Integration:
     Integrate comitted changes from Weblate into the Integration repository
  4. Integration → Canonical:
     Push up-to-date integration repository to canonical repository. After this step canonical has everything merged from Weblate.
  5. Canonical → Weblate:
     Pull from canonical and update the Weblate components.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
148
     repository
Sandro Knauß's avatar
Sandro Knauß committed
149
150
  6. Run [`manage.py update_index`](https://docs.weblate.org/en/weblate-2.20/admin/management.html#update-index).
     Updates index for fulltext search, it is recommended to run it every 5 mins.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
151
152
153

Whenever a contributor modifies a markdown (`*.mdwn`) file, and pushes
to master, the corresponding POT and PO files are updated, that is: the
Ulrike Uhlig's avatar
Ulrike Uhlig committed
154
translatable English strings within those files are updated. This
Ulrike Uhlig's avatar
Ulrike Uhlig committed
155
156
157
158
159
160
161
162
163
164
165
update happens on the production server itself, when [[building the
wiki|contribute/build/website]] for languages that are enabled on the
production website.

We need to ensure on the translation platform server, that PO files for
additional languages (that are enabled on Weblate but not on the
production website) are equally updated, committed locally, pushed to
the canonical Git repository. On top of this we need to update Weblate's
database accordingly, so that translations can be added for new or
modified English strings in those files, in all languages.

166
167
### Step 1: Canonical → Integration

Ulrike Uhlig's avatar
Ulrike Uhlig committed
168
169
**Update the integration repository with changes made on the canonical
repository**
Ulrike Uhlig's avatar
Ulrike Uhlig committed
170
171
172

The script fetches from the canonical (remote) repository and tries to
merge changes into the (local) integration repository. The merge
Ulrike Uhlig's avatar
Ulrike Uhlig committed
173
strategy used for this step is defined in [`update_weblate_git.py`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/update_weblate_git.py): (XXX: Shouldn't this script be called update_integration_git.py according to this documentation?)
Ulrike Uhlig's avatar
Ulrike Uhlig committed
174

Sandro Knauß's avatar
Sandro Knauß committed
175
When this script is executed, it merges changes in PO files based on
Ulrike Uhlig's avatar
Ulrike Uhlig committed
176
177
178
179
180
181
single translation units (`msgids`). Merge conflicts occur when the same
translation unit has been changed in the canonical and the integration
repository (in the latter case, this would mean that the change has been
done via Weblate). In such a case, we always prefer the canonical
version. This makes sure that Tails developers can fix issues in
translations and have priority over Weblate.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
182
183
184
185

Due to this procedure we never end up with broken PO files, however, we
may loose a translation done on Weblate.

Ulrike Uhlig's avatar
Ulrike Uhlig committed
186
Until here, only PO files of languages that are activated on our
Ulrike Uhlig's avatar
Ulrike Uhlig committed
187
188
189
190
191
192
193
194
195
196
197
production website will be merged, as the production website, i.e. the
canonical Git repository does not regenerate those of non activated
languages.

Hence, after the activated language PO files are merged, the script
checks if PO files of additional, non-production activated languages
need updating. We do this by generating POT files out of a PO file that
we've previously defined as a default language.  If the actual POT file,
generated on the production server differs from the POT file we've just
created, then every additional language PO file needs to be updated.

Ulrike Uhlig's avatar
Ulrike Uhlig committed
198
199
200
On top of this, if the PO file of the default language (that is, its
markdown file) have been renamed, moved or deleted, than the PO files of
additional languages need to follow this step.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
201

Sandro Knauß's avatar
Sandro Knauß committed
202
Our script actually applies all changes from the default language to the additional languages and afterwards we can look if we actually introduce any change in the PO files.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
203

Sandro Knauß's avatar
Sandro Knauß committed
204
The described mechanisms always touching files and changes mtime etc, that's why git would always create a new commit. Often those commits don't change anything on the files. To get rid of those empty not needed commits the script detects, when a fast-forward is possible (the master  branch is updated to HEAD of canonical or integration branch). This is possible, if only Weblate introduce new commits or only the canonical.
205

Sandro Knauß's avatar
Sandro Knauß committed
206
### Step 3: Weblate → Integration
207

Ulrike Uhlig's avatar
Ulrike Uhlig committed
208
209
**Merging changes from Weblate's Git repository into the integration
repository**
210

Ulrike Uhlig's avatar
Ulrike Uhlig committed
211
212
213
214
Weblate's Git repository is not bare. Hence we need to pull changes
committed to Weblate's Git repository and merge them into the
integration repository. This is done by the script
[`merge_weblate_changes.py`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/merge_weblate_changes.py).
215

Ulrike Uhlig's avatar
Ulrike Uhlig committed
216
217
218
219
220
221
Changes already present in the integration repository are preferred over
the changes from the remote, Weblate repository. This is to allow fixes
done to PO files manually, via the canonical Git repository.

Again, PO file merges are done on translation units (`msgids`).

Ulrike Uhlig's avatar
Ulrike Uhlig committed
222
Furthermore, we make sure via the script that Weblate has only modified
Ulrike Uhlig's avatar
Ulrike Uhlig committed
223
224
225
PO files; indeed we automatically reset everything else to the version
that exists in canonical.

Sandro Knauß's avatar
Sandro Knauß committed
226
### Step 4: Integration → Canonical
227

Ulrike Uhlig's avatar
Ulrike Uhlig committed
228
229
**Pushing from the integration repository to our canonical repository,
aka "production"**
230

Sandro Knauß's avatar
Sandro Knauß committed
231
232
233
After updating the Integration repository, we push the changes back to
Canonical aka puppet-git.lizard. After this the Canonical repository has
everything integrated from Weblate.
234

Ulrike Uhlig's avatar
Ulrike Uhlig committed
235
236
237
238
239
240
241
242
On the side of the canonical Git repository, gitolite has a special
hook,
[`tails-weblate-update.hook`](https://git-tails.immerda.ch/puppet-tails/tree/files/gitolite/hooks/tails-weblate-update.hook),
to make sure that Weblate is only allowed to push changes on PO files.
This hook also checks and verifies the committer of each commit, to make
sure only translations made on the Weblate platform are automatically
pushed, and no other changes than those on PO files accepted. Otherwise
the push is rejected, for security reasons.
243

Sandro Knauß's avatar
Sandro Knauß committed
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
### Step 5: Canonical → Weblate

**Integrating the changes made in the Canonical Git repository into
Weblate repository**

After having merged changes from the canonical Git repository into the
integration Git repository, and integrated changes from Weblate there,
we can assume that every PO file now is up-to-date (in the Integration
and Canonical repository). Hence we can try to pull from the Canonical
repository using a fast-forward only (`git pull --ff-only`). Canonical and the
Weblate repositories can get new commits everytime, also while the cronjob is
running. It can happen, that a new commit on one side (Cannonical or Weblate)
makes it impossible to perform a fast-forward. If we can't fast-forward the git
repository, the cronjob is run 5min later anyways again, Than step 1,3 and 4 of
the cronjob fixes the reason why the fast-forward was not possible this time.

If the fast-forward was successful, we need to update Weblate's components
to reflect the modifications that happened on the side of Git, such as
string and file updates, removals, renames, or additions. This is
handled by another script:
[`update_weblate_components.py`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/update_weblate_components.py).

We are not the only process touching the Weblate repository, as Weblate itself
creating commits and updating the master branch That's why the script is using
an own Git remote named `cron` to keep track of which commits need to look at
for Weblate component changes. This remote name is set in the
[weblate.pp](https://git-tails.immerda.ch/puppet-tails/tree/manifests/weblate.pp)
and used in the cronjob `update_weblate_components.py
--remoteBranch=cron/master [...]`.

274
275
<a id="staging-website"></a>

276
277
278
Staging website
---------------

Ulrike Uhlig's avatar
Ulrike Uhlig committed
279
280
281
282
283
In order to allow translators to see their non committed suggestions as
well as languages which are not activated on https://tails.boum.org we
have put in place a [staging website](https://staging.tails.boum.org/) .
It is a clone of our production website and is regularly refreshed.

284
On top of what our production website has, it includes:
285

286
287
288
289
 - all languages available on Weblate, even those that are not enabled
   on our production website yet;

 - all translation suggestions made on Weblate.
290

291
This allows:
292

293
294
 - translators to check how the result of their work will look like
   on our website;
295

296
297
 - reviewers to check how translation suggestions look like on the
   website, before validating them.
298

299
 - check the sanity-check-website report:
300

Ulrike Uhlig's avatar
Ulrike Uhlig committed
301
   [https://staging.tails.boum.org/last-sanity-errors.txt](https://staging.tails.boum.org/last-sanity-errors.txt)
302

Ulrike Uhlig's avatar
Ulrike Uhlig committed
303
### What is done behind the scene to generate a new version of the staging website?
304

305
The cronjob
Ulrike Uhlig's avatar
Ulrike Uhlig committed
306
[`update-staging-website.sh`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/update-staging-website.sh)
307
is run.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
308

309
310
311
312
This cronjob calls a script that extracts suggestions from Weblate's
database and applies them to a local clone of Weblate's Git repository,
after having updated the clone with newer data from Weblate's VCS.
[`save-suggestions.py`](https://git-tails.immerda.ch/puppet-tails/tree/files/weblate/scripts/save-suggestions.py)
Ulrike Uhlig's avatar
Ulrike Uhlig committed
313

Ulrike Uhlig's avatar
Ulrike Uhlig committed
314
After that we run `ikiwiki --refresh` using an dedicated `ikiwiki.setup`
315
file for the staging website.
316

317
318
319
None of the changes on this repository clone are fed back anywhere and they
should not.

320
321
<a id="access-control"></a>

322
Access control on the Weblate platform
Ulrike Uhlig's avatar
Ulrike Uhlig committed
323
======================================
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
344

### Requirements

- Every translation change must be reviewed by another person before
  it's validated (and thus committed by Weblate and pushed to our
  production website).

  - This requirement must be enforced via technical means, for
    translators that are not particularly trusted (e.g. new user
    accounts). For example, it must be impossible for an attacker to
    pretend to be that second person and validate their own changes,
    simply by creating a second user account.

  - It's acceptable that this requirement is enforced only via social
    rules, and not via technical means, for a set of
    trusted translators.

- We need to be able to bootstrap a new language and give its
  translators sufficient access rights so that they can do their job,
  even without anyone at Tails personally knowing any of them.

intrigeri's avatar
intrigeri committed
345
346
347
- Suggested translations are used to build the [[staging
  website|translation_platform#staging-website]].

Ulrike Uhlig's avatar
Ulrike Uhlig committed
348
349
Currently implemented proposal
------------------------------
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369

- In Weblate lingo, we use the [dedicated
  reviewers](https://docs.weblate.org/en/latest/workflows.html#dedicated-reviewers)
  workflow: it's the only one that protects us against an adversary
  who's ready to create multiple user accounts.

- When not logged in, a visitor is in the `Guests` group and is
  only allowed to suggest translations.

- Every logged in user is in the `Users` group. Members of this group
  are allowed to suggest translations but not to accept suggestions
  nor to directly save new translations of their own.

- A reviewer, i.e. a member of the `@Review` group in Weblate, is
  allowed to accept suggestions.

  Limitations:

  - Technically, reviewers are also allowed to directly save new
    translations of their own, edit existing translations, and
370
    accept their own suggestions; we ask them in our
371
372
373
374
375
376
377
378
379
380
    documentation to use this privilege sparingly, only to fix
    important and obvious problems.

	Even if we forbid reviewers to accept their own suggestions,
    nothing would prevent them from creating another account, making
    the suggestion from there, and then accepting it with their
    reviewer account.

  - Reviewer status is global to our Weblate instance, and not
    per-language, so technically, a reviewer can very well accept
381
    suggestions for a language they don't speak. We will them in
382
383
384
385
386
387
388
389
390
391
392
393
394
395
    our documentation to _not_ do that, except to fix important and
    obvious problems that don't require knowledge of that language
    (for example, broken syntax for ikiwiki directives).

	If this ever causes actual problems, this could be fixed with
    [group-based access
    control](https://docs.weblate.org/en/weblate-2.20/admin/access.html#groupacl)

- How one gets reviewer status:

  - We will port to Weblate semantics the pre-existing trust
    relationship we already have towards translation teams that have
    been using Git so far: they all become reviewers.

396
	To this aim, we have asked them to create an account on Weblate
397
398
399
	and tell us what their user name is.

  - One can request reviewer status to Weblate administrators, who
intrigeri's avatar
intrigeri committed
400
401
402
403
404
405
406
407
    will:
    1. Accept this request if, and only if, a sufficient amount of
       work was done by the requesting translator (this can be checked on
       the user's page, e.g.
       [intrigeri's](https://translate.tails.boum.org/user/intrigeri/).
       In other words, we use proof-of-work to increase the cost of attacks.
    2. Let <tails-l10n@boum.org> and all the other Weblate reviewers
       know about this status change.
408
409
410
411
412

- Bootstrapping a new language

  As a result of this access control setup, translators for a new
  language can only make suggestions until they have done a sufficient
413
414
415
  amount of work and two of them are granted reviewer status. In the
  meantime, they can see the output of their work on the [[staging
  website|blueprint/translation_platform#staging-website]].
416
417

  Pending questions:
418

419
420
421
  - Is the resulting UX good enough? Would it help if we allowed them
    to vote up suggestions, even if this does not result in the
    suggestion to be accepted as a validated translation?
intrigeri's avatar
intrigeri committed
422
    (At the moment, suggestion voting is disabled.)
423

424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
450
451
452
Machine translation
===================

This is important because it saves time for the translators, especially
in cumbersome documents, and helps us to be consistent not only with our
translations but, for example, with the Debian locales if we feed them
to the tmserver.

It is a very subtle way of increasing the quality of our translations.

It should give suggestions when you are translating, under the translation
window, tab 'Machine translation'.

We use tmserver for 'Machine translation'. You find the documentation
[here](https://docs.weblate.org/en/weblate-2.20/admin/machine.html#tmserver).

In order to update the suggestion we run
[`update_tm.sh`](templates/weblate/update_tm.sh.erb) via cronjob every month.

The tmserver can be queried like this [(see
`tmserver.service`)](manifests/weblate.pp):

<pre>
http://localhost:8080/tmserver/en/de/unit/contribute
</pre>

It should give suggestions when you are translating, under the translation
window, tab 'Machine translation'.

453
454
455
456
457
458
459
Weblate administration
======================

- [[Enabling a new language|contribute/l10n_tricks#weblate-administration]]
  Make sure to enable languages only if they are part of our tier-1
  list or discuss the matter on the l10n-mailing list.

460

461
462
463
464
- Sysadmin: This documentation currently still lives in
  translate-server.git and should be moved somewhere else.
  [[!tails_ticket 15086]]

465
466
467
468
469
470
471
472
473
474
475
476
477
478
479
480
481
482
483
484
485
486
487
488
Manually fix issues
-------------------

We have our weblate codebase at

    /usr/local/share/weblate

If commands have to be run, they should be run as user weblate (sudo -u weblate $COMMAND).

However, this VM is supposed to run smoothly without human
intervention, so be careful with what you do and please document
modifications you make so that they can be fed back to puppet.git or other
places if necessary.

Reload translations from VCS and cleanup orphaned checks and suggestions
------------------------------------------------------------------------

The following commands mayby run manually if something went wrong:

- Reload all translations from proper folder on disk (eg. in case you
  did some updates in VCS repository) to components of weblate
    `sudo -u weblate ./manage.py loadpo --all`


Ulrike Uhlig's avatar
Ulrike Uhlig committed
489
490
491
492
493
Weblate installation and maintenance
====================================

A hybrid approach
-----------------
494

Ulrike Uhlig's avatar
Ulrike Uhlig committed
495
496
497
498
499
500
501
502
503
504
505
506
507
508
509
510
511
512
The Tails infrastructure uses Puppet to make it easier to enforce and
replicate system configuration, and usually relies on Debian packages to
ensure stability of the system. But the effort to maintain a stable
system somehow conflicts with installing and maintaining Weblate, a
Python web application, which requires using up-to-date versions of
Weblate itself and of its dependencies.

Having that in mind, and taking into account that we already started
using Docker to replicate the translation server environment to
experiment with upgrading and running an up-to-date version of Weblate,
it can be a good trade-off to use Puppet to provide an environment to
run Docker, and to use a Docker container to actually run an up-to-date
Weblate installation.

From the present state of the Docker image, which currently uses
(slightly modified/updated) Puppet code to configure the environment and
then sets up Weblate, the following steps could be taken to achieve a
new service configuration as described above:
513
514

* Move the database to a separate Docker service.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
515
516
517
518
519
* Remove all Puppet code from the Docker image: inherit from the
  simplest possible Docker image and setup a Weblate Docker image with
  all needed dependencies.
* Modify the Puppet code to account for setting up an environment that
  has Docker installed and that runs the Weblate Docker image.
520
521
* Set up persistence for the Weblate git repository and configuration.
* Set up persistence and backups for the database service.
Ulrike Uhlig's avatar
Ulrike Uhlig committed
522
523
524
525
526
527
528
529
* Update the Puppet code to run tmserver (if/when it's needed -- latest
  Weblate accounts for basic suggestions using its own database).

After that, we should have a clear separation between stable
infrastructure maintenance using Debian+Puppet in one side and
up-to-date Weblate application deployment using Docker in the other
side. The Docker image would have to be constantly maintained to account
for Weblate upgrades, but that should be easier cleaner than deploying
530
531
532
533
534
535
536
537
Weblate directly on the server.

Long-term maintenance plan
--------------------------

This is work in progress. A plan for the future maintenance of our
Weblate instance will be worked on in November 2019 and laid out here
before the end of the year.
Ulrike Uhlig's avatar
Reorder    
Ulrike Uhlig committed
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590

Choosing a translation web platform
===================================

These are the requirements that we have defined for our translation web platform.

MUST
----

* provide a usable easy web interface
* be usable from Tor Browser
* automatic pull from main Git repo
* provide a common glossary for each language, easy to use and improve
* allow translators to view, in the correct order, all strings that
  come from the entire page being translated, both in English and in
  the target language
* make it easy to view the translations in context i.e. when translating
  an entire page, all strings to be translated should only come from
  this page. translators should be able to view the page in context.
* provide user roles (admin, reviewer, translator)

SHOULD
------

* be "privacy sensitive", i.e. be operated by a non-profit
* allow translators to push translations through Git (so that core
  developers only have to fetch reviewed translations from there)
* provide support for Git standard development branches (devel, stable,
  and testing) but we could also agree upon translation only master
  through this interface
* provide checks for inconsistent translations
* provide feature to write/read comments between translators

MAY
---

* allow translating topic branches without confusing translators,
  causing duplicate/premature work, fragmentation or merge conflicts
  -- e.g. present only new or updated strings in topic branches;
  see <https://mailman.boum.org/pipermail/tails-l10n/2015-March/002102.html>
  for details
* provide a feature to easily see what is new, what needs updating, what are translation priorities
* provide possibility to set up new languages easily
* send email notifications
  - to reviewers whenever new strings have been translated or updated
  - to translators whenever a resource is updated
* respect authorship (different committers?)
* provide statistics about the percentage of translated and fuzzy strings
* Letting translators report about problems in original strings, e.g.
  with a "Report a problem in the original English text" link, that
  e.g. results in an email being sent to -l10n@ or -support-private@.
  If we don't have that, then [[contribute/how/translate]] MUST
  document how to report issues in the original English text.