tails issueshttps://gitlab.tails.boum.org/tails/tails/-/issues2022-02-18T10:59:09Zhttps://gitlab.tails.boum.org/tails/tails/-/issues/18855Consider giving translators the "Define author of translation upload" permission2022-02-18T10:59:09ZintrigeriConsider giving translators the "Define author of translation upload" permissionFrom tails/tails#17649:
To be discussed:
* Add “Define author of translation upload” to translators. It means, when you upload a po file, you should be able to give an author for it. This is good for attribution, because sometimes you ...From tails/tails#17649:
To be discussed:
* Add “Define author of translation upload” to translators. It means, when you upload a po file, you should be able to give an author for it. This is good for attribution, because sometimes you add files somebody else has translated. This permission should be for translators, not for reviewers. You don’t need to upload a translation while you review.
Also, every change decided here would have to be implemented in tails/sysadmin#17338.Translation Platform in Productionhttps://gitlab.tails.boum.org/tails/tails/-/issues/17871Allow translators to access the Machine translation suggestions2020-08-21T21:36:59ZemmapeelAllow translators to access the Machine translation suggestionsAt the moment translators have no access to machine translation suggestions while translating:
<img src="/uploads/ce4652d1fecd9bf95beabcd5518d5930/machine-translation-missing.png" width="400">
This makes their work less comfortable, an...At the moment translators have no access to machine translation suggestions while translating:
<img src="/uploads/ce4652d1fecd9bf95beabcd5518d5930/machine-translation-missing.png" width="400">
This makes their work less comfortable, and makes the translation less consistent.
We should review the permissions translators have and allow them to access the 'Machine translation' tab in Weblate, as my user with more privileges has:
<img src="/uploads/e16679b118e89cac83225067057895ed/machine-translation-on.png" width="400">Translation Platform in Productionemmapeelemmapeelhttps://gitlab.tails.boum.org/tails/tails/-/issues/17870merge_canonical_changes.py adds German translations to not-yet-published lang...2021-03-22T12:07:05Zemmapeelmerge_canonical_changes.py adds German translations to not-yet-published languagesSpotted by sajolida, it seems the script that refreshes the still not published translations is adding German translations to the languages we have in Weblate but we don't yet have on the main repository. This problem does not affect
Spa...Spotted by sajolida, it seems the script that refreshes the still not published translations is adding German translations to the languages we have in Weblate but we don't yet have on the main repository. This problem does not affect
Spanish, German, Persian, French, Italian or Portuguese.
Examples are:
[wiki/src/contribute.*.po](https://gitlab.tails.boum.org/tails/tails/-/blob/stable/wiki/src/contribute.ca.po#L459) / https://translate.tails.boum.org/translate/tails/contribute/pl/?checksum=f274a1056febc396#translations
[wiki/src/doc/first_steps/bug_reporting.*.po](https://gitlab.tails.boum.org/tails/tails/-/blame/master/wiki/src/doc/first_steps/bug_reporting.ca.po#L404) / https://translate.tails.boum.org/translate/tails/bug_reporting/tr/?checksum=649206e70a2f2c59#translations - https://gitlab.tails.boum.org/tails/tails/-/blame/master/wiki/src/doc/first_steps/bug_reporting.ca.po#L404
A sample commit is 41bf7a19394 (look for 'Starten von DVD mit einem Mac' )
I am not sure how merge_canonical_changes.py works, but it seems sometimes the msgstr is being copied from German along the msgid.
Related issues
--------------
* #17910Translation Platform in ProductionZen FuZen Fuhttps://gitlab.tails.boum.org/tails/tails/-/issues/17771tmserver breaks on missing dependency2020-06-15T21:51:15Zgroentetmserver breaks on missing dependencyHello translatey people,
Starting tmserver fails with:
pkg_resources.DistributionNotFound: The 'six>=1.11.0' distribution was not found and is required by translate-toolkit
Indeed the debian package python3-six we now have installed i...Hello translatey people,
Starting tmserver fails with:
pkg_resources.DistributionNotFound: The 'six>=1.11.0' distribution was not found and is required by translate-toolkit
Indeed the debian package python3-six we now have installed is 1.10.0, do you want to work your own python magic or is this the point where we want to upgrade to buster, which ships with 1.12.0?Translation Platform in ProductionZen FuZen Fuhttps://gitlab.tails.boum.org/tails/tails/-/issues/17661Weblate: ikiwiki should not stop completly when file is wrong2020-11-30T20:01:16ZhefeeWeblate: ikiwiki should not stop completly when file is wrong_Originally created by @hefee on [#17661 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17661)_
Currently ikiwiki run fails, when there is one failed
file:
weblate@translate:~/repositories/vcs/staging$ iki..._Originally created by @hefee on [#17661 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17661)_
Currently ikiwiki run fails, when there is one failed
file:
weblate@translate:~/repositories/vcs/staging$ ikiwiki --setup /var/lib/weblate/config/ikiwiki.setup --rebuild
perl: warning: Setting locale failed.
perl: warning: Please check that your locale settings:
LANGUAGE = (unset),
LC_ALL = (unset),
LANG = "de_DE.UTF-8"
are supported and installed on your system.
perl: warning: Falling back to the standard locale ("C").
Invalid po file /tmp/ikiwiki-po-filter-in.9r75CG2SFn:
/tmp/ikiwiki-po-filter-in.9r75CG2SFn:400: 'msgid' and 'msgstr' entries do not both end with '\n'
msgfmt: found 1 fatal error
things to improve:
1\. just continue with the other files
2\. print the file source that is triggering that failure, so it
possible to fix that file
### Attachments
* [patch](https://redmine.tails.boum.org/code/attachments/download/2589/patch)
### Related issues
- **Related to** tails/sysadmin#17560Translation Platform in ProductionZen FuZen Fuhttps://gitlab.tails.boum.org/tails/tails/-/issues/17655Review Weblate's translation workflow and its integration with Git repos and ...2023-08-15T20:44:17ZZen FuReview Weblate's translation workflow and its integration with Git repos and websites_Originally created by @zen on [#17655 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17655)_
For the integration of the Translation Platform with the main website,
we originally wanted that:
* Anyone can suggest...._Originally created by @zen on [#17655 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17655)_
For the integration of the Translation Platform with the main website,
we originally wanted that:
* Anyone can suggest.
* Logged in users can vote on suggestions.
* Most voted suggestions would appear in the staging website.
* Reviewers can accept translations.
* Accepted translations appear in the main website automatically.
# Current state: Weblate workflow vs Git repository vs website rendering
The following table attempts to address the following questions:
* Which of the possible states of strings (Needs editing, Waiting for review, Approved) gets committed to the repo?
* Which of these states gets committed but is marked as fuzzy?
* Which of these states get rendered in the main website?
* Which of these states get rendered in the staging website?
The Weblate workflow did not change since we first installed ([3.5.1](https://docs.weblate.org/en/weblate-3.5.1/workflows.html)) it up to the latest version ([4.12.2](https://docs.weblate.org/en/weblate-4.12.2/workflows.html) nowadays).
Using the info from [Weblate Doc](https://docs.weblate.org/en/latest/workflows.html?highlight=fuzzy#translation-states) and comparing strings in Weblate, Tails Git repo and the main website, this is the current state of affairs:
| Weblate state | Committed to Git? | Marked as fuzzy? | Rendered in the main website? | Rendered in staging website? |
| ------------------ | ----------------- | ---------------- | ------------------------- | ----------------- |
| Suggestions | ❌ | (doesn't apply) | ❌ | ✅ |
| Untranslated | ❌ | (doesn't apply) | ❌ | ❌ |
| Needs editing /Fuzzy | ✅ | ✅ | ❌ | ❌ |
| Waiting for review / Translated | ✅ | ❌ | ✅ | ✅ (if not committed to git) |
| Approved | ✅ | ❌ | ✅ | ✅ (if not committed to git already) |
# Analysis of the current state
- There's a difference between what Weblate and what Tails understand as "reviewed strings":
- For Weblate, a string "Waiting for review" [is a valid translation](https://docs.weblate.org/en/weblate-4.9.1/workflows.html#translation-states):
- "It is stored in the file as a valid translation" and, because of that, it gets rendered in our main website (as well as in the staging website, unless there are suggestions for it).
- The difference between "Waiting for review" and "Approved" only lives in the Weblate database.
- For Tails, an unreviewed string needs "[to be accepted by reviewers to go live onto the main Tails website](https://tails.boum.org/contribute/how/translate/with_translation_platform/#index3h2)".
- Right now:
- We're only using [3 "kinds" of roles](https://gitlab.tails.boum.org/tails/puppet-tails/-/blob/aa4ed4f4f87e0faae31feea62eeed2a180661ce4/files/weblate/config/weblate_permissions.yaml) in Weblate related to actually translating (not counting other functionalities as managing glossaries, etc): Guests, Users and (global and per-language) Reviewers.
- `Guests` and `Users` can only suggest, and suggestions determine what gets rendered on the staging website.
- `Reviewers` can accept suggestions to turn them into any of the other states described in the table above.
- As emmapeel [wrote](https://gitlab.tails.boum.org/tails/tails/-/issues/17655#note_149467):
- This workflow probably wasn't in place when Weblate started to be evaluated.
- We currently use a bad mapping from Weblate workflow features to Tails translation workflow.
- The current setup causes several usability problems.
## Assumptions made by our current integration code
**Weblate → Canonical:**
1. **Assumption:** All changes made via Weblate web interface that are committed to the Weblate Git repo will eventually be merged into the Canonical repo (unless they conflict with changes made to Canonical in the meantime).
**Consequence:** Canonical gets all translation content and state changes made in the Weblate interface (not counting conflicts and suggestions): "Needs edit" translations are committed as fuzzy (and thus invalidated, rendering the main language instead), and "Needs review" and "Approved" translations are committed as "valid" translations.
2. **Assumption:** Weblate and our custom code only change `.po` files in the Weblate and Integration repos.
**Consequence:** Pushing changes to Canonical fails if anything other than `.po` files is changed.
3. **Assumption:** When there are conflicts, changes from Canonical have precedence over changes from Weblate.
**Consequence:** Conflicting translations made through Weblate are lost.
**Canonical → Weblate:**
1. **Assumption:** The update of the translations in the Weblate backend is not made automatically by Weblate (i.e. `AUTO_UPDATE` is off, so Weblate-initiated Git pulls and component discovery are never triggered automatically).
**Consequences:**
- We need a custom mechanism to update the Weblate backend with changes from Canonical.
- Such mechanism is responsible for maintaining integrity between the Weblate backend and repository, that is, to ensure that when it finishes all translations content and state are the same in both places.
2. **Assumption:** Source string changes are not automatically reflected in the Canonical repo for languages that are not activated in the main website. (The `.po` files for languages that are activated in the main website are automatically updated by the production Ikiwiki.)
**Consequence:** Our custom code needs to propagate relevant changes to source strings to the languages that are not activated in the main website, but that are activated in Weblate and the staging website.
3. **Assumption:** Changes in any source string or translation in a component cause the whole component to be updated in the Weblate backend.
**Consequence:** If the state of a translation in the Weblate backend differs from its state in the `.po` file (for example if we implement only committing of "Approved" strings), then a change in another translation in the same component may cause data loss in the Weblate backend.
# Analysis of the proposals
In terms of UX, @emmapeel proposes:
- `Guests` and unprivileged `Users` can only suggest, and suggestions only live in Weblate.
- `Translators` (which are "approved" `Users`, added to the `Translators` group) can determine what gets rendered in the staging website.
- `Reviewers` can determine what is accepted and included in the main website.
Implementation-wise, because there's no easy way to change how Weblate behaves, we have to face the problem that "Waiting for review" strings are treated as "valid" translations, thus always get committed to the Weblate repository without the fuzzy tag (see the table above).
From the discussion below in this issue, i understand that @emmapeel's idea is to either:
1. Change the logic of our integration scripts and integration repository(ies) in a way that only "Approved" strings are propagated from Weblate to the Tails canonical repository, or
2. Change Weblate behavior upstream, which would mean convince Weblate devs that only strings in "Approved" state should be committed to the repository (effectively changing the first and second columns in the table above).
In any of the above cases:
- The new `Translator` role would be able to add strings directly to the "Waiting for review" state.
- Such strings would be rendered in the staging website only.
- Only `Reviewers` would be able to turn such strings into "Approved" state.
## Proposal 1: Replace merging from Weblate by importing "Approved" strings
- Replace the "Merge Weblate Changes" step by an "Integrate Approved Strings" step.
- Update the "Update Weblate Components" step to account for the fact that the Weblate repo will never converge with the Canonical repo.
- Change Weblate roles/permissions so that Translators can add strings to "Needs review".
- Change the generation of the Staging website to use strings in "Needs review" state instead of suggestions.
Pros: improved UX, satisfies security policy.
Cons: high cost of implementation, needs changes in many places of our integration code.
## Proposal 2: Foster the implementation of a write-policy in Weblate
- Wait or help Weblate to implement a [write-policy](https://github.com/WeblateOrg/weblate/issues/3745).
- Change Weblate config so only "Approved" strings get committed to the Weblate repository.
- Profit.
Pros: improved UX, satisfies security policy, benefits more Weblate users.
Cons: needs knowledge of Weblate codebase, needs acceptance by Weblate, needs currently unknown amount of resources.
## Proposal 3: Stick with the same workflow, improve communication with Reviewers
Train Reviewers to:
- Do their own translations by means of suggestions.
- Periodically check for new suggestions.
- Do not use the "Needs review" state: every accepted suggestion must be reviewed by the time of acceptance and immediately marked and saved as "Approved".
Pros: cheap solution.
Cons: bad UX.
# Next steps
Preparation and evaluation:
- [x] Create a table comparing weblate states, storage in file in Git repo, fuzziness, and rendering in the main and staging websites.
- [x] Compare the current state with emmapeel's proposal
- [x] List what would need to be changed to implement the proposal
- [x] Ask for review, confirmation and input from folks, maybe iterate to improve the proposal
- [x] Write a proposal that only changes workflow and permissions
- [x] Map assumptions that our code/design relies on
- [x] Maybe: Prototype something that commits only "Approved" strings to Git
- [ ] Decide next stepsTranslation Platform in Productionhttps://gitlab.tails.boum.org/tails/tails/-/issues/17653Update Translation Platform design doc2022-03-11T11:04:49ZZen FuUpdate Translation Platform design doc_Originally created by @zen on [#17653 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17653)_
The [Translation Platform design
doc](https://tails.boum.org/contribute/design/translation_platform/)
documents the intr..._Originally created by @zen on [#17653 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17653)_
The [Translation Platform design
doc](https://tails.boum.org/contribute/design/translation_platform/)
documents the intricacies of the integration of Weblate with the Tails
repository and website. We need to:
* [x] Ensure that [the
documentation](https://tails.boum.org/contribute/design/translation_platform/)
is up to date.
* [x] Add missing context so the documentation is self contained.
* ~~Improve documentation of
[scripts](https://git.tails.boum.org/puppet-tails/tree/files/weblate/scripts)
and refer to the design doc.~~ -- #18879Translation Platform in Productionhttps://gitlab.tails.boum.org/tails/tails/-/issues/17649Adapt Weblate permissions to a better translation workflow2022-02-18T10:59:29ZZen FuAdapt Weblate permissions to a better translation workflow_Originally created by @zen on [#17649 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17649)_
After having used Weblate for some months, some changes to groups and
permissions might be needed to reach a better tran..._Originally created by @zen on [#17649 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17649)_
After having used Weblate for some months, some changes to groups and
permissions might be needed to reach a better translation workflow for
translators and reviewers.
This issue will be used to track such discussions and modifications.
To be discussed:
* Add “Define author of translation upload” to translators. It means,
when you upload a po file, you should be able to give an author for it.
This is good for attribution, because sometimes you add files somebody
else has translated. This permission should be for translators, not for
reviewers. You don’t need to upload a translation while you review.
Also, every change decided here would have to be implemented in tails/sysadmin#17338.
### Subtasks
- [x] tails/sysadmin#17515 - Remove “Overwrite existing translation with upload” (not needed
when reviewing strings). Many translators have asked for the possibility of uploading translations, most notably the Chinese translators. As the rendering of the pages is very slow, it makes sense to give this possiblity to
translators. They should be able to upload a .po file and add as suggestion.
- [x] tails/#17836 - Have different groups for each language (so, for example, Italian reviewers
can only delete Italian suggestions).Translation Platform in Productionemmapeelemmapeelhttps://gitlab.tails.boum.org/tails/tails/-/issues/17619update_weblate_components.py broken due to unclean local Git state2020-06-04T13:38:27Zintrigeriupdate_weblate_components.py broken due to unclean local Git state_Originally created by @intrigeri on [#17619 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17619)_
`cron.sh` has been failing loudly every 5 minutes since at least 24h.
This looks related to PO files having been ..._Originally created by @intrigeri on [#17619 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17619)_
`cron.sh` has been failing loudly every 5 minutes since at least 24h.
This looks related to PO files having been renamed in our Git repo.Translation Platform in Productionhefeehefeehttps://gitlab.tails.boum.org/tails/tails/-/issues/17377Weblate: Generate list of Python dependencies automatically2020-06-26T06:43:56ZZen FuWeblate: Generate list of Python dependencies automatically_Originally created by @zen on [#17377 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17377)_
For maintaining the Weblate installation, we chose to:
\* install Python dependencies from Debian repo as much as possi..._Originally created by @zen on [#17377 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17377)_
For maintaining the Weblate installation, we chose to:
\* install Python dependencies from Debian repo as much as possible;
and
\* pin dependencies’ versions and install from upstream repositories
whenever they are not available in Debian. (The reason for installing
from repositories is that the pip repo distributes binaries which have
no way to be verified.)
Currently, we’re manually maintaining the Puppet code for installing
Weblate dependencies either as [Debian
packages](https://git.tails.boum.org/puppet-tails/tree/manifests/weblate/debian_packages.pp?id=23a39bda33ccbf7fc433686ca3b84a550a5d363f)
or [Python
modules](https://git.tails.boum.org/puppet-tails/tree/manifests/weblate/python_modules.pp?id=23a39bda33ccbf7fc433686ca3b84a550a5d363f),
with the aid of a script. That is hard to maintain and error prone, so
we should ideally generate the list of dependencies automatically.
We need have a script that, given a Weblate version and a list of extra
packages (example: redis is needed for celery which is needed — see
tails/sysadmin#17347), will traverse the whole dependency trees and generate whatever
is needed to have a fixed list of Debian packages and Python modules
installed — and nothing else.
Feature Branch: https://salsa.debian.org/hefee/puppet-tails/:hefee/update_puppet_dependecy_filesTranslation Platform in ProductionZen FuZen Fuhttps://gitlab.tails.boum.org/tails/tails/-/issues/17344Update translation platform user doc after Weblate upgrade to 4.x2022-06-02T13:35:59ZZen FuUpdate translation platform user doc after Weblate upgrade to 4.x_Originally created by @zen on [#17344 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17344)_
The [translations platform’s user
documentation](https://tails.boum.org/contribute/how/translate/with_translation_platfo..._Originally created by @zen on [#17344 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17344)_
The [translations platform’s user
documentation](https://tails.boum.org/contribute/how/translate/with_translation_platform/)
references various features and interfaces of Weblate. As we’ve upgraded
Weblate one major version recently, we have to verify whether something
has to be updated in the user doc.Translation Platform in Productionemmapeelemmapeelhttps://gitlab.tails.boum.org/tails/tails/-/issues/16872Make it obvious to visitors of the staging website that it is not trustworthy2023-07-20T21:16:51ZAnonymousMake it obvious to visitors of the staging website that it is not trustworthy_Originally created by @Anonymous on [#16872 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16872)_
This was mentioned by hefee as a nice to have.
### Related issues
- **Related to** tails/tails#150..._Originally created by @Anonymous on [#16872 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16872)_
This was mentioned by hefee as a nice to have.
### Related issues
- **Related to** tails/tails#15080
- **Related to** tails/sysadmin#15360
- **Related to** tails/tails#15080Translation Platform in Production