Evaluate the practical impact of having containerized Weblate
As per our meeting last week, we should evaluate the impact with these 2 criteria:
- lowering the maintenance cost
- non-root folks can help with the "backend" part
We said we wanted to do it twice:
- after the 1st post-containerization upgrade (probably only applicable to the maintenance cost criterion)
- after using it to do a few upgrades and folks had enough time to contribute to the platform without being root
Notes about 1st post-containerization upgrade (3.11 to 4.9)
For this upgrade the following steps had to be repeated 10 times (one for each intermediate version, as they can't be skipped):
- Update
settings.php
with the new version. - Fix any issues with our custom scripts, for example:
- Account for any changes in Weblate/Django API.
- Make sure hardcoded Python paths are correct.
- Check for important changes in the image (eg. changes in the base image).
- Bump the container image version.
- Run Puppet.
- Watch database migrations being applied in the container's log.
This is considerably less work than in the previous setup in which we had to, on top of the above, also handle Python module dependency resolution as well as versions and repositories pinning via Puppet-managed calls to pip
. The previous setup also didn't implement purging of old/unneeded Python packages, leaving old dependencies behind in the /usr/local/lib/python3*
directories. This could've been implemented but there were other pressing issues (such as the shutdown of the Pypi API which made automation even more difficult, the speed of Weblate releases, etc) and we decided to solve several issues at once with the move to a containerized service.
After some preparation, the actual upgrade process took around 5h, counting creation of backups (which i did a total of 3 times because of mistake and over-caution), doing all of the 10 upgrades and then checking that the system was working correctly and fixing small bits. I used a local copy of the Puppet code repo plus some simple tooling to automate some of the steps, because I wanted to be able to revert or restart the upgrade process without having to deal with the production repo during the process.
The next upgrade will need some adaptation to our custom scripts because there's a change in the base image which affects Python paths. Shouldn't be much work, though, and the dev/test server is serving as an important tool to make such work significantly smoother.
Notes about 2nd post-containerization upgrade (4.9 → 5.3)
This upgrade was much easier than the last one:
- Setting up a dev env was fairly simple, using the tails/puppet-weblate> module. Performing this step was important to help fix our integration scripts tests and the CI pipeline before deploying to production. It can probably be skipped when the CI is green. I did not use the test env to perform a test upgrade for the last upgrade of this batch (5.2 → 5.3) and just did it directly in prod.
- Some new improvements to the containerized setup made upgrading easier:
- There's no need to maintain hardcoded Python paths anymore (puppet-weblate@3c090df9, puppet-weblate@fb050932)
- Moved to a separate settings override file (puppet-weblate@fb050932)
- Upgrading can now be almost as easy as updating the versions in Puppet and deploying:
- Upgrade example: puppet-weblate@f0c817dc
- Upgrade instructions: puppet-weblate@6ae3c589
- I still had to go through more than one upgrade, but that's because there was a major version upgrade in between:
- The first one (4.9 → 5.0) took long because of database upgrade (~1h30m)
- The next ones (5.0 → 5.2 and 5.2 → 5.3) were fast and only needed an update to the container image version.
- I could've done only 2 upgrades instead of 3, but the image for 5.3 was released yesterday, right after I had already upgraded to 5.2. The good news is that the last upgrade took about 5 min to complete and about 2 min downtime.
Overall, the move to a containerized setup had a significant impact in upgrades and maintainability of our Weblate setup.
It's hard to evaluate impacts in external contributions because the Translation Team is not functional anymore as a team.
Next steps
-
Upgrade Weblate to latest -
Describe the experience in this issue