Migrate the Translation Platform to use a containerized Weblate

• Part of: #17782 (closed)
• MR: puppet-tails!58 (merged)

Migration plan

Preliminary steps:

• Deploy the container-based platform in development environment, test all functionality.
• Schedule downtime period for migration: Oct 28-29 2021 → Message sent to tails-l10n@.

Migration steps:

1. Create a backup of the last working state:
   • Disable Puppet Agent.
   • Disable the website.
   • Create backups of data and database (takes a bit less than 1h)
   • Copy the backups elsewhere.
   • Shutdown the VM.
   • Snapshot the disk volume.
2. Install the new platform:
   • Install a fresh Debian Bullseye VM.
   • Merge puppet-tails!58 (merged)
   • Add the newly installed VM to Puppet.
   • Run Puppet until it converges.
3. Restore Weblate from backups:
   • Disable Puppet Agent.
   • Protect Weblate with a temporary password.
   • Restore database.
   • Restore repositories and fix ownership and permissions.
4. Check that the platform works fine:
   • Accepting a suggestion creates a commit in the Git repo.
   • Check that integration with the website works (tip: create new SSH key and push to Gitolite config).
   • Ensure that building the staging wiki works.
   • Check updating of Translation Memory works.
   • Check that permissions enforcement works.
   • Get help to double check whether everything works fine.
5. Enable the platform:
   • Enable Puppet Agent.
   • Open Weblate to the public.
6. Wrap up:
   • Fix ownership of /var/log/weblate/update.log (+ other logfiles): should be 2001000:weblate.
   • Fix monitoring of translate.lizard.
   • Remove celery monitoring.
   • Check old logs for clues about unexpected permissions.
   • Double check VM creation checklist
   • Update PXE installer with newer images (i.e. Bullseye).
   • Check re. need of puppetizing loginctl enable-linger 2000000.
   • Fix Puppet error:

     Error: Facter: error while resolving custom fact "podman":
     cannot merge "/run/podman/podman.sock":String and "/run/user/0/podman/podman.sock":String

   • Document some commands (tails!660 (merged)):

     # Some of the commands below have to be executed in a directory readable to the user `weblate` (eg. /tmp).
     sudo -u weblate XDG_RUNTIME_DIR=/run/user/2000000 systemctl --user stop podman-weblate
     cd /tmp; sudo -u weblate podman logs -f --tail=1 weblate
     sudo -u weblate podman exec -t -i weblate /bin/bash
     sudo tail -f /var/log/weblate/update.log
     sudo -u weblate /var/lib/weblate/scripts/run_in_container.sh /scripts/cron.sh
     sudo -u weblate /var/lib/weblate/scripts/run_in_container.sh /scripts/weblate_permissions.py --enforce
     sudo -u weblate /var/lib/weblate/scripts/update_tm.sh

   • Document or puppetize generation of Weblate SSH key for weblate-gatekeeper.git.
   • Allow service admins to read Apache logs
   • Document that /var/log/weblate/weblate.log is not used anymore (and ditch it).
   • Check how/if to access podman logs via file instead of "podman logs"
   • Fix permissions of /var/lib/tmserver.
   • Check if some container mounts should be mounted read-only on run_in_container.sh -- puppet-tails!74 (merged)
   • Whitelist modsec rules that triggered after migration.
   • Create an issue to re-evaluate ModSec config: #17874
   • Fix checks for existence of configured remotes in Git repos.
   • Do not send messages from update-staging-website.py when there are no errors.
   • Fix mail queue.
   • Fix changing permissions of executable files in the repositories: chmod 664 → chmod g+w.
   • Merge Weblate design docs into the website: tails!592 (merged)
   • Turn on ModSec in "log-only" mode: puppet-tails!73 (merged)
   • Fix the pipeline (i.e. make successful use of tails::apt when deploying in CI) -- puppet-tails!75 (merged)
   • Send e-mail to interested parties re. reproducibility of the platform.
   • Configure backups for translate.lizard's data disk.
   • Delete old/backup LVM volumes (+ snapshots).