GitLab

We need to account for longer version strings than the one provided by
the Pypi JSON for some packages.

In the case at hand, Pypi will return "2.1" for latest user-agents
version, but we should be looking for "2.1.0" in GitHub tags.

The version in the Pypi JSON is "2.1" but there's no such tag in the
repo and our dep update script does not currently account for such
differences.

The deps update script does not correctly check for compatibility with
Python for direct dependencies of Weblate (such check is only done for
dependencies of dependencies).

Therefore, we manually pin openpyxl version to <3.0 as bigger versions
all depend on Python >= 3.6 which is not available in Debian Stretch.

In its previous location, the dependencies update script was being
served in the Puppet file server, but we don't need that, as such script
is meant to be executed in the admin machine to generate Puppet code.

This is what I ran:

  python3 files/weblate/external_scripts/update_puppet_dependecy_files.py \
    --weblate-version 3.5.1 \
    --debian-suite stretch \
    --weblate-repo ~/weblate/

Apache packages are declared in tails::weblate::webserver and memcached
is declared in tails::weblate::config.

Extra dependencies are listed in the Pypi JSON with the keyword "extra",
as such (example taken from https://pypi.org/pypi/celery/json):

    "cryptography ; extra == 'auth'",

This means that someone can run `pip install celery[auth]` and will get
the cryptography package installed.

But this scheme also allows for variations, as the following (from the
same URL as above):

    "couchbase-cffi (<3.0.0) ; (platform_python_implementation == \"PyPy\") and extra == 'couchbase'",

The current implementation of detection of "extra" dependencies does not
account for the second format, so I'm fixing it in this commit.

The regular expression matching package versions was not prepared to see
"rc" versions, and this made it fail to parse versions of dependencies.

The dependencies update script had a hardcoded path making it unable to
work in different environments. This commit adds a command line argument
to the script allowing to pass the weblate repo path.

See tails/tails!19 for the corresponding change in tails/tails.

We don't use this underlay anymore, and its mere presence currently
breaks tails-website-update-underlays: we pointed the remote
to that repo to our GitLab, but we did not migrate that repo
to GitLab.

The pgbouncer pooler is reching its max number of connections and
causing weblate to periodically 503:

  2020-05-28 07:16:07.044 27409 WARNING C-0x5584b39df7d0:
  (nodb)/(nouser)@unix(14953):6432 Pooler Error: no more connections
  allowed (max_client_conn)

This commit doubles the number of connections to the pooler, from 100 to
200.

--

Context: GitLab migration -> Translation Platform integration.

This commit makes the gatekeeper class extend the website params class,
so the Git update hook will have the language information it needs to
make the Weblate Gatekeeper work as expected.

See: https://salsa.debian.org/tails-team/gitlab-migration/-/issues/56

This is an attempt to create a smoother workflow by working around a
known issue:
tails/sysadmin#17713

The glue scripts were running every 5min, and could block weblate for
several minutes rendering the site unusable for translation.

This is not a proper solution to the problem, just an attempt to make
using the website less frustrating.

I see these scripts fail on Jenkins with:

  ValueError: Invalid header value b'SECRET\n'
                                           ^^

I suppose that's because our apikey file has a trailing newline.
So let's strip trailing whitespace after reading that file.

This ports the changes from tails/tails'
dfbdedbfc493d770c5108840acfbe4d616c432e5 here. Quoting that commit:

URLs without the /-/ routing separator will incrementally become deprecated.

Reference: https://gitlab.com/gitlab-org/gitlab/-/issues/214217

Thanks to kibi for the pointer :)

I've just spent too much time trying to understand why this hook
works just fine on one repo, but not on another, while their
configuration is almost identical. I suspect some shell weirdness
is happening, that I'm not skilled enough to understand, so let's
try to convert this code to something simpler and see how it goes.

refs: https://salsa.debian.org/tails-team/gitlab-migration/-/issues/49

… since tails/tails#15460 was resolved.

They now have been run/deployed so they're not needed anymore.

It's unused since 553ac8e3.