sysadmin issueshttps://gitlab.tails.boum.org/tails/sysadmin/-/issues2023-08-22T14:22:28Zhttps://gitlab.tails.boum.org/tails/sysadmin/-/issues/17830Missing latest.* symlinks for the build_Tails_ISO_stable Jenkins job2023-08-22T14:22:28ZintrigeriMissing latest.* symlinks for the build_Tails_ISO_stable Jenkins jobThe https://nightly.tails.boum.org/build_Tails_ISO_stable/lastSuccessful/archive/latest.packages symlink does not exist, while it should.
I have to workaround this problem during the release process. So far it's been simple, I'll update...The https://nightly.tails.boum.org/build_Tails_ISO_stable/lastSuccessful/archive/latest.packages symlink does not exist, while it should.
I have to workaround this problem during the release process. So far it's been simple, I'll update this issue if it gets worse.https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17826Some not-yet-published languages have lingering strings (updates broken?)2022-02-18T10:39:18ZemmapeelSome not-yet-published languages have lingering strings (updates broken?)A new reviewer for Russian found a this bug:
The ['wiki/src/index.*.po' weblate component](https://translate.tails.boum.org/projects/tails/index/) has 44 strings for Russian, Polish, Indonesian, Chinese, Chinese TW, Turkish but only 37 ...A new reviewer for Russian found a this bug:
The ['wiki/src/index.*.po' weblate component](https://translate.tails.boum.org/projects/tails/index/) has 44 strings for Russian, Polish, Indonesian, Chinese, Chinese TW, Turkish but only 37 strings for Spanish, French, Arabic, German, Portuguese.
When passing from the string https://translate.tails.boum.org/translate/tails/index/ru/?&offset=37 to the next, users receive the 'Internal error message'. In the maintainers mailing list we receive an email saying:
```Internal Server Error: /translate/tails/index/ru/
DoesNotExist at /translate/tails/index/ru/
Unit matching query does not exist.
```
I am not sure why some of the strings are still there. Maybe weblate could not delete them from the database because they had suggestions pending? I cannot look at the strings (I get the aforementioned error) but the other strings of the component have suggestions, so I think it is possible that the contributors did suggestions on all strings of the component.https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17814Monitor packages that can't be upgraded for some reason2023-03-31T17:44:55ZZen FuMonitor packages that can't be upgraded for some reasonPrior to a recent Puppet monitoring code refactor, we had an APT "upgradable" packages check that ensured we noticed if there were packages that couldn't be upgraded for some reason, for example because of outdated APT pinnings.
We need...Prior to a recent Puppet monitoring code refactor, we had an APT "upgradable" packages check that ensured we noticed if there were packages that couldn't be upgraded for some reason, for example because of outdated APT pinnings.
We need to reimplement this check in the new monitoring code.https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17809Prevent translations that fail the automated tests from being merged into tai...2022-12-15T14:23:05ZemmapeelPrevent translations that fail the automated tests from being merged into tails/masterSince a while there are [several automated tests triggered by merges from the translation platform](https://gitlab.tails.boum.org/tails/tails/-/pipelines?page=1&scope=all&username=role-weblate-gatekeeper).
It is very nice to receive an ...Since a while there are [several automated tests triggered by merges from the translation platform](https://gitlab.tails.boum.org/tails/tails/-/pipelines?page=1&scope=all&username=role-weblate-gatekeeper).
It is very nice to receive an alert when these checks fail. But you know what would be better? To not merge those [commits that fail the tests](https://gitlab.tails.boum.org/tails/tails/-/pipelines?page=1&scope=all&username=role-weblate-gatekeeper&status=failed)!https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17807Make weblate reviewers aware of accepted translations breaking the production...2022-12-15T14:23:04ZemmapeelMake weblate reviewers aware of accepted translations breaking the production websiteIn the weblate maintainers mailing list we receive an alert when [the pipelines triggered by role-weblate-gatekeeper](https://gitlab.tails.boum.org/tails/tails/-/pipelines?page=1&scope=all&username=role-weblate-gatekeeper) fail. Unless g...In the weblate maintainers mailing list we receive an alert when [the pipelines triggered by role-weblate-gatekeeper](https://gitlab.tails.boum.org/tails/tails/-/pipelines?page=1&scope=all&username=role-weblate-gatekeeper) fail. Unless gitlabCI gives a false positive, this means that a faulty translation made it to the Tails website.
The tests that are related to translation are: check-po-meta-date , check-po-msgfmt, check-translatable-live-website-urls , lint-po (please correct if wrong).
This are **not suggestions**, this are strings that have been accepted by a reviewer and are already on the website.
This information does not belong on the tails-weblate mailing list: it should reach translators, and especially **reviewers**, as sometimes translators may not be able to fix the errors by themselves because they lack the technical expertise to find the error, and at least one reviewer has approved the string that is triggering the error.
As the commits arrive to gitlabCI in batches, it is hard to know which especific commit from the batch has triggered the fail. But we can get the failing string on the tests.
The reviewers are divided in language groups, and there are some 'superreviewers' that can review string in many languages. We could have an alias with all the reviewers emails extracted from the weblate database, or different alias depending on the language. But it will be much more work to decide which language the error is in to send to those reviewers especially. I propose we add all reviewers to one mailing list and let them know about the failures.https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17801Weblate Permission: Automatic Group Assignment not covered2022-02-18T10:39:32ZhefeeWeblate Permission: Automatic Group Assignment not coveredThe Automatic Group Assignment are not covered/checked by weblate-permissions. This is not that important, because next run of weblate-permissions will sort things out. But in terms of having complete control over permissions this should...The Automatic Group Assignment are not covered/checked by weblate-permissions. This is not that important, because next run of weblate-permissions will sort things out. But in terms of having complete control over permissions this should be checked too.
```
from weblate.auth.models import AutoGroup
g = Group.objects.filter(name="Users").first()
for a in g.autogroup_set.all():
print(a.match)
```
output: "^.*$"https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17767Allow translators to use Automatic translation2022-02-18T10:40:22ZemmapeelAllow translators to use Automatic translationAutomatic translation in weblate translates strings that have been already translated in other components.
This is very useful specially on common titles or XML tags. I have been translating automatically for a while, for example when t...Automatic translation in weblate translates strings that have been already translated in other components.
This is very useful specially on common titles or XML tags. I have been translating automatically for a while, for example when there is a new release all the titles for the post get translated in most of our languages (except 'Tails x.x is out', because the number for the release changes, so it does not get automatically translated, although the suggestion appears on the Machine translation tab for translators to copy and edit).
I think it would be interesting for translators (not reviewers, but plain translators) to have the posibility of doing such automatic translation, as this will prevent wasting their time on repetitive translations and prevent inconsistencies.https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17723Make our onion services single-hop2023-01-19T20:01:36ZintrigeriMake our onion services single-hopWe're running a number of onion services on our infrastructure. As of today, the reason to run these as onion services is never to hide their location, so we could get better performance (and lighter load on the Tor network) if we set th...We're running a number of onion services on our infrastructure. As of today, the reason to run these as onion services is never to hide their location, so we could get better performance (and lighter load on the Tor network) if we set them up as single-hop onion services.
Originally reported on tails/sysadmin#16409:
> Also, have you considered running the Tails repos as single onion services ? Skipping the onion routing on the server side would make more bandwidth available. I feel like the repos were super slow from over here, but maybe it’s just my connection? https://blog.torproject.org/whats-new-tor-0298
Once single-hop is no longer an experimental feature, we should enable it for http-hidden and whisperbackhttps://gitlab.tails.boum.org/tails/sysadmin/-/issues/17364The build of our production website should be self-healing2024-03-12T14:03:39ZintrigeriThe build of our production website should be self-healing_Originally created by @intrigeri on [#17364 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17364)_
In a variety of situations, an ikiwiki refresh triggered by a Git push
fails, leaving it in an unclean state, and ..._Originally created by @intrigeri on [#17364 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17364)_
In a variety of situations, an ikiwiki refresh triggered by a Git push
fails, leaving it in an unclean state, and then the only way to recover
is to ssh into the machine and manually start a full rebuild. This is
painful because:
- When this happens during a release process, the release can be left
half-published, until someone fixes this. That’s not fun for the RM.
- It puts timing/availability/expectations pressure on sysadmins.
- I suspect our technical writers have grown wary of pushing some
kinds of changes that typically trigger this sort of problems. Not
being able to do one’s job with a reasonable amount of confidence in
oneself and in our infra is surely not fun.
Ideally, somehow our infra would notice this situation and run a full
rebuild itself.
### Related issues
- **Related to** tails/tails#17361https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17274Inform Weblate reviewers about suggestions that don't pass our checks (check_...2022-02-18T10:47:53ZhefeeInform Weblate reviewers about suggestions that don't pass our checks (check_PO, sanity-check-website)_Originally created by @hefee on [#17274 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17274)_
At the moment only tails-translators get informed when the sanity check
of the the staging website breaks.
But these..._Originally created by @hefee on [#17274 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17274)_
At the moment only tails-translators get informed when the sanity check
of the the staging website breaks.
But these failures needs normally interaction by the langue teams,
that’s why tails-l10n should also be informed about those failures.
Additionally tails-l10n leans about those checks.
Blocked by tails/sysadmin#17793https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17216Make the test suite clean up after itself even in most tricky failure modes2022-08-04T05:53:59ZintrigeriMake the test suite clean up after itself even in most tricky failure modes_Originally created by @intrigeri on [#17216 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17216)_
Goals:
- avoid leaving zombie processes behind (e.g. Xvfb, unclutter, tor
processes started by Chutney, avc..._Originally created by @intrigeri on [#17216 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17216)_
Goals:
- avoid leaving zombie processes behind (e.g. Xvfb, unclutter, tor
processes started by Chutney, avconv, local services run by
cucumber, etc.)
- drop the whole “reboot before running the test suite” dance in our
CI (tails/sysadmin#10601, tails/sysadmin#11295)
- have CI nodes that can run both builds & tests, which would provide
great performance improvements to our feedback loop: we often have
build jobs in the queue while isotesters are idling, or test jobs in
the queue while builders are idling
How to do so? Wrap the test suite with something that will track
resources and clean them up, such as:
- `systemd-run`, which could a not-super-expensive incremental step
forward: https://gitlab.tails.boum.org/tails/sysadmin/-/issues/11295#note_61201
- Docker, which would allow us to have multiple Jenkins executors on
one single big node (instead of a number of isotesterN VMs with one
executor each), resulting in better usage of our hardware
resources:
- <https://www.projectatomic.io/blog/2014/10/libvirtd_in_containers/>
- <https://github.com/fuzzyhandle/libvirtd-in-docker>
- <https://github.com/kubevirt/libvirt>
Scope:
- Initially: our CI.
- Ideally, the solution would also work for developers who run the
test suite locally.
- Ideally, the solution would not depend on Jenkins much, if at all,
so we benefit from it even if we migrate to another kind of CI some
day.https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17071Jenkins does not readily expose the full CI pipeline progress & outcome2020-05-15T12:52:19ZintrigeriJenkins does not readily expose the full CI pipeline progress & outcome_Originally created by @intrigeri on [#17071 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17071)_
As a reviewer of a given branch, I’m supposed to check the results of 3
different Jenkins jobs. There’s no single ..._Originally created by @intrigeri on [#17071 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17071)_
As a reviewer of a given branch, I’m supposed to check the results of 3
different Jenkins jobs. There’s no single place that shows me this
information: I need to separately check 3 different jobs.
As a developer of a branch, I’m also supposed to check whether the full
CI pipeline passed, before I submit my MR. For this I need to track
progress of the full pipeline until it has finished running. Currently
this means having 3 extra tabs open in my web browser.
I believe that Jenkins pipelines solve this UX problem, but this would
imply to rework how we generate jobs.
GitLab CI is not affected by this problem.
### Related issues
- **Related to** tails/tails#16959
- **Related to** tails/sysadmin#17070https://gitlab.tails.boum.org/tails/sysadmin/-/issues/17070Finding the Jenkins jobs corresponding to a given branch is bothersome2020-05-15T12:53:22ZintrigeriFinding the Jenkins jobs corresponding to a given branch is bothersome_Originally created by @intrigeri on [#17070 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17070)_
As a developer or reviewer, I need to go to
<https://jenkins.tails.boum.org/> and look for the 3 jobs (build, RB,
..._Originally created by @intrigeri on [#17070 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/17070)_
As a developer or reviewer, I need to go to
<https://jenkins.tails.boum.org/> and look for the 3 jobs (build, RB,
test) whose name looks like the name of the branch I’m respectively
working on or reviewing.
In systems that better integrate Git with CI, instead I have direct
links from a MR to the corresponding CI jobs (and even CI results
directly in the MR, but that’s out of scope for this ticket), which is
much more convenient.
This problem is not trivial to fix with our current Redmine + Jenkins
setup. Presumably it would require a custom Redmine plugin or bot that
knows how we map branch names to Jenkins job names.
Once our issues & MRs are on GitLab, the best option might be to have a
bot that adds these links to the MR (ideally, in the description; worst
case, as a comment).
If we switched to GitLab CI, this problem would be solved for free.
### Related issues
- **Related to** tails/tails#16959
- **Related to** tails/sysadmin#9719
- **Related to** tails/sysadmin#17071https://gitlab.tails.boum.org/tails/sysadmin/-/issues/16958Fix the design of our Puppet codebase & document design guidelines2024-03-21T14:18:28ZintrigeriFix the design of our Puppet codebase & document design guidelinesGanttStart: 2022-06-01
_Originally created by @intrigeri on [#16958 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16958)_
References:
* https://puppet.com/docs/pe/2018.1/the_roles_and_profiles_method.html
* http...GanttStart: 2022-06-01
_Originally created by @intrigeri on [#16958 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16958)_
References:
* https://puppet.com/docs/pe/2018.1/the_roles_and_profiles_method.html
* https://puppet.com/docs/pe/latest/roles_and_profiles_example.html
* https://puppet.com/docs/pe/2016.4/r_n_p_full_example.html
* https://medium.com/@sebolabs/puppet-roles-profiles-opinion-essay-2d081d9e62bf
### Subtasks
- [x] tails/sysadmin#16934+
- [x] tails/sysadmin#16927+
- [x] tails/sysadmin#16953+
- [x] tails/sysadmin#17942+
- [x] tails/sysadmin#17931+
- [x] #18014+
- [x] #18015+
- [x] Move Weblate to a separate module -- puppet-tails!112+
- [x] Consider refactoring VPN/Tinc to a separate submodule + #11253+
- [x] Document design guidelines
- [x] #17975+
- [x] #17848+
- ~~[ ] #9616+~~
- [x] #17938+
- [x] #17941+
- [x] puppet-tails!121+
- [x] #18028+
- [ ] Jenkins code (`tails::jenkins` and `tails::tester`)
### Related issues
- **Related to** tails/sysadmin#6921Refactor our Puppet codebasegroentegroentehttps://gitlab.tails.boum.org/tails/sysadmin/-/issues/17752Sysadmin "What to do in case of fire" checklist2023-11-29T14:50:57ZintrigeriSysadmin "What to do in case of fire" checklistGanttStart: 2022-08-01
_Originally created by @intrigeri on [#16957 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16957)_
Based on the risk analysis, make a plan how to deal with certain threat
scenarios coming t...GanttStart: 2022-08-01
_Originally created by @intrigeri on [#16957 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16957)_
Based on the risk analysis, make a plan how to deal with certain threat
scenarios coming true. have a concise bullet-point-style checklist of
what needs to be done in case of security breaches, hardware failure,
human downtime, and those sort of emergencies.
The plan is to do this during one of our 2019-2020 sysadmin team
sprints.
### Related issuesgroentegroentehttps://gitlab.tails.boum.org/tails/sysadmin/-/issues/16956Make our infrastructure more redundant2024-03-19T19:13:21ZintrigeriMake our infrastructure more redundant_Originally created by @intrigeri on [#16956 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16956)_
Problems we already know about:
- it would be nice to be able to reboot lizard without people
suffering fro..._Originally created by @intrigeri on [#16956 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16956)_
Problems we already know about:
- it would be nice to be able to reboot lizard without people
suffering from the Tails website being down;
- DDoS resistance: we’re in a very bad place when lizard or SeaCCP are
DDoS’ed.
So at least we would like to make our website hosted in more than one
location. Hosting our website elsewhere too *might* require dropping the ikiwiki cgi, which itself requires:
- [x] move blueprints out to a separate wiki
- [x] switch to an external search engine
- [ ] replace our usage of the ikiwiki `ping` plugin with something else
Depending on the outcome of the risk analysis, there’s probably things
we should make more redundant, if this fits into our 2019-2020 sysadmin
budget.
### Related issues
- **Related to** tails/tails#12406
- [ ] **Blocked by** tails/sysadmin#15097Zen FuZen Fuhttps://gitlab.tails.boum.org/tails/sysadmin/-/issues/16126no alerts when icinga2 is down2022-12-15T14:22:31Zgroenteno alerts when icinga2 is down_Originally created by @groente on [#16126 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16126)_
icinga2 on monitor was down for several days without us noticing. the
web frontend showed no indication of the backe..._Originally created by @groente on [#16126 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/16126)_
icinga2 on monitor was down for several days without us noticing. the
web frontend showed no indication of the backend being down and there
seem to be no other checks outside of icinga to keep an eye whether our
monitoring is still actually functioning.
let’s set an hourly cron for a simple script called that attempts to
connect to monitor on port 5665 and mails tails-sysadmins on failure.
i’d propose running this script on ecours, what do you think?https://gitlab.tails.boum.org/tails/sysadmin/-/issues/15097Risk analysis on our infrastructure2024-02-02T18:39:08ZgroenteRisk analysis on our infrastructureAudit the risks the project is facing and prioritize mitigations.
Loosely based on OCTAVE/Allegro, this would involve:
- https://gitlab.tails.boum.org/tails/sysadmin/-/issues/18057 establishing risk measurement criteria
- https://...Audit the risks the project is facing and prioritize mitigations.
Loosely based on OCTAVE/Allegro, this would involve:
- https://gitlab.tails.boum.org/tails/sysadmin/-/issues/18057 establishing risk measurement criteria
- https://gitlab.tails.boum.org/tails/sysadmin/-/issues/18056 identifying assets and their criteria
(confidentiality/availability/integrity)
- establishing threat trees
- calculate risks as the product of probability and impact of threat
scenario’s
- identify possible mitigations and their cost
- prioritize mitigations as a function of risk-reduction and cost
### Related issues
- [ ] **Blocked by** tails/sysadmin#15096
- **Blocks** tails/tails#9802
- **Blocks** tails/sysadmin#16956groentegroentehttps://gitlab.tails.boum.org/tails/sysadmin/-/issues/12113Make sure tails.net is available over IPv62024-02-29T11:45:53ZusulMake sure tails.net is available over IPv6_Originally created by @usul on [#12113 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/12113)_
The internet is moving from v4 to v6. making the website and some
mirrors available over v6 would probably help some pe..._Originally created by @usul on [#12113 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/12113)_
The internet is moving from v4 to v6. making the website and some
mirrors available over v6 would probably help some people bypass ipv4
only censor tools.
For the record my mirror is ipv6 accessible.
### Related issues
- **Related to** tails/sysadmin#16767
- [x] **Blocked by** tails/sysadmin#14588https://gitlab.tails.boum.org/tails/sysadmin/-/issues/11880Centralize our servers' logs2023-11-29T14:21:22ZintrigeriCentralize our servers' logs_Originally created by @intrigeri on [#11880 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/11880)_
Right now we have volatile Journal + some persistent log files managed
by rsyslog and individual applications. Thi..._Originally created by @intrigeri on [#11880 (Redmine)](https://public-redmine-archive.tails.boum.org/code/issues/11880)_
Right now we have volatile Journal + some persistent log files managed
by rsyslog and individual applications. This sometimes makes it painful
to debug problems since one has to cross-match info from various
sources. It would be nicer if all our logs landed in a single place.
An initial idea to start brainstorming about it would be:
- on each of our systems, send all logs to journald
- configure these journalds to have volatile storage only (that’s the
default and what we currently do) and to forward them to a single
journald instance running in a central place
- in the central logging location, either have journald store logs in
a persistent manner, or forward them to a fancy system like Graylog
(that seems much easier to setup than an ELK stack)