[[_TOC_]] # Why ## Main reason When we designed our current mirror pool, we dismissed the server-side options mainly because of reasons that are all now moot: - We did not host our website ourselves → we now do. - We had no place of our own to host such a redirector → we now have this. - We lacked sysadmin resources to implement something server-side → we could now decide to prioritize this if we wanted. - We were worried about our web server becoming a SPOF → our current client-side mirror picking code relies on our web server and has the same problem. Then we decided to pick the mirror on the client-side. It felt pretty cool at the time but experience shows that as a consequence of this solution, we have a whole set of other problems to solve: - https://gitlab.tails.boum.org/tails/blueprints/-/wikis/HTTP_mirror_pool#improve-ux-and-lower-maintenance-cost-2021 - We have to maintain our mirror pool manually, which is tedious, repetitive, and stressful - We have to write, maintain, and improve, custom code to: - monitor our mirrors (check-mirrors + integration into our infra) - pick a mirror on the client side, using programming languages very few of us are comfortable with (JavaScript + integration into our Perl Upgrader) - Even seemingly simple changes require careful planning and coordination between developers, technical writers, mirror pool maintainers, and release managers. Quite often the update must be released in several stages. This sometimes leads technical writers to instead implement JS/CSS hacks to bypass the design, instead of integrating their change into the design, which suggests the design is not great. - Only 1 person has the big picture in mind. All of these need more work in order to make our current implementation sustainable. So I think it's time to consider cutting down our costs, stopping investing into this design, and ditching the whole thing in favor of a server-side redirector. ## Additional benefits - As we see all downloads, we can do stats about downloads and upgrades ⇒ UX has more useful data to do their job - We can ditch the DNS fallback pool eventually, so: - One less piece of infra (Git hook & code that updates the dl.a.b.o DNS zone) - No more hard-coded IP addresses that may change without upstream notifying us - We can simplify our website: no need to differentiate clients with/without JavaScript anymore, one less piece of JS to integrate # Candidates The most popular option these days seems to be [Mirrorbits](https://github.com/etix/mirrorbits) ([in Debian Bookworm](https://tracker.debian.org/pkg/mirrorbits)). # Implementation - [x] Notify Mirrorbits author that we're doing it - [x] Setup a test installation - [x] Install and configure Redis - [x] Install Mirrorbits (from Bookworm) - [x] Manually download GeoLite2 databases - [x] Add [mirrors](https://gitlab.tails.boum.org/tails/mirror-pool/-/blob/master/mirrors.json) to Mirrorbits configuration and document - [x] Test downloading from different locations - [x] Determine where Mirrorbits will be run in production (needs access to a local copy of the mirrored folder) - → Let's do it in `rsync.lizard` (rationale: already has the data, it's ok to be upgraded to bookworm/testing) - [x] Decide mirrors to use as "Fallback mirrors" - Let's use the ones from our current DNS pool (rationale: they are considered fast and reliable, they are already used as fallback for some use cases). - [x] Decide how to feed the Mirrorbits database (Manually? From `mirrors.json`? From elsewhere?) - Let's feed it from `mirrors.json` initially. - [x] Decide how to feed the Mirrorbits repository (maybe not needed if already available, eg. in `rsync.lizard`) - → Not needed, as we'll run it in `rsync.lizard` - [x] Decide how to handle lack of Rsync/FTP for some mirrors - [x] Check need for setting RPC auth - [x] Add `/trace` file config - [x] Add automatic update of GeoLite2 databases - [x] Add Rsync URLs to `mirrors.json` → mirror-pool!20 - [x] Automatically update the Mirrorbits database with data from `mirrors.json` - [x] Communicate with mirrors - [x] Subscribe Zen-Fu to `tails-mirrors@` (temporarily) - [x] Draft a message explaining that to continue in the pool they'd need to serve Tails via Rsync. - [x] Send to a subset of the mirrors that don't currently serve Tails via Rsync (those with `weight > 0` + tails.as1101.net, tails.darklab.sh, and mirror.alpha-labs.net) - [x] Follow-up, if needed # Deployment ## Stage 0: Infrastructure - [x] Deploy Mirrorbits in `rsync.lizard` → puppet-tails!99 - [x] Decide on what hostname to use for download URLs → `download.tails.net` - [x] Setup reverse proxy for `download.tails.net` to `rsync.lizard` (with TLS and `http` → `https` redirection) - [x] Make `rsync_url` mandatory in `mirrors.json` (and remove non-compliant mirrors) ## Stage 1: Use the mirror redirector for downloads from our website (simple, low-risk, reverting is cheap) In a MR targeting `master` (tails/tails!945), switch from http://dl.amnesia.boum.org to https://download.tails.net in: - [x] Update website download URLs - [x] Testing doc: `wiki/src/contribute/how/testing.mdwn` - [x] Release process doc: `wiki/src/contribute/release_process` - [x] Image URL includes: `wiki/src/inc/stable_amd64_{img,iso}_url.html` - [x] Stop using the JS code that picks a random mirror - Drop obsolete `use-mirror-pool` CSS class and the corresponding CSS & JS code - Keep `submodules/mirror-pool-dispatcher` around for now: it's still used by the Upgrader - [x] IDF file: `wiki/src/install/v2/Tails/amd64/stable/latest.json` - [x] Test that the download + verify JS code supports redirections - [x] Make sure @sajolida reviews this before merging - [x] CSS: `wiki/src/local.css` (add the new URL) - [x] Call for testing template: `config/release_management/templates/call_for_testing.mdwn` - [x] Update mirrors & design doc: include !898 in this MR - [x] Scripts that generate website contents: `bin/idf-content`, `bin/prepare-included-website-for-release` Once satisfied that it works well: - [x] Drop the ikiwiki underlay for `mirror-pool-dispatcher` ## Stage 2: Make the Upgrader use the mirror redirector (not entirely simple, reverting needs a release) MR: tails/tails!983 This is a tricky topic, as the Upgrader we've shipped in already released Tails 5.x must allow upgrading to any future 5.y. Thankfully, we're in luck: - The `transformURL` JS function only uses the *length* of `fallback_download_url_prefix` (hardcoded to `http://dl.amnesia.boum.org/tails` in already released 5.x), not the actual contents of the string. - `https://download.tails.net/tails` is the same length as `http://dl.amnesia.boum.org/tails` So, once we generate UDFs that use the new URL prefix, the old Upgrader will keep working exactly the same way as it currently does: it'll replace that new URL prefix with a random one from `mirrors.json`. And it'll keep falling back to the DNS pool on failure. To make newer Upgrader versions actually use the mirror redirector: - [x] Ensure the Upgrader follows redirections - We don't override `LWP::UserAgent`'s default settings wrt. redirections, which are: max redirect = 7. - This works: `PERL5LIB=config/chroot_local-includes/usr/src/iuk/lib DISABLE_PROXY=1 ./config/chroot_local-includes/usr/src/iuk/bin/tails-iuk-get-target-file --uri https://download.tails.net/tails/project/trace --fallback-uri=https://download.tails.net/tails/project/trace --hash-type=sha256 --hash-value=77f44a44a342a12e6894f144bdd92075549bb33b7788ae7557c2bd344c766b27 --output-file=/tmp/trace --size=11` - [x] Generate UDFs that point to the mirror redirector - Code: `config/chroot_local-includes/usr/src/iuk/lib/Tails/IUK/UpgradeDescriptionFile/Generate.pm` - [x] Update example UDFs in `wiki/src/contribute/design/upgrades.mdwn` - [x] Drop the "replace URL with a random one from the mirror pool JSON" and "fallback to DNS pool" mechanisms: instead, use the mirror redirector - Drop `config/chroot_local-includes/usr/src/perl5lib/lib/Tails/MirrorPool.pm`, that'll become unused - [x] Update mirrors design docs accordingly (`wiki/src/contribute/design/`) - [x] Drop `submodules/mirror-pool-dispatcher` - [x] Remove obsolete dependencies (Node.js?) ## Wrap up - [x] Consider using `LocalJSPath` and the bundled `fetchfiles.sh` to serve JS/CSS/Fonts for Mirrorbits UI from our server. → Tracked separately at: sysadmin#17971 - [x] Document and plan periodic procedures for: - [x] Updating the GeoLite2 databases - [x] Updating the mirror pool - [x] Unsubscribe Zen-Fu from `tails-mirrors` ## What we won't update and why - Pre-existing upgrade YAML files: `wiki/src/upgrade` (most URLs are broken, and we can start using the new one for next releases) - One very old report: `wiki/src/news/report_2013_11.mdwn` (historic data) - News entries: `wiki/src/news/` (most URLs are broken, and we can start using the new one for next - Sandbox: `wiki/src/sandbox.*.po` (maybe add the new URL just for reference?) # Follow-ups These should be tracked elsewhere as they should not block us from closing this issue: ## Optimize Upgrader download Status: done on 653cbdee8a. If it's any more complicated than this, let's create an issue to track it. Context: https://gitlab.tails.boum.org/tails/tails/-/issues/18263#note_193021 Currently `Tails::IUK::TargetFile::Download` downloads IUKs using: SocksPort 127.0.0.1:9062 IsolateDestAddr IsolateDestPort We should probably use another `SocksPort`, that has `IsolateDestAddr IsolateDestPort`, disabled, so there's at least a chance that the circuit used to connect to Mirrorbits is reused for the actual download, thus benefiting from the fact Mirrorbits uses GeoIP to select the mirror. ## Deprecate the fallback DNS pool tails/tails#19333 ## Rethink how we monitor our mirror pool tails/tails#19334