Slow downloads from some mirrors to Lizard

Problem statement

Since at least 2021-09-21 our check-mirrors cronjob reports that downloads from a number of mirrors are much slower than expected.

Output of 4 of these jobs:

[https://tails.interpipe.net/tails/] Slow mirror: 2.41 MB/s
[https://tails.braingap.uk/tails/] Slow mirror: 178 KB/s
[https://mirrors.ocf.berkeley.edu/tails/] Slow mirror: 593 KB/s
[https://tails.osuosl.org/] Slow mirror: 6.27 MB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 5.66 MB/s

[https://tails.interpipe.net/tails/] Slow mirror: 2.35 MB/s        
[https://tails.braingap.uk/tails/] Slow mirror: 476 KB/s        
[https://ftp.nluug.nl/os/Linux/distr/tails/tails/] Slow mirror: 5.89 MB/s
[https://mirror.netcologne.de/tails/tails/] Slow mirror: 551 KB/s
[https://mirrors.dotsrc.org/tails/] Slow mirror: 6.47 MB/s
[https://ftp.acc.umu.se/tails/] Slow mirror: 6.45 MB/s
[https://tails.osuosl.org/] Slow mirror: 948 KB/s
[https://tails.mirror.iphh.net/tails/] Slow mirror: 4.83 MB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 1.33 MB/s

[https://tails.interpipe.net/tails/] Slow mirror: 2.33 MB/s
[https://tails.braingap.uk/tails/] Slow mirror: 374 KB/s
[https://tails.as1101.net/tails/] Slow mirror: 5.86 MB/s
[https://mirrors.ocf.berkeley.edu/tails/] Slow mirror: 647 KB/s
[https://mirror.netcologne.de/tails/tails/] Slow mirror: 1.21 MB/s
[https://ftp.acc.umu.se/tails/] Slow mirror: 5.73 MB/s
[https://tails.osuosl.org/] Slow mirror: 962 KB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 3.35 MB/s

[https://tails.interpipe.net/tails/] Slow mirror: 1.07 MB/s
[https://tails.braingap.uk/tails/] Slow mirror: 406 KB/s
[https://mirrors.ocf.berkeley.edu/tails/] Ignoring, mirror marked as failed
[https://ftp.fau.de/tails/] Slow mirror: 5.70 MB/s
[https://mirror.netcologne.de/tails/tails/] Slow mirror: 509 KB/s
[https://ftp.acc.umu.se/tails/] Slow mirror: 6.87 MB/s
[https://mirror.alpix.eu/tails/tails/] Slow mirror: 1.12 MB/s
[https://tails.osuosl.org/] Slow mirror: 3.38 MB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 2.07 MB/s

This situation:

• Fully breaks pool monitoring.
• So if a mirror is broken we'll keep sending users to it.
• This breaks the "user acquisition" path.

This is how our script currently checks for mirrors speeds:

• Fetch: https://tails.boum.org/mirrors.json
• Go through one (with some exceptions to ignore some errors)
• Report if download_speed < 7 MB/s

Tasks

• Compare download speed from different places
• Fix munin in lizard.
• Enable munin in iguana
• Find and fix problem with slower network in some VMs (eg. misc.lizard)