Slow downloads from some mirrors to Lizard

Problem statement

Since at least 2021-09-21 our check-mirrors cronjob reports that downloads from a number of mirrors are much slower than expected.

Output of 4 of these jobs:

[https://tails.interpipe.net/tails/] Slow mirror: 2.41 MB/s
[https://tails.braingap.uk/tails/] Slow mirror: 178 KB/s
[https://mirrors.ocf.berkeley.edu/tails/] Slow mirror: 593 KB/s
[https://tails.osuosl.org/] Slow mirror: 6.27 MB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 5.66 MB/s

[https://tails.interpipe.net/tails/] Slow mirror: 2.35 MB/s        
[https://tails.braingap.uk/tails/] Slow mirror: 476 KB/s        
[https://ftp.nluug.nl/os/Linux/distr/tails/tails/] Slow mirror: 5.89 MB/s
[https://mirror.netcologne.de/tails/tails/] Slow mirror: 551 KB/s
[https://mirrors.dotsrc.org/tails/] Slow mirror: 6.47 MB/s
[https://ftp.acc.umu.se/tails/] Slow mirror: 6.45 MB/s
[https://tails.osuosl.org/] Slow mirror: 948 KB/s
[https://tails.mirror.iphh.net/tails/] Slow mirror: 4.83 MB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 1.33 MB/s

[https://tails.interpipe.net/tails/] Slow mirror: 2.33 MB/s
[https://tails.braingap.uk/tails/] Slow mirror: 374 KB/s
[https://tails.as1101.net/tails/] Slow mirror: 5.86 MB/s
[https://mirrors.ocf.berkeley.edu/tails/] Slow mirror: 647 KB/s
[https://mirror.netcologne.de/tails/tails/] Slow mirror: 1.21 MB/s
[https://ftp.acc.umu.se/tails/] Slow mirror: 5.73 MB/s
[https://tails.osuosl.org/] Slow mirror: 962 KB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 3.35 MB/s

[https://tails.interpipe.net/tails/] Slow mirror: 1.07 MB/s
[https://tails.braingap.uk/tails/] Slow mirror: 406 KB/s
[https://mirrors.ocf.berkeley.edu/tails/] Ignoring, mirror marked as failed
[https://ftp.fau.de/tails/] Slow mirror: 5.70 MB/s
[https://mirror.netcologne.de/tails/tails/] Slow mirror: 509 KB/s
[https://ftp.acc.umu.se/tails/] Slow mirror: 6.87 MB/s
[https://mirror.alpix.eu/tails/tails/] Slow mirror: 1.12 MB/s
[https://tails.osuosl.org/] Slow mirror: 3.38 MB/s
[https://tailsde.freedif.org/tails/] Slow mirror: 2.07 MB/s

This situation:

  • Fully breaks pool monitoring.
  • So if a mirror is broken we'll keep sending users to it.
  • This breaks the "user acquisition" path.

This is how our script currently checks for mirrors speeds:

Tasks

  • Compare download speed from different places
  • Fix munin in lizard.
  • Enable munin in iguana
  • Find and fix problem with slower network in some VMs (eg. misc.lizard)
Edited by Zen Fu