Make our infrastructure more redundant

Originally created by @intrigeri on #16956 (Redmine)

Problems we already know about:

• it would be nice to be able to reboot lizard without people suffering from the Tails website being down;
• DDoS resistance: we’re in a very bad place when lizard or SeaCCP are DDoS’ed.

So at least we would like to make our website hosted in more than one location. Hosting our website elsewhere too might require dropping the ikiwiki cgi, which itself requires:

• move blueprints out to a separate wiki
• switch to an external search engine
• replace our usage of the ikiwiki ping plugin with something else → #17364 (closed)

Depending on the outcome of the risk analysis, there’s probably things we should make more redundant, if this fits into our 2019-2020 sysadmin budget.

Related issues

• Related to tails#12406
• Blocked by #15097 (closed) (sysadmins already discussed internally the security implications and will document accordingly)

Note: For S11, this fits in:

• B.2 - Keep our infrastructure up-to-date and secure: Improving the redundancy of some parts of our infra benefits the availability of those parts and consequently the security and UX of the operating system.

To-do

• Setup a copy of the website in a different network
• Wait for Build our production website in GitLab CI (#17364 - closed)
• Make the DNS point to that copy when www.lizard is down
• Make sure mirror stats are reported accordingly
• Document the setup
• Defer redundancy of download.tails.net to a separate issue → #18124
• Defer replacement of ikiwiki ping to #17364 (closed)
• Figure out about DDoS resistance
• Figure out about other bits of redundancy