sysadmins.mdwn 12.5 KB
Newer Older
1
2
3
4
5
6
7
8
9
[[!meta title="System administrators"]]

[[!toc levels=2]]

<a id="goals"></a>

# Goals

The Tails system administrators set up and maintain the infrastructure
Tails developers's avatar
Tails developers committed
10
11
that supports the development and operations of Tails. We aim at
making the life of Tails contributors easier, and to improve the quality of
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
the Tails releases.

<a id="principles"></a>

# Principles

## Infrastructure as code

We want to treat system administration like a (free) software
development project:

* We want to enable people to participate without needing an account
  on the Tails servers.
* We want to review the changes that are applied to our systems.
* We want to be able to easily reproduce our systems via
  automatic deployment.
* We want to share knowledge with other people.

This is why we try to publish as much as possible of our systems
configuration, and to manage our whole infrastructure with
configuration management tools. That is, without needing to log
into hosts.

## Free Software

We use Free Software, as defined by the [Debian Free Software
Guidelines](https://www.debian.org/social_contract#guidelines).  
The firmware our systems might need are the only exception to
this rule.

## Relationships with upstream

The [[principles used by the broader Tails
project|contribute/relationship_with_upstream]] also apply for
system administration.

intrigeri's avatar
intrigeri committed
48
49
<a id="duties"></a>

intrigeri's avatar
intrigeri committed
50
# Duties
intrigeri's avatar
intrigeri committed
51

intrigeri's avatar
intrigeri committed
52
53
54
55
56
57
58
59
60
61
62
63
64
## In general

As said above, "set up and maintain the infrastructure". This implies
for example:

* dealing with hardware purchase, upgrades and failures;
* upgrading our systems to a new version of Debian.

## During sysadmin shifts

* create Git repositories when requested
* update access control lists to resources we manage, as requested by
  the corresponding teams
intrigeri's avatar
intrigeri committed
65
66
* keep systems up-to-date, reboot them as needed
* keep backups up-to-date
67
68
69
70
71
72
73
74
75
* keep Jenkins plugins up-to-date, by upgrading any plugin that satisfies
  at least one of these conditions:
   - only brings security fixes
   - fixes bugs we're affected by
   - brings new feature we are interested in, without breaking the ones we rely on
   - is needed to upgrade another plugin that we want to upgrade
   - is required by a system upgrade (e.g. of the Jenkins packages)
* report bugs identified in Jenkins plugins after they have been upgraded (both
  on the upstream bug tracker and on our own one)
76
77
* act as the de facto interface between Tails and the servers hosting
  our services (boum.org, immerda.ch) for non-trivial requests
78
79
* when a sysadmin shift includes the beginning of a yearly quarter, ensure that
  sysadmin shifts are filled and agreed on for the next two quarters
intrigeri's avatar
intrigeri committed
80

81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
<a id="tools"></a>

# Tools

The main tools used to manage the Tails infrastructure are:

* [Debian](https://www.debian.org/) GNU/Linux; in the vast majority of
  cases, we run the current stable release
* [Puppet](http://projects.puppetlabs.com/projects/puppet),
  a configuration management system
* [Git](http://git-scm.com/) to host and deploy configuration,
  including our [[Puppet modules|contribute/git#puppet]]

<a id="communication"></a>

# Communication

A few people have write access to the puppetmasters, and can log into
the hosts.  
sajolida's avatar
sajolida committed
100
They read the <tails-sysadmins@boum.org> encrypted mailing list.
101
102
103
104
105
106
107
108
109
110
111
112

We use Redmine tickets for public discussion and tasks management:

* [tasks requiring *Sysadmin*
  work](https://labs.riseup.net/code/projects/tails/issues?query_id=113)
* [tasks belonging to the *Infrastructure*
  category](https://labs.riseup.net/code/projects/tails/issues?query_id=140)

<a id="services"></a>

# Services

intrigeri's avatar
intrigeri committed
113
114
115
116
117
118
119
120
121
Below, importance level is evaluated based on:

* users' needs: e.g. if the APT repository is down, then the
  "Additional Software Packages" persistence feature is broken;
* developers' needs: e.g. if the ISO build fails, then developers
  cannot work;
* the release process' needs: we want to be able to do an emergency
  release at any time when critical security issues are published.

122
123
## APT repositories

124
125
<a id="custom-apt-repository"></a>

126
### Custom APT repository
127
128

* purpose: host Tails-specific Debian packages
129
* [[documentation|contribute/APT repository/custom]]
130
131
* access: anyone can read, Tails core developers can write
* tools: [[!debpts reprepro]]
132
133
134
* configuration:
  - `tails::reprepro::custom` class in [[!tails_gitweb_repo puppet-tails]]
  - signing keys are managed with the `tails_secrets_apt` Puppet module
intrigeri's avatar
intrigeri committed
135
* importance: critical (needed by users, and to build & release a Tails ISO)
136

137
138
### Time-based snapshots of APT repositories

139
140
141
142
* purpose: host full snapshots of the upstream APT repositories we
  need, which provides the freezable APT repositories feature needed
  by the Tails development and QA processes
* [[documentation|contribute/APT repository/time-based snapshots]]
143
* access: anyone can read, release managers have write access
144
* tools: [[!debpts reprepro]]
145
146
147
148
* configuration:
  - `tails::reprepro::snapshots::time_based` class in
    [[!tails_gitweb_repo puppet-tails]]
  - signing keys are managed with the `tails_secrets_apt` Puppet module
intrigeri's avatar
intrigeri committed
149
* importance: critical (needed to build a Tails ISO)
150
151
152

### Tagged snapshots of APT repositories

153
154
155
* purpose: host partial snapshots of the upstream APT repositories we
  need, for historical purposes and compliance with some licenses
* [[documentation|contribute/APT repository/tagged snapshots]]
156
157
* access: anyone can read, release managers can create and publish new
  snapshots
158
* tools: [[!debpts reprepro]]
159
160
161
162
* configuration:
  - `tails::reprepro::snapshots::tagged` class in
    [[!tails_gitweb_repo puppet-tails]]
  - signing keys are managed with the `tails_secrets_apt` Puppet module
intrigeri's avatar
intrigeri committed
163
* importance: critical (needed by users and to release Tails)
164

165
166
167
168
169
170
## Bitcoind

* purpose: handle the Tails Bitcoin wallet
* access: Tails core developers only
* tools: [[!debpts bitcoind]]
* configuration: `bitcoind` class in [[!tails_gitweb_repo puppet-bitcoind]]
intrigeri's avatar
intrigeri committed
171
* importance: medium
172
173
174
175

## BitTorrent

* purpose: seed the new ISO image when preparing a release
176
* [[documentation|contribute/release_process]]
177
178
179
* access: anyone can read, Tails core developers can write
* tools: [[!debpts transmission-daemon]]
* configuration: done by hand ([[!tails_ticket 6926]])
intrigeri's avatar
intrigeri committed
180
* importance: low
181
182
183
184
185
186
187
188
189

## Gitolite

* purpose: host Git repositories used by the puppetmaster and other
  services; mostly useless for humans
* access: Tails core developers only
* tools: [[!debpts gitolite]]
* configuration: `tails::gitolite` class in [[!tails_gitweb_repo
  puppet-tails]]
intrigeri's avatar
intrigeri committed
190
* importance: high (needed to release Tails)
191

192
193
## git-annex

194
195
* purpose: host the full history of Tails released images and Tor
  Browser tarballs
196
197
* access: Tails core developers only
* tools: [[!debpts git-annex]]
198
199
200
201
202
* configuration:
  - `tails::git_annex` and `tails::gitolite` classes in
    [[!tails_gitweb_repo puppet-tails]]
  - `tails::git_annex::mirror` defined resource in
    [[!tails_gitweb_repo puppet-tails]]
intrigeri's avatar
intrigeri committed
203
* importance: high (needed to release Tails)
204

205
206
<a id="icinga2"></a>

207
208
209
210
211
## Icinga2

* purpose: Monitor Tails online services and systems.
* access: only Tails core developers can read-only the Icingaweb2 interface,
  sysadmins are RW and receive notifications by email.
bertagaz's avatar
bertagaz committed
212
213
214
215
216
217
218
219
* setup: We have one Icinga2 instance installed on a dedicated system
  used as the master of all our Icinga2 zones. We use a VM on the other
  bare-metal host as the Icinga2 satellite of our master. Icinga2 agents are
  installed on every other VM and the host itself. They report back to
  the satellite, which transmits to the master. We spread the Icinga2
  configuration with Puppet. This way, we achieve a certain isolation
  where the master or the satellite have no right to configure agents or
  run arbitrary commands on them.
220
* tools: [[!debpts icinga2 desc="Icinga2"]], [[!debpts icingaweb2]]
221
* configuration:
222
223
224
225
226
227
228
229
230
231
232
233
  - master:
    * `tails::monitoring::master` class in [[!tails_gitweb_repo puppet-tails]].
    * some configuration in the ecours.tails.boum.org node manifest.
    * See Vpn section.
  - web server:
    * `tails::monitoring::icingaweb2` class in [[!tails_gitweb_repo puppet-tails]],
       that wraps around [upstream `icingaweb2` module](https://git.icinga.org/puppet-icingaweb2.git).
    * some configuration in the ecours.tails.boum.org node manifest.
  - satellite:
    * `tails::monitoring::satellite` class in [[!tails_gitweb_repo puppet-tails]],
  - agents:
    * `tails::monitoring::agent` class in [[!tails_gitweb_repo puppet-tails]]
234
  - private keys are managed with the `tails_secrets_monitoring` Puppet module
235
236
* documentation:
  - [[How to add checks to our monitoring setup|roles/sysadmins/adding_icinga2_checks]]
intrigeri's avatar
intrigeri committed
237
* importance: critical (needed to ensure that other, critical services are working)
238

239
240
241
242
243
244
245
246
247
248
## Jenkins

* purpose: continuous integration, e.g. build Tails ISO images from
  source and run test suites
* access: only Tails core developers can see the Jenkins web interface
  ([[!tails_ticket 6270]]); anyone can [[download the built
  products|contribute/how/testing]]
* tools: [[!debpts jenkins desc="Jenkins"]], [[!debpts jenkins-job-builder]]
* configuration:
  - master:
Tails developers's avatar
Tails developers committed
249
    * `jenkins` class in [[!tails_gitweb_repo puppet-jenkins]]
250
    * `tails::jenkins::master` class in [[!tails_gitweb_repo puppet-tails]]
Tails developers's avatar
Tails developers committed
251
    * a few Jenkins plugins installed with `jenkins::plugin`
252
253
254
255
    * YAML jobs configuration lives in a
      [[!tails_gitweb_repo jenkins-jobs desc="dedicated Git repository"]];
      [Jenkins Job Builder](http://ci.openstack.org/jenkins-job-builder/)
      uses it to configure Jenkins
256
  - slaves:
257
    * `tails::builder`, `tails::jenkins::slave`,
258
      `tails::jenkins::slave::iso_builder` and `tails::tester` classes in
259
      [[!tails_gitweb_repo puppet-tails]]
260
    * some configuration in the manifest ([[!tails_ticket 7106]])
261
    * signing keys are managed with the `tails_secrets_jenkins` Puppet module
262
263
  - web server:
    * some configuration in the manifest ([[!tails_ticket 7107]])
intrigeri's avatar
intrigeri committed
264
* design documentation:
intrigeri's avatar
intrigeri committed
265
  - [[sysadmins/automated_builds_in_Jenkins]]
intrigeri's avatar
intrigeri committed
266
  - [[sysadmins/automated_tests_in_Jenkins]]
intrigeri's avatar
intrigeri committed
267
* importance: critical (as a key component of our development process)
268

269
270
271
272
## Mumble

* purpose: internal communication for the Fundraising team
* access: Fundraising team members
273
* tools: [[!debpts mumble-server]]
274
275
276
* configuration:
  - <https://github.com/voxpupuli/puppet-mumble>
  - `mumble::*` parameters in Hiera
intrigeri's avatar
intrigeri committed
277
* importance: low
278

intrigeri's avatar
intrigeri committed
279
280
<a id="rsync"></a>

281
282
283
284
285
286
287
## rsync

* purpose: provide content to the public rsync server, from which all
  HTTP mirrors in turn pull
* access: read-only for those who need it, read-write for Tails core
  developers
* tools: [[!debpts rsync]]
288
289
290
291
* configuration:
  - `tails::rsync` in [[!tails_gitweb_repo puppet-tails]]
  - users and credentials are managed with the `tails_secrets_rsync`
    Puppet module
intrigeri's avatar
intrigeri committed
292
* importance: critical (needed to release Tails)
293
294
295
296
297
298
299

## Tor bridge

* purpose: provide a Tor bridge that Tails contributors can easily use
  for testing
* access: anyone who gets it from
  [BridgeDB](https://bridges.torproject.org/)
Tails developers's avatar
Tails developers committed
300
* tools: [[!debpts tor]], [[!debpts obfs4proxy]]
301
302
303
304
* configuration:
  - `tails::apt::repository::torproject` in
    [[!tails_gitweb_repo puppet-tails]]
  - `tor::daemon::relay` in [[!tails_gitweb_repo puppet-tor]]
intrigeri's avatar
intrigeri committed
305
* importance: low
306

307
308
309
310
311
312
313
314
315
## VPN

* purpose: flow through VPN traffic the connections between our
  different remote systems. Mainly used by the monitoring service.
* access: private network.
* tools: [[!debpts tinc]]
* configuration:
  - `tails::vpn::instance` class in the [[!tails_gitweb_repo puppet-tails]]
     repo.
intrigeri's avatar
intrigeri committed
316
* importance: transitively critical (as a dependency of our monitoring system)
317

318
319
320
321
322
323
324
325
326
## Web server

* purpose: serve web content for any other service that need it
* access: depending on the service
* tools: [[!debpts nginx]]
* configuration:
  - `nginx` class in [[!tails_gitweb_repo puppet-nginx]]
  - hard-coded manifest snippets and files on the puppetmaster
    ([[!tails_ticket 6938]])
intrigeri's avatar
intrigeri committed
327
* importance: transitively critical (as a dependency of Jenkins)
328

intrigeri's avatar
intrigeri committed
329
330
<a id="weblate"></a>

intrigeri's avatar
intrigeri committed
331
332
333
334
335
336
337
338
## Weblate

* URL: <https://translate.tails.boum.org/>
* purpose: web interface for translators
* admins: emmapeel, spriver
* tools: [Weblate](https://weblate.org/)
* configuration:
  - `tails::weblate` class in [[!tails_gitweb_repo puppet-tails]]
intrigeri's avatar
intrigeri committed
339
* importance: low (not in production yet)
intrigeri's avatar
intrigeri committed
340

341
342
343
344
345
## WhisperBack relay

* purpose: forward bug reports sent with WhisperBack to <tails-bugs@boum.org>
* access: public; WhisperBack (and hence, any bug reporter) uses it
* tools: [[!debpts postfix desc="Postfix"]]
346
347
348
349
* configuration:
  - `tails::whisperback::relay` in [[!tails_gitweb_repo puppet-tails]]
  - private keys are managed with the `tails_secrets_whisperback`
    Puppet module
intrigeri's avatar
intrigeri committed
350
* importance: high
351
352
353
354

# Other pages

[[!map pages="contribute/working_together/roles/sysadmins/*"]]