sysadmins.mdwn 18.9 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
[[!meta title="System administrators"]]

[[!toc levels=2]]

<a id="goals"></a>

# Goals

The Tails system administrators set up and maintain the infrastructure
that supports the development and operations of Tails. We aim at
making the life of Tails contributors easier, and to improve the quality of
the Tails releases.

<a id="principles"></a>

# Principles

## Infrastructure as code

We want to treat system administration like a (free) software
development project:

* We want to enable people to participate without needing an account
  on the Tails servers.
* We want to review the changes that are applied to our systems.
* We want to be able to easily reproduce our systems via
  automatic deployment.
* We want to share knowledge with other people.

This is why we try to publish as much as possible of our systems
configuration, and to manage our whole infrastructure with
configuration management tools. That is, without needing to log
into hosts.

## Free Software

We use Free Software, as defined by the [Debian Free Software
Guidelines](https://www.debian.org/social_contract#guidelines).  
The firmware our systems might need are the only exception to
this rule.

## Relationships with upstream

The [[principles used by the broader Tails
project|contribute/relationship_with_upstream]] also apply for
system administration.

<a id="duties"></a>

# Duties

## In general

As said above, "set up and maintain the infrastructure". This implies
for example:

* dealing with hardware purchase, upgrades and failures;
* upgrading our systems to a new version of Debian.

## During sysadmin shifts

* create Git repositories when requested
* update access control lists to resources we manage, as requested by
  the corresponding teams
* keep systems up-to-date, reboot them as needed
* keep Jenkins plugins up-to-date, by upgrading any plugin that satisfies
  at least one of these conditions:
   - brings security fixes
   - fixes bugs we're affected by
   - brings new feature we are interested in, without breaking the ones we rely on
   - is needed to upgrade another plugin that we want to upgrade
   - is required by a system upgrade (e.g. of the Jenkins packages)
* report bugs identified in Jenkins plugins after they have been upgraded (both
  on the upstream bug tracker and on our own one)
* act as the de facto interface between Tails and the people hosting
  our services (boum.org, immerda.ch) for non-trivial requests
* when a sysadmin shift includes the beginning of a yearly quarter, ensure that
  sysadmin shifts are filled and agreed on for the next two quarters
* quarterly: self-evaluate our work and report to the -summit@ mailing list
* When the deadline for taking over a given maintenance task (see
  below) has passed, the sysadmin on duty must make it clear s·he's
  handling the problem before starting to work on it, in order to
  avoid work duplication.

## Outside of sysadmin shifts

* Read email at least twice a week to check if the sysadmin currently
  on duty needs help.

* Once 48 hours have passed after a problem was identified, the
  sysadmins not currently on duty can/should take over maintenance
  tasks if the on duty sysadmin is MIA; for critical problems this
  delay shall be reduced.


<a id="tools"></a>

# Tools

The main tools used to manage the Tails infrastructure are:

* [Debian](https://www.debian.org/) GNU/Linux; in the vast majority of
  cases, we run the current stable release
* [Puppet](http://projects.puppetlabs.com/projects/puppet),
  a configuration management system
  - our [[Puppet code|contribute/git#puppet]]
* [Git](http://git-scm.com/) to host and deploy configuration,
  including our Puppet code

110
111
112
Sysadmins can login to all hosts and have write access to the Puppet masters'
Git repositories.

113
114
115
116
<a id="communication"></a>

# Communication

117
118
119
In order to get in touch with Tails sysadmins, you can:

* Create an issue in the [[!tails_gitlab tails/sysadmin]] project
Zen Fu's avatar
Zen Fu committed
120
* Ping all sysadmins anywhere in our [[!tails_gitlab desc="GitLab"]] by mentioning the `@sysadmin-team` group
121
122
* See if one of us is on shift in [[one of our chat rooms|about/contact#chat]]
* Send an e-mail to [[the sysadmin's mailing list|about/contact#tails-sysadmins]]
123

124
The following lists of issues are also of interest to sysadmins:
125

126
* [[!tails_gitlab
127
128
129
  groups/tails/-/issues?label_name%5B%5D=Core+Work%3ASysadmin
  desc="issues that should be taken care of as part of sysadmin shifts
  or are on the sysadmin team's roadmap"]]
130
131
132
* [[!tails_gitlab
  groups/tails/-/issues?label_name%5B%5D=C%3AInfrastructure
  desc="tasks that belong to the *Infrastructure* category"]]
133
134
135
136
137
138
139
140

<a id="services"></a>

# Services

Below, importance level is evaluated based on:

* users' needs: e.g. if the APT repository is down, then the
141
  _Additional Software_ feature is broken;
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
* developers' needs: e.g. if the ISO build fails, then developers
  cannot work;
* the release process' needs: we want to be able to do an emergency
  release at any time when critical security issues are published.

## APT repositories

<a id="custom-apt-repository"></a>

### Custom APT repository

* purpose: host Tails-specific Debian packages
* [[documentation|contribute/APT repository/custom]]
* access: anyone can read, Tails core developers can write
* tools: [[!debpts reprepro]]
* configuration:
158
  - [[!tails_gitweb_puppet_tails manifests/reprepro/custom.pp
159
    desc="`tails::reprepro::custom` class"]]
160
161
162
163
164
165
166
167
168
169
170
171
  - signing keys are managed with the `tails_secrets_apt` Puppet module
* importance: critical (needed by users, and to build & release a Tails ISO)

### Time-based snapshots of APT repositories

* purpose: host full snapshots of the upstream APT repositories we
  need, which provides the freezable APT repositories feature needed
  by the Tails development and QA processes
* [[documentation|contribute/APT repository/time-based snapshots]]
* access: anyone can read, release managers have write access
* tools: [[!debpts reprepro]]
* configuration:
172
  - [[!tails_gitweb_puppet_tails manifests/reprepro/snapshots/time_based.pp
173
    desc="`tails::reprepro::snapshots::time_based` class"]]
174
175
176
177
178
179
180
181
182
183
184
185
  - signing keys are managed with the `tails_secrets_apt` Puppet module
* importance: critical (needed to build a Tails ISO)

### Tagged snapshots of APT repositories

* purpose: host partial snapshots of the upstream APT repositories we
  need, for historical purposes and compliance with some licenses
* [[documentation|contribute/APT repository/tagged snapshots]]
* access: anyone can read, release managers can create and publish new
  snapshots
* tools: [[!debpts reprepro]]
* configuration:
186
  - [[!tails_gitweb_puppet_tails manifests/reprepro/snapshots/tagged.pp
187
    desc="`tails::reprepro::snapshots::tagged` class"]]
188
189
190
191
192
193
194
195
  - signing keys are managed with the `tails_secrets_apt` Puppet module
* importance: critical (needed by users and to release Tails)

## Bitcoind

* purpose: handle the Tails Bitcoin wallet
* access: Tails core developers only
* tools: [[!debpts bitcoind]]
196
197
198
* configuration:
  [[!tails_gitlab tails/puppet-bitcoind/-/blob/master/manifests/init.pp
  desc="`bitcoind` class"]]
intrigeri's avatar
intrigeri committed
199
* Vcs-Git: [[!tails_gitweb_repo bitcoin]] and [[!tails_gitweb_repo libunivalue]]
200
* importance: medium
201
202
203
204
* To save disk space: as the `bitcoin@bitcoin.lizard` user, run
  `bitcoin-cli getblockcount` to get the ID of the last block,
  then run `bitcoin-cli pruneblockchain XYZ`, with `XYZ` being
  a Unix timestamp that's at least 5 months in the past.
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225

## BitTorrent

* purpose: seed the new ISO image when preparing a release
* [[documentation|contribute/release_process]]
* access: anyone can read, Tails core developers can write
* tools: [[!debpts transmission-daemon]]
* configuration: done by hand ([[!tails_ticket 6926]])
* importance: low

## DNS

* purpose: authoritative nameserver for the `tails.boum.org` and
  `amnesia.boum.org` zones
* access:
  - anyone can query this nameserver
  - members of the mirrors team control some of the content of the
    `dl.amnesia.boum.org` sub-zone
  - Tails sysadmins can edit the zones with `pdnsutil edit-zone`
* tools: [[!debpts pdns]] with its MySQL backend
* configuration:
226
  - [[!tails_gitweb_puppet_tails manifests/pdns.pp
227
228
229
230
    desc="`tails::pdns` class"]]
    and [[!tails_gitlab tails/puppet-tails/-/tree/master/manifests/pdns
    desc="`tails::pdns::*` resources"]]
  - [`powerdns` Puppet module](https://github.com/sensson/puppet-powerdns)
231
232
233
* importance: critical (most of our other services are not available
  if this one is not working)

intrigeri's avatar
intrigeri committed
234
235
<a id="gitlab"></a>

236
237
238
239
240
241
## GitLab

* purpose:
  - host Tails issues
  - host most Tails [[Git repositories|contribute/git]]
* access: public + some data with more restricted access
242
* operations documentation: [[contribute/working_together/roles/sysadmins/gitlab]]
intrigeri's avatar
intrigeri committed
243
* end-user documentation: [[contribute/working_together/GitLab]]
244
245
246
247
248
* configuration:
  - immerda hosts our GitLab instance using [this Puppet
    code](https://code.immerda.ch/immerda/ibox/puppet-modules/-/blob/master/ib_gitlab/manifests/instance.pp).
  - We don't have shell access.
  - Tails system administrators have administrator credentials inside GitLab.
intrigeri's avatar
intrigeri committed
249
250
  - Groups, projects, and access control:
     - [[high-level documentation|working_together/GitLab#access-control]]
251
     - configuration: [[!tails_gitlab tails/gitlab-config]]
252
253
* importance: critical (needed to release Tails)
* Tails system administrators administrate this GitLab instance.
254
* See our [[documentation about GitLab for Tails sysadmins|contribute/working_together/roles/sysadmins/gitlab]].
255

256
257
258
259
## Gitolite

* purpose:
  - host Git repositories used by the puppetmaster and other services
260
261
  - host mirrors of various Git repositories needed on lizard,
    and whose canonical copy lives on GitLab
262
* access: Tails core developers only
intrigeri's avatar
intrigeri committed
263
* tools: [[!debpts gitolite3]]
264
* configuration:
265
  [[!tails_gitweb_puppet_tails manifests/gitolite.pp
266
  desc="`tails::gitolite` class"]]
267
268
269
270
271
272
273
274
275
* importance: high (needed to release Tails)

## git-annex

* purpose: host the full history of Tails released images and Tor
  Browser tarballs
* access: Tails core developers only
* tools: [[!debpts git-annex]]
* configuration:
276
  - [[!tails_gitweb_puppet_tails manifests/git_annex.pp
277
    desc="`tails::git_annex` class"]]
278
  - [[!tails_gitweb_puppet_tails manifests/gitolite.pp
279
    desc="`tails::gitolite` class"]]
280
  - [[!tails_gitweb_puppet_tails manifests/git_annex/mirror.pp
281
    desc="`tails::git_annex::mirror` defined resource"]]
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
* importance: high (needed to release Tails)

<a id="icinga2"></a>

## Icinga2

* purpose: Monitor Tails online services and systems.
* access: only Tails core developers can read-only the Icingaweb2 interface,
  sysadmins are RW and receive notifications by email.
* setup: We have one Icinga2 instance installed on a dedicated system
  used as the master of all our Icinga2 zones. We use a VM on the other
  bare-metal host as the Icinga2 satellite of our master. Icinga2 agents are
  installed on every other VM and the host itself. They report back to
  the satellite, which transmits to the master. We spread the Icinga2
  configuration with Puppet. This way, we achieve a certain isolation
  where the master or the satellite have no right to configure agents or
  run arbitrary commands on them.
* tools: [[!debpts icinga2 desc="Icinga2"]], [[!debpts icingaweb2]]
* configuration:
  - master:
302
    * [[!tails_gitweb_puppet_tails manifests/monitoring/master.pp
303
      desc="`tails::monitoring::master` class"]].
304
305
306
    * some configuration in the ecours.tails.boum.org node manifest.
    * See Vpn section.
  - web server:
307
    * [[!tails_gitweb_puppet_tails manifests/monitoring/icingaweb2.pp
308
309
      desc="`tails::monitoring::icingaweb2` class"]],
      that wraps around [upstream `icingaweb2` module](https://git.icinga.org/puppet-icingaweb2.git).
310
311
    * some configuration in the ecours.tails.boum.org node manifest.
  - satellite:
312
    * [[!tails_gitweb_puppet_tails manifests/monitoring/satellite.pp
313
      desc="`tails::monitoring::satellite` class"]]
314
  - agents:
315
    * [[!tails_gitweb_puppet_tails manifests/monitoring/agent.pp
316
      desc="`tails::monitoring::agent` class"]]
317
318
319
320
321
322
323
324
325
326
327
328
  - private keys are managed with the `tails_secrets_monitoring` Puppet module
* documentation:
  - [[How to add checks to our monitoring setup|roles/sysadmins/adding_icinga2_checks]]
* importance: critical (needed to ensure that other, critical services are working)

## Internal XMPP service

* purpose: an internal XMPP service that can be used by Tails developers and some contributors.
* access: at the moment everyone that is on the tails-summit mailinglist has and/or can
  request an account.
* tools: prosody
* configuration:
329
  - [[!tails_gitweb_puppet_tails manifests/prosody.pp
330
    desc="`tails::prosody` class"]]
331
332
333
334
335
336
337
338
339
340
341
342
* importance: low

## Jenkins

* purpose: continuous integration, e.g. build Tails ISO images from
  source and run test suites
* access: only Tails core developers can see the Jenkins web interface
  ([[!tails_ticket 6270]]); anyone can [[download the built
  products|contribute/how/testing]]
* tools: [[!debpts jenkins desc="Jenkins"]], [[!debpts jenkins-job-builder]]
* configuration:
  - master:
343
344
    * [[!tails_gitlab tails/puppet-jenkins/-/blob/master/manifests/init.pp
      desc="`jenkins` class"]]
345
    * [[!tails_gitweb_puppet_tails manifests/jenkins/master.pp
346
      desc="`tails::jenkins::master` class"]]
347
348
349
350
351
352
    * a few Jenkins plugins installed with `jenkins::plugin`
    * YAML jobs configuration lives in a
      [[!tails_gitweb_repo jenkins-jobs desc="dedicated Git repository"]];
      [Jenkins Job Builder](http://ci.openstack.org/jenkins-job-builder/)
      uses it to configure Jenkins
  - slaves:
353
    * [[!tails_gitweb_puppet_tails manifests/iso_builder.pp
intrigeri's avatar
intrigeri committed
354
      desc="`tails::iso_builder`"]],
355
      [[!tails_gitweb_puppet_tails manifests/jenkins/slave.pp
356
      desc="`tails::jenkins::slave`"]],
357
      [[!tails_gitweb_puppet_tails manifests/jenkins/slave/iso_builder.pp
358
      desc="`tails::jenkins::slave::iso_builder`"]],
359
      [[!tails_gitweb_puppet_tails manifests/jenkins/slave/iso_tester.pp
360
      desc="`tails::jenkins::slave::iso_tester`"]],
361
      and [[!tails_gitweb_puppet_tails manifests/tester.pp
362
363
      desc="`tails::tester`"]]
      classes
364
365
    * signing keys are managed with the `tails_secrets_jenkins` Puppet module
  - web server:
366
    * [[!tails_gitweb_puppet_tails manifests/jenkins/reverse_proxy.pp
367
      desc="`tails::jenkins::reverse_proxy` class"]]
368
* design documentation: [[sysadmins/Jenkins]]
369
370
371
372
373
374
375
376
* importance: critical (as a key component of our development process)

## Mail

* purpose: handle incoming and outgoing email for some of our
  [[Schleuder lists|sysadmins#schleuder]]
* access: public MTA listening on `mail.tails.boum.org`
* tools: [[!debpts postfix]], [[!debpts amavisd-new]], [[!debpts spamassassin]]
377
* configuration:
378
  [[!tails_gitweb_puppet_tails manifests/postfix.pp
379
  desc="`tails::postfix`"]],
380
  [[!tails_gitweb_puppet_tails manifests/amavisd_new.pp
381
382
  desc="`tails::amavisd_new`"]],
  and
383
  [[!tails_gitweb_puppet_tails manifests/spamassassin.pp
384
385
  desc="`tails::spamassassin`"]]
  classes
386
387
* importance: high (at least because WhisperBack bug reports go through this MTA)

388
389
390
391
392
393
394
395
<a id="meeting-reminder"></a>

## Meeting reminder

* purpose: send email reminders, for example about upcoming meetings
* access: not applicable
* configuration:
  - to add a new reminder, or modify an existing one:
396
    - [[!tails_gitweb_puppet_tails manifests/meeting/reminders.pp
397
398
399
400
      desc="`tails::meeting::reminders`"]]
    - [[!tails_gitlab tails/puppet-tails/-/tree/master/files/meeting
      desc="email templates"]]
  - implementation:
401
    [[!tails_gitweb_puppet_tails manifests/meeting.pp
402
    desc="`tails::meeting`"]],
403
    [[!tails_gitweb_puppet_tails manifests/meeting/reminder.pp
404
405
    desc="`tails::meeting::reminder`"]],
    and
406
    [[!tails_gitweb_puppet_tails files/meeting/meeting.py
407
    desc="`meeting.py` script"]]
408
409
* importance: to be defined

410
411
412
413
414
415
416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
<a id="mumble"></a>

## Mumble

* purpose: internal communication for some internal teams
* access: members of some internal teams
* tools: [[!debpts mumble-server]]
* configuration:
  - <https://github.com/voxpupuli/puppet-mumble>
  - `mumble::*` parameters in Hiera
* importance: low

<a id="rsync"></a>

## rsync

* purpose: provide content to the public rsync server, from which all
  HTTP mirrors in turn pull
* access: read-only for those who need it, read-write for Tails core
  developers
* tools: [[!debpts rsync]]
* configuration:
432
  - [[!tails_gitweb_puppet_tails manifests/rsync.pp
433
    desc="`tails::rsync`"]]
434
435
436
437
438
439
440
441
442
443
444
  - users and credentials are managed with the `tails_secrets_rsync`
    Puppet module
* importance: critical (needed to release Tails)

<a id="schleuder"></a>

## Schleuder

* purpose: host some of our Schleuder mailing lists
* access: anyone can send email to these lists
* tools: [[!debpts schleuder]]
445
* configuration:
446
  - [[!tails_gitweb_puppet_tails manifests/schleuder.pp
447
448
    desc="`tails::schleuder` class"]]
  - `tails::schleuder::lists` Hiera setting
449
450
451
452
453
454
455
456
457
458
* importance: high (at least because WhisperBack bug reports go through this service)

## Tor bridge

* purpose: provide a Tor bridge that Tails contributors can easily use
  for testing
* access: anyone who gets it from
  [BridgeDB](https://bridges.torproject.org/)
* tools: [[!debpts tor]], [[!debpts obfs4proxy]]
* configuration:
459
  - [[!tails_gitweb_puppet_tails manifests/apt/repository/torproject.pp
460
461
462
    desc="`tails::apt::repository::torproject`"]]
  - [[!tails_gitlab tails/puppet-tor/-/blob/master/manifests/daemon/relay.pp
    desc="`tor::daemon::relay`"]]
463
464
465
466
467
468
469
470
471
* importance: low

## VPN

* purpose: flow through VPN traffic the connections between our
  different remote systems. Mainly used by the monitoring service.
* access: private network.
* tools: [[!debpts tinc]]
* configuration:
472
  - [[!tails_gitweb_puppet_tails manifests/vpn/instance.pp
473
    desc="`tails::vpn::instance` class"]]
474
475
476
477
478
479
480
481
* importance: transitively critical (as a dependency of our monitoring system)

## Web server

* purpose: serve web content for any other service that need it
* access: depending on the service
* tools: [[!debpts nginx]]
* configuration:
482
483
  - [[!tails_gitlab tails/puppet-nginx/-/blob/master/manifests/init.pp
    desc="`nginx` class"]]
484
485
486
487
488
489
490
491
* importance: transitively critical (as a dependency of Jenkins)

<a id="weblate"></a>

## Weblate

* URL: <https://translate.tails.boum.org/>
* purpose: web interface for translators
intrigeri's avatar
intrigeri committed
492
493
* [[design documentation|contribute/design/translation_platform]]
* [[usage documentation|contribute/how/translate/with_translation_platform]]
494
* admins: to be defined ([[!tails_ticket 17050]])
495
496
* tools: [Weblate](https://weblate.org/)
* configuration:
497
  - [[!tails_gitweb_puppet_tails manifests/weblate.pp
498
    desc="`tails::weblate` class"]]
499
* importance: to be defined
500
501
502
503
504
505
506

## WhisperBack relay

* purpose: forward bug reports sent with WhisperBack to <tails-bugs@boum.org>
* access: public; WhisperBack (and hence, any bug reporter) uses it
* tools: [[!debpts postfix desc="Postfix"]]
* configuration:
507
  - [[!tails_gitweb_puppet_tails manifests/whisperback/relay.pp
508
    desc="`tails::whisperback::relay` class"]]
509
510
511
512
513
514
515
  - private keys are managed with the `tails_secrets_whisperback`
    Puppet module
* importance: high

# Other pages

[[!map pages="contribute/working_together/roles/sysadmins/*"]]