sysadmins.mdwn 19.6 KB
Newer Older
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113
[[!meta title="System administrators"]]

[[!toc levels=2]]

<a id="goals"></a>

# Goals

The Tails system administrators set up and maintain the infrastructure
that supports the development and operations of Tails. We aim at
making the life of Tails contributors easier, and to improve the quality of
the Tails releases.

<a id="principles"></a>

# Principles

## Infrastructure as code

We want to treat system administration like a (free) software
development project:

* We want to enable people to participate without needing an account
  on the Tails servers.
* We want to review the changes that are applied to our systems.
* We want to be able to easily reproduce our systems via
  automatic deployment.
* We want to share knowledge with other people.

This is why we try to publish as much as possible of our systems
configuration, and to manage our whole infrastructure with
configuration management tools. That is, without needing to log
into hosts.

## Free Software

We use Free Software, as defined by the [Debian Free Software
Guidelines](https://www.debian.org/social_contract#guidelines).  
The firmware our systems might need are the only exception to
this rule.

## Relationships with upstream

The [[principles used by the broader Tails
project|contribute/relationship_with_upstream]] also apply for
system administration.

<a id="duties"></a>

# Duties

## In general

As said above, "set up and maintain the infrastructure". This implies
for example:

* dealing with hardware purchase, upgrades and failures;
* upgrading our systems to a new version of Debian.

## During sysadmin shifts

* create Git repositories when requested
* update access control lists to resources we manage, as requested by
  the corresponding teams
* keep systems up-to-date, reboot them as needed
* keep Jenkins plugins up-to-date, by upgrading any plugin that satisfies
  at least one of these conditions:
   - brings security fixes
   - fixes bugs we're affected by
   - brings new feature we are interested in, without breaking the ones we rely on
   - is needed to upgrade another plugin that we want to upgrade
   - is required by a system upgrade (e.g. of the Jenkins packages)
* report bugs identified in Jenkins plugins after they have been upgraded (both
  on the upstream bug tracker and on our own one)
* act as the de facto interface between Tails and the people hosting
  our services (boum.org, immerda.ch) for non-trivial requests
* when a sysadmin shift includes the beginning of a yearly quarter, ensure that
  sysadmin shifts are filled and agreed on for the next two quarters
* quarterly: self-evaluate our work and report to the -summit@ mailing list
* When the deadline for taking over a given maintenance task (see
  below) has passed, the sysadmin on duty must make it clear s·he's
  handling the problem before starting to work on it, in order to
  avoid work duplication.

## Outside of sysadmin shifts

* Read email at least twice a week to check if the sysadmin currently
  on duty needs help.

* Once 48 hours have passed after a problem was identified, the
  sysadmins not currently on duty can/should take over maintenance
  tasks if the on duty sysadmin is MIA; for critical problems this
  delay shall be reduced.


<a id="tools"></a>

# Tools

The main tools used to manage the Tails infrastructure are:

* [Debian](https://www.debian.org/) GNU/Linux; in the vast majority of
  cases, we run the current stable release
* [Puppet](http://projects.puppetlabs.com/projects/puppet),
  a configuration management system
  - our [[Puppet code|contribute/git#puppet]]
* [Git](http://git-scm.com/) to host and deploy configuration,
  including our Puppet code

<a id="communication"></a>

# Communication

intrigeri's avatar
intrigeri committed
114
Tails system administrators have write access to the puppetmasters, and can log into
115 116 117
the hosts.  
They read the <tails-sysadmins@boum.org> encrypted mailing list.

118
We use [[!tails_gitlab desc="GitLab"]] issues for public discussion
119 120
and tasks management:

121 122 123 124 125 126 127 128 129 130 131 132 133 134
* To bring an issue to the attention of system administrators,
  mention the `@sysadmins-team` group.
* [[!tails_gitlab
  groups/tails/-/issues?label_name%5B%5D=Core+Work%3ASysadmin+%28maintenance%29
  desc="issues that should be taken care of as part of sysadmin shifts"]]
* [[!tails_gitlab
  groups/tails/-/issues?label_name%5B%5D=Core+Work%3ASysadmin+%28adapt%29
  desc="issues that are on the sysadmin team's roadmap"]]
* [[!tails_gitlab
  groups/tails/-/issues?label_name%5B%5D=T%3ASysadmin
  desc="tasks that require *Sysadmin* work"]]
* [[!tails_gitlab
  groups/tails/-/issues?label_name%5B%5D=C%3AInfrastructure
  desc="tasks that belong to the *Infrastructure* category"]]
135 136 137
* [[!tails_gitlab
  groups/tails/-/issues?scope=all&utf8=%E2%9C%93&state=opened&assignee_username=Sysadmins
  desc="issues assigned to the *Sysadmins* user"]]
138 139 140 141 142 143 144 145

<a id="services"></a>

# Services

Below, importance level is evaluated based on:

* users' needs: e.g. if the APT repository is down, then the
146
  _Additional Software_ feature is broken;
147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162
* developers' needs: e.g. if the ISO build fails, then developers
  cannot work;
* the release process' needs: we want to be able to do an emergency
  release at any time when critical security issues are published.

## APT repositories

<a id="custom-apt-repository"></a>

### Custom APT repository

* purpose: host Tails-specific Debian packages
* [[documentation|contribute/APT repository/custom]]
* access: anyone can read, Tails core developers can write
* tools: [[!debpts reprepro]]
* configuration:
163 164
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/reprepro/custom.pp
    desc="`tails::reprepro::custom` class"]]
165 166 167 168 169 170 171 172 173 174 175 176
  - signing keys are managed with the `tails_secrets_apt` Puppet module
* importance: critical (needed by users, and to build & release a Tails ISO)

### Time-based snapshots of APT repositories

* purpose: host full snapshots of the upstream APT repositories we
  need, which provides the freezable APT repositories feature needed
  by the Tails development and QA processes
* [[documentation|contribute/APT repository/time-based snapshots]]
* access: anyone can read, release managers have write access
* tools: [[!debpts reprepro]]
* configuration:
177 178
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/reprepro/snapshots/time_based.pp
    desc="`tails::reprepro::snapshots::time_based` class"]]
179 180 181 182 183 184 185 186 187 188 189 190
  - signing keys are managed with the `tails_secrets_apt` Puppet module
* importance: critical (needed to build a Tails ISO)

### Tagged snapshots of APT repositories

* purpose: host partial snapshots of the upstream APT repositories we
  need, for historical purposes and compliance with some licenses
* [[documentation|contribute/APT repository/tagged snapshots]]
* access: anyone can read, release managers can create and publish new
  snapshots
* tools: [[!debpts reprepro]]
* configuration:
191 192
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/reprepro/snapshots/tagged.pp
    desc="`tails::reprepro::snapshots::tagged` class"]]
193 194 195 196 197 198 199 200
  - signing keys are managed with the `tails_secrets_apt` Puppet module
* importance: critical (needed by users and to release Tails)

## Bitcoind

* purpose: handle the Tails Bitcoin wallet
* access: Tails core developers only
* tools: [[!debpts bitcoind]]
201 202 203
* configuration:
  [[!tails_gitlab tails/puppet-bitcoind/-/blob/master/manifests/init.pp
  desc="`bitcoind` class"]]
intrigeri's avatar
intrigeri committed
204
* Vcs-Git: [[!tails_gitweb_repo bitcoin]] and [[!tails_gitweb_repo libunivalue]]
205
* importance: medium
206 207 208 209
* To save disk space: as the `bitcoin@bitcoin.lizard` user, run
  `bitcoin-cli getblockcount` to get the ID of the last block,
  then run `bitcoin-cli pruneblockchain XYZ`, with `XYZ` being
  a Unix timestamp that's at least 5 months in the past.
210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230

## BitTorrent

* purpose: seed the new ISO image when preparing a release
* [[documentation|contribute/release_process]]
* access: anyone can read, Tails core developers can write
* tools: [[!debpts transmission-daemon]]
* configuration: done by hand ([[!tails_ticket 6926]])
* importance: low

## DNS

* purpose: authoritative nameserver for the `tails.boum.org` and
  `amnesia.boum.org` zones
* access:
  - anyone can query this nameserver
  - members of the mirrors team control some of the content of the
    `dl.amnesia.boum.org` sub-zone
  - Tails sysadmins can edit the zones with `pdnsutil edit-zone`
* tools: [[!debpts pdns]] with its MySQL backend
* configuration:
231 232 233 234 235
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/pdns.pp
    desc="`tails::pdns` class"]]
    and [[!tails_gitlab tails/puppet-tails/-/tree/master/manifests/pdns
    desc="`tails::pdns::*` resources"]]
  - [`powerdns` Puppet module](https://github.com/sensson/puppet-powerdns)
236 237 238
* importance: critical (most of our other services are not available
  if this one is not working)

239 240 241 242 243 244
## GitLab

* purpose:
  - host Tails issues
  - host most Tails [[Git repositories|contribute/git]]
* access: public + some data with more restricted access
intrigeri's avatar
intrigeri committed
245 246
* operations documentation: [[contribute/working_together/GitLab#operations]]
* end-user documentation: [[contribute/working_together/GitLab]]
247 248 249 250 251
* configuration:
  - immerda hosts our GitLab instance using [this Puppet
    code](https://code.immerda.ch/immerda/ibox/puppet-modules/-/blob/master/ib_gitlab/manifests/instance.pp).
  - We don't have shell access.
  - Tails system administrators have administrator credentials inside GitLab.
intrigeri's avatar
intrigeri committed
252 253 254
  - Groups, projects, and access control:
     - [[high-level documentation|working_together/GitLab#access-control]]
     - configuration: `gitlab-config` repository (hosted on `puppet-git.lizard`)
255 256 257
* importance: critical (needed to release Tails)
* Tails system administrators administrate this GitLab instance.

258 259 260 261
## Gitolite

* purpose:
  - host Git repositories used by the puppetmaster and other services
262 263
  - host mirrors of various Git repositories needed on lizard,
    and whose canonical copy lives on GitLab
264
* access: Tails core developers only
intrigeri's avatar
intrigeri committed
265
* tools: [[!debpts gitolite3]]
266 267 268
* configuration:
  [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/gitolite.pp
  desc="`tails::gitolite` class"]]
269 270 271 272 273 274 275 276 277
* importance: high (needed to release Tails)

## git-annex

* purpose: host the full history of Tails released images and Tor
  Browser tarballs
* access: Tails core developers only
* tools: [[!debpts git-annex]]
* configuration:
278 279 280 281 282 283
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/git_annex.pp
    desc="`tails::git_annex` class"]]
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/gitolite.pp
    desc="`tails::gitolite` class"]]
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/git_annex/mirror.pp
    desc="`tails::git_annex::mirror` defined resource"]]
284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 301 302 303
* importance: high (needed to release Tails)

<a id="icinga2"></a>

## Icinga2

* purpose: Monitor Tails online services and systems.
* access: only Tails core developers can read-only the Icingaweb2 interface,
  sysadmins are RW and receive notifications by email.
* setup: We have one Icinga2 instance installed on a dedicated system
  used as the master of all our Icinga2 zones. We use a VM on the other
  bare-metal host as the Icinga2 satellite of our master. Icinga2 agents are
  installed on every other VM and the host itself. They report back to
  the satellite, which transmits to the master. We spread the Icinga2
  configuration with Puppet. This way, we achieve a certain isolation
  where the master or the satellite have no right to configure agents or
  run arbitrary commands on them.
* tools: [[!debpts icinga2 desc="Icinga2"]], [[!debpts icingaweb2]]
* configuration:
  - master:
304 305
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/monitoring/master.pp
      desc="`tails::monitoring::master` class"]].
306 307 308
    * some configuration in the ecours.tails.boum.org node manifest.
    * See Vpn section.
  - web server:
309 310 311
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/monitoring/icingaweb2.pp
      desc="`tails::monitoring::icingaweb2` class"]],
      that wraps around [upstream `icingaweb2` module](https://git.icinga.org/puppet-icingaweb2.git).
312 313
    * some configuration in the ecours.tails.boum.org node manifest.
  - satellite:
314 315
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/monitoring/satellite.pp
      desc="`tails::monitoring::satellite` class"]]
316
  - agents:
317 318
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/monitoring/agent.pp
      desc="`tails::monitoring::agent` class"]]
319 320 321 322 323 324 325 326 327 328 329 330
  - private keys are managed with the `tails_secrets_monitoring` Puppet module
* documentation:
  - [[How to add checks to our monitoring setup|roles/sysadmins/adding_icinga2_checks]]
* importance: critical (needed to ensure that other, critical services are working)

## Internal XMPP service

* purpose: an internal XMPP service that can be used by Tails developers and some contributors.
* access: at the moment everyone that is on the tails-summit mailinglist has and/or can
  request an account.
* tools: prosody
* configuration:
331 332
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/prosody.pp
    desc="`tails::prosody` class"]]
333 334 335 336 337 338 339 340 341 342 343 344
* importance: low

## Jenkins

* purpose: continuous integration, e.g. build Tails ISO images from
  source and run test suites
* access: only Tails core developers can see the Jenkins web interface
  ([[!tails_ticket 6270]]); anyone can [[download the built
  products|contribute/how/testing]]
* tools: [[!debpts jenkins desc="Jenkins"]], [[!debpts jenkins-job-builder]]
* configuration:
  - master:
345 346 347 348
    * [[!tails_gitlab tails/puppet-jenkins/-/blob/master/manifests/init.pp
      desc="`jenkins` class"]]
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/jenkins/master.pp
      desc="`tails::jenkins::master` class"]]
349 350 351 352 353 354
    * a few Jenkins plugins installed with `jenkins::plugin`
    * YAML jobs configuration lives in a
      [[!tails_gitweb_repo jenkins-jobs desc="dedicated Git repository"]];
      [Jenkins Job Builder](http://ci.openstack.org/jenkins-job-builder/)
      uses it to configure Jenkins
  - slaves:
intrigeri's avatar
intrigeri committed
355 356
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/iso_builder.pp
      desc="`tails::iso_builder`"]],
357 358 359 360
      [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/jenkins/slave.pp
      desc="`tails::jenkins::slave`"]],
      [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/jenkins/slave/iso_builder.pp
      desc="`tails::jenkins::slave::iso_builder`"]],
361 362
      [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/jenkins/slave/iso_tester.pp
      desc="`tails::jenkins::slave::iso_tester`"]],
363 364 365
      and [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/tester.pp
      desc="`tails::tester`"]]
      classes
366 367
    * signing keys are managed with the `tails_secrets_jenkins` Puppet module
  - web server:
368 369
    * [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/jenkins/reverse_proxy.pp
      desc="`tails::jenkins::reverse_proxy` class"]]
370
* design documentation: [[sysadmins/Jenkins]]
371 372 373 374 375 376 377 378
* importance: critical (as a key component of our development process)

## Mail

* purpose: handle incoming and outgoing email for some of our
  [[Schleuder lists|sysadmins#schleuder]]
* access: public MTA listening on `mail.tails.boum.org`
* tools: [[!debpts postfix]], [[!debpts amavisd-new]], [[!debpts spamassassin]]
379 380 381 382 383 384 385 386 387
* configuration:
  [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/postfix.pp
  desc="`tails::postfix`"]],
  [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/amavisd_new.pp
  desc="`tails::amavisd_new`"]],
  and
  [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/spamassassin.pp
  desc="`tails::spamassassin`"]]
  classes
388 389
* importance: high (at least because WhisperBack bug reports go through this MTA)

390 391 392 393 394 395 396 397
<a id="meeting-reminder"></a>

## Meeting reminder

* purpose: send email reminders, for example about upcoming meetings
* access: not applicable
* configuration:
  - to add a new reminder, or modify an existing one:
398 399 400 401 402 403 404 405 406 407 408 409
    - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/meeting/reminders.pp
      desc="`tails::meeting::reminders`"]]
    - [[!tails_gitlab tails/puppet-tails/-/tree/master/files/meeting
      desc="email templates"]]
  - implementation:
    [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/meeting.pp
    desc="`tails::meeting`"]],
    [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/meeting/reminder.pp
    desc="`tails::meeting::reminder`"]],
    and
    [[!tails_gitlab tails/puppet-tails/-/blob/master/files/meeting/meeting.py
    desc="`meeting.py` script"]]
410 411
* importance: to be defined

412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433
<a id="mumble"></a>

## Mumble

* purpose: internal communication for some internal teams
* access: members of some internal teams
* tools: [[!debpts mumble-server]]
* configuration:
  - <https://github.com/voxpupuli/puppet-mumble>
  - `mumble::*` parameters in Hiera
* importance: low

<a id="rsync"></a>

## rsync

* purpose: provide content to the public rsync server, from which all
  HTTP mirrors in turn pull
* access: read-only for those who need it, read-write for Tails core
  developers
* tools: [[!debpts rsync]]
* configuration:
434 435
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/rsync.pp
    desc="`tails::rsync`"]]
436 437 438 439 440 441 442 443 444 445 446
  - users and credentials are managed with the `tails_secrets_rsync`
    Puppet module
* importance: critical (needed to release Tails)

<a id="schleuder"></a>

## Schleuder

* purpose: host some of our Schleuder mailing lists
* access: anyone can send email to these lists
* tools: [[!debpts schleuder]]
447 448 449 450
* configuration:
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/schleuder.pp
    desc="`tails::schleuder` class"]]
  - `tails::schleuder::lists` Hiera setting
451 452 453 454 455 456 457 458 459 460
* importance: high (at least because WhisperBack bug reports go through this service)

## Tor bridge

* purpose: provide a Tor bridge that Tails contributors can easily use
  for testing
* access: anyone who gets it from
  [BridgeDB](https://bridges.torproject.org/)
* tools: [[!debpts tor]], [[!debpts obfs4proxy]]
* configuration:
461 462 463 464
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/apt/repository/torproject.pp
    desc="`tails::apt::repository::torproject`"]]
  - [[!tails_gitlab tails/puppet-tor/-/blob/master/manifests/daemon/relay.pp
    desc="`tor::daemon::relay`"]]
465 466 467 468 469 470 471 472 473
* importance: low

## VPN

* purpose: flow through VPN traffic the connections between our
  different remote systems. Mainly used by the monitoring service.
* access: private network.
* tools: [[!debpts tinc]]
* configuration:
474 475
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/vpn/instance.pp
    desc="`tails::vpn::instance` class"]]
476 477 478 479 480 481 482 483
* importance: transitively critical (as a dependency of our monitoring system)

## Web server

* purpose: serve web content for any other service that need it
* access: depending on the service
* tools: [[!debpts nginx]]
* configuration:
484 485
  - [[!tails_gitlab tails/puppet-nginx/-/blob/master/manifests/init.pp
    desc="`nginx` class"]]
486 487 488 489 490 491 492 493
* importance: transitively critical (as a dependency of Jenkins)

<a id="weblate"></a>

## Weblate

* URL: <https://translate.tails.boum.org/>
* purpose: web interface for translators
intrigeri's avatar
intrigeri committed
494 495
* [[design documentation|contribute/design/translation_platform]]
* [[usage documentation|contribute/how/translate/with_translation_platform]]
496
* admins: to be defined ([[!tails_ticket 17050]])
497 498
* tools: [Weblate](https://weblate.org/)
* configuration:
499 500
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/weblate.pp
    desc="`tails::weblate` class"]]
501
* importance: to be defined
502 503 504 505 506 507 508

## WhisperBack relay

* purpose: forward bug reports sent with WhisperBack to <tails-bugs@boum.org>
* access: public; WhisperBack (and hence, any bug reporter) uses it
* tools: [[!debpts postfix desc="Postfix"]]
* configuration:
509 510
  - [[!tails_gitlab tails/puppet-tails/-/blob/master/manifests/whisperback/relay.pp
    desc="`tails::whisperback::relay` class"]]
511 512 513 514 515 516 517
  - private keys are managed with the `tails_secrets_whisperback`
    Puppet module
* importance: high

# Other pages

[[!map pages="contribute/working_together/roles/sysadmins/*"]]