release_process.mdwn 77 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
<div class="caution">
8
9
Read the remainder of this document from the branch used to prepare the release
after having merged the current master branch into it!
10
11
</div>

12
13
14
Requirements
============

15
16
17
Packages
--------

18
19
To release Tails you'll need some packages installed:

20
* `docker.io gitlab-cli jq tidy mktorrent python3-debian python3-gitlab python3-jinja2 transmission-cli`
21
22
* [[!debpts squashfs-tools]] 1:4.4-1+0.tails1
  from our custom `iukbuilder-stretch` APT suite.
23
24
* `iuk` [[dependencies|contribute/release_process/tails-iuk#build-deps]]
* `perl5lib` [[dependencies|contribute/release_process/perl5lib#build-deps]]
25
* `po4a` 0.55: different versions extract Markdown headings
26
   in a different way, which makes tons of strings fuzzy.
27
* packages to [[build a local version of the website|contribute/build/website/]]
28

29
30
31
32
33
34
35
36
37
38
39
40
Configuration files
-------------------

To release Tails you need:

* `~/.python-gitlab.cfg`

  You need at least this content:

        [global]
        ssl_verify = true

41
        [TailsRM]
42
43
44
45
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

46
47
48
49
50
51
52
53
54
55
56
57
58
        [Tails]
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

   Then:

   - In the `TailsRM` section, set the value of the `private_token` option to
     the `role-release-automation` GitLab user's API token, which you'll find in
     `rm.git`'s keyringer.

   - In the `Tails` section, set the value of the `private_token` option to a
     GitLab API token for your own user.
59

60
61
62
Environment
===========

63
64
65
66
67
68
69
To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the
value of one such variable, treat it as a constant: else,
inconsistency will surely arise, which can cause trouble later on.

70
71
72
Version numbers
---------------

73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
Note:

* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

Export the following environment variables:
90

91
92
93
94
* `VERSION`: the version you're preparing (see
  [[contribute/release_schedule#versioning]])
* `PREVIOUS_VERSION`: the last released version (which could be a release
  candidate, in case you're preparing a major release)
95
96
97
* `PREVIOUS_STABLE_VERSION`: the current stable release of Tails. This
  will be the same as `PREVIOUS_VERSION` except when preparing a major
  release.
98
* tags:
99

100
101
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
102

103
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
104
105
106
  *major* Tails release; if you're preparing a RC for a major release,
  use that major release; otherwise, use whatever the next planned
  major release is
107
108
* `SECOND_NEXT_PLANNED_MAJOR_VERSION`: if you're preparing the RC
  for a major release, set this to the version number of
109
110
111
112
  the second next *major* Tails release; e.g. if preparing the RC for
  the 3.9 major release, then set this to 3.12 (3.9 is the next major
  release, 3.10 and 3.11 are bugfix releases, 3.12 is a major
  release).
113
* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
114
  scheduled *bugfix* Tails release
115
116
* `NEXT_PLANNED_VERSION`: use whichever is the next scheduled release
   among `$NEXT_PLANNED_BUGFIX_VERSION` and `$NEXT_PLANNED_MAJOR_VERSION`
117
* `NEXT_POTENTIAL_EMERGENCY_VERSION`: set to the version number we'll give
118
119
  to the next emergency release if we have to put one out; unset for a
  release candidate
120
121
* `NEXT_STABLE_CHANGELOG_VERSION`: if `$NEXT_PLANNED_BUGFIX_VERSION` is the next
  scheduled release, use it; otherwise, use `$NEXT_POTENTIAL_EMERGENCY_VERSION`
122
123
124
125
126
127

Other variables
---------------

Also export the following environment variables:

128
129
* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
130
* `ISOS`: the directory where one stores `tails-amd64-*`
131
  sub-directories like the ones downloaded with BitTorrent.
132
133
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
134
135
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
intrigeri's avatar
intrigeri committed
136
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
137
138
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
139
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
140
* `TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}" | perl -nE 'say substr($_, -17)')`
141
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
142
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
143
* `export IUKS_DIR="${ISOS:?}/iuks/v2"`
144
* `export IUKS_HASHES="${IUKS_DIR:?}/to_${VERSION}.sha256sum"`
145
* `export MILESTONE=$(echo "${VERSION:?}" | sed -e 's,~.*,,')`
146

147
148
149
Pre-freeze
==========

150
151
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
152
153
154
155
156
157

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

158
159
160
161
162
163
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

164
165
166
167
168
169
Freeze
======

Major release
-------------

170
171
If we are at freeze time for a major release (i.e. preparing its
release candidate):
172
173
174

1. Merge the `master` Git branch into `devel`:

175
        git checkout devel && git fetch origin && git merge origin/devel && git merge --no-ff origin/master
176

intrigeri's avatar
intrigeri committed
177
178
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
179
   APT suite; set `BRANCH=testing` instead of the default `BRANCH=devel`.
180
181
182

3. Merge the `devel` Git branch into the `testing` one:

183
        git checkout testing && git merge origin/testing && git merge devel
184
185
186
187

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

188
189
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
190

anonym's avatar
anonym committed
191
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
192
193
194
195
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
196
   long enough, by bumping their `Valid-Until` to 10 days after the
197
   second next major release (the one _after_ the one you're preparing)'s
198
   scheduled date:
199
200
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

201

202
203
Bugfix release
--------------
204

205
If we are at freeze time for a bugfix release:
206
207
208

1. Merge the `master` Git branch into `stable`:

209
        git checkout stable && git fetch && git merge origin/stable && git merge --no-ff origin/master
210

intrigeri's avatar
intrigeri committed
211
212
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
213
214
   APT suite.

215
216
Common steps for bugfix and major releases
------------------------------------------
217

218
Reset the release branch's `config/base_branch`:
219

220
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
221
           git commit config/base_branch \
222
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
223

224
225
226
227
228
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
229
230
3. Send the pad URL in an **encrypted** mail to the manual testers:
   <tails-manual-testers@boum.org>
231

Tails developers's avatar
Tails developers committed
232
233
234
Update included files
=====================

235
236
<a id="upgrade-custom-debs"></a>

237
238
239
240
241
242
243
244
245
246
Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[tor-browser_AppArmor_patch]]

Bessemer's avatar
Bessemer committed
247
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
248
249
--------------------------------------

250
Skip this section if you are preparing a bugfix release.
251
252
253

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:
254

255
256
257
 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
258
 - If you are preparing a final major release, build at least the packages
259
260
261
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)
262

263
264
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
intrigeri's avatar
intrigeri committed
265
import translations from Transifex and sanity-check them:
266

267
268
269
	cd whisperback && \
	git checkout master && \
	git pull && \
intrigeri's avatar
intrigeri committed
270
	"${RELEASE_CHECKOUT:?}"/import-translations && \
271
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po
272

273
274
275
Then, for every PO file that has issues:

1. Rollback changes to that file: `git checkout po/LL.po`
276
2. Run `lint_po` again. It should pass this time.
277

278
And finally, commit:
279

280
    git add po && git commit \
281
	    -m "Update POT and PO files, pull updated translations from Transifex."
282

283
Then see the relevant release processes, and upload the packages to
284
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
285

286
* [[tails-installer]]
287
* whisperback:
288
  * follow [upstream release process](https://gitlab.tails.boum.org/tails/whisperback/-/blob/master/HACKING.md#release)
289
290
  * build a Debian package in an amd64 chroot of the Debian release
    the Tails version you're preparing is based on
Tails developers's avatar
Tails developers committed
291

292
293
294
295
296
Upgrade custom packages for VeraCrypt integration
-------------------------------------------------

See the dedicated page: [[veracrypt]]

297
298
Update PO files
---------------
299

300
301
302
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
303

304
	cd "${RELEASE_CHECKOUT:?}" && \
305
306
	./import-translations  && \
	./refresh-translations && \
307
	./submodules/jenkins-tools/slaves/lint_po && \
308
	git add po && git commit -m 'Update PO files.'
309

310
If `lint_po` complains:
311

312
* rollback the offending PO files and retry; worst case, delete it
313
* send a note to <tails-l10n@boum.org> [public] so that they get in touch with
314
315
  whoever can fix them.

316
317
When preparing an actual release
================================
318

319
If we're about to prepare the images for a final (non-RC) release, then
320
321
322
323
324
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
325
326
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
327
custom APT suite.
328

329
330
Bugfix release
--------------
331

332
<div class="note">
333
For bugfix releases, we generally do not put any RC out, so freeze time
334
335
336
337
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
338

intrigeri's avatar
intrigeri committed
339
340
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
341
custom APT suite.
342

anonym's avatar
anonym committed
343
344
Update other base branches
==========================
345
346

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
347
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
348

349
2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
350
   branch, the time-based APT repository snapshots being used
351
352
   during the freeze. It's fine if that results in a no-op
   (it depends on how exactly previous operations were performed).
intrigeri's avatar
intrigeri committed
353

354
3. Merge `devel` into `feature/bullseye` (if it exists), *without* following the instructions for
intrigeri's avatar
intrigeri committed
355
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
356
   (For now `feature/bullseye` is handled as any other topic branch
357
   forked off `devel`: its base branch is set to `devel`.)
358
359
360
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.
361

362
4. Ensure that the release, `devel` and `feature/bullseye` (if it exists) branches
363
364
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
365

366
5. Push the modified branches to Git:
367

368
        git push origin                          \
369
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
370
           $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
371
           devel:devel
372
373
374

Update more included files
==========================
375

376
377
Changelog
---------
Tails developers's avatar
Tails developers committed
378

379
Update the Changelog entry for the release you're preparing:
380

381
	git checkout "${RELEASE_BRANCH:?}" && \
382
	./bin/update-changelog --version "${MILESTONE:?}"
383

384
385
Then, gather other useful information from:

386
* the diff between the previous version's `.packages` file and the one
387
  from the to-be-released images; look for:
388
389
390
391
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
intrigeri's avatar
intrigeri committed
392
* the [[!tails_gitlab groups/tails/-/milestones desc="GitLab milestone"]].
393

394
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
395

396
397
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
398
	else
399
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
400
	fi
Tails developers's avatar
Tails developers committed
401

Bessemer's avatar
Bessemer committed
402
Included website
403
404
----------------

Tails developers's avatar
Tails developers committed
405
406
407
408
409
410
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

411
412
### version number

413
414
If preparing a RC, skip this part.

415
In the branch used to build the release, update the `wiki/src/inc/*` files to
416
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
417
at least 24 hours in the future! Between tests and mirror synchronization,
418
the build will not be released on the same day. Try to make sure it
419
matches the date of the future signature.
420

sajolida's avatar
sajolida committed
421
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
422

intrigeri's avatar
intrigeri committed
423
424
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html && \
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html && \
425
426
427
	for type in img iso; do
	   basename="tails-amd64-${VERSION:?}"
	   filename="${basename:?}.${type:?}"
428
	   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
429
430
	        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
	   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
intrigeri's avatar
intrigeri committed
431
	        > wiki/src/inc/stable_amd64_${type:?}_url.html && \
432
	   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
intrigeri's avatar
intrigeri committed
433
	        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html && \
434
435
	   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
	        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
intrigeri's avatar
intrigeri committed
436
437
	done && \
	./build-website --rebuild && \
438
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
439

440
441
442
443
444
Signing key downloaded by the Upgrader
--------------------------------------

    TMP_GNUPG_HOME=$(mktemp -d)
    gpg --homedir "${TMP_GNUPG_HOME:?}" --import wiki/src/tails-signing.key && \
445
446
447
    gpg --homedir "${TMP_GNUPG_HOME:?}" \
        --export-filter drop-subkey="revoked == 1" \
        --export-options export-minimal \
448
449
450
451
        --armor --export "${TAILS_SIGNATURE_KEY:?}" \
        > wiki/src/tails-signing-minimal.key && \
    git commit wiki/src/tails-signing-minimal.key \
        -m "Update signing key used by the Upgrader"
452
    rm -rf "${TMP_GNUPG_HOME:?}"
453

454
455
Website translations
--------------------
456

457
458
459
460
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
461

intrigeri's avatar
intrigeri committed
462
463
464
465
    ./build-website && \
    git add wiki/src && \
    git commit -m 'Update website PO files.'
    git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
466

467
468
469
Call for translation
====================

470
471
472
If at freeze time for a major release, send a call for translations to
<tails-l10n@boum.org> [public], making it clear what Git branch the
translations must be based on, and what are the priorities.
473

474
To get a list of changes on the website:
475

intrigeri's avatar
intrigeri committed
476
    git diff --stat ${PREVIOUS_TAG:?}.. -- \
477
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
478
479
480
481
482
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
483

484
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
485
486
======================

487
488
489
490
### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
491
pro tip is to never plug it unless prompted which `gpg` will do for you.
492
493
494
495
496
497

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
498

Tails developers's avatar
Tails developers committed
499
You should never import the Tails signing key into your own keyring,
500
501
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
502
503

    export GNUPGHOME=$(mktemp -d)
504
505
506
507
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
508
509
510
511
512
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

513
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
514

515
516
Build the almost-final images
=============================
517

518
519
520
521
522
1. [[Build ISO and USB images|contribute/build]] from the release branch,
   with `$TAILS_BUILD_OPTIONS` set like this:
   - Set `defaultcomp`, so we can more accurately optimize our
     SquashFS file ordering.
   - Do _not_ set `keeprunning` nor `rescue`.
523
2. Carefully read the build logs to make sure nothing bad happened.
524
3. Keep the resulting build artifacts until the end of this release process.
525
526
4. Record where the manifest of needed packages is stored:

527
528
        export BUILD_MANIFEST=XXX ; \
        [ -f "${BUILD_MANIFEST:?}" ] || echo "ERROR: BUILD_MANIFEST is incorrect"
529
530
        echo "${BUILD_MANIFEST:?}" | grep -E -qs '\.build-manifest$' \
           || echo "ERROR: BUILD_MANIFEST does not have the .build-manifest extension"
531

532
533
534
Tag the release in Git
======================

535
536
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
537
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
538
539
540
541
542
543
544

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

545
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
intrigeri's avatar
intrigeri committed
546
547
to continue if the last changelog entry is unreleased but corresponds to
an existing tag. There are workarounds we need to decide and implement.
548

549
550
551
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
552
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
553

554
* Prepare tagged snapshots of upstream APT repositories:
555

556
          ./bin/tag-apt-snapshots "${BUILD_MANIFEST:?}" "${TAG:?}"
557

558
  Note:
559

560
561
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
562
    [[custom APT repository|APT_repository/custom]] are
563
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
564
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
565
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
566
    Now, we have a "safety" net, in case you don't notice such a problem: if
567
568
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
569

Tails developers's avatar
Tails developers committed
570
571
572
Build images
============

573
574
575
Sanity check
------------

576
Verify that the Tor Browser release used in Tails still is the most
577
578
579
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:
580

581
* <https://gitweb.torproject.org/builders/tor-browser-build.git>
582
583
* <https://gitweb.torproject.org/tor-browser.git>

584
A new tag may indicate that a new Tor Browser release or rebuild is imminent.
585
586
587

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
588
589
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
590

591
1. Install the almost-final USB image to a USB stick.
592
593
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
594
1. Set up a wired connection to avoid having to deal with wireless settings.
595
1. Boot this USB stick **on bare metal** again.
596
1. Add `profile` to the kernel command-line.
597
1. Login with the default settings in the Welcome Screen (e.g. do not configure
598
   an _Administration Password_).
599
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
600
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
601
1. A few minutes later, once the `boot-profile` process has been
602
   killed, retrieve the new sort file from `/var/log/boot-profile`.
intrigeri's avatar
intrigeri committed
603
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
604
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
605
606
607
608
609
610
1. Remove runtime-generated files that don't exist in the rootfs,
   in order to avoid confusing noise in the build output:

           perl -ni -E 'chomp; say unless m{(?:
                  [.]pyc\s+\d+\z
               | \Alib/live/mount/medium/live/(?:filesystem[.]squashfs|initrd[.]img)\s
611
612
613
614
615
616
617
618
619
               | \Alib/live/mount/overlay/rw/etc/fstab\s
               | \Alib/live/mount/overlay/rw/etc/console-setup/cached_\S+[.](?:gz|sh)\s
               | \Alib/live/mount/overlay/rw/etc/machine-id\s
               | \Alib/live/mount/overlay/rw/etc/network/interfaces\s
               | \Alib/live/mount/overlay/rw/var/log/wtmp\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/apparmor[.]d/cache/[.]features\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/(?:group|gshadow|passwd|shadow)-\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/resolv-over-clearnet[.]conf\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/skel/[.]config/autostart/end-profile[.]desktop\s
620
               | \Alib/modules/.*/kernel/drivers/(?:cpufreq|net)/
621
               | \Arun/
intrigeri's avatar
intrigeri committed
622
               | \Avar/lib/AccountsService/users/Debian-gdm\s
623
               | \Avar/lib/gdm3/[#]\d+\s
624
625
626
627
628
               | \Avar/log/live/config[.]pipe\s
           )}xms' config/binary_rootfs/squashfs.sort

1. Remove the bits about `kill-boot-profile` at the end: they're
   only useful when profiling the boot.
intrigeri's avatar
intrigeri committed
629
1. Inspect the Git diff (including diff stat), apply common sense:
630

631
        diff -NaurB \
intrigeri's avatar
intrigeri committed
632
633
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
634
            | less
635

636
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`
637
1. Clean up: `rm -f config/binary_rootfs/squashfs.sort.old`
638

639
640
Build the final images
----------------------
641

Bessemer's avatar
Bessemer committed
642
Then all included files should be up-to-date and the versioned APT
643
644
suite should be ready, so it is time to:

645
1. Mark the version as "released" in the changelog:
646

647
        dch --release --no-force-save-on-release --maintmaint && \
648
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
649

650
1. Export `SOURCE_DATE_EPOCH`:
651

652
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
653

654
1. tag the release *again*, with all included files in:
655

656
657
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
658
659
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
660

661
662
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
663
   commands by e.g. committing a commit in between `git tag` and the
664
665
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
666

667
1. build the final images!
668
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
669
670
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.
671

672
673
674
675
1. Make sure the Jenkins build starts. Until the hook is back in place
   ([[!tails_ticket 17745]]), starting it manually may avoid up to 15
   minutes of waiting.

intrigeri's avatar
intrigeri committed
676
1. Compare the new build manifest with the one from the previous,
677
   almost-final build:
intrigeri's avatar
intrigeri committed
678
679

        diff -Naur \
680
           "${BUILD_MANIFEST:?}" \
681
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
intrigeri's avatar
intrigeri committed
682
683
684
685

   They should be identical, except that the `debian-security` serial might be higher.

1. To ensure we publish the final build's `.build-manifest`, run:
686

687
        export BUILD_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
688

intrigeri's avatar
intrigeri committed
689
<a id="reproducibility-sanity-check-iso"></a>
690

intrigeri's avatar
intrigeri committed
691
692
693
694
Verify that Jenkins reproduced your images
------------------------------------------

to verify that Jenkins reproduced your images:
695

intrigeri's avatar
intrigeri committed
696
1. Visit the URL printed by this command:
anonym's avatar
xxx    
anonym committed
697
698
699

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

intrigeri's avatar
intrigeri committed
700
2. Find the job (probably the last one)
701
702
   and make sure the ISO and USB images built by Jenkins
   have the same hash (in the `.shasum` file) as the images you built.
703

intrigeri's avatar
intrigeri committed
704
3. Then:
705

706
707
   - If the ISO and USB images hashes match: yay, we're good to go!
     The `.build-manifest` may differ — that's OK.
708

709
     Set the `$MATCHING_JENKINS_IMAGES_BUILD_ID` environment variable
710
711
     to the ID of this job (an integer).

712
   - If there is a hash mismatch for one of the images: ouch! Now we are in a
713
714
715
716
717
718
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
719

anonym's avatar
anonym committed
720
721
722
723
724
725
726
727
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
728

729
730
     Do the same for the USB image as well.

731
     Then carefully investigate the `diffoscope` report:
732

733
734
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
intrigeri's avatar
intrigeri committed
735
         release. Halt the release, involve the rest of <tails@boum.org>, and then
736
737
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
738

739
       - Otherwise, if the change is definitely harmless:
740

741
         * If the source of non-determinism is identified quickly and
742
           is easy and fast to fix, *and* the QA of the current images
743
744
745
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
746

747
748
749
750
751
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
752

753
754
755
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
intrigeri's avatar
intrigeri committed
756
           release notes, linking to the issue where
757
758
           the nature of the reproducibility failure is clearly
           described.
759

intrigeri's avatar
intrigeri committed
760
761
Initialize the website release branch
-------------------------------------
intrigeri's avatar
intrigeri committed
762

763
764
765
766
767
768
From now on, we don't want to push new commits on `$RELEASE_BRANCH`
until the new release is out. Otherwise, this would break its build
and the build of every branch based on it, which would effectively
block other development work. So the final steps towards publishing
the release will be done in a new, dedicated branch.

intrigeri's avatar
intrigeri committed
769
770
If preparing anything but a final release (e.g. an alpha, beta
or RC):
771
772
773
774

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

intrigeri's avatar
intrigeri committed
775
Else, if preparing a final release:
776

777
778
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
779

780
781
782
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
783
Create a directory with a suitable name, go there, move the built
784
785
786
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
intrigeri's avatar
intrigeri committed
787
788
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
789

790
791
792
793
794
795
796
    mkdir -p "${ISOS:?}/tails-amd64-${VERSION:?}" && \
    for type in iso img ; do
       cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
       mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
       gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
       rename 's,\.asc$,.sig,' *.asc && \
       tmp="$(mktemp -d)" && \
797
798
       mkdir -p "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       cd "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
799
       for x in "${ISOS:?}/tails-amd64-${VERSION:?}"/*.${type:?}*; do
800
801
802
803
804
805
           ln -s ${x} .
       done && \
       mktorrent \
          -o "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" \
          -a 'udp://tracker.torrent.eu.org:451'   \
          -a 'udp://tracker.coppersurfer.tk:6969' \
806
          "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
807
       transmission-show "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" && \
808
       cd - && \
809
810
       rm -rf "${tmp:?}"
    done
811
812
    mv "${ARTIFACTS:?}"/tails-amd64-${VERSION:?}.{apt-sources,build-manifest,buildlog,packages} \
       "${ISOS:?}/tails-amd64-${VERSION:?}"
813

814
815
816
817
818
Due to various directory changes, one needs to manually go back to the
desired directory, likely using:

    cd ${RELEASE_CHECKOUT?:}

anonym's avatar
anonym committed
819
820
Lastly, let's set some variables to be used later:

821
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
822
823
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
824
    IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.img"
825
826
    IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"
anonym's avatar
anonym committed
827

828
829
<a id="prepare-iuk"></a>

830
831
832
Prepare incremental upgrades
============================

833
834
835
836
Since Tails 4.2, we use a new upgrade scheme, which fundamentally
changes what the source version number of an upgrade means: it's now
the version that was *initially installed* and *not* the currently
running version. If this is news to you, see:
837

838
839
* the document that explains the benefits for our users:
  [[blueprint/Endless_upgrades]];
840

841
842
* the corresponding
  [[design documentation|contribute/design/incremental_upgrades]].
843

segfault's avatar
segfault committed
844
The main practical implications at release time are:
845

846
847
848
* The Release Manager has to publish more IUKs than they used to.
  But they can now publish IUKs (reproducibly) built on Jenkins,
  instead of having to upload those they've built locally.
849

850
* The Release Manager has to sign more UDFs than they used to.
851

852
853
854
855
856
857
858
859
860
861
* Computing `$IUK_SOURCE_VERSIONS` is now straightforward enough
  that it was automated :)

Prepare the environment
-----------------------

Compute the list of initial version install to build IUKs for:

    cd "${RELEASE_CHECKOUT:?}" && \
	export IUK_SOURCE_VERSIONS=$(./bin/iuk-source-versions ${VERSION:?})
862
    echo "${IUK_SOURCE_VERSIONS?:}"
863

864
865
866
Even if it's computed automatically, remember to store it in the file
holding environment variables, it will be used in various places below.

867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
Sanity checks
-------------

Check that you have the correct version of `squashfs-tools` installed:

    [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
      = '1:4.4-1+0.tails1' \
    ] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for <https://iso-history.tails.boum.org> (it
will be used by `wrap_tails_create_iuks` below, in order to download any ISO
that is not present locally yet):

    if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
        echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
        echo "The corresponding login and password are in the RMs' keyringer."
    fi

885
886
Build the Incremental Upgrade Kits locally
------------------------------------------
887

888
889
890
891
892
893
894
895
896
You're encouraged to enable parallelism to avoid waiting for a very
long, serial build (which is still the default at the moment). As
discussed in [[!tails_ticket 17657]], it seems running as many jobs as
there are physical cores is a nice rule of thumb.

For example, set:

    JOBS="--jobs 4"

anonym's avatar
anonym committed
897
898
899
900
or, attempt to automatically set it to the number of physical cores:

    JOBS="--jobs $(grep '^core id' /proc/cpuinfo | sort -u | wc -l)"

901
902
before starting the wrapper from `puppet-tails`:

903
904
905
    (
       set -eu
       WORK_DIR=$(mktemp -d)
906
       TAILS_REMOTE="$(git -C "${RELEASE_CHECKOUT?:}" remote get-url origin)"
907
       PUPPET_TAILS_REMOTE=$(echo -n "${TAILS_REMOTE?:}" | perl -p -E 's,:tails/tails(?:[.]git)?\z,:tails/puppet-tails,')
908
       cd "${WORK_DIR?:}"
909
       git clone "$PUPPET_TAILS_REMOTE"
910
       sudo                                                                   \
911
       time                                                                   \
912
       ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
913
914
915
916
           --tails-git-remote "file://${RELEASE_CHECKOUT?:}/.git"             \
           --tails-git-commit "${TAG?:}"                                      \
           --source-date-epoch "${SOURCE_DATE_EPOCH?:}"                       \
           --local-isos-dir "${ISOS?:}"                                       \
917
           --tmp-dir "${TMPDIR:-/tmp}"                                        \
918
919
920
           --output-dir "${IUKS_DIR?:}"                                       \
           --source-versions "${IUK_SOURCE_VERSIONS?:}"                       \
           --new-version "${VERSION?:}"                                       \
921
           --verbose ${JOBS:-}
922
923
       cd "${IUKS_DIR?:}"
       sha256sum Tails_amd64_*_to_${VERSION?:}.iuk > "${IUKS_HASHES?:}"
924
    )
925

926
This command takes a long time. In parallel, while it is running,
927
you can follow the next step:
928
929
930
931
932
933
934
935
936
937

 - Build the Incremental Upgrade Kits on Jenkins

ISO history
-----------

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

938
 - The Jenkins `parallel_build_IUKs` job can fetch them.
intrigeri's avatar
intrigeri committed
939
 - Our isotesters can fetch them from there for their testing.
940

941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
Make sure that Jenkins has finished building so you have set
`MATCHING_JENKINS_IMAGES_BUILD_ID` (see above), then run:

    ssh misc.lizard mkdir tails-amd64-${VERSION:?} && \
    scp "${ISOS:?}"/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.{apt-sources,build-manifest,buildlog,packages,iso.sig,img.sig} \
        misc.lizard:tails-amd64-${VERSION:?} && \
    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh misc.lizard gpg --import && \
    ssh misc.lizard << EOF && \
       cd tails-amd64-${VERSION:?} && \
       wget --quiet \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_IMAGES_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_IMAGES_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.img" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,} && \
       gpg --verify tails-amd64-${VERSION:?}.img{.sig,}
    EOF
    ssh misc.lizard << EOF
       ( [ -d isos ] || git clone gitolite@puppet-git.lizard:isos.git ) && \
       cd isos && \
960
       git annex init && \
961
962
963
964
965
966
967
968
969
       git annex sync && \
       git annex import ../tails-amd64-${VERSION:?} && \
       rmdir ../tails-amd64-${VERSION:?} && \
       git commit -m "Add Tails ${VERSION:?}" && \
       git annex sync && \
       git annex copy tails-amd64-${VERSION:?} --to origin && \
       git annex drop tails-amd64-${VERSION:?} && \
       git annex sync
    EOF
970

971
Then, wait (a few minutes, */15 crontab) until the images appear
972
on <https://iso-history.tails.boum.org/>.
973
974
975
976

<a id="build-iuks-on-jenkins"></a>

Build the Incremental Upgrade Kits on Jenkins
977
978
979
980
981
---------------------------------------------

1. Make sure the push to ISO history (started in the previous section)
   has finished, and that images have appeared on the web server:
   <https://iso-history.tails.boum.org/>
982

983
984
985
986
987
1. On <https://jenkins.tails.boum.org/job/parallel_build_IUKs/configure>, adjust
   the `SOURCE_VERSION` axis to list all versions in `$IUK_SOURCE_VERSIONS`,
   and save the updated configuration.

1. On <https://jenkins.tails.boum.org/job/parallel_build_IUKs/build?delay=0sec>,
988
989
990
   fill the form with these values:

    - `TAILS_GIT_COMMIT`: the value of `$TAG` in your release environment
991
    - `SOURCE_DATE_EPOCH`: the value of `$SOURCE_DATE_EPOCH` in your
992
993
994
995
996
997
998
999
1000
      release environment
    - `NEW_VERSION`: the value of `$VERSION` in your release environment
    - `EXTRA_ARGS`: leave it blank

2. Click the _Build_ button

3. After a few seconds, a new build appears on top of the _Build
   History_ sidebar. Click on the progress bar of this new build.

1001
1002
1003
3. **Verify:** A downstream `parallel_collect_IUKs` job should appear,
   to be triggered once the `parallel_build_IUKs` has completed successfully.

1004
4. Set the `$CANDIDATE_JENKINS_IUKS_BUILD_ID` environment variable
1005
   to the ID of that downstream job (an integer).
1006

1007
5. Wait until both `parallel_build_IUKs` and `parallel_collect_IUKs` jobs complete successfully.
1008
   It should take about 10-15 minutes for each member of