release_process.mdwn 77.5 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
<div class="caution">
8
9
Read the remainder of this document from the branch used to prepare the release
after having merged the current master branch into it!
10
11
</div>

12
13
14
Requirements
============

15
16
17
Packages
--------

18
19
To release Tails you'll need some packages installed:

20
* `docker.io gitlab-cli jq tidy mktorrent python3-debian python3-gitlab python3-jinja2 transmission-cli`
21
22
* [[!debpts squashfs-tools]] 1:4.4-1+0.tails1
  from our custom `iukbuilder-stretch` APT suite.
23
24
* `iuk` [[dependencies|contribute/release_process/tails-iuk#build-deps]]
* `perl5lib` [[dependencies|contribute/release_process/perl5lib#build-deps]]
25
* `po4a` 0.55: different versions extract Markdown headings
26
   in a different way, which makes tons of strings fuzzy.
27
* packages to [[build a local version of the website|contribute/build/website/]]
28

29
30
31
32
33
34
35
36
37
38
39
40
Configuration files
-------------------

To release Tails you need:

* `~/.python-gitlab.cfg`

  You need at least this content:

        [global]
        ssl_verify = true

41
        [TailsRM]
42
43
44
45
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

46
47
48
49
50
51
52
53
54
55
56
57
58
        [Tails]
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

   Then:

   - In the `TailsRM` section, set the value of the `private_token` option to
     the `role-release-automation` GitLab user's API token, which you'll find in
     `rm.git`'s keyringer.

   - In the `Tails` section, set the value of the `private_token` option to a
     GitLab API token for your own user.
59

60
61
62
Environment
===========

63
64
65
66
67
68
69
To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the
value of one such variable, treat it as a constant: else,
inconsistency will surely arise, which can cause trouble later on.

70
71
72
Version numbers
---------------

73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
Note:

* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

Export the following environment variables:
90

91
92
93
94
95
* `VERSION`: the version you're preparing (see
  [[contribute/release_schedule#versioning]])
* `PREVIOUS_VERSION`: the last released version (which could be a release
  candidate, in case you're preparing a major release)
* tags:
96

97
98
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
99

100
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
101
102
103
  *major* Tails release; if you're preparing a RC for a major release,
  use that major release; otherwise, use whatever the next planned
  major release is
104
105
* `SECOND_NEXT_PLANNED_MAJOR_VERSION`: if you're preparing the RC
  for a major release, set this to the version number of
106
107
108
109
  the second next *major* Tails release; e.g. if preparing the RC for
  the 3.9 major release, then set this to 3.12 (3.9 is the next major
  release, 3.10 and 3.11 are bugfix releases, 3.12 is a major
  release).
110
* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
111
  scheduled *bugfix* Tails release
112
113
* `NEXT_PLANNED_VERSION`: use whichever is the next scheduled release
   among `$NEXT_PLANNED_BUGFIX_VERSION` and `$NEXT_PLANNED_MAJOR_VERSION`
114
* `NEXT_POTENTIAL_EMERGENCY_VERSION`: set to the version number we'll give
115
116
  to the next emergency release if we have to put one out; unset for a
  release candidate
117
118
* `NEXT_STABLE_CHANGELOG_VERSION`: if `$NEXT_PLANNED_BUGFIX_VERSION` is the next
  scheduled release, use it; otherwise, use `$NEXT_POTENTIAL_EMERGENCY_VERSION`
119
120
121
122
123
124

Other variables
---------------

Also export the following environment variables:

125
126
* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
127
* `ISOS`: the directory where one stores `tails-amd64-*`
128
  sub-directories like the ones downloaded with BitTorrent.
129
130
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
131
132
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
intrigeri's avatar
intrigeri committed
133
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
134
135
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
136
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
137
* `TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}" | perl -nE 'say substr($_, -17)')`
138
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
139
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
140
* `export IUKS_DIR="${ISOS:?}/iuks/v2"`
141
* `export IUKS_HASHES="${IUKS_DIR:?}/to_${VERSION}.sha256sum"`
142
* `export MILESTONE=$(echo "${VERSION:?}" | sed -e 's,~.*,,')`
143

144
145
146
Pre-freeze
==========

147
148
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
149
150
151
152
153
154

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

155
156
157
158
159
160
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

161
162
163
164
165
166
Freeze
======

Major release
-------------

167
168
If we are at freeze time for a major release (i.e. preparing its
release candidate):
169
170
171

1. Merge the `master` Git branch into `devel`:

172
        git checkout devel && git fetch origin && git merge origin/devel && git merge --no-ff origin/master
173

intrigeri's avatar
intrigeri committed
174
175
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
176
   APT suite; set `BRANCH=testing` instead of the default `BRANCH=devel`.
177
178
179

3. Merge the `devel` Git branch into the `testing` one:

180
        git checkout testing && git merge origin/testing && git merge devel
181
182
183
184

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

185
186
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
187

anonym's avatar
anonym committed
188
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
189
190
191
192
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
193
   long enough, by bumping their `Valid-Until` to 10 days after the
194
   second next major release (the one _after_ the one you're preparing)'s
195
   scheduled date:
196
197
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

198

199
200
Bugfix release
--------------
201

202
If we are at freeze time for a bugfix release:
203
204
205

1. Merge the `master` Git branch into `stable`:

206
        git checkout stable && git fetch && git merge origin/stable && git merge --no-ff origin/master
207

intrigeri's avatar
intrigeri committed
208
209
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
210
211
   APT suite.

212
213
Common steps for bugfix and major releases
------------------------------------------
214

215
Reset the release branch's `config/base_branch`:
216

217
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
218
           git commit config/base_branch \
219
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
220

221
222
223
224
225
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
226
227
3. Send the pad URL in an **encrypted** mail to the manual testers:
   <tails-manual-testers@boum.org>
228

Tails developers's avatar
Tails developers committed
229
230
231
Update included files
=====================

232
233
<a id="upgrade-custom-debs"></a>

234
235
236
237
238
239
240
241
242
243
Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[tor-browser_AppArmor_patch]]

Bessemer's avatar
Bessemer committed
244
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
245
246
--------------------------------------

247
Skip this section if you are preparing a bugfix release.
248
249
250

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:
251

252
253
254
 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
255
 - If you are preparing a final major release, build at least the packages
256
257
258
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)
259

260
261
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
intrigeri's avatar
intrigeri committed
262
import translations from Transifex and sanity-check them:
263

264
265
266
	cd whisperback && \
	git checkout master && \
	git pull && \
intrigeri's avatar
intrigeri committed
267
	"${RELEASE_CHECKOUT:?}"/import-translations && \
268
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po
269

270
271
272
Then, for every PO file that has issues:

1. Rollback changes to that file: `git checkout po/LL.po`
273
2. Run `lint_po` again. It should pass this time.
274

275
And finally, commit:
276

277
    git add po && git commit \
278
	    -m "Update POT and PO files, pull updated translations from Transifex."
279

280
Then see the relevant release processes, and upload the packages to
281
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
282

283
* [[tails-installer]]
284
* whisperback:
285
  * follow [upstream release process](https://gitlab.tails.boum.org/tails/whisperback/-/blob/master/HACKING.md#release)
286
287
  * build a Debian package in an amd64 chroot of the Debian release
    the Tails version you're preparing is based on
Tails developers's avatar
Tails developers committed
288

289
290
291
292
293
Upgrade custom packages for VeraCrypt integration
-------------------------------------------------

See the dedicated page: [[veracrypt]]

294
295
Update PO files
---------------
296

297
298
299
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
300

301
	cd "${RELEASE_CHECKOUT:?}" && \
302
303
	./import-translations  && \
	./refresh-translations && \
304
	./submodules/jenkins-tools/slaves/lint_po && \
305
	git add po && git commit -m 'Update PO files.'
306

307
If `lint_po` complains:
308

309
* rollback the offending PO files and retry; worst case, delete it
310
* send a note to <tails-l10n@boum.org> [public] so that they get in touch with
311
312
  whoever can fix them.

313
314
When preparing an actual release
================================
315

316
If we're about to prepare the images for a final (non-RC) release, then
317
318
319
320
321
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
322
323
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
324
custom APT suite.
325

326
327
Bugfix release
--------------
328

329
<div class="note">
330
For bugfix releases, we generally do not put any RC out, so freeze time
331
332
333
334
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
335

intrigeri's avatar
intrigeri committed
336
337
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
338
custom APT suite.
339

anonym's avatar
anonym committed
340
341
Update other base branches
==========================
342
343

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
344
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
345

346
2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
347
   branch, the time-based APT repository snapshots being used
348
349
   during the freeze. It's fine if that results in a no-op
   (it depends on how exactly previous operations were performed).
intrigeri's avatar
intrigeri committed
350

351
3. Merge `devel` into `feature/bullseye` (if it exists), *without* following the instructions for
intrigeri's avatar
intrigeri committed
352
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
353
   (For now `feature/bullseye` is handled as any other topic branch
354
   forked off `devel`: its base branch is set to `devel`.)
355
356
357
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.
358

359
4. Ensure that the release, `devel` and `feature/bullseye` (if it exists) branches
360
361
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
362

363
5. Push the modified branches to Git:
364

365
        git push origin                          \
366
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
367
           $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
368
           devel:devel
369
370
371

Update more included files
==========================
372

373
374
Changelog
---------
Tails developers's avatar
Tails developers committed
375

376
Update the Changelog entry for the release you're preparing:
377

378
	git checkout "${RELEASE_BRANCH:?}" && \
379
	./bin/update-changelog --version "${MILESTONE:?}"
380

381
382
Then, gather other useful information from:

383
* the diff between the previous version's `.packages` file and the one
384
  from the to-be-released images; look for:
385
386
387
388
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
intrigeri's avatar
intrigeri committed
389
* the [[!tails_gitlab groups/tails/-/milestones desc="GitLab milestone"]].
390

391
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
392

393
394
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
395
	else
396
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
397
	fi
Tails developers's avatar
Tails developers committed
398

Bessemer's avatar
Bessemer committed
399
Included website
400
401
----------------

Tails developers's avatar
Tails developers committed
402
403
404
405
406
407
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

408
409
### version number

410
411
If preparing a RC, skip this part.

412
In the branch used to build the release, update the `wiki/src/inc/*` files to
413
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
414
at least 24 hours in the future! Between tests and mirror synchronization,
415
the build will not be released on the same day. Try to make sure it
416
matches the date of the future signature.
417

sajolida's avatar
sajolida committed
418
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
419

intrigeri's avatar
intrigeri committed
420
421
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html && \
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html && \
422
423
424
	for type in img iso; do
	   basename="tails-amd64-${VERSION:?}"
	   filename="${basename:?}.${type:?}"
425
	   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
426
427
	        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
	   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
intrigeri's avatar
intrigeri committed
428
	        > wiki/src/inc/stable_amd64_${type:?}_url.html && \
429
	   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
intrigeri's avatar
intrigeri committed
430
	        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html && \
431
432
	   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
	        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
intrigeri's avatar
intrigeri committed
433
434
	done && \
	./build-website --rebuild && \
435
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
436

437
438
439
440
441
Signing key downloaded by the Upgrader
--------------------------------------

    TMP_GNUPG_HOME=$(mktemp -d)
    gpg --homedir "${TMP_GNUPG_HOME:?}" --import wiki/src/tails-signing.key && \
442
443
444
    gpg --homedir "${TMP_GNUPG_HOME:?}" \
        --export-filter drop-subkey="revoked == 1" \
        --export-options export-minimal \
445
446
447
448
        --armor --export "${TAILS_SIGNATURE_KEY:?}" \
        > wiki/src/tails-signing-minimal.key && \
    git commit wiki/src/tails-signing-minimal.key \
        -m "Update signing key used by the Upgrader"
449
    rm -rf "${TMP_GNUPG_HOME:?}"
450

451
452
Website translations
--------------------
453

454
455
456
457
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
458

intrigeri's avatar
intrigeri committed
459
460
461
462
    ./build-website && \
    git add wiki/src && \
    git commit -m 'Update website PO files.'
    git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
463

464
465
466
Call for translation
====================

467
468
469
If at freeze time for a major release, send a call for translations to
<tails-l10n@boum.org> [public], making it clear what Git branch the
translations must be based on, and what are the priorities.
470

471
To get a list of changes on the website:
472

intrigeri's avatar
intrigeri committed
473
    git diff --stat ${PREVIOUS_TAG:?}.. -- \
474
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
475
476
477
478
479
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
480

481
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
482
483
======================

484
485
486
487
### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
488
pro tip is to never plug it unless prompted which `gpg` will do for you.
489
490
491
492
493
494

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
495

Tails developers's avatar
Tails developers committed
496
You should never import the Tails signing key into your own keyring,
497
498
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
499
500

    export GNUPGHOME=$(mktemp -d)
501
502
503
504
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
505
506
507
508
509
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

510
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
511

512
513
Build the almost-final images
=============================
514

515
516
517
518
519
1. [[Build ISO and USB images|contribute/build]] from the release branch,
   with `$TAILS_BUILD_OPTIONS` set like this:
   - Set `defaultcomp`, so we can more accurately optimize our
     SquashFS file ordering.
   - Do _not_ set `keeprunning` nor `rescue`.
520
2. Carefully read the build logs to make sure nothing bad happened.
521
3. Keep the resulting build artifacts until the end of this release process.
522
523
4. Record where the manifest of needed packages is stored:

524
525
        export BUILD_MANIFEST=XXX ; \
        [ -f "${BUILD_MANIFEST:?}" ] || echo "ERROR: BUILD_MANIFEST is incorrect"
526
527
        echo "${BUILD_MANIFEST:?}" | grep -E -qs '\.build-manifest$' \
           || echo "ERROR: BUILD_MANIFEST does not have the .build-manifest extension"
528

529
530
531
Tag the release in Git
======================

532
533
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
534
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
535
536
537
538
539
540
541

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

542
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
intrigeri's avatar
intrigeri committed
543
544
to continue if the last changelog entry is unreleased but corresponds to
an existing tag. There are workarounds we need to decide and implement.
545

546
547
548
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
549
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
550

551
* Prepare tagged snapshots of upstream APT repositories:
552

553
          ./bin/tag-apt-snapshots "${BUILD_MANIFEST:?}" "${TAG:?}"
554

555
  Note:
556

557
558
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
559
    [[custom APT repository|APT_repository/custom]] are
560
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
561
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
562
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
563
    Now, we have a "safety" net, in case you don't notice such a problem: if
564
565
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
566

Tails developers's avatar
Tails developers committed
567
568
569
Build images
============

570
571
572
Sanity check
------------

573
Verify that the Tor Browser release used in Tails still is the most
574
575
576
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:
577

578
* <https://gitweb.torproject.org/builders/tor-browser-build.git>
579
580
* <https://gitweb.torproject.org/tor-browser.git>

581
A new tag may indicate that a new Tor Browser release or rebuild is imminent.
582
583
584

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
585
586
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
587

588
1. Install the almost-final USB image to a USB stick.
589
590
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
591
1. Set up a wired connection to avoid having to deal with wireless settings.
592
1. Boot this USB stick **on bare metal** again.
593
1. Add `profile` to the kernel command-line.
594
1. Login with the default settings in the Welcome Screen (e.g. do not configure
595
   an _Administration Password_).
596
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
597
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
598
1. A few minutes later, once the `boot-profile` process has been
599
   killed, retrieve the new sort file from `/var/log/boot-profile`.
intrigeri's avatar
intrigeri committed
600
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
601
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
602
603
604
605
606
607
1. Remove runtime-generated files that don't exist in the rootfs,
   in order to avoid confusing noise in the build output:

           perl -ni -E 'chomp; say unless m{(?:
                  [.]pyc\s+\d+\z
               | \Alib/live/mount/medium/live/(?:filesystem[.]squashfs|initrd[.]img)\s
608
609
610
611
612
613
614
615
616
               | \Alib/live/mount/overlay/rw/etc/fstab\s
               | \Alib/live/mount/overlay/rw/etc/console-setup/cached_\S+[.](?:gz|sh)\s
               | \Alib/live/mount/overlay/rw/etc/machine-id\s
               | \Alib/live/mount/overlay/rw/etc/network/interfaces\s
               | \Alib/live/mount/overlay/rw/var/log/wtmp\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/apparmor[.]d/cache/[.]features\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/(?:group|gshadow|passwd|shadow)-\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/resolv-over-clearnet[.]conf\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/skel/[.]config/autostart/end-profile[.]desktop\s
617
               | \Alib/modules/.*/kernel/drivers/(?:cpufreq|net)/
618
               | \Arun/
intrigeri's avatar
intrigeri committed
619
               | \Avar/lib/AccountsService/users/Debian-gdm\s
620
               | \Avar/lib/gdm3/[#]\d+\s
621
622
623
624
625
               | \Avar/log/live/config[.]pipe\s
           )}xms' config/binary_rootfs/squashfs.sort

1. Remove the bits about `kill-boot-profile` at the end: they're
   only useful when profiling the boot.
626
   XXX: automate during next release process ([[!tails_ticket 16467]])
intrigeri's avatar
intrigeri committed
627
1. Inspect the Git diff (including diff stat), apply common sense:
628

629
        diff -NaurB \
intrigeri's avatar
intrigeri committed
630
631
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
632
            | less
633

634
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`
635
1. Clean up: `rm -f config/binary_rootfs/squashfs.sort.old`
636

637
638
Build the final images
----------------------
639

Bessemer's avatar
Bessemer committed
640
Then all included files should be up-to-date and the versioned APT
641
642
suite should be ready, so it is time to:

643
1. Mark the version as "released" in the changelog:
644

645
        dch --release --no-force-save-on-release --maintmaint && \
646
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
647

648
1. Export `SOURCE_DATE_EPOCH`:
649

650
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
651

652
1. tag the release *again*, with all included files in:
653

654
655
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
656
657
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
658

659
660
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
661
   commands by e.g. committing a commit in between `git tag` and the
662
663
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
664

665
1. build the final images!
666
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
667
668
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.
669

670
671
672
673
1. Make sure the Jenkins build starts. Until the hook is back in place
   ([[!tails_ticket 17745]]), starting it manually may avoid up to 15
   minutes of waiting.

intrigeri's avatar
intrigeri committed
674
1. Compare the new build manifest with the one from the previous,
675
   almost-final build:
intrigeri's avatar
intrigeri committed
676
677

        diff -Naur \
678
           "${BUILD_MANIFEST:?}" \
679
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
intrigeri's avatar
intrigeri committed
680
681
682
683

   They should be identical, except that the `debian-security` serial might be higher.

1. To ensure we publish the final build's `.build-manifest`, run:
684

685
        export BUILD_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
686

intrigeri's avatar
intrigeri committed
687
<a id="reproducibility-sanity-check-iso"></a>
688

intrigeri's avatar
intrigeri committed
689
690
691
692
Verify that Jenkins reproduced your images
------------------------------------------

to verify that Jenkins reproduced your images:
693

intrigeri's avatar
intrigeri committed
694
1. Visit the URL printed by this command:
anonym's avatar
xxx    
anonym committed
695
696
697

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

intrigeri's avatar
intrigeri committed
698
2. Find the job (probably the last one)
699
700
   and make sure the ISO and USB images built by Jenkins
   have the same hash (in the `.shasum` file) as the images you built.
701

intrigeri's avatar
intrigeri committed
702
3. Then:
703

704
705
   - If the ISO and USB images hashes match: yay, we're good to go!
     The `.build-manifest` may differ — that's OK.
706

707
     Set the `$MATCHING_JENKINS_IMAGES_BUILD_ID` environment variable
708
709
     to the ID of this job (an integer).

710
   - If there is a hash mismatch for one of the images: ouch! Now we are in a
711
712
713
714
715
716
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
717

anonym's avatar
anonym committed
718
719
720
721
722
723
724
725
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
726

727
728
     Do the same for the USB image as well.

729
     Then carefully investigate the `diffoscope` report:
730

731
732
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
intrigeri's avatar
intrigeri committed
733
         release. Halt the release, involve the rest of <tails@boum.org>, and then
734
735
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
736

737
       - Otherwise, if the change is definitely harmless:
738

739
         * If the source of non-determinism is identified quickly and
740
           is easy and fast to fix, *and* the QA of the current images
741
742
743
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
744

745
746
747
748
749
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
750

751
752
753
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
intrigeri's avatar
intrigeri committed
754
           release notes, linking to the issue where
755
756
           the nature of the reproducibility failure is clearly
           described.
757

intrigeri's avatar
intrigeri committed
758
759
Initialize the website release branch
-------------------------------------
intrigeri's avatar
intrigeri committed
760

761
762
763
764
765
766
From now on, we don't want to push new commits on `$RELEASE_BRANCH`
until the new release is out. Otherwise, this would break its build
and the build of every branch based on it, which would effectively
block other development work. So the final steps towards publishing
the release will be done in a new, dedicated branch.

intrigeri's avatar
intrigeri committed
767
768
If preparing anything but a final release (e.g. an alpha, beta
or RC):
769
770
771
772

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

intrigeri's avatar
intrigeri committed
773
Else, if preparing a final release:
774

775
776
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
777

778
779
780
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
781
Create a directory with a suitable name, go there, move the built
782
783
784
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
intrigeri's avatar
intrigeri committed
785
786
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
787

788
789
790
791
792
793
794
    mkdir -p "${ISOS:?}/tails-amd64-${VERSION:?}" && \
    for type in iso img ; do
       cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
       mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
       gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
       rename 's,\.asc$,.sig,' *.asc && \
       tmp="$(mktemp -d)" && \
795
796
       mkdir -p "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       cd "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
797
       for x in "${ISOS:?}/tails-amd64-${VERSION:?}"/*.${type:?}*; do
798
799
800
801
802
803
           ln -s ${x} .
       done && \
       mktorrent \
          -o "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" \
          -a 'udp://tracker.torrent.eu.org:451'   \
          -a 'udp://tracker.coppersurfer.tk:6969' \
804
          "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
805
       transmission-show "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" && \
806
       cd - && \
807
808
       rm -rf "${tmp:?}"
    done
809

810
811
812
813
814
Due to various directory changes, one needs to manually go back to the
desired directory, likely using:

    cd ${RELEASE_CHECKOUT?:}

anonym's avatar
anonym committed
815
816
Lastly, let's set some variables to be used later:

817
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
818
819
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
820
    IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.img"
821
822
    IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"
anonym's avatar
anonym committed
823

824
825
826
827
828
829
830
Push images to ISO history
==========================

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

831
 - The Jenkins `parallel_build_IUKs` job can fetch them.
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
 - Our isotesters can fetch them from there for their testing.

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

If the release manager has `$ISOS` pointing to a checkout of the ISO
history repository, the following commands might be useful. For the
record, images and their signatures are already in that directory, and
one just needs to ship some metadata along:

    (cd ${ISOS?:}/tails-amd64-${VERSION?:} && \
    cp "${ARTIFACTS:?}"/tails-amd64-${VERSION:?}{.buildlog,.build-manifest,.packages} . && \
    git annex sync && \
    git annex add . && \
    git commit -m "Add Tails ${VERSION?:} images and metadata." && \
    git annex sync && \
    git annex copy --to origin .)

This transfer can take a while, and it's possible to proceed with the
next section in the meanwhile, up to the following step (included):

 - Build the Incremental Upgrade Kits locally

However, the following step will require those files being transferred
and made available on the relevant web server:

 - Build the Incremental Upgrade Kits on Jenkins

860
861
<a id="prepare-iuk"></a>

862
863
864
Prepare incremental upgrades
============================

865
866
867
868
Since Tails 4.2, we use a new upgrade scheme, which fundamentally
changes what the source version number of an upgrade means: it's now
the version that was *initially installed* and *not* the currently
running version. If this is news to you, see:
869

870
871
* the document that explains the benefits for our users:
  [[blueprint/Endless_upgrades]];
872

873
874
* the corresponding
  [[design documentation|contribute/design/incremental_upgrades]].
875

segfault's avatar
segfault committed
876
The main practical implications at release time are:
877

878
879
880
* The Release Manager has to publish more IUKs than they used to.
  But they can now publish IUKs (reproducibly) built on Jenkins,
  instead of having to upload those they've built locally.
881

882
* The Release Manager has to sign more UDFs than they used to.
883

884
885
886
887
888
889
890
891
892
893
* Computing `$IUK_SOURCE_VERSIONS` is now straightforward enough
  that it was automated :)

Prepare the environment
-----------------------

Compute the list of initial version install to build IUKs for:

    cd "${RELEASE_CHECKOUT:?}" && \
	export IUK_SOURCE_VERSIONS=$(./bin/iuk-source-versions ${VERSION:?})
894
    echo "${IUK_SOURCE_VERSIONS?:}"
895

896
897
898
Even if it's computed automatically, remember to store it in the file
holding environment variables, it will be used in various places below.

899
900
901
902
903
904
905
906
907
908
909
910
911
912
913
914
915
916
Sanity checks
-------------

Check that you have the correct version of `squashfs-tools` installed:

    [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
      = '1:4.4-1+0.tails1' \
    ] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for <https://iso-history.tails.boum.org> (it
will be used by `wrap_tails_create_iuks` below, in order to download any ISO
that is not present locally yet):

    if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
        echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
        echo "The corresponding login and password are in the RMs' keyringer."
    fi

917
918
Build the Incremental Upgrade Kits locally
------------------------------------------
919

920
921
922
923
924
925
926
927
928
You're encouraged to enable parallelism to avoid waiting for a very
long, serial build (which is still the default at the moment). As
discussed in [[!tails_ticket 17657]], it seems running as many jobs as
there are physical cores is a nice rule of thumb.

For example, set:

    JOBS="--jobs 4"

anonym's avatar
anonym committed
929
930
931
932
or, attempt to automatically set it to the number of physical cores:

    JOBS="--jobs $(grep '^core id' /proc/cpuinfo | sort -u | wc -l)"

933
934
before starting the wrapper from `puppet-tails`:

935
936
937
    (
       set -eu
       WORK_DIR=$(mktemp -d)
938
       TAILS_REMOTE="$(git -C "${RELEASE_CHECKOUT?:}" remote get-url origin)"
939
       PUPPET_TAILS_REMOTE=$(echo -n "${TAILS_REMOTE?:}" | perl -p -E 's,:tails/tails(?:[.]git)?\z,:tails/puppet-tails,')
940
       cd "${WORK_DIR?:}"
941
       git clone "$PUPPET_TAILS_REMOTE"
942
       sudo                                                                   \
943
       time                                                                   \
944
       ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
945
946
947
948
           --tails-git-remote "file://${RELEASE_CHECKOUT?:}/.git"             \
           --tails-git-commit "${TAG?:}"                                      \
           --source-date-epoch "${SOURCE_DATE_EPOCH?:}"                       \
           --local-isos-dir "${ISOS?:}"                                       \
949
           --tmp-dir "${TMPDIR:-/tmp}"                                        \
950
951
952
           --output-dir "${IUKS_DIR?:}"                                       \
           --source-versions "${IUK_SOURCE_VERSIONS?:}"                       \
           --new-version "${VERSION?:}"                                       \
953
           --verbose ${JOBS:-}
954
955
       cd "${IUKS_DIR?:}"
       sha256sum Tails_amd64_*_to_${VERSION?:}.iuk > "${IUKS_HASHES?:}"
956
    )
957

958
This command takes a long time. In parallel, while it is running,
959
you can follow the next step:
960
961
962
963
964
965
966
967
968
969

 - Build the Incremental Upgrade Kits on Jenkins

ISO history
-----------

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

970
 - The Jenkins `parallel_build_IUKs` job can fetch them.
intrigeri's avatar
intrigeri committed
971
 - Our isotesters can fetch them from there for their testing.
972

intrigeri's avatar
intrigeri committed
973
974
975
976
XXX: Dear first-RM-who-reads-this, please replace the next step
with new ones that are not affected by `git-annex` performance issues
([[!tails_ticket 17686#note_154190]]).

977
978
979
How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

980
Then, wait (a few minutes, `*/15` crontab) until the images appear
981
on <https://iso-history.tails.boum.org/>.
982
983
984
985

<a id="build-iuks-on-jenkins"></a>

Build the Incremental Upgrade Kits on Jenkins
986
987
988
989
990
---------------------------------------------

1. Make sure the push to ISO history (started in the previous section)
   has finished, and that images have appeared on the web server:
   <https://iso-history.tails.boum.org/>
991

992
993
994
995
996
1. On <https://jenkins.tails.boum.org/job/parallel_build_IUKs/configure>, adjust
   the `SOURCE_VERSION` axis to list all versions in `$IUK_SOURCE_VERSIONS`,
   and save the updated configuration.

1. On <https://jenkins.tails.boum.org/job/parallel_build_IUKs/build?delay=0sec>,
997
998
999
   fill the form with these values:

    - `TAILS_GIT_COMMIT`: the value of `$TAG` in your release environment
1000
    - `SOURCE_DATE_EPOCH`: the value of `$SOURCE_DATE_EPOCH` in your
1001
1002
1003
1004
1005
1006
1007
1008
1009
      release environment
    - `NEW_VERSION`: the value of `$VERSION` in your release environment
    - `EXTRA_ARGS`: leave it blank

2. Click the _Build_ button

3. After a few seconds, a new build appears on top of the _Build
   History_ sidebar. Click on the progress bar of this new build.

1010
1011
1012
3. **Verify:** A downstream `parallel_collect_IUKs` job should appear,
   to be triggered once the `parallel_build_IUKs` has completed successfully.

1013
4. Set the `$CANDIDATE_JENKINS_IUKS_BUILD_ID` environment variable
1014
   to the ID of that downstream job (an integer).
1015

1016
5. Wait until both `parallel_build_IUKs` and `parallel_collect_IUKs` jobs complete successfully.
1017
   It should take about 10-15 minutes for each member of