release_process.mdwn 76.2 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
<div class="caution">
8
9
Read the remainder of this document from the branch used to prepare the release
after having merged the current master branch into it!
10
11
</div>

12
13
14
Requirements
============

15
16
17
Packages
--------

18
19
To release Tails you'll need some packages installed:

20
* `gitlab-cli jq tidy mktorrent transmission-cli`
21
22
* [[!debpts squashfs-tools]] 1:4.4-1+0.tails1
  from our custom `iukbuilder-stretch` APT suite.
23
24
* `iuk` [[dependencies|contribute/release_process/tails-iuk#build-deps]]
* `perl5lib` [[dependencies|contribute/release_process/perl5lib#build-deps]]
25
* `po4a` 0.55: different versions extract Markdown headings
26
   in a different way, which makes tons of strings fuzzy.
27
* packages to [[build a local version of the website|contribute/build/website/]]
28

29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
Configuration files
-------------------

To release Tails you need:

* `~/.python-gitlab.cfg`

  You need at least this content:

        [global]
        ssl_verify = true

        [Tails]
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

   [[!tails_gitlab profile/personal_access_tokens
   desc="Generate a _Personal Access Token_"]] with at least the `api` scope
   enabled, and set it as the value of the `private_token` option
   in your `~/.python-gitlab.cfg`.

51
52
53
Environment
===========

54
55
56
57
58
59
60
To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the
value of one such variable, treat it as a constant: else,
inconsistency will surely arise, which can cause trouble later on.

61
62
63
Version numbers
---------------

64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
Note:

* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

Export the following environment variables:
81

82
* version numbers (see [[contribute/release_schedule#versioning]]):
83

84
85
86
87
        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
88

89
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
90
91
92
  *major* Tails release; if you're preparing a RC for a major release,
  use that major release; otherwise, use whatever the next planned
  major release is
93
94
* `SECOND_NEXT_PLANNED_MAJOR_VERSION`: if you're preparing the RC
  for a major release, set this to the version number of
95
96
97
98
  the second next *major* Tails release; e.g. if preparing the RC for
  the 3.9 major release, then set this to 3.12 (3.9 is the next major
  release, 3.10 and 3.11 are bugfix releases, 3.12 is a major
  release).
99
* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
100
  scheduled *bugfix* Tails release
101
* `NEXT_POTENTIAL_EMERGENCY_VERSION`: set to the version number we'll give
102
103
  to the next emergency release if we have to put one out; unset for a
  release candidate
104
105
* `NEXT_STABLE_CHANGELOG_VERSION`: if `$NEXT_PLANNED_BUGFIX_VERSION` is the next
  scheduled release, use it; otherwise, use `$NEXT_POTENTIAL_EMERGENCY_VERSION`
106
107
108
109
110
111

Other variables
---------------

Also export the following environment variables:

112
113
* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
114
* `ISOS`: the directory where one stores `tails-amd64-*`
115
  sub-directories like the ones downloaded with BitTorrent.
116
117
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
118
119
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
intrigeri's avatar
intrigeri committed
120
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
121
122
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
123
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
124
* `TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}" | perl -nE 'say substr($_, -17)')`
125
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
126
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
127
* `export IUKS_DIR="${ISOS:?}/iuks/v2"`
128
* `export IUKS_HASHES="${IUKS_DIR:?}/to_${VERSION}.sha256sum"`
129

130
131
132
Pre-freeze
==========

133
134
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
135
136
137
138
139
140

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

141
142
143
144
145
146
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

147
148
149
150
151
152
Freeze
======

Major release
-------------

153
154
If we are at freeze time for a major release (i.e. preparing its
release candidate):
155
156
157

1. Merge the `master` Git branch into `devel`:

158
        git checkout devel && git fetch origin && git merge origin/devel && git merge --no-ff origin/master
159

intrigeri's avatar
intrigeri committed
160
161
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
162
   APT suite; set `BRANCH=testing` instead of the default `BRANCH=devel`.
163
164
165

3. Merge the `devel` Git branch into the `testing` one:

166
        git checkout testing && git merge origin/testing && git merge devel
167
168
169
170

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

171
172
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
173

anonym's avatar
anonym committed
174
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
175
176
177
178
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
179
   long enough, by bumping their `Valid-Until` to 10 days after the
180
   second next major release (the one _after_ the one you're preparing)'s
181
   scheduled date:
182
183
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

184

185
186
Bugfix release
--------------
187

188
If we are at freeze time for a bugfix release:
189
190
191

1. Merge the `master` Git branch into `stable`:

192
        git checkout stable && git fetch && git merge origin/stable && git merge --no-ff origin/master
193

intrigeri's avatar
intrigeri committed
194
195
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
196
197
   APT suite.

198
199
Common steps for bugfix and major releases
------------------------------------------
200

201
Reset the release branch's `config/base_branch`:
202

203
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
204
           git commit config/base_branch \
205
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
206

207
208
209
210
211
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
212
213
3. Send the pad URL in an **encrypted** mail to the manual testers:
   <tails-manual-testers@boum.org>
214

Tails developers's avatar
Tails developers committed
215
216
217
Update included files
=====================

218
219
<a id="upgrade-custom-debs"></a>

220
221
222
223
224
225
226
227
228
229
Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[tor-browser_AppArmor_patch]]

Bessemer's avatar
Bessemer committed
230
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
231
232
--------------------------------------

233
Skip this section if you are preparing a bugfix release.
234
235
236

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:
237

238
239
240
 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
241
 - If you are preparing a final major release, build at least the packages
242
243
244
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)
245

246
247
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
intrigeri's avatar
intrigeri committed
248
import translations from Transifex and sanity-check them:
249

250
251
252
	cd whisperback && \
	git checkout master && \
	git pull && \
intrigeri's avatar
intrigeri committed
253
	"${RELEASE_CHECKOUT:?}"/import-translations && \
254
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po
255

256
257
258
Then, for every PO file that has issues:

1. Rollback changes to that file: `git checkout po/LL.po`
259
2. Run `lint_po` again. It should pass this time.
260

261
And finally, commit:
262

263
    git add po && git commit \
264
	    -m "Update POT and PO files, pull updated translations from Transifex."
265

266
Then see the relevant release processes, and upload the packages to
267
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
268

269
* [[tails-installer]]
270
* whisperback:
271
  * follow [upstream release process](https://gitlab.tails.boum.org/tails/whisperback/-/blob/master/HACKING.md#release)
272
273
  * build a Debian package in an amd64 chroot of the Debian release
    the Tails version you're preparing is based on
Tails developers's avatar
Tails developers committed
274

275
276
277
278
279
Upgrade custom packages for VeraCrypt integration
-------------------------------------------------

See the dedicated page: [[veracrypt]]

280
281
Update PO files
---------------
282

283
284
285
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
286

287
	cd "${RELEASE_CHECKOUT:?}" && \
288
289
	./import-translations  && \
	./refresh-translations && \
290
	./submodules/jenkins-tools/slaves/lint_po && \
291
	git add po && git commit -m 'Update PO files.'
292

293
If `lint_po` complains:
294

295
* rollback the offending PO files and retry; worst case, delete it
296
* send a note to <tails-l10n@boum.org> [public] so that they get in touch with
297
298
  whoever can fix them.

299
300
When preparing an actual release
================================
301

302
If we're about to prepare the images for a final (non-RC) release, then
303
304
305
306
307
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
308
309
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
310
custom APT suite.
311

312
313
Bugfix release
--------------
314

315
<div class="note">
316
For bugfix releases, we generally do not put any RC out, so freeze time
317
318
319
320
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
321

intrigeri's avatar
intrigeri committed
322
323
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
324
custom APT suite.
325

anonym's avatar
anonym committed
326
327
Update other base branches
==========================
328
329

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
330
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
331

332
2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
333
   branch, the time-based APT repository snapshots being used
334
335
   during the freeze. It's fine if that results in a no-op
   (it depends on how exactly previous operations were performed).
intrigeri's avatar
intrigeri committed
336

337
3. Merge `devel` into `feature/bullseye` (if it exists), *without* following the instructions for
intrigeri's avatar
intrigeri committed
338
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
339
   (For now `feature/bullseye` is handled as any other topic branch
340
   forked off `devel`: its base branch is set to `devel`.)
341
342
343
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.
344

345
4. Ensure that the release, `devel` and `feature/bullseye` (if it exists) branches
346
347
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
348

349
5. Push the modified branches to Git:
350

351
        git push origin                          \
352
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
353
           $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
354
           devel:devel
355
356
357

Update more included files
==========================
358

359
360
Changelog
---------
Tails developers's avatar
Tails developers committed
361

362
363
364
Make sure to switch back to the release branch, spawn an editor to
remove the placeholder entry for then next release in `debian/changelog`,
and gather data from the Git repository:
365

366
	git checkout "${RELEASE_BRANCH:?}" && \
367
	dch -e && \
368
	DEBEMAIL='tails@boum.org' DEBFULLNAME='Tails developers' \
369
	./release ${VERSION:?} ${PREVIOUS_TAG:?}
Tails developers's avatar
Tails developers committed
370
371
372

This populates the Changelog with the Git log entries.

373
374
375
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
376

377
378
Changelog entries can be dispatched into those usual sections:

379
380
381
382
383
384
385
386
387
388
389
390
391
<pre>
  * Major changes

  * Security fixes

  * Bugfixes

  * Minor improvements and updates

  * Build system

  * Test suite
</pre>
392

393
394
Then, gather other useful information from:

395
* every custom bundled package's own Changelog (Tails Installer, etc.)
396
* the diff between the previous version's `.packages` file and the one
397
  from the to-be-released images; look for:
398
399
400
401
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
intrigeri's avatar
intrigeri committed
402
* the [[!tails_gitlab groups/tails/-/milestones desc="GitLab milestone"]].
403

404
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
405

406
407
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
408
	else
409
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
410
	fi
Tails developers's avatar
Tails developers committed
411

Bessemer's avatar
Bessemer committed
412
Included website
413
414
----------------

Tails developers's avatar
Tails developers committed
415
416
417
418
419
420
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

421
422
### version number

423
424
If preparing a RC, skip this part.

425
In the branch used to build the release, update the `wiki/src/inc/*` files to
426
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
427
at least 24 hours in the future! Between tests and mirror synchronization,
428
the build will not be released on the same day. Try to make sure it
429
matches the date of the future signature.
430

sajolida's avatar
sajolida committed
431
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
432

intrigeri's avatar
intrigeri committed
433
434
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html && \
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html && \
435
436
437
	for type in img iso; do
	   basename="tails-amd64-${VERSION:?}"
	   filename="${basename:?}.${type:?}"
438
	   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
439
440
	        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
	   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
intrigeri's avatar
intrigeri committed
441
	        > wiki/src/inc/stable_amd64_${type:?}_url.html && \
442
	   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
intrigeri's avatar
intrigeri committed
443
	        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html && \
444
445
	   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
	        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
intrigeri's avatar
intrigeri committed
446
447
	done && \
	./build-website --rebuild && \
448
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
449

450
451
452
453
454
455
456
457
458
459
Signing key downloaded by the Upgrader
--------------------------------------

    TMP_GNUPG_HOME=$(mktemp -d)
    gpg --homedir "${TMP_GNUPG_HOME:?}" --import wiki/src/tails-signing.key && \
    gpg --homedir "${TMP_GNUPG_HOME:?}" --export-options export-minimal \
        --armor --export "${TAILS_SIGNATURE_KEY:?}" \
        > wiki/src/tails-signing-minimal.key && \
    git commit wiki/src/tails-signing-minimal.key \
        -m "Update signing key used by the Upgrader"
460
    rm -rf "${TMP_GNUPG_HOME:?}"
461

462
463
Website translations
--------------------
464

465
466
467
468
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
469

intrigeri's avatar
intrigeri committed
470
471
472
473
    ./build-website && \
    git add wiki/src && \
    git commit -m 'Update website PO files.'
    git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
474

475
476
477
Call for translation
====================

478
479
480
If at freeze time for a major release, send a call for translations to
<tails-l10n@boum.org> [public], making it clear what Git branch the
translations must be based on, and what are the priorities.
481

482
To get a list of changes on the website:
483

intrigeri's avatar
intrigeri committed
484
    git diff --stat ${PREVIOUS_TAG:?}.. -- \
485
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
486
487
488
489
490
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
491

492
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
493
494
======================

495
496
497
498
### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
499
pro tip is to never plug it unless prompted which `gpg` will do for you.
500
501
502
503
504
505

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
506

Tails developers's avatar
Tails developers committed
507
You should never import the Tails signing key into your own keyring,
508
509
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
510
511

    export GNUPGHOME=$(mktemp -d)
512
513
514
515
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
516
517
518
519
520
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

521
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
522

523
524
Build the almost-final images
=============================
525

526
527
528
529
530
1. [[Build ISO and USB images|contribute/build]] from the release branch,
   with `$TAILS_BUILD_OPTIONS` set like this:
   - Set `defaultcomp`, so we can more accurately optimize our
     SquashFS file ordering.
   - Do _not_ set `keeprunning` nor `rescue`.
531
2. Carefully read the build logs to make sure nothing bad happened.
532
3. Keep the resulting build artifacts until the end of this release process.
533
534
4. Record where the manifest of needed packages is stored:

535
536
        export BUILD_MANIFEST=XXX ; \
        [ -f "${BUILD_MANIFEST:?}" ] || echo "ERROR: BUILD_MANIFEST is incorrect"
537
538
        echo "${BUILD_MANIFEST:?}" | grep -E -qs '\.build-manifest$' \
           || echo "ERROR: BUILD_MANIFEST does not have the .build-manifest extension"
539

540
541
542
Tag the release in Git
======================

543
544
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
545
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
546
547
548
549
550
551
552

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

553
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
intrigeri's avatar
intrigeri committed
554
555
to continue if the last changelog entry is unreleased but corresponds to
an existing tag. There are workarounds we need to decide and implement.
556

557
558
559
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
560
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
561

562
* Prepare tagged snapshots of upstream APT repositories:
563

564
          ./bin/tag-apt-snapshots "${BUILD_MANIFEST:?}" "${TAG:?}"
565

566
  Note:
567

568
569
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
570
    [[custom APT repository|APT_repository/custom]] are
571
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
572
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
573
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
574
    Now, we have a "safety" net, in case you don't notice such a problem: if
575
576
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
577

Tails developers's avatar
Tails developers committed
578
579
580
Build images
============

581
582
583
Sanity check
------------

584
Verify that the Tor Browser release used in Tails still is the most
585
586
587
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:
588

589
* <https://gitweb.torproject.org/builders/tor-browser-build.git>
590
591
* <https://gitweb.torproject.org/tor-browser.git>

592
A new tag may indicate that a new Tor Browser release or rebuild is imminent.
593
594
595

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
596
597
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
598

599
1. Install the almost-final USB image to a USB stick.
600
601
602
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
1. Boot this USB stick **on bare metal** again.
603
1. Add `profile` to the kernel command-line.
604
1. Login with the default settings in the Welcome Screen (e.g. do not configure
605
   an _Administration Password_).
606
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
607
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
608
1. A few minutes later, once the `boot-profile` process has been
609
   killed, retrieve the new sort file from `/var/log/boot-profile`.
intrigeri's avatar
intrigeri committed
610
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
611
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
612
613
614
615
616
617
1. Remove runtime-generated files that don't exist in the rootfs,
   in order to avoid confusing noise in the build output:

           perl -ni -E 'chomp; say unless m{(?:
                  [.]pyc\s+\d+\z
               | \Alib/live/mount/medium/live/(?:filesystem[.]squashfs|initrd[.]img)\s
618
619
620
621
622
623
624
625
626
               | \Alib/live/mount/overlay/rw/etc/fstab\s
               | \Alib/live/mount/overlay/rw/etc/console-setup/cached_\S+[.](?:gz|sh)\s
               | \Alib/live/mount/overlay/rw/etc/machine-id\s
               | \Alib/live/mount/overlay/rw/etc/network/interfaces\s
               | \Alib/live/mount/overlay/rw/var/log/wtmp\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/apparmor[.]d/cache/[.]features\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/(?:group|gshadow|passwd|shadow)-\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/resolv-over-clearnet[.]conf\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/skel/[.]config/autostart/end-profile[.]desktop\s
627
               | \Arun/
intrigeri's avatar
intrigeri committed
628
               | \Avar/lib/AccountsService/users/Debian-gdm\s
629
               | \Avar/lib/gdm3/[#]\d+\s
630
631
632
633
634
               | \Avar/log/live/config[.]pipe\s
           )}xms' config/binary_rootfs/squashfs.sort

1. Remove the bits about `kill-boot-profile` at the end: they're
   only useful when profiling the boot.
intrigeri's avatar
intrigeri committed
635
1. Inspect the Git diff (including diff stat), apply common sense:
636

637
        diff -NaurB \
intrigeri's avatar
intrigeri committed
638
639
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
640
            | less
641

642
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`
643
1. Clean up: `rm -f config/binary_rootfs/squashfs.sort.old`
644

645
646
Build the final images
----------------------
647

Bessemer's avatar
Bessemer committed
648
Then all included files should be up-to-date and the versioned APT
649
650
suite should be ready, so it is time to:

651
1. Mark the version as "released" in the changelog:
652

653
        dch --release --no-force-save-on-release --maintmaint && \
654
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
655

656
1. Export `SOURCE_DATE_EPOCH`:
657

658
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
659

660
1. tag the release *again*, with all included files in:
661

662
663
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
664
665
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
666

667
668
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
669
   commands by e.g. committing a commit in between `git tag` and the
670
671
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
672

673
1. build the final images!
674
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
675
676
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.
677

678
679
680
681
1. Make sure the Jenkins build starts. Until the hook is back in place
   ([[!tails_ticket 17745]]), starting it manually may avoid up to 15
   minutes of waiting.

intrigeri's avatar
intrigeri committed
682
1. Compare the new build manifest with the one from the previous,
683
   almost-final build:
intrigeri's avatar
intrigeri committed
684
685

        diff -Naur \
686
           "${BUILD_MANIFEST:?}" \
687
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
intrigeri's avatar
intrigeri committed
688
689
690
691

   They should be identical, except that the `debian-security` serial might be higher.

1. To ensure we publish the final build's `.build-manifest`, run:
692

693
        export BUILD_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
694

intrigeri's avatar
intrigeri committed
695
<a id="reproducibility-sanity-check-iso"></a>
696

intrigeri's avatar
intrigeri committed
697
698
699
700
Verify that Jenkins reproduced your images
------------------------------------------

to verify that Jenkins reproduced your images:
701

intrigeri's avatar
intrigeri committed
702
1. Visit the URL printed by this command:
anonym's avatar
xxx    
anonym committed
703
704
705

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

intrigeri's avatar
intrigeri committed
706
2. Find the job (probably the last one)
707
708
   and make sure the ISO and USB images built by Jenkins
   have the same hash (in the `.shasum` file) as the images you built.
709

intrigeri's avatar
intrigeri committed
710
3. Then:
711

712
713
   - If the ISO and USB images hashes match: yay, we're good to go!
     The `.build-manifest` may differ — that's OK.
714

715
     Set the `$MATCHING_JENKINS_IMAGES_BUILD_ID` environment variable
716
717
     to the ID of this job (an integer).

718
   - If there is a hash mismatch for one of the images: ouch! Now we are in a
719
720
721
722
723
724
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
725

anonym's avatar
anonym committed
726
727
728
729
730
731
732
733
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
734

735
736
     Do the same for the USB image as well.

737
     Then carefully investigate the `diffoscope` report:
738

739
740
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
intrigeri's avatar
intrigeri committed
741
         release. Halt the release, involve the rest of <tails@boum.org>, and then
742
743
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
744

745
       - Otherwise, if the change is definitely harmless:
746

747
         * If the source of non-determinism is identified quickly and
748
           is easy and fast to fix, *and* the QA of the current images
749
750
751
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
752

753
754
755
756
757
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
758

759
760
761
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
intrigeri's avatar
intrigeri committed
762
           release notes, linking to the issue where
763
764
           the nature of the reproducibility failure is clearly
           described.
765

intrigeri's avatar
intrigeri committed
766
767
Initialize the website release branch
-------------------------------------
intrigeri's avatar
intrigeri committed
768

769
770
771
772
773
774
From now on, we don't want to push new commits on `$RELEASE_BRANCH`
until the new release is out. Otherwise, this would break its build
and the build of every branch based on it, which would effectively
block other development work. So the final steps towards publishing
the release will be done in a new, dedicated branch.

intrigeri's avatar
intrigeri committed
775
776
If preparing anything but a final release (e.g. an alpha, beta
or RC):
777
778
779
780

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

intrigeri's avatar
intrigeri committed
781
Else, if preparing a final release:
782

783
784
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
785

786
787
788
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
789
Create a directory with a suitable name, go there, move the built
790
791
792
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
intrigeri's avatar
intrigeri committed
793
794
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
795

796
797
798
799
800
801
802
    mkdir -p "${ISOS:?}/tails-amd64-${VERSION:?}" && \
    for type in iso img ; do
       cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
       mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
       gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
       rename 's,\.asc$,.sig,' *.asc && \
       tmp="$(mktemp -d)" && \
803
804
       mkdir -p "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       cd "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
805
       for x in "${ISOS:?}/tails-amd64-${VERSION:?}"/*.${type:?}*; do
806
807
808
809
810
811
           ln -s ${x} .
       done && \
       mktorrent \
          -o "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" \
          -a 'udp://tracker.torrent.eu.org:451'   \
          -a 'udp://tracker.coppersurfer.tk:6969' \
812
          "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
813
       transmission-show "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" && \
814
       cd - && \
815
816
       rm -rf "${tmp:?}"
    done
817

818
819
820
821
822
Due to various directory changes, one needs to manually go back to the
desired directory, likely using:

    cd ${RELEASE_CHECKOUT?:}

anonym's avatar
anonym committed
823
824
Lastly, let's set some variables to be used later:

825
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
826
827
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
828
    IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.img"
829
830
    IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"
anonym's avatar
anonym committed
831

832
833
834
835
836
837
838
Push images to ISO history
==========================

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

839
 - The Jenkins `parallel_build_IUKs` job can fetch them.
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
 - Our isotesters can fetch them from there for their testing.

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

If the release manager has `$ISOS` pointing to a checkout of the ISO
history repository, the following commands might be useful. For the
record, images and their signatures are already in that directory, and
one just needs to ship some metadata along:

    (cd ${ISOS?:}/tails-amd64-${VERSION?:} && \
    cp "${ARTIFACTS:?}"/tails-amd64-${VERSION:?}{.buildlog,.build-manifest,.packages} . && \
    git annex sync && \
    git annex add . && \
    git commit -m "Add Tails ${VERSION?:} images and metadata." && \
    git annex sync && \
    git annex copy --to origin .)

This transfer can take a while, and it's possible to proceed with the
next section in the meanwhile, up to the following step (included):

 - Build the Incremental Upgrade Kits locally

However, the following step will require those files being transferred
and made available on the relevant web server:

 - Build the Incremental Upgrade Kits on Jenkins

868
869
<a id="prepare-iuk"></a>

870
871
872
Prepare incremental upgrades
============================

873
874
875
876
Since Tails 4.2, we use a new upgrade scheme, which fundamentally
changes what the source version number of an upgrade means: it's now
the version that was *initially installed* and *not* the currently
running version. If this is news to you, see:
877

878
879
* the document that explains the benefits for our users:
  [[blueprint/Endless_upgrades]];
880

881
882
* the corresponding
  [[design documentation|contribute/design/incremental_upgrades]].
883

segfault's avatar
segfault committed
884
The main practical implications at release time are:
885

886
887
888
* The Release Manager has to publish more IUKs than they used to.
  But they can now publish IUKs (reproducibly) built on Jenkins,
  instead of having to upload those they've built locally.
889

890
* The Release Manager has to sign more UDFs than they used to.
891

892
893
894
895
896
897
898
899
900
901
* Computing `$IUK_SOURCE_VERSIONS` is now straightforward enough
  that it was automated :)

Prepare the environment
-----------------------

Compute the list of initial version install to build IUKs for:

    cd "${RELEASE_CHECKOUT:?}" && \
	export IUK_SOURCE_VERSIONS=$(./bin/iuk-source-versions ${VERSION:?})
902
    echo "${IUK_SOURCE_VERSIONS?:}"
903

904
905
906
Even if it's computed automatically, remember to store it in the file
holding environment variables, it will be used in various places below.

907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
924
Sanity checks
-------------

Check that you have the correct version of `squashfs-tools` installed:

    [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
      = '1:4.4-1+0.tails1' \
    ] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for <https://iso-history.tails.boum.org> (it
will be used by `wrap_tails_create_iuks` below, in order to download any ISO
that is not present locally yet):

    if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
        echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
        echo "The corresponding login and password are in the RMs' keyringer."
    fi

925
926
Build the Incremental Upgrade Kits locally
------------------------------------------
927

928
929
930
931
932
933
934
935
936
937
938
You're encouraged to enable parallelism to avoid waiting for a very
long, serial build (which is still the default at the moment). As
discussed in [[!tails_ticket 17657]], it seems running as many jobs as
there are physical cores is a nice rule of thumb.

For example, set:

    JOBS="--jobs 4"

before starting the wrapper from `puppet-tails`:

939
940
941
    (
       set -eu
       WORK_DIR=$(mktemp -d)
942
943
944
       TAILS_REMOTE="$(git -C "${RELEASE_CHECKOUT?:}" remote get-url origin)"
       PUPPET_TAILS_REMOTE=$(echo -n "${TAILS_REMOTE?:}" | perl -p -E 's,:tails(:?[.]git)?\z,:puppet-tails,')
       cd "${WORK_DIR?:}"
945
       git clone "$PUPPET_TAILS_REMOTE"
946
       sudo -l
947
       ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
948
949
950
951
           --tails-git-remote "file://${RELEASE_CHECKOUT?:}/.git"             \
           --tails-git-commit "${TAG?:}"                                      \
           --source-date-epoch "${SOURCE_DATE_EPOCH?:}"                       \
           --local-isos-dir "${ISOS?:}"                                       \
952
           --tmp-dir "${TMPDIR:-/tmp}"                                        \
953
954
955
           --output-dir "${IUKS_DIR?:}"                                       \
           --source-versions "${IUK_SOURCE_VERSIONS?:}"                       \
           --new-version "${VERSION?:}"                                       \
956
           --verbose ${JOBS:-}
957
958
       cd "${IUKS_DIR?:}"
       sha256sum Tails_amd64_*_to_${VERSION?:}.iuk > "${IUKS_HASHES?:}"
959
    )
960

961
This command takes a long time. In parallel, while it is running,
962
you can follow the next step:
963
964
965
966
967
968
969
970
971
972

 - Build the Incremental Upgrade Kits on Jenkins

ISO history
-----------

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

973
 - The Jenkins `parallel_build_IUKs` job can fetch them.
intrigeri's avatar
intrigeri committed
974
 - Our isotesters can fetch them from there for their testing.
975
976
977
978

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

979
Then, wait (a few minutes, */15 crontab) until the images appear
980
on <https://iso-history.tails.boum.org/>.
981
982
983
984

<a id="build-iuks-on-jenkins"></a>

Build the Incremental Upgrade Kits on Jenkins
985
986
987
988
989
---------------------------------------------

1. Make sure the push to ISO history (started in the previous section)
   has finished, and that images have appeared on the web server:
   <https://iso-history.tails.boum.org/>