release_process.mdwn 70.5 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
<div class="caution">
8
9
Read the remainder of this document from the branch used to prepare the release
after having merged the current master branch into it!
10
11
</div>

12
13
14
15
16
Requirements
============

To release Tails you'll need some packages installed:

17
* `tidy mktorrent transmission-cli`
18
19
* [[!debpts squashfs-tools]] 1:4.4-1+0.tails1
  from our custom `iukbuilder-stretch` APT suite.
20
21
* `iuk` [[dependencies|contribute/release_process/tails-iuk#build-deps]]
* `perl5lib` [[dependencies|contribute/release_process/perl5lib#build-deps]]
22
23
* `po4a` _from Stretch_: the version in testing/sid extracts Markdown headings
   in a different way, which makes tons of strings fuzzy.
24

25
26
27
Environment
===========

28
29
30
31
32
33
34
To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the
value of one such variable, treat it as a constant: else,
inconsistency will surely arise, which can cause trouble later on.

35
36
37
Version numbers
---------------

38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
Note:

* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

Export the following environment variables:
55

56
* version numbers (see [[contribute/release_schedule#versioning]]):
57

58
59
60
61
        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
62

63
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
64
65
66
  *major* Tails release; if you're preparing a RC for a major release,
  use that major release; otherwise, use whatever the next planned
  major release is
67
68
* `SECOND_NEXT_PLANNED_MAJOR_VERSION`: if you're preparing the RC
  for a major release, set this to the version number of
69
70
71
72
  the second next *major* Tails release; e.g. if preparing the RC for
  the 3.9 major release, then set this to 3.12 (3.9 is the next major
  release, 3.10 and 3.11 are bugfix releases, 3.12 is a major
  release).
73
* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
74
  scheduled *bugfix* Tails release
75
76
* `NEXT_POTENTIAL_EMERGENCY_VERSION`: set to the version number we'll give
  to the next emergency release if we have to put one out
77
78
* `NEXT_STABLE_CHANGELOG_VERSION`: if `$NEXT_PLANNED_BUGFIX_VERSION` is the next
  scheduled release, use it; otherwise, use `$NEXT_POTENTIAL_EMERGENCY_VERSION`
79
80
81
82
83
84

Other variables
---------------

Also export the following environment variables:

85
86
* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
87
* `ISOS`: the directory where one stores `tails-amd64-*`
88
  sub-directories like the ones downloaded with BitTorrent.
89
90
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
91
92
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
intrigeri's avatar
intrigeri committed
93
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
94
95
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
96
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
97
* `TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}" | perl -nE 'say substr($_, -17)')`
98
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
99
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
100
* `export IUKS_DIR="${ISOS:?}/iuks/v2"`
101
* `export IUKS_HASHES="${IUKS_DIR:?}/to_${VERSION}.sha256sum"`
102

103
104
105
Pre-freeze
==========

106
107
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
108
109
110
111
112
113

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

114
115
116
117
118
119
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

120
121
122
123
124
125
126
127
128
129
Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

130
        git checkout devel && git fetch origin && git merge --no-ff origin/master
131

intrigeri's avatar
intrigeri committed
132
133
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
134
135
136
137
138
139
140
141
142
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

143
144
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
145

anonym's avatar
anonym committed
146
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
147
148
149
150
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
151
152
153
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
154
155
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

156

157
158
Bugfix release
--------------
159

160
If we are at freeze time for a bugfix release:
161
162
163

1. Merge the `master` Git branch into `stable`:

164
        git checkout stable && git fetch && git merge --no-ff origin/master
165

intrigeri's avatar
intrigeri committed
166
167
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
168
169
   APT suite.

170
171
Common steps for bugfix and major releases
------------------------------------------
172

173
Reset the release branch's `config/base_branch`:
174

175
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
176
           git commit config/base_branch \
177
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
178

179
180
181
182
183
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
184
185
3. Send the pad URL in an **encrypted** mail to the manual testers:
   <tails-manual-testers@boum.org>
186

Tails developers's avatar
Tails developers committed
187
188
189
Update included files
=====================

190
191
<a id="upgrade-custom-debs"></a>

192
193
194
195
196
197
198
199
200
201
Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[tor-browser_AppArmor_patch]]

Bessemer's avatar
Bessemer committed
202
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
203
204
--------------------------------------

205
Skip this section if you are preparing a bugfix release.
206
207
208

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:
209

210
211
212
213
214
215
216
 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
 - If you are preparing a major release, build at least the packages
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)
217

218
219
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
intrigeri's avatar
intrigeri committed
220
import translations from Transifex and sanity-check them:
221

222
223
224
	cd whisperback && \
	git checkout master && \
	git pull && \
intrigeri's avatar
intrigeri committed
225
	"${RELEASE_CHECKOUT:?}"/import-translations && \
226
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po
227

228
229
230
Then, for every PO file that has issues:

1. Rollback changes to that file: `git checkout po/LL.po`
231
2. Run `lint_po` again. It should pass this time.
232

233
And finally, commit:
234

235
    git add po && git commit \
236
	    -m "Update POT and PO files, pull updated translations from Transifex."
237

238
Then see the relevant release processes, and upload the packages to
239
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
240

241
* [[tails-installer]]
242
* whisperback:
243
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
244
245
  * build a Debian package in an amd64 chroot of the Debian release
    the Tails version you're preparing is based on
Tails developers's avatar
Tails developers committed
246

247
248
249
250
251
Upgrade custom packages for VeraCrypt integration
-------------------------------------------------

See the dedicated page: [[veracrypt]]

252
253
Update PO files
---------------
254

255
256
257
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
258

259
	cd "${RELEASE_CHECKOUT:?}" && \
260
261
	./import-translations  && \
	./refresh-translations && \
262
	./submodules/jenkins-tools/slaves/lint_po && \
263
	git add po && git commit -m 'Update PO files.'
264

265
If `lint_po` complains:
266

267
* rollback the offending PO files and retry; worst case, delete it
268
* send a note to <tails-l10n@boum.org> [public] so that they get in touch with
269
270
  whoever can fix them.

271
272
When preparing an actual release
================================
273

274
If we're about to prepare the images for a final (non-RC) release, then
275
276
277
278
279
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
280
281
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
282
custom APT suite.
283

284
285
Bugfix release
--------------
286

287
<div class="note">
288
For bugfix releases, we generally do not put any RC out, so freeze time
289
290
291
292
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
293

intrigeri's avatar
intrigeri committed
294
295
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
296
custom APT suite.
297

anonym's avatar
anonym committed
298
299
Update other base branches
==========================
300
301

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
302
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
303

304
2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
intrigeri's avatar
intrigeri committed
305
   branch, the time-based APT repository snapshots that were used
306
307
   during the freeze. It's fine if that results in a no-op
   (it depends on how exactly previous operations were performed).
intrigeri's avatar
intrigeri committed
308

309
3. Merge `devel` into `feature/bullseye` (if it exists), *without* following the instructions for
intrigeri's avatar
intrigeri committed
310
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
311
   (For now `feature/bullseye` is handled as any other topic branch
312
   forked off `devel`: its base branch is set to `devel`.)
313
314
315
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.
316

317
4. Ensure that the release, `devel` and `feature/bullseye` (if it exists) branches
318
319
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
320

321
5. Push the modified branches to Git:
322

323
        git push origin                          \
324
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
325
           $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
326
           devel:devel
327
328
329

Update more included files
==========================
330

331
332
Changelog
---------
Tails developers's avatar
Tails developers committed
333

334
335
336
Remove the placeholder entry for next release in `debian/changelog`,
and then:

337
	git checkout "${RELEASE_BRANCH:?}" && \
338
	DEBEMAIL='tails@boum.org' DEBFULLNAME='Tails developers' \
339
	./release ${VERSION:?} ${PREVIOUS_TAG:?}
Tails developers's avatar
Tails developers committed
340
341
342

This populates the Changelog with the Git log entries.

343
344
345
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
346

347
348
Changelog entries can be dispatched into those usual sections:

349
350
351
352
353
354
355
356
357
358
359
360
361
<pre>
  * Major changes

  * Security fixes

  * Bugfixes

  * Minor improvements and updates

  * Build system

  * Test suite
</pre>
362

363
364
Then, gather other useful information from:

365
* every custom bundled package's own Changelog (Tails Installer, etc.)
366
* the diff between the previous version's `.packages` file and the one
367
  from the to-be-released images; look for:
368
369
370
371
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
372
* the [*Completed for the next release*](https://redmine.tails.boum.org/code/projects/tails/issues?query_id=327) Redmine view.
373

374
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
375

376
377
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
378
	else
379
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
380
	fi
Tails developers's avatar
Tails developers committed
381

Bessemer's avatar
Bessemer committed
382
Included website
383
384
----------------

Tails developers's avatar
Tails developers committed
385
386
387
388
389
390
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

391
392
### version number

393
394
If preparing a RC, skip this part.

395
In the branch used to build the release, update the `wiki/src/inc/*` files to
396
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
397
at least 24 hours in the future! Between tests and mirror synchronization,
398
the build will not be released on the same day. Try to make sure it
399
matches the date of the future signature.
400

sajolida's avatar
sajolida committed
401
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
402

intrigeri's avatar
intrigeri committed
403
404
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html && \
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html && \
405
406
407
	for type in img iso; do
	   basename="tails-amd64-${VERSION:?}"
	   filename="${basename:?}.${type:?}"
408
	   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
409
410
	        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
	   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
intrigeri's avatar
intrigeri committed
411
	        > wiki/src/inc/stable_amd64_${type:?}_url.html && \
412
	   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
intrigeri's avatar
intrigeri committed
413
	        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html && \
414
415
	   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
	        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
intrigeri's avatar
intrigeri committed
416
417
	done && \
	./build-website --rebuild && \
418
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
419

420
421
422
423
424
425
426
427
428
429
Signing key downloaded by the Upgrader
--------------------------------------

    TMP_GNUPG_HOME=$(mktemp -d)
    gpg --homedir "${TMP_GNUPG_HOME:?}" --import wiki/src/tails-signing.key && \
    gpg --homedir "${TMP_GNUPG_HOME:?}" --export-options export-minimal \
        --armor --export "${TAILS_SIGNATURE_KEY:?}" \
        > wiki/src/tails-signing-minimal.key && \
    git commit wiki/src/tails-signing-minimal.key \
        -m "Update signing key used by the Upgrader"
430
    rm -rf "${TMP_GNUPG_HOME:?}"
431

432
433
Website translations
--------------------
434

435
436
437
438
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
439

intrigeri's avatar
intrigeri committed
440
441
442
443
    ./build-website && \
    git add wiki/src && \
    git commit -m 'Update website PO files.'
    git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
444

445
446
447
Call for translation
====================

448
449
If at freeze time for a major release, send a call for translations
to tails-l10n, making it clear what
450
Git branch the translations must be based on, and what are the
451
priorities.
452

453
To get a list of changes on the website:
454

intrigeri's avatar
intrigeri committed
455
    git diff --stat ${PREVIOUS_TAG:?}.. -- \
456
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
457
458
459
460
461
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
462

463
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
464
465
======================

466
467
468
469
### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
470
pro tip is to never plug it unless prompted which `gpg` will do for you.
471
472
473
474
475
476

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
477

Tails developers's avatar
Tails developers committed
478
You should never import the Tails signing key into your own keyring,
479
480
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
481
482

    export GNUPGHOME=$(mktemp -d)
483
484
485
486
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
487
488
489
490
491
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

492
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
493

494
495
Build the almost-final images
=============================
496

497
498
499
500
501
1. [[Build ISO and USB images|contribute/build]] from the release branch,
   with `$TAILS_BUILD_OPTIONS` set like this:
   - Set `defaultcomp`, so we can more accurately optimize our
     SquashFS file ordering.
   - Do _not_ set `keeprunning` nor `rescue`.
502
2. Carefully read the build logs to make sure nothing bad happened.
503
3. Keep the resulting build artifacts until the end of this release process.
504
505
4. Record where the manifest of needed packages is stored:

506
507
        export BUILD_MANIFEST=XXX ; \
        [ -f "${BUILD_MANIFEST:?}" ] || echo "ERROR: BUILD_MANIFEST is incorrect"
508
509
        echo "${BUILD_MANIFEST:?}" | grep -E -qs '\.build-manifest$' \
           || echo "ERROR: BUILD_MANIFEST does not have the .build-manifest extension"
510

511
512
513
Tag the release in Git
======================

514
515
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
516
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
517
518
519
520
521
522
523

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

524
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
intrigeri's avatar
intrigeri committed
525
526
to continue if the last changelog entry is unreleased but corresponds to
an existing tag. There are workarounds we need to decide and implement.
527

528
529
530
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
531
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
532

533
* Prepare tagged snapshots of upstream APT repositories:
534

535
          ./bin/tag-apt-snapshots "${BUILD_MANIFEST:?}" "${TAG:?}"
536

537
  Note:
538

539
540
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
541
    [[custom APT repository|APT_repository/custom]] are
542
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
543
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
544
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
545
    Now, we have a "safety" net, in case you don't notice such a problem: if
546
547
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
548

Tails developers's avatar
Tails developers committed
549
550
551
Build images
============

552
553
554
Sanity check
------------

555
Verify that the Tor Browser release used in Tails still is the most
556
557
558
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:
559

560
* <https://gitweb.torproject.org/builders/tor-browser-build.git>
561
562
* <https://gitweb.torproject.org/tor-browser.git>

563
A new tag may indicate that a new Tor Browser release or rebuild is imminent.
564
565
566

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
567
568
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
569

570
1. Install the almost final USB image to a USB stick.
571
572
573
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
1. Boot this USB stick **on bare metal** again.
574
1. Add `profile` to the kernel command-line.
575
1. Login with the default settings in the Welcome Screen (e.g. do not configure
576
   an _Administration Password_).
577
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
578
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
579
1. A few minutes later, once the `boot-profile` process has been
580
   killed, retrieve the new sort file from `/var/log/boot-profile`.
intrigeri's avatar
intrigeri committed
581
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
582
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
583
584
585
586
587
588
1. Remove runtime-generated files that don't exist in the rootfs,
   in order to avoid confusing noise in the build output:

           perl -ni -E 'chomp; say unless m{(?:
                  [.]pyc\s+\d+\z
               | \Alib/live/mount/medium/live/(?:filesystem[.]squashfs|initrd[.]img)\s
589
590
591
592
593
594
595
596
597
               | \Alib/live/mount/overlay/rw/etc/fstab\s
               | \Alib/live/mount/overlay/rw/etc/console-setup/cached_\S+[.](?:gz|sh)\s
               | \Alib/live/mount/overlay/rw/etc/machine-id\s
               | \Alib/live/mount/overlay/rw/etc/network/interfaces\s
               | \Alib/live/mount/overlay/rw/var/log/wtmp\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/apparmor[.]d/cache/[.]features\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/(?:group|gshadow|passwd|shadow)-\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/resolv-over-clearnet[.]conf\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/skel/[.]config/autostart/end-profile[.]desktop\s
598
               | \Arun/
intrigeri's avatar
intrigeri committed
599
               | \Avar/lib/AccountsService/users/Debian-gdm\s
600
               | \Avar/lib/gdm3/[#]\d+\s
601
602
603
604
605
               | \Avar/log/live/config[.]pipe\s
           )}xms' config/binary_rootfs/squashfs.sort

1. Remove the bits about `kill-boot-profile` at the end: they're
   only useful when profiling the boot.
intrigeri's avatar
intrigeri committed
606
1. Inspect the Git diff (including diff stat), apply common sense:
607

608
        diff -NaurB \
intrigeri's avatar
intrigeri committed
609
610
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
611
            | less
612

613
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`
614
1. Clean up: `rm -f config/binary_rootfs/squashfs.sort.old`
615

616
617
Build the final images
----------------------
618

Bessemer's avatar
Bessemer committed
619
Then all included files should be up-to-date and the versioned APT
620
621
suite should be ready, so it is time to:

622
1. Mark the version as "released" in the changelog:
623

624
        dch --release --no-force-save-on-release --maintmaint && \
625
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
626

627
1. Export `SOURCE_DATE_EPOCH`:
628

629
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
630

631
1. tag the release *again*, with all included files in:
632

633
634
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
635
636
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
637

638
639
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
640
   commands by e.g. committing a commit in between `git tag` and the
641
642
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
643

644
1. build the final images!
645
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
646
647
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.
648

intrigeri's avatar
intrigeri committed
649
650
651
652
1. Compare the new build manifest with the one from the previous,
   almost final build:

        diff -Naur \
653
           "${BUILD_MANIFEST:?}" \
654
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
intrigeri's avatar
intrigeri committed
655
656
657
658

   They should be identical, except that the `debian-security` serial might be higher.

1. To ensure we publish the final build's `.build-manifest`, run:
659

660
        export BUILD_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
661

intrigeri's avatar
intrigeri committed
662
<a id="reproducibility-sanity-check-iso"></a>
663

intrigeri's avatar
intrigeri committed
664
665
666
667
Verify that Jenkins reproduced your images
------------------------------------------

to verify that Jenkins reproduced your images:
668

intrigeri's avatar
intrigeri committed
669
1. Visit the URL printed by this command:
anonym's avatar
xxx    
anonym committed
670
671
672

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

intrigeri's avatar
intrigeri committed
673
2. Find the job (probably the last one)
674
675
   and make sure the ISO and USB images built by Jenkins
   have the same hash (in the `.shasum` file) as the images you built.
676

intrigeri's avatar
intrigeri committed
677
3. Then:
678

679
680
   - If the ISO and USB images hashes match: yay, we're good to go!
     The `.build-manifest` may differ — that's OK.
681

682
     Set the `$MATCHING_JENKINS_IMAGES_BUILD_ID` environment variable
683
684
     to the ID of this job (an integer).

685
   - If there is a hash mismatch for one of the images: ouch! Now we are in a
686
687
688
689
690
691
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
692

anonym's avatar
anonym committed
693
694
695
696
697
698
699
700
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
701

702
703
     Do the same for the USB image as well.

704
     Then carefully investigate the `diffoscope` report:
705

706
707
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
intrigeri's avatar
intrigeri committed
708
         release. Halt the release, involve the rest of <tails@boum.org>, and then
709
710
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
711

712
       - Otherwise, if the change is definitely harmless:
713

714
         * If the source of non-determinism is identified quickly and
715
           is easy and fast to fix, *and* the QA of the current images
716
717
718
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
719

720
721
722
723
724
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
725

726
727
728
729
730
731
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
           release notes, linking to the ticket where
           the nature of the reproducibility failure is clearly
           described.
732

intrigeri's avatar
intrigeri committed
733
734
Initialize the website release branch
-------------------------------------
intrigeri's avatar
intrigeri committed
735

736
737
738
739
740
741
From now on, we don't want to push new commits on `$RELEASE_BRANCH`
until the new release is out. Otherwise, this would break its build
and the build of every branch based on it, which would effectively
block other development work. So the final steps towards publishing
the release will be done in a new, dedicated branch.

intrigeri's avatar
intrigeri committed
742
743
If preparing anything but a final release (e.g. an alpha, beta
or RC):
744
745
746
747

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

intrigeri's avatar
intrigeri committed
748
Else, if preparing a final release:
749

750
751
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
752

753
754
755
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
756
Create a directory with a suitable name, go there, move the built
757
758
759
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
intrigeri's avatar
intrigeri committed
760
761
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
762

763
764
765
766
767
768
769
    mkdir -p "${ISOS:?}/tails-amd64-${VERSION:?}" && \
    for type in iso img ; do
       cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
       mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
       gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
       rename 's,\.asc$,.sig,' *.asc && \
       tmp="$(mktemp -d)" && \
770
771
       mkdir -p "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       cd "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
772
       for x in "${ISOS:?}/tails-amd64-${VERSION:?}"/*.${type:?}*; do
773
774
775
776
777
778
           ln -s ${x} .
       done && \
       mktorrent \
          -o "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" \
          -a 'udp://tracker.torrent.eu.org:451'   \
          -a 'udp://tracker.coppersurfer.tk:6969' \
779
          "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
780
       transmission-show "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" && \
781
       cd - && \
782
783
       rm -rf "${tmp:?}"
    done
784

anonym's avatar
anonym committed
785
786
Lastly, let's set some variables to be used later:

787
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
788
789
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
790
    IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.img"
791
792
    IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"
anonym's avatar
anonym committed
793

794
795
<a id="prepare-iuk"></a>

796
797
798
Prepare incremental upgrades
============================

799
800
801
802
Since Tails 4.2, we use a new upgrade scheme, which fundamentally
changes what the source version number of an upgrade means: it's now
the version that was *initially installed* and *not* the currently
running version. If this is news to you, see:
803

804
805
* the document that explains the benefits for our users:
  [[blueprint/Endless_upgrades]];
806

807
808
* the corresponding
  [[design documentation|contribute/design/incremental_upgrades]].
809

segfault's avatar
segfault committed
810
The main practical implications at release time are:
811

812
813
814
* The Release Manager has to publish more IUKs than they used to.
  But they can now publish IUKs (reproducibly) built on Jenkins,
  instead of having to upload those they've built locally.
815

816
* The Release Manager has to sign more UDFs than they used to.
817

818
819
820
821
822
823
824
825
826
827
* Computing `$IUK_SOURCE_VERSIONS` is now straightforward enough
  that it was automated :)

Prepare the environment
-----------------------

Compute the list of initial version install to build IUKs for:

    cd "${RELEASE_CHECKOUT:?}" && \
	export IUK_SOURCE_VERSIONS=$(./bin/iuk-source-versions ${VERSION:?})
828
    echo "${IUK_SOURCE_VERSIONS?:}"
829

830
831
832
Even if it's computed automatically, remember to store it in the file
holding environment variables, it will be used in various places below.

833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
Sanity checks
-------------

Check that you have the correct version of `squashfs-tools` installed:

    [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
      = '1:4.4-1+0.tails1' \
    ] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for <https://iso-history.tails.boum.org> (it
will be used by `wrap_tails_create_iuks` below, in order to download any ISO
that is not present locally yet):

    if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
        echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
        echo "The corresponding login and password are in the RMs' keyringer."
    fi

851
852
Build the Incremental Upgrade Kits locally
------------------------------------------
853

854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
    (
       set -eu
       WORK_DIR=$(mktemp -d)
       TAILS_REMOTE="$(git -C "$RELEASE_CHECKOUT" remote get-url origin)"
       PUPPET_TAILS_REMOTE=$(echo -n "$TAILS_REMOTE" | perl -p -E 's,:tails(:?[.]git)?\z,:puppet-tails,')
       cd "$WORK_DIR"
       git clone "$PUPPET_TAILS_REMOTE"
       ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
           --tails-git-remote "file://${RELEASE_CHECKOUT}/.git"             \
           --tails-git-commit "$TAG"                                      \
           --source-date-epoch "$SOURCE_DATE_EPOCH"                       \
           --local-isos-dir "$ISOS"                                       \
           --tmp-dir "${TMPDIR:-/tmp}"                                        \
           --output-dir "$IUKS_DIR"                                       \
           --source-versions "$IUK_SOURCE_VERSIONS"                       \
           --new-version "$VERSION"                                       \
           --verbose
       cd "$IUKS_DIR"
       sha256sum Tails_amd64_*_to_${VERSION}.iuk > "$IUKS_HASHES"
   )
874

875
This command takes a long time. In parallel, while it is running,
876
877
878
879
880
881
882
883
884
885
886
887
you can follow the next 2 steps:

 - Push the images to our ISO history git-annex repo
 - Build the Incremental Upgrade Kits on Jenkins

ISO history
-----------

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

intrigeri's avatar
intrigeri committed
888
889
 - The Jenkins `build_IUKs` job can fetch them.
 - Our isotesters can fetch them from there for their testing.
890
891
892
893
894
895

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

Then, wait (a few minutes) until the images appear
on <https://iso-history.tails.boum.org/>.
896
897
898
899
900
901
902
903
904
905

<a id="build-iuks-on-jenkins"></a>

Build the Incremental Upgrade Kits on Jenkins
------------------------------------------

1. On <https://jenkins.tails.boum.org/job/build_IUKs/build?delay=0sec>,
   fill the form with these values:

    - `TAILS_GIT_COMMIT`: the value of `$TAG` in your release environment
906
    - `SOURCE_DATE_EPOCH`: the value of `$SOURCE_DATE_EPOCH` in your
907
908
909
910
911
912
913
914
915
916
917
918
919
920
921
922
923
      release environment
    - `SOURCE_VERSIONS`: the value of `$IUK_SOURCE_VERSIONS` in your
      release environment
    - `NEW_VERSION`: the value of `$VERSION` in your release environment
    - `EXTRA_ARGS`: leave it blank

2. Click the _Build_ button

3. After a few seconds, a new build appears on top of the _Build
   History_ sidebar. Click on the progress bar of this new build.

4. Set the `$CANDIDATE_JENKINS_IUKS_BUILD_ID` environment variable
   to the ID of this job (an integer).

5. Wait until this *build_IUKs* job completes successfully.
   It should take about 10-15 minutes for each member of
   the `$IUK_SOURCE_VERSIONS` list.
924

925
926
<a id="reproducibility-sanity-check-iuk"></a>

927
928
929
930
931
Verify that Jenkins reproduced your IUKs
----------------------------------------

    "${RELEASE_CHECKOUT:?}"/bin/copy-iuks-to-rsync-server-and-verify \
       --hashes-file "${IUKS_HASHES:?}" \
932
       --work-dir /srv/tmp \
933
934
935
936
937
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
       --jenkins-build-id "${CANDIDATE_JENKINS_IUKS_BUILD_ID:?}"

If this verification succeeds, move on to the next section.

Else, if this verification fails, then:

1. Visit the web page whose URL is printed by the following command:

        echo "https://jenkins.tails.boum.org/job/build_IUKs/${CANDIDATE_JENKINS_IUKS_BUILD_ID:?}/parameters/"

   It tells you which parameters you've passed to the Jenkins job.

2. Double-check that you've passed the correct parameters.

   If you notice you made a mistake, [[build the IUKs on Jenkins
   again|release_process#build-iuks-on-jenkins]], and do the
   verification again.

   Else, if the parameters where correct, then follow the next steps.

3. File a ticket about this problem.

   Specify:

    - Which set of parameters you've passed to the *build_IUKs*
      job, so that the person who'll investigate the problem
      can reproduce it.
    - The ID of the build that failed to reproduce your
      locally-built IUKs.

4. Later on, after you're done with OpenPGP signing,
   you will upload the IUKs you've built locally.
965

Tails developers's avatar
Tails developers committed
966
<a id="prepare-upgrade-description-files"></a>
967

Tails developers's avatar
Tails developers committed
968
969
Prepare upgrade-description files
---------------------------------
970

Tails developers's avatar
Tails developers committed
971
1. Prepare upgrade-description files (see the [[upgrade-description
972
   files
Tails developers's avatar
Tails developers committed
973
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
974
975
976
977
978
979
980
981
982
983
984
985
986
987
988
989
   for details).

   <div class="caution">
   In <emph>most</emph> cases the example command below is exactly the
   one you should run. But in order to tell whether you're in one of
   the exceptional cases when you have to adjust that command, it's
   important that you understand what follows.
   </div>

   At this step, we use the
   `tails-iuk-generate-upgrade-description-files` tool in order to:

   1. Create a new upgrade-description for the version being released
      and for the next one, that expresses that *no* upgrade is
      available for these ones yet.

990
      This is what the `--version` and `--next_version` arguments
991
992
      in the example command below do. You do not need to modify them.

993
994
995
    2. For every recent previous release that's _not_ listed in
       `$IUK_SOURCE_VERSIONS` (i.e. those based on an older Debian
       release): tell the users of that old version they need to
996
       manually upgrade to the version being released.
997
998
999
1000

       To do so, pass this previous version to `--previous_version`.
       You need to do this manually as the example command below won't
       do it automatically.
1001
1002

   Run this command, after adjusting it if needed as explained above:
1003

1004
        ${RELEASE_CHECKOUT:?}/config/chroot_local-includes/usr/src/iuk/bin/tails-iuk-generate-upgrade-description-files \
1005
            --version "${VERSION:?}" \
1006
1007
1008
            $( \
                if [ "${DIST:?}" = stable ]; then
                    echo \
1009
1010
1011
1012
                        --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
                        --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}~rc1" \
                        --next_version "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
                        --next_version "${NEXT_POTENTIAL_EMERGENCY_VERSION:?}"
1013
                else
1014
1015
1016
                    echo \
                        --next_version $(echo ${VERSION:?} | sed -e 's,~.*$,,') \
                        --next_version "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}"
1017
1018
                fi
            ) \
1019
            $( \
1020
                for version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
1021
                   echo "--previous_version ${version:?}"
1022
1023
                done \
            ) \
anonym's avatar
anonym committed
1024
            --iso "${ISO_PATH:?}" \
1025
            --iuks "${IUKS_DIR:?}" \
1026
1027
            --release_checkout "${RELEASE_CHECKOUT:?}" \
            --major_release "${MAJOR_RELEASE:?}" \