release_process.mdwn 62.3 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
8
9
10
<div class="caution">
Read the remainder of this document from the branch used to prepare the release!
</div>

11
12
13
14
15
Requirements
============

To release Tails you'll need some packages installed:

16
* `tidy mktorrent transmission-cli`
intrigeri's avatar
intrigeri committed
17
* aufs DKMS module for your running kernel.
intrigeri's avatar
intrigeri committed
18
* [[!debpts squashfs-tools]] that honors `$SOURCE_DATE_EPOCH`.
19
  Install it from our custom `devel` APT suite.
intrigeri's avatar
intrigeri committed
20
* `tails-iuk` dependencies, including suggested packages (see
21
22
23
  `debian/control` in the `debian` branch of its repo)
* `tails-perl5lib` dependencies (same trick as `tails-iuk` to get the
  list)
24
25
* `po4a` _from Stretch_: the version in testing/sid extracts Markdown headings
   in a different way, which makes tons of strings fuzzy.
26

27
28
29
Environment
===========

30
31
32
33
34
35
36
To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the
value of one such variable, treat it as a constant: else,
inconsistency will surely arise, which can cause trouble later on.

37
38
39
Version numbers
---------------

40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
Note:

* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

Export the following environment variables:
57

58
* version numbers (see [[contribute/release_schedule#versioning]]):
59

60
61
62
63
        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
64

65
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
66
67
68
  *major* Tails release; if you're preparing a RC for a major release,
  use that major release; otherwise, use whatever the next planned
  major release is
69
70
* `SECOND_NEXT_PLANNED_MAJOR_VERSION`: if you're preparing the RC
  for a major release, set this to the version number of
71
72
73
74
  the second next *major* Tails release; e.g. if preparing the RC for
  the 3.9 major release, then set this to 3.12 (3.9 is the next major
  release, 3.10 and 3.11 are bugfix releases, 3.12 is a major
  release).
75
76
* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
  *bugfix* Tails release; if the next release is a bugfix release, use
77
  that one; otherwise, use `${VERSION}.1`
78
79
* `NEXT_POTENTIAL_EMERGENCY_VERSION`: set to the version number we'll give
  to the next emergency release if we have to put one out
80
81
82
83
84
85

Other variables
---------------

Also export the following environment variables:

86
87
* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
88
* `ISOS`: the directory where one stores `tails-amd64-*`
89
  sub-directories like the ones downloaded with BitTorrent.
90
91
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
92
93
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
intrigeri's avatar
intrigeri committed
94
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
95
96
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
97
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
98
* `TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}"perl -nE 'say substr($_, -17)')`
99
100
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
101
102
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
103
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
104
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
105

106
107
108
Pre-freeze
==========

109
110
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
111
112
113
114
115
116

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

117
118
119
120
121
122
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

123
124
125
126
127
128
129
130
131
132
Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

133
        git checkout devel && git fetch origin && git merge --no-ff origin/master
134

intrigeri's avatar
intrigeri committed
135
136
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
137
138
139
140
141
142
143
144
145
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

146
147
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
148

anonym's avatar
anonym committed
149
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
150
151
152
153
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
154
155
156
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
157
158
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

159

160
161
Bugfix release
--------------
162

163
If we are at freeze time for a bugfix release:
164
165
166

1. Merge the `master` Git branch into `stable`:

167
        git checkout stable && git fetch && git merge --no-ff origin/master
168

intrigeri's avatar
intrigeri committed
169
170
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
171
172
   APT suite.

173
174
Common steps for bugfix and major releases
------------------------------------------
175

176
Reset the release branch's `config/base_branch`:
177

178
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
179
           git commit config/base_branch \
180
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
181

182
183
184
185
186
187
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
3. Send the pad URL to the usual testers (see `manual_testers.mdwn` in
188
   the RM team's Git repository), end-to-end encrypted.
189

Tails developers's avatar
Tails developers committed
190
191
192
Update included files
=====================

193
194
<a id="upgrade-custom-debs"></a>

Bessemer's avatar
Bessemer committed
195
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
196
197
--------------------------------------

198
Skip this section if you are preparing a bugfix release.
199
200
201

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:
202

203
204
205
206
207
208
209
 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
 - If you are preparing a major release, build at least the packages
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)
210

211
212
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
intrigeri's avatar
intrigeri committed
213
import translations from Transifex and sanity-check them:
214
215

	cd whisperback
intrigeri's avatar
intrigeri committed
216
	"${RELEASE_CHECKOUT:?}"/import-translations && \
217
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po
218

intrigeri's avatar
intrigeri committed
219
Then, `git rm` the PO files that have issues (alternatively, if you
220
221
feel like it you can fix them but your changes will be overwritten
next time we import translations from Transifex).
222

223
And finally, commit:
224

225
    git add po && git commit \
226
	    -m "Update POT and PO files, pull updated translations from Transifex."
227

228
Then see the relevant release processes, and upload the packages to
229
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
230

231
* [[tails-installer]]
232
* [[tails-greeter]]
233
* [[perl5lib]]
234
* [[persistence-setup]]
235
* [[tails-iuk]]
236
* whisperback:
237
238
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package
Tails developers's avatar
Tails developers committed
239

intrigeri's avatar
intrigeri committed
240
241
Upgrade Tor Browser
-------------------
242
243
244

See the dedicated page: [[tor-browser]]

245
246
247
Upgrade Tor Browser AppArmor profile
------------------------------------

248
See the dedicated page: [[tor-browser_AppArmor_patch]]
249

250
Upgrade Thunderbird
251
252
---------------

253
See the dedicated page: [[thunderbird]]
254

255
256
257
258
259
Upgrade custom packages for VeraCrypt integration
-------------------------------------------------

See the dedicated page: [[veracrypt]]

260
261
Update PO files
---------------
262

263
264
265
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
266

267
	cd "${RELEASE_CHECKOUT:?}" && \
268
269
	./import-translations  && \
	./refresh-translations && \
270
	./submodules/jenkins-tools/slaves/check_po && \
271
	git add po && git commit -m 'Update PO files.'
272

273
274
275
276
277
278
If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

279
280
When preparing an actual release
================================
281

282
If we're about to prepare the images for a final (non-RC) release, then
283
284
285
286
287
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
288
289
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
290
custom APT suite.
291

292
293
Bugfix release
--------------
294

295
<div class="note">
296
For bugfix releases, we generally do not put any RC out, so freeze time
297
298
299
300
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
301

intrigeri's avatar
intrigeri committed
302
303
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
304
custom APT suite.
305

anonym's avatar
anonym committed
306
307
Update other base branches
==========================
308
309

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
310
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
311

312
2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
intrigeri's avatar
intrigeri committed
313
   branch, the time-based APT repository snapshots that were used
314
315
   during the freeze. It's fine if that results in a no-op
   (it depends on how exactly previous operations were performed).
intrigeri's avatar
intrigeri committed
316
317

3. Merge `devel` into `feature/buster`, *without* following the instructions for
intrigeri's avatar
intrigeri committed
318
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
319
320
   (For now `feature/buster` is handled as any other topic branch
   forked off `devel`: its base branch is set to `devel`.)
321
322
323
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.
324

intrigeri's avatar
intrigeri committed
325
4. Ensure that the release, `devel` and `feature/buster` branches
326
327
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
328

329
5. Push the modified branches to Git:
330

331
        git push origin                          \
332
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
333
           feature/buster:feature/buster       \
334
           devel:devel
335
336
337

Update more included files
==========================
338

339
340
Changelog
---------
Tails developers's avatar
Tails developers committed
341

342
343
344
Remove the placeholder entry for next release in `debian/changelog`,
and then:

345
	git checkout "${RELEASE_BRANCH:?}" && \
346
	DEBEMAIL='tails@boum.org' DEBFULLNAME='Tails developers' \
347
	./release ${VERSION:?} ${PREVIOUS_TAG:?}
Tails developers's avatar
Tails developers committed
348
349
350

This populates the Changelog with the Git log entries.

351
352
353
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
354

355
356
357
358
359
360
361
362
363
Changelog entries can be dispatched into those usual sections:

 * Major changes
 * Security fixes
 * Bugfixes
 * Minor improvements and updates
 * Build system
 * Test suite

364
365
366
367
Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
368
* the diff between the previous version's `.packages` file and the one
369
  from the to-be-released images; look for:
370
371
372
373
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
374
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
375
  in Redmine.
376

377
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
378

379
380
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
381
	else
382
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
383
	fi
Tails developers's avatar
Tails developers committed
384

Bessemer's avatar
Bessemer committed
385
Included website
386
387
----------------

Tails developers's avatar
Tails developers committed
388
389
390
391
392
393
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

394
395
### version number

396
397
If preparing a RC, skip this part.

398
In the branch used to build the release, update the `wiki/src/inc/*` files to
399
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
400
at least 24 hours in the future! Between tests and mirror synchronization,
401
the build will not be released on the same day. Try to make sure it
402
matches the date of the future signature.
403

sajolida's avatar
sajolida committed
404
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
405

406
407
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html
408
409
410
	for type in img iso; do
	   basename="tails-amd64-${VERSION:?}"
	   filename="${basename:?}.${type:?}"
411
	   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
412
413
414
415
416
417
418
419
	        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
	   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
	        > wiki/src/inc/stable_amd64_${type:?}_url.html
	   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
	        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html
	   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
	        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
	done
elouann's avatar
elouann committed
420
	./build-website
421
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
422

423
424
Website translations
--------------------
425

426
427
428
429
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
430

elouann's avatar
elouann committed
431
	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
432
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
433

434
435
436
Call for translation
====================

437
438
439
440
441
442
If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].
443

444
To get a list of changes on the website:
445

intrigeri's avatar
intrigeri committed
446
    git diff --stat ${PREVIOUS_TAG:?}.. -- \
447
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
448
449
450
451
452
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
453

454
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
455
456
======================

457
458
459
460
461
462
463
464
465
466
467
468
### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
pro tip is to never plug it unless prompted which `gpg` will do for
you. Then just unplug it as soon as the `.sig` is done.

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
469

Tails developers's avatar
Tails developers committed
470
You should never import the Tails signing key into your own keyring,
471
472
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
473
474

    export GNUPGHOME=$(mktemp -d)
475
476
477
478
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
479
480
481
482
483
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

484
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
485

486
487
Build the almost-final images
=============================
488

489
1. [[Build ISO and USB images|contribute/build]] from the release branch.
490
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
491
2. Carefully read the build logs to make sure nothing bad happened.
492
3. Keep the resulting build artifacts until the end of this release process.
493
494
495
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
496
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"
497
498


499
500
501
Tag the release in Git
======================

502
503
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
504
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
505
506
507
508
509
510
511

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

512
513
514
515
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

516
517
518
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
519
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
520

521
* Prepare tagged snapshots of upstream APT repositories:
522

523
          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"
524

525
  Note:
526

527
528
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
529
    [[custom APT repository|APT_repository/custom]] are
530
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
531
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
532
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
533
    Now, we have a "safety" net, in case you don't notice such a problem: if
534
535
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
536

Tails developers's avatar
Tails developers committed
537
538
539
Build images
============

540
541
542
Sanity check
------------

543
Verify that the Tor Browser release used in Tails still is the most
544
545
546
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:
547

548
* <https://gitweb.torproject.org/builders/tor-browser-build.git>
549
550
* <https://gitweb.torproject.org/tor-browser.git>

551
A new tag may indicate that a new Tor Browser release or rebuild is imminent.
552
553
554

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
555
556
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
557

558
1. Install the almost final USB image to a USB stick.
559
560
561
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
1. Boot this USB stick **on bare metal** again.
562
1. Add `profile` to the kernel command-line.
563
564
1. Login with the default settings in the Greeter (e.g. do not configure
   an _Administration Password_).
565
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
566
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
567
1. A few minutes later, once the `boot-profile` process has been
568
   killed, retrieve the new sort file from `/var/log/boot-profile`.
intrigeri's avatar
intrigeri committed
569
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
570
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
571
572
573
574
575
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
intrigeri's avatar
intrigeri committed
576
1. Inspect the Git diff (including diff stat), apply common sense:
577

578
        diff -NaurB \
intrigeri's avatar
intrigeri committed
579
580
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
581
            | less
582

583
584
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

585
586
Build the final images
----------------------
587

Bessemer's avatar
Bessemer committed
588
Then all included files should be up-to-date and the versioned APT
589
590
suite should be ready, so it is time to:

591
1. Mark the version as "released" in the changelog:
592

593
        dch --release --no-force-save-on-release --maintmaint && \
594
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
595

596
1. Export `SOURCE_DATE_EPOCH`:
597

598
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
599

600
1. tag the release *again*, with all included files in:
601

602
603
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
604
605
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
606

607
608
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
609
   commands by e.g. committing a commit in between `git tag` and the
610
611
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
612

613
1. build the final images!
614
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
615
616
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.
617

intrigeri's avatar
intrigeri committed
618
619
620
621
622
1. Compare the new build manifest with the one from the previous,
   almost final build:

        diff -Naur \
           "${PACKAGES_MANIFEST:?}" \
623
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
intrigeri's avatar
intrigeri committed
624
625
626
627

   They should be identical, except that the `debian-security` serial might be higher.

1. To ensure we publish the final build's `.build-manifest`, run:
628

629
        export PACKAGES_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
630

631

632
1. <a id="reproducibility-sanity-check-iso"></a>
633
   Let's sanity check that Jenkins reproduced your images.
634

anonym's avatar
xxx    
anonym committed
635
636
637
638
639
   Visit the URL printed by this command:

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

   Find the job (probably the last one)
640
641
   and make sure the ISO and USB images built by Jenkins
   have the same hash (in the `.shasum` file) as the images you built.
642

643
   Then:
644

645
   - If all hashes match: yay, we're good to go!
646

647
648
649
     Set the `$MATCHING_JENKINS_BUILD_ID` environment variable
     to the ID of this job (an integer).

650
   - If there is a hash mismatch for one of the images: ouch! Now we are in a
651
652
653
654
655
656
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
657

anonym's avatar
anonym committed
658
659
660
661
662
663
664
665
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
666

667
668
     Do the same for the USB image as well.

669
     Then carefully investigate the `diffoscope` report:
670

671
672
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
intrigeri's avatar
intrigeri committed
673
         release. Halt the release, involve the rest of <tails@boum.org>, and then
674
675
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
676

677
       - Otherwise, if the change is definitely harmless:
678

679
         * If the source of non-determinism is identified quickly and
680
           is easy and fast to fix, *and* the QA of the current images
681
682
683
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
684

685
686
687
688
689
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
690

691
692
693
694
695
696
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
           release notes, linking to the ticket where
           the nature of the reproducibility failure is clearly
           described.
697

698
1. check out a new branch:
intrigeri's avatar
intrigeri committed
699

700
701
702
703
704
705
706
707
   If preparing anything but a final release (e.g. an alpha, beta
   or RC):

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

   Else, if preparing a final release:

708
709
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
710

711
   (as soon as a new commit is created on `$RELEASE_BRANCH`, its
anonym's avatar
anonym committed
712
713
714
   build will start failing until a new changelog entry is created,
   which we don't want to do on `$RELEASE_BRANCH` before it's merged
   into `master` at release time)
anonym's avatar
anonym committed
715

716
717
718
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
719
Create a directory with a suitable name, go there, move the built
720
721
722
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
intrigeri's avatar
intrigeri committed
723
724
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
725

726
	for type in iso img ; do
727
728
	   mkdir "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}" && \
	   cd "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}" && \
intrigeri's avatar
intrigeri committed
729
	   mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
730
	   gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
731
732
733
	   rename 's,\.asc$,.sig,' *.asc && \
	   cd .. && \
	   mktorrent \
intrigeri's avatar
intrigeri committed
734
735
	      -a 'udp://tracker.torrent.eu.org:451'   \
	      -a 'udp://tracker.coppersurfer.tk:6969' \
736
	      "tails-amd64-${VERSION:?}.${type:?}" && \
737
	   transmission-show tails-amd64-${VERSION:?}.${type:?}.torrent
738
	done
739

anonym's avatar
anonym committed
740
741
Lastly, let's set some variables to be used later:

742
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}.iso/tails-amd64-${VERSION:?}.iso"
743
744
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
745
    IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}.img/tails-amd64-${VERSION:?}.img"
746
747
    IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"
anonym's avatar
anonym committed
748

749
750
<a id="prepare-iuk"></a>

751
752
753
Prepare incremental upgrades
============================

754
755
Build the Incremental Upgrade Kits
----------------------------------
756

757
758
759
760
Incremental upgrades may be skipped if the delta is too big (like when
migrating to a new Debian release) or if there are changes outside of
the scope for IUKs (like partition table changes). Use common sense!

761
762
Use `tails-create-iuk` to build the following IUKs:

763
764
765
766
* From the two previous *planned* releases, and any emergency releases
  in between and after. This should be, more or less, all releases for
  the last 12 weeks (although irregularities in Firefox release
  schedule may add or remove a few weeks).
767
768
769
770
771

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
772
  next version.
773

774
Include each such version in a white-space separated list called
775
`IUK_SOURCE_VERSIONS`, (e.g. `IUK_SOURCE_VERSIONS="2.8 2.9 2.9.1 2.10~rc1"`), optionally setting `TMPDIR` to an existing absolute path where disk space is available,
776
and run the following:
777

778
    for source_version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
779
780
781
782
        if [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" != 1:4.3-3.0tails4 ]; then
            echo 'ERROR! Your squashfs-tools probably does not honor SOURCE_DATE_EPOCH so any generated IUKs will *not* be reproducible!'
            break
        fi
783
        echo "Generating IUK file from ${source_version:?} to ${VERSION:?}"
784
        sudo su -c "cd ${IUK_CHECKOUT:?} && \
785
786
          SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH \
          LC_ALL=C \
787
          TMPDIR=\"${TMPDIR:-/tmp}\" \
788
789
790
          PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
            ./bin/tails-create-iuk \
               --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
791
792
               --old-iso \"${ISOS:?}/tails-amd64-${source_version:?}.iso/tails-amd64-${source_version:?}.iso\" \
               --new-iso \"${ISOS:?}/tails-amd64-${VERSION:?}.iso/tails-amd64-${VERSION:?}.iso\"          \
793
               --outfile \"${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk\""
794
    done
795
796

Note that developer tools for creating IUK and upgrade-description
797
798
799
files are primarily developed and tested on Debian sid. They may
therefore occasionally be broken on Debian stable. As of December 2018
they work fine on Debian Stretch.
800

801
802
803
804
<a id="reproducibility-sanity-check-iuk"></a>

Note that we do not yet build IUKs on Jenkins, otherwise here would be
a great point to compare its IUKs with yours.
805

Tails developers's avatar
Tails developers committed
806
<a id="prepare-upgrade-description-files"></a>
807

Tails developers's avatar
Tails developers committed
808
809
Prepare upgrade-description files
---------------------------------
810

Tails developers's avatar
Tails developers committed
811
1. Prepare upgrade-description files (see the [[upgrade-description
812
   files
Tails developers's avatar
Tails developers committed
813
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
814
815
816
817
818
819
820
821
822
823
824
825
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854
855
856
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
   for details).

   <div class="caution">
   In <emph>most</emph> cases the example command below is exactly the
   one you should run. But in order to tell whether you're in one of
   the exceptional cases when you have to adjust that command, it's
   important that you understand what follows.
   </div>

   At this step, we use the
   `tails-iuk-generate-upgrade-description-files` tool in order to:

   1. Create a new upgrade-description for the version being released
      and for the next one, that expresses that *no* upgrade is
      available for these ones yet.

      This is what the `--version` and `--next-version` arguments
      in the example command below do. You do not need to modify them.

   2. For every previous release listed in `$IUK_SOURCE_VERSIONS`,
      i.e. those that will have a direct incremental upgrade path to
      the version being released: create or update its
      upgrade-description file to advertise this direct incremental
      upgrade path.

      This is what the `--previous-version` arguments generated by the
      example command below do. You do not need to modify that part.

   3. For every recently supported previous release that's _not_
      listed in `$IUK_SOURCE_VERSIONS` (i.e. releases that have no
      _direct_ incremental upgrade path to the version being
      released):

       - If there's a _multi-step incremental upgrade path_ from this
         previous version to the version being released (e.g. by first
         upgrading to a version that's in `$IUK_SOURCE_VERSIONS`): tell
         the users of that old version to go through this multi-step
         incremental upgrade path, i.e. leave its upgrade-description
         file unmodified (it already advertises the first step of
         this multi-step path).

         To do so, ensure you do not manually pass this previous
         release to `--previous-version`. The example command below
         does the right thing in this respect already, so this
         constraint only matters if you have other reasons to modify
         the command.

       - Else, if there's _no incremental upgrade path, be it direct
         or multi-step_, from this previous version to the version
         being released (which happens for example when releasing
         Tails based on a new version of Debian and when we
         unintentionally broke incremental upgrades recently): tell the users
         of that old version they need to manually upgrade to the version
         being released.

         To do so, pass this previous version to `--previous-version`.
         You need to do this manually as the example command below
         won't do it automatically.

   Run this command, after adjusting it if needed as explained above:
874

875
876
877
878
        ( cd ${IUK_CHECKOUT:?} && \
        ./bin/tails-iuk-generate-upgrade-description-files \
            --build-target amd64 \
            --version "${VERSION:?}" \
879
880
881
882
883
884
885
886
887
888
889
            $( \
                if [ "${DIST:?}" = stable ]; then
                    echo \
                        --next-version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
                        --next-version "${NEXT_PLANNED_MAJOR_VERSION:?}~rc1" \
                        --next-version "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
                        --next-version "${NEXT_POTENTIAL_EMERGENCY_VERSION:?}" \
                else
                    echo --next-version $(echo ${VERSION:?} | sed -e 's,~.*$,,')
                fi
            ) \
890
            $( \
891
                for version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
892
893
894
                   echo "--previous-version ${version:?}"
                done \
            ) \
anonym's avatar
anonym committed
895
            --iso "${ISO_PATH:?}" \
896
897
898
            --iuks "${ISOS:?}" \
            --release-checkout "${RELEASE_CHECKOUT:?}" \
            --major-release "${MAJOR_RELEASE:?}" \
899
            --channel "${DIST:?}"
900
       )
901
902
903
904

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

905
        cd "${RELEASE_CHECKOUT:?}" && \
906
907
908
909
        find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
           -type f -name upgrades.yml | \
           while read udf; do
               if [ -n "$(git status --porcelain "${udf:?}")" ]; then
910
911
912
913
914
                   for x in 1 2 3; do
                       gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor \
                           --detach-sign "${udf:?}" \
                       && break
                   done
915
916
917
918
919
920
921
                   mv --force "${udf:?}.asc" "${udf:?}.pgp"
                   ( \
                     cd ${IUK_CHECKOUT:?} && \
                     ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
                   ) || break
               fi
           done
922

Tails developers's avatar
Tails developers committed
923
1. Add and commit the upgrade-description files and their detached
924
925
   signatures to the Git branch used to prepare the release
   (`$WEBSITE_RELEASE_BRANCH`):
Tails developers's avatar
Tails developers committed
926

927
928
929
930
931
        ( \
          cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
          git commit -m "Update upgrade-description files." && \
          git push origin ${WEBSITE_RELEASE_BRANCH:?} \
        )
Tails developers's avatar
Tails developers committed
932

933
934
935
1. Copy the generated UDFs for the previous releases to the *test*
   channel in `$MASTER_CHECKOUT`, modify their content accordingly,
   sign them, commit and push:
Tails developers's avatar
Tails developers committed
936

intrigeri's avatar
intrigeri committed
937
938
939
        ( \
          cd ${MASTER_CHECKOUT:?} && \
          git fetch && \
940
          git merge origin/master && \
941
          for old_version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
942
            release_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/${DIST:?}/upgrades.yml" && \
intrigeri's avatar
intrigeri committed
943
944
            test_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/test/upgrades.yml" && \
            mkdir -p "$(dirname "$test_udf")" && \
945
946
            git show origin/${WEBSITE_RELEASE_BRANCH:?}:${release_udf:?} \
              | sed -e "s/channel: ${DIST:?}/channel: test/" > ${test_udf:?} && \
intrigeri's avatar
intrigeri committed
947
948
949
950
951
952
953
954
            gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${test_udf:?} && \
            mv ${test_udf:?}.asc ${test_udf:?}.pgp && \
            git add ${test_udf:?}* ; \
          done && \
          git commit -m "Add incremental upgrades on the test channel for Tails ${VERSION:?}" && \
          git push origin master:master \
        )

955
956
Prepare the image description file for *Tails Verification*
-----------------------------------------------------------
957

anonym's avatar
anonym committed
958
959
If preparing a RC, skip this part.

960
Update the image description file (IDF) used by the browser extension:
961

962
963
964
    ./bin/idf-content \
       --version "${VERSION:?}" \
       --iso "${ISO_PATH:?}" \
965
       --img "${IMG_PATH:?}" \
966
       > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v2/Tails/amd64/stable/latest.json && \
967
    ( cd "${RELEASE_CHECKOUT:?}" && \
968
      git add wiki/src/install/v2/Tails/amd64/stable/latest.json && \
969
      git commit -m "Update IDF file for Tails Verification." )
970

971
972
973
Done with OpenPGP signing
=========================

974
975
Unplug your OpenPGP smartcard and store it away, so you don't plug it
back semi-mechanically later on.
976

977
978
**Beware!** If your have to plug your OpenPGP smart card or reassemble the
key again after this point it invalidates *everything* done for
979
980
981
982
983
984
985
986
987
988
the [[reproduction of this release|test#reproducibility-final-check]]
so it has to be started from the beginning:

* the original text is restored on the pad, and
* some tester follows it from scratch, and
* the Trusted Reproducer follows awaits the new input from said tester
  and then starts from scratch.

So please try to avoid this!

Tails developers's avatar
Tails developers committed
989
990
991
Upload images
=============

Tails developers's avatar
Tails developers committed
992
993
994
Sanity check
------------

995
Verify once more that the Tor Browser we ship is still the most recent (see
996
above).
Tails developers's avatar
Tails developers committed
997

998
999
<a id="publish-iuk"></a>

intrigeri's avatar
intrigeri committed
1000
1001
Publish the ISO, IMG, and IUKs over HTTP
----------------------------------------
Tails developers's avatar
Tails developers committed
1002

1003
Upload the IUKs to our rsync server:
intrigeri's avatar
intrigeri committed
1004

1005
    for source_version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
intrigeri's avatar
intrigeri committed
1006
1007
1008
1009
1010
       rsync --partial --inplace --progress -v \
          "${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk" \
          rsync.lizard:
    done

1011
1012
While waiting for the IUKs to be uploaded, you can proceed with the next steps.

1013
Upload the ISO and USB image signatures to our rsync server:
1014

1015
    scp "${ISO_PATH:?}.sig" "${IMG_PATH:?}.sig" rsync.lizard:
1016

1017
Copy the ISO and USB images to our rsync server, verify their signature,
1018
1019
1020
move them in place with proper ownership and permissions
and update the time in `project/trace` file on our rsync server
and on the live website (even for a release candidate):
1021
1022
1023
1024
1025

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh rsync.lizard gpg --import
    ssh rsync.lizard << EOF
       wget \
1026
1027
1028
1029
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.img" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,} && \
       gpg --verify tails-amd64-${VERSION:?}.img{.sig,}
1030
    EOF
intrigeri's avatar
intrigeri committed
1031

1032
    ssh rsync.lizard << EOF
1033
1034
      sudo install -o root -g rsync_tails -m 0755 -d \
         /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
1035
1036
1037
      sudo chown root:rsync_tails tails-amd64-${VERSION:?}.{iso,img}* && \
      sudo chmod u=rwX,go=rX tails-amd64-${VERSION:?}.{iso,img}* && \
      sudo mv tails-amd64-${VERSION:?}.{iso,img}* \