release_process.mdwn 52.5 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
8
9
10
11
Requirements
============

To release Tails you'll need some packages installed:

12
* `tidy mktorrent transmission-cli`
13
* Aufs DKMS module for your running kernel.
intrigeri's avatar
intrigeri committed
14
* [[!debpts squashfs-tools]] that honors `$SOURCE_DATE_EPOCH`.
15
  Install it from our custom `devel` APT suite.
intrigeri's avatar
intrigeri committed
16
* `tails-iuk` dependencies, including suggested packages (see
17
18
19
  `debian/control` in the `debian` branch of its repo)
* `tails-perl5lib` dependencies (same trick as `tails-iuk` to get the
  list)
20

21
22
23
24
25
26
Environment
===========

Export the following environment variables to be able to copy'n'paste
the scripts snippets found on this page:

27
* version numbers (see [[contribute/release_schedule#versioning]]):
28

29
30
31
32
        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
33

34
* `NEXT_PLANNED_VERSION`: set to the version number of the next Tails release
35
  (e.g. 0.23 when releasing 0.22.1, and 1.3 when releasing 1.2)
36
37
38
39
40
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
  *major* Tails release
* `NEXT_PLANNED_MINOR_VERSION`: set to the version number of the next
  *minor* Tails release; if the next release is a point-release, use
  that one; otherwise, use `${VERSION}.1`
41
* `MAJOR_RELEASE`: set to 1 if preparing a major release, to 0 else
42
* `ISOS`: the directory where one stores `tails-amd64-*`
43
  sub-directories like the ones downloaded with BitTorrent.
44
45
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
46
47
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
48
49
* `RELEASE_BRANCH`: the name of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
50
51
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
52
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
53
54
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
55
56
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
57
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
intrigeri's avatar
intrigeri committed
58
59
* `export DEBEMAIL='tails@boum.org'`
* `export DEBFULLNAME='Tails developers'`
60
* `export WEBSITE_RELEASE_BRANCH="web/release-${VERSION:?}"`
61

62
63
64
Pre-freeze
==========

65
66
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
67
68
69
70
71
72

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

73
74
75
Select the right branch
=======================

76
77
What we refer to as the "release branch" (and `RELEASE_BRANCH`) should
be `testing` for major releases, and `stable` for point-releases.
78

79
<div class="caution">
80
Read the remainder of this document from the branch used to prepare the release!
81
82
</div>

83
84
85
86
87
88
89
90
91
92
93
94
Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
95
96
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
97
98
99
100
101
102
103
104
105
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

106
107
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
108

anonym's avatar
anonym committed
109
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
110
111
112
113
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
114
115
116
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
117
118
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

119
120
121
122
123
124
125
126
127
128

Point-release
-------------

If we are at freeze time for a point-release:

1. Merge the `master` Git branch into `stable`:

        git checkout stable && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
129
130
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
131
132
133
134
135
   APT suite.

Common steps for point and major releases
-----------------------------------------

136
Reset the release branch's `config/base_branch`:
137

138
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
139
           git commit config/base_branch \
140
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
141

142
143
144
145
146
147
148
149
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
3. Send the pad URL to the usual testers (see `manual_testers.mdwn` in
   the `internal.git` repository).

Tails developers's avatar
Tails developers committed
150
151
152
Update included files
=====================

spriver's avatar
spriver committed
153
uBlock patterns and settings file
Tails developers's avatar
Tails developers committed
154
155
----------------

spriver's avatar
spriver committed
156
157
The patterns+settings file is stored as a converted sqlite text dump in
`config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`.
Tails developers's avatar
Tails developers committed
158

159
1. Boot Tails
spriver's avatar
spriver committed
160
161
162
163
2. Start the Tor Browser and open the uBlock dashboard by clicking on the uBlock icon.
3. Select the tab *3rd-party filters*
4. Click on the button *Update now* to update all used patterns
5. Close the Tor Browser
164
165
166
167
7. Copy the `.tor-browser/profile.default/extension-data/ublock0.sqlite`
   from this Tor Browser instance into the root of Tails' Git repo and
   run the following command:

168
169
170
171
172
173
174
175
176
177
178
179
180
181
        # For posterity: the general idea is to introduce \r\n as a
        # token where we have made a line break to make the dump more
        # diff-friendly (and, hence, Git-friendly). The most complex
        # expression is the one done with perl, where we employ
        # negative lookahead. What it means, is: replace single
        # occurrences of | except when followed by \\n.
        echo '.dump' | sqlite3 ublock0.sqlite | \
            grep -v "cached_asset_content://cache://compiled-" | \
            awk '!/^INSERT/; /^INSERT/ {print $0 | "sort -n"}' | \
            sed 's_\\n_\\n\r\n_g' | \
            sed 's_,_,\r\n_g' | \
            perl -p -e 's/([^|])\|((?!\||\\n).)/\1\|\r\n\2/g' | \
            sed "/^INSERT INTO \"settings\" VALUES('\(remoteBlacklists\|cached_asset_entries\)'/"'s_,_,\r\n_g' > \
            config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump
182

183
8. Commit:
Tails developers's avatar
Tails developers committed
184

185
186
187
        git commit -m 'Update uBlock Origin patterns + settings file.' \
           config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump \
        && rm ublock0.sqlite
188

Bessemer's avatar
Bessemer committed
189
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
190
191
--------------------------------------

192
193
194
Skip this section unless we are at freeze time for a major release
(i.e. we are about to prepare a release candidate).

195
That is: make sure the bundled binary Debian packages contain
196
197
up-to-date localization files.

198
199
200
201
202
203
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
and then run the `import-translations` script that is in the
main Tails repository. For example:

	cd whisperback
204
	"${RELEASE_CHECKOUT:?}"/import-translations
205

WinterFairy's avatar
WinterFairy committed
206
207
If the `import-translations` script fails to import translations for
the current package, manually copy updated PO files from the
208
Transifex branches of `git://git.torproject.org/translation.git` (e.g.
209
210
`whisperback_completed`) instead. In this case, skip PO files for
[[translation teams that use Git|contribute/how/translate#translate]].
211

212
Add and commit.
213

214
215
Then check the PO files:

216
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po
217
218

Correct any displayed error, then commit the changes if any.
219

220
Then see the relevant release processes, and upload the packages to
221
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
222

223
* [[tails-installer]]
224
* [[tails-greeter]]
225
* [[perl5lib]]
226
* [[persistence-setup]]
227
* [[tails-iuk]]
228
* whisperback:
229
230
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package
Tails developers's avatar
Tails developers committed
231

intrigeri's avatar
intrigeri committed
232
233
Upgrade Tor Browser
-------------------
234
235
236

See the dedicated page: [[tor-browser]]

237
238
239
240
241
Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[browser-apparmor-patch]]

242
Upgrade Thunderbird
243
244
---------------

245
See the dedicated page: [[thunderbird]]
246

247
248
Update PO files
---------------
249

250
251
252
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
253

254
	cd "${RELEASE_CHECKOUT:?}" && \
255
256
	./import-translations  && \
	./refresh-translations && \
257
	./submodules/jenkins-tools/slaves/check_po && \
258
	git add po && git commit -m 'Update PO files.'
259

260
261
262
263
264
265
If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

266
267
When preparing an actual release
================================
268

269
270
271
272
273
274
If we're about to prepare an image for a final (non-RC) release, then
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
275
276
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
277
custom APT suite.
278
279
280
281

Point-release
-------------

282
283
284
285
286
287
<div class="note">
For point-releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
288

intrigeri's avatar
intrigeri committed
289
290
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
291
custom APT suite.
292

anonym's avatar
anonym committed
293
294
Update other base branches
==========================
295
296

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
297
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
298

299
2. Merge `devel` into `feature/buster` following the instructions for
intrigeri's avatar
intrigeri committed
300
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
301
302
303
   Given that these two branches' APT suites have diverged a lot, and
   that `tails-merge-suite` currently happily overwrites newer
   packages in the target with older packages from the source, it's
anonym's avatar
anonym committed
304
   probably easier to just merge each individual APT overlay that was
305
   just merged into the release branch into `feature/buster`'s APT
306
   suite. Also, most of our just upgraded bundled packages
307
   (e.g. `tails-greeter`) may need to be rebuilt for Stretch.
308

309
3. Ensure that the release, `devel` and `feature/buster` branches
310
311
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
312

313
4. Push the modified branches to Git:
314

315
        git push origin                          \
316
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
317
           feature/buster:feature/buster       \
318
           devel:devel
319
320
321

Update more included files
==========================
322

323
324
Changelog
---------
Tails developers's avatar
Tails developers committed
325

326
327
328
Remove the placeholder entry for next release in `debian/changelog`,
and then:

329
	git checkout "${RELEASE_BRANCH:?}" && \
330
	./release ${VERSION:?} ${PREVIOUS_TAG:?}
Tails developers's avatar
Tails developers committed
331
332
333

This populates the Changelog with the Git log entries.

334
335
336
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
337

338
339
340
341
Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
342
343
* the diff between the previous version's `.packages` file and the one
  from the to-be-released ISO;
344
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
345
  in Redmine.
346

347
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
348

349
350
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
351
	else
352
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
353
	fi
Tails developers's avatar
Tails developers committed
354

Bessemer's avatar
Bessemer committed
355
Included website
356
357
----------------

Tails developers's avatar
Tails developers committed
358
359
360
361
362
363
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

364
365
### version number

366
367
If preparing a RC, skip this part.

368
In the branch used to build the release, update the `wiki/src/inc/*` files to
369
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
370
at least 24 hours in the future! Between tests and mirror synchronization,
371
the build will not be released on the same day. Try to make sure it
372
matches the date of the future signature.
373

sajolida's avatar
sajolida committed
374
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
375

376
377
378
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html
	sed -ri "s%news/version_.*]]%news/version_${VERSION:?}]]%" wiki/src/inc/stable_amd64_release_notes.*
379
	${EDITOR:?} wiki/src/inc/*.html
elouann's avatar
elouann committed
380
	./build-website
381
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
382

383
384
Website translations
--------------------
385

386
387
388
389
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
390

elouann's avatar
elouann committed
391
	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
392
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
393

394
395
396
Call for translation
====================

397
398
399
400
401
402
If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].
403

404
To get a list of changes on the website:
405

406
    git diff --stat ${PREVIOUS_VERSION:?}.. -- \
407
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
408
409
410
411
412
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
413

Tails developers's avatar
Tails developers committed
414
415
416
Import the signing key
======================

417
418
419
420
421
422
Skip this part if you have a Tails signing subkey on an OpenPGPG
smartcard, i.e. if you are one of the usual release managers. This is
only relevant when the master key has been reassembled, e.g. for
signing a Tails emergency release where none of the usual release
managers are available.

Tails developers's avatar
Tails developers committed
423
You should never import the Tails signing key into your own keyring,
424
425
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
426
427

    export GNUPGHOME=$(mktemp -d)
428
429
430
431
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
432
433
434
435
436
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

437
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
438

439
440
441
442
443
444
445
446
447
448
Build the almost-final image
============================

1. [[Build an ISO image|contribute/build]] from the release branch.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep at least the resulting ISO image and the manifest of needed
   packages until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
449
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"
450
451


452
453
454
Tag the release in Git
======================

455
456
457
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
	git push --tags origin "${RELEASE_BRANCH:?}"
458
459
460
461
462
463
464

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

465
466
467
468
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

469
470
471
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
472
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
473

474
* Prepare tagged snapshots of upstream APT repositories:
475

476
          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"
477

478
  Note:
479

480
481
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
482
    [[custom APT repository|APT_repository/custom]] are
483
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
484
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
485
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
486
    Now, we have a "safety" net, in case you don't notice such a problem: if
487
488
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
489

Tails developers's avatar
Tails developers committed
490
491
492
Build images
============

493
494
495
Sanity check
------------

496
497
498
Verify that the Tor Browser release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag for the targeted
Tor Browser version in these Git repositories:
499
500
501
502

* <https://gitweb.torproject.org/builders/tor-browser-bundle.git>
* <https://gitweb.torproject.org/tor-browser.git>

503
A new tag may indicate that a new Tor Browser release is imminent.
504
505
506

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
507
508
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
509

510
1. Burn the almost final ISO image to a DVD.
511
512
1. Boot this DVD **on bare metal**.
1. Add `profile` to the kernel command-line.
513
514
515
1. Login.
1. Wait for the "Tor is ready" notification.
1. Start the web browser.
Tails developers's avatar
Tails developers committed
516
1. A few minutes later, once the `boot-profile` process has been
517
   killed, retrieve the new sort file from `/var/log/boot-profile`.
518
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
519
520
521
522
523
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
524
1. Inspect the Git diff (including diff stat), apply common sense.
525
526
527
   The following command is also helpful but requires that you save a
   copy of the old sort file into `/tmp/squashfs.sort.old`:

528
529
530
531
        diff -NaurB \
            <( cut -d' ' -f1 /tmp/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort | sort ) \
            | less
532

533
534
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

535
536
537
Build the final image
---------------------

Bessemer's avatar
Bessemer committed
538
Then all included files should be up-to-date and the versioned APT
539
540
suite should be ready, so it is time to:

541
542
* Mark the version as "released" in the changelog:

543
544
        dch --release --no-force-save-on-release --maintmaint
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
545

546
547
* Export `SOURCE_DATE_EPOCH`:

548
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
549

550
551
* tag the release *again*, with all included files in:
  
552
553
554
555
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
        git push --tags --force && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
556

557
558
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
559
   commands by e.g. committing a commit in between `git tag` and the
560
561
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
562

563
* build the final image!
564

565
566
567
568
* make sure at least two `tails@` members will build the ISO
  image. Example: if you, the RM, is a `tails@` member, you only have
  to ask one more member.

intrigeri's avatar
intrigeri committed
569
* compare the new build manifest with the one from the previous,
570
  almost final build; they should be identical, except that the
571
572
  `debian-security` serial might be higher. To ensure we publish
  the final build's `.build-manifest`, please run:
573

574
575
576
        export PACKAGES_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso.build-manifest"

* check out a new branch:
intrigeri's avatar
intrigeri committed
577

578
579
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
580

581
582
583
584
  (as soon as a new commit is created on `$RELEASE_BRANCH`, its ISO
  build will start failing until a new changelog entry is created,
  which we don't want to do on `$RELEASE_BRANCH` before it's merged
  into `master` at release time)
anonym's avatar
anonym committed
585

586
587
588
589
590
Generate the OpenPGP signatures and Torrents
============================================

First, create a directory with a suitable name and go there:

591
592
	mkdir "${ISOS:?}/tails-amd64-${VERSION:?}" && \
	cd "${ISOS:?}/tails-amd64-${VERSION:?}"
593

anonym's avatar
anonym committed
594
Second, move the built image to this brand new directory:
595

596
597
	mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso" \
	   "${ISOS:?}/tails-amd64-${VERSION:?}/"
598
599
600
601
602

Third, generate detached OpenPGP signatures for the image to be
published, in the same directory as the image and with a `.sig`
extension; e.g.

603
	gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *.iso
604
605
606
607
608
609
	rename 's,\.asc$,.sig,' *.asc

Fourth, go up to the parent directory, create a `.torrent` file and
check the generated `.torrent` files metainfo:

	cd .. && \
intrigeri's avatar
intrigeri committed
610
	mktorrent \
intrigeri's avatar
intrigeri committed
611
	  -a 'udp://tracker.torrent.eu.org:451' \
intrigeri's avatar
intrigeri committed
612
	  -a 'udp://tracker.coppersurfer.tk:6969'   \
613
614
	  "tails-amd64-${VERSION:?}" && \
	transmission-show tails-amd64-${VERSION:?}.torrent
615

anonym's avatar
anonym committed
616
617
Lastly, let's set some variables to be used later:

618
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
619
620
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
anonym's avatar
anonym committed
621

622
623
<a id="prepare-iuk"></a>

624
625
626
Prepare incremental upgrades
============================

627
628
Build the Incremental Upgrade Kits
----------------------------------
629

630
631
632
633
Incremental upgrades may be skipped if the delta is too big (like when
migrating to a new Debian release) or if there are changes outside of
the scope for IUKs (like partition table changes). Use common sense!

634
635
Use `tails-create-iuk` to build the following IUKs:

636
637
638
639
* From the two previous *planned* releases, and any emergency releases
  in between and after. This should be, more or less, all releases for
  the last 12 weeks (although irregularities in Firefox release
  schedule may add or remove a few weeks).
640
641
642
643
644

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
645
  next version.
646

647
648
649
Include each such version in a white-space separated list called
`IUK_SOURCE_VERSIONS`, (e.g. `IUK_SOURCE_VERSIONS="2.8 2.9 2.9.1 2.10~rc1"`)
and run the following:
650

651
652
    for source_version in ${IUK_SOURCE_VERSIONS}; do
        sudo su -c "cd ${IUK_CHECKOUT:?} && \
653
654
          SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH \
          LC_ALL=C \
655
656
657
          PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
            ./bin/tails-create-iuk \
               --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
658
659
660
               --old-iso \"${ISOS:?}/tails-amd64-${source_version:?}/tails-amd64-${source_version:?}.iso\" \
               --new-iso \"${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso\"          \
               --outfile \"${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk\""
661
    done
662
663

Note that developer tools for creating IUK and upgrade-description
Tails developers's avatar
Tails developers committed
664
files were only tested on Debian sid. It should hopefully work well on
665
Jessie too.
666

667
668
669
Make sure at least two `tails@` members reproduce the same IUKs.
Provide them with the value of `IUK_SOURCE_VERSIONS` that you used!

Tails developers's avatar
Tails developers committed
670
<a id="prepare-upgrade-description-files"></a>
671

Tails developers's avatar
Tails developers committed
672
673
Prepare upgrade-description files
---------------------------------
674

Tails developers's avatar
Tails developers committed
675
1. Prepare upgrade-description files (see the [[upgrade-description
676
   files
Tails developers's avatar
Tails developers committed
677
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
678
679
680
681
682
683
   for details). The idea is to:

   * update (create if needed) an upgrade-description file for every
     *previous* supported release (e.g. N~rc1, N-1, N-1~rc2) that replaces
     all existing upgrade paths with the path to the version being
     released;
684
685
686
   * create a new upgrade-description for the version being released
     and for the next one, that expresses that *no* upgrade is
     available for these ones yet.
687

intrigeri's avatar
intrigeri committed
688
   This is what `tails-iuk-generate-upgrade-description-files` tool
689
690
   does:

691
692
693
694
695
696
697
        ( cd ${IUK_CHECKOUT:?} && \
        ./bin/tails-iuk-generate-upgrade-description-files \
            --build-target amd64 \
            --version "${VERSION:?}" \
            --next-version "${NEXT_PLANNED_VERSION:?}" \
            --next-version "${NEXT_PLANNED_VERSION:?}~rc1" \
            --next-version "${VERSION:?}.1" \
698
            --iso "${ISOS_PATH:?}" \
699
700
701
702
703
            --previous-version "${PREVIOUS_VERSION:?}" \
            --previous-version "${VERSION:?}~rc1" \
            --iuks "${ISOS:?}" \
            --release-checkout "${RELEASE_CHECKOUT:?}" \
            --major-release "${MAJOR_RELEASE:?}" \
704
       )
705
706
707
708
709
710
711

   Note:
   * The `--iuks` argument must point to the directory where the IUKs
     generated at the previous step are stored.
   * At least the last stable release and the previous release
     candidates for the version being released must be passed to
     `--previous-version`.
712
713
714
715
716
717
718
719
720
   * Older versions for which there is no incremental upgrade path to
     the new release must be passed with `--previous-version`, so that
     users who skipped a release or two are informed of the new one.
     Note that multi-steps incremental upgrade paths are valid and
     supported: e.g. when releasing 1.1.2, 1.1 users should still be
     able to incrementally upgrade to 1.1.1, and in turn to 1.1.2; to
     make this work, one must _not_ pass `--previous-version 1.1`,
     that would remove the existing incremental upgrade path from 1.1
     to 1.1.1.
721
722
723
724
   * If preparing anything but a final release (e.g. an alpha, beta
     or RC), add `--channel alpha`
   * If preparing anything but a final release (e.g. an alpha, beta
     or RC), drop all `--next-version`
725
     arguments, and instead pass (**untested!**)
anonym's avatar
anonym committed
726
     `--next-version $(echo ${VERSION:?} | sed -e 's,~rc.*$,,')`
727
   * If preparing a point-release, pass neither
728
729
     `--next-version "${VERSION:?}.1"`,
     nor `--next-version "${VERSION:?}.1~rc1"`
730
731
732
733

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

734
735
736
737
738
739
740
741
742
743
744
745
        find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
           -type f -name upgrades.yml | \
           while read udf; do
               if [ -n "$(git status --porcelain "${udf:?}")" ]; then
                   gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign "${udf:?}"
                   mv --force "${udf:?}.asc" "${udf:?}.pgp"
                   ( \
                     cd ${IUK_CHECKOUT:?} && \
                     ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
                   ) || break
               fi
           done
746

Tails developers's avatar
Tails developers committed
747
1. Add and commit the upgrade-description files and their detached
748
749
   signatures to the Git branch used to prepare the release
   (`$WEBSITE_RELEASE_BRANCH`):
Tails developers's avatar
Tails developers committed
750

751
752
753
754
755
        ( \
          cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
          git commit -m "Update upgrade-description files." && \
          git push origin ${WEBSITE_RELEASE_BRANCH:?} \
        )
Tails developers's avatar
Tails developers committed
756

757
1. If preparing anything but a final release (e.g. an alpha, beta
758
759
760
   or RC), copy the generated or updated files to
   `${MASTER_CHECKOUT:?}`, replace `channel: alpha` with `channel:
   test`, sign them, commit and push.
Tails developers's avatar
Tails developers committed
761

intrigeri's avatar
intrigeri committed
762
763
764
1. Else, if preparing a final release, copy the generated UDF for the previous
   release to the *test* channel in `$MASTER_CHECKOUT`, modify its content
   accordingly, sign it, commit and push:
765

766
767
768
        ( \
          cd ${MASTER_CHECKOUT:?} && \
          git fetch && \
anonym's avatar
anonym committed
769
          for old_version in ${IUK_SOURCE_VERSIONS:?}; do
770
771
772
773
774
775
776
777
778
779
780
781
            stable_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/stable/upgrades.yml" && \
            test_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/test/upgrades.yml" && \
            mkdir -p "$(dirname "$test_udf")" && \
            git show origin/${WEBSITE_RELEASE_BRANCH:?}:${stable_udf:?} \
              | sed -e 's/channel: stable/channel: test/' > ${test_udf:?} && \
            gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${test_udf:?} && \
            mv ${test_udf:?}.asc ${test_udf:?}.pgp && \
            git add ${test_udf:?}* ; \
          done && \
          git commit -m "Add incremental upgrades on the test channel for Tails ${VERSION:?}" && \
          git push origin master:master \
        )
782
783


784
785
786
Prepare the ISO description file for DAVE
-----------------------------------------

anonym's avatar
anonym committed
787
788
If preparing a RC, skip this part.

789
790
Update the ISO description file (IDF) used by the browser extension:

791
792
793
794
795
796
797
798
799
800
801
    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/i386/stable/latest.yml <<EOF
    ---
    build-target: amd64
    channel: stable
    product-name: Tails
    version: '${VERSION:?}'
    target-files:
    - sha256: ${ISO_SHA256SUM}
      size: ${ISO_SIZE_IN_BYTES:?}
      url: http://dl.amnesia.boum.org/tails/stable/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso
    EOF
802
    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/amd64/stable/latest.yml <<EOF
803
    ---
804
    build-target: amd64
805
806
    channel: stable
    product-name: Tails
807
    version: '${VERSION:?}'
808
809
    target-files:
    - sha256: ${ISO_SHA256SUM}
810
      size: ${ISO_SIZE_IN_BYTES:?}
811
      url: http://dl.amnesia.boum.org/tails/stable/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso
812
    EOF
813
    ( cd "${RELEASE_CHECKOUT:?}" && \
anonym's avatar
anonym committed
814
      git add wiki/src/install/v1/Tails/{i386,amd64}/stable/latest.yml && \
815
816
      git commit -m "Update IDF file for DAVE." )

Tails developers's avatar
Tails developers committed
817
818
819
Upload images
=============

Tails developers's avatar
Tails developers committed
820
821
822
Sanity check
------------

823
Verify once more that the Tor Browser we ship is still the most recent (see
824
above).
Tails developers's avatar
Tails developers committed
825

826
827
828
829
Reproducibility
---------------

Previously you have asked `tails@` members to reproduce the Tails ISO
830
831
image and all IUKs; now tell all participants to send you the
`SHA-512` hashes of their ISO and IUKs over signed email.
832

833
* If all hashes match: yay, we're good to go!
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848

* If the reproduction attempts haven't been completed yet: continue at
  your own risk! If you get a negative answer later you might have to
  undo everything done from this point on. It's still a good idea to
  optimistically assume success in order to not be blocked at this
  point. However, be mentally prepared that it might have to be done
  once more, and make sure to return to this section once everyone is
  done with their reproduction attempts, and certainly before making
  the release public.

* If there is a hash mismatch for the ISO: ouch!  Now we are in a
  tricky situation: on the one hand it seems like a poor idea to block
  users from benefiting from this release's security updates, but on
  the other hand the failure might imply that something nefarious is
  going on. At this stage, no matter what, immediately compare the
849
850
  ISOs (using `diffoscope`) and try to rule out build system
  compromise.
851
852
853
854

  - If something seemingly malicious is found, then let's take a step
    back: we might be compromised, so we are in no position to
    release. Halt the release, involve the rest of `tails@`, and then
855
856
    try to re-establish trust in all build machines and infra
    involved, etc. Have fun!
857
858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889

  - Otherwise:

    * If the source of non-determinism is identified quickly and is
      easy and fast to fix, *and* the QA of the current ISO has not
      gone very far (so at least that time is not wasted), then you
      should consider abandoning the current version, and immediately
      start preparing an emergency release with:

      - the reproducibility fix

      - a new changelog entry,

      - adjustments to the release notes so they are re-purposed for
        this emergency release (the abandoned release gets none, since
        it effectively never will be released publicly).

    * Otherwise, let's release any way. But let's add a known issue
      about "This Tails release has reproducibility issues" to the
      release notes, linking to the ticket(s) (or similar) where the
      nature of the reproducibility failure is clearly described.

* If there is a hash mismatch for an IUK (or several): proceed with
  the release, except for the problematic IUK(s); remove them from the
  mirrors, and remove the affected incremental upgrade paths from the
  UDFs! In parallel with the rest of the release process, try to
  figure out what the problem with the IUK is and fix the cause, so a
  good IUK can be released as soon as possible. If a seemingly
  malicious difference is found, then immediately halt the release and
  go to the "If something seemingly malicious is found" case for the
  ISO above. Because of this it is advisable to not publicly release
  until malicious differences have been ruled out.

890
891
892
893
<a id="publish-iuk"></a>

Publish the ISO and IUK over HTTP
---------------------------------
Tails developers's avatar
Tails developers committed
894

intrigeri's avatar
intrigeri committed
895
896
897
898
899
900
901
902
Upload the IUKs to the primary rsync mirror:

    for source_version in ${IUK_SOURCE_VERSIONS}; do
       rsync --partial --inplace --progress -v \
          "${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk" \
          rsync.lizard:
    done

903
904
Upload the ISO signature to the primary rsync mirror:

905
    scp "${ISO_PATH:?}.sig" rsync.lizard:
906
907
908
909
910
911

Pick a build from `$RELEASE_BRANCH` that produced an ISO identical to
the one you've built locally (`XXX` must be the job ID, i.e.
an integer):

    MATCHING_JENKINS_BUILD_ID=XXX
912

913
914
915
916
917
918
919
920
921
Copy the ISO to the primary rsync mirror and verify the signature:

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh rsync.lizard gpg --import
    ssh rsync.lizard << EOF
       wget \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,}
    EOF
intrigeri's avatar
intrigeri committed
922
923
924

Move files in place with proper ownership and permissions:

925
    ssh rsync.lizard << EOF
926
927
928
929
930
931
932
933
934
935
936
      sudo install -o root -g rsync_tails -m 0755 -d \
         /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
      sudo chown root:rsync_tails \
         tails-amd64-${VERSION:?}.iso* \
         Tails_amd64_*_to_${VERSION:?}.iuk && \
      sudo chmod u=rwX,go=rX \
         tails-amd64-${VERSION:?}.iso* \
         Tails_amd64_*_to_${VERSION:?}.iuk && \
      sudo mv tails-amd64-${VERSION:?}.iso* \
              /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
      sudo mv Tails_amd64_*_to_${VERSION:?}.iuk \
937
              /srv/rsync/tails/tails/${DIST:?}/iuk/
938
    EOF
939
940
941
942
943

Update the time in `project/trace` file on the primary rsync mirror
and on the live wiki (even for a release candidate):

	TRACE_TIME=$(date +%s) &&
944
945
946
	echo ${TRACE_TIME:?} | ssh rsync.lizard "cat > /srv/rsync/tails/tails/project/trace" && \
	[ -n "${MASTER_CHECKOUT:?}" ] && \
	echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
947
	(
948
	   cd "${MASTER_CHECKOUT:?}" && \
949
	   git commit wiki/src/inc/trace \
950
	      -m "Updating trace file after uploading ${VERSION:?}." && \
951
	   git push origin master
952
953
	)

954
955
956
957
958
959
960

## Announce, seed and test the Torrents

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh bittorrent.lizard gpg --import
    scp \
       "${ISOS:?}/tails-amd64-${VERSION:?}.torrent" \
961
       "${ISOS_PATH:?}.sig" \
962
963
964
965
966
967
968
969
970
971
972
       bittorrent.lizard: && \
    ssh bittorrent.lizard << EOF
       mkdir --mode 0755 "tails-amd64-${VERSION:?}" && \
       mv "tails-amd64-${VERSION:?}.iso.sig" \
          "tails-amd64-${VERSION:?}/" && \
       cd "tails-amd64-${VERSION:?}" && \
       wget \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,} && \
       cd && \
       chmod -R go+rX "tails-amd64-${VERSION:?}" && \
anonym's avatar
anonym committed
973
       mv \
974
975
976
977
978
979
980
981
          "tails-amd64-${VERSION:?}" \
          /var/lib/transmission-daemon/downloads/ && \
       transmission-remote --add tails-amd64-${VERSION:?}.torrent \
                           --find /var/lib/transmission-daemon/downloads/
    EOF

Test that you can start downloading the ISO with a BitTorrent client.

982
983
984
985
ISO history
-----------

Push the released ISO to our Tails ISO history git-annex repo, so that
986
987
our isotesters can fetch it from there for their testing. How to do so
is described in our internal Git repo.
988

989
990
991
992
993
994
995
Testing
=======

1. Using `check-mirrors`, choose a fast mirror that already has the
   tentative ISO. E.g. <https://mirrors.kernel.org/tails/> or
   <https://mirrors.wikimedia.org/tails/> are reliable and have plenty
   of bandwidth.
anonym's avatar
anonym committed
996
997
998
999
1000

        ./check-mirrors.rb --allow-multiple --channel ${DIST:?} \
            --ip $(dig +short mirrors.kernel.org | tail -n1) \
            tails-amd64-${VERSION:?}

For faster browsing, not all history is shown. View entire blame