release_process.mdwn 54.8 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
8
9
10
<div class="caution">
Read the remainder of this document from the branch used to prepare the release!
</div>

11
12
13
14
15
Requirements
============

To release Tails you'll need some packages installed:

16
* `tidy mktorrent transmission-cli`
intrigeri's avatar
intrigeri committed
17
* aufs DKMS module for your running kernel.
intrigeri's avatar
intrigeri committed
18
* [[!debpts squashfs-tools]] that honors `$SOURCE_DATE_EPOCH`.
19
  Install it from our custom `devel` APT suite.
intrigeri's avatar
intrigeri committed
20
* `tails-iuk` dependencies, including suggested packages (see
21
22
23
  `debian/control` in the `debian` branch of its repo)
* `tails-perl5lib` dependencies (same trick as `tails-iuk` to get the
  list)
24

25
26
27
28
29
30
Environment
===========

Export the following environment variables to be able to copy'n'paste
the scripts snippets found on this page:

31
* version numbers (see [[contribute/release_schedule#versioning]]):
32

33
34
35
36
        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')
37

38
* `NEXT_PLANNED_VERSION`: set to the version number of the next Tails release
39
  (e.g. 0.23 when releasing 0.22.1, and 1.3 when releasing 1.2)
40
41
42
43
44
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
  *major* Tails release
* `NEXT_PLANNED_MINOR_VERSION`: set to the version number of the next
  *minor* Tails release; if the next release is a point-release, use
  that one; otherwise, use `${VERSION}.1`
45
46
* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
47
* `ISOS`: the directory where one stores `tails-amd64-*`
48
  sub-directories like the ones downloaded with BitTorrent.
49
50
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
51
52
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
intrigeri's avatar
intrigeri committed
53
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
54
55
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
56
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
57
58
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
59
60
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
61
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
intrigeri's avatar
intrigeri committed
62
63
* `export DEBEMAIL='tails@boum.org'`
* `export DEBFULLNAME='Tails developers'`
64
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
65

66
67
68
Pre-freeze
==========

69
70
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
71
72
73
74
75
76

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

77
78
79
80
81
82
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

83
84
85
86
87
88
89
90
91
92
93
94
Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
95
96
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
97
98
99
100
101
102
103
104
105
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

106
107
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
108

anonym's avatar
anonym committed
109
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
110
111
112
113
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
114
115
116
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
117
118
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

119
120
121
122
123
124
125
126

Point-release
-------------

If we are at freeze time for a point-release:

1. Merge the `master` Git branch into `stable`:

127
        git checkout stable && git fetch && git merge --no-ff origin/master
128

intrigeri's avatar
intrigeri committed
129
130
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
131
132
133
134
135
   APT suite.

Common steps for point and major releases
-----------------------------------------

136
Reset the release branch's `config/base_branch`:
137

138
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
139
           git commit config/base_branch \
140
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
141

142
143
144
145
146
147
148
149
Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
3. Send the pad URL to the usual testers (see `manual_testers.mdwn` in
   the `internal.git` repository).

Tails developers's avatar
Tails developers committed
150
151
152
Update included files
=====================

spriver's avatar
spriver committed
153
uBlock patterns and settings file
Tails developers's avatar
Tails developers committed
154
155
----------------

156
The patterns+settings file is stored as a SQLite text dump in
spriver's avatar
spriver committed
157
`config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`.
Tails developers's avatar
Tails developers committed
158

159
160
161
162
163
164
165
1. Start Tails
2. Start *Tor Browser*
2. Click on the uBlock icon and then click on the gears icon to open
   the uBlock dashboard
3. Open the *3rd-party filters* tab
4. Click on the button *Update now* to update all filters
5. Close *Tor Browser*
166
167
168
169
7. Copy the `.tor-browser/profile.default/extension-data/ublock0.sqlite`
   from this Tor Browser instance into the root of Tails' Git repo and
   run the following command:

170
171
172
        ./bin/convert-ublock-settings ublock0.sqlite > \
            config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump \
        && git commit -m 'Update uBlock Origin patterns + settings file.' \
173
174
           config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump \
        && rm ublock0.sqlite
175

Bessemer's avatar
Bessemer committed
176
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
177
178
--------------------------------------

179
180
181
182
183
184
185
186
187
188
189
190
Skip this section if you are preparing a point-release.

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:

 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
 - If you are preparing a major release, build at least the packages
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)
191

192
193
194
195
196
197
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
and then run the `import-translations` script that is in the
main Tails repository. For example:

	cd whisperback
198
	"${RELEASE_CHECKOUT:?}"/import-translations
199

WinterFairy's avatar
WinterFairy committed
200
201
If the `import-translations` script fails to import translations for
the current package, manually copy updated PO files from the
202
Transifex branches of `git://git.torproject.org/translation.git` (e.g.
203
204
`whisperback_completed`) instead. In this case, skip PO files for
[[translation teams that use Git|contribute/how/translate#translate]].
205

206
Add and commit.
207

208
209
Then check the PO files:

210
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po
211
212

Correct any displayed error, then commit the changes if any.
213

214
Then see the relevant release processes, and upload the packages to
215
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
216

217
* [[tails-installer]]
218
* [[tails-greeter]]
219
* [[perl5lib]]
220
* [[persistence-setup]]
221
* [[tails-iuk]]
222
* whisperback:
223
224
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package
Tails developers's avatar
Tails developers committed
225

intrigeri's avatar
intrigeri committed
226
227
Upgrade Tor Browser
-------------------
228
229
230

See the dedicated page: [[tor-browser]]

231
232
233
234
235
Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[browser-apparmor-patch]]

236
Upgrade Thunderbird
237
238
---------------

239
See the dedicated page: [[thunderbird]]
240

241
242
Update PO files
---------------
243

244
245
246
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
247

248
	cd "${RELEASE_CHECKOUT:?}" && \
249
250
	./import-translations  && \
	./refresh-translations && \
251
	./submodules/jenkins-tools/slaves/check_po && \
252
	git add po && git commit -m 'Update PO files.'
253

254
255
256
257
258
259
If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

260
261
When preparing an actual release
================================
262

263
264
265
266
267
268
If we're about to prepare an image for a final (non-RC) release, then
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
269
270
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
271
custom APT suite.
272
273
274
275

Point-release
-------------

276
277
278
279
280
281
<div class="note">
For point-releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
282

intrigeri's avatar
intrigeri committed
283
284
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
285
custom APT suite.
286

anonym's avatar
anonym committed
287
288
Update other base branches
==========================
289
290

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
291
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
292

293
2. Merge `devel` into `feature/buster`, *without* following the instructions for
intrigeri's avatar
intrigeri committed
294
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
295
296
   (For now `feature/buster` is handled as any other topic branch
   forked off `devel`: its base branch is set to `devel`.)
297

298
3. Ensure that the release, `devel` and `feature/buster` branches
299
300
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
301

302
303
304
305
306
4. [[Thaw|APT_repository/time-based snapshots#thaw]], on the devel
   branch, the time-based APT repository snapshots that were used
   during the freeze.

5. Push the modified branches to Git:
307

308
        git push origin                          \
309
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
310
           feature/buster:feature/buster       \
311
           devel:devel
312
313
314

Update more included files
==========================
315

316
317
Changelog
---------
Tails developers's avatar
Tails developers committed
318

319
320
321
Remove the placeholder entry for next release in `debian/changelog`,
and then:

322
	git checkout "${RELEASE_BRANCH:?}" && \
323
	./release ${VERSION:?} ${PREVIOUS_TAG:?}
Tails developers's avatar
Tails developers committed
324
325
326

This populates the Changelog with the Git log entries.

327
328
329
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
330

331
332
333
334
Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
335
* the diff between the previous version's `.packages` file and the one
336
337
338
339
340
  from the to-be-released ISO; look for:
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
341
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
342
  in Redmine.
343

344
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
345

346
347
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
348
	else
349
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
350
	fi
Tails developers's avatar
Tails developers committed
351

Bessemer's avatar
Bessemer committed
352
Included website
353
354
----------------

Tails developers's avatar
Tails developers committed
355
356
357
358
359
360
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

361
362
### version number

363
364
If preparing a RC, skip this part.

365
In the branch used to build the release, update the `wiki/src/inc/*` files to
366
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
367
at least 24 hours in the future! Between tests and mirror synchronization,
368
the build will not be released on the same day. Try to make sure it
369
matches the date of the future signature.
370

sajolida's avatar
sajolida committed
371
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
372

373
374
375
	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html
	sed -ri "s%news/version_.*]]%news/version_${VERSION:?}]]%" wiki/src/inc/stable_amd64_release_notes.*
376
	${EDITOR:?} wiki/src/inc/*.html
elouann's avatar
elouann committed
377
	./build-website
378
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
379

380
381
Website translations
--------------------
382

383
384
385
386
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
387

elouann's avatar
elouann committed
388
	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
389
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
390

391
392
393
Call for translation
====================

394
395
396
397
398
399
If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].
400

401
To get a list of changes on the website:
402

intrigeri's avatar
intrigeri committed
403
    git diff --stat ${PREVIOUS_TAG:?}.. -- \
404
        wiki/src/'*'.{mdwn,html} \
anonym's avatar
anonym committed
405
406
407
408
409
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
410

411
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
412
413
======================

414
415
416
417
418
419
420
421
422
423
424
425
### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
pro tip is to never plug it unless prompted which `gpg` will do for
you. Then just unplug it as soon as the `.sig` is done.

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
426

Tails developers's avatar
Tails developers committed
427
You should never import the Tails signing key into your own keyring,
428
429
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
430
431

    export GNUPGHOME=$(mktemp -d)
432
433
434
435
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
436
437
438
439
440
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

441
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
442

443
444
445
446
447
448
449
450
451
452
Build the almost-final image
============================

1. [[Build an ISO image|contribute/build]] from the release branch.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep at least the resulting ISO image and the manifest of needed
   packages until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
453
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"
454
455


456
457
458
Tag the release in Git
======================

459
460
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
461
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
462
463
464
465
466
467
468

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

469
470
471
472
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

473
474
475
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
476
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
477

478
* Prepare tagged snapshots of upstream APT repositories:
479

480
          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"
481

482
  Note:
483

484
485
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
486
    [[custom APT repository|APT_repository/custom]] are
487
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
488
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
489
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
490
    Now, we have a "safety" net, in case you don't notice such a problem: if
491
492
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
493

Tails developers's avatar
Tails developers committed
494
495
496
Build images
============

497
498
499
Sanity check
------------

500
501
502
Verify that the Tor Browser release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag for the targeted
Tor Browser version in these Git repositories:
503
504
505
506

* <https://gitweb.torproject.org/builders/tor-browser-bundle.git>
* <https://gitweb.torproject.org/tor-browser.git>

507
A new tag may indicate that a new Tor Browser release is imminent.
508
509
510

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
511
512
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
513

514
1. Burn the almost final ISO image to a DVD.
515
516
1. Boot this DVD **on bare metal**.
1. Add `profile` to the kernel command-line.
517
518
1. Login.
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
519
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
520
1. A few minutes later, once the `boot-profile` process has been
521
   killed, retrieve the new sort file from `/var/log/boot-profile`.
522
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
523
524
525
526
527
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
528
1. Inspect the Git diff (including diff stat), apply common sense.
529
530
531
   The following command is also helpful but requires that you save a
   copy of the old sort file into `/tmp/squashfs.sort.old`:

532
533
534
535
        diff -NaurB \
            <( cut -d' ' -f1 /tmp/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort | sort ) \
            | less
536

537
538
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

539
540
541
Build the final image
---------------------

Bessemer's avatar
Bessemer committed
542
Then all included files should be up-to-date and the versioned APT
543
544
suite should be ready, so it is time to:

545
1. Mark the version as "released" in the changelog:
546

547
        dch --release --no-force-save-on-release --maintmaint && \
548
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
549

550
1. Export `SOURCE_DATE_EPOCH`:
551

552
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
553

554
1. tag the release *again*, with all included files in:
555

556
557
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
558
559
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
560

561
562
   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
intrigeri's avatar
intrigeri committed
563
   commands by e.g. committing a commit in between `git tag` and the
564
565
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!
anonym's avatar
anonym committed
566

567
1. build the final image!
568

569
1. compare the new build manifest with the one from the previous,
anonym's avatar
anonym committed
570
571
572
   almost final build; they should be identical, except that the
   `debian-security` serial might be higher. To ensure we publish
   the final build's `.build-manifest`, please run:
573

574
575
        export PACKAGES_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso.build-manifest"

576

577
578
1. <a id="reproducibility-sanity-check-iso"></a>
   Let's sanity check that Jenkins reproduced your image.
579

anonym's avatar
xxx    
anonym committed
580
581
582
583
584
   Visit the URL printed by this command:

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

   Find the job (probably the last one)
585
   and make sure the image built by Jenkins:
586

587
588
589
   - was built from the correct Git commit
   - has the same file size as the image you built
   - has the same hash (in the `.shasum` file) as the image you built
590

591
   Then:
592

593
   - If all hashes match: yay, we're good to go!
594

595
596
597
598
599
600
601
   - If there is a hash mismatch for the image: ouch! Now we are in a
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
602

anonym's avatar
anonym committed
603
604
605
606
607
608
609
610
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
611

612
     Then carefully investigate the `diffoscope` report:
613

614
615
616
617
618
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
         release. Halt the release, involve the rest of `tails@`, and then
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
619

620
       - Otherwise, if the change is definitely harmless:
621

622
623
624
625
626
         * If the source of non-determinism is identified quickly and
           is easy and fast to fix, *and* the QA of the current image
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
627

628
629
630
631
632
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
633

634
635
636
637
638
639
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
           release notes, linking to the ticket where
           the nature of the reproducibility failure is clearly
           described.
640

641
1. check out a new branch:
intrigeri's avatar
intrigeri committed
642

643
644
645
646
647
648
649
650
   If preparing anything but a final release (e.g. an alpha, beta
   or RC):

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

   Else, if preparing a final release:

651
652
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
653

anonym's avatar
anonym committed
654
655
656
657
   (as soon as a new commit is created on `$RELEASE_BRANCH`, its ISO
   build will start failing until a new changelog entry is created,
   which we don't want to do on `$RELEASE_BRANCH` before it's merged
   into `master` at release time)
anonym's avatar
anonym committed
658

659
660
661
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
662
663
664
665
666
667
Create a directory with a suitable name, go there, move the built
image to this brand new directory, generate detached OpenPGP
signatures for the image to be published (in the same directory as the
image and with a `.sig` extension), then go up to the parent
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
668

669
	mkdir "${ISOS:?}/tails-amd64-${VERSION:?}" && \
intrigeri's avatar
intrigeri committed
670
	cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
671
	mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso" \
intrigeri's avatar
intrigeri committed
672
673
674
	   "${ISOS:?}/tails-amd64-${VERSION:?}/" && \
	gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *.iso && \
	rename 's,\.asc$,.sig,' *.asc && \
675
	cd .. && \
intrigeri's avatar
intrigeri committed
676
	mktorrent \
intrigeri's avatar
intrigeri committed
677
	  -a 'udp://tracker.torrent.eu.org:451' \
intrigeri's avatar
intrigeri committed
678
	  -a 'udp://tracker.coppersurfer.tk:6969'   \
679
680
	  "tails-amd64-${VERSION:?}" && \
	transmission-show tails-amd64-${VERSION:?}.torrent
681

anonym's avatar
anonym committed
682
683
Lastly, let's set some variables to be used later:

684
    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
685
686
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
anonym's avatar
anonym committed
687

688
689
<a id="prepare-iuk"></a>

690
691
692
Prepare incremental upgrades
============================

693
694
Build the Incremental Upgrade Kits
----------------------------------
695

696
697
698
699
Incremental upgrades may be skipped if the delta is too big (like when
migrating to a new Debian release) or if there are changes outside of
the scope for IUKs (like partition table changes). Use common sense!

700
701
Use `tails-create-iuk` to build the following IUKs:

702
703
704
705
* From the two previous *planned* releases, and any emergency releases
  in between and after. This should be, more or less, all releases for
  the last 12 weeks (although irregularities in Firefox release
  schedule may add or remove a few weeks).
706
707
708
709
710

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
711
  next version.
712

713
714
715
Include each such version in a white-space separated list called
`IUK_SOURCE_VERSIONS`, (e.g. `IUK_SOURCE_VERSIONS="2.8 2.9 2.9.1 2.10~rc1"`)
and run the following:
716

717
    for source_version in ${IUK_SOURCE_VERSIONS}; do
718
719
720
721
        if [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" != 1:4.3-3.0tails4 ]; then
            echo 'ERROR! Your squashfs-tools probably does not honor SOURCE_DATE_EPOCH so any generated IUKs will *not* be reproducible!'
            break
        fi
722
        sudo su -c "cd ${IUK_CHECKOUT:?} && \
723
724
          SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH \
          LC_ALL=C \
725
726
727
          PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
            ./bin/tails-create-iuk \
               --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
728
729
730
               --old-iso \"${ISOS:?}/tails-amd64-${source_version:?}/tails-amd64-${source_version:?}.iso\" \
               --new-iso \"${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso\"          \
               --outfile \"${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk\""
731
    done
732
733

Note that developer tools for creating IUK and upgrade-description
Tails developers's avatar
Tails developers committed
734
files were only tested on Debian sid. It should hopefully work well on
735
Debian stable too.
736

737
738
739
740
<a id="reproducibility-sanity-check-iuk"></a>

Note that we do not yet build IUKs on Jenkins, otherwise here would be
a great point to compare its IUKs with yours.
741

Tails developers's avatar
Tails developers committed
742
<a id="prepare-upgrade-description-files"></a>
743

Tails developers's avatar
Tails developers committed
744
745
Prepare upgrade-description files
---------------------------------
746

Tails developers's avatar
Tails developers committed
747
1. Prepare upgrade-description files (see the [[upgrade-description
748
   files
Tails developers's avatar
Tails developers committed
749
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
750
751
752
753
754
755
   for details). The idea is to:

   * update (create if needed) an upgrade-description file for every
     *previous* supported release (e.g. N~rc1, N-1, N-1~rc2) that replaces
     all existing upgrade paths with the path to the version being
     released;
756
757
758
   * create a new upgrade-description for the version being released
     and for the next one, that expresses that *no* upgrade is
     available for these ones yet.
759

intrigeri's avatar
intrigeri committed
760
   This is what `tails-iuk-generate-upgrade-description-files` tool
761
762
   does:

763
764
765
766
        ( cd ${IUK_CHECKOUT:?} && \
        ./bin/tails-iuk-generate-upgrade-description-files \
            --build-target amd64 \
            --version "${VERSION:?}" \
767
768
769
            --next-version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
            --next-version "${NEXT_PLANNED_MAJOR_VERSION:?}~rc1" \
            --next-version "${NEXT_PLANNED_MINOR_VERSION:?}" \
770
            --next-version "${VERSION:?}.1" \
anonym's avatar
anonym committed
771
            --iso "${ISO_PATH:?}" \
772
773
774
775
776
            --previous-version "${PREVIOUS_VERSION:?}" \
            --previous-version "${VERSION:?}~rc1" \
            --iuks "${ISOS:?}" \
            --release-checkout "${RELEASE_CHECKOUT:?}" \
            --major-release "${MAJOR_RELEASE:?}" \
777
       )
778
779
780
781
782
783
784

   Note:
   * The `--iuks` argument must point to the directory where the IUKs
     generated at the previous step are stored.
   * At least the last stable release and the previous release
     candidates for the version being released must be passed to
     `--previous-version`.
785
786
787
788
789
790
791
792
793
   * Older versions for which there is no incremental upgrade path to
     the new release must be passed with `--previous-version`, so that
     users who skipped a release or two are informed of the new one.
     Note that multi-steps incremental upgrade paths are valid and
     supported: e.g. when releasing 1.1.2, 1.1 users should still be
     able to incrementally upgrade to 1.1.1, and in turn to 1.1.2; to
     make this work, one must _not_ pass `--previous-version 1.1`,
     that would remove the existing incremental upgrade path from 1.1
     to 1.1.1.
794
795
796
797
   * If preparing anything but a final release (e.g. an alpha, beta
     or RC), add `--channel alpha`
   * If preparing anything but a final release (e.g. an alpha, beta
     or RC), drop all `--next-version`
798
     arguments, and instead pass (**untested!**)
anonym's avatar
anonym committed
799
     `--next-version $(echo ${VERSION:?} | sed -e 's,~rc.*$,,')`
800
801
802
803
   * Adjust `--next-version "${VERSION:?}.1"` so it matches the next
     potential emergency release. E.g. when releasing 3.7.1,
     pass `--next-version 3.7.2` and when releasing 3.8, pass
     `--next-version 3.8.1`.
804
805
806
807

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

808
809
810
811
        find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
           -type f -name upgrades.yml | \
           while read udf; do
               if [ -n "$(git status --porcelain "${udf:?}")" ]; then
812
813
814
815
816
                   for x in 1 2 3; do
                       gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor \
                           --detach-sign "${udf:?}" \
                       && break
                   done
817
818
819
820
821
822
823
                   mv --force "${udf:?}.asc" "${udf:?}.pgp"
                   ( \
                     cd ${IUK_CHECKOUT:?} && \
                     ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
                   ) || break
               fi
           done
824

Tails developers's avatar
Tails developers committed
825
1. Add and commit the upgrade-description files and their detached
826
827
   signatures to the Git branch used to prepare the release
   (`$WEBSITE_RELEASE_BRANCH`):
Tails developers's avatar
Tails developers committed
828

829
830
831
832
833
        ( \
          cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
          git commit -m "Update upgrade-description files." && \
          git push origin ${WEBSITE_RELEASE_BRANCH:?} \
        )
Tails developers's avatar
Tails developers committed
834

835
1. If preparing anything but a final release (e.g. an alpha, beta
836
837
838
   or RC), copy the generated or updated files to
   `${MASTER_CHECKOUT:?}`, replace `channel: alpha` with `channel:
   test`, sign them, commit and push.
Tails developers's avatar
Tails developers committed
839

intrigeri's avatar
intrigeri committed
840
841
842
1. Else, if preparing a final release, copy the generated UDF for the previous
   release to the *test* channel in `$MASTER_CHECKOUT`, modify its content
   accordingly, sign it, commit and push:
843

844
845
846
        ( \
          cd ${MASTER_CHECKOUT:?} && \
          git fetch && \
anonym's avatar
anonym committed
847
          for old_version in ${IUK_SOURCE_VERSIONS:?}; do
848
849
850
851
852
853
854
855
856
857
858
859
            stable_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/stable/upgrades.yml" && \
            test_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/test/upgrades.yml" && \
            mkdir -p "$(dirname "$test_udf")" && \
            git show origin/${WEBSITE_RELEASE_BRANCH:?}:${stable_udf:?} \
              | sed -e 's/channel: stable/channel: test/' > ${test_udf:?} && \
            gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${test_udf:?} && \
            mv ${test_udf:?}.asc ${test_udf:?}.pgp && \
            git add ${test_udf:?}* ; \
          done && \
          git commit -m "Add incremental upgrades on the test channel for Tails ${VERSION:?}" && \
          git push origin master:master \
        )
860
861


862
863
Prepare the ISO description file for *Tails Verification*
---------------------------------------------------------
864

anonym's avatar
anonym committed
865
866
If preparing a RC, skip this part.

867
868
Update the ISO description file (IDF) used by the browser extension:

869
    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/amd64/stable/latest.yml <<EOF
870
    ---
871
    build-target: amd64
872
873
    channel: stable
    product-name: Tails
874
    version: '${VERSION:?}'
875
876
    target-files:
    - sha256: ${ISO_SHA256SUM}
877
      size: ${ISO_SIZE_IN_BYTES:?}
878
      url: http://dl.amnesia.boum.org/tails/stable/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso
879
    EOF
880
    ( cd "${RELEASE_CHECKOUT:?}" && \
anonym's avatar
anonym committed
881
      git add wiki/src/install/v1/Tails/amd64/stable/latest.yml && \
882
      git commit -m "Update IDF file for Tails Verification." )
883

884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
899
900
901
902
903
904
905
Done with OpenPGP signing
=========================

By now you are done with Tails signing key, so please make sure it is
not usable by your system any more.

<div class="note">

Beware! If your have to plug your OpenPGP smart card or reassemble the
key again after this point it invalidates <i>everything</i> done for
the [[reproduction of this release|test#reproducibility-final-check]]
so it has to be started from the beginning:

* the original text is restored on the pad, and
* some tester follows it from scratch, and
* the Trusted Reproducer follows awaits the new input from said tester
  and then starts from scratch.

So please try to avoid this!

</div>

Tails developers's avatar
Tails developers committed
906
907
908
Upload images
=============

Tails developers's avatar
Tails developers committed
909
910
911
Sanity check
------------

912
Verify once more that the Tor Browser we ship is still the most recent (see
913
above).
Tails developers's avatar
Tails developers committed
914

915
916
<a id="publish-iuk"></a>

intrigeri's avatar
intrigeri committed
917
918
Publish the ISO and IUKs over HTTP
----------------------------------
Tails developers's avatar
Tails developers committed
919

920
Upload the IUKs to our rsync server:
intrigeri's avatar
intrigeri committed
921
922
923
924
925
926
927

    for source_version in ${IUK_SOURCE_VERSIONS}; do
       rsync --partial --inplace --progress -v \
          "${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk" \
          rsync.lizard:
    done

928
Upload the ISO signature to our rsync server:
929

930
    scp "${ISO_PATH:?}.sig" rsync.lizard:
931
932
933
934
935
936

Pick a build from `$RELEASE_BRANCH` that produced an ISO identical to
the one you've built locally (`XXX` must be the job ID, i.e.
an integer):

    MATCHING_JENKINS_BUILD_ID=XXX
937

938
Copy the ISO to our rsync server and verify the signature:
939
940
941
942
943
944
945
946

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh rsync.lizard gpg --import
    ssh rsync.lizard << EOF
       wget \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,}
    EOF
intrigeri's avatar
intrigeri committed
947
948
949

Move files in place with proper ownership and permissions:

950
    ssh rsync.lizard << EOF
951
952
953
954
955
956
957
958
959
960
961
      sudo install -o root -g rsync_tails -m 0755 -d \
         /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
      sudo chown root:rsync_tails \
         tails-amd64-${VERSION:?}.iso* \
         Tails_amd64_*_to_${VERSION:?}.iuk && \
      sudo chmod u=rwX,go=rX \
         tails-amd64-${VERSION:?}.iso* \
         Tails_amd64_*_to_${VERSION:?}.iuk && \
      sudo mv tails-amd64-${VERSION:?}.iso* \
              /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
      sudo mv Tails_amd64_*_to_${VERSION:?}.iuk \
962
              /srv/rsync/tails/tails/${DIST:?}/iuk/
963
    EOF
964

965
Update the time in `project/trace` file on our rsync server
966
967
968
and on the live wiki (even for a release candidate):

	TRACE_TIME=$(date +%s) &&
969
970
971
	echo ${TRACE_TIME:?} | ssh rsync.lizard "cat > /srv/rsync/tails/tails/project/trace" && \
	[ -n "${MASTER_CHECKOUT:?}" ] && \
	echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
972
	(
973
	   cd "${MASTER_CHECKOUT:?}" && \
974
	   git commit wiki/src/inc/trace \
975
	      -m "Updating trace file after uploading ${VERSION:?}." && \
976
	   git push origin master
977
978
	)

979
980
981
982
983
984
985

## Announce, seed and test the Torrents

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh bittorrent.lizard gpg --import
    scp \
       "${ISOS:?}/tails-amd64-${VERSION:?}.torrent" \
intrigeri's avatar
intrigeri committed
986
       "${ISO_PATH:?}.sig" \
987
988
989
990
991
992
993
994
995
996
       bittorrent.lizard: && \
    ssh bittorrent.lizard << EOF
       mkdir --mode 0755 "tails-amd64-${VERSION:?}" && \
       mv "tails-amd64-${VERSION:?}.iso.sig" \
          "tails-amd64-${VERSION:?}/" && \
       cd "tails-amd64-${VERSION:?}" && \
       wget \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,} && \
       cd && \
997
998
       chgrp -R debian-transmission "tails-amd64-${VERSION:?}" && \
       chmod -R go+rX,g+w "tails-amd64-${VERSION:?}" && \
anonym's avatar
anonym committed
999
       mv \
1000
          "tails-amd64-${VERSION:?}" \
For faster browsing, not all history is shown. View entire blame