release_process.mdwn 64.9 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
8
9
10
11
12
13
14
15
16
17
18
Prepare your checklist
======================

1. Merge the current `master` branch into the branch used to prepare the release
   (`stable` for a bugfix release, `testing` for a RC or a major release).
2. Copy this very file from the branch used to prepare the release
   to some place outside of Git.
3. Close the version of this file that's in Git.
4. From now on, follow the instructions in your own local copy, using it as
   a checklist: delete steps once you've successfully completed them.
   This ensures you won't mistakenly skip a step, without relying on memory, so
   "oops, I forgot X" cannot happen.
19

20
21
22
Requirements
============

23
24
25
Packages
--------

26
27
To release Tails you'll need some packages installed:

28
* `docker.io gitlab-cli jq tidy mktorrent python3-bs4 python3-debian python3-gitlab python3-jinja2 python3-voluptuous transmission-cli`
29
30
* [[!debpts squashfs-tools]] 1:4.4-1+0.tails1
  from our custom `iukbuilder-stretch` APT suite.
31
* `perl5lib` [[dependencies|contribute/release_process/perl5lib#build-deps]]
intrigeri's avatar
intrigeri committed
32
* `iuk` [[dependencies|contribute/release_process/tails-iuk#build-deps]]
33
34
35
36
37
* `po4a` 0.55:
   - Different versions extract Markdown headings in a different way, which
     makes tons of strings fuzzy.
   - Different versions emit different sets of message flags, some of which
     are not supported by _i18nspector_ yet.
38
* `i18nspector` 0.26 or newer
39
* packages to [[build a local version of the website|contribute/build/website/]]
40

41
42
43
44
45
46
47
48
49
50
51
52
Configuration files
-------------------

To release Tails you need:

* `~/.python-gitlab.cfg`

  You need at least this content:

        [global]
        ssl_verify = true

53
        [TailsRM]
54
55
56
57
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

58
59
60
61
62
63
64
65
66
67
68
69
70
        [Tails]
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

   Then:

   - In the `TailsRM` section, set the value of the `private_token` option to
     the `role-release-automation` GitLab user's API token, which you'll find in
     `rm.git`'s keyringer.

   - In the `Tails` section, set the value of the `private_token` option to a
     GitLab API token for your own user.
71

72
73
74
75
76
77
* `~/.config/tails/release_management/local.yml`

  If you have no such file yet, copy
  `config/release_management/examples/local.yml` there and adjust to your
  local environment.

78
79
80
Environment
===========

81
82
83
To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

84
### Generate the base environment
85

86
87
1. To generate the `~/.config/tails/release_management/current.yml` template,
   run:
88

89
90
91
92
        ./bin/rm-config generate-boilerplate

2. Edit `~/.config/tails/release_management/current.yml` and replace
  `FIXME`:s:
93

94
95
96
97
98
99
100
        "${EDITOR:?}" ~/.config/tails/release_management/current.yml

3. Generate the resulting environment variables and export them into your
   environment:

        . $(./bin/rm-config generate-environment)

101
### Notes
102

103
104
105
* Unless the release process explicitly instructs you to change the
  value of one such variable, treat it as a constant: else,
  inconsistency will surely arise, which can cause trouble later on.
106
107
108
109
110
111
112
113
114
115
116
117
118
119
* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

120
121
122
Pre-freeze
==========

123
124
125
126
127
Check the Release Manager role
------------------------------

Check the [[contribute/working_together/roles/release_manager]] role
documentation: it has more tasks that should be done early enough.
128
129
130
131

Coordinate with Debian security updates
---------------------------------------

132
133
134
135
136
137
138
Look at the [list of upcoming Debian Security Advisories (DSA)][DSA needed]
found in the Debian [security tracker][web]'s [Git repository][git].

[web]: http://security-tracker.debian.org/tracker/
[git]: https://salsa.debian.org/security-tracker-team/security-tracker
[DSA needed]: https://salsa.debian.org/security-tracker-team/security-tracker/blob/master/data/dsa-needed.txt

139
Rationale: we don't want to release Tails and learn 2 days later about
140
141
important security issues fixed by Debian security updates. In some cases, it
may be worth delaying a Tails release a bit, while waiting for a DSA to happen.
142

143
144
145
146
Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
147
148
149
150
151
152
153
and check that the jobs for the release branch have good enough results:

 - the last build and reproducibility testing builds have succeeded;
 - look at the last build of the relevant `test_Tails_ISO_*` job;
   if anything looks unusual or fishy, check the few previous builds
   to see if it's a one-off robustness issue, a serious regression,
   or what not.
154

155
156
157
158
159
160
Freeze
======

Major release
-------------

161
162
Skip this if you're preparing the images for a final (non-RC) major release.

163
164
If we are at freeze time for a major release (i.e. preparing its
release candidate):
165
166
167

1. Merge the `master` Git branch into `devel`:

168
        git checkout devel && git fetch origin && git merge origin/devel && git merge --no-ff origin/master
169

170
171
172
173
2. Merge each APT overlay suite listed in the `devel` branch's
   `config/APT_overlays.d/` into the `devel` APT suite:

        ./bin/merge-APT-overlays devel
174
175
176

3. Merge the `devel` Git branch into the `testing` one:

177
        git checkout testing && git merge origin/testing && git merge devel
178
179
180
181

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

182
183
184
185
4. Hard reset the `testing` custom APT suite to the current state of the
   `devel` one:

        ./bin/reset-custom-APT-suite testing devel
186

187
188
189
5. Freeze the [[time-based APT repository
   snapshots|APT_repository/time-based_snapshots]] that shall be used
   during the code freeze, and those used by our builder VM:
190

191
        ./bin/freeze-all-APT-snapshots
192
193

6. Make it so the time-based APT repository snapshots are kept around
194
   long enough, by bumping their `Valid-Until` half a year from now:
195

196
197
        git checkout "$RELEASE_BRANCH" && \
        ./bin/bump-APT-snapshots-expiration-date 183
198

199
200
Bugfix release
--------------
201

202
If we are at freeze time for a bugfix release:
203
204
205

1. Merge the `master` Git branch into `stable`:

206
        git checkout stable && git fetch && git merge origin/stable && git merge --no-ff origin/master
207

208
209
210
211
2. Merge each APT overlay suite listed in the `stable` branch's
   `config/APT_overlays.d/` into the `stable` APT suite:

        ./bin/merge-APT-overlays stable
212

213
214
Common steps for bugfix and major releases
------------------------------------------
215

216
Reset the release branch's `config/base_branch`:
217

218
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
219
           git commit config/base_branch \
220
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
221

222
223
224
Bootstrap manual testing coordination:

1. Create a pad.
225
226
227
228
229
230
2. Set the `PAD` environment variable to the URL of this pad.
3. Copy the [[manual test suite|contribute/release_process/test]]
   into this pad.
4. Generate boilerplate for the call for manual testing:

        ./bin/generate-call-for-manual-testers \
231
           --pad "${PAD:?}" \
232
233
234
235
           --version "${VERSION:?}"

5. Send the call for manual testing:

intrigeri's avatar
intrigeri committed
236
    - encrypt and sign your email
237
    - use the headers and body generated above
238

Tails developers's avatar
Tails developers committed
239
240
241
Update included files
=====================

242
243
244
Upgrade Tor Browser
-------------------

245
If you did not import the new Tor Browser yet, [[do so now|tor-browser]].
246

247
248
Update PO files
---------------
249

250
251
252
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
253

254
	cd "${RELEASE_CHECKOUT:?}" && \
255
256
	./import-translations  && \
	./refresh-translations && \
257
	./submodules/jenkins-tools/slaves/lint_po && \
258
	git add po && git commit -m 'Update PO files.'
259

260
If `lint_po` complains:
261

262
* rollback the offending PO files and retry; worst case, delete it
263
* send a note to <tails-l10n@boum.org> [public] so that they get in touch with
264
265
  whoever can fix them.

266
267
When preparing an actual release
================================
268

269
If we're about to prepare the images for a final (non-RC) release, then
270
271
272
273
274
follow these instructions:

Major release
-------------

275
276
277
278
Merge each APT overlay suite listed in the `testing` branch's
`config/APT_overlays.d/` into the `testing` custom APT suite:

        ./bin/merge-APT-overlays testing
279

280
281
Bugfix release
--------------
282

283
<div class="note">
284
For bugfix releases, we generally do not put any RC out, so freeze time
285
286
287
288
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
289

290
291
292
293
Merge each APT overlay suite listed in the `stable` branch's
`config/APT_overlays.d/` into the `stable` custom APT suite:

        ./bin/merge-APT-overlays stable
294

anonym's avatar
anonym committed
295
296
Update other base branches
==========================
297

298
299
300
1. Merge the release branch into `devel`:

        ./bin/merge-main-branch "${RELEASE_BRANCH:?}" devel
301

302
2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
303
   branch, the time-based APT repository snapshots being used
304
305
306
307
308
309
310
311
312
313
   during the freeze:

        git checkout devel && \
        ./auto/scripts/apt-snapshots-serials thaw && \
        git commit \
            -m "Thaw APT snapshots." \
            config/APT_snapshots.d/*/serial || :

   It's fine if that results in a no-op (it depends on how exactly previous
   operations were performed).
intrigeri's avatar
intrigeri committed
314

intrigeri's avatar
intrigeri committed
315
316
317
318
319
320
321
322
323
324
3. Try to merge the `devel` Git branch into `feature/bullseye` (if it exists),
   *without* merging the corresponding APT suite (for now `feature/bullseye` is
   handled as any other topic branch forked off `devel`: its base branch is set
   to `devel`):

        if git describe feature/bullseye >/dev/null 2>&1; then
           git checkout feature/bullseye && \
           git pull && \
           git merge devel
        fi
intrigeri's avatar
intrigeri committed
325

326
327
328
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.
329

330
4. Ensure that the release, `devel` and `feature/bullseye` (if it exists) branches
331
332
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
333

334
5. Push the modified branches to Git:
335

336
        git push origin                          \
337
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
338
           $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
339
           devel:devel
340
341
342

Update more included files
==========================
343

344
345
Changelog
---------
Tails developers's avatar
Tails developers committed
346

347
348
349
If `debian/changelog` on `$RELEASE_BRANCH` has an `UNRELEASED` entry
for a version lower than `$VERSION`, remove it.

350
Update the Changelog entry for the release you're preparing:
351

352
	git checkout "${RELEASE_BRANCH:?}" && \
353
	./bin/update-changelog --version "${MILESTONE:?}"
354

355
Then, gather other useful information from the diff between the previous
intrigeri's avatar
intrigeri committed
356
357
358
359
360
361
362
363
364
365
version's `.packages` file and the one from the to-be-released images:

1. Generate the diff:

        diff --color=never -u \
           "wiki/src/torrents/files/tails-amd64-${PREVIOUS_STABLE_VERSION}.packages" \
           <(curl --silent "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH}/lastSuccessful/archive/latest.packages") \
        | wdiff --diff-input  --terminal

2. In the diff, look for:
366

367
368
369
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
370

intrigeri's avatar
intrigeri committed
371
Finally, sanity check the version, commit, and push:
Tails developers's avatar
Tails developers committed
372

373
374
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
375
	else
376
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
intrigeri's avatar
intrigeri committed
377
378
	fi && \
    git push origin "${RELEASE_BRANCH:?}"
Tails developers's avatar
Tails developers committed
379

sajolida's avatar
sajolida committed
380
381
382
Notify technical writers on the GitLab issue about the release notes
that the changelog is ready.

Bessemer's avatar
Bessemer committed
383
Included website
384
385
----------------

386
    ./bin/prepare-included-website-for-release
387

388
389
390
While the website is building, if this is not a good time for taking a break,
you can follow the "Enable OpenPGP signing" instructions.

391
Enable OpenPGP signing
Tails developers's avatar
Tails developers committed
392
393
======================

394
395
### If you have an OpenPGP smart card

396
397
398
399
If you have an OpenPGP smart card with a Tails signing subkey,
go fetch it.

Only plug it when `gpg` asks you to.
400
401
402
403
404
405

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.
406

Tails developers's avatar
Tails developers committed
407
You should never import the Tails signing key into your own keyring,
408
409
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
410
411

    export GNUPGHOME=$(mktemp -d)
412
413
414
415
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
416
417
418
419
420
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

421
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
422

423
424
Build the almost-final images
=============================
425

426
427
428
429
430
1. [[Build ISO and USB images|contribute/build]] from the release branch,
   with `$TAILS_BUILD_OPTIONS` set like this:
   - Set `defaultcomp`, so we can more accurately optimize our
     SquashFS file ordering.
   - Do _not_ set `keeprunning` nor `rescue`.
431

432
2. Keep the resulting build artifacts until the end of this release process.
433

434
3. Record where the manifest of needed packages is stored:
435

436
437
438
   1. To update the `~/.config/tails/release_management/current.yml` template
      with newly required variables, run:

439
440
        ./bin/rm-config generate-boilerplate --stage built-almost-final

441
   2. Edit `~/.config/tails/release_management/current.yml` and replace
442
443
      `FIXME`:s:

444
445
        "${EDITOR:?}" ~/.config/tails/release_management/current.yml

446
447
448
   3. Generate the resulting environment variables and export them
      into your environment:

449
        . $(./bin/rm-config generate-environment --stage built-almost-final)
450

451
452
453
Tag the release in Git
======================

454
455
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
456
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"
457

458
Known limitations:
459

460
461
462
463
- Pushing the tag is needed so that the APT repository is updated, and the Tails
  APT configuration works at build and boot time. It might be premature, as
  testing might reveal critical issues, but this is a signed tag, so it can be
  overridden later.
464

465
466
467
- From this push of a tag, the release branch will fail to build because the
  last changelog entry is unreleased but corresponds to an existing tag.
  Don't worry about it, this will be fixed shortly.
468

469
470
471
Prepare the versioned APT suites
================================

472
473
Versioned APT suite in our custom APT repository
------------------------------------------------
474

475
476
477
Within a few minutes after pushing the new release's Git tag, a cronjob
creates a new APT suite in our custom APT repository. This new APT suite is
called `$TAG` and is initially empty.
478

479
480
481
Wait for this APT suite to be created and initialize it with the packages
currently found in the APT suite corresponding to the branch used to prepare
the release:
482

483
484
485
486
487
      while ! ssh reprepro@incoming.deb.tails.boum.org reprepro list "${TAG:?}" >/dev/null 2>&1; do
         sleep 5
      done && \
      ssh reprepro@incoming.deb.tails.boum.org \
         tails-merge-suite "$RELEASE_BRANCH" "$TAG"
488

489
490
491
492
Tagged snapshots of upstream APT repositories
---------------------------------------------

Create tagged snapshots of upstream APT repositories:
493

494
      ./bin/tag-apt-snapshots "${ALMOST_FINAL_BUILD_MANIFEST:?}" "${TAG:?}"
495

496
Note:
497

intrigeri's avatar
intrigeri committed
498
  - This command takes a while (about a dozen minutes).
499
500
    While it's running, if you don't feel the need to take a break,
    you can proceed with the "SquashFS file order" section below.
501
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
502
    [[custom APT repository|APT_repository/custom]] are
503
    listed under "some packages were not found anywhere" (because we
intrigeri's avatar
intrigeri committed
504
    are currently not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
505
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
506
    Now, we have a "safety" net, in case you don't notice such a problem: if
507
508
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
509

Tails developers's avatar
Tails developers committed
510
511
512
Build images
============

513
514
515
Sanity check
------------

516
Verify that the Tor Browser release used in Tails still is the most
517
518
519
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:
520

521
* <https://gitweb.torproject.org/builders/tor-browser-build.git>
522
  (`maint-X.Y-desktop` branch)
523
524
* <https://gitweb.torproject.org/tor-browser.git>

525
A new tag may indicate that a new Tor Browser release or rebuild is imminent.
526
527
528

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
529
530
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
531

532
533
Feel free to skip this section when preparing an emergency release.

534
1. Install the almost-final USB image to a USB stick.
535
536
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
537
538
1. If possible, set up a wired connection to avoid having to deal
   with wireless settings.
539
1. Boot this USB stick **on bare metal** again.
540
1. Add `profile` to the kernel command-line.
541
1. Login with the default settings in the Welcome Screen (e.g. do not configure
542
   an _Administration Password_).
543
1. Unless you've set up a wired connection, connect to Wi-Fi.
544
1. Wait for the "Tor is ready" notification.
intrigeri's avatar
intrigeri committed
545
1. Start *Tor Browser*.
Tails developers's avatar
Tails developers committed
546
1. A few minutes later, once the `boot-profile` process has been
547
   killed, retrieve the new sort file from `/var/log/boot-profile`.
intrigeri's avatar
intrigeri committed
548
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
549
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
550
1. To remove:
551

552
553
554
555
556
557
558
559
   - runtime-generated files that don't exist in the rootfs,
     in order to avoid confusing noise in the build output
   - the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
   - some hardware-dependent files

   Run this command:

560
           ./bin/clean-squashfs-sort-file config/binary_rootfs/squashfs.sort
561

intrigeri's avatar
intrigeri committed
562
1. Inspect the Git diff (including diff stat), apply common sense:
563

564
        diff -NaurB \
intrigeri's avatar
intrigeri committed
565
566
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
567
            | less
568

569
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`
570
1. Clean up: `rm -f config/binary_rootfs/squashfs.sort.old`
571

572
573
Build the final images
----------------------
574

Bessemer's avatar
Bessemer committed
575
Then all included files should be up-to-date and the versioned APT
576
577
suite should be ready, so it is time to:

578
1. Mark the version as "released" in the changelog:
579

580
        dch --release --no-force-save-on-release --maintmaint && \
581
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
582

583
1. Export `SOURCE_DATE_EPOCH`:
584

585
        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')
586

587
1. tag the release *again*, with all included files in:
588

589
590
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
591
592
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
593

594
1. build the final images!
595
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
596
597
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.
598

599
600
1. Make sure the Jenkins build starts as soon as possible. If needed, remove
   other builds from the _Build Queue_ or cancel other running builds.
601

intrigeri's avatar
intrigeri committed
602
1. Compare the new build manifest with the one from the previous,
603
   almost-final build:
intrigeri's avatar
intrigeri committed
604
605

        diff -Naur \
606
           "${ALMOST_FINAL_BUILD_MANIFEST:?}" \
607
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"
intrigeri's avatar
intrigeri committed
608
609
610

   They should be identical, except that the `debian-security` serial might be higher.

intrigeri's avatar
intrigeri committed
611
<a id="reproducibility-sanity-check-iso"></a>
612

intrigeri's avatar
intrigeri committed
613
614
615
Verify that Jenkins reproduced your images
------------------------------------------

intrigeri's avatar
intrigeri committed
616
To verify that Jenkins reproduced your images:
617

intrigeri's avatar
intrigeri committed
618
1. Visit the URL printed by this command:
anonym's avatar
xxx    
anonym committed
619
620
621

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

622
2. Find the relevant build (probably the last one)
623
   and compare the SHA512 hashes of the ISO and USB images built by Jenkins
intrigeri's avatar
intrigeri committed
624
625
   (in the `tails-build-artifacts.shasum` build artifact) with the ones
   of the images you built.
626

intrigeri's avatar
intrigeri committed
627
3. Then:
628

629
630
   - If the ISO and USB images hashes match: yay, we're good to go!
     The `.build-manifest` may differ — that's OK.
631
632
633
634
     Then:

     1. To update the `~/.config/tails/release_management/current.yml` template
        with newly required variables, run:
635

636
637
638
639
            ./bin/rm-config generate-boilerplate --stage reproduced-images

     2. Edit `~/.config/tails/release_management/current.yml`
        and set the `matching_jenkins_images_build_id` value
640
641
642
        to the ID of this job (an integer):

            "${EDITOR:?}" ~/.config/tails/release_management/current.yml
643
644
645

     3. Generate the resulting environment variables and export them
        into your environment:
646

647
            . $(./bin/rm-config generate-environment --stage reproduced-images)
648

649
   - If there is a hash mismatch for one of the images: ouch! Now we are in a
650
651
652
653
654
655
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:
656

anonym's avatar
anonym committed
657
658
659
660
661
662
663
664
          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso
665

666
667
     Do the same for the USB image as well.

668
     Then carefully investigate the `diffoscope` report:
669

670
671
       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
intrigeri's avatar
intrigeri committed
672
         release. Halt the release, involve the rest of <tails@boum.org>, and then
673
674
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!
675

676
       - Otherwise, if the change is definitely harmless:
677

678
         * If the source of non-determinism is identified quickly and
679
           is easy and fast to fix, *and* the QA of the current images
680
681
682
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:
683

684
685
686
687
688
           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).
689

690
691
692
         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
intrigeri's avatar
intrigeri committed
693
           release notes, linking to the issue where
694
695
           the nature of the reproducibility failure is clearly
           described.
696

intrigeri's avatar
intrigeri committed
697
698
Initialize the website release branch
-------------------------------------
intrigeri's avatar
intrigeri committed
699

700
701
702
703
704
705
From now on, we don't want to push new commits on `$RELEASE_BRANCH`
until the new release is out. Otherwise, this would break its build
and the build of every branch based on it, which would effectively
block other development work. So the final steps towards publishing
the release will be done in a new, dedicated branch.

intrigeri's avatar
intrigeri committed
706
707
If preparing anything but a final release (e.g. an alpha, beta
or RC):
708
709
710
711

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

intrigeri's avatar
intrigeri committed
712
Else, if preparing a final release:
713

714
715
        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
716

717
718
719
Generate the OpenPGP signatures and Torrents
============================================

intrigeri's avatar
intrigeri committed
720
Create a directory with a suitable name, go there, move the built
721
722
723
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
intrigeri's avatar
intrigeri committed
724
725
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:
726

727
    ./bin/generate-images-signatures-and-torrents
728

729
730
<a id="prepare-iuk"></a>

731
732
733
Prepare incremental upgrades
============================

734
735
736
Since Tails 4.2, we use a new upgrade scheme, which fundamentally
changes what the source version number of an upgrade means: it's now
the version that was *initially installed* and *not* the currently
737
738
running version. For details, see the corresponding
[[design documentation|contribute/design/incremental_upgrades]].
739

740
741
742
743
744
745
746
747
748
749
750
751
752
753
754
755
756
757
Sanity checks
-------------

Check that you have the correct version of `squashfs-tools` installed:

    [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
      = '1:4.4-1+0.tails1' \
    ] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for <https://iso-history.tails.boum.org> (it
will be used by `wrap_tails_create_iuks` below, in order to download any ISO
that is not present locally yet):

    if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
        echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
        echo "The corresponding login and password are in the RMs' keyringer."
    fi

758
759
Build the Incremental Upgrade Kits locally
------------------------------------------
760

761
762
763
764
765
766
767
768
769
Verify there's enough free disk space in `$IUKS_DIR`:

    MIN_DISK_SPACE=15000
    FREE_DISK_SPACE=$(/bin/df --block-size=M --output=avail "${IUKS_DIR:?}" \
       | tail -n1 | sed --regexp-extended 's,M$,,')
    if [ "$FREE_DISK_SPACE" -lt "$MIN_DISK_SPACE" ]; then
       echo "ERROR! Not enough free space in ${IUKS_DIR:?}"
    fi

intrigeri's avatar
intrigeri committed
770
To build the IUKS, run this command:
771

772
773
774
    (
       set -eu
       WORK_DIR=$(mktemp -d)
775
       TAILS_REMOTE="$(git -C "${RELEASE_CHECKOUT?:}" remote get-url origin)"
776
       PUPPET_TAILS_REMOTE=$(echo -n "${TAILS_REMOTE?:}" | perl -p -E 's,:tails/tails(?:[.]git)?\z,:tails/puppet-tails,')
777
       cd "${WORK_DIR?:}"
778
       git clone "$PUPPET_TAILS_REMOTE"
779
       time                                                                   \
780
       sudo                                                                   \
781
       ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
782
783
784
785
           --tails-git-remote "file://${RELEASE_CHECKOUT?:}/.git"             \
           --tails-git-commit "${TAG?:}"                                      \
           --source-date-epoch "${SOURCE_DATE_EPOCH?:}"                       \
           --local-isos-dir "${ISOS?:}"                                       \
786
           --tmp-dir "${TMPDIR:-/tmp}"                                        \
787
788
789
           --output-dir "${IUKS_DIR?:}"                                       \
           --source-versions "${IUK_SOURCE_VERSIONS?:}"                       \
           --new-version "${VERSION?:}"                                       \
790
791
           --verbose                                                          \
           --jobs "$(grep '^core id' /proc/cpuinfo | sort -u | wc -l)"
intrigeri's avatar
intrigeri committed
792
    ) && \
793
794
795
     cd "${IUKS_DIR?:}" && \
     sha256sum Tails_amd64_*_to_${VERSION?:}.iuk > "${IUKS_HASHES?:}"

796
797
798
Take note of the `time` output above and add it to the table in
[[!tails_ticket 17750]].

799
This command takes a long time. In parallel, while it is running,
intrigeri's avatar
intrigeri committed
800
you can follow the next steps:
801

intrigeri's avatar
intrigeri committed
802
 - ISO history
803
804
805
806
807
 - Build the Incremental Upgrade Kits on Jenkins

ISO history
-----------

808
In this section, you will push the released ISO and USB images and their artifacts (`.buildlog`,
809
810
811
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

812
 - The Jenkins `build_IUKs` job can fetch them.
intrigeri's avatar
intrigeri committed
813
 - Our isotesters can fetch them from there for their testing.
814

815
816
Make sure that Jenkins has finished building so you have set
`MATCHING_JENKINS_IMAGES_BUILD_ID` (see above), then run:
intrigeri's avatar
intrigeri committed
817

818
819
820
821
822
823
    cd "${RELEASE_CHECKOUT:?}" && \
    ./bin/add-release-to-iso-history \
        --version "${VERSION:?}" \
        --isos "${ISOS:?}" \
        --release-branch "${RELEASE_BRANCH:?}" \
        --matching-jenkins-images-build-id "${MATCHING_JENKINS_IMAGES_BUILD_ID:?}"
824

825
Then, wait (a few minutes, `*/15` crontab) until the images appear
826
on <https://iso-history.tails.boum.org/?C=M&O=D>.
827

828
While waiting, you can prepare the `build_IUKs` job and build in the
829
830
next section. But do **not** click _Build_ until the images have appeared in the
ISO history.
intrigeri's avatar
intrigeri committed
831

832
833
834
<a id="build-iuks-on-jenkins"></a>

Build the Incremental Upgrade Kits on Jenkins
835
836
837
838
839
---------------------------------------------

1. Make sure the push to ISO history (started in the previous section)
   has finished, and that images have appeared on the web server:
   <https://iso-history.tails.boum.org/>
840

841
1. On <https://jenkins.tails.boum.org/job/build_IUKs/configure>, adjust
842
843
844
   the `SOURCE_VERSION` axis to list all versions in `$IUK_SOURCE_VERSIONS`,
   and save the updated configuration.

845
1. On <https://jenkins.tails.boum.org/job/build_IUKs/build?delay=0sec>,
846
847
848
   fill the form with these values:

    - `TAILS_GIT_COMMIT`: the value of `$TAG` in your release environment
849
    - `SOURCE_DATE_EPOCH`: the value of `$SOURCE_DATE_EPOCH` in your
850
851
852
853
854
855
856
857
858
      release environment
    - `NEW_VERSION`: the value of `$VERSION` in your release environment
    - `EXTRA_ARGS`: leave it blank

2. Click the _Build_ button

3. After a few seconds, a new build appears on top of the _Build
   History_ sidebar. Click on the progress bar of this new build.

859
4. Record the ID of the candidate Jenkins `build_IUKs` build:
860
861
862
863
864
865
866
867

   1. To update the `~/.config/tails/release_management/current.yml` template
      with newly required variables, run:

          ./bin/rm-config generate-boilerplate --stage built-iuks

   2. Edit `~/.config/tails/release_management/current.yml`
      and set the `candidate_jenkins_iuks_build_id` value
868
      to the ID of that `build_IUKs` build (an integer):
869
870

          "${EDITOR:?}" ~/.config/tails/release_management/current.yml
871
872
873
874

   3. Generate the resulting environment variables and export them
      into your environment:

intrigeri's avatar
intrigeri committed
875
          . $(./bin/rm-config generate-environment --stage built-iuks)
876

877
5. Wait until the `build_IUKs` build completes successfully.
878
   It should take about 10-15 minutes for each member of
879
   the `$IUK_SOURCE_VERSIONS` list, distributed across `isobuilderN` workers.
880

881
6. Add the time it took for `build_IUKs` to the table of [[!tails_ticket 17750]].
882

883
884
<a id="reproducibility-sanity-check-iuk"></a>

885
886
887
888
889
Verify that Jenkins reproduced your IUKs
----------------------------------------

    "${RELEASE_CHECKOUT:?}"/bin/copy-iuks-to-rsync-server-and-verify \
       --hashes-file "${IUKS_HASHES:?}" \
890
       --work-dir /srv/tmp \
891
892
893
894
895
896
       --jenkins-build-id "${CANDIDATE_JENKINS_IUKS_BUILD_ID:?}"

If this verification succeeds, move on to the next section.

Else, if this verification fails, then:

897
1. Visit this page: `https://jenkins.tails.boum.org/job/build_IUKs/lastBuild/parameters/`
898

899
900
   It tells you which parameters you've passed to the upstream Jenkins
   job, which spawned the many workers. Make sure to use the build
901
   history browser to find the right `build_IUKs` job in case
902
903
   there were several attempts; the URL above assumes the last build
   is what you're interested in (see `lastBuild` part).
904
905
906
907
908
909
910
911
912

2. Double-check that you've passed the correct parameters.

   If you notice you made a mistake, [[build the IUKs on Jenkins
   again|release_process#build-iuks-on-jenkins]], and do the
   verification again.

   Else, if the parameters where correct, then follow the next steps.

intrigeri's avatar
intrigeri committed
913
914
915
916
3. Leave the IUKs built by Jenkins in `rsync.lizard:/srv/tmp`: at least in some
   cases, this will speed up uploading your own IUKs later by a 500+ factor,
   thanks to `rsync --partial --inplace`.

intrigeri's avatar
intrigeri committed
917
3. File an issue about this problem.
918
919
920

   Specify:

921
    - Which set of parameters you've passed to the *build_IUKs*
922
923
924
925
926
927
928
      job, so that the person who'll investigate the problem
      can reproduce it.
    - The ID of the build that failed to reproduce your
      locally-built IUKs.

4. Later on, after you're done with OpenPGP signing,
   you will upload the IUKs you've built locally.
929

Tails developers's avatar
Tails developers committed
930
<a id="prepare-upgrade-description-files"></a>
931

Tails developers's avatar
Tails developers committed
932
933
Prepare upgrade-description files
---------------------------------
934

Tails developers's avatar
Tails developers committed
935
1. Prepare upgrade-description files (see the [[upgrade-description
936
   files
Tails developers's avatar
Tails developers committed
937
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
938
939
940
941
942
943
944
945
946
947
948
949
950
951
952
953
   for details).

   <div class="caution">
   In <emph>most</emph> cases the example command below is exactly the
   one you should run. But in order to tell whether you're in one of
   the exceptional cases when you have to adjust that command, it's
   important that you understand what follows.
   </div>

   At this step, we use the
   `tails-iuk-generate-upgrade-description-files` tool in order to:

   1. Create a new upgrade-description for the version being released
      and for the next one, that expresses that *no* upgrade is
      available for these ones yet.

954
      This is what the `--version` and `--next_version` arguments
955
956
      in the example command below do. You do not need to modify them.

957
958
959
    2. For every recent previous release that's _not_ listed in
       `$IUK_SOURCE_VERSIONS` (i.e. those based on an older Debian
       release): tell the users of that old version they need to
960
       manually upgrade to the version being released.
961
962
963
964

       To do so, pass this previous version to `--previous_version`.
       You need to do this manually as the example command below won't
       do it automatically.
965
966

   Run this command, after adjusting it if needed as explained above:
967

968
        ${RELEASE_CHECKOUT:?}/config/chroot_local-includes/usr/src/iuk/bin/tails-iuk-generate-upgrade-description-files \
969
            --version "${VERSION:?}" \
970
971
972
            $( \
                if [ "${DIST:?}" = stable ]; then
                    echo \
973
974
975
976
                        --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
                        --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}~rc1" \
                        --next_version "${NEXT_PLANNED_BUGFIX_VERSION:?}" \
                        --next_version "${NEXT_POTENTIAL_EMERGENCY_VERSION:?}"
977
                else
978
                    echo \
979
                        --next_version "${NEXT_PLANNED_MAJOR_VERSION:?}" \
980
                        --next_version "${SECOND_NEXT_PLANNED_MAJOR_VERSION:?}"
981
982
                fi
            ) \
983
            $( \
984
                for version in $(echo ${IUK_SOURCE_VERSIONS:?}); do
985
                   echo "--previous_version ${version:?}"
986
987
                done \
            ) \
anonym's avatar
anonym committed
988
            --iso "${ISO_PATH:?}" \
989
            --iuks "${IUKS_DIR:?}" \
990
991
            --release_checkout "${RELEASE_CHECKOUT:?}" \
            --major_release "${MAJOR_RELEASE:?}" \
992
            --channel "${DIST:?}"
993
994
995
996

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

997
        ./bin/sign-updated-udfs
998

Tails developers's avatar
Tails developers committed
999
1. Add and commit the upgrade-description files and their detached
1000
   signatures to the Git branch used to prepare the release
For faster browsing, not all hist