release_process.mdwn 43.5 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
8
9
10
11
12
13
14
15
16
Requirements
============

To release Tails you'll need some packages installed:

* mktorrent and transmission-cli
* Aufs DKMS module for your running kernel.
* tails-iuk dependencies (see debian/control in the `debian` branch of its repo)
* tails-perl5lib dependencies (same trick than tails-iuk to get the list)

17
18
19
20
21
22
Environment
===========

Export the following environment variables to be able to copy'n'paste
the scripts snippets found on this page:

23
* version numbers (see [[contribute/release_schedule#versioning]]):
24
25

      export VERSION=$(dpkg-parsechangelog -SVersion)
26
      export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
27
28
      export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)

29
* `NEXT_PLANNED_VERSION`: set to the version number of the next Tails release
30
  (e.g. 0.23 when releasing 0.22.1, and 1.3 when releasing 1.2)
31
* `MAJOR_RELEASE`: set to 1 if preparing a major release, to 0 else
32
33
* `ISOS`: the directory where one stores `tails-i386-*`
  sub-directories like the ones downloaded with BitTorrent.
34
35
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
36
37
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
38
39
* `RELEASE_BRANCH`: the name of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
40
41
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
42
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
43
44
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
45
46
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
47
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
48

49
50
51
52
Also configure dch for this session:

    export DEBEMAIL='Tails developers <tails@boum.org>'

53
54
55
Pre-freeze
==========

56
57
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
58

59
60
61
62
63
64
Update Icedove preferences
------------------------------

* update `extensions.enigmail.configuredVersion` in
  `config/chroot_local-includes/etc/skel/.icedove/profile.default/preferences/0000tails.js`

65
66
67
68
69
Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

70
71
72
Select the right branch
=======================

73
74
What we refer to as the "release branch" (and `RELEASE_BRANCH`) should
be `testing` for major releases, and `stable` for point-releases.
75

76
<div class="caution">
77
Read the remainder of this document from the branch used to prepare the release!
78
79
</div>

80
81
82
83
84
85
86
87
88
89
90
91
Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
92
93
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
94
95
96
97
98
99
100
101
102
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

103
104
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
105

anonym's avatar
anonym committed
106
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
107
108
109
110
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
111
112
113
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
114
115
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

116
117
118
119
120
121
122
123
124
125

Point-release
-------------

If we are at freeze time for a point-release:

1. Merge the `master` Git branch into `stable`:

        git checkout stable && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
126
127
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
128
129
130
131
132
   APT suite.

Common steps for point and major releases
-----------------------------------------

133
Reset the release branch's `config/base_branch`:
134

135
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
136
           git commit config/base_branch \
137
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
138

Tails developers's avatar
Tails developers committed
139
140
141
Update included files
=====================

spriver's avatar
spriver committed
142
uBlock patterns and settings file
Tails developers's avatar
Tails developers committed
143
144
----------------

spriver's avatar
spriver committed
145
146
The patterns+settings file is stored as a converted sqlite text dump in
`config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`.
Tails developers's avatar
Tails developers committed
147

148
1. Boot Tails
spriver's avatar
spriver committed
149
150
151
152
2. Start the Tor Browser and open the uBlock dashboard by clicking on the uBlock icon.
3. Select the tab *3rd-party filters*
4. Click on the button *Update now* to update all used patterns
5. Close the Tor Browser
153
154
155
156
157
158
159
7. Copy the `.tor-browser/profile.default/extension-data/ublock0.sqlite`
   from this Tor Browser instance into the root of Tails' Git repo and
   run the following command:

       echo '.dump' | sqlite3 ublock0.sqlite | \
           grep -v "cached_asset_content://cache://compiled-" | \
           awk '!/^INSERT/; /^INSERT/ {print $0 | "sort -n"}' | \
160
161
           sed 's_\\n_\\n\r\n_g' | \
           sed "/^INSERT INTO \"settings\" VALUES('\(remoteBlacklists\|cached_asset_entries\)'/"'s_,_,\r\n_g' > \
162
163
           config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`

164
8. Commit:
Tails developers's avatar
Tails developers committed
165

166
       git commit -m 'Update uBlock Origin patterns + settings file.' \
spriver's avatar
spriver committed
167
168
          config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump

169
9. Remove the original `ublock0.sqlite` from the Git root.
170

Bessemer's avatar
Bessemer committed
171
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
172
173
--------------------------------------

174
175
176
Skip this section unless we are at freeze time for a major release
(i.e. we are about to prepare a release candidate).

177
That is: make sure the bundled binary Debian packages contain
178
179
up-to-date localization files.

180
181
182
183
184
185
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
and then run the `import-translations` script that is in the
main Tails repository. For example:

	cd whisperback
186
	"${RELEASE_CHECKOUT:?}"/import-translations
187

WinterFairy's avatar
WinterFairy committed
188
189
If the `import-translations` script fails to import translations for
the current package, manually copy updated PO files from the
190
Transifex branches of `git://git.torproject.org/translation.git` (e.g.
191
192
`whisperback_completed`) instead. In this case, skip PO files for
[[translation teams that use Git|contribute/how/translate#translate]].
193

194
Add and commit.
195

196
197
Then check the PO files:

198
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po
199
200

Correct any displayed error, then commit the changes if any.
201

202
Then see the relevant release processes, and upload the packages to
203
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
204

205
* [[tails-installer]]
206
* [[tails-greeter]]
207
* [[perl5lib]]
208
* [[persistence-setup]]
209
* [[tails-iuk]]
210
* whisperback:
211
212
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package
Tails developers's avatar
Tails developers committed
213

intrigeri's avatar
intrigeri committed
214
215
Upgrade Tor Browser
-------------------
216
217
218

See the dedicated page: [[tor-browser]]

219
220
221
222
223
Upgrade Icedove
---------------

See the dedicated page: [[icedove]]

224
225
Update PO files
---------------
226

227
228
229
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
230

231
	cd "${RELEASE_CHECKOUT:?}" && \
232
233
	./import-translations  && \
	./refresh-translations && \
234
	./submodules/jenkins-tools/slaves/check_po && \
235
	git add po && git commit -m 'Update PO files.'
236

237
238
239
240
241
242
If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

243
244
When preparing an actual release
================================
245

246
247
248
249
250
251
If we're about to prepare an image for a final (non-RC) release, then
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
252
253
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
254
custom APT suite.
255
256
257
258

Point-release
-------------

259
260
261
262
263
264
<div class="note">
For point-releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
265

intrigeri's avatar
intrigeri committed
266
267
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
268
custom APT suite.
269

anonym's avatar
anonym committed
270
271
Update other base branches
==========================
272
273

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
274
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
275

276
2. Merge `devel` into `feature/stretch` following the instructions for
intrigeri's avatar
intrigeri committed
277
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
278
279
280
   Given that these two branches' APT suites have diverged a lot, and
   that `tails-merge-suite` currently happily overwrites newer
   packages in the target with older packages from the source, it's
anonym's avatar
anonym committed
281
   probably easier to just merge each individual APT overlay that was
282
   just merged into the release branch into `feature/stretch`'s APT
283
   suite. Also, most of our just upgraded bundled packages
284
   (e.g. `tails-greeter`) may need to be rebuilt for Stretch.
285

286
3. Ensure that the release, `devel` and `feature/stretch` branches
287
288
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
289

290
4. Push the modified branches to Git:
291

292
        git push origin                          \
293
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
294
295
           feature/stretch:feature/stretch       \
           devel:devel
296
297
298

Update more included files
==========================
299

300
301
Changelog
---------
Tails developers's avatar
Tails developers committed
302

303
304
305
Remove the placeholder entry for next release in `debian/changelog`,
and then:

306
307
	git checkout "${RELEASE_BRANCH:?}" && \
	./release ${VERSION:?} ${PREVIOUS_VERSION:?}
Tails developers's avatar
Tails developers committed
308
309
310

This populates the Changelog with the Git log entries.

311
312
313
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
314

315
316
317
318
Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
319
320
* the diff between the previous version's `.packages` file and the one
  from the to-be-released ISO;
321
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
322
  in Redmine.
323

324
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
325

326
327
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
328
	else
329
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
330
	fi
Tails developers's avatar
Tails developers committed
331

Bessemer's avatar
Bessemer committed
332
Included website
333
334
----------------

Tails developers's avatar
Tails developers committed
335
336
337
338
339
340
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

341
342
### version number

343
344
If preparing a RC, skip this part.

345
In the branch used to build the release, update the `wiki/src/inc/*` files to
346
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
347
at least 24 hours in the future! Between tests and mirror synchronization,
348
the build will not be released on the same day. Try to make sure it
349
matches the date of the future signature.
350

sajolida's avatar
sajolida committed
351
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
352

353
354
355
356
	echo "${VERSION:?}"      > wiki/src/inc/stable_i386_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_i386_date.html
	sed -ri "s%news/version_.*]]%news/version_${VERSION:?}]]%" wiki/src/inc/stable_i386_release_notes.*
	${EDITOR:?} wiki/src/inc/*.html
elouann's avatar
elouann committed
357
	./build-website
358
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
359

360
361
### features and design documentation

362
363
Read the Changelog carefully, and update [[doc/about/features]]
pages accordingly.
364

365
366
Website translations
--------------------
367

368
369
370
371
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
372

elouann's avatar
elouann committed
373
	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
374
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
375

376
377
378
Call for translation
====================

379
380
381
382
383
384
If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].
385

386
To get a list of changes on the website:
387

388
    git diff --stat ${PREVIOUS_VERSION:?}.. -- \
anonym's avatar
anonym committed
389
390
391
392
393
394
        *.{mdwn,html} \
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
395

Tails developers's avatar
Tails developers committed
396
397
398
Import the signing key
======================

399
400
401
402
403
404
Skip this part if you have a Tails signing subkey on an OpenPGPG
smartcard, i.e. if you are one of the usual release managers. This is
only relevant when the master key has been reassembled, e.g. for
signing a Tails emergency release where none of the usual release
managers are available.

Tails developers's avatar
Tails developers committed
405
You should never import the Tails signing key into your own keyring,
406
407
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
408
409

    export GNUPGHOME=$(mktemp -d)
410
411
412
413
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
414
415
416
417
418
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

419
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
420

421
422
423
424
425
426
427
428
429
430
Build the almost-final image
============================

1. [[Build an ISO image|contribute/build]] from the release branch.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep at least the resulting ISO image and the manifest of needed
   packages until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
431
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"
432
433


434
435
436
Tag the release in Git
======================

437
438
439
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
	git push --tags origin "${RELEASE_BRANCH:?}"
440
441
442
443
444
445
446

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

447
448
449
450
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

451
452
453
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
454
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
455

456
* Prepare tagged snapshots of upstream APT repositories:
457

458
          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"
459

460
  Note:
461

462
463
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
464
    [[custom APT repository|APT_repository/custom]] are
465
466
    listed under "some packages were not found anywhere" (because we
    are current not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
467
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
468
    Now, we have a "safety" net, in case you don't notice such a problem: if
469
470
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
471

Tails developers's avatar
Tails developers committed
472
473
474
Build images
============

475
476
477
478
479
480
481
482
483
484
485
486
487
488
Sanity check
------------

Verify that the TBB release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag for the targetted TBB
and Tor Browser versions in their respective Git repos:

* <https://gitweb.torproject.org/builders/tor-browser-bundle.git>
* <https://gitweb.torproject.org/tor-browser.git>

A new tag may indicate that a new TBB release is imminent.

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
489
490
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
491

492
1. Burn the almost final ISO image to a DVD.
493
494
1. Boot this DVD **on bare metal**.
1. Add `profile` to the kernel command-line.
495
496
497
1. Login.
1. Wait for the "Tor is ready" notification.
1. Start the web browser.
Tails developers's avatar
Tails developers committed
498
1. A few minutes later, once the `boot-profile` process has been
499
   killed, retrieve the new sort file from `/var/log/boot-profile`.
500
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
501
502
503
504
505
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
506
1. Inspect the Git diff (including diff stat), apply common sense.
507
508
509
510
511
512
513
514
   The following command is also helpful but requires that you save a
   copy of the old sort file into `/tmp/squashfs.sort.old`:

       diff -NaurB \
           <( cut -d' ' -f1 /tmp/squashfs.sort.old | sort ) \
           <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort | sort ) \
           | less

515
516
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

517
518
519
Build the final image
---------------------

Bessemer's avatar
Bessemer committed
520
Then all included files should be up-to-date and the versioned APT
521
522
suite should be ready, so it is time to:

523
524
525
* Mark the version as "released" in the changelog:

      dch --release --no-force-save-on-release --maintmaint
526
      git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
527

528
529
* tag the release *again*, with all included files in:
  
530
531
532
      git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
              -m "tagging version ${VERSION:?}" "${TAG:?}" && \
      git push origin "${RELEASE_BRANCH:?}" && \
533
      git push --tags --force
anonym's avatar
anonym committed
534
535
536

* check out the release tag:

537
      git checkout "${TAG:?}"
anonym's avatar
anonym committed
538

539
* build the final image!
540

intrigeri's avatar
intrigeri committed
541
* compare the new build manifest with the one from the previous,
anonym's avatar
anonym committed
542
543
  almost final build; they should be identical (XXX: except for
  the `debian-security` serial, right?).
intrigeri's avatar
intrigeri committed
544

anonym's avatar
anonym committed
545
546
* check out the release branch again:

547
      git checkout "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
548

549
550
551
552
553
Generate the OpenPGP signatures and Torrents
============================================

First, create a directory with a suitable name and go there:

554
555
	mkdir "${ISOS:?}/tails-i386-${VERSION:?}" && \
	cd "${ISOS:?}/tails-i386-${VERSION:?}"
556

anonym's avatar
anonym committed
557
Second, move the built image to this brand new directory:
558

559
560
	mv "${ARTIFACTS:?}/tails-i386-${VERSION:?}.iso" \
	   "${ISOS:?}/tails-i386-${VERSION:?}/"
561
562
563
564
565

Third, generate detached OpenPGP signatures for the image to be
published, in the same directory as the image and with a `.sig`
extension; e.g.

566
	gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *.iso
567
568
569
570
571
572
	rename 's,\.asc$,.sig,' *.asc

Fourth, go up to the parent directory, create a `.torrent` file and
check the generated `.torrent` files metainfo:

	cd .. && \
intrigeri's avatar
intrigeri committed
573
	mktorrent \
intrigeri's avatar
intrigeri committed
574
	  -a 'udp://tracker.torrent.eu.org:451' \
intrigeri's avatar
intrigeri committed
575
	  -a 'udp://tracker.coppersurfer.tk:6969'   \
576
577
	  "tails-i386-${VERSION:?}" && \
	transmission-show tails-i386-${VERSION:?}.torrent
578

anonym's avatar
anonym committed
579
580
Lastly, let's set some variables to be used later:

581
582
583
    ISO_PATH="${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso"
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
anonym's avatar
anonym committed
584

585
586
<a id="prepare-iuk"></a>

587
588
589
Prepare incremental upgrades
============================

590
591
Build the Incremental Upgrade Kits
----------------------------------
592

593
594
595
596
597
598
599
600
601
602
603
Use `tails-create-iuk` to build the following IUKs:

* From the previous stable release, e.g. 1.0 to 1.0.1 or 1.0 to
  1.0.1~rc1. This may be skipped if the delta is too big (like when
  migrating to a new Debian release) or if there are changes outside
  of the scope for IUKs (like partition table changes).

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
604
  next version.
605

606
Example (for RC, replace `${PREVIOUS_VERSION:?}` with e.g. `${VERSION:?}~rc1`
607
below):
608

609
    sudo su -c "cd ${IUK_CHECKOUT:?} && \
anonym's avatar
anonym committed
610
      PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
611
        ./bin/tails-create-iuk \
612
613
614
615
           --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
           --old-iso \"${ISOS:?}/tails-i386-${PREVIOUS_VERSION:?}/tails-i386-${PREVIOUS_VERSION:?}.iso\" \
           --new-iso \"${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso\"          \
           --outfile \"${ISOS:?}/Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk\""
616
617

Note that developer tools for creating IUK and upgrade-description
Tails developers's avatar
Tails developers committed
618
files were only tested on Debian sid. It should hopefully work well on
619
Jessie too.
620

Tails developers's avatar
Tails developers committed
621
<a id="prepare-upgrade-description-files"></a>
622

Tails developers's avatar
Tails developers committed
623
624
Prepare upgrade-description files
---------------------------------
625

Tails developers's avatar
Tails developers committed
626
1. Prepare upgrade-description files (see the [[upgrade-description
627
   files
Tails developers's avatar
Tails developers committed
628
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
629
630
631
632
633
634
635
636
637
638
639
640
   for details). The idea is to:

   * update (create if needed) an upgrade-description file for every
     *previous* supported release (e.g. N~rc1, N-1, N-1~rc2) that replaces
     all existing upgrade paths with the path to the version being
     released;
   * create a new upgrade-description for the version being released,
     that expresses that *no* upgrade is available for that one yet.

   This is what `tails-iuk-generate-ugrade-description-files` tool
   does:

641
       ( cd ${IUK_CHECKOUT:?} && \
642
       ./bin/tails-iuk-generate-upgrade-description-files \
643
644
645
646
647
648
649
650
651
652
           --version "${VERSION:?}" \
           --next-version "${NEXT_PLANNED_VERSION:?}" \
           --next-version "${NEXT_PLANNED_VERSION:?}~rc1" \
           --next-version "${VERSION:?}.1" \
           --iso "${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso" \
           --previous-version "${PREVIOUS_VERSION:?}" \
           --previous-version "${VERSION:?}~rc1" \
           --iuks "${ISOS:?}" \
           --release-checkout "${RELEASE_CHECKOUT:?}" \
           --major-release "${MAJOR_RELEASE:?}" \
653
       )
654
655
656
657
658
659
660

   Note:
   * The `--iuks` argument must point to the directory where the IUKs
     generated at the previous step are stored.
   * At least the last stable release and the previous release
     candidates for the version being released must be passed to
     `--previous-version`.
661
662
663
664
665
666
667
668
669
   * Older versions for which there is no incremental upgrade path to
     the new release must be passed with `--previous-version`, so that
     users who skipped a release or two are informed of the new one.
     Note that multi-steps incremental upgrade paths are valid and
     supported: e.g. when releasing 1.1.2, 1.1 users should still be
     able to incrementally upgrade to 1.1.1, and in turn to 1.1.2; to
     make this work, one must _not_ pass `--previous-version 1.1`,
     that would remove the existing incremental upgrade path from 1.1
     to 1.1.1.
670
   * If preparing a release candidate, add `--channel alpha`
671
672
   * If preparing a release candidate, drop all `--next-version`
     arguments, and instead pass (**untested!**)
673
     `--next-version $(echo ${VERSION:?} | sed -e 's,~rc*$,,')`
674
   * If preparing a point-release, pass neither
675
676
     `--next-version "${VERSION:?}.1"`,
     nor `--next-version "${VERSION:?}.1~rc1"`
677
678
679
680

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

681
       find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
anonym's avatar
anonym committed
682
683
          -type f -name upgrades.yml | \
          while read udf; do
684
685
686
              if [ -n "$(git status --porcelain "${udf:?}")" ]; then
                  gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign "${udf:?}"
                  mv "${udf:?}.asc" "${udf:?}.pgp"
anonym's avatar
anonym committed
687
                  ( \
688
689
                    cd ${IUK_CHECKOUT:?} && \
                    ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
anonym's avatar
anonym committed
690
                  ) || break
anonym's avatar
anonym committed
691
692
              fi
          done
693

Tails developers's avatar
Tails developers committed
694
695
1. Add and commit the upgrade-description files and their detached
   signatures to the Git branch used to prepare the release (`stable`
696
   or `testing`):
Tails developers's avatar
Tails developers committed
697

anonym's avatar
anonym committed
698
699
700
701
702
       ( \
         cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
         git commit -m "Update upgrade-description files." && \
         git push origin ${RELEASE_BRANCH:?} \
       )
Tails developers's avatar
Tails developers committed
703

704
1. If preparing a release candidate, move the generated or updated
705
   files to `${MASTER_CHECKOUT:?}`, commit and push: given the updates are
706
707
708
   advertised on the *alpha* channel, while all users use the *stable*
   one by default, this will allow you to more easily test the IUK
   without impacting anyone.
Tails developers's avatar
Tails developers committed
709

710
711
712
713
1. If preparing a point release, copy the generated UDF for the previous
   release in the alpha channel in the master branch, modify its content
   accordingly, sign it, commit and push

anonym's avatar
anonym committed
714
715
716
717
       # If more old versions are supported, add them (whitespace
       # separated) to this variable
       SUPPORTED_OLD_VERSIONS="${PREVIOUS_VERSION:?}"

anonym's avatar
anonym committed
718
719
720
       ( \
         cd ${MASTER_CHECKOUT:?} && \
         git fetch && \
anonym's avatar
anonym committed
721
722
723
724
725
726
727
728
729
         for old_version in ${SUPPORTED_OLD_VERSIONS:?}; do
           stable_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/i386/stable/upgrades.yml" && \
           alpha_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/i386/alpha/upgrades.yml" && \
           git show origin/${RELEASE_BRANCH:?}:${stable_udf:?} \
             | sed -e 's/channel: stable/channel: alpha/' > ${alpha_udf:?} && \
           gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${alpha_udf:?} && \
           mv ${alpha_udf:?}.asc ${alpha_udf:?}.pgp && \
           git add ${alpha_udf:?}* ; \
         done && \
anonym's avatar
anonym committed
730
         git commit -m "Add incremental upgrades on the alpha channel for Tails ${VERSION:?}" && \
anonym's avatar
anonym committed
731
732
         git push origin master:master \
       )
733
734


735
736
737
Prepare the ISO description file for DAVE
-----------------------------------------

anonym's avatar
anonym committed
738
739
If preparing a RC, skip this part.

740
741
Update the ISO description file (IDF) used by the browser extension:

742
    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/i386/stable/latest.yml <<EOF
743
744
745
746
    ---
    build-target: i386
    channel: stable
    product-name: Tails
747
    version: '${VERSION:?}'
748
749
    target-files:
    - sha256: ${ISO_SHA256SUM}
750
751
      size: ${ISO_SIZE_IN_BYTES:?}
      url: http://dl.amnesia.boum.org/tails/stable/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso
752
    EOF
753
    ( cd "${RELEASE_CHECKOUT:?}" && \
754
755
756
      git add wiki/src/install/v1/Tails/i386/stable/latest.yml && \
      git commit -m "Update IDF file for DAVE." )

Tails developers's avatar
Tails developers committed
757
758
759
Upload images
=============

Tails developers's avatar
Tails developers committed
760
761
762
Sanity check
------------

763
764
Verify once more that the TBB we ship is still the most recent (see
above).
Tails developers's avatar
Tails developers committed
765

766
767
## Announce, seed and test the Torrents

Tails developers's avatar
Tails developers committed
768
Announce and seed the Torrents.
769
770
771

Test them with a BitTorrent client running in a different place.

772
773
## Download and seed image from lizard

774
    scp "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
775
       bittorrent.lizard: && \
776
       ssh bittorrent.lizard \
777
         transmission-remote --add tails-i386-${VERSION:?}.torrent \
778
           --find /var/lib/transmission-daemon/downloads/
779

780
781
782
783
<a id="publish-iuk"></a>

Publish the ISO and IUK over HTTP
---------------------------------
Tails developers's avatar
Tails developers committed
784

785
786
787
788
789
790
Upload the images to the primary rsync mirror. Best practice is to first
let bittorrent.lizard download the image, and then copy it from there to
rsync.lizard:

    ssh lizard.tails.boum.org \
        scp -3 -r \
791
            bittorrent.lizard:/var/lib/transmission-daemon/downloads/tails-i386-${VERSION:?} \
792
793
            rsync.lizard:
    ssh rsync.lizard << EOF
794
      sudo chown -R root:rsync_tails \
795
796
         tails-i386-${VERSION:?} \
         Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk && \
797
      sudo chmod -R u=rwX,go=rX \
798
799
800
801
802
803
         tails-i386-${VERSION:?} \
         Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk && \
      sudo mv tails-i386-${VERSION:?} \
              /srv/rsync/tails/tails/${DIST:?}/ && \
      sudo mv Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk \
              /srv/rsync/tails/tails/${DIST:?}/iuk/
804
    EOF
805
806
807
808
809

Update the time in `project/trace` file on the primary rsync mirror
and on the live wiki (even for a release candidate):

	TRACE_TIME=$(date +%s) &&
810
811
812
	echo ${TRACE_TIME:?} | ssh rsync.lizard "cat > /srv/rsync/tails/tails/project/trace" && \
	[ -n "${MASTER_CHECKOUT:?}" ] && \
	echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
813
	(
814
	   cd "${MASTER_CHECKOUT:?}" && \
815
	   git commit wiki/src/inc/trace \
816
	      -m "Updating trace file after uploading ${VERSION:?}." && \
817
	   git push origin master
818
819
	)

820
821
822
823
ISO history
-----------

Push the released ISO to our Tails ISO history git-annex repo, so that
824
825
our isotesters can fetch it from there for their testing. How to do so
is described in our internal Git repo.
826

Tails developers's avatar
Tails developers committed
827
828
829
Update the website and Git repository
=====================================

830
What follows in this section happens on the release branch in
831
`${RELEASE_CHECKOUT:?}`.
832
833
834
835
836
837
838
839
840

If preparing a final release
----------------------------

Skip this part if preparing a RC.

Rename the `.packages` file to remove the `.iso` and build date parts
of its name:

841
842
	mv "${ARTIFACTS:?}"/tails-i386-"${VERSION:?}".iso.packages \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.packages"
843

844
845
Rename the manifest of needed packages as well:

846
	mv "${PACKAGES_MANIFEST:?}" "${ARTIFACTS:?}/tails-i386-${VERSION:?}.build-manifest"
847
848
849

Copy the `.iso.sig`, `.build-manifest`, `.packages`, `.torrent` and
`.torrent.sig` files into the website repository:
Tails developers's avatar
Tails developers committed
850

851
852
853
854
855
	cp "${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso.sig" \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.build-manifest" \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.packages" \
	   "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
	   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/"
856

857
858
859
Remove from `wiki/src/torrents/files/` any remaining file from the
previous release (including any RC).

860
861
Update the size of the ISO image in `inc/*`:

862
        LC_NUMERIC=C ls -l -h ${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso | \
863
          cut -f 5 -d ' ' | sed -r 's/(.+)([MG])/\1 \2iB/' \
864
          > "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_i386_iso_size.html"
865

866
867
Generate the expected signature verification output:

868
      gpg --keyid-format 0xlong --verify "${ISO_PATH:?}.sig" "${ISO_PATH:?}" 2>&1 | \
869
      sed 's/ /\&nbsp;/g;s/</\&lt;/;s/>/\&gt;/;s/$/<br\/>/g' > \
870
      "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_i386_gpg_signature_output.html"
871

872
873
874
875
Update the [[support/known_issues]] page:

- Add regressions brought by the new release.
- Remove older known issues that are fixed by the new release.
876

877
Write the announcement for the release in
878
`wiki/src/news/version_${TAG:?}.mdwn`. See the [[release notes
879
documentation|contribute/how/documentation/release_notes]]
880

881
882
883
884
885
886
XXX: we should probably merge that into the above liked documentation.
**Check you correctly:**

- Updated the `meta title` directive.
- Updated the `meta date` directive.
- Made sure there's an `announce` tag to have an email sent to the
sajolida's avatar
sajolida committed
887
  news mailing list.
888
- Documented important config changes that persistence users have to do
Tails developers's avatar
Tails developers committed
889
  themselves (e.g. the Pidgin proxy settings change in
890
  [[!tails_gitweb_commit 9925321]] breaks all existing persistent
Tails developers's avatar
Tails developers committed
891
  profiles).
892
- Documented known issues.
Tails developers's avatar
Tails developers committed
893

Tails developers's avatar
Tails developers committed
894
Write an announcement listing the security bugs affecting the previous
895
version in
896
`wiki/src/security/Numerous_security_holes_in_${PREVIOUS_VERSION:?}.mdwn`
897
in order to let the users of the old versions
898
know that they have to upgrade. Date it a few days before the ISO
899
900
image to be released was *built*. Including:

901
902
- if we are shipping Linux from testing/sid, the list of CVE fixed in
  Linux since the one shipped in the previous release of Tails:
intrigeri's avatar
intrigeri committed
903
  <http://metadata.ftp-master.debian.org/changelogs/main/l/linux/stable_changelog>
904
- the list of DSA fixed in packages we ship since those that were in
intrigeri's avatar
intrigeri committed
905
  the previous release of Tails: <https://www.debian.org/security/#DSAS>
Tails developers's avatar
Tails developers committed
906
907
908
- the list of BSA fixed in packages we ship since those that were in
  the previous release of Tails:
  <https://lists.debian.org/debian-backports-announce/>
909
- the list of MFSA fixed by the Tor Browser update:
910
  <https://www.mozilla.org/security/announce/>
Tails developers's avatar
Tails developers committed
911

912
913
914
915
916
917
918
If preparing a release candidate
--------------------------------

Skip this part if preparing a final release.

Copy the `.iso.sig` file into the website repository:

919
920
921
	cp "${ISO_PATH:?}.sig" \
	   "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
	      "${MASTER_CHECKOUT:?}/wiki/src/torrents/files/"
922
923

Write the announcement for the release in
924
`${MASTER_CHECKOUT:?}/wiki/src/news/test_${TAG:?}.mdwn`, including:
925
926
927
928
929
930
931
932

- Update the `meta title` directive.
- Update the `meta date` directive.
- Document important config changes that persistence users have to do
  themselves (e.g. the Pidgin proxy settings change in
  [[!tails_gitweb_commit 9925321]] breaks all existing persistent
  profiles).
- Document known issues.
anonym's avatar
anonym committed
933
934
935
936
- This snippet can help to convert the copied changelog's ticket
  references to links:

      sed -i 's@#\([0-9]\{4,5\}\)@[[!tails_ticket \1]]@g' \
937
          wiki/src/news/test_${TAG:?}.mdwn
938
939
940
941

In any case
-----------

elouann's avatar
elouann committed
942
Generate PO files for the announcements with `./build-website`.
943

944
945
946
947
Then, send them to <tails-l10n@boum.org> so that they get translated
shortly, perhaps even soon enough to integrate them before pushing the
release out officially.

948
Then, record the last commit before putting the release out for real:
Tails developers's avatar
Tails developers committed
949

950
	git add wiki/src && \
951
	git commit -m "releasing version ${VERSION:?}"
Tails developers's avatar
Tails developers committed
952

953
954
955
Testing
=======

956
957
958
959
960
961
1. Using `check-mirrors`, choose a fast mirror that already has the
   tentative ISO. E.g. <https://mirrors.kernel.org/tails/> or
   <https://mirrors.wikimedia.org/tails/> are reliable and have plenty
   of bandwidth.
1. Email <tails-testers@boum.org> to ask them to test the tentative
   ISO, pointing them to the up-to-date mirror you've found previously.
962
1. Set up a pad and copy the [[manual test
963
   suite|contribute/release_process/test]] in it.
intrigeri's avatar
intrigeri committed
964
1. Email <tails@boum.org> and potential contributors (see
965
966
   `manual_testers.mdwn` in the internal Git repository) that tests
   may start:
967
   - point them to the up-to-date mirror you've found previously
968
969
970
971
972
973
974
975
   - make it clear what's the deadline
   - make it clear where and how you expect to get feedback
   - attach the Torrent
   - attach the `.packages` file
1. Make sure someone is committed to run the automated test suite.
1. Make sure that enough people are here to run the tests, that they
   report their results in due time, and that they make it clear when
   they're leaving for good.
intrigeri's avatar
intrigeri committed
976
977
978
979
980
1. Fill the holes and make sure that the manual test suite is done in
   due time.
1. Triage test results, reproduce bugs as needed, decide what the next
   step is and make sure it happens: add to known issues? file ticket?
   release blocker?
981

Tails developers's avatar
Tails developers committed
982
983
984
Go wild!
========

985
986
987
988
989
Wait for the HTTP mirrors to catch up
-------------------------------------

Test downloading the ISO and IUK over HTTP.

990
Make sure every active mirror in the pool has the new version:
991

992
	./check-mirrors.rb --allow-multiple --fast tails-i386-${VERSION:?}
993

994
995
996
Ask <tails-mirrors@boum.org> to drop those that are lagging behind and
notify their administrators.

997
998
999
Sanity check
------------

1000
Verify once more that the TBB we ship is still the most recent (see
For faster browsing, not all history is shown. View entire blame