release_process.mdwn 44.4 KB
Newer Older
Tails developers's avatar
Tails developers committed
1
2
[[!meta title="Release process"]]

Tails developers's avatar
Tails developers committed
3
[[!toc levels=2]]
Tails developers's avatar
Tails developers committed
4

Tails developers's avatar
Tails developers committed
5
6
See the [[release_schedule]].

7
8
9
10
11
12
13
14
15
16
Requirements
============

To release Tails you'll need some packages installed:

* mktorrent and transmission-cli
* Aufs DKMS module for your running kernel.
* tails-iuk dependencies (see debian/control in the `debian` branch of its repo)
* tails-perl5lib dependencies (same trick than tails-iuk to get the list)

17
18
19
20
21
22
Environment
===========

Export the following environment variables to be able to copy'n'paste
the scripts snippets found on this page:

23
* version numbers (see [[contribute/release_schedule#versioning]]):
24
25

      export VERSION=$(dpkg-parsechangelog -SVersion)
26
      export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
27
28
      export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)

29
* `NEXT_PLANNED_VERSION`: set to the version number of the next Tails release
30
  (e.g. 0.23 when releasing 0.22.1, and 1.3 when releasing 1.2)
31
* `MAJOR_RELEASE`: set to 1 if preparing a major release, to 0 else
32
33
* `ISOS`: the directory where one stores `tails-i386-*`
  sub-directories like the ones downloaded with BitTorrent.
34
35
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
36
37
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
38
39
* `RELEASE_BRANCH`: the name of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
40
41
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
42
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
43
44
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
45
46
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
47
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
intrigeri's avatar
intrigeri committed
48
49
* `export DEBEMAIL='tails@boum.org'`
* `export DEBFULLNAME='Tails developers'`
50

51
52
53
Pre-freeze
==========

54
55
The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.
56

57
58
59
60
61
62
Update Icedove preferences
------------------------------

* update `extensions.enigmail.configuredVersion` in
  `config/chroot_local-includes/etc/skel/.icedove/profile.default/preferences/0000tails.js`

63
64
65
66
67
Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

68
69
70
Select the right branch
=======================

71
72
What we refer to as the "release branch" (and `RELEASE_BRANCH`) should
be `testing` for major releases, and `stable` for point-releases.
73

74
<div class="caution">
75
Read the remainder of this document from the branch used to prepare the release!
76
77
</div>

78
79
80
81
82
83
84
85
86
87
88
89
Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
90
91
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
92
93
94
95
96
97
98
99
100
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

101
102
4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.
103

anonym's avatar
anonym committed
104
5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
105
106
107
108
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
109
110
111
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
112
113
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]

114
115
116
117
118
119
120
121
122
123

Point-release
-------------

If we are at freeze time for a point-release:

1. Merge the `master` Git branch into `stable`:

        git checkout stable && git merge --no-ff origin/master

intrigeri's avatar
intrigeri committed
124
125
2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
126
127
128
129
130
   APT suite.

Common steps for point and major releases
-----------------------------------------

131
Reset the release branch's `config/base_branch`:
132

133
        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
134
           git commit config/base_branch \
135
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."
136

Tails developers's avatar
Tails developers committed
137
138
139
Update included files
=====================

spriver's avatar
spriver committed
140
uBlock patterns and settings file
Tails developers's avatar
Tails developers committed
141
142
----------------

spriver's avatar
spriver committed
143
144
The patterns+settings file is stored as a converted sqlite text dump in
`config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`.
Tails developers's avatar
Tails developers committed
145

146
1. Boot Tails
spriver's avatar
spriver committed
147
148
149
150
2. Start the Tor Browser and open the uBlock dashboard by clicking on the uBlock icon.
3. Select the tab *3rd-party filters*
4. Click on the button *Update now* to update all used patterns
5. Close the Tor Browser
151
152
153
154
7. Copy the `.tor-browser/profile.default/extension-data/ublock0.sqlite`
   from this Tor Browser instance into the root of Tails' Git repo and
   run the following command:

155
156
157
158
159
160
       # For posterity: the general idea is to introduce \r\n as a
       # token where we have made a line break to make the dump more
       # diff-friendly (and, hence, Git-friendly). The most complex
       # expression is the one done with perl, where we employ
       # negative lookahead. What it means, is: replace single
       # occurrences of | except when followed by \\n.
161
162
163
       echo '.dump' | sqlite3 ublock0.sqlite | \
           grep -v "cached_asset_content://cache://compiled-" | \
           awk '!/^INSERT/; /^INSERT/ {print $0 | "sort -n"}' | \
164
           sed 's_\\n_\\n\r\n_g' | \
165
           sed 's_,_,\r\n_g' | \
anonym's avatar
anonym committed
166
           perl -pi -e 's/([^|])\|((?!\||\\n).)/\1\|\r\n\2/g' | \
167
           sed "/^INSERT INTO \"settings\" VALUES('\(remoteBlacklists\|cached_asset_entries\)'/"'s_,_,\r\n_g' > \
anonym's avatar
anonym committed
168
           config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump
169

170
8. Commit:
Tails developers's avatar
Tails developers committed
171

172
       git commit -m 'Update uBlock Origin patterns + settings file.' \
spriver's avatar
spriver committed
173
174
          config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump

175
9. Remove the original `ublock0.sqlite` from the Git root.
176

Bessemer's avatar
Bessemer committed
177
Upgrade bundled binary Debian packages
Tails developers's avatar
Tails developers committed
178
179
--------------------------------------

180
181
182
Skip this section unless we are at freeze time for a major release
(i.e. we are about to prepare a release candidate).

183
That is: make sure the bundled binary Debian packages contain
184
185
up-to-date localization files.

186
187
188
189
190
191
For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
and then run the `import-translations` script that is in the
main Tails repository. For example:

	cd whisperback
192
	"${RELEASE_CHECKOUT:?}"/import-translations
193

WinterFairy's avatar
WinterFairy committed
194
195
If the `import-translations` script fails to import translations for
the current package, manually copy updated PO files from the
196
Transifex branches of `git://git.torproject.org/translation.git` (e.g.
197
198
`whisperback_completed`) instead. In this case, skip PO files for
[[translation teams that use Git|contribute/how/translate#translate]].
199

200
Add and commit.
201

202
203
Then check the PO files:

204
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po
205
206

Correct any displayed error, then commit the changes if any.
207

208
Then see the relevant release processes, and upload the packages to
209
the release branch's custom APT suite:
Tails developers's avatar
Tails developers committed
210

211
* [[tails-installer]]
212
* [[tails-greeter]]
213
* [[perl5lib]]
214
* [[persistence-setup]]
215
* [[tails-iuk]]
216
* whisperback:
217
218
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package
Tails developers's avatar
Tails developers committed
219

intrigeri's avatar
intrigeri committed
220
221
Upgrade Tor Browser
-------------------
222
223
224

See the dedicated page: [[tor-browser]]

225
226
227
228
229
Upgrade Icedove
---------------

See the dedicated page: [[icedove]]

230
231
Update PO files
---------------
232

233
234
235
Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:
236

237
	cd "${RELEASE_CHECKOUT:?}" && \
238
239
	./import-translations  && \
	./refresh-translations && \
240
	./submodules/jenkins-tools/slaves/check_po && \
241
	git add po && git commit -m 'Update PO files.'
242

243
244
245
246
247
248
If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

249
250
When preparing an actual release
================================
251

252
253
254
255
256
257
If we're about to prepare an image for a final (non-RC) release, then
follow these instructions:

Major release
-------------

intrigeri's avatar
intrigeri committed
258
259
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
260
custom APT suite.
261
262
263
264

Point-release
-------------

265
266
267
268
269
270
<div class="note">
For point-releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>
271

intrigeri's avatar
intrigeri committed
272
273
[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
274
custom APT suite.
275

anonym's avatar
anonym committed
276
277
Update other base branches
==========================
278
279

1. Merge the release branch into `devel` following the instructions for
intrigeri's avatar
intrigeri committed
280
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
281

282
2. Merge `devel` into `feature/stretch` following the instructions for
intrigeri's avatar
intrigeri committed
283
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
284
285
286
   Given that these two branches' APT suites have diverged a lot, and
   that `tails-merge-suite` currently happily overwrites newer
   packages in the target with older packages from the source, it's
anonym's avatar
anonym committed
287
   probably easier to just merge each individual APT overlay that was
288
   just merged into the release branch into `feature/stretch`'s APT
289
   suite. Also, most of our just upgraded bundled packages
290
   (e.g. `tails-greeter`) may need to be rebuilt for Stretch.
291

292
3. Ensure that the release, `devel` and `feature/stretch` branches
293
294
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.
295

296
4. Push the modified branches to Git:
297

298
        git push origin                          \
299
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
300
301
           feature/stretch:feature/stretch       \
           devel:devel
302
303
304

Update more included files
==========================
305

306
307
Changelog
---------
Tails developers's avatar
Tails developers committed
308

309
310
311
Remove the placeholder entry for next release in `debian/changelog`,
and then:

312
313
	git checkout "${RELEASE_BRANCH:?}" && \
	./release ${VERSION:?} ${PREVIOUS_VERSION:?}
Tails developers's avatar
Tails developers committed
314
315
316

This populates the Changelog with the Git log entries.

317
318
319
Then, launch an editor for the needed cleanup of the result:

	dch -e
Tails developers's avatar
Tails developers committed
320

321
322
323
324
Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
325
326
* the diff between the previous version's `.packages` file and the one
  from the to-be-released ISO;
327
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
328
  in Redmine.
329

330
Finally, sanity check the version and commit:
Tails developers's avatar
Tails developers committed
331

332
333
	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
334
	else
335
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
336
	fi
Tails developers's avatar
Tails developers committed
337

Bessemer's avatar
Bessemer committed
338
Included website
339
340
----------------

Tails developers's avatar
Tails developers committed
341
342
343
344
345
346
### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

347
348
### version number

349
350
If preparing a RC, skip this part.

351
In the branch used to build the release, update the `wiki/src/inc/*` files to
352
match the *version number* and *date* of the new release. Set the date
Tails developers's avatar
Tails developers committed
353
at least 24 hours in the future! Between tests and mirror synchronization,
354
the build will not be released on the same day. Try to make sure it
355
matches the date of the future signature.
356

sajolida's avatar
sajolida committed
357
	RELEASE_DATE='2015-11-03'
Tails developers's avatar
Tails developers committed
358

359
360
361
362
	echo "${VERSION:?}"      > wiki/src/inc/stable_i386_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_i386_date.html
	sed -ri "s%news/version_.*]]%news/version_${VERSION:?}]]%" wiki/src/inc/stable_i386_release_notes.*
	${EDITOR:?} wiki/src/inc/*.html
elouann's avatar
elouann committed
363
	./build-website
364
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."
Tails developers's avatar
Tails developers committed
365

366
367
### features and design documentation

368
369
Read the Changelog carefully, and update [[doc/about/features]]
pages accordingly.
370

371
372
Website translations
--------------------
373

374
375
376
377
Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:
378

elouann's avatar
elouann committed
379
	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
380
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"
381

382
383
384
Call for translation
====================

385
386
387
388
389
390
If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].
391

392
To get a list of changes on the website:
393

394
    git diff --stat ${PREVIOUS_VERSION:?}.. -- \
anonym's avatar
anonym committed
395
396
397
398
399
400
        *.{mdwn,html} \
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'
401

Tails developers's avatar
Tails developers committed
402
403
404
Import the signing key
======================

405
406
407
408
409
410
Skip this part if you have a Tails signing subkey on an OpenPGPG
smartcard, i.e. if you are one of the usual release managers. This is
only relevant when the master key has been reassembled, e.g. for
signing a Tails emergency release where none of the usual release
managers are available.

Tails developers's avatar
Tails developers committed
411
You should never import the Tails signing key into your own keyring,
412
413
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:
Tails developers's avatar
Tails developers committed
414
415

    export GNUPGHOME=$(mktemp -d)
416
417
418
419
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
Tails developers's avatar
Tails developers committed
420
421
422
423
424
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

425
    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"
Tails developers's avatar
Tails developers committed
426

427
428
429
430
431
432
433
434
435
436
Build the almost-final image
============================

1. [[Build an ISO image|contribute/build]] from the release branch.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep at least the resulting ISO image and the manifest of needed
   packages until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
437
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"
438
439


440
441
442
Tag the release in Git
======================

443
444
445
	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
	git push --tags origin "${RELEASE_BRANCH:?}"
446
447
448
449
450
451
452

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

453
454
455
456
XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

457
458
459
Prepare the versioned APT suites
================================

intrigeri's avatar
intrigeri committed
460
* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].
461

462
* Prepare tagged snapshots of upstream APT repositories:
463

464
          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"
465

466
  Note:
467

468
469
  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
intrigeri's avatar
intrigeri committed
470
    [[custom APT repository|APT_repository/custom]] are
471
472
    listed under "some packages were not found anywhere" (because we
    are current not using time-based snapshots for our custom APT
intrigeri's avatar
intrigeri committed
473
    repository). However, _no other package should be on that list_.
intrigeri's avatar
intrigeri committed
474
    Now, we have a "safety" net, in case you don't notice such a problem: if
475
476
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.
477

Tails developers's avatar
Tails developers committed
478
479
480
Build images
============

481
482
483
484
485
486
487
488
489
490
491
492
493
494
Sanity check
------------

Verify that the TBB release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag for the targetted TBB
and Tor Browser versions in their respective Git repos:

* <https://gitweb.torproject.org/builders/tor-browser-bundle.git>
* <https://gitweb.torproject.org/tor-browser.git>

A new tag may indicate that a new TBB release is imminent.

Better catch this before people spend time doing manual tests.

Bessemer's avatar
Bessemer committed
495
496
SquashFS file order
-------------------
Tails developers's avatar
Tails developers committed
497

498
1. Burn the almost final ISO image to a DVD.
499
500
1. Boot this DVD **on bare metal**.
1. Add `profile` to the kernel command-line.
501
502
503
1. Login.
1. Wait for the "Tor is ready" notification.
1. Start the web browser.
Tails developers's avatar
Tails developers committed
504
1. A few minutes later, once the `boot-profile` process has been
505
   killed, retrieve the new sort file from `/var/log/boot-profile`.
506
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
507
508
509
510
511
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
512
1. Inspect the Git diff (including diff stat), apply common sense.
513
514
515
516
517
518
519
520
   The following command is also helpful but requires that you save a
   copy of the old sort file into `/tmp/squashfs.sort.old`:

       diff -NaurB \
           <( cut -d' ' -f1 /tmp/squashfs.sort.old | sort ) \
           <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort | sort ) \
           | less

521
522
1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

523
524
525
Build the final image
---------------------

Bessemer's avatar
Bessemer committed
526
Then all included files should be up-to-date and the versioned APT
527
528
suite should be ready, so it is time to:

529
530
531
* Mark the version as "released" in the changelog:

      dch --release --no-force-save-on-release --maintmaint
532
      git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog
533

534
535
* tag the release *again*, with all included files in:
  
536
537
538
      git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
              -m "tagging version ${VERSION:?}" "${TAG:?}" && \
      git push origin "${RELEASE_BRANCH:?}" && \
539
      git push --tags --force
anonym's avatar
anonym committed
540
541
542

* check out the release tag:

543
      git checkout "${TAG:?}"
anonym's avatar
anonym committed
544

545
* build the final image!
546

intrigeri's avatar
intrigeri committed
547
* compare the new build manifest with the one from the previous,
548
549
  almost final build; they should be identical, except that the
  `debian-security` serial/reference might be higher. To ensure we get
intrigeri's avatar
intrigeri committed
550
  the final build's .build-manifest, please run:
551
552

      export PACKAGES_MANIFEST="${ARTIFACTS:?}/tails-i386-${VERSION:?}.iso.build-manifest"
intrigeri's avatar
intrigeri committed
553

anonym's avatar
anonym committed
554
555
* check out the release branch again:

556
      git checkout "${RELEASE_BRANCH:?}"
anonym's avatar
anonym committed
557

558
559
560
561
562
Generate the OpenPGP signatures and Torrents
============================================

First, create a directory with a suitable name and go there:

563
564
	mkdir "${ISOS:?}/tails-i386-${VERSION:?}" && \
	cd "${ISOS:?}/tails-i386-${VERSION:?}"
565

anonym's avatar
anonym committed
566
Second, move the built image to this brand new directory:
567

568
569
	mv "${ARTIFACTS:?}/tails-i386-${VERSION:?}.iso" \
	   "${ISOS:?}/tails-i386-${VERSION:?}/"
570
571
572
573
574

Third, generate detached OpenPGP signatures for the image to be
published, in the same directory as the image and with a `.sig`
extension; e.g.

575
	gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *.iso
576
577
578
579
580
581
	rename 's,\.asc$,.sig,' *.asc

Fourth, go up to the parent directory, create a `.torrent` file and
check the generated `.torrent` files metainfo:

	cd .. && \
intrigeri's avatar
intrigeri committed
582
	mktorrent \
intrigeri's avatar
intrigeri committed
583
	  -a 'udp://tracker.torrent.eu.org:451' \
intrigeri's avatar
intrigeri committed
584
	  -a 'udp://tracker.coppersurfer.tk:6969'   \
585
586
	  "tails-i386-${VERSION:?}" && \
	transmission-show tails-i386-${VERSION:?}.torrent
587

anonym's avatar
anonym committed
588
589
Lastly, let's set some variables to be used later:

590
591
592
    ISO_PATH="${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso"
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
anonym's avatar
anonym committed
593

594
595
<a id="prepare-iuk"></a>

596
597
598
Prepare incremental upgrades
============================

599
600
Build the Incremental Upgrade Kits
----------------------------------
601

602
603
604
605
Incremental upgrades may be skipped if the delta is too big (like when
migrating to a new Debian release) or if there are changes outside of
the scope for IUKs (like partition table changes). Use common sense!

606
607
Use `tails-create-iuk` to build the following IUKs:

608
609
610
611
* From the two previous *planned* releases, and any emergency releases
  in between and after. This should be, more or less, all releases for
  the last 12 weeks (although irregularities in Firefox release
  schedule may add or remove a few weeks).
612
613
614
615
616

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
617
  next version.
618

619
620
621
622
623
624
625
626
627
628
629
630
631
Include each such version in a white-space separated list called
`IUK_SOURCE_VERSIONS`, (e.g. `IUK_SOURCE_VERSIONS="2.8 2.9 2.9.1 2.10~rc1"`)
and run the following:

    for source_version in ${IUK_SOURCE_VERSIONS}; do
        sudo su -c "cd ${IUK_CHECKOUT:?} && \
          PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
            ./bin/tails-create-iuk \
               --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
               --old-iso \"${ISOS:?}/tails-i386-${source_version:?}/tails-i386-${source_version:?}.iso\" \
               --new-iso \"${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso\"          \
               --outfile \"${ISOS:?}/Tails_i386_${source_version:?}_to_${VERSION:?}.iuk\""
    done
632
633

Note that developer tools for creating IUK and upgrade-description
Tails developers's avatar
Tails developers committed
634
files were only tested on Debian sid. It should hopefully work well on
635
Jessie too.
636

Tails developers's avatar
Tails developers committed
637
<a id="prepare-upgrade-description-files"></a>
638

Tails developers's avatar
Tails developers committed
639
640
Prepare upgrade-description files
---------------------------------
641

Tails developers's avatar
Tails developers committed
642
1. Prepare upgrade-description files (see the [[upgrade-description
643
   files
Tails developers's avatar
Tails developers committed
644
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
645
646
647
648
649
650
651
652
653
   for details). The idea is to:

   * update (create if needed) an upgrade-description file for every
     *previous* supported release (e.g. N~rc1, N-1, N-1~rc2) that replaces
     all existing upgrade paths with the path to the version being
     released;
   * create a new upgrade-description for the version being released,
     that expresses that *no* upgrade is available for that one yet.

intrigeri's avatar
intrigeri committed
654
   This is what `tails-iuk-generate-upgrade-description-files` tool
655
656
   does:

657
       ( cd ${IUK_CHECKOUT:?} && \
658
       ./bin/tails-iuk-generate-upgrade-description-files \
659
660
661
662
663
664
665
666
667
668
           --version "${VERSION:?}" \
           --next-version "${NEXT_PLANNED_VERSION:?}" \
           --next-version "${NEXT_PLANNED_VERSION:?}~rc1" \
           --next-version "${VERSION:?}.1" \
           --iso "${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso" \
           --previous-version "${PREVIOUS_VERSION:?}" \
           --previous-version "${VERSION:?}~rc1" \
           --iuks "${ISOS:?}" \
           --release-checkout "${RELEASE_CHECKOUT:?}" \
           --major-release "${MAJOR_RELEASE:?}" \
669
       )
670
671
672
673
674
675
676

   Note:
   * The `--iuks` argument must point to the directory where the IUKs
     generated at the previous step are stored.
   * At least the last stable release and the previous release
     candidates for the version being released must be passed to
     `--previous-version`.
677
678
679
680
681
682
683
684
685
   * Older versions for which there is no incremental upgrade path to
     the new release must be passed with `--previous-version`, so that
     users who skipped a release or two are informed of the new one.
     Note that multi-steps incremental upgrade paths are valid and
     supported: e.g. when releasing 1.1.2, 1.1 users should still be
     able to incrementally upgrade to 1.1.1, and in turn to 1.1.2; to
     make this work, one must _not_ pass `--previous-version 1.1`,
     that would remove the existing incremental upgrade path from 1.1
     to 1.1.1.
686
   * If preparing a release candidate, add `--channel alpha`
687
688
   * If preparing a release candidate, drop all `--next-version`
     arguments, and instead pass (**untested!**)
anonym's avatar
anonym committed
689
     `--next-version $(echo ${VERSION:?} | sed -e 's,~rc.*$,,')`
690
   * If preparing a point-release, pass neither
691
692
     `--next-version "${VERSION:?}.1"`,
     nor `--next-version "${VERSION:?}.1~rc1"`
693
694
695
696

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

697
       find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
anonym's avatar
anonym committed
698
699
          -type f -name upgrades.yml | \
          while read udf; do
700
701
702
              if [ -n "$(git status --porcelain "${udf:?}")" ]; then
                  gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign "${udf:?}"
                  mv "${udf:?}.asc" "${udf:?}.pgp"
anonym's avatar
anonym committed
703
                  ( \
704
705
                    cd ${IUK_CHECKOUT:?} && \
                    ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
anonym's avatar
anonym committed
706
                  ) || break
anonym's avatar
anonym committed
707
708
              fi
          done
709

Tails developers's avatar
Tails developers committed
710
711
1. Add and commit the upgrade-description files and their detached
   signatures to the Git branch used to prepare the release (`stable`
712
   or `testing`):
Tails developers's avatar
Tails developers committed
713

anonym's avatar
anonym committed
714
715
716
717
718
       ( \
         cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
         git commit -m "Update upgrade-description files." && \
         git push origin ${RELEASE_BRANCH:?} \
       )
Tails developers's avatar
Tails developers committed
719

720
1. If preparing a release candidate, move the generated or updated
721
   files to `${MASTER_CHECKOUT:?}`, commit and push: given the updates are
722
723
724
   advertised on the *alpha* channel, while all users use the *stable*
   one by default, this will allow you to more easily test the IUK
   without impacting anyone.
Tails developers's avatar
Tails developers committed
725

726
727
728
729
1. If preparing a point release, copy the generated UDF for the previous
   release in the alpha channel in the master branch, modify its content
   accordingly, sign it, commit and push

anonym's avatar
anonym committed
730
731
732
733
       # If more old versions are supported, add them (whitespace
       # separated) to this variable
       SUPPORTED_OLD_VERSIONS="${PREVIOUS_VERSION:?}"

anonym's avatar
anonym committed
734
735
736
       ( \
         cd ${MASTER_CHECKOUT:?} && \
         git fetch && \
anonym's avatar
anonym committed
737
738
739
740
741
742
743
744
745
         for old_version in ${SUPPORTED_OLD_VERSIONS:?}; do
           stable_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/i386/stable/upgrades.yml" && \
           alpha_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/i386/alpha/upgrades.yml" && \
           git show origin/${RELEASE_BRANCH:?}:${stable_udf:?} \
             | sed -e 's/channel: stable/channel: alpha/' > ${alpha_udf:?} && \
           gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${alpha_udf:?} && \
           mv ${alpha_udf:?}.asc ${alpha_udf:?}.pgp && \
           git add ${alpha_udf:?}* ; \
         done && \
anonym's avatar
anonym committed
746
         git commit -m "Add incremental upgrades on the alpha channel for Tails ${VERSION:?}" && \
anonym's avatar
anonym committed
747
748
         git push origin master:master \
       )
749
750


751
752
753
Prepare the ISO description file for DAVE
-----------------------------------------

anonym's avatar
anonym committed
754
755
If preparing a RC, skip this part.

756
757
Update the ISO description file (IDF) used by the browser extension:

758
    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/i386/stable/latest.yml <<EOF
759
760
761
762
    ---
    build-target: i386
    channel: stable
    product-name: Tails
763
    version: '${VERSION:?}'
764
765
    target-files:
    - sha256: ${ISO_SHA256SUM}
766
767
      size: ${ISO_SIZE_IN_BYTES:?}
      url: http://dl.amnesia.boum.org/tails/stable/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso
768
    EOF
769
    ( cd "${RELEASE_CHECKOUT:?}" && \
770
771
772
      git add wiki/src/install/v1/Tails/i386/stable/latest.yml && \
      git commit -m "Update IDF file for DAVE." )

Tails developers's avatar
Tails developers committed
773
774
775
Upload images
=============

Tails developers's avatar
Tails developers committed
776
777
778
Sanity check
------------

779
780
Verify once more that the TBB we ship is still the most recent (see
above).
Tails developers's avatar
Tails developers committed
781

782
783
## Announce, seed and test the Torrents

Tails developers's avatar
Tails developers committed
784
Announce and seed the Torrents.
785
786
787

Test them with a BitTorrent client running in a different place.

788
789
## Download and seed image from lizard

790
    scp "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
791
       bittorrent.lizard: && \
792
       ssh bittorrent.lizard \
793
         transmission-remote --add tails-i386-${VERSION:?}.torrent \
794
           --find /var/lib/transmission-daemon/downloads/
795

796
797
798
799
<a id="publish-iuk"></a>

Publish the ISO and IUK over HTTP
---------------------------------
Tails developers's avatar
Tails developers committed
800

801
802
803
804
805
806
Upload the images to the primary rsync mirror. Best practice is to first
let bittorrent.lizard download the image, and then copy it from there to
rsync.lizard:

    ssh lizard.tails.boum.org \
        scp -3 -r \
807
            bittorrent.lizard:/var/lib/transmission-daemon/downloads/tails-i386-${VERSION:?} \
808
809
            rsync.lizard:
    ssh rsync.lizard << EOF
810
      sudo chown -R root:rsync_tails \
811
812
         tails-i386-${VERSION:?} \
         Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk && \
813
      sudo chmod -R u=rwX,go=rX \
814
815
816
817
818
819
         tails-i386-${VERSION:?} \
         Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk && \
      sudo mv tails-i386-${VERSION:?} \
              /srv/rsync/tails/tails/${DIST:?}/ && \
      sudo mv Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk \
              /srv/rsync/tails/tails/${DIST:?}/iuk/
820
    EOF
821
822
823
824
825

Update the time in `project/trace` file on the primary rsync mirror
and on the live wiki (even for a release candidate):

	TRACE_TIME=$(date +%s) &&
826
827
828
	echo ${TRACE_TIME:?} | ssh rsync.lizard "cat > /srv/rsync/tails/tails/project/trace" && \
	[ -n "${MASTER_CHECKOUT:?}" ] && \
	echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
829
	(
830
	   cd "${MASTER_CHECKOUT:?}" && \
831
	   git commit wiki/src/inc/trace \
832
	      -m "Updating trace file after uploading ${VERSION:?}." && \
833
	   git push origin master
834
835
	)

836
837
838
839
ISO history
-----------

Push the released ISO to our Tails ISO history git-annex repo, so that
840
841
our isotesters can fetch it from there for their testing. How to do so
is described in our internal Git repo.
842

Tails developers's avatar
Tails developers committed
843
844
845
Update the website and Git repository
=====================================

846
What follows in this section happens on the release branch in
847
`${RELEASE_CHECKOUT:?}`.
848
849
850
851
852
853
854
855
856

If preparing a final release
----------------------------

Skip this part if preparing a RC.

Rename the `.packages` file to remove the `.iso` and build date parts
of its name:

857
858
	mv "${ARTIFACTS:?}"/tails-i386-"${VERSION:?}".iso.packages \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.packages"
859

860
861
Rename the manifest of needed packages as well:

862
	mv "${PACKAGES_MANIFEST:?}" "${ARTIFACTS:?}/tails-i386-${VERSION:?}.build-manifest"
863
864
865

Copy the `.iso.sig`, `.build-manifest`, `.packages`, `.torrent` and
`.torrent.sig` files into the website repository:
Tails developers's avatar
Tails developers committed
866

867
868
869
870
871
	cp "${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso.sig" \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.build-manifest" \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.packages" \
	   "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
	   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/"
872

873
874
875
Remove from `wiki/src/torrents/files/` any remaining file from the
previous release (including any RC).

876
877
Update the size of the ISO image in `inc/*`:

878
        LC_NUMERIC=C ls -l -h ${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso | \
879
          cut -f 5 -d ' ' | sed -r 's/(.+)([MG])/\1 \2iB/' \
880
          > "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_i386_iso_size.html"
881

882
883
Generate the expected signature verification output:

884
      gpg --keyid-format 0xlong --verify "${ISO_PATH:?}.sig" "${ISO_PATH:?}" 2>&1 | \
885
      sed 's/ /\&nbsp;/g;s/</\&lt;/;s/>/\&gt;/;s/$/<br\/>/g' > \
886
      "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_i386_gpg_signature_output.html"
887

888
889
890
891
Update the [[support/known_issues]] page:

- Add regressions brought by the new release.
- Remove older known issues that are fixed by the new release.
892

893
Write the announcement for the release in
894
`wiki/src/news/version_${TAG:?}.mdwn`. See the [[release notes
895
documentation|contribute/how/documentation/release_notes]]
896

897
898
899
900
901
902
XXX: we should probably merge that into the above liked documentation.
**Check you correctly:**

- Updated the `meta title` directive.
- Updated the `meta date` directive.
- Made sure there's an `announce` tag to have an email sent to the
sajolida's avatar
sajolida committed
903
  news mailing list.
904
- Documented important config changes that persistence users have to do
Tails developers's avatar
Tails developers committed
905
  themselves (e.g. the Pidgin proxy settings change in
906
  [[!tails_gitweb_commit 9925321]] breaks all existing persistent
Tails developers's avatar
Tails developers committed
907
  profiles).
908
- Documented known issues.
Tails developers's avatar
Tails developers committed
909

Tails developers's avatar
Tails developers committed
910
Write an announcement listing the security bugs affecting the previous
911
version in
912
`wiki/src/security/Numerous_security_holes_in_${PREVIOUS_VERSION:?}.mdwn`
913
in order to let the users of the old versions
914
know that they have to upgrade. Date it a few days before the ISO
915
916
image to be released was *built*. Including:

917
918
- if we are shipping Linux from testing/sid, the list of CVE fixed in
  Linux since the one shipped in the previous release of Tails:
intrigeri's avatar
intrigeri committed
919
  <http://metadata.ftp-master.debian.org/changelogs/main/l/linux/stable_changelog>
920
- the list of DSA fixed in packages we ship since those that were in
intrigeri's avatar
intrigeri committed
921
  the previous release of Tails: <https://www.debian.org/security/#DSAS>
Tails developers's avatar
Tails developers committed
922
923
924
- the list of BSA fixed in packages we ship since those that were in
  the previous release of Tails:
  <https://lists.debian.org/debian-backports-announce/>
925
- the list of MFSA fixed by the Tor Browser update:
926
  <https://www.mozilla.org/security/announce/>
Tails developers's avatar
Tails developers committed
927

928
929
930
931
932
933
934
If preparing a release candidate
--------------------------------

Skip this part if preparing a final release.

Copy the `.iso.sig` file into the website repository:

935
936
937
	cp "${ISO_PATH:?}.sig" \
	   "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
	      "${MASTER_CHECKOUT:?}/wiki/src/torrents/files/"
938
939

Write the announcement for the release in
940
`${MASTER_CHECKOUT:?}/wiki/src/news/test_${TAG:?}.mdwn`, including:
941
942
943
944
945
946
947
948

- Update the `meta title` directive.
- Update the `meta date` directive.
- Document important config changes that persistence users have to do
  themselves (e.g. the Pidgin proxy settings change in
  [[!tails_gitweb_commit 9925321]] breaks all existing persistent
  profiles).
- Document known issues.
anonym's avatar
anonym committed
949
950
951
952
- This snippet can help to convert the copied changelog's ticket
  references to links:

      sed -i 's@#\([0-9]\{4,5\}\)@[[!tails_ticket \1]]@g' \
953
          wiki/src/news/test_${TAG:?}.mdwn
954
955
956
957

In any case
-----------

elouann's avatar
elouann committed
958
Generate PO files for the announcements with `./build-website`.
959

960
961
962
963
Then, send them to <tails-l10n@boum.org> so that they get translated
shortly, perhaps even soon enough to integrate them before pushing the
release out officially.

964
Then, record the last commit before putting the release out for real:
Tails developers's avatar
Tails developers committed
965

966
	git add wiki/src && \
967
	git commit -m "releasing version ${VERSION:?}"
Tails developers's avatar
Tails developers committed
968

969
970
971
Testing
=======

972
973
974
975
976
977
1. Using `check-mirrors`, choose a fast mirror that already has the
   tentative ISO. E.g. <https://mirrors.kernel.org/tails/> or
   <https://mirrors.wikimedia.org/tails/> are reliable and have plenty
   of bandwidth.
1. Email <tails-testers@boum.org> to ask them to test the tentative
   ISO, pointing them to the up-to-date mirror you've found previously.
978
1. Set up a pad and copy the [[manual test
979
   suite|contribute/release_process/test]] in it.
intrigeri's avatar
intrigeri committed
980
1. Email <tails@boum.org> and potential contributors (see
981
982
   `manual_testers.mdwn` in the internal Git repository) that tests
   may start:
983
   - point them to the up-to-date mirror you've found previously
984
985
986
987
988
989
990
991
   - make it clear what's the deadline
   - make it clear where and how you expect to get feedback
   - attach the Torrent
   - attach the `.packages` file
1. Make sure someone is committed to run the automated test suite.
1. Make sure that enough people are here to run the tests, that they
   report their results in due time, and that they make it clear when
   they're leaving for good.
intrigeri's avatar
intrigeri committed
992
993
994
995
996
1. Fill the holes and make sure that the manual test suite is done in
   due time.
1. Triage test results, reproduce bugs as needed, decide what the next
   step is and make sure it happens: add to known issues? file ticket?
   release blocker?
997

Tails developers's avatar
Tails developers committed
998
999
1000
Go wild!
========

For faster browsing, not all history is shown. View entire blame