release_process.mdwn

[[!meta title="Release process"]]

[[!toc levels=2]]

See the [[release_schedule]].

Requirements
============

To release Tails you'll need some packages installed:

* `tidy mktorrent transmission-cli`
* Aufs DKMS module for your running kernel.
* [[!debpts squashfs-tools]] that honors `$SOURCE_DATE_EPOCH`.
  Install it from our custom `devel` APT suite.
* `tails-iuk` dependencies, including suggested packages (see
  `debian/control` in the `debian` branch of its repo)
* `tails-perl5lib` dependencies (same trick as `tails-iuk` to get the
  list)

Environment
===========

Export the following environment variables to be able to copy'n'paste
the scripts snippets found on this page:

* version numbers (see [[contribute/release_schedule#versioning]]):

        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')

* `NEXT_PLANNED_VERSION`: set to the version number of the next Tails release
  (e.g. 0.23 when releasing 0.22.1, and 1.3 when releasing 1.2)
* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
  *major* Tails release
* `NEXT_PLANNED_MINOR_VERSION`: set to the version number of the next
  *minor* Tails release; if the next release is a point-release, use
  that one; otherwise, use `${VERSION}.1`
* `MAJOR_RELEASE`: set to 1 if preparing a major release, to 0 else
* `ISOS`: the directory where one stores `tails-amd64-*`
  sub-directories like the ones downloaded with BitTorrent.
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
* `RELEASE_BRANCH`: the name of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
* `export DEBEMAIL='tails@boum.org'`
* `export DEBFULLNAME='Tails developers'`
* `export WEBSITE_RELEASE_BRANCH="web/release-${VERSION:?}"`

Pre-freeze
==========

The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

Select the right branch
=======================

What we refer to as the "release branch" (and `RELEASE_BRANCH`) should
be `testing` for major releases, and `stable` for point-releases.

<div class="caution">
Read the remainder of this document from the branch used to prepare the release!
</div>

Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git merge --no-ff origin/master

2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.

5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]


Point-release
-------------

If we are at freeze time for a point-release:

1. Merge the `master` Git branch into `stable`:

        git checkout stable && git merge --no-ff origin/master

2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
   APT suite.

Common steps for point and major releases
-----------------------------------------

Reset the release branch's `config/base_branch`:

        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
           git commit config/base_branch \
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."

Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
3. Send the pad URL to the usual testers (see `manual_testers.mdwn` in
   the `internal.git` repository).

Update included files
=====================

uBlock patterns and settings file
----------------

The patterns+settings file is stored as a converted sqlite text dump in
`config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`.

1. Boot Tails
2. Start the Tor Browser and open the uBlock dashboard by clicking on the uBlock icon.
3. Select the tab *3rd-party filters*
4. Click on the button *Update now* to update all used patterns
5. Close the Tor Browser
7. Copy the `.tor-browser/profile.default/extension-data/ublock0.sqlite`
   from this Tor Browser instance into the root of Tails' Git repo and
   run the following command:

        # For posterity: the general idea is to introduce \r\n as a
        # token where we have made a line break to make the dump more
        # diff-friendly (and, hence, Git-friendly). The most complex
        # expression is the one done with perl, where we employ
        # negative lookahead. What it means, is: replace single
        # occurrences of | except when followed by \\n.
        echo '.dump' | sqlite3 ublock0.sqlite | \
            grep -v "cached_asset_content://cache://compiled-" | \
            awk '!/^INSERT/; /^INSERT/ {print $0 | "sort -n"}' | \
            sed 's_\\n_\\n\r\n_g' | \
            sed 's_,_,\r\n_g' | \
            perl -p -e 's/([^|])\|((?!\||\\n).)/\1\|\r\n\2/g' | \
            sed "/^INSERT INTO \"settings\" VALUES('\(remoteBlacklists\|cached_asset_entries\)'/"'s_,_,\r\n_g' > \
            config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump

8. Commit:

        git commit -m 'Update uBlock Origin patterns + settings file.' \
           config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump \
        && rm ublock0.sqlite

Upgrade bundled binary Debian packages
--------------------------------------

Skip this section unless we are at freeze time for a major release
(i.e. we are about to prepare a release candidate).

That is: make sure the bundled binary Debian packages contain
up-to-date localization files.

For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
and then run the `import-translations` script that is in the
main Tails repository. For example:

	cd whisperback
	"${RELEASE_CHECKOUT:?}"/import-translations

If the `import-translations` script fails to import translations for
the current package, manually copy updated PO files from the
Transifex branches of `git://git.torproject.org/translation.git` (e.g.
`whisperback_completed`) instead. In this case, skip PO files for
[[translation teams that use Git|contribute/how/translate#translate]].

Add and commit.

Then check the PO files:

	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po

Correct any displayed error, then commit the changes if any.

Then see the relevant release processes, and upload the packages to
the release branch's custom APT suite:

* [[tails-installer]]
* [[tails-greeter]]
* [[perl5lib]]
* [[persistence-setup]]
* [[tails-iuk]]
* whisperback:
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package

Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[browser-apparmor-patch]]

Upgrade Thunderbird
---------------

See the dedicated page: [[thunderbird]]

Update PO files
---------------

Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:

	cd "${RELEASE_CHECKOUT:?}" && \
	./import-translations  && \
	./refresh-translations && \
	./submodules/jenkins-tools/slaves/check_po && \
	git add po && git commit -m 'Update PO files.'

If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

When preparing an actual release
================================

If we're about to prepare an image for a final (non-RC) release, then
follow these instructions:

Major release
-------------

[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
custom APT suite.

Point-release
-------------

<div class="note">
For point-releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>

[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
custom APT suite.

Update other base branches
==========================

1. Merge the release branch into `devel` following the instructions for
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].

2. Merge `devel` into `feature/buster` following the instructions for
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
   Given that these two branches' APT suites have diverged a lot, and
   that `tails-merge-suite` currently happily overwrites newer
   packages in the target with older packages from the source, it's
   probably easier to just merge each individual APT overlay that was
   just merged into the release branch into `feature/buster`'s APT
   suite. Also, most of our just upgraded bundled packages
   (e.g. `tails-greeter`) may need to be rebuilt for Stretch.

3. Ensure that the release, `devel` and `feature/buster` branches
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.

4. Push the modified branches to Git:

        git push origin                          \
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
           feature/buster:feature/buster       \
           devel:devel

Update more included files
==========================

Changelog
---------

Remove the placeholder entry for next release in `debian/changelog`,
and then:

	git checkout "${RELEASE_BRANCH:?}" && \
	./release ${VERSION:?} ${PREVIOUS_TAG:?}

This populates the Changelog with the Git log entries.

Then, launch an editor for the needed cleanup of the result:

	dch -e

Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
* the diff between the previous version's `.packages` file and the one
  from the to-be-released ISO;
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
  in Redmine.

Finally, sanity check the version and commit:

	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
	else
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
	fi

Included website
----------------

### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

### version number

If preparing a RC, skip this part.

In the branch used to build the release, update the `wiki/src/inc/*` files to
match the *version number* and *date* of the new release. Set the date
at least 24 hours in the future! Between tests and mirror synchronization,
the build will not be released on the same day. Try to make sure it
matches the date of the future signature.

	RELEASE_DATE='2015-11-03'

	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html
	sed -ri "s%news/version_.*]]%news/version_${VERSION:?}]]%" wiki/src/inc/stable_amd64_release_notes.*
	${EDITOR:?} wiki/src/inc/*.html
	./build-website
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."

Website translations
--------------------

Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:

	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"

Call for translation
====================

If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].

To get a list of changes on the website:

    git diff --stat ${PREVIOUS_VERSION:?}.. -- \
        wiki/src/'*'.{mdwn,html} \
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'

Import the signing key
======================

Skip this part if you have a Tails signing subkey on an OpenPGPG
smartcard, i.e. if you are one of the usual release managers. This is
only relevant when the master key has been reassembled, e.g. for
signing a Tails emergency release where none of the usual release
managers are available.

You should never import the Tails signing key into your own keyring,
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:

    export GNUPGHOME=$(mktemp -d)
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"

Build the almost-final image
============================

1. [[Build an ISO image|contribute/build]] from the release branch.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep at least the resulting ISO image and the manifest of needed
   packages until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"


Tag the release in Git
======================

	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
	git push --tags origin "${RELEASE_BRANCH:?}"

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

Prepare the versioned APT suites
================================

* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].

* Prepare tagged snapshots of upstream APT repositories:

          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"

  Note:

  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
    [[custom APT repository|APT_repository/custom]] are
    listed under "some packages were not found anywhere" (because we
    are currently not using time-based snapshots for our custom APT
    repository). However, _no other package should be on that list_.
    Now, we have a "safety" net, in case you don't notice such a problem: if
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.

Build images
============

Sanity check
------------

Verify that the Tor Browser release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag for the targeted
Tor Browser version in these Git repositories:

* <https://gitweb.torproject.org/builders/tor-browser-bundle.git>
* <https://gitweb.torproject.org/tor-browser.git>

A new tag may indicate that a new Tor Browser release is imminent.

Better catch this before people spend time doing manual tests.

SquashFS file order
-------------------

1. Burn the almost final ISO image to a DVD.
1. Boot this DVD **on bare metal**.
1. Add `profile` to the kernel command-line.
1. Login.
1. Wait for the "Tor is ready" notification.
1. Start the web browser.
1. A few minutes later, once the `boot-profile` process has been
   killed, retrieve the new sort file from `/var/log/boot-profile`.
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
1. Inspect the Git diff (including diff stat), apply common sense.
   The following command is also helpful but requires that you save a
   copy of the old sort file into `/tmp/squashfs.sort.old`:

        diff -NaurB \
            <( cut -d' ' -f1 /tmp/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort | sort ) \
            | less

1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

Build the final image
---------------------

Then all included files should be up-to-date and the versioned APT
suite should be ready, so it is time to:

* Mark the version as "released" in the changelog:

        dch --release --no-force-save-on-release --maintmaint
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog

* Export `SOURCE_DATE_EPOCH`:

        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')

* tag the release *again*, with all included files in:
  
        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
        git push --tags --force && \
        git push origin "${RELEASE_BRANCH:?}"

   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
   commands by e.g. committing a commit in between `git tag` and the
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!

* build the final image!

* make sure at least two `tails@` members will build the ISO
  image. Example: if you, the RM, is a `tails@` member, you only have
  to ask one more member.

* compare the new build manifest with the one from the previous,
  almost final build; they should be identical, except that the
  `debian-security` serial might be higher. To ensure we publish
  the final build's `.build-manifest`, please run:

        export PACKAGES_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso.build-manifest"

* check out a new branch:

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

  (as soon as a new commit is created on `$RELEASE_BRANCH`, its ISO
  build will start failing until a new changelog entry is created,
  which we don't want to do on `$RELEASE_BRANCH` before it's merged
  into `master` at release time)

Generate the OpenPGP signatures and Torrents
============================================

First, create a directory with a suitable name and go there:

	mkdir "${ISOS:?}/tails-amd64-${VERSION:?}" && \
	cd "${ISOS:?}/tails-amd64-${VERSION:?}"

Second, move the built image to this brand new directory:

	mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.iso" \
	   "${ISOS:?}/tails-amd64-${VERSION:?}/"

Third, generate detached OpenPGP signatures for the image to be
published, in the same directory as the image and with a `.sig`
extension; e.g.

	gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *.iso
	rename 's,\.asc$,.sig,' *.asc

Fourth, go up to the parent directory, create a `.torrent` file and
check the generated `.torrent` files metainfo:

	cd .. && \
	mktorrent \
	  -a 'udp://tracker.torrent.eu.org:451' \
	  -a 'udp://tracker.coppersurfer.tk:6969'   \
	  "tails-amd64-${VERSION:?}" && \
	transmission-show tails-amd64-${VERSION:?}.torrent

Lastly, let's set some variables to be used later:

    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"

<a id="prepare-iuk"></a>

Prepare incremental upgrades
============================

Build the Incremental Upgrade Kits
----------------------------------

Incremental upgrades may be skipped if the delta is too big (like when
migrating to a new Debian release) or if there are changes outside of
the scope for IUKs (like partition table changes). Use common sense!

Use `tails-create-iuk` to build the following IUKs:

* From the two previous *planned* releases, and any emergency releases
  in between and after. This should be, more or less, all releases for
  the last 12 weeks (although irregularities in Firefox release
  schedule may add or remove a few weeks).

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
  next version.

Include each such version in a white-space separated list called
`IUK_SOURCE_VERSIONS`, (e.g. `IUK_SOURCE_VERSIONS="2.8 2.9 2.9.1 2.10~rc1"`)
and run the following:

    for source_version in ${IUK_SOURCE_VERSIONS}; do
        sudo su -c "cd ${IUK_CHECKOUT:?} && \
          SOURCE_DATE_EPOCH=$SOURCE_DATE_EPOCH \
          LC_ALL=C \
          PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
            ./bin/tails-create-iuk \
               --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
               --old-iso \"${ISOS:?}/tails-amd64-${source_version:?}/tails-amd64-${source_version:?}.iso\" \
               --new-iso \"${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso\"          \
               --outfile \"${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk\""
    done

Note that developer tools for creating IUK and upgrade-description
files were only tested on Debian sid. It should hopefully work well on
Jessie too.

Make sure at least two `tails@` members reproduce the same IUKs.
Provide them with the value of `IUK_SOURCE_VERSIONS` that you used!

<a id="prepare-upgrade-description-files"></a>

Prepare upgrade-description files
---------------------------------

1. Prepare upgrade-description files (see the [[upgrade-description
   files
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
   for details). The idea is to:

   * update (create if needed) an upgrade-description file for every
     *previous* supported release (e.g. N~rc1, N-1, N-1~rc2) that replaces
     all existing upgrade paths with the path to the version being
     released;
   * create a new upgrade-description for the version being released
     and for the next one, that expresses that *no* upgrade is
     available for these ones yet.

   This is what `tails-iuk-generate-upgrade-description-files` tool
   does:

        ( cd ${IUK_CHECKOUT:?} && \
        ./bin/tails-iuk-generate-upgrade-description-files \
            --build-target amd64 \
            --version "${VERSION:?}" \
            --next-version "${NEXT_PLANNED_VERSION:?}" \
            --next-version "${NEXT_PLANNED_VERSION:?}~rc1" \
            --next-version "${VERSION:?}.1" \
            --iso "${ISOS_PATH:?}" \
            --previous-version "${PREVIOUS_VERSION:?}" \
            --previous-version "${VERSION:?}~rc1" \
            --iuks "${ISOS:?}" \
            --release-checkout "${RELEASE_CHECKOUT:?}" \
            --major-release "${MAJOR_RELEASE:?}" \
       )

   Note:
   * The `--iuks` argument must point to the directory where the IUKs
     generated at the previous step are stored.
   * At least the last stable release and the previous release
     candidates for the version being released must be passed to
     `--previous-version`.
   * Older versions for which there is no incremental upgrade path to
     the new release must be passed with `--previous-version`, so that
     users who skipped a release or two are informed of the new one.
     Note that multi-steps incremental upgrade paths are valid and
     supported: e.g. when releasing 1.1.2, 1.1 users should still be
     able to incrementally upgrade to 1.1.1, and in turn to 1.1.2; to
     make this work, one must _not_ pass `--previous-version 1.1`,
     that would remove the existing incremental upgrade path from 1.1
     to 1.1.1.
   * If preparing anything but a final release (e.g. an alpha, beta
     or RC), add `--channel alpha`
   * If preparing anything but a final release (e.g. an alpha, beta
     or RC), drop all `--next-version`
     arguments, and instead pass (**untested!**)
     `--next-version $(echo ${VERSION:?} | sed -e 's,~rc.*$,,')`
   * If preparing a point-release, pass neither
     `--next-version "${VERSION:?}.1"`,
     nor `--next-version "${VERSION:?}.1~rc1"`

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

        find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
           -type f -name upgrades.yml | \
           while read udf; do
               if [ -n "$(git status --porcelain "${udf:?}")" ]; then
                   gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign "${udf:?}"
                   mv --force "${udf:?}.asc" "${udf:?}.pgp"
                   ( \
                     cd ${IUK_CHECKOUT:?} && \
                     ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
                   ) || break
               fi
           done

1. Add and commit the upgrade-description files and their detached
   signatures to the Git branch used to prepare the release
   (`$WEBSITE_RELEASE_BRANCH`):

        ( \
          cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
          git commit -m "Update upgrade-description files." && \
          git push origin ${WEBSITE_RELEASE_BRANCH:?} \
        )

1. If preparing anything but a final release (e.g. an alpha, beta
   or RC), copy the generated or updated files to
   `${MASTER_CHECKOUT:?}`, replace `channel: alpha` with `channel:
   test`, sign them, commit and push.

1. Else, if preparing a final release, copy the generated UDF for the previous
   release to the *test* channel in `$MASTER_CHECKOUT`, modify its content
   accordingly, sign it, commit and push:

        ( \
          cd ${MASTER_CHECKOUT:?} && \
          git fetch && \
          for old_version in ${IUK_SOURCE_VERSIONS:?}; do
            stable_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/stable/upgrades.yml" && \
            test_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/amd64/test/upgrades.yml" && \
            mkdir -p "$(dirname "$test_udf")" && \
            git show origin/${WEBSITE_RELEASE_BRANCH:?}:${stable_udf:?} \
              | sed -e 's/channel: stable/channel: test/' > ${test_udf:?} && \
            gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${test_udf:?} && \
            mv ${test_udf:?}.asc ${test_udf:?}.pgp && \
            git add ${test_udf:?}* ; \
          done && \
          git commit -m "Add incremental upgrades on the test channel for Tails ${VERSION:?}" && \
          git push origin master:master \
        )


Prepare the ISO description file for DAVE
-----------------------------------------

If preparing a RC, skip this part.

Update the ISO description file (IDF) used by the browser extension:

    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/i386/stable/latest.yml <<EOF
    ---
    build-target: amd64
    channel: stable
    product-name: Tails
    version: '${VERSION:?}'
    target-files:
    - sha256: ${ISO_SHA256SUM}
      size: ${ISO_SIZE_IN_BYTES:?}
      url: http://dl.amnesia.boum.org/tails/stable/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso
    EOF
    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/amd64/stable/latest.yml <<EOF
    ---
    build-target: amd64
    channel: stable
    product-name: Tails
    version: '${VERSION:?}'
    target-files:
    - sha256: ${ISO_SHA256SUM}
      size: ${ISO_SIZE_IN_BYTES:?}
      url: http://dl.amnesia.boum.org/tails/stable/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso
    EOF
    ( cd "${RELEASE_CHECKOUT:?}" && \
      git add wiki/src/install/v1/Tails/{i386,amd64}/stable/latest.yml && \
      git commit -m "Update IDF file for DAVE." )

Upload images
=============

Sanity check
------------

Verify once more that the Tor Browser we ship is still the most recent (see
above).

Reproducibility
---------------

Previously you have asked `tails@` members to reproduce the Tails ISO
image and all IUKs; now tell all participants to send you the
`SHA-512` hashes of their ISO and IUKs over signed email.

* If all hashes match: yay, we're good to go!

* If the reproduction attempts haven't been completed yet: continue at
  your own risk! If you get a negative answer later you might have to
  undo everything done from this point on. It's still a good idea to
  optimistically assume success in order to not be blocked at this
  point. However, be mentally prepared that it might have to be done
  once more, and make sure to return to this section once everyone is
  done with their reproduction attempts, and certainly before making
  the release public.

* If there is a hash mismatch for the ISO: ouch!  Now we are in a
  tricky situation: on the one hand it seems like a poor idea to block
  users from benefiting from this release's security updates, but on
  the other hand the failure might imply that something nefarious is
  going on. At this stage, no matter what, immediately compare the
  ISOs (using `diffoscope`) and try to rule out build system
  compromise.

  - If something seemingly malicious is found, then let's take a step
    back: we might be compromised, so we are in no position to
    release. Halt the release, involve the rest of `tails@`, and then
    try to re-establish trust in all build machines and infra
    involved, etc. Have fun!

  - Otherwise:

    * If the source of non-determinism is identified quickly and is
      easy and fast to fix, *and* the QA of the current ISO has not
      gone very far (so at least that time is not wasted), then you
      should consider abandoning the current version, and immediately
      start preparing an emergency release with:

      - the reproducibility fix

      - a new changelog entry,

      - adjustments to the release notes so they are re-purposed for
        this emergency release (the abandoned release gets none, since
        it effectively never will be released publicly).

    * Otherwise, let's release any way. But let's add a known issue
      about "This Tails release has reproducibility issues" to the
      release notes, linking to the ticket(s) (or similar) where the
      nature of the reproducibility failure is clearly described.

* If there is a hash mismatch for an IUK (or several): proceed with
  the release, except for the problematic IUK(s); remove them from the
  mirrors, and remove the affected incremental upgrade paths from the
  UDFs! In parallel with the rest of the release process, try to
  figure out what the problem with the IUK is and fix the cause, so a
  good IUK can be released as soon as possible. If a seemingly
  malicious difference is found, then immediately halt the release and
  go to the "If something seemingly malicious is found" case for the
  ISO above. Because of this it is advisable to not publicly release
  until malicious differences have been ruled out.

<a id="publish-iuk"></a>

Publish the ISO and IUK over HTTP
---------------------------------

Upload the IUKs to the primary rsync mirror:

    for source_version in ${IUK_SOURCE_VERSIONS}; do
       rsync --partial --inplace --progress -v \
          "${ISOS:?}/Tails_amd64_${source_version:?}_to_${VERSION:?}.iuk" \
          rsync.lizard:
    done

Upload the ISO signature to the primary rsync mirror:

    scp "${ISO_PATH:?}.sig" rsync.lizard:

Pick a build from `$RELEASE_BRANCH` that produced an ISO identical to
the one you've built locally (`XXX` must be the job ID, i.e.
an integer):

    MATCHING_JENKINS_BUILD_ID=XXX

Copy the ISO to the primary rsync mirror and verify the signature:

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh rsync.lizard gpg --import
    ssh rsync.lizard << EOF
       wget \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,}
    EOF

Move files in place with proper ownership and permissions:

    ssh rsync.lizard << EOF
      sudo install -o root -g rsync_tails -m 0755 -d \
         /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
      sudo chown root:rsync_tails \
         tails-amd64-${VERSION:?}.iso* \
         Tails_amd64_*_to_${VERSION:?}.iuk && \
      sudo chmod u=rwX,go=rX \
         tails-amd64-${VERSION:?}.iso* \
         Tails_amd64_*_to_${VERSION:?}.iuk && \
      sudo mv tails-amd64-${VERSION:?}.iso* \
              /srv/rsync/tails/tails/${DIST:?}/tails-amd64-${VERSION:?} && \
      sudo mv Tails_amd64_*_to_${VERSION:?}.iuk \
              /srv/rsync/tails/tails/${DIST:?}/iuk/
    EOF

Update the time in `project/trace` file on the primary rsync mirror
and on the live wiki (even for a release candidate):

	TRACE_TIME=$(date +%s) &&
	echo ${TRACE_TIME:?} | ssh rsync.lizard "cat > /srv/rsync/tails/tails/project/trace" && \
	[ -n "${MASTER_CHECKOUT:?}" ] && \
	echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
	(
	   cd "${MASTER_CHECKOUT:?}" && \
	   git commit wiki/src/inc/trace \
	      -m "Updating trace file after uploading ${VERSION:?}." && \
	   git push origin master
	)


## Announce, seed and test the Torrents

    cat "${RELEASE_CHECKOUT:?}/wiki/src/tails-signing.key" \
       | ssh bittorrent.lizard gpg --import
    scp \
       "${ISOS:?}/tails-amd64-${VERSION:?}.torrent" \
       "${ISOS_PATH:?}.sig" \
       bittorrent.lizard: && \
    ssh bittorrent.lizard << EOF
       mkdir --mode 0755 "tails-amd64-${VERSION:?}" && \
       mv "tails-amd64-${VERSION:?}.iso.sig" \
          "tails-amd64-${VERSION:?}/" && \
       cd "tails-amd64-${VERSION:?}" && \
       wget \
          "https://nightly.tails.boum.org/build_Tails_ISO_${RELEASE_BRANCH:?}/builds/${MATCHING_JENKINS_BUILD_ID:?}/archive/build-artifacts/tails-amd64-${VERSION:?}.iso" && \
       gpg --verify tails-amd64-${VERSION:?}.iso{.sig,} && \
       cd && \
       chmod -R go+rX "tails-amd64-${VERSION:?}" && \
       mv \
          "tails-amd64-${VERSION:?}" \
          /var/lib/transmission-daemon/downloads/ && \
       transmission-remote --add tails-amd64-${VERSION:?}.torrent \
                           --find /var/lib/transmission-daemon/downloads/
    EOF

Test that you can start downloading the ISO with a BitTorrent client.

ISO history
-----------

Push the released ISO to our Tails ISO history git-annex repo, so that
our isotesters can fetch it from there for their testing. How to do so
is described in our internal Git repo.

Testing
=======

1. Using `check-mirrors`, choose a fast mirror that already has the
   tentative ISO. E.g. <https://mirrors.kernel.org/tails/> or
   <https://mirrors.wikimedia.org/tails/> are reliable and have plenty
   of bandwidth.

        ./check-mirrors.rb --allow-multiple --channel ${DIST:?} \
            --ip $(dig +short mirrors.kernel.org | tail -n1) \
            tails-amd64-${VERSION:?}