release_process.mdwn

[[!meta title="Release process"]]

[[!toc levels=2]]

See the [[release_schedule]].

Requirements
============

To release Tails you'll need some packages installed:

* mktorrent and transmission-cli
* Aufs DKMS module for your running kernel.
* tails-iuk dependencies (see debian/control in the `debian` branch of its repo)
* tails-perl5lib dependencies (same trick than tails-iuk to get the list)

Environment
===========

Export the following environment variables to be able to copy'n'paste
the scripts snippets found on this page:

* version numbers (see [[contribute/release_schedule#versioning]]):

      export VERSION=$(dpkg-parsechangelog -SVersion)
      export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
      export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)

* `NEXT_PLANNED_VERSION`: set to the version number of the next Tails release
  (e.g. 0.23 when releasing 0.22.1, and 1.3 when releasing 1.2)
* `MAJOR_RELEASE`: set to 1 if preparing a major release, to 0 else
* `ISOS`: the directory where one stores `tails-i386-*`
  sub-directories like the ones downloaded with BitTorrent.
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
* `RELEASE_BRANCH`: the name of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
* `IUK_CHECKOUT`: a checkout of the relevant tag of the `iuk`
  Git repository.
* `PERL5LIB_CHECKOUT`: a checkout of the relevant tag of the
  `perl5lib` Git repository.
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
* `export DEBEMAIL='tails@boum.org'`
* `export DEBFULLNAME='Tails developers'`

Pre-freeze
==========

The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.

Update Icedove preferences
------------------------------

* update `extensions.enigmail.configuredVersion` in
  `config/chroot_local-includes/etc/skel/.icedove/profile.default/preferences/0000tails.js`

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

Select the right branch
=======================

What we refer to as the "release branch" (and `RELEASE_BRANCH`) should
be `testing` for major releases, and `stable` for point-releases.

<div class="caution">
Read the remainder of this document from the branch used to prepare the release!
</div>

Freeze
======

Major release
-------------

If we are at freeze time for a major release:

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git merge --no-ff origin/master

2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
   APT suite.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.

5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
   long enough, by bumping their `Valid-Until` to 10 days after the
   next major release (the one _after_ the one you're preparing)'s
   scheduled date:
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]


Point-release
-------------

If we are at freeze time for a point-release:

1. Merge the `master` Git branch into `stable`:

        git checkout stable && git merge --no-ff origin/master

2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
   APT suite.

Common steps for point and major releases
-----------------------------------------

Reset the release branch's `config/base_branch`:

        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
           git commit config/base_branch \
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."

Update included files
=====================

uBlock patterns and settings file
----------------

The patterns+settings file is stored as a converted sqlite text dump in
`config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump`.

1. Boot Tails
2. Start the Tor Browser and open the uBlock dashboard by clicking on the uBlock icon.
3. Select the tab *3rd-party filters*
4. Click on the button *Update now* to update all used patterns
5. Close the Tor Browser
7. Copy the `.tor-browser/profile.default/extension-data/ublock0.sqlite`
   from this Tor Browser instance into the root of Tails' Git repo and
   run the following command:

       # For posterity: the general idea is to introduce \r\n as a
       # token where we have made a line break to make the dump more
       # diff-friendly (and, hence, Git-friendly). The most complex
       # expression is the one done with perl, where we employ
       # negative lookahead. What it means, is: replace single
       # occurrences of | except when followed by \\n.
       echo '.dump' | sqlite3 ublock0.sqlite | \
           grep -v "cached_asset_content://cache://compiled-" | \
           awk '!/^INSERT/; /^INSERT/ {print $0 | "sort -n"}' | \
           sed 's_\\n_\\n\r\n_g' | \
           sed 's_,_,\r\n_g' | \
           perl -pi -e 's/([^|])\|((?!\||\\n).)/\1\|\r\n\2/g' | \
           sed "/^INSERT INTO \"settings\" VALUES('\(remoteBlacklists\|cached_asset_entries\)'/"'s_,_,\r\n_g' > \
           config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump

8. Commit:

       git commit -m 'Update uBlock Origin patterns + settings file.' \
          config/chroot_local-includes/usr/share/tails/ublock-origin/ublock0.dump

9. Remove the original `ublock0.sqlite` from the Git root.

Upgrade bundled binary Debian packages
--------------------------------------

Skip this section unless we are at freeze time for a major release
(i.e. we are about to prepare a release candidate).

That is: make sure the bundled binary Debian packages contain
up-to-date localization files.

For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
and then run the `import-translations` script that is in the
main Tails repository. For example:

	cd whisperback
	"${RELEASE_CHECKOUT:?}"/import-translations

If the `import-translations` script fails to import translations for
the current package, manually copy updated PO files from the
Transifex branches of `git://git.torproject.org/translation.git` (e.g.
`whisperback_completed`) instead. In this case, skip PO files for
[[translation teams that use Git|contribute/how/translate#translate]].

Add and commit.

Then check the PO files:

	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/check_po

Correct any displayed error, then commit the changes if any.

Then see the relevant release processes, and upload the packages to
the release branch's custom APT suite:

* [[tails-installer]]
* [[tails-greeter]]
* [[perl5lib]]
* [[persistence-setup]]
* [[tails-iuk]]
* whisperback:
  * follow [upstream release process](https://git-tails.immerda.ch/whisperback/plain/HACKING)
  * build a Debian package

Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Icedove
---------------

See the dedicated page: [[icedove]]

Update PO files
---------------

Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:

	cd "${RELEASE_CHECKOUT:?}" && \
	./import-translations  && \
	./refresh-translations && \
	./submodules/jenkins-tools/slaves/check_po && \
	git add po && git commit -m 'Update PO files.'

If `check_po` complains:

* delete the offending PO files;
* send a note to <tails-l10n@boum.org> so that they get in touch with
  whoever can fix them.

When preparing an actual release
================================

If we're about to prepare an image for a final (non-RC) release, then
follow these instructions:

Major release
-------------

[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
custom APT suite.

Point-release
-------------

<div class="note">
For point-releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>

[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
custom APT suite.

Update other base branches
==========================

1. Merge the release branch into `devel` following the instructions for
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].

2. Merge `devel` into `feature/stretch` following the instructions for
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
   Given that these two branches' APT suites have diverged a lot, and
   that `tails-merge-suite` currently happily overwrites newer
   packages in the target with older packages from the source, it's
   probably easier to just merge each individual APT overlay that was
   just merged into the release branch into `feature/stretch`'s APT
   suite. Also, most of our just upgraded bundled packages
   (e.g. `tails-greeter`) may need to be rebuilt for Stretch.

3. Ensure that the release, `devel` and `feature/stretch` branches
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.

4. Push the modified branches to Git:

        git push origin                          \
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
           feature/stretch:feature/stretch       \
           devel:devel

Update more included files
==========================

Changelog
---------

Remove the placeholder entry for next release in `debian/changelog`,
and then:

	git checkout "${RELEASE_BRANCH:?}" && \
	./release ${VERSION:?} ${PREVIOUS_VERSION:?}

This populates the Changelog with the Git log entries.

Then, launch an editor for the needed cleanup of the result:

	dch -e

Then, gather other useful information from:

* every custom bundled package's own Changelog (Greeter, Persistent
  Volume Assistant, etc.);
* the diff between the previous version's `.packages` file and the one
  from the to-be-released ISO;
* the "Fix committed" section on the *Release Manager View for ${VERSION:?}*
  in Redmine.

Finally, sanity check the version and commit:

	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
	else
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
	fi

Included website
----------------

### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

### version number

If preparing a RC, skip this part.

In the branch used to build the release, update the `wiki/src/inc/*` files to
match the *version number* and *date* of the new release. Set the date
at least 24 hours in the future! Between tests and mirror synchronization,
the build will not be released on the same day. Try to make sure it
matches the date of the future signature.

	RELEASE_DATE='2015-11-03'

	echo "${VERSION:?}"      > wiki/src/inc/stable_i386_version.html
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_i386_date.html
	sed -ri "s%news/version_.*]]%news/version_${VERSION:?}]]%" wiki/src/inc/stable_i386_release_notes.*
	${EDITOR:?} wiki/src/inc/*.html
	./build-website
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."

### features and design documentation

Read the Changelog carefully, and update [[doc/about/features]]
pages accordingly.

Website translations
--------------------

Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:

	./build-website && git add wiki/src && git commit -m 'Update website PO files.'
	git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"

Call for translation
====================

If at freeze time, send a call for translations to tails-l10n, making it clear what
Git branch the translations must be based on, and what are the
priorities. Also, add a few words to remember the translation teams
using Git that they should regularly contact Transifex translators,
as detailed on the [[documentation for
translators|contribute/how/translate]].

To get a list of changes on the website:

    git diff --stat ${PREVIOUS_VERSION:?}.. -- \
        *.{mdwn,html} \
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'

Import the signing key
======================

Skip this part if you have a Tails signing subkey on an OpenPGPG
smartcard, i.e. if you are one of the usual release managers. This is
only relevant when the master key has been reassembled, e.g. for
signing a Tails emergency release where none of the usual release
managers are available.

You should never import the Tails signing key into your own keyring,
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:

    export GNUPGHOME=$(mktemp -d)
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"

Build the almost-final image
============================

1. [[Build an ISO image|contribute/build]] from the release branch.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep at least the resulting ISO image and the manifest of needed
   packages until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export PACKAGES_MANIFEST=XXX ; \
        [ -f "${PACKAGES_MANIFEST:?}" ] || echo "ERROR: PACKAGES_MANIFEST is incorrect"


Tag the release in Git
======================

	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
	git push --tags origin "${RELEASE_BRANCH:?}"

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last debian/changelog entry has a tag. There are workarounds
we need to decide and implement.

Prepare the versioned APT suites
================================

* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].

* Prepare tagged snapshots of upstream APT repositories:

          ./bin/tag-apt-snapshots "${PACKAGES_MANIFEST:?}" "${TAG:?}"

  Note:

  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
    [[custom APT repository|APT_repository/custom]] are
    listed under "some packages were not found anywhere" (because we
    are current not using time-based snapshots for our custom APT
    repository). However, _no other package should be on that list_.
    Now, we have a "safety" net, in case you don't notice such a problem: if
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.

Build images
============

Sanity check
------------

Verify that the TBB release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag for the targetted TBB
and Tor Browser versions in their respective Git repos:

* <https://gitweb.torproject.org/builders/tor-browser-bundle.git>
* <https://gitweb.torproject.org/tor-browser.git>

A new tag may indicate that a new TBB release is imminent.

Better catch this before people spend time doing manual tests.

SquashFS file order
-------------------

1. Burn the almost final ISO image to a DVD.
1. Boot this DVD **on bare metal**.
1. Add `profile` to the kernel command-line.
1. Login.
1. Wait for the "Tor is ready" notification.
1. Start the web browser.
1. A few minutes later, once the `boot-profile` process has been
   killed, retrieve the new sort file from `/var/log/boot-profile`.
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
1. Cleanup a bit:
   - remove `var/log/live/config.pipe`: otherwise the boot is broken
     or super-slow
   - remove the bits about `kill-boot-profile` at the end: they're
     only useful when profiling the boot
1. Inspect the Git diff (including diff stat), apply common sense.
   The following command is also helpful but requires that you save a
   copy of the old sort file into `/tmp/squashfs.sort.old`:

       diff -NaurB \
           <( cut -d' ' -f1 /tmp/squashfs.sort.old | sort ) \
           <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort | sort ) \
           | less

1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`

Build the final image
---------------------

Then all included files should be up-to-date and the versioned APT
suite should be ready, so it is time to:

* Mark the version as "released" in the changelog:

      dch --release --no-force-save-on-release --maintmaint
      git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog

* tag the release *again*, with all included files in:
  
      git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
              -m "tagging version ${VERSION:?}" "${TAG:?}" && \
      git push origin "${RELEASE_BRANCH:?}" && \
      git push --tags --force

* check out the release tag:

      git checkout "${TAG:?}"

* build the final image!

* compare the new build manifest with the one from the previous,
  almost final build; they should be identical, except that the
  `debian-security` serial/reference might be higher. To ensure we get
  the final build's .build-manifest, please run:

      export PACKAGES_MANIFEST="${ARTIFACTS:?}/tails-i386-${VERSION:?}.iso.build-manifest"

* check out the release branch again:

      git checkout "${RELEASE_BRANCH:?}"

Generate the OpenPGP signatures and Torrents
============================================

First, create a directory with a suitable name and go there:

	mkdir "${ISOS:?}/tails-i386-${VERSION:?}" && \
	cd "${ISOS:?}/tails-i386-${VERSION:?}"

Second, move the built image to this brand new directory:

	mv "${ARTIFACTS:?}/tails-i386-${VERSION:?}.iso" \
	   "${ISOS:?}/tails-i386-${VERSION:?}/"

Third, generate detached OpenPGP signatures for the image to be
published, in the same directory as the image and with a `.sig`
extension; e.g.

	gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *.iso
	rename 's,\.asc$,.sig,' *.asc

Fourth, go up to the parent directory, create a `.torrent` file and
check the generated `.torrent` files metainfo:

	cd .. && \
	mktorrent \
	  -a 'udp://tracker.torrent.eu.org:451' \
	  -a 'udp://tracker.coppersurfer.tk:6969'   \
	  "tails-i386-${VERSION:?}" && \
	transmission-show tails-i386-${VERSION:?}.torrent

Lastly, let's set some variables to be used later:

    ISO_PATH="${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso"
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"

<a id="prepare-iuk"></a>

Prepare incremental upgrades
============================

Build the Incremental Upgrade Kits
----------------------------------

Incremental upgrades may be skipped if the delta is too big (like when
migrating to a new Debian release) or if there are changes outside of
the scope for IUKs (like partition table changes). Use common sense!

Use `tails-create-iuk` to build the following IUKs:

* From the two previous *planned* releases, and any emergency releases
  in between and after. This should be, more or less, all releases for
  the last 12 weeks (although irregularities in Firefox release
  schedule may add or remove a few weeks).

* From the last RC for the version being released, e.g. 1.0~rc1 to
  1.0. This should be done even if there was no IUK generated from the
  previous stable release since it is a good way to test the iuk code
  that'll be used for the incremental upgrade paths to the
  next version.

Include each such version in a white-space separated list called
`IUK_SOURCE_VERSIONS`, (e.g. `IUK_SOURCE_VERSIONS="2.8 2.9 2.9.1 2.10~rc1"`)
and run the following:

    for source_version in ${IUK_SOURCE_VERSIONS}; do
        sudo su -c "cd ${IUK_CHECKOUT:?} && \
          PERL5LIB=\"${PERL5LIB_CHECKOUT:?}/lib\" \
            ./bin/tails-create-iuk \
               --squashfs-diff-name \"${VERSION:?}.squashfs\"           \
               --old-iso \"${ISOS:?}/tails-i386-${source_version:?}/tails-i386-${source_version:?}.iso\" \
               --new-iso \"${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso\"          \
               --outfile \"${ISOS:?}/Tails_i386_${source_version:?}_to_${VERSION:?}.iuk\""
    done

Note that developer tools for creating IUK and upgrade-description
files were only tested on Debian sid. It should hopefully work well on
Jessie too.

<a id="prepare-upgrade-description-files"></a>

Prepare upgrade-description files
---------------------------------

1. Prepare upgrade-description files (see the [[upgrade-description
   files
   specification|contribute/design/incremental_upgrades#upgrade-description-files]]
   for details). The idea is to:

   * update (create if needed) an upgrade-description file for every
     *previous* supported release (e.g. N~rc1, N-1, N-1~rc2) that replaces
     all existing upgrade paths with the path to the version being
     released;
   * create a new upgrade-description for the version being released,
     that expresses that *no* upgrade is available for that one yet.

   This is what `tails-iuk-generate-upgrade-description-files` tool
   does:

       ( cd ${IUK_CHECKOUT:?} && \
       ./bin/tails-iuk-generate-upgrade-description-files \
           --version "${VERSION:?}" \
           --next-version "${NEXT_PLANNED_VERSION:?}" \
           --next-version "${NEXT_PLANNED_VERSION:?}~rc1" \
           --next-version "${VERSION:?}.1" \
           --iso "${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso" \
           --previous-version "${PREVIOUS_VERSION:?}" \
           --previous-version "${VERSION:?}~rc1" \
           --iuks "${ISOS:?}" \
           --release-checkout "${RELEASE_CHECKOUT:?}" \
           --major-release "${MAJOR_RELEASE:?}" \
       )

   Note:
   * The `--iuks` argument must point to the directory where the IUKs
     generated at the previous step are stored.
   * At least the last stable release and the previous release
     candidates for the version being released must be passed to
     `--previous-version`.
   * Older versions for which there is no incremental upgrade path to
     the new release must be passed with `--previous-version`, so that
     users who skipped a release or two are informed of the new one.
     Note that multi-steps incremental upgrade paths are valid and
     supported: e.g. when releasing 1.1.2, 1.1 users should still be
     able to incrementally upgrade to 1.1.1, and in turn to 1.1.2; to
     make this work, one must _not_ pass `--previous-version 1.1`,
     that would remove the existing incremental upgrade path from 1.1
     to 1.1.1.
   * If preparing a release candidate, add `--channel alpha`
   * If preparing a release candidate, drop all `--next-version`
     arguments, and instead pass (**untested!**)
     `--next-version $(echo ${VERSION:?} | sed -e 's,~rc.*$,,')`
   * If preparing a point-release, pass neither
     `--next-version "${VERSION:?}.1"`,
     nor `--next-version "${VERSION:?}.1~rc1"`

1. Create an armoured detached signature for each created or modified
   upgrade-description file.

       find "${RELEASE_CHECKOUT:?}/wiki/src/upgrade/" \
          -type f -name upgrades.yml | \
          while read udf; do
              if [ -n "$(git status --porcelain "${udf:?}")" ]; then
                  gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign "${udf:?}"
                  mv "${udf:?}.asc" "${udf:?}.pgp"
                  ( \
                    cd ${IUK_CHECKOUT:?} && \
                    ./bin/tails-iuk-check-upgrade-description-file "${udf:?}" \
                  ) || break
              fi
          done

1. Add and commit the upgrade-description files and their detached
   signatures to the Git branch used to prepare the release (`stable`
   or `testing`):

       ( \
         cd "${RELEASE_CHECKOUT:?}" && git add wiki/src/upgrade && \
         git commit -m "Update upgrade-description files." && \
         git push origin ${RELEASE_BRANCH:?} \
       )

1. If preparing a release candidate, move the generated or updated
   files to `${MASTER_CHECKOUT:?}`, commit and push: given the updates are
   advertised on the *alpha* channel, while all users use the *stable*
   one by default, this will allow you to more easily test the IUK
   without impacting anyone.

1. If preparing a point release, copy the generated UDF for the previous
   release in the alpha channel in the master branch, modify its content
   accordingly, sign it, commit and push

       # If more old versions are supported, add them (whitespace
       # separated) to this variable
       SUPPORTED_OLD_VERSIONS="${PREVIOUS_VERSION:?}"

       ( \
         cd ${MASTER_CHECKOUT:?} && \
         git fetch && \
         for old_version in ${SUPPORTED_OLD_VERSIONS:?}; do
           stable_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/i386/stable/upgrades.yml" && \
           alpha_udf="wiki/src/upgrade/v1/Tails/${old_version:?}/i386/alpha/upgrades.yml" && \
           git show origin/${RELEASE_BRANCH:?}:${stable_udf:?} \
             | sed -e 's/channel: stable/channel: alpha/' > ${alpha_udf:?} && \
           gpg -u "${TAILS_SIGNATURE_KEY:?}" --armor --detach-sign ${alpha_udf:?} && \
           mv ${alpha_udf:?}.asc ${alpha_udf:?}.pgp && \
           git add ${alpha_udf:?}* ; \
         done && \
         git commit -m "Add incremental upgrades on the alpha channel for Tails ${VERSION:?}" && \
         git push origin master:master \
       )


Prepare the ISO description file for DAVE
-----------------------------------------

If preparing a RC, skip this part.

Update the ISO description file (IDF) used by the browser extension:

    cat > "${RELEASE_CHECKOUT:?}"/wiki/src/install/v1/Tails/i386/stable/latest.yml <<EOF
    ---
    build-target: i386
    channel: stable
    product-name: Tails
    version: '${VERSION:?}'
    target-files:
    - sha256: ${ISO_SHA256SUM}
      size: ${ISO_SIZE_IN_BYTES:?}
      url: http://dl.amnesia.boum.org/tails/stable/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso
    EOF
    ( cd "${RELEASE_CHECKOUT:?}" && \
      git add wiki/src/install/v1/Tails/i386/stable/latest.yml && \
      git commit -m "Update IDF file for DAVE." )

Upload images
=============

Sanity check
------------

Verify once more that the TBB we ship is still the most recent (see
above).

## Announce, seed and test the Torrents

Announce and seed the Torrents.

Test them with a BitTorrent client running in a different place.

## Download and seed image from lizard

    scp "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
       bittorrent.lizard: && \
       ssh bittorrent.lizard \
         transmission-remote --add tails-i386-${VERSION:?}.torrent \
           --find /var/lib/transmission-daemon/downloads/

<a id="publish-iuk"></a>

Publish the ISO and IUK over HTTP
---------------------------------

Upload the images to the primary rsync mirror. Best practice is to first
let bittorrent.lizard download the image, and then copy it from there to
rsync.lizard:

    ssh lizard.tails.boum.org \
        scp -3 -r \
            bittorrent.lizard:/var/lib/transmission-daemon/downloads/tails-i386-${VERSION:?} \
            rsync.lizard:
    ssh rsync.lizard << EOF
      sudo chown -R root:rsync_tails \
         tails-i386-${VERSION:?} \
         Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk && \
      sudo chmod -R u=rwX,go=rX \
         tails-i386-${VERSION:?} \
         Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk && \
      sudo mv tails-i386-${VERSION:?} \
              /srv/rsync/tails/tails/${DIST:?}/ && \
      sudo mv Tails_i386_${PREVIOUS_VERSION:?}_to_${VERSION:?}.iuk \
              /srv/rsync/tails/tails/${DIST:?}/iuk/
    EOF

Update the time in `project/trace` file on the primary rsync mirror
and on the live wiki (even for a release candidate):

	TRACE_TIME=$(date +%s) &&
	echo ${TRACE_TIME:?} | ssh rsync.lizard "cat > /srv/rsync/tails/tails/project/trace" && \
	[ -n "${MASTER_CHECKOUT:?}" ] && \
	echo ${TRACE_TIME:?} > "${MASTER_CHECKOUT:?}/wiki/src/inc/trace" &&
	(
	   cd "${MASTER_CHECKOUT:?}" && \
	   git commit wiki/src/inc/trace \
	      -m "Updating trace file after uploading ${VERSION:?}." && \
	   git push origin master
	)

ISO history
-----------

Push the released ISO to our Tails ISO history git-annex repo, so that
our isotesters can fetch it from there for their testing. How to do so
is described in our internal Git repo.

Update the website and Git repository
=====================================

What follows in this section happens on the release branch in
`${RELEASE_CHECKOUT:?}`.

If preparing a final release
----------------------------

Skip this part if preparing a RC.

Rename the `.packages` file to remove the `.iso` and build date parts
of its name:

	mv "${ARTIFACTS:?}"/tails-i386-"${VERSION:?}".iso.packages \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.packages"

Rename the manifest of needed packages as well:

	mv "${PACKAGES_MANIFEST:?}" "${ARTIFACTS:?}/tails-i386-${VERSION:?}.build-manifest"

Copy the `.iso.sig`, `.build-manifest`, `.packages`, `.torrent` and
`.torrent.sig` files into the website repository:

	cp "${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso.sig" \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.build-manifest" \
	   "${ARTIFACTS:?}/tails-i386-${VERSION:?}.packages" \
	   "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
	   "${RELEASE_CHECKOUT:?}/wiki/src/torrents/files/"

Remove from `wiki/src/torrents/files/` any remaining file from the
previous release (including any RC).

Update the size of the ISO image in `inc/*`:

        LC_NUMERIC=C ls -l -h ${ISOS:?}/tails-i386-${VERSION:?}/tails-i386-${VERSION:?}.iso | \
          cut -f 5 -d ' ' | sed -r 's/(.+)([MG])/\1 \2iB/' \
          > "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_i386_iso_size.html"

Generate the expected signature verification output:

      gpg --keyid-format 0xlong --verify "${ISO_PATH:?}.sig" "${ISO_PATH:?}" 2>&1 | \
      sed 's/ /\&nbsp;/g;s/</\&lt;/;s/>/\&gt;/;s/$/<br\/>/g' > \
      "${RELEASE_CHECKOUT:?}/wiki/src/inc/stable_i386_gpg_signature_output.html"

Update the [[support/known_issues]] page:

- Add regressions brought by the new release.
- Remove older known issues that are fixed by the new release.

Write the announcement for the release in
`wiki/src/news/version_${TAG:?}.mdwn`. See the [[release notes
documentation|contribute/how/documentation/release_notes]]

XXX: we should probably merge that into the above liked documentation.
**Check you correctly:**

- Updated the `meta title` directive.
- Updated the `meta date` directive.
- Made sure there's an `announce` tag to have an email sent to the
  news mailing list.
- Documented important config changes that persistence users have to do
  themselves (e.g. the Pidgin proxy settings change in
  [[!tails_gitweb_commit 9925321]] breaks all existing persistent
  profiles).
- Documented known issues.

Write an announcement listing the security bugs affecting the previous
version in
`wiki/src/security/Numerous_security_holes_in_${PREVIOUS_VERSION:?}.mdwn`
in order to let the users of the old versions
know that they have to upgrade. Date it a few days before the ISO
image to be released was *built*. Including:

- if we are shipping Linux from testing/sid, the list of CVE fixed in
  Linux since the one shipped in the previous release of Tails:
  <http://metadata.ftp-master.debian.org/changelogs/main/l/linux/stable_changelog>
- the list of DSA fixed in packages we ship since those that were in
  the previous release of Tails: <https://www.debian.org/security/#DSAS>
- the list of BSA fixed in packages we ship since those that were in
  the previous release of Tails:
  <https://lists.debian.org/debian-backports-announce/>
- the list of MFSA fixed by the Tor Browser update:
  <https://www.mozilla.org/security/announce/>

If preparing a release candidate
--------------------------------

Skip this part if preparing a final release.

Copy the `.iso.sig` file into the website repository:

	cp "${ISO_PATH:?}.sig" \
	   "${ISOS:?}/tails-i386-${VERSION:?}.torrent" \
	      "${MASTER_CHECKOUT:?}/wiki/src/torrents/files/"

Write the announcement for the release in
`${MASTER_CHECKOUT:?}/wiki/src/news/test_${TAG:?}.mdwn`, including:

- Update the `meta title` directive.
- Update the `meta date` directive.
- Document important config changes that persistence users have to do
  themselves (e.g. the Pidgin proxy settings change in
  [[!tails_gitweb_commit 9925321]] breaks all existing persistent
  profiles).
- Document known issues.
- This snippet can help to convert the copied changelog's ticket
  references to links:

      sed -i 's@#\([0-9]\{4,5\}\)@[[!tails_ticket \1]]@g' \
          wiki/src/news/test_${TAG:?}.mdwn

In any case
-----------

Generate PO files for the announcements with `./build-website`.

Then, send them to <tails-l10n@boum.org> so that they get translated
shortly, perhaps even soon enough to integrate them before pushing the
release out officially.

Then, record the last commit before putting the release out for real:

	git add wiki/src && \
	git commit -m "releasing version ${VERSION:?}"

Testing
=======

1. Using `check-mirrors`, choose a fast mirror that already has the
   tentative ISO. E.g. <https://mirrors.kernel.org/tails/> or
   <https://mirrors.wikimedia.org/tails/> are reliable and have plenty
   of bandwidth.
1. Email <tails-testers@boum.org> to ask them to test the tentative
   ISO, pointing them to the up-to-date mirror you've found previously.
1. Set up a pad and copy the [[manual test
   suite|contribute/release_process/test]] in it.
1. Email <tails@boum.org> and potential contributors (see
   `manual_testers.mdwn` in the internal Git repository) that tests
   may start:
   - point them to the up-to-date mirror you've found previously
   - make it clear what's the deadline
   - make it clear where and how you expect to get feedback
   - attach the Torrent
   - attach the `.packages` file
1. Make sure someone is committed to run the automated test suite.
1. Make sure that enough people are here to run the tests, that they
   report their results in due time, and that they make it clear when
   they're leaving for good.
1. Fill the holes and make sure that the manual test suite is done in
   due time.
1. Triage test results, reproduce bugs as needed, decide what the next
   step is and make sure it happens: add to known issues? file ticket?
   release blocker?

Go wild!
========