release_process.mdwn

[[!meta title="Release process"]]

[[!toc levels=2]]

See the [[release_schedule]].

<div class="caution">
Read the remainder of this document from the branch used to prepare the release
after having merged the current master branch into it!
</div>

Requirements
============

Packages
--------

To release Tails you'll need some packages installed:

* `gitlab-cli jq tidy mktorrent transmission-cli`
* [[!debpts squashfs-tools]] 1:4.4-1+0.tails1
  from our custom `iukbuilder-stretch` APT suite.
* `iuk` [[dependencies|contribute/release_process/tails-iuk#build-deps]]
* `perl5lib` [[dependencies|contribute/release_process/perl5lib#build-deps]]
* `po4a` 0.55: different versions extract Markdown headings
   in a different way, which makes tons of strings fuzzy.
* packages to [[build a local version of the website|contribute/build/website/]]

Configuration files
-------------------

To release Tails you need:

* `~/.python-gitlab.cfg`

  You need at least this content:

        [global]
        ssl_verify = true

        [Tails]
        url = https://gitlab.tails.boum.org
        per_page = 100
        private_token = XXX

   [[!tails_gitlab profile/personal_access_tokens
   desc="Generate a _Personal Access Token_"]] with at least the `api` scope
   enabled, and set it as the value of the `private_token` option
   in your `~/.python-gitlab.cfg`.

Environment
===========

To be able to copy'n'paste the code snippets found on this page,
you need to set a bunch of environment variables.

Unless the release process explicitly instructs you to change the
value of one such variable, treat it as a constant: else,
inconsistency will surely arise, which can cause trouble later on.

Version numbers
---------------

Note:

* Regarding version numbers, what follows supports just fine the case
  when we do something else than alternating bugfix and major releases
  consistently. For example, if the next two releases are bugfix ones,
  do not set `$NEXT_PLANNED_MAJOR_VERSION` to one of these
  bugfix releases. Instead, set it to the version number of the next
  major release.
* The `$NEXT*VERSION` constants are used only for two types of
  operations: preparing upgrade-description files and adding changelog
  entries. This two types of operations have to be consistent with
  each other: for example, if one adds a dummy entry for version X in
  a changelog, an UDF must exist for version X as well… hence the use
  of shared constants to encode the values that must be the same on
  both sides :)

Export the following environment variables:

* version numbers (see [[contribute/release_schedule#versioning]]):

        export VERSION=$(dpkg-parsechangelog -SVersion)
        export TAG=$(echo "${VERSION:?}" | sed -e 's,~,-,')
        export PREVIOUS_VERSION=$(dpkg-parsechangelog --offset 1 --count 1 -SVersion)
        export PREVIOUS_TAG=$(echo "${PREVIOUS_VERSION:?}" | sed -e 's,~,-,')

* `NEXT_PLANNED_MAJOR_VERSION`: set to the version number of the next
  *major* Tails release; if you're preparing a RC for a major release,
  use that major release; otherwise, use whatever the next planned
  major release is
* `SECOND_NEXT_PLANNED_MAJOR_VERSION`: if you're preparing the RC
  for a major release, set this to the version number of
  the second next *major* Tails release; e.g. if preparing the RC for
  the 3.9 major release, then set this to 3.12 (3.9 is the next major
  release, 3.10 and 3.11 are bugfix releases, 3.12 is a major
  release).
* `NEXT_PLANNED_BUGFIX_VERSION`: set to the version number of the next
  scheduled *bugfix* Tails release
* `NEXT_POTENTIAL_EMERGENCY_VERSION`: set to the version number we'll give
  to the next emergency release if we have to put one out; unset for a
  release candidate
* `NEXT_STABLE_CHANGELOG_VERSION`: if `$NEXT_PLANNED_BUGFIX_VERSION` is the next
  scheduled release, use it; otherwise, use `$NEXT_POTENTIAL_EMERGENCY_VERSION`

Other variables
---------------

Also export the following environment variables:

* `MAJOR_RELEASE`: set to 1 if preparing a major release or a release
  candidate for a major release, to 0 otherwise
* `ISOS`: the directory where one stores `tails-amd64-*`
  sub-directories like the ones downloaded with BitTorrent.
* `ARTIFACTS`: the directory where build artifacts (e.g.
  the `.packages` file) land.
* `MASTER_CHECKOUT`: a checkout of the `master` branch of the main
  Tails Git repository.
* `RELEASE_BRANCH=$(if [ "$MAJOR_RELEASE" = 1 ]; then echo -n testing; else echo -n stable; fi)`
* `RELEASE_CHECKOUT`: a checkout of the branch of the main Tails Git
  repository used to prepare the release (`stable` or `testing`).
* `TAILS_SIGNATURE_KEY=A490D0F4D311A4153E2BB7CADBB802B258ACD84F`
* `TAILS_SIGNATURE_KEY_LONG_ID=$(echo "${TAILS_SIGNATURE_KEY:?}" | perl -nE 'say substr($_, -17)')`
* `DIST`: either 'alpha' (for RC:s) or 'stable' (for actual releases)
* `export WEBSITE_RELEASE_BRANCH="web/release-${TAG:?}"`
* `export IUKS_DIR="${ISOS:?}/iuks/v2"`
* `export IUKS_HASHES="${IUKS_DIR:?}/to_${VERSION}.sha256sum"`

Pre-freeze
==========

The [[contribute/working_together/roles/release_manager]] role
documentation has more tasks that should be done early enough.

Coordinate with Debian security updates
---------------------------------------

See [[release_process/Debian_security_updates]].

Sanity check
============

Visit the [Jenkins RM view](https://jenkins.tails.boum.org/view/RM/)
and check that the jobs for the release branch have good enough results.

Freeze
======

Major release
-------------

If we are at freeze time for a major release (i.e. preparing its
release candidate):

1. Merge the `master` Git branch into `devel`:

        git checkout devel && git fetch origin && git merge origin/devel && git merge --no-ff origin/master

2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `devel` branch's `config/APT_overlays.d/` into the `devel`
   APT suite; set `BRANCH=testing` instead of the default `BRANCH=devel`.

3. Merge the `devel` Git branch into the `testing` one:

        git checkout testing && git merge origin/testing && git merge devel

   ... and check that the resulting `config/APT_overlays.d/` in the
   `testing` branch is empty.

4. [[Hard reset|APT_repository/custom#workflow-reset]] the `testing`
   custom APT suite to the current state of the `devel` one.

5. [[Freeze|APT_repository/time-based_snapshots#freeze]] the
   time-based APT repository snapshots that shall be used
   during the freeze.

6. Make it so the time-based APT repository snapshots are kept around
   long enough, by bumping their `Valid-Until` to 10 days after the
   second next major release (the one _after_ the one you're preparing)'s
   scheduled date:
   [[APT_repository/time-based_snapshots#bump-expiration-date-for-all-snapshots]]


Bugfix release
--------------

If we are at freeze time for a bugfix release:

1. Merge the `master` Git branch into `stable`:

        git checkout stable && git fetch && git merge origin/stable && git merge --no-ff origin/master

2. [[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
   listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
   APT suite.

Common steps for bugfix and major releases
------------------------------------------

Reset the release branch's `config/base_branch`:

        echo "${RELEASE_BRANCH:?}" > config/base_branch && \
           git commit config/base_branch \
               -m "Restore ${RELEASE_BRANCH:?}'s base branch."

Bootstrap manual testing coordination:

1. Create a pad.
2. Copy the [[manual test suite|contribute/release_process/test]]
   into it.
3. Send the pad URL in an **encrypted** mail to the manual testers:
   <tails-manual-testers@boum.org>

Update included files
=====================

<a id="upgrade-custom-debs"></a>

Upgrade Tor Browser
-------------------

See the dedicated page: [[tor-browser]]

Upgrade Tor Browser AppArmor profile
------------------------------------

See the dedicated page: [[tor-browser_AppArmor_patch]]

Upgrade bundled binary Debian packages
--------------------------------------

Skip this section if you are preparing a bugfix release.

The goal here is to make sure the bundled binary Debian packages contain
up-to-date localization files, so:

 - If you are preparing a release candidate, build at least the packages
   that change user-visible strings, so that translators can use the RC
   to check the status of their work and identify what's left to do.
 - If you are preparing a final major release, build at least the packages
   that got translation updates since the RC: we've sent a call for
   translation while releasing the RC so the least we can do is to
   incorporate the work that ensued into our final release :)

For each bundled Debian package, `cd` into the package's root
directory (e.g. a checkout of the `whisperback` repository),
import translations from Transifex and sanity-check them:

	cd whisperback && \
	git checkout master && \
	git pull && \
	"${RELEASE_CHECKOUT:?}"/import-translations && \
	"${RELEASE_CHECKOUT:?}"/submodules/jenkins-tools/slaves/lint_po

Then, for every PO file that has issues:

1. Rollback changes to that file: `git checkout po/LL.po`
2. Run `lint_po` again. It should pass this time.

And finally, commit:

    git add po && git commit \
	    -m "Update POT and PO files, pull updated translations from Transifex."

Then see the relevant release processes, and upload the packages to
the release branch's custom APT suite:

* [[tails-installer]]
* whisperback:
  * follow [upstream release process](https://gitlab.tails.boum.org/tails/whisperback/-/blob/master/HACKING.md#release)
  * build a Debian package in an amd64 chroot of the Debian release
    the Tails version you're preparing is based on

Upgrade custom packages for VeraCrypt integration
-------------------------------------------------

See the dedicated page: [[veracrypt]]

Update PO files
---------------

Pull updated translations for languages translated in Transifex,
refresh the code PO files,
and commit the result, including new PO files:

	cd "${RELEASE_CHECKOUT:?}" && \
	./import-translations  && \
	./refresh-translations && \
	./submodules/jenkins-tools/slaves/lint_po && \
	git add po && git commit -m 'Update PO files.'

If `lint_po` complains:

* rollback the offending PO files and retry; worst case, delete it
* send a note to <tails-l10n@boum.org> [public] so that they get in touch with
  whoever can fix them.

When preparing an actual release
================================

If we're about to prepare the images for a final (non-RC) release, then
follow these instructions:

Major release
-------------

[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `testing` branch's `config/APT_overlays.d/` into the `testing`
custom APT suite.

Bugfix release
--------------

<div class="note">
For bugfix releases, we generally do not put any RC out, so freeze time
is the same as preparing the actual release. Hence, the following
steps have already been done above, and this section is a noop in the
general case.
</div>

[[Merge each APT overlay suite|APT_repository/custom#workflow-merge-overlays]]
listed in the `stable` branch's `config/APT_overlays.d/` into the `stable`
custom APT suite.

Update other base branches
==========================

1. Merge the release branch into `devel` following the instructions for
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].

2. [[Thaw|APT_repository/time-based_snapshots#thaw]], on the devel
   branch, the time-based APT repository snapshots being used
   during the freeze. It's fine if that results in a no-op
   (it depends on how exactly previous operations were performed).

3. Merge `devel` into `feature/bullseye` (if it exists), *without* following the instructions for
   [[merging base branches|APT_repository/custom#workflow-merge-main-branch]].
   (For now `feature/bullseye` is handled as any other topic branch
   forked off `devel`: its base branch is set to `devel`.)
   If the merge conflicts don't look like something you feel confident
   resolving properly, abort this merge and let the Foundations
   Team know.

4. Ensure that the release, `devel` and `feature/bullseye` (if it exists) branches
   have the expected content in `config/APT_overlays.d/`: e.g. it must
   not list any overlay APT suite that has been merged already.

5. Push the modified branches to Git:

        git push origin                          \
           "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}" \
           $(if git describe feature/bullseye >/dev/null 2>&1; then echo feature/bullseye:feature/bullseye ; fi) \
           devel:devel

Update more included files
==========================

Changelog
---------

Make sure to switch back to the release branch, spawn an editor to
remove the placeholder entry for then next release in `debian/changelog`,
and gather data from the Git repository:

	git checkout "${RELEASE_BRANCH:?}" && \
	dch -e && \
	DEBEMAIL='tails@boum.org' DEBFULLNAME='Tails developers' \
	./release ${VERSION:?} ${PREVIOUS_TAG:?}

This populates the Changelog with the Git log entries.

Then, launch an editor for the needed cleanup of the result:

	dch -e

Changelog entries can be dispatched into those usual sections:

<pre>
  * Major changes

  * Security fixes

  * Bugfixes

  * Minor improvements and updates

  * Build system

  * Test suite
</pre>

Then, gather other useful information from:

* every custom bundled package's own Changelog (Tails Installer, etc.)
* the diff between the previous version's `.packages` file and the one
  from the to-be-released images; look for:
  - security fixes
  - new upstream releases of applications mentioned in [[doc/about/features]]
  - new upstream releases of other important components such as the
    Linux kernel
* the [[!tails_gitlab groups/tails/-/milestones desc="GitLab milestone"]].

Finally, sanity check the version and commit:

	if [ "$(dpkg-parsechangelog -SVersion)" = "${VERSION:?}" ]; then
	    git commit debian/changelog -m "Update changelog for ${VERSION:?}."
	else
	    echo 'Error: version mismatch: please compare ${VERSION:?} with the last entry in debian/changelog'
	fi

Included website
----------------

### Merge master

Merge `master` into the branch used for the release:

	git fetch origin && git merge origin/master

### version number

If preparing a RC, skip this part.

In the branch used to build the release, update the `wiki/src/inc/*` files to
match the *version number* and *date* of the new release. Set the date
at least 24 hours in the future! Between tests and mirror synchronization,
the build will not be released on the same day. Try to make sure it
matches the date of the future signature.

	RELEASE_DATE='2015-11-03'

	echo "${VERSION:?}"      > wiki/src/inc/stable_amd64_version.html && \
	echo -n "${RELEASE_DATE:?}" > wiki/src/inc/stable_amd64_date.html && \
	for type in img iso; do
	   basename="tails-amd64-${VERSION:?}"
	   filename="${basename:?}.${type:?}"
	   echo "TZ=UTC gpg --no-options --keyid-format long --verify ${filename:?}.sig ${filename:?}" \
	        > wiki/src/inc/stable_amd64_${type:?}_gpg_verify.html && \
	   echo "http://dl.amnesia.boum.org/tails/stable/${basename:?}/${filename:?}" \
	        > wiki/src/inc/stable_amd64_${type:?}_url.html && \
	   echo "https://tails.boum.org/torrents/files/${filename:?}.sig" \
	        > wiki/src/inc/stable_amd64_${type:?}_sig_url.html && \
	   echo "https://tails.boum.org/torrents/files/${filename:?}.torrent" \
	        > wiki/src/inc/stable_amd64_${type:?}_torrent_url.html
	done && \
	./build-website --rebuild && \
	git commit wiki/src/inc/ -m "Update version and date for ${VERSION:?}."

Signing key downloaded by the Upgrader
--------------------------------------

    TMP_GNUPG_HOME=$(mktemp -d)
    gpg --homedir "${TMP_GNUPG_HOME:?}" --import wiki/src/tails-signing.key && \
    gpg --homedir "${TMP_GNUPG_HOME:?}" --export-options export-minimal \
        --armor --export "${TAILS_SIGNATURE_KEY:?}" \
        > wiki/src/tails-signing-minimal.key && \
    git commit wiki/src/tails-signing-minimal.key \
        -m "Update signing key used by the Upgrader"
    rm -rf "${TMP_GNUPG_HOME:?}"

Website translations
--------------------

Refresh the website PO files and commit the ones corresponding to
pages that were added or changed accordingly to changes coming with
the new release. This e.g. ensures that the RC call for translation
points translators to up-to-date PO files:

    ./build-website && \
    git add wiki/src && \
    git commit -m 'Update website PO files.'
    git push origin "${RELEASE_BRANCH:?}:${RELEASE_BRANCH:?}"

Call for translation
====================

If at freeze time for a major release, send a call for translations to
<tails-l10n@boum.org> [public], making it clear what Git branch the
translations must be based on, and what are the priorities.

To get a list of changes on the website:

    git diff --stat ${PREVIOUS_TAG:?}.. -- \
        wiki/src/'*'.{mdwn,html} \
        ':!wiki/src/blueprint*' \
        ':!wiki/src/contribute*' \
        ':!wiki/src/inc' \
        ':!wiki/src/news*' \
        ':!wiki/src/security*'

Enable OpenPGP signing
======================

### If you have an OpenPGP smart card

If you have an OpenPGP smart card (i.e. if you are one of the usual
release managers) go fetch it. Remember to only plug it when needed! A
pro tip is to never plug it unless prompted which `gpg` will do for you.

### Otherwise: importing the signing key

This is only relevant when the master key has been reassembled,
e.g. for signing a Tails emergency release where none of the usual
release managers are available.

You should never import the Tails signing key into your own keyring,
and a good practice is to import it to a tmpfs to limit the risks that
the private key material is written to disk:

    export GNUPGHOME=$(mktemp -d)
    sudo mount -t ramfs ramfs "${GNUPGHOME:?}"
    sudo chown $(id -u):$(id -g) "${GNUPGHOME:?}"
    sudo chmod 0700 "${GNUPGHOME:?}"
    gpg --homedir ${HOME:?}/.gnupg --export ${TAILS_SIGNATURE_KEY:?} | gpg --import
    gpg --import path/to/private-key

Let's also ensure that strong digest algorithms are used for our
signatures, like the defaults we set in Tails:

    cp config/chroot_local-includes/etc/skel/.gnupg/gpg.conf "${GNUPGHOME:?}"

Build the almost-final images
=============================

1. [[Build ISO and USB images|contribute/build]] from the release branch,
   with `$TAILS_BUILD_OPTIONS` set like this:
   - Set `defaultcomp`, so we can more accurately optimize our
     SquashFS file ordering.
   - Do _not_ set `keeprunning` nor `rescue`.
2. Carefully read the build logs to make sure nothing bad happened.
3. Keep the resulting build artifacts until the end of this release process.
4. Record where the manifest of needed packages is stored:

        export BUILD_MANIFEST=XXX ; \
        [ -f "${BUILD_MANIFEST:?}" ] || echo "ERROR: BUILD_MANIFEST is incorrect"
        echo "${BUILD_MANIFEST:?}" | grep -E -qs '\.build-manifest$' \
           || echo "ERROR: BUILD_MANIFEST does not have the .build-manifest extension"

Tag the release in Git
======================

	git tag -u "${TAILS_SIGNATURE_KEY:?}" \
	  -m "tagging version ${VERSION:?}" "${TAG:?}" && \
	git push origin "${TAG:?}" "${RELEASE_BRANCH:?}"

(Pushing the tag is needed so that the APT repository is updated, and
the Tails APT configuration works at build and boot time. It might be
premature, as testing might reveal critical issues, but this is
a signed tag, so it can be overridden later. Yes, there is room for
improvement here.)

XXX: From this push of a tag, the builds in Jenkins fail because we prevent it
to continue if the last changelog entry is unreleased but corresponds to
an existing tag. There are workarounds we need to decide and implement.

Prepare the versioned APT suites
================================

* [[Prepare the versioned APT suite in our custom APT repository|APT_repository/custom#workflow-post-tag]].

* Prepare tagged snapshots of upstream APT repositories:

          ./bin/tag-apt-snapshots "${BUILD_MANIFEST:?}" "${TAG:?}"

  Note:

  - This command can take a while (about a dozen minutes).
  - It's expected that the packages that were pulled from our
    [[custom APT repository|APT_repository/custom]] are
    listed under "some packages were not found anywhere" (because we
    are currently not using time-based snapshots for our custom APT
    repository). However, _no other package should be on that list_.
    Now, we have a "safety" net, in case you don't notice such a problem: if
    other packages are missing, the next build (that will use the
    newly created partial, tagged APT repository) will fail.

Build images
============

Sanity check
------------

Verify that the Tor Browser release used in Tails still is the most
recent. Also look if there's a new `-buildX` tag (e.g.
`tor-browser-60.3.0esr-8.0-1-build1`) for the Firefox version the Tor
Browser we want to ship is based on in these Git repositories:

* <https://gitweb.torproject.org/builders/tor-browser-build.git>
* <https://gitweb.torproject.org/tor-browser.git>

A new tag may indicate that a new Tor Browser release or rebuild is imminent.

Better catch this before people spend time doing manual tests.

SquashFS file order
-------------------

1. Install the almost-final USB image to a USB stick.
1. Boot this USB stick a first time to trigger re-partitioning.
1. Shut down this Tails.
1. Boot this USB stick **on bare metal** again.
1. Add `profile` to the kernel command-line.
1. Login with the default settings in the Welcome Screen (e.g. do not configure
   an _Administration Password_).
1. Wait for the "Tor is ready" notification.
1. Start *Tor Browser*.
1. A few minutes later, once the `boot-profile` process has been
   killed, retrieve the new sort file from `/var/log/boot-profile`.
1. Backup the old sort file: `cp config/binary_rootfs/squashfs.sort{,.old}`
1. Copy the new sort file to `config/binary_rootfs/squashfs.sort`.
1. Remove runtime-generated files that don't exist in the rootfs,
   in order to avoid confusing noise in the build output:

           perl -ni -E 'chomp; say unless m{(?:
                  [.]pyc\s+\d+\z
               | \Alib/live/mount/medium/live/(?:filesystem[.]squashfs|initrd[.]img)\s
               | \Alib/live/mount/overlay/rw/etc/fstab\s
               | \Alib/live/mount/overlay/rw/etc/console-setup/cached_\S+[.](?:gz|sh)\s
               | \Alib/live/mount/overlay/rw/etc/machine-id\s
               | \Alib/live/mount/overlay/rw/etc/network/interfaces\s
               | \Alib/live/mount/overlay/rw/var/log/wtmp\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/apparmor[.]d/cache/[.]features\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/(?:group|gshadow|passwd|shadow)-\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/resolv-over-clearnet[.]conf\s
               | \A(?:lib/live/mount/overlay/rw/)?etc/skel/[.]config/autostart/end-profile[.]desktop\s
               | \Arun/
               | \Avar/lib/AccountsService/users/Debian-gdm\s
               | \Avar/lib/gdm3/[#]\d+\s
               | \Avar/log/live/config[.]pipe\s
           )}xms' config/binary_rootfs/squashfs.sort

1. Remove the bits about `kill-boot-profile` at the end: they're
   only useful when profiling the boot.
1. Inspect the Git diff (including diff stat), apply common sense:

        diff -NaurB \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort.old | sort ) \
            <( cut -d' ' -f1 config/binary_rootfs/squashfs.sort     | sort ) \
            | less

1. `git commit -m 'Updating SquashFS sort file' config/binary_rootfs/squashfs.sort`
1. Clean up: `rm -f config/binary_rootfs/squashfs.sort.old`

Build the final images
----------------------

Then all included files should be up-to-date and the versioned APT
suite should be ready, so it is time to:

1. Mark the version as "released" in the changelog:

        dch --release --no-force-save-on-release --maintmaint && \
        git commit -m "Mark Tails ${VERSION:?} as released." debian/changelog

1. Export `SOURCE_DATE_EPOCH`:

        export SOURCE_DATE_EPOCH=$(date --utc --date="$(dpkg-parsechangelog --show-field=Date)" '+%s')

1. tag the release *again*, with all included files in:

        git tag -f -u "${TAILS_SIGNATURE_KEY:?}" \
                -m "tagging version ${VERSION:?}" "${TAG:?}" && \
        git push --force origin "${TAG:?}" && \
        git push origin "${RELEASE_BRANCH:?}"

   Note: for Jenkins to build the release you must push the release
   branch with its tip tagged. I.e. if you deviate from the above
   commands by e.g. committing a commit in between `git tag` and the
   first `git push` then Jenkins won't build from the tag -- please
   avoid that!

1. build the final images!
   Do _not_ set `keeprunning` nor `rescue` in `$TAILS_BUILD_OPTIONS`.
   Our build system will apply the correct compression settings automatically
   so don't bother setting it yourself.

1. Make sure the Jenkins build starts. Until the hook is back in place
   ([[!tails_ticket 17745]]), starting it manually may avoid up to 15
   minutes of waiting.

1. Compare the new build manifest with the one from the previous,
   almost-final build:

        diff -Naur \
           "${BUILD_MANIFEST:?}" \
           "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"

   They should be identical, except that the `debian-security` serial might be higher.

1. To ensure we publish the final build's `.build-manifest`, run:

        export BUILD_MANIFEST="${ARTIFACTS:?}/tails-amd64-${VERSION:?}.build-manifest"

<a id="reproducibility-sanity-check-iso"></a>

Verify that Jenkins reproduced your images
------------------------------------------

to verify that Jenkins reproduced your images:

1. Visit the URL printed by this command:

       echo "https://jenkins.tails.boum.org/job/build_Tails_ISO_${RELEASE_BRANCH}/"

2. Find the job (probably the last one)
   and make sure the ISO and USB images built by Jenkins
   have the same hash (in the `.shasum` file) as the images you built.

3. Then:

   - If the ISO and USB images hashes match: yay, we're good to go!
     The `.build-manifest` may differ — that's OK.

     Set the `$MATCHING_JENKINS_IMAGES_BUILD_ID` environment variable
     to the ID of this job (an integer).

   - If there is a hash mismatch for one of the images: ouch! Now we are in a
     tricky situation: on the one hand it seems like a poor idea to
     block users from benefiting from this release's security updates,
     but on the other hand the failure might imply that something
     nefarious is going on. At this stage, no matter what, immediately
     fetch Jenkins' image, compare it with your, and try to rule out
     build system compromise:

          sudo diffoscope \
              --text diffoscope.txt \
              --html diffoscope.html \
              --max-report-size 262144000 \
              --max-diff-block-lines 10000 \
              --max-diff-input-lines 10000000 \
                  path/to/your/tails-amd64-${VERSION:?}.iso \
                  path/to/jenkins/tails-amd64-${VERSION:?}.iso

     Do the same for the USB image as well.

     Then carefully investigate the `diffoscope` report:

       - If you cannot rule out that the difference is harmful: let's take
         a step back; we might be compromised, so we are in no position to
         release. Halt the release, involve the rest of <tails@boum.org>, and then
         try to re-establish trust in all build machines and infra
         involved, etc. Have fun!

       - Otherwise, if the change is definitely harmless:

         * If the source of non-determinism is identified quickly and
           is easy and fast to fix, *and* the QA of the current images
           has not gone very far (so at least that time is not wasted),
           then you should consider abandoning the current version, and
           immediately start preparing an emergency release with:

           - the reproducibility fix,
           - a new changelog entry,
           - adjustments to the release notes so they are re-purposed for
             this emergency release (the abandoned release gets none, since
             it effectively never will be released publicly).

         * Otherwise, if the fix looks time-consuming or difficult,
           let's release anyway. But let's add a known issue about
           "This Tails release does not build reproducibility" to the
           release notes, linking to the issue where
           the nature of the reproducibility failure is clearly
           described.

Initialize the website release branch
-------------------------------------

From now on, we don't want to push new commits on `$RELEASE_BRANCH`
until the new release is out. Otherwise, this would break its build
and the build of every branch based on it, which would effectively
block other development work. So the final steps towards publishing
the release will be done in a new, dedicated branch.

If preparing anything but a final release (e.g. an alpha, beta
or RC):

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" origin/master && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

Else, if preparing a final release:

        git checkout -b "${WEBSITE_RELEASE_BRANCH:?}" "${TAG:?}" && \
        git push -u origin "${WEBSITE_RELEASE_BRANCH:?}"

Generate the OpenPGP signatures and Torrents
============================================

Create a directory with a suitable name, go there, move the built
images to this brand new directory, generate detached OpenPGP
signatures for the images to be published (in the same directory as the
images and with a `.sig` extension), then go up to the parent
directory, create a `.torrent` file and check the generated `.torrent`
files metadata:

    mkdir -p "${ISOS:?}/tails-amd64-${VERSION:?}" && \
    for type in iso img ; do
       cd "${ISOS:?}/tails-amd64-${VERSION:?}" && \
       mv "${ARTIFACTS:?}/tails-amd64-${VERSION:?}.${type:?}" . && \
       gpg --armor --default-key "${TAILS_SIGNATURE_KEY:?}" --detach-sign *".${type:?}" && \
       rename 's,\.asc$,.sig,' *.asc && \
       tmp="$(mktemp -d)" && \
       mkdir -p "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       cd "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       for x in "${ISOS:?}/tails-amd64-${VERSION:?}"/*.${type:?}*; do
           ln -s ${x} .
       done && \
       mktorrent \
          -o "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" \
          -a 'udp://tracker.torrent.eu.org:451'   \
          -a 'udp://tracker.coppersurfer.tk:6969' \
          "${tmp:?}/tails-amd64-${VERSION:?}-${type:?}" && \
       transmission-show "${ISOS:?}/tails-amd64-${VERSION:?}.${type:?}.torrent" && \
       cd - && \
       rm -rf "${tmp:?}"
    done

Due to various directory changes, one needs to manually go back to the
desired directory, likely using:

    cd ${RELEASE_CHECKOUT?:}

Lastly, let's set some variables to be used later:

    ISO_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.iso"
    ISO_SHA256SUM="$(sha256sum "${ISO_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    ISO_SIZE_IN_BYTES="$(stat -c %s "${ISO_PATH:?}")"
    IMG_PATH="${ISOS:?}/tails-amd64-${VERSION:?}/tails-amd64-${VERSION:?}.img"
    IMG_SHA256SUM="$(sha256sum "${IMG_PATH:?}" | cut -f 1 -d ' ' | tr -d '\n')"
    IMG_SIZE_IN_BYTES="$(stat -c %s "${IMG_PATH:?}")"

Push images to ISO history
==========================

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

 - The Jenkins `parallel_build_IUKs` job can fetch them.
 - Our isotesters can fetch them from there for their testing.

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

If the release manager has `$ISOS` pointing to a checkout of the ISO
history repository, the following commands might be useful. For the
record, images and their signatures are already in that directory, and
one just needs to ship some metadata along:

    (cd ${ISOS?:}/tails-amd64-${VERSION?:} && \
    cp "${ARTIFACTS:?}"/tails-amd64-${VERSION:?}{.buildlog,.build-manifest,.packages} . && \
    git annex sync && \
    git annex add . && \
    git commit -m "Add Tails ${VERSION?:} images and metadata." && \
    git annex sync && \
    git annex copy --to origin .)

This transfer can take a while, and it's possible to proceed with the
next section in the meanwhile, up to the following step (included):

 - Build the Incremental Upgrade Kits locally

However, the following step will require those files being transferred
and made available on the relevant web server:

 - Build the Incremental Upgrade Kits on Jenkins

<a id="prepare-iuk"></a>

Prepare incremental upgrades
============================

Since Tails 4.2, we use a new upgrade scheme, which fundamentally
changes what the source version number of an upgrade means: it's now
the version that was *initially installed* and *not* the currently
running version. If this is news to you, see:

* the document that explains the benefits for our users:
  [[blueprint/Endless_upgrades]];

* the corresponding
  [[design documentation|contribute/design/incremental_upgrades]].

The main practical implications at release time are:

* The Release Manager has to publish more IUKs than they used to.
  But they can now publish IUKs (reproducibly) built on Jenkins,
  instead of having to upload those they've built locally.

* The Release Manager has to sign more UDFs than they used to.

* Computing `$IUK_SOURCE_VERSIONS` is now straightforward enough
  that it was automated :)

Prepare the environment
-----------------------

Compute the list of initial version install to build IUKs for:

    cd "${RELEASE_CHECKOUT:?}" && \
	export IUK_SOURCE_VERSIONS=$(./bin/iuk-source-versions ${VERSION:?})
    echo "${IUK_SOURCE_VERSIONS?:}"

Even if it's computed automatically, remember to store it in the file
holding environment variables, it will be used in various places below.

Sanity checks
-------------

Check that you have the correct version of `squashfs-tools` installed:

    [ "$(dpkg-query --showformat '${Version}\n' --show squashfs-tools)" \
      = '1:4.4-1+0.tails1' \
    ] || echo 'ERROR! Your squashfs-tools is not the required version, so any generated IUKs will *not* be reproducible!'

Check that you have the password for <https://iso-history.tails.boum.org> (it
will be used by `wrap_tails_create_iuks` below, in order to download any ISO
that is not present locally yet):

    if ! grep -F --line-regexp -qs 'machine iso-history.tails.boum.org' ~/.netrc; then
        echo "ERROR! Add a section for iso-history.tails.boum.org' to your ~/.netrc"
        echo "The corresponding login and password are in the RMs' keyringer."
    fi

Build the Incremental Upgrade Kits locally
------------------------------------------

You're encouraged to enable parallelism to avoid waiting for a very
long, serial build (which is still the default at the moment). As
discussed in [[!tails_ticket 17657]], it seems running as many jobs as
there are physical cores is a nice rule of thumb.

For example, set:

    JOBS="--jobs 4"

before starting the wrapper from `puppet-tails`:

    (
       set -eu
       WORK_DIR=$(mktemp -d)
       TAILS_REMOTE="$(git -C "${RELEASE_CHECKOUT?:}" remote get-url origin)"
       PUPPET_TAILS_REMOTE=$(echo -n "${TAILS_REMOTE?:}" | perl -p -E 's,:tails(:?[.]git)?\z,:puppet-tails,')
       cd "${WORK_DIR?:}"
       git clone "$PUPPET_TAILS_REMOTE"
       sudo -l
       ./puppet-tails/files/jenkins/slaves/isobuilders/wrap_tails_create_iuks \
           --tails-git-remote "file://${RELEASE_CHECKOUT?:}/.git"             \
           --tails-git-commit "${TAG?:}"                                      \
           --source-date-epoch "${SOURCE_DATE_EPOCH?:}"                       \
           --local-isos-dir "${ISOS?:}"                                       \
           --tmp-dir "${TMPDIR:-/tmp}"                                        \
           --output-dir "${IUKS_DIR?:}"                                       \
           --source-versions "${IUK_SOURCE_VERSIONS?:}"                       \
           --new-version "${VERSION?:}"                                       \
           --verbose ${JOBS:-}
       cd "${IUKS_DIR?:}"
       sha256sum Tails_amd64_*_to_${VERSION?:}.iuk > "${IUKS_HASHES?:}"
    )

This command takes a long time. In parallel, while it is running,
you can follow the next step:

 - Build the Incremental Upgrade Kits on Jenkins

ISO history
-----------

Push the released ISO and USB images and their artifacts (`.buildlog`,
`.build-manifest`, and `.packages` files) to our Tails ISO history git-annex
repo, so that:

 - The Jenkins `parallel_build_IUKs` job can fetch them.
 - Our isotesters can fetch them from there for their testing.

How to do so is described in the `ISO_history.mdwn` document in the RM team's
Git repo.

Then, wait (a few minutes, */15 crontab) until the images appear
on <https://iso-history.tails.boum.org/>.

<a id="build-iuks-on-jenkins"></a>

Build the Incremental Upgrade Kits on Jenkins
---------------------------------------------

1. Make sure the push to ISO history (started in the previous section)
   has finished, and that images have appeared on the web server:
   <https://iso-history.tails.boum.org/>