Commit e0127968 authored by intrigeri's avatar intrigeri
Browse files

Merge remote-tracking branch 'origin/master' into stable

parents 8de77a32 13d5ab61
......@@ -141,7 +141,7 @@ gitmaster_branch: master
# htmlscrubber plugin
# PageSpec specifying pages not to scrub
#htmlscrubber_skip: '!*/Discussion'
htmlscrubber_skip: 'download or download.* or install or install.* or install/* or upgrade or upgrade.* or upgrade/*'
# inline plugin
# enable rss feeds by default?
......@@ -225,7 +225,7 @@ po_slave_languages:
- fr|Français
- pt|Português
# PageSpec controlling which pages are translatable
po_translatable_pages: '!security/audits and !security/audits/* and (about or bugs or chat or contribute or contribute/how/donate or doc or doc/* or download or getting_started or inc/release_notes/* or inc/stable_i386_date or inc/stable_i386_release_notes or index or news or news/* or press or press/* or security or security/* or sidebar or support or support/* or todo or torrents or wishlist or misc or misc/*)'
po_translatable_pages: '!security/audits and !security/audits/* and (about or bugs or chat or contribute or contribute/how/donate or doc or doc/* or download or download.inline or getting_started or inc/release_notes/* or inc/stable_i386_date or inc/stable_i386_release_notes or index or news or news/* or press or press/* or security or security/* or sidebar or support or support/* or todo or torrents or wishlist or misc or misc/* or install or install/* or upgrade or upgrade/*)'
# internal linking behavior (default/current/negotiated)
po_link_to: current
......
......@@ -118,7 +118,7 @@ allow_symlinks_before_srcdir: 1
# htmlscrubber plugin
# PageSpec specifying pages not to scrub
htmlscrubber_skip: '!/misc/unsafe_browser_warning'
htmlscrubber_skip: 'misc/unsafe_browser_warning or misc/unsafe_browser_warning.* or download or download.* or install or install.* or install/* or upgrade or upgrade.* or upgrade/*'
# inline plugin
# enable rss feeds by default?
......@@ -202,7 +202,7 @@ po_slave_languages:
- fr|Français
- pt|Português
# PageSpec controlling which pages are translatable
po_translatable_pages: '!security/audits and !security/audits/* and (about or bugs or chat or contribute or contribute/how/donate or doc or doc/* or download or getting_started or inc/release_notes/* or inc/stable_i386_date or inc/stable_i386_release_notes or index or news or news/* or press or press/* or security or security/* or sidebar or support or support/* or todo or torrents or wishlist or misc or misc/*)'
po_translatable_pages: '!security/audits and !security/audits/* and (about or bugs or chat or contribute or contribute/how/donate or doc or doc/* or download or download.inline or getting_started or inc/release_notes/* or inc/stable_i386_date or inc/stable_i386_release_notes or index or news or news/* or press or press/* or security or security/* or sidebar or support or support/* or todo or torrents or wishlist or misc or misc/* or install or install/* or upgrade or upgrade/*)'
# internal linking behavior (default/current/negotiated)
po_link_to: current
......
......@@ -19,23 +19,283 @@ ticket #6938 can be seen on <https://labs.riseup.net/code/issues/6938>.
# A. Replace Claws Mail with Icedove
## A.n. description of subsection
- A.1.3. Integrate Icedove into Tails
- A.n.m. description of deliverable: ticket numbers
We configured Icedove in Tails to match our security goals (#6151),
as part of this we disabled:
status summary:
- The remote email account feature (#10009)
- The automatic add-on updates (#10428)
* what was done
* what is the outcome (how it makes Tails better)
* what was not done, and why
We adapted Torbirdy to suit our needs in Tails. (#10007)
During this work we discovered a bug in Torbirdy sometimes
choosing a wrong language, which we patched upstream. Once our
patch was merged, we updated the Torbirdy Debian package to a new
version which we could also include in Tails. (#9821)
- A.1.4. Provide a migration path for our users from Claws Mail to Icedove
We studied the most common Claws Mail persistence configuration
(#9494, #5663), then provided a script (#10453) to migrate POP
mailboxes to Icedove and wrote
[migration instructions](https://tails.boum.org/doc/anonymous_internet/claws_mail_to_icedove/)
(#9495) for users to migrate their mailboxes, email account
settings, and address books.
To make this migration easier to users, we enabled automatically the
Icedove persistence feature when the Claws Mail persistence was
enabled (#9498) and temporarily added the `nmh` package to Tails.
- A.1.5. Update Icedove documentation
The previous Thunderbird documentation was [completely
rewritten](https://tails.boum.org/doc/anonymous_internet/icedove/). (#7158)
- A.1.6. Release Icedove in Tails
Icedove has been added to Tails 1.7. (#10285)
It is advertised as a technology preview in our
[release notes](https://tails.boum.org/news/version_1.7/). We want
to have some more advanced users test our configuration and the
migration process before we make Icedove the default email client
and remove Claws Mail.
# B. Improve our quality assurance process
## B.1. Automatically build ISO images for all the branches of our source code that are under active development
We installed the GlobalBuildStat Jenkins plugin back in February, which helps
us collect the number of ISO images successfully built automatically by our
servers since the deployment of the automatic ISO builds:
- 2015-07: 101
- 2015-08: 195
- 2015-09: 470
- 2015-10: 935
## B.2. Continuously run our entire test suite on all those ISO images once they are built
- B.2.3. Research and design a solution to generate a set of Jenkins ISO test jobs
Early in October we finally settled to the best option we found to
reboot the VM between each run of the test suite in Jenkins (#9486).
We finished evaluating the solution set up last month to share
artifacts between the build jobs and the corresponding test jobs,
and concluded that it works correctly. (#9597)
We had to upgrade our jenkins-job-builder setup to the new 1.3.0 version,
which was required to configure some Jenkins plugins our test jobs need. (#9646)
We also pushed upstream a patch to handle the CucumberTestResult Jenkins
plugin, so that we can easily use it in our setup and have a nice display of
Cucumber reports. It has been merged and will be released with
jenkins-job-builder 1.4.0: <https://review.openstack.org/#/c/218668/>.
We worked together with the test suite team on getting video artifacts
for failing scenarios of a test job stored so that we could better debug. (#10154)
With all this, we were able to deploy automated tests of ISO builds
automatically for our main release branches at the end of the first week of
October, which helped us confirm our design was working. (#10117)
<https://tails.boum.org/blueprint/automated_builds_and_tests/jenkins/>
- B.2.4. Deploy the best solution from this research
With this first initial and limited deployment, we corrected some tiny
configuration details we didn't catch during the design step. (#10215, #10417)
We stumbled upon a bug due to hardware or kernel troubles with nested
virtualization we had to workaround. (#10229)
It also helped us fix some issues in the integration of our test
in the Jenkins environment. (#10356, #10359)
- B.2.5. Write library code that maps Jenkins jobs from building to testing
Now that we had a good example of how to manage a test job with our
jenkins-job-builder setup, triggered by its upstream build job, we
adapted our code that generates build jobs for active branches to
output their corresponding test jobs. We also ended up using the power
of a more recent jenkins-job-builder version to simplify
this code quite a lot. (#10118, #10119)
It was already clear that we wouldn't be able to notify the
committers on test suite failures at first, as the number of false
positives was too high, and we didn't want to train them to ignore
such notifications. So we planned a first iteration during which
only the test suite team would receive notifications upon failure,
so that they'll be able to work on the remaining tests being too
fragile. (#10287)
By October 15th, we had the automated test deployed live in Jenkins
for all active branches. However, we quickly understood that our Git
branches based on our maintenance one were too old to run the test
suite in Jenkins correctly. These topic branches were mainly
documentation ones so we could turn off their automated testing
temporarily (this could be reverted once Tails 1.7 was released).
According to the GlobalBuildStat Jenkins plugin, in the second half of
October, our Jenkins instance ran the test suite 240 times.
We're also discussing on our public mailing list whether it makes sense to
run our whole test suite on ISO images built from
documentation branches. (#10492)
## B.3. Extend the coverage of our test suite
- B.3.5. Write tests for I2P
Basic I2P functionality and configuration that is important in the
context of Tails is now tested (#6306):
<https://git-tails.immerda.ch/tails/tree/features/i2p.feature?h=stable>
This was part of Milestone II (July 15, 2015).
- B.3.6. Fix newly identified issues to make our test suite more robust
and fast enough to be part of a continuous integration process
A plan was devised to leverage out automated testing
infrastructure for identifying robustness issues is our test suite, and isolating
them into separate branches, where they can be dealt with in
isolation. (#10288)
Other improvements:
* The new snapshot system improved robustness noticeably, and
reduced the amount of time needed for a full run with at least
30%. (#6094)
* Restart Tor if bootstrapping stalls for too long.
This started out as a robustness improvement in the test suite,
since many tests failed randomly due to Tor. We then implemented
it as a feature in Tails, since it arguably should improve the
user experience too (#9516).
- B.3.9. Optimize tests that need a persistent volume
The new snapshot system was finished early (part of Milestone IV,
January 15, 2015). (#6094, #8008)
- B.3.13. Split the "various checks" feature
Started early, but not finished; so far only `mat.feature` and
`localization.feature` have been created from split out
scenarios. Thanks to #6094 the remainder of this work will be
trivial.
- Some other new tests that were written:
* Write initial Icedove tests. (#10332)
* Write a test for the fix of a symlink attack. It was added at
the same time as the fix, and this is the ideal workflow we hope
to be ready to adopt soon. (#10333)
- Some improvements for test suite developers:
* Organize automated test suite artifacts in per-run directories,
and improve the naming so that it's easier to see e.g. which video
belongs to which failed scenario. (#10151)
* Clean up automated test suite dependencies. (#10208)
* Force UTF-8 locale in automated test suite, to avoid implicitly
relying on the system locales configuration. (#10359)
* Run the automated test suite's VNC server forever. (#10345)
* Don't print test suite information to STDERR, but use the
Cucumber logging facilities so this information is included when
logging to file. (#10342)
- Bugs and regressions identified by our test suite
We just started taking notes of them, so these records are
probably not complete.
* "Restart Tor if bootstrapping stalls for too long", mentioned
above (#9516).
* In September, implementing the automated MAC spoofing tests led
us to notice that the MAC spoofing's panic mode was broken (#10160).
## B.4. Freezable APT repository
- B.4.1. Specify when we want to import foreign packages into which
APT suites (#9488),
B.4.4. Design freezable APT repository (#5926),
and B.4.5. Implement processes and tools for importing and freezing
those packages (#9487, #6295, #7427, #6906)
Some initial research and experiments were conducted.
The proposed design is described on a blueprint:
<https://tails.boum.org/blueprint/freezable_APT_repository/>.
A server-side implementation (#6296, #6299) has been drafted in the
`tails::reprepro::snapshots::*` Puppet classes in the
<https://git-tails.immerda.ch/puppet-tails/> module.
- B.4.2. Implement a mechanism to save the list of packages used at
ISO build time (#6297)
We have found ways to implement this, and have a working
proof-of-concept, that now needs to be refined. This implied adding
functionality we need to `debootstrap` upstream, and uploading
a backport for Debian Jessie.
- B.4.6. Adjust the rest of our ecosystem to the freezable APT repository
We adopted a documentation-driven approach, and started drafting
documentation for the tools we will need to write:
<https://git-tails.immerda.ch/tails/log/?h=feature/5926-freezable-APT-repository>.
# C. Scale our infrastructure
## C.1. Change in depth the infrastructure of our pool of mirrors
- C.1.2. Write & audit the code that makes the redirection decision
Our prototype code (#8639) has been refined and will be adapted to work
with our new installation assistant.
## C.2. Be able to detect within hours failures and malfunction on our services
- C.2.1. Research and decide what monitoring solution to use: #8645
A little bit more research was done to ensure that our current best
option will satisfy our needs.
## C.4. Maintain our already existing services
This covers "C.4.3. Administer our services upto milestone III" until
the end of October.
* We kept on answering to the request from the community as well as taking
care of security updates.
* We had to reinstall one of our ISO builder VM that was failing too
often. (#10334)
* The OpenPGP key of our Debian repository was going to expire, so we had to
update it. (#10419)
* We thought a bit further how we could make our Jenkins instance public
(#6270) which probably means to update to the packages provided by
upstream. (#10068)
# D. Migration to Debian Jessie
## D. Migration to Debian Jessie
- D.3.1 Update our test suite for Tails Jessie
* Fix memory erasure tests in Debian Jessie. (#8317, #9705)
* Work around screen blanking when testing Synaptic. (#10403)
# E. Release management
* Fix a robustness issue with the "Applications" menu
handling. (#9706)
......@@ -16,7 +16,7 @@ The extension will be used as well by people doing full upgrades until
we automate the download and verification steps of a full upgrade in
Tails Upgrader. See the blueprint on [[upgrades|upgrade]].
[[!toc]]
[[!toc levels=2]]
Vision
======
......
[[!meta title="Download and verify"]]
<div id="extension-version">1.0</div>
<div id="download-and-verify">
<div id="extension-version">1.0</div>
<div id="undetected-browser">
<p>We failed to detect your browser vendor, maybe because JavaScript is disabled.</p>
<p>You can download and verify the ISO image via:</p>
<ul class="download-options">
<li class="extension">
<h2>Browser extension</h2>
<p>(for Firefox, Tor Browser, or Chrome)</p>
<a href="">I'm already in Firefox or Tor Broswer</a>
<a href="">I'm already in Chrome</a>
<p>Or copy and paste this link in Firefox, Tor Browser, or Chrome:</p>
<p>https://tails.boum.org/download</p>
</li>
<li class="bittorrent">
<h2>BitTorrent</h2>
<a href=[[!inline pages="inc/stable_i386_torrent_url" raw="yes"]]>Download Torrent file</a>
</li>
</ul>
</div>
<div id="unsupported-browser">
<p>You can download and verify the ISO image via:</p>
<ul class="download-options">
<li class="extension">
<h2>Browser extension</h2>
<p>(for Firefox, Tor Browser, or Chrome)</p>
<p>Copy and paste this link in Firefox, Tor Browser, or Chrome:</p>
<p>https://tails.boum.org/download</p>
</li>
<li class="bittorrent">
<h2>BitTorrent</h2>
<a href=[[!inline pages="inc/stable_i386_torrent_url" raw="yes"]]>Download Torrent file</a>
</li>
</ul>
</div>
<div id="supported-browser">
<div id="use">
<p>We detected that you are running Firefox or Tor Browser and already have our Firefox extension installed.</p>
<a id="use-button" href="">
<span id="use-button-label">Use Firefox extension</span>
<div id="use-button-state>
<span class="state already">Already installed</span>
<div id="undetected-browser">
<p>We failed to detect your browser vendor, maybe because JavaScript is disabled.</p>
<p>You can download and verify the ISO image via:</p>
<div class="row download-options">
<div class="col-md-6 extension">
<h2>Browser extension</h2>
<p>(for Firefox, Tor Browser, or Chrome)</p>
<a href="" class="btn btn-lg btn-primary">I'm already in Firefox or Tor Browser</a>
<a href="" class="btn btn-lg btn-primary">I'm already in Chrome</a>
<p>Or copy and paste this link in Firefox, Tor Browser, or Chrome:</p>
<code>https://tails.boum.org/download</code>
</div>
</a>
<div id="use-text">
<span id="use-text-label">Use Firefox extension</span>
<div id="use-text-state>
<span class="state done">Done</span>
<div class="col-md-6 bittorrent">
<h2>BitTorrent</h2>
<a href="[[!inline pages="inc/stable_i386_torrent_url" raw="yes"]]" class="btn btn-lg btn-primary">Download Torrent file</a>
</div>
</div>
</div>
<div id="update">
<p>We detected that you are running Firefox of Tor Browser but have an outdated version of our Firefox extension.</p>
<a id="update-button" href="">
<span id="update-button-label">Update Firefox extension</span>
<div id="update-state>
<span class="state restartless">No restart</span>
<div id="unsupported-browser">
<p>You can download and verify the ISO image via:</p>
<div class="row download-options">
<div class="col-md-6 extension">
<h2>Browser extension</h2>
<p>(for Firefox, Tor Browser, or Chrome)</p>
<p>Copy and paste this link in Firefox, Tor Browser, or Chrome:</p>
<code>https://tails.boum.org/download</code>
</div>
</a>
<div id="update-text">
<span id="update-button-label">Update Firefox extension</span>
<div id="update-text-state>
<span class="state done">Done</span>
<div class="col-md-6 bittorrent">
<h2>BitTorrent</h2>
<a href="[[!inline pages="inc/stable_i386_torrent_url" raw="yes"]]" class="btn btn-lg btn-primary">Download Torrent file</a>
</div>
</div>
</div>
<div id="install">
<p>We detected that you are running Firefox or Tor Browser.</p>
<p>You can download and verify the ISO image via our Firefox extension.</p>
<a id="install-button" href="">
<span id="install-button-label">Install Firefox extension</span>
<div id="install-button-state>
<span class="state restartless">No restart</span>
<div id="supported-browser">
<div id="use">
<p>We detected that you are running Firefox or Tor Browser and already have our Firefox extension installed.</p>
<a id="use-button" href="" class="btn btn-lg btn-primary clearfix">
<span id="use-button-label" class="pull-left">Use Firefox extension</span>
<div id="use-button-state" class="pull-right">
<span class="label label-default state already">Already installed</span>
</div>
</a>
<div id="use-text" class="btn btn-lg disabled clearfix">
<span id="use-text-label" class="pull-left">Use Firefox extension</span>
<div id="use-text-state" class="pull-right">
<span class="label label-success state done">Done</span>
</div>
</div>
</a>
<div id="install-text">
<span id="install-text-label">Install Firefox extension</span>
<div id="install-text-state>
<span class="state done">Done</span>
</div>
<div id="update">
<p>We detected that you are running Firefox of Tor Browser but have an outdated version of our Firefox extension.</p>
<a id="update-button" href="" class="btn btn-lg btn-primary clearfix">
<span id="update-button-label" class="pull-left">Update Firefox extension</span>
<div id="update-state" class="pull-right">
<span class="label label-default state restartless">No restart</span>
</div>
</a>
<div id="update-text" class="btn btn-lg disabled clearfix">
<span id="update-button-label" class="pull-left">Update Firefox extension</span>
<div id="update-text-state" class="pull-right">
<span class="label label-success state done">Done</span>
</div>
</div>
</div>
<div id="install">
<p>We detected that you are running Firefox or Tor Browser.</p>
<p>You can download and verify the ISO image via our Firefox extension.</p>
<a id="install-button" href="" class="btn btn-lg btn-primary clearfix">
<span id="install-button-label" class="pull-left">Install Firefox extension</span>
<div id="install-button-state" class="pull-right">
<span class="label label-default state restartless">No restart</span>
</div>
</a>
<div id="install-text" class="btn btn-lg disabled clearfix">
<span id="install-text-label" class="pull-left">Install Firefox extension</span>
<div id="install-text-state" class="pull-right">
<span class="label label-success state done">Done</span>
</div>
</div>
</div>
<p id="bittorrent-minor">or <a href="[[!inline pages="inc/stable_i386_torrent_url" raw="yes"]]" >Download via BitTorrent</a></p>
</div>
<p id="bittorrent-minor">or <a href=[[!inline pages="inc/stable_i386_torrent_url" raw="yes"]]>Download via BitTorrent</a></p>
</div>
<a href="">I already have an ISO image.</a>
<a id="i_have_iso" href="">I already have an ISO image.</a>
<div id="download">
<a id="download-button" href=[[!inline pages="inc/stable_i386_iso_url" raw="yes"]]>
<span id="download-button-label">Download Tails [[!inline pages="inc/stable_i386_version" raw="yes"]] ISO image</span>
<span id="download-button-size">[[!inline pages="inc/stable_i386_iso_size" raw="yes"]]</span>
<div id="download-button-state">
<span id="download-button-state-retry">Retry</span>
<div id="download">
<a id="download-button" href="[[!inline pages="inc/stable_i386_iso_url" raw="yes"]]" class="btn btn-lg btn-primary clearfix">
<span id="download-button-label" class="pull-left">Download Tails [[!inline pages="inc/stable_i386_version" raw="yes"]] ISO image
<small id="download-button-size" >[[!inline pages="inc/stable_i386_iso_size" raw="yes"]]</small>
</span>
<div id="download-button-state" class="pull-right">
<span id="download-button-state-retry" class="label label-warning">Retry</span>
<span id="download-button-state-resume" class="label label-info">Resume</span>
</div>
</a>
<div id="download-text" class="btn btn-lg clearfix">
<div id="download-text-left" class="pull-left clearfix">
<div id="download-text-label" class="pull-left">Download Tails[[!inline pages="inc/stable_i386_version" raw="yes"]] ISO image
<small id="download-eta">
240 Ko/s - 360,7/[[!inline pages="inc/stable_i386_iso_size" raw="yes"]], 10 minutes left
</small>
</div>
<div id="download-progress" class="progress">
<div class="progress-bar progress-bar-striped active" role="progressbar" aria-valuenow="45" aria-valuemin="0" aria-valuemax="100" style="width: 45%">
<span class="sr-only">45% Complete</span>
45%
</div>
</div>
<small id="download-path">Downloading to /home/amnesia/Tor Browser/tails-i386-1.3.2.iso</small>
</div>
<div id="download-text-state" class="pull-right">
<span id="download-text-pause">Pause</span>
<span id="download-text-done" class="label label-success state">Done</span>
<span id="download-text-failed" class="label label-warning state">Failed</span>
</div>
</div>
</a>
<div id="download-text">
<div id="download-text-label">Download Tails [[!inline pages="inc/stable_i386_version" raw="yes"]] ISO image</div>
<div id="download-text-size">[[!inline pages="inc/stable_i386_iso_size" raw="yes"]]</div>
<div id="download-eta">10 minutes left</div>
<div id="download-path">Downloading to /home/amnesia/Tor Browser/tails-i386-1.3.2.iso</div>
<div id="download-text-state">
<span class="state">Pause</span>
<span class="state">Done</span>
<div id="download-message">
<div id="download-message-paused">
<p>The download as been paused. Click "resume" to go on.</p>
</div>
<div id="download-message-failed">
<p>The download of the ISO image failed! Please check your network connection and try to resume...</p>
</div>
</div>
</div>
</div>
<div id="verify">
<div id="verify-text">
<span id="verify-text-label">Verify ISO image</span>
<div id="verify-text-state">
<span class="state calculating">Calculating</span>
<span class="state success">Done</span>
<span class="state failed">Failed</span>
<div id="verify">
<div id="verify-text" class="btn btn-lg clearfix">
<div class="pull-left">
<span id="verify-text-label">Verify ISO image</span>
<small id="verify-text-calculating">Calculating SHA-256 checksum...</small>
</div>
<div id="verify-text-state" class="pull-right">
<span id="verify-text-state-calculating" class="state calculating label label-info">Calculating</span>
<span id="verify-text-state-done" class="state success label label-success">Done</span>
<span id="verify-text-state-failed" class="state failed label label-danger">Failed</span>
</div>
</div>
<span id="verify-text-calculating">Calculating SHA-256 checksum...</span>
<div id="verify-text-success">
<p>If you are knowledgeable about OpenPGP, you can do additional verification using the <a href=[[!inline pages="inc/stable_i386_iso_sig_url" raw="yes"]]>OpenPGP signature</a>.</p>
<a href="">Learn how to do that</a>
<a href="">Next <span>Copy ISO image on first USB stick</span></a>
<div id="verify-text-success" class="bg-success">
<p>If you are knowledgeable about OpenPGP, you can do additional verification using the <a href=[[!inline pages="inc/stable_i386_iso_sig_url" raw="yes"]]>OpenPGP signature</a>.
<a href="">Learn how to do that</a></p>
<a href="" class="btn btn-lg btn-primary">Next <span>Copy ISO image on first USB stick</span></a>
</div>
<div id="verify-text-failure">
<div id="verify-text-failure" class="bg-warning">
<p>The verification of the ISO image failed! Please try to download again&hellip;</p>
</div>
<div id="verify-text-failure-again">
<div id="verify-text-failure-again" class="bg-danger">
<p>The verification of the ISO image failed again! Please try to download again from a different place or a different computer&hellip;</p>
</div>
</div>
......
[[!tails_ticket 5785]]
For users that haven't read the documentation about the unsafe browser
and/or just don't understand when it's necessary, it would be good if
Tails does a reasonable job to try to detect whether a captive portal
......@@ -60,6 +62,10 @@ hellais and friends are working on
[ooni-probe](https://gitweb.torproject.org/ooni-probe.git) which may be
interesting, depeding on how stealthy the probe is.