Commit 64ddfbb4 authored by intrigeri's avatar intrigeri
Browse files

Merge branch 'devel' into feature/jessie

parents ffe87a7d b3e51f09
......@@ -26,6 +26,7 @@ Feature: Various checks
Given I have started Tails from DVD without network and logged in
Then the shipped Debian repository key will be valid for the next 3 months
@doc
Scenario: The "Report an Error" launcher will open the support documentation
Given I have started Tails from DVD without network and logged in
And the network is plugged
......
......@@ -4,6 +4,7 @@ Feature: Localization
I want Tails to be localized in my native language
And various Tails features should still work
@doc
Scenario: The Report an Error launcher will open the support documentation in supported non-English locales
Given I have started Tails from DVD without network and stopped at Tails Greeter's login screen
And the network is plugged
......
......@@ -3,6 +3,7 @@ Feature: check PO files
As a Tails developer, when I build Tails, I want to make sure
the PO files in use are correct.
@doc
Scenario: check all PO files
Given I am in the Git branch being tested
Then all the PO files should be correct
......@@ -94,6 +94,7 @@ Feature: Browsing the web using the Tor Browser
When I open the address "file:///tmp/synaptic.html" in the Tor Browser
Then I do not see "TorBrowserSynapticManual.png" after at most 5 seconds
@doc
Scenario: The "Tails documentation" link on the Desktop works
Given I have started Tails from DVD and logged in and the network is connected
When I double-click on the "Tails documentation" link on the Desktop
......
jenkins-tools @ acb5e3f0
Subproject commit fa46523fb297c21a975a39140676708c9eedbfee
Subproject commit acb5e3f0427021ef34514558700570002405c4a1
[[!toc levels=2]]
<div class="caution">
<strong>Deadline: 2015-09-05</strong>
<strong>Deadline: 2015-12-05</strong>
</div>
<div class="note">
......@@ -17,25 +17,237 @@ Note: the numbers preceded with a `#` correspond to tickets in our bug
tracker which contains more technical details and timeline. For example,
ticket #6938 can be seen on <https://labs.riseup.net/code/issues/6938>.
# A. Replace Claws Mail with Icedove
# B. Improve our quality assurance process
## A.n. description of subsection
## B.1. Automatically build ISO images for all the branches of our source code that are under active development
- A.n.m. description of deliverable: ticket numbers
In November, XXX ISO images were automatically built on our Jenkins setup.
status summary:
## B.2. Continuously run our entire test suite on all those ISO images once they are built
* what was done
* what is the outcome (how it makes Tails better)
* what was not done, and why
In November, **314 ISOs** were tested by our Jenkins instance.
# B. Improve our quality assurance process
- B.2.4. Implement and deploy the best solution from this research
With the 1.7 release, we were able to unify back the way the branches
based on our maintenance branch and the ones based on our development
branch were tested. (#10409)
We have deployed the feature that prints out in the test suite logs in
Jenkins a direct link to the collected artifacts of the failing step.
(#10445)
We found out that Sikuli was sometimes leaving PNG files in the Jenkins
workspaces, and we worked on a fix. (#10476)
We are being short again in disk space and need to anticipate that future
infrastructure features will require more, so we're sorting out what's the real
needs of our isotesters on this side, now that we have more data points.
(#10396)
We are also short on isotesters to really test every single automatically
built ISO image. So we're discussing if we need new hardware to cope with the
load, and if so, which kind. (#9264)
Meanwhile, our automated test suite will still be a bit more efficient
on testing branches that actually change Tails' code, while testing
only some subset for the branches that only change our website or
documentation. (#10492)
We spent a lot of the CI time gathering statistics about the test suite
runs in Jenkins in order to help the team writing the tests to identify
which scenarios were fragile, going forward to the release of the
notifications to contributors.
XXX: #10601
## B.3. Extend the coverage of our test suite
* First of all, it should be noted that our automated test suite has
been key to identify regressions introduced by porting Tails to
Debian Jessie. A number of such bugs have been found early, that we
would otherwise not have discovered before the first release
candidate of Tails based or Jessie at best, and quite possibly after
the first actual release of Tails based on Jessie.
* B.3.11. Fix newly identified issues to make our test suite more robust and faster
- Increased the number of Tor circuit retries in the test suite (#10375).
- Work is in progress to make some test cases, that cause false
positives, more robust: SSH (#10498), Git (#10444), Windows
Camouflage (#10493), Seahorse (#9095, #10500, #10501).
- More such fragile test cases were identified, and are momentarily
disabled in our Jenkins environment. They will be worked on later:
Electrum tests (#10475) due to us shipping a too old version of
this piece of software, connecting to the #i2p IRC channel
(#10474), Synaptic (#10441), APT (#10496), ICMP filtering
(#10499), whois (#10523), `wait_until_tor_is_working` helper
(#10497), time synchronization (#10440, #10494, #10495), Pidgin
connecting to OFTC (#10443), watching a WebM video over HTTPS
(#10442).
# C. Scale our infrastructure
## C.2. Be able to detect within hours failures and malfunction on our services
- C.2.1. Research and decide what monitoring solution to use
what tools and abstraction layer to use for configuring it,
and where to host it.
We researched more deeply how the different candidates were respecting
our specifications, especially about the trust that we will put
into the monitoring machine.
## C.4. Maintain our already existing services
We kept on answering the requests from the community as well as taking
as well as taking care of security updates as covered by "C.4.4.
Administer our services upto milestone IV" until the end of November.
We settled on a date for a sysadmin sprint in December.
# D. Migration to Debian Jessie
## D.1. Adjust to the change of desktop environment to GNOME Shell
We finished porting Tails to GNOME Shell by fixing the remaining
identified regression: raw HTML was displayed in desktop notifications
(#7989).
## D.2. Take advantage of systemd to improve the internals of Tails
The Tor daemon is now managed with systemd, that completes supervision
of critical services (#5750).
We also polished code that was previously ported to systemd:
* Introduced a dedicated systemd target for "Tor has bootstrapped"
state (#9393).
* Call out to the shell less often in htpdate.service (#10612).
* Converted the `21-gdm_unit_file` hook to a drop-in override (#10606).
## D.3. Update our test suite for Tails Jessie
This is about ticket #7563 and subtasks. Good progress was made
on this front in November.
These were resolved:
* Updated the memory erasure automated tests for Jessie (#9705, #8317);
in passing, made it more robust and accurate.
* Disabled screen blanking during the test suite, since it broke some
tests on Jessie (#10403); and in passing, we have refactored how we
re-use GNOME/X's environment from scripts in our codebase.
* Fixed the "I open the address" step by pasting URLs instead of
typing them char by char (#10467).
* Updated the "Connect to server" tests for Jessie (#10325).
* Adapted GNOME notification handling (#8782) and applications menu
handling (#9706) for Jessie in the test suite.
* Ported tests that used syslog to instead use the systemd Journal.
* Updated Icedove tests for Jessie.
Additionally, we made progress on:
* With respect to "iptables_parse is buggy for IPv6" (#9704): sadly,
we could not find an iptables rules parser in Ruby (to nicely
integrate into our test suite's codebase) that clearly states that
it's compatible with IPv6. So we started looking for candidate
solutions in Python and Perl.
* The GnuPG keyserver interaction test cases are being updated for
Jessie (#9791).
* Updating tests for: USB installation and upgrade, Tor Browser
tests, ICMP filtering.
## D.4.1. Document the changes implied by the move to Jessie on our website
XXX
## D.5.1. Release an official version of Tails based on Jessie
Our two usual release managers also did a full code review of the
changes introduced on our Jessie branch so far. No serious problem was
identified, but lots of small polishing tasks have been added.
Based on these good results and the ones we see in our partly ported
automated test suite, we decided a [release schedule for Tails
Jessie](https://mailman.boum.org/pipermail/tails-dev/2015-November/009790.html)
which will be released on January 26 and be numbered 2.0. We renamed
accordingly the versions coming after 2.0. Here is the new version
numbers of the releases mentioned in past reports and budgets:
- Tails 1.9 will be Tails 2.0
- Tails 1.10 will be Tails 2.2 (the second major version in a row)
- Tails 1.11 will be Tails 2.3
- Tails 1.12 will be Tails 2.4
- Tails 1.13 will be Tails 2.5
## Various porting to Jessie
A lot of other porting to Jessie work was done, that does not really
fit into any of the above categories:
* Merged and forward-ported changes introduced up to Tails 1.7.
* Ported our automatic (incremental) upgrade system to Jessie (#8083),
and made its check for free memory smarter (#8263, #10540).
* Fixed synchronizing OpenPGP keys in Seahorse (#9792).
* Fixed Totem access to the network (#10593, #10603).
* Ensured we use a current version of obfs4proxy (#10605).
* Made AppArmor profile for cupsd work in read-only persistence mode
(#10591).
* Fixed string encoding regression that was introduced when porting
WhisperBack to Python 3 (#10577).
* Restored association of application/pgp-keys MIME type with
Seahorse's "Import Key" application (#10571).
* Restored AppArmor confinement of Tor on Jessie (#10528) that briefly
disappeared with the upgrade to Tor 0.2.7.
* Adjusted to the fact that `.xsession-errors` does not exist anymore
on Jessie (#9966).
* Restored unmuting and sanitizing ALSA mixer levels at boot time (#7591).
* Wrote a test case to ensure that NetworkManager change of behaviour
wrt. watching configuration files does not affect us (#7966).
And some code was cleaned up in passing:
* Removed obsolete bits of APT pinning (#10607).
* Removed obsolete workaround for the HashType enum in Tails Upgrader
(#7042).
* Dropped custom syslinux package with patches for Chromebooks, that
we had included in a Debian Jessie update (#9292).
We researched some candidate changed and investigated a few potential
problems:
* Made sure the services we disable at boot time indeed don't start on
Jessie (#8313).
* Considered replacing some of our code with the GDM to GNOME session
keyboard layout pass-through (#8713).
* Clarified `ttdnsd.service` (#10608) and `laptop-mode.service`
(#9967) situation on Jessie.
* Looked for screen reader support regressions compared to Wheezy
(#8371).
* Made sure that we have no regression wrt. logging into a text
console on Jessie (#9408).
* Check if we need to keep memlockd running longer (#8260).
* Made sure that GNOME 3.14's captive portal handling is disabled on
Tails/Jessie (#10526).
* Evaluated the Caribou virtual keyboard in Tails/Jessie (#8281).
* Considered replacing sound-juicer with goobox (#8393).
# E. Release management
- Tails 1.7 was released on 2015-11-03 [1]:
* Add an offline mode to disable all networking.
* Add Icedove as a technology preview.
* Improve the wording of the first screen of Tails Installer.
* Upgrade Tor Browser to version 5.0.4 (based on Firefox 38.4.0 ESR).
* Update Tor to 0.2.7.4.
* Restart Tor automatically if connecting to the Tor network takes too long. (#9516)
* Prevent wget from leaking the IP address when using the FTP protocol. (#10364)
* Prevent symlink attack on ~/.xsession-errors via tails-debugging-info.
* Force synchronization of data on the USB stick at the end of automatic upgrades.
* Make the "I2P is ready" notification more reliable.
[1] <https://tails.boum.org/news/version_1.7/>
[[!meta title="User interface for additional software packages"]]
The persistence feature for additional software packages is a great tool
to make Tails more flexible for diverse scenarios without having to
bloat the ISO image.
The current limitations include:
- No user interface. Currently you have to edit a file as root. ([[!tails_ticket 5996 desc="#5996"]])
- Their Installation locks the opening of the desktop. ([[!tails_ticket 9059 desc="#9059"]])
- They are checked for updates every time Tor is restated. ([[!tails_ticket 9819 desc="#9819"]])
Proposed user experience
========================
1. When installing a new package, either through the command line or
through Synaptic, the user is asked whether she wants to make it
persistent.
2. When removing a persistent package, the user is asked whether she
wants to remove it from the list of persistent packages.
3. Have a list of the persistent packages visible in the persistence
wizard. As the user need to be able to check the state of this feature
outside of APT operations.
4. Allow removing packages from the list in 3 (optional).
5. Allow adding packages to the list in 3 (optional).
Implementation
--------------
- 1 and 2 might be possible to implement using APT hooks. We need to
investigate how these APT hooks would communicate with the desktop
notifications.
- 3 might require modifying the general concept of the persistence
wizard which is currently only a list of features that are activated
or not, without feedback on the information managed by each of them.
- 4 should be easy to implement once we have 3 as removing packages
from the list doesn't need any validity check. Note that we would
always answer Yes to debconf questions.
- 5 would require validating the packages added to the list to make
sure that they can be installed. Installing packages on the fly as
they are added to the list might help solving this.
- We could merge both the **APT Packages** and **APT Lists**
persistence features
Speed up installation
=====================
To solve [[!tails_ticket 9059 desc="#9059"]], which can currently block
the opening of the desktop for several minutes, we should investigate:
- Starting reading packages lists and building cache on GDM PostLogin.
For that we need an APT mechanism to do all this without installing
or removing anything.
- Installing packages once the session has started.
- Using `nice` to not slow down the desktop too much in competition with
`tails-upgrade-frontend`.
- What kind of packages would suffer from being installed after the
session started.
- A notification mechanism for APT to be started after the session is
ready.
Rationale
=========
In [[!tails_ticket 9264]], we're discussing and drafting our needs for more
isotesters. From some statistics of the number of automated builds per month,
we concluded we need to be able to run at least 1000 more test suites per
month, which means begin able to host at least 8 more isotesters.
Estimates
=========
As a reminder, we are discussing in [[!tails_ticket 10396]] that one
isotester means:
* 20G RAM
* 25-30G HDD (10G system+data, 15-20G tmp)
* 3 CPUs (+1 for the qemu process on the host)
So 8 isotesters would mean:
* 160G RAM
* 200-240G HDD
* 32 CPUs max
From there, it seems unlikely we could host that on Lizard, and we are
back on the research about new hardware for isotesters.
Hardware
========
Given the amount of RAM and CPUs required, it seems close to [[Lizard's
specs|blueprint/hardware_for_automated_tests]]. If we stick on the CPUs, we
probably need the same kind, with 12 cores and in the end 48 CPU threads.
If we settle on this, we would have a bit of room for the future to be able to
run up to 11 isotesters (44 CPUs, 275-330G HDD, 220G RAM), running theorically
1320 test suites per month on this sole box.
We still need to decide if we would use faster (as in Ghz) CPUs than the one
Lizard has (Low voltages versions). In this case the price jumps quite a bit,
and the electic bill will too (120W each for that kind of CPUs, against the 65W
each of Lizard's ones).
The following price estimations are based on interpromicro.com ones.
## Option A: Low voltage CPUs
That's essentially a Lizard clone, minus SSD HDDs.
* 1 x Super X10DRi, 2x GB/i350 LAN, 10x SATA3(C612)+R, IPMI, DDR4(1TB) - $390
* 2 x Intel® E5-2650Lv3-1.8Ghz/12C, 30M, 9.6GT/s, 65W LGA2011 CPU - $2,860
* 16 x 16GB D4-2133Mhz, ECC/Reg 288-Pin Sdram - $2100
* 2 x Samsung 850 EVO, 500GB 2.5" SATA3 SSD(3D V-NAMD) - $400
* 2 x Heatsinks - $100
* 1 x Supermicro SC113TQ-600W, 1U RM, WIO, 8x 2.5" SAS/SATA 600W - $350
Total --- **$6200**
## Option B: Faster CPUs
* 1 x Super X10DRi, 2x GB/i350 LAN, 10x SATA3(C612)+R, IPMI, DDR4(1TB) - $390
* 2 x Intel® E5-2670v3-2.3Ghz/12C, 30M, 9.6GT/s, 120W LGA2011 CPU - $3,300
* 16 x 16GB D4-2133Mhz, ECC/Reg 288-Pin Sdram - $2100
* 2 x Samsung 850 EVO, 500GB 2.5" SATA3 SSD(3D V-NAMD) - $400
* 2 x Heatsinks - $100
* 1 x Supermicro SC113TQ-600W, 1U RM, WIO, 8x 2.5" SAS/SATA 600W - $350
Total --- **$6650**
......@@ -10,7 +10,6 @@ Availability and plans for the next weeks
month
- Volunteers to handle important tickets flagged for next release,
but without assignee
- Availability and plans for monthly low-hanging fruits meeting
- Availability and plans until the next meeting
Discussions
......
......@@ -8,9 +8,11 @@ This is about [[!tails_ticket 10491]].
<https://mailman.boum.org/pipermail/tails-dev/2015-October/009593.html>
- Digital rewrite by SpencerOne:
- Digital rewrite by Spencer:
<a href="https://labs.riseup.net/code/attachments/download/1035/20151005-spencer.png"><img src="https://labs.riseup.net/code/attachments/download/1035/20151005-spencer.png" width="300px"></a>
<a href="https://mailman.boum.org/pipermail/tails-ux/attachments/20151204/aba95e82/attachment-0001.png"><img src="https://mailman.boum.org/pipermail/tails-ux/attachments/20151204/aba95e82/attachment-0001.png" width="300px"></a>
<https://mailman.boum.org/pipermail/tails-ux/2015-December/000812.html>
Current issues in Tails
=======================
......@@ -73,3 +75,4 @@ Related work
============
- [[Captive portal detection|detect_captive_portals]]
- [[UX testing of circumvention features of Tor Browser|https://github.com/lindanlee/circumvention-ux-tor]]
......@@ -58,9 +58,9 @@ FIXME
Metrics
=======
* Tails has been started more than FIXME times this month. This makes FIXME boots a day on average.
* Tails has been started more than 500,551 times this month. This makes 16,147 boots a day on average.
* FIXME downloads of the OpenPGP signature of Tails ISO from our website.
* 25,790 downloads of the OpenPGP signature of Tails ISO from our website.
* FIXME bug reports were received through WhisperBack.
......
......@@ -9,7 +9,7 @@ Releases
* [[Tails 1.8 was released on December 15, 2015|news/version_1.8]] (minor release).
* The next release (1.9) is [[planned for January 26|https://tails.boum.org/contribute/calendar/]].
* The next release (2.0) is [[planned for January 26|https://tails.boum.org/contribute/calendar/]].
Code
====
......
[[!meta title="WhisperBack for frontdesk"]]
At the 2015 summit, we identified a few improvements to WhisperBack that
would make the life of our support team easier. Namely:
- Have user language information in Whisperback ([[!tails_ticket 9799 desc="#9799"]])
- Add some OpenPGP key checks to WhisperBack ([[!tails_ticket 9800 desc="#9800"]])
Here are some notes and design ideas on how to solve these.
Language information
====================
Our support team can understand and write several languages. But not everybody
in the team has the same linguistic abilities and they also work on shifts.
The process of answering requests would be faster if our team could:
- Filter incoming requests based on the language they are written in.
- Know whether the person sending the request understands other languages.
Language of the report
----------------------
We could determine the language in which the report is written by:
- Using a language detection library.
- Referring to the LOCALE of the session.
- Asking the user.
Relevant design question:
- None of these techniques would be perfect but that's ok.
- The set possible languages should be limited by the set of languages
understood by the support team.
- The user should be aware this information so as to correct it manually or
to adjust her writing to the languages understood by the support team.
Languages understood by the user
--------------------------------
This is based on the assumption that many people are more comfortable
understanding than writing another language than their native language. This
being true both for people writing us and people working in the support team.
For example, it might be all-right for someone writing us in French to receive
an answer in English or for someone writing us in Portuguese to receive an
answer in Spanish.
we could determine the languages understood by the user by:
- Referring to the LOCALE of the session.
- Asking the user.
Privacy concerns
----------------
As we received the reports encrypted, it would be most efficient for the
support team to know the language of the report from the email headers. For
example as a flag in the subject of the email as this would allow using email
filters. But this might have privacy concerns.
- When the user sends the report, we would be telling our WhisperBack relays
(hidden services), boum.org, and the email providers of the user support team
that someone sent an report in a given language amongst the one understood by
the team. This seems acceptable.
- When answering the report:
- If the answer is sent in plain text, the language of the report
would already be known by all the machines relaying the email.
- If the answer is sent encrypted, we would revealed this language to the
email provider of the user. This is more serious and should be avoided.
As a workaround, we can decide not to flag in the email headers, the language
of reports that are sent along with an OpenPGP key.
OpenPGP checks
==============
We want to:
1. Verify that the key matches the email address of the user.
2. Verify that the key exists.
3. Be nice to people sending many reports and avoid asking them to paste the
armored version every time.
In order to ensure 1 and 2, we need access to the public key itself (not only
it's ID).
To solve 3 we could:
1. Look for a key in the keyring of the user. But we need to ask permission
before adding it to the report.
2. Search for a key on the public key servers. But we need to ask permission
before doing the search.
Proposed design
===============
Mockup
------
<pre>
Summary: [ ]
Email: [ ] [ OpenPGP key (optional) ]
Language: [ English |v] [ ] I understand English.
French
Spanish
Italian
Description of the problem:
[ ]
[ ]
</pre>
Description of the widgets
--------------------------
a. `[ OpenPGP key (optional)]` is a button which opens a dialog:
1. If an OpenPGP key matching the email is found in the keyring
of the user, then we ask for permission before adding it to the
report.
2. Otherwise we ask permission to search for a key in the public key
servers and then attach it to the report.
3. If the search fails we ask for an armored OpenPGP key and
validate it against the email of the user.
4. If several valid keys are found in step 1 or 2 we ask the user to select one.
5. Once an OpenPGP key is selected the button changes caption