test.mdwn 13.8 KB
Newer Older
1 2
[[!meta title="Manual test suite"]]

3 4
[[!toc levels=1]]

Tails developers's avatar
Tails developers committed
5 6
Some [[test results]] that might be useful to keep are saved.

7 8 9 10
<div class="caution">
Read this document from the branch used to prepare the release.
</div>

11 12 13 14 15 16 17 18
# Changes

Keeping an eye on the changes between released versions is one of the
many safeguards against releasing crap.

## Source


Tails developers's avatar
Tails developers committed
19 20 21 22 23 24 25
Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the
`tails-version` command.

Then, from the source tree, see the diff:

26
	git diff --find-renames <old ISO commit>..<ISO commit>
Tails developers's avatar
Tails developers committed
27

28
e.g. `git diff --find-renames 334e1c485a3a79be9fff899d4dc9d2db89cdc9e1..cfbde80925fdd0af008f10bc90c8a91a578c58e3`
29 30 31

## Result

32 33
Compare the list of bundled packages and versions with the one shipped last
time. `.packages` are usually attached to the email announcing the ISO is ready.
34

35
	/usr/bin/diff -u \
36 37
	    wiki/src/torrents/files/tails-amd64-3.1.packages \
	    tails-amd64-3.2.packages \
38
	    | wdiff --diff-input  --terminal
39 40 41 42 43 44 45 46 47 48

Check the output for:

- new packages that may cause harm or make the images unnecessarily
  big
- packages that could be erroneously removed
- new versions of software we might not have audited yet (including:
  does the combination of our configuration with software X version
  Y+1 achieve the same wished results as with software X version Y?)

49 50 51 52
## Image size

Check the image size has not changed much since the last release.

53 54
In a directory with many Tails ISO images:

55
    find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 5
56

57 58
# Automated test suite

59 60 61 62 63
Our long term goal is to eliminate the manual test suite (except the
parts which require real hardware) and have the automated test suite
run all our tests. It's design, and how to write new tests, are
documented on a [[dedicated page|test/automated_tests]].

64
## Running the automated test suite
65

66
See [[test/setup]] and [[test/usage]].
67

intrigeri's avatar
intrigeri committed
68
Do point `--old-iso` to the ISO of the previous stable release.
69

70 71
## Automated test suite migration progress

72 73 74 75 76
The manual test suite below either contains tests that cannot be
automated, has no automated test implemented yet, or has a test
implemented, but it either hasn't been reviewed, had a confirmed pass
by someone other than the test author, or has issues. The latter is
tracked by tickets prefixed with `todo/test_suite:`.
77

78
# Tor Browser
79

80 81
## Miscellaneous functionality

intrigeri's avatar
intrigeri committed
82 83 84 85 86 87
* Test if _uBlock_ works:
  - The _uBlock_ icon must be visible.
  - Visit a website that normally displays ads, such is
    <https://www.nytimes.com/>. The ads should not be displayed and
    the uBlock icon should display a strictly positive number of
    blocked elements.
88

89 90
## Security and fingerprinting

intrigeri's avatar
intrigeri committed
91
* Run the [tests the Tor Browser folks
92
  use](https://trac.torproject.org/projects/tor/wiki/doc/build/BuildSignoff#TestPagestoUse).
93
  (automate: [[!tails_ticket 10260]])
intrigeri's avatar
intrigeri committed
94
* Compare the fingerprint of Tails and the latest Tor Browser using at least
95
  <https://panopticlick.eff.org/> (automate: [[!tails_ticket 10262]])
intrigeri's avatar
intrigeri committed
96
  - The exposed User-Agent should match the latest Tor Browser's one.
Tails developers's avatar
Tails developers committed
97
  - Update the [[fingerprint section|support/known_issues#fingerprint]] of the
98
    known issues page if needed.
99
* WebRTC should be disabled: (automate: [[!tails_ticket 10264]])
Tails developers's avatar
Tails developers committed
100 101 102 103 104 105 106
  - In `about:config` check that `media.peerconnection.enabled` is set to
    `false`.
  - <http://mozilla.github.io/webrtc-landing/>, especially the `getUserMedia`
    test. It's expected that the audio test works if you agree to share a
    microphone with the remote website; anything else should fail.
  - <http://net.ipcalf.com/> should display
    `ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1`
107
* Running `/usr/local/lib/getTorBrowserUserAgent` should produce the User-Agent set by the
108
  installed version of Torbutton, and used in the Tor Browser. (automate: [[!tails_ticket 10268]])
109

110
# Thunderbird
111

112
* Check mail over IMAP using:
Ulrike Uhlig's avatar
Ulrike Uhlig committed
113 114 115
  - a hidden service IMAP server (e.g. Riseup, zsolxunfmbfuq7wf.onion with SSL).
* Check mail over POP using:
  - a hidden service POP server (e.g. Riseup, zsolxunfmbfuq7wf.onion with SSL).
116 117
* Send an email using:
  - a hidden service SMTP server (see above).
Ulrike Uhlig's avatar
Ulrike Uhlig committed
118

Tails developers's avatar
Tails developers committed
119
* Check that the profile works and is torified:
120
  1. Send an email using Thunderbird and a non-anonymizing SMTP relay (a
Tails developers's avatar
Tails developers committed
121 122
     SMTP relay that writes the IP address of the client it is
     relaying email for in the Received header).
Tails developers's avatar
Tails developers committed
123
  1. Then check that email's headers once received, especially the
Tails developers's avatar
Tails developers committed
124
     `Received:` one.
125
* Also check that the EHLO/HELO SMTP message is not leaking anything
Tails developers's avatar
Tails developers committed
126
  at the application level:
127 128
  1. Start Thunderbird using the GNOME Applications menu.
  2. Disable SSL/TLS for SMTP in Thunderbird (so take precautions for not
Tails developers's avatar
Tails developers committed
129
     leaking your password in plaintext by either changing it
130 131
     temporarily or using a disposable account). Or better, configure
     StartTLS, since it will send two EHLO/HELO: one before TLS is
132
     initiated; one after. The assumption here is that Thunderbird will
133
     send the same both times.
anonym's avatar
anonym committed
134 135 136
  3. Run `sudo tcpdump -n -i lo -w dump` while sending an email to
     capture the packets before Tor encrypts it, then close
     tcpdump. Note that the packet containing EHLO/HELO will be sent
anonym's avatar
anonym committed
137
     really early, so even if the email failed (e.g. because the mail
anonym's avatar
anonym committed
138
     server doesn't support plaintext SMTP on port 587) we are ok.
Tails developers's avatar
Tails developers committed
139
  4. Check the dump for the HELO/EHLO message and
anonym's avatar
anonym committed
140
     verify that it only contains `127.0.0.1`:
anonym's avatar
anonym committed
141
     `sudo tcpdump -A -r dump | grep EHLO`
142
* Make sure Thunderbird Mail use its dedicated `SocksPort` (see "SocksPort
anonym's avatar
anonym committed
143
  for the MUA" in `/etc/tor/torrc`) when connecting to IMAP / POP3 /
anonym's avatar
anonym committed
144 145
  SMTP servers (both clearnet and hidden services) by monitoring the
  output of this command:
146

147
      sudo watch -n 0.1 'ss -taupen | grep thunderbird'
Ulrike Uhlig's avatar
Ulrike Uhlig committed
148

149
# Tor
Ulrike Uhlig's avatar
Ulrike Uhlig committed
150

Tails developers's avatar
Tails developers committed
151
* The version of Tor should be the latest stable one, which is the highest version number
152 153
  before alpha releases on <http://deb.torproject.org/torproject.org/pool/main/t/tor/>. (automate:
  [[!tails_ticket 10259]])
154

155
# WhisperBack
156

157
* I should be able to send a bug report with WhisperBack.
sajolida's avatar
sajolida committed
158
* When we receive this bug report on the tails-bugs mailing list,
159
  Schleuder tells us that it was sent encrypted.
160

161 162
# Root access control

anonym's avatar
anonym committed
163
* Check you cannot login as root with `su` neither with the `amnesia` password nor
164
  with the `live` one. (automate: [[!tails_ticket 10274]])
165

166 167
# Virtualization support

168
* Test that Tails starts and the browser launches in VirtualBox.
169

170
# APT (automate: [[!tails_ticket 8164 desc="#8164"]])
171

172
     grep -r jenw7xbd6tf7vfhp.onion /etc/apt/sources.list*
Tails developers's avatar
Tails developers committed
173 174 175

* Make sure the Tails repository suite in matching the release tag (for example
  the release version number) is in APT sources.
176 177
* Make sure the Tails repository unversioned suites (e.g. `testing`,
  `stable` and `devel`) are *not* in APT sources.
178

179
<a id="incremental-upgrades"></a>
180

anonym's avatar
anonym committed
181
# Incremental upgrades
182

Tails developers's avatar
Tails developers committed
183 184 185
* List the versions from which an upgrade paths to this one is described.
  In the `stable` or `testing` branch:

186
      git grep -l "  version: '\?0.23'\?" wiki/src/upgrade/
Tails developers's avatar
Tails developers committed
187 188 189 190 191 192 193

* For each description file, open it and verify if it allows incremental upgrade
  or only full upgrade.

* For each previous version from which an upgrade paths is described, install it
  and try to upgrade:
  * For every incremental upgrade path: make sure the resulting updated
194 195
    system "works fine" (boots, pretends to be the correct version,
    and the following components work fine: Tor, Tor Browser, Unsafe Browser).
Tails developers's avatar
Tails developers committed
196 197 198
  * For upgrade paths that only propose a full upgrade: make sure the
    user is told to do a manual upgrade.

199
  If:
Ulrike Uhlig's avatar
Ulrike Uhlig committed
200

201
  * the update-description files have been published on the
202
    *test* channel already (see <https://tails.boum.org/upgrade/v1/Tails/>)
203 204 205
  * and the IUK has been published already (see
    <https://archive.torproject.org/amnesia.boum.org/tails/alpha/>
    and <https://archive.torproject.org/amnesia.boum.org/tails/stable/>):
Tails developers's avatar
Tails developers committed
206

207
  then:
Ulrike Uhlig's avatar
Ulrike Uhlig committed
208

anonym's avatar
anonym committed
209 210 211 212 213 214 215 216 217 218
        # Set TAILS_CHANNEL accordingly:
        
        # For actual releases:
        TAILS_CHANNEL=test
        
        # For other (~rc, ~alpha...) releases:
        TAILS_CHANNEL=alpha
        
        # Run:
        echo "TAILS_CHANNEL=\"${TAILS_CHANNEL}\"" | sudo tee --append /etc/os-release && \
219
        tails-upgrade-frontend-wrapper
Tails developers's avatar
Tails developers committed
220 221 222 223

  Else, use a local test setup:

  * A web server on the LAN.
Tails developers's avatar
Tails developers committed
224
  * A copy of `wiki/src/upgrade` from the `stable` or `testing` branch,
225
    for example in `/var/www/tails/upgrade/v1/Tails/3.14~rc2/amd64/stable/updates.yml`
Tails developers's avatar
Tails developers committed
226
  * A copy of the `iuk` directory of our HTTP mirrors,
227
    for example in `/var/www/tails/stable/iuk/Tails_amd64_3.14-rc2_to_3.14.iuk`.
228

Tails developers's avatar
Tails developers committed
229
    To synchronize your local copy:
230

Tails developers's avatar
Tails developers committed
231
        torsocks rsync -rt --progress --delete rsync.torproject.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/
232

Tails developers's avatar
Tails developers committed
233
  * Patch `/etc/hosts` in Tails to point to your web server:
234

235
        echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts
236

237 238 239 240 241 242 243
  * Patch sudo configuration to allow passing arbitrary arguments to
    `tails-upgrade-frontend`:

        sudo sed -i \
            -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
            /etc/sudoers.d/zzz_upgrade

Tails developers's avatar
Tails developers committed
244 245 246
  * Call the upgrader must be called, from inside the system to upgrade,
    with every needed option to use the local web server rather than the
    online one, for example:
247

Tails developers's avatar
Tails developers committed
248 249 250
        DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
        tails-upgrade-frontend-wrapper --override-baseurl \
        http://192.168.1.4/tails
251

anonym's avatar
anonym committed
252 253
# Unsafe Web Browser

254
* Browsing (by IP) a FTP server on the LAN should be possible. (automate: [[!tails_ticket 10252]])
anonym's avatar
anonym committed
255

Tails developers's avatar
Tails developers committed
256 257 258
# Real (non-VM) hardware

`[can't-automate]`
259

260 261 262 263 264
* Boot on bare-metal from USB. Measure the boot time (from the
  syslinux menu until the GNOME desktop is ready -- quickly press
  ENTER in the Greeter) and compare with the boot time of the previous
  Tails version. The new one should not be significantly slower to
  start.
anonym's avatar
anonym committed
265 266 267 268 269 270
* Boot on bare-metal from DVD. Measure the boot time (from the
  syslinux menu until the GNOME desktop is ready -- quickly press
  ENTER in the Greeter) and compare with the boot time of the previous
  Tails version. The new one should not be significantly slower to
  start (for release candidates we do not always update the squashfs
  sort file, so then it might be ok if somewhat slower).
271 272

# Documentation
273

Tails developers's avatar
Tests++  
Tails developers committed
274 275 276 277
* The "Tails documentation" desktop launcher should open the
  [[getting started]] page (automate: [[!tails_ticket 8788]]):
  - in one language to which the website is translated
  - in one language to which the website is not translated (=> English)
278
* Browse around in the documentation shipped in the image. Internal
279
  links should be fine. (automate: [[!tails_ticket 10254]])
280 281 282

# Internationalization

283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298
Boot and check basic functionality is working for these (language,
region) tuples:

 - Arabic - Egypt
 - Chinese - China
 - Deutsch - Deutschland
 - English - USA
 - Español - España
 - Français - France
 - Italiano - Italia
 - Persian - Iran
 - Português - Brasil
 - Russian - Russian Federation
 - Tiếng Việt - Vietnam

You *really* have to reboot between each language.
299

300
* The chosen keyboard layout must be applied. (automate: [[!tails_ticket 10261]])
301 302 303 304 305
* The screen keyboard must (automate: [[!tails_ticket 10263]]):
  - work in Tor Browser when activated after the browser has started;
  - work in Thunderbird when activated after Thunderbird has started;
  - be auto-configured to use the same keyboard layout as the
    X session.
anonym's avatar
anonym committed
306
* In the Tor Browser:
anonym's avatar
anonym committed
307
  - DuckDuckGo must be the default, pre-selected search plugin. (automate: [[!tails_ticket 10265]])
anonym's avatar
anonym committed
308 309 310
  - the search plugins must be localized for the expected locales
    (automate: [[!tails_ticket 10267]]).
    
anonym's avatar
anonym committed
311 312 313
    StartPage should have localized *user interface* for (run this in
    a Tails Git checkout of the commit the release under testing was built
    from):
anonym's avatar
anonym committed
314 315 316 317 318 319 320 321 322 323 324 325 326 327 328 329
    
        grep --extended-regexp "[^:]*:[^:]*:[^:]*:[^:]*:[^:]+" \
            config/chroot_local-includes/usr/share/tails/browser-localization/descriptions | \
            sed -n --regexp-extended 's/^([^:]+):.*$/\1/p'
    
    StartPage should have localized *search results* for:
    
        grep --extended-regexp "[^:]*:[^:]*:[^:]*:[^:]+:[^:]*" \
            config/chroot_local-includes/usr/share/tails/browser-localization/descriptions | \
            sed -n --regexp-extended 's/^([^:]+):.*$/\1/p'
    
    DDG should have localized user interface *and* search results, and
    Wikipedia should have a localized plugin, for:
    
        sed -n --regexp-extended 's/^([^:]+):.*$/\1/p' \
            config/chroot_local-includes/usr/share/tails/browser-localization/descriptions
330 331 332

## Spellchecking

anonym's avatar
anonym committed
333 334 335 336 337 338 339 340 341 342 343
To see which among the supported locales there should be no
spellchecker, run this in a Tails Git checkout of the commit the
release under testing was built from:

    git grep NO_SPELLCHECKER_LOCALES= config/chroot_local-hooks/11-localize_browser

Then do the follow in the same Tor Browser session running in the
`en_US.UTF-8` locale (or whatever locale you are most comfortable
identifying other language names in):

* Check that the expected languages are listed in the list of languages for
344
  spell checking. (automate: [[!tails_ticket 10269]])
345 346 347 348 349 350
  - Visit <https://translate.google.com/>.
  - Right-click and choose "Check spelling".
  - Right-click and check the list of available languages.
* For a few languages, check the spell checking:
  - Type something in the textarea.
  - Right-click and select a language.
351
  - Verify that the spelling suggestion are from that language. (automate: [[!tails_ticket 10271]])
352 353 354

# Misc

355
* Check that Tails Greeter's "more options" screen displays properly
356
  on a display with 600 px height, preferably in a language that's
357
  more verbose than English (e.g. French). (automate: [[!tails_ticket 10276]])
358
* Check that all seems well during init: (automate: [[!tails_ticket 10277]])
359 360
  - `systemctl --failed --all` should say `0 loaded units listed`
  - the output of `journalctl` should seem OK.