test.mdwn 22.6 KB
Newer Older
1
2
[[!meta title="Manual test suite"]]

3
4
[[!toc levels=1]]

Tails developers's avatar
Tails developers committed
5
6
Some [[test results]] that might be useful to keep are saved.

7
8
9
10
<div class="caution">
Read this document from the branch used to prepare the release.
</div>

11
12
13
14
15
16
17
18
# Changes

Keeping an eye on the changes between released versions is one of the
many safeguards against releasing crap.

## Source


Tails developers's avatar
Tails developers committed
19
20
21
22
23
24
25
Compare the to-be-released source code with previous version's one e.g.:

Boot the candidate ISO and find the commit it was build from with the
`tails-version` command.

Then, from the source tree, see the diff:

26
	git diff --find-renames <old tag>..<ISO commit>
Tails developers's avatar
Tails developers committed
27
28

e.g. `git diff 0.17..06fa1ab80d55c9f29274b7459bd198edb1a8d53d`
29
30
31

## Result

32
33
Compare the list of bundled packages and versions with the one shipped last
time. `.packages` are usually attached to the email announcing the ISO is ready.
34

35
36
37
38
	/usr/bin/diff -u \
	    wiki/src/torrents/files/tails-i386-0.16.packages \
	    tails-i386-0.17.packages \
	    | wdiff --diff-input  --terminal
39
40
41
42
43
44
45
46
47
48

Check the output for:

- new packages that may cause harm or make the images unnecessarily
  big
- packages that could be erroneously removed
- new versions of software we might not have audited yet (including:
  does the combination of our configuration with software X version
  Y+1 achieve the same wished results as with software X version Y?)

49
50
51
52
## Image size

Check the image size has not changed much since the last release.

53
54
In a directory with many Tails ISO images:

55
    find -iname "tails*.iso" -exec ls -lh '{}' \; | sort -rhk 5
56

57
58
# Automated test suite

59
60
61
62
63
Our long term goal is to eliminate the manual test suite (except the
parts which require real hardware) and have the automated test suite
run all our tests. It's design, and how to write new tests, are
documented on a [[dedicated page|test/automated_tests]].

64
## Running the automated test suite
65

66
See [[test/setup]] and [[test/usage]].
67

68
69
## Automated test suite migration progress

70
71
72
73
74
The manual test suite below either contains tests that cannot be
automated, has no automated test implemented yet, or has a test
implemented, but it either hasn't been reviewed, had a confirmed pass
by someone other than the test author, or has issues. The latter is
tracked by tickets prefixed with `todo/test_suite:`.
75

76
# Tor Browser
77

78
79
80
81
## Security and fingerprinting

* Run the [tests the TBB folks
  use](https://trac.torproject.org/projects/tor/wiki/doc/build/BuildSignoff#TestPagestoUse).
82
* Compare the fingerprint of Tails and the latest TBB using at least
83
84
  <https://panopticlick.eff.org/>
  - The exposed User-Agent should match the latest TBB's one.
Tails developers's avatar
Tails developers committed
85
86
  - Update the [[fingerprint section|support/known_issues#fingerprint]] of the
    known issues page if needed.
87
* WebRTC should be disabled:
Tails developers's avatar
Tails developers committed
88
89
90
91
92
93
94
  - In `about:config` check that `media.peerconnection.enabled` is set to
    `false`.
  - <http://mozilla.github.io/webrtc-landing/>, especially the `getUserMedia`
    test. It's expected that the audio test works if you agree to share a
    microphone with the remote website; anything else should fail.
  - <http://net.ipcalf.com/> should display
    `ifconfig | grep inet | grep -v inet6 | cut -d" " -f2 | tail -n1`
Tails developers's avatar
Tests++  
Tails developers committed
95
* One should be able to switch identities from the web browser.
96
* Running `getTorBrowserUserAgent` should produce the User-Agent set by the
97
  installed version of Torbutton, and used in the Tor Browser.
98
99
100
101
102

## Functionality

* Browsing (by IP) a HTTP or HTTPS server on the LAN should be possible.
* Browsing (by IP) a FTP server on the LAN should be possible.
103
104
105

# Pidgin

Tails developers's avatar
Tails developers committed
106
107
(automate: [[!tails_ticket 7820]])

108
* Check that you can initiate an OTR conversation.
Tails developers's avatar
Tails developers committed
109
* Check that XMPP is working with a new test profile.
110
111
112
113
114
115
116
117
  For example using Riseup:
  - Username: username
  - Domain: riseup.net
  - Connect server: 4cjw6cwpeaeppfqz.onion
  - Then try to create and connect to a new room:
    - Room: testing
    - Server: conference.riseup.net
    - Handle: username
118
* Check that Pidgin doesn't leak too much information when replying to
119
  CTCP requests:
120
  * Start Tails, launch Pidgin, and join #tails.
121
122
123
  * Also join #tails from the webchat of OFTC on <https://webchat.oftc.net/>
    using another nickname.
  * Try to send `/ctcp <Tails_account_nick> COMMAND` from the webchat to pidgin:
Tails developers's avatar
Tails developers committed
124
125
    - You should get no answer apart for the commands listed in [[!tails_ticket
      5823]].
126
127
128
129
130
131
132
    - List of `/ctcp` commands, see [this page](http://www.wikkedwire.com/irccommands):
      - PING
      - VERSION
      - FINGER
      - USERINFO
      - CLIENTINFO
      - TIME
133

134
# Tor
135

Tails developers's avatar
Tails developers committed
136
137
(automate: [[!tails_ticket 7821]])

Tails developers's avatar
Tails developers committed
138
* The version of Tor should be the latest stable one, which is the highest version number
139
  before alpha releases on <http://deb.torproject.org/torproject.org/pool/main/t/tor/>.
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
* Check that the firewall-level Tor enforcement is effective:
  - check output of `iptables -L -n -v`
  - check output of `iptables -t nat -L -n -v`
  - try connecting to the Internet after unsetting `$http_proxy` and
    `$HTTP_PROXY` using a piece of software that does not obey the
    GNOME proxy settings, *and* is not explicitly torified in Tails:
    
        	unset http_proxy ; unset HTTP_PROXY
        	wget --no-proxy http://monip.org/
    
    ... should only give you "Connection refused" error message.
* Check that IPv6 traffic is blocked:
  - check output of `ip6tables -L -n`
  - at a place with working IPv6: try connecting to a known-working
    IPv6-enabled server on its IPv6 address over TCP and icmp6.
* After DHCP has been set up, `/etc/resolv.conf` must read `nameserver 127.0.0.1`.
* Before DHCP has been set up, `/etc/resolv.conf` must read `nameserver 127.0.0.1`.
Tails developers's avatar
Tails developers committed
157
* [[doc/first_steps/startup_options/bridge_mode]] should work:
Tails developers's avatar
Tails developers committed
158
  1. Set up an administrator password.
159
  1. Enable network configuration in Tails Greeter.
160
161
162
163
164
165
  1. Configure a few bridges in Tor Launcher:
     
         	bridge 198.252.153.59:9001
         	obfs2 198.252.153.59:16492
        	obfs3 198.252.153.59:16493
     
Tails developers's avatar
Tails developers committed
166
  1. Use the Internet.
167
168
169
170
171
  1. Check that the only outgoing direct network connections go to the
     configured bridges:
     
         	sudo watch "netstat -taupen | grep ESTABLISHED"

Tails developers's avatar
Tails developers committed
172
* Verify that all destinations reached from an intensive Tails session
173
  are tor routers or
174
  authorities:
Tails developers's avatar
Tails developers committed
175
  1. Boot Tails without the network in.
176
  1. Set up an administration password.
Tails developers's avatar
Tails developers committed
177
178
179
180
181
  1. Start dumping your whole session's network activity with `sudo
     tcpdump -n -i any -w dump` (or better, do the dump on another machine,
     or on the host OS if Tails is running in a VM).
  1. Plug the network.
  1. Wait for Tor to be functional.
182
  1. Save `/var/lib/tor/cached-microdesc-consensus` out of the VM (it's needed
Tails developers's avatar
Tails developers committed
183
184
     to analyze the network dump later on).
  1. Do *a lot* of network stuff (why not run do this while doing all
185
186
     the other tests **but** I2P and the unsafe browser, which would
     show many false positives?)
Tails developers's avatar
Tails developers committed
187
188
  1. Then check all destinations, e.g. by using tshark and the script below:

Tails developers's avatar
Tails developers committed
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
         # set DUMP to the output of tcpdump above
         DUMP=dump
         # set CONSENSUS to Tor's consensus from the Tails session
         CONSENSUS=cached-microdesc-consensus
         NODES=$(mktemp)
         awk '/^r / { print $6 }' ${CONSENSUS} > ${NODES}
         # Note that these default directory authorities may change! To be
         # sure, check in Tor's source, src/or/config.c:~900
         DIR_AUTHS="
         128.31.0.39
         86.59.21.38
         194.109.206.212
         82.94.251.203
         76.73.17.194
         212.112.245.170
         193.23.244.244
         208.83.223.34
         171.25.193.9
         154.35.32.5
         "
         tshark -r ${DUMP} -T fields -e ip.dst | sort | uniq | \
         while read x; do
             ip_expr=$(echo ${x} | sed -e "s@\.@\\\.@g")
             if echo ${DIR_AUTHS} | grep -qe "${ip_expr}"; then
                 continue
             fi
             if ! grep -qe "^${ip_expr}$" ${NODES}; then
                 echo "${x} is bad"
             fi
         done
         rm ${NODES}
Tails developers's avatar
Tails developers committed
220
221

     Note that this script will produce some false positives, like your
222
     gateway, broadcasts, etc.
223

224
225
226
227
228
229
230
231
232
233
## Stream isolation

See our [[stream isolation design
page|contribute/design/stream_isolation]] for details such as port
numbers, that are not duplicated here to avoid desynchronization.

Assumptions for the following tests: first, Tor stream isolation
features properly do their work; second, our `torrc` sets the right
`SocksPort` options to implement what we want.

234
235
236
**Note**: the following commands would advantageously be replaced with
the appropriate tcpdump or tshark filters.

237
* Make sure Claws Mail use its dedicated `SocksPort` when connecting
238
  to IMAP / POP3 / SMTP servers:
239

240
      sudo watch -n 0.1 'netstat -taupen | grep claws'
241

242
* Make sure these use the `SocksPort` dedicated for Tails-specific applications:
243
244
245
246
247
248
  - htpdate — as root, run:

        service htpdate stop \
           && rm -f /var/run/htpdate/{done,success} \
           && service htpdate start

249
    ... with the following command running in another terminal:
250

251
        sudo watch -n 0.1 'netstat -taupen | grep curl'
252
253

  - security check — run `tails-security-check` with the following
254
    command running in another terminal:
255

256
        sudo watch -n 0.1 'netstat -taupen | grep perl'
257
258

  - incremental upgrades — run `tails-upgrade-frontend-wrapper` with
259
    the following command running in another terminal:
260

261
        sudo watch -n 0.1 'netstat -taupen | grep perl'
262

263
* Make sure the Tor Browser uses its dedicated `SocksPort`: quit the Tor Browser
264
  then start it with the following command running in another
265
266
  terminal:

267
      sudo watch -n 0.1 'netstat -taupen | grep firefox'
268
269
270
271
272
273
274

* Make sure other applications use the default system-wide
  `SocksPort`:
  - Polipo — run:

        wget https://tails.boum.org/

275
    ... with the following command running in another terminal:
276

277
        sudo watch -n 0.1 'netstat -taupen | grep polipo'
278
279

  - Gobby 0.5 — start Gobby 0.5 from the *Applications* menu and
280
    connect to a server (for example `gobby.debian.org`), with the following command running in
281
282
    another terminal:

283
        sudo watch -n 0.1 'netstat -taupen | grep gobby'
284
285
286
287
288

  - SSH — run (no need to authenticate the server or to login):

        ssh lizard.tails.boum.org

289
    ... with the following command running in another terminal:
290

291
        sudo watch -n 0.1 'netstat -taupen | grep -E "connect-proxy|ssh"'
292
293
294
295
296

  - whois — run:

        whois example.com

297
    ... with the following command running in another terminal:
298

299
        sudo watch -n 0.1 'netstat -taupen | grep whois'
300

301
* Make sure a random application run using `torify` and `torsocks`
302
303
304
305
  uses the default system-wide `SocksPort`. Run:

      torify /usr/bin/gobby-0.5

306
307
    ... and connect to a server (for example `gobby.debian.org`), with the following command running
    in another terminal:
308

309
        sudo watch -n 0.1 'netstat -taupen | grep gobby'
310
311
312
313

    Then do the same test for:

      torsocks /usr/bin/gobby-0.5
314

315
316
# Use of untrusted partitions

Tails developers's avatar
Tails developers committed
317
318
(automate: [[!tails_ticket 7822]])

Tails developers's avatar
Tails developers committed
319
* Is any local hard-disk swap partition used as swap?
320
  boot on a (possibly virtual) machine that has a cleartext swap
Tails developers's avatar
Tails developers committed
321
322
323
324
325
  partition not managed by LVM. To verify that a local GTP partition is swap,
  check its type code with `sgdisk -p`, Linux swap is code 8200.

  This swap partition must not be used by Tails. Run `cat /proc/swaps`.

Tails developers's avatar
Tails developers committed
326
* Is a persistence volume on a local hard-disk partition used?
Tails developers's avatar
Tails developers committed
327
328
329
330
331
  (Hint: setup a libvirt USB disk with GPT and a partition labeled
  `TailsData`, set the `removable` flag on it, check that
  tails-greeter proposes to enable persistence. Then remove the
  `removable` flag, and check that tails-greeter does not propose to
  enable persistence anymore.)
332
333
334

# Claws

335
336
337
* Check mail over IMAP using:
  - a "clearnet" IMAP server.
  - a hidden service IMAP server (e.g. TorMail, jhiwjjlqpyawmpjx.onion, or
338
    Riseup, zsolxunfmbfuq7wf.onion with SSL).
339
340
341
* Send an email using:
  - a "clearnet" SMTP server.
  - a hidden service SMTP server (see above).
Tails developers's avatar
Tails developers committed
342
343
344
345
* Check that the profile works and is torified:
  1. Send an email using Claws and a non-anonymizing SMTP relay (a
     SMTP relay that writes the IP address of the client it is
     relaying email for in the Received header).
Tails developers's avatar
Tails developers committed
346
  1. Then check that email's headers once received, especially the
Tails developers's avatar
Tails developers committed
347
     `Received:` one.
348
* Also check that the EHLO/HELO SMTP message is not leaking anything
Tails developers's avatar
Tails developers committed
349
  at the application level:
Tails developers's avatar
Tails developers committed
350
351
  1. Start Claws using the panel icon.
  2. Disable SSL/TLS for SMTP in Claws (so take precautions for not
Tails developers's avatar
Tails developers committed
352
353
     leaking your password in plaintext by either changing it
     temporarily or using a disposable account).
Tails developers's avatar
Tails developers committed
354
355
356
  3. Run `sudo tcpdump -n -i lo -w dump` to capture the packets before
     Tor encrypts it, then close tcpdump
  4. Check the dump for the HELO/EHLO message and
Tails developers's avatar
Tails developers committed
357
     verify that it only contains `localhost`: `tcpdump -A -r dump`
Tails developers's avatar
Tails developers committed
358
359
  5. Check the `Received:` and `Message-Id` fields in the received
     message: it must not leak the hostname, nor the local IP.
360

361
# WhisperBack
362

363
364
365
* I should be able to send a bug report with WhisperBack.
* When we receive this bug report on the tails-bugs mailing-list,
  Schleuder tells us that it was sent encrypted.
366

367
368
# Time

Tails developers's avatar
Tails developers committed
369
370
(automate: [[!tails_ticket 5836]])

Tails developers's avatar
Tails developers committed
371
372
373
374
1. Boot Tails without a network cable connected.
   (e.g. `virsh domif-setlink tails-dev 52:54:00:05:17:62 down`.)
2. Set an administration password.
3. set the time to an obviously wrong one:
375

Tails developers's avatar
Tails developers committed
376
           date --set="Mon, 01 Mar 2000 15:45:34 - 0800"
Tails developers's avatar
Tails developers committed
377

Tails developers's avatar
Tails developers committed
378
379
4. Connect the network cable.
   (e.g. `virsh domif-setlink tails-dev 52:54:00:05:17:62 up`)
Tails developers's avatar
Tails developers committed
380
381

=> the date should be corrected and Tor/Vidalia should start
Tails developers's avatar
Tails developers committed
382
correctly.
Tails developers's avatar
Tails developers committed
383

384
# Erase memory on shutdown
Tails developers's avatar
Tails developers committed
385

386
387
388
- `memlockd` must be running
- `udev-watchdog` must monitor the right device when booted off USB (automate: [[!tails_ticket 5560]])
- `udev-watchdog` must monitor the right device when booted off DVD (automate: [[!tails_ticket 5560]])
Tails developers's avatar
Tails developers committed
389
390
- After booting from DVD, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).
391
  (automate: [[!tails_ticket 5472]])
392
393
- After booting from USB, remove Tails boot medium and check that the
  memory erasure process is started (`Loading new kernel`, at least).
394

395
396
# Root access control

397
* Check you can login as root with `su` neither with the `amnesia` password nor
398
  with the `live` one.
Tails developers's avatar
Tails developers committed
399
* Check that the `$TAILS_USER_PASSWORD` variable, if still existing in the system
400
401
  environment after the boot has finished, does not contain the clear text
  password.
402

403
404
# Virtualization support

405
* Test that Tails starts and the browser launches in VirtualBox.
406

407
408
# I2P

409
410
411
412
413
414
Make sure that I2P is up-to-date, at least if the
[changelogs](https://geti2p.net/en/blog/) mention that
security critical bugs were fixed.

Start I2P by appending `i2p` to the kernel command line.

415
* Check that I2P starts when a network interface is up:
416
417
  - Within 30 seconds you should get the "I2P router console is ready"
    pop-up
418
419
420
421
422
423
424
  - Start the I2P Browser via "Applications -> Internet -> I2P Browser":
    * You get the "Starting I2P Browser..." pop-up.
    * The router console (<http://127.0.0.1:7657>) opens successfully
      upon success.
    * On exiting I2P Browser, check that its chroot gets properly torn
      down on exit (there should be nothing mounted inside
      `/var/lib/i2p-browser`).
425
  - After a few minutes you should get the "I2P is ready" pop-up
426
427
428
429
430
431
432
433
  - Go to <http://127.0.0.1:7657/i2ptunnelmgr> in the I2P Browser:
    * You should get "Network: Hidden" in the "General" section.
    * The numbers in the "Peers" section of the sidebar should be
      non-zero.
    * Check that you can reach some eepsites within Iceweasel, like
      <http://i2p-projekt.i2p> and <http://forum.i2p>.
  - Check that you can connect to the I2P IRC server through Pidgin
    and the preconfigured IRC account on 127.0.0.1.
434
435
436
* Check I2P failure modes:
  - Router console failure:
    * Boot without network so I2P doesn't start automatically.
437
    * Block the router console port: `nc -l -p 7657 -t 127.0.0.1`
438
439
440
441
442
443
444
445
446
447
    * Plug the network
    * You should get the "I2P failed to start" pop-up, and I2P should
      not be running (check with `service i2p status`)
  - Bootstrap failure:
    * Detach the network immediately after getting the "I2P router
      console is ready" pop-up
    * Wait for up to six minutes
    * You should get the "I2P is not ready" pop-up
    * The I2P router console should still be accessible on
      <http://127.0.0.1:7657>
448

Tails developers's avatar
Tails developers committed
449
450
# Git

Tails developers's avatar
Tails developers committed
451
* clone a repository over `git://`
Tails developers's avatar
Tails developers committed
452
453
454

  git clone git://git.tails.boum.org/htp

Tails developers's avatar
Tails developers committed
455
* clone a repository over `https://`
Tails developers's avatar
Tails developers committed
456

457
  git clone https://git-tails.immerda.ch/htp
Tails developers's avatar
Tails developers committed
458

Tails developers's avatar
Tails developers committed
459
460
* clone a repository over SSH

461
462
463
464
465
466
467
468
469
470
# SSH

* Connecting over SSH to a server on the Internet should work (and
  appear in Vidalia's connections list).
* Connecting (by IP) over SSH to a server on the LAN should work.
* Connecting to a sftp server on the Internet using GNOME's "Connect
  to a server" should work.

# APT

Tails developers's avatar
Tails developers committed
471
472
473
474
     grep -r deb.tails.boum.org /etc/apt/sources.list*

* Make sure the Tails repository suite in matching the release tag (for example
  the release version number) is in APT sources.
475
476
* Make sure the Tails repository unversioned suites (e.g. `testing`,
  `stable` and `devel`) are *not* in APT sources.
477

478
<a id="incremental-upgrades"></a>
479

480
# Incremental upgrades
481

Tails developers's avatar
Tails developers committed
482
483
484
* List the versions from which an upgrade paths to this one is described.
  In the `stable` or `testing` branch:

485
      git grep -l "  version: '\?0.23'\?" wiki/src/upgrade/
Tails developers's avatar
Tails developers committed
486
487
488
489
490
491
492
493
494
495
496
497
498
499
500
501
502
503
504
505
506

* For each description file, open it and verify if it allows incremental upgrade
  or only full upgrade.

* For each previous version from which an upgrade paths is described, install it
  and try to upgrade:
  * For every incremental upgrade path: make sure the resulting updated
    system "works fine" (boots and pretends to be the correct version).
  * For upgrade paths that only propose a full upgrade: make sure the
    user is told to do a manual upgrade.

  If the IUKs and update-description files have been published on the
  *alpha* channel already (see
  <https://archive.torproject.org/amnesia.boum.org/tails/alpha/>):

      echo 'TAILS_CHANNEL="alpha"' | sudo tee --append /etc/os-release && \
      tails-upgrade-frontend-wrapper

  Else, use a local test setup:

  * A web server on the LAN.
Tails developers's avatar
Tails developers committed
507
508
  * A copy of `wiki/src/upgrade` from the `stable` or `testing` branch,
    for example in `/var/www/tails/upgrade/v1/Tails/0.14~rc2/i386/stable/updates.yml`
Tails developers's avatar
Tails developers committed
509
510
  * A copy of the `iuk` directory of our HTTP mirrors,
    for example in `/var/www/tails/stable/iuk/Tails_i386_0.14-rc2_to_0.14.iuk`.
511

Tails developers's avatar
Tails developers committed
512
    To synchronize your local copy:
513

Tails developers's avatar
Tails developers committed
514
        torsocks rsync -rt --progress --delete rsync.torproject.org::amnesia-archive/tails/stable/iuk/ /var/www/tails/stable/iuk/
515

Tails developers's avatar
Tails developers committed
516
  * Patch `/etc/hosts` in Tails to point to your web server:
517

518
        echo "192.168.1.4    dl.amnesia.boum.org" | sudo tee --append /etc/hosts
519

520
521
522
523
524
525
526
  * Patch sudo configuration to allow passing arbitrary arguments to
    `tails-upgrade-frontend`:

        sudo sed -i \
            -e 's,/usr/bin/tails-upgrade-frontend ""$,/usr/bin/tails-upgrade-frontend,' \
            /etc/sudoers.d/zzz_upgrade

Tails developers's avatar
Tails developers committed
527
528
529
  * Call the upgrader must be called, from inside the system to upgrade,
    with every needed option to use the local web server rather than the
    online one, for example:
530

Tails developers's avatar
Tails developers committed
531
532
533
        DISABLE_PROXY=1 SSL_NO_VERIFY=1 \
        tails-upgrade-frontend-wrapper --override-baseurl \
        http://192.168.1.4/tails
534

Tails developers's avatar
Tails developers committed
535
536
# Windows Camouflage

537
Enable Windows camouflage via the Tails Greeter checkbox and:
538

Tails developers's avatar
Tails developers committed
539
* Tails OpenPGP Applet's context menu should look readable
540
* The Tor Browser should use a Internet Explorer theme
541
* The Unsafe Browser has no scary red theme
Tails developers's avatar
Tails developers committed
542
543
544

# Unsafe Web Browser

Tails developers's avatar
Tails developers committed
545
546
(automate: [[!tails_ticket 7823]])

547
548
* On start, if no DNS server was configured in NetworkManager
  (e.g. if there's no network connection), there must be an error.
Tails developers's avatar
Tails developers committed
549
* Once started, check that:
550
  - the Tor Browser instance runs as the `clearnet` user.
551
  - it has no proxy configured.
552
  - no extensions are installed.
Tails developers's avatar
Tails developers committed
553
554
555
556
557
  - there are no bookmarks.
* On exit, check that:
  - make sure that its chroot gets properly teared down on exit (there
    should be nothing mounted inside `/var/lib/unsafe-browser`).

Tails developers's avatar
Tails developers committed
558
559
560
# Real (non-VM) hardware

`[can't-automate]`
561

562
563
* Boot on bare-metal on USB.
* Boot on bare-metal on DVD.
Tails developers's avatar
Tails developers committed
564
565
566
* Measure boot time (from syslinux menu the GNOME dektop ready - quickly press
  enter in the greeter), then on some reference bare metal hardware, and
  compare with previous version. The new one should not be significantly
567
568
569
  slower to start.

# Documentation
570

Tails developers's avatar
Tails developers committed
571
* Check that links to the online website (`Mirror:`) at the bottom of
Tails developers's avatar
Tails developers committed
572
  bundled static web pages (`/usr/share/doc/tails/website/`) are working. Else, it probably means the
573
  wiki was not built with a recent enough ikiwiki.
574
575
576
577
578
579
* Browse around in the documentation shipped in the image. Internal
  links should be fine.

# Internationalization

Boot and check basic functionality is working for every supported
580
language. You *really* have to reboot between each language.
581
582

* The chosen keyboard layout must be applied.
583
584
585
* The virtual keyboard must work and be auto-configured to use the same keyboard
  layout as the X session.
* The Startpage search engine must be localized for the languages we ship a
586
  search plugin for:
587

588
      find /usr/local/lib/tor-browser/distribution/searchplugins/locale -iname startpage-*.xml
589
590

* The Wikipedia search engine must be localized for all languages.
Tails developers's avatar
Tails developers committed
591
  - Except for Farsi, see [[!tails_ticket 6884]]
592
593
594
595
596
597
598
599
600
601
602
603
604
605

## Spellchecking

* Check that every supported language is listed in the list of languages for
  spell checking.
  - Visit <https://translate.google.com/>.
  - Right-click and choose "Check spelling".
  - Right-click and check the list of available languages.
* For a few languages, check the spell checking:
  - Type something in the textarea.
  - Right-click and select a language.
  - Verify that the spelling suggestion are from that language.
* Once [[!tails_ticket 5962]] is fixed, the browser spelling dictionary must be
  localized (for languages that are supported by our branding extension).
606
607
608

# Misc

609
610
* Check that Tails Greeter's "more options" screen displays properly
  on a display with 600 px height.
611
* Check that all seems well during init (mostly that all services
612
  start without errors), and that `/var/log/syslog` seems OK.
613
* MAT should be able to clean a PDF file, such as:
Tails developers's avatar
Tails developers committed
614
615
616
  <http://examples.itextpdf.com/results/part3/chapter12/pdf_metadata.pdf>
* The Tails signing key shipped should be up-to-date (that is, neither it nor
  one its subkeys must have expired, or be about to expire any time soon).
617
  - `gpg --list-keys --with-colons 1202821CBE2CD9C1`
Tails developers's avatar
Tails developers committed
618
* The "Report an error" desktop launcher should open the [[support]]
619
620
  page, both in English and in one language to which the website is
  translated (automate: [[!tails_ticket 6904]]).
621
* One should be able to refresh the GnuPG keyring in Seahorse (with
Tails developers's avatar
Tails developers committed
622
623
  the workaround documented in comment 4 on [[!tails_ticket 7051]],
  until that ticket is fixed for real).