|
|
[[!tag archived]]
|
|
|
|
|
|
[[!meta title="Automated testing tools"]]
|
|
|
|
|
|
We already have [[an automated test suite|contribute/release_process/test/automated_tests]].
|
|
|
This page is about tools that could allow us to improve it.
|
|
|
|
|
|
[[!toc levels=2]]
|
|
|
|
|
|
# Tools
|
|
|
|
|
|
## behave
|
|
|
|
|
|
* [homepage](https://github.com/behave/behave)
|
|
|
* Cucumber-like, in Python
|
|
|
* used by GNOME
|
|
|
* examples from the eog source tree, that use *behave* and *dogtail*:
|
|
|
- [feature](https://git.gnome.org/browse/eog/tree/tests/actions.feature)
|
|
|
- [steps definition](https://git.gnome.org/browse/eog/tree/tests/steps/steps.py)
|
|
|
* In [Debian](https://packages.qa.debian.org/b/behave.html) (in unstable since 2017/07/20)
|
|
|
* Python (with Jython) is now Sikuli's preferred scripting language;
|
|
|
it's also the language that has the best maintained bindings to
|
|
|
interact with libvirt, accessibility technologies, and more
|
|
|
* does *behave* work fine under Jython?
|
|
|
|
|
|
## dogtail
|
|
|
|
|
|
* [homepage](https://gitlab.com/dogtail/dogtail)
|
|
|
* actively maintained upstream as of 2017-05-31
|
|
|
* GUI test tool and automation framework written in Python
|
|
|
* uses Accessibility (a11y) technologies to communicate with
|
|
|
desktop applications
|
|
|
* used by GNOME in combination with *behave*: see the section about
|
|
|
that one
|
|
|
* in Debian Wheezy, Jessie, and (as of 2016-02-14) Stretch;
|
|
|
but the package is orphaned
|
|
|
* how much do we still need Sikuli if we have dogtail?
|
|
|
* ships a `sniff` GUI that allows one to explore GUI applications
|
|
|
widgets and content using a11y technologies: very useful if we use
|
|
|
dogtail, or any similar tool
|
|
|
* most of the work is done by RedHat
|
|
|
|
|
|
## Misc helpers
|
|
|
|
|
|
* The pyatspi and Accessibility Python modules can be helpful
|
|
|
regardless of the actual tool we use. E.g. pyatspi can allow us to
|
|
|
listen for GUI events. Also see [[!debpkg gir1.2-atspi-2.0]] and
|
|
|
[[!debpkg gir1.2-wnck-3.0]].
|
|
|
* [Accerciser](https://wiki.gnome.org/Apps/Accerciser), an interactive
|
|
|
accessibility explorer for the GNOME desktop
|
|
|
|
|
|
## autopilot
|
|
|
|
|
|
* [homepage](https://launchpad.net/autopilot)
|
|
|
* "A tool for functional testing of GUI application."
|
|
|
* in Ubuntu, but not in Debian
|
|
|
* [usage documentation](https://developer.ubuntu.com/api/autopilot/python/1.5.0/)
|
|
|
|
|
|
## LDTP
|
|
|
|
|
|
LDTP is an open source testing tool that uses computer assistive
|
|
|
technology (accessibility) to automate GUIs. It is (was?) used by GNOME,
|
|
|
Mozilla and others:
|
|
|
|
|
|
* [[!wikipedia Linux_Desktop_Testing_Project]]
|
|
|
* [homepage](http://ldtp.freedesktop.org/wiki/)
|
|
|
* latest upstream release (as of 2016-02-14) is 3.5, released mid-2013
|
|
|
* Debian Wheezy, Jessie and (as of 2016-02-14) Stretch have 2.3.1,
|
|
|
released 2012-02-26
|
|
|
* [tutorial](http://download.freedesktop.org/ldtp/doc/ldtp-tutorial.pdf)
|
|
|
* The main bindings are Python, but there also are a Ruby client and
|
|
|
Perl bindings in the [Git repo](http://cgit.freedesktop.org/ldtp/ldtp2/tree/ldtp)
|
|
|
* The LDTP dev mailing list is very quiet, and it's unclear whether
|
|
|
GNOME still uses it, or instead switched to dogtail.
|
|
|
|
|
|
## misc
|
|
|
|
|
|
- Martin Pitt
|
|
|
[announces](http://www.piware.de/2013/02/umockdev-record-and-mock-hardware-for-debugging-and-testing/)
|
|
|
umockdev ([source code](https://github.com/martinpitt/umockdev)),
|
|
|
a set of tools to record and mock hardware for debugging and testing
|
|
|
|
|
|
# Open questions
|
|
|
|
|
|
## Using accessibility technologies?
|
|
|
|
|
|
[[!tails_ticket 10721]]
|
|
|
|
|
|
In some cases, it could simplify some testing steps, such as anything
|
|
|
about navigating menus, that we're currently mostly avoiding since
|
|
|
it's hard to do in a robust way with Sikuli.
|
|
|
|
|
|
A downside is that we're not exactly testing how most users interact
|
|
|
with the software. Some upsides are that it would ensure that our
|
|
|
stuff does support accessibility technologies, and that we would have
|
|
|
to maintain much less pictures.
|
|
|
|
|
|
## How to do it
|
|
|
|
|
|
We want to build a high-level Ruby library that helps us generate bits
|
|
|
of Python (that use e.g. dogtail) that we'll then send to the system
|
|
|
under testing (using the remote shell we have already), and run there.
|
|
|
|
|
|
For example, one feature this library should provide would be to
|
|
|
select a particular conversation tab in Pidgin, based on the
|
|
|
conversation tab's name; that is, using _text_ and not a picture; this
|
|
|
can be especially useful when we learn what name we are looking for
|
|
|
dynamically, at run time, and thus can't add the corresponding picture
|
|
|
to Git.
|
|
|
|
|
|
Some stuff won't be doable in a nice, high-level way, so likely we
|
|
|
will need, for some corner cases, lower-level access to the generation
|
|
|
of the Python code that will be sent and run. |