Other tools for Acceptance Testing

There are hundreds of tools out there that aim to do automated testing at a more or less high level. This is only a brief survey of what we consider to be the most interesting ones (other than our own) and how they compare to using the tools available here.

It seems simplest to break the problem down into application types, which have different considerations and in some cases different tools.

  1. Non-interactive calculators. Here I mean a tool which essentially takes some input, performs some calculation and produces some output without any interaction from a user other than starting it. Such tools may not have a GUI, or the GUI may be entirely focussed on setting up the input and displaying the output, and as such testing the GUI can be considered as a separate problem.

  2. Interactive tools with a custom GUI. Here I include anything using a standard GUI toolkit, and also interactive console programs controlled via standard input responses. Acceptance Testing them is viewed as testing their “use cases” - modelling a series of GUI actions from a user (workflow).

  3. Web applications. Anything run over HTTP in a standard web browser - not including Flash, ActiveX and things like that.

Non-interactive calculators

Here our candidate tool is TextTest alone, with no need for simulating user input. The main alternative I have come across is Fit using ColumnFixture (or FitLibrary using CalculateFixture, but these are not very different as far as I can see). Here is an example:




This is a very compact and user-friendly format for specifying lots of tests for a small calculator. The first three columns specify numeric inputs and the final column the expected output. There is a limit to how big such a table can get though before it ceases to be a nice expression format, both in terms of numbers of relevant input and output values and their size (very long strings will probably not read well in this format). For a larger “calculator”, it is better to separate each test and then TextTest seems a better candidate – it's easy to keep all the details of the data out of the way unless you really want to look at it.

While it may be relatively unusual that an entire system is of this form, it is much more common that parts of it are. It can be very useful to identify them and test them separately in this way.

Here is a table comparing the tools.

Custom GUIs

Here our candidate tool is TextTest with PyUseCase or JUseCase. Suddenly there are more alternatives, both that try to make use of the GUI and that bypass it and call into the system via an API instead. I have picked out four that I regard as especially interesting.

  • MarathonMan – a record/playback tool for Java Swing GUIs that was originally built by two guys at Thoughtworks aiming to make record/playback a bit more agile. Tests are recorded in Python in terms of the GUI mechanics: rather than user intent. Useful mainly as a comparison with other modern record/playback tools. Naturally there are many other tools of this type, of which Abbot is probably the most popular.

  • Fit with ActionFixture – uses HTML tables to define tests against an API (i.e. bypasses the GUI). ActionFixture is essentially a simple “test language” with four GUI-like keywords: (check, press, select and enter) which calls into the system under test via an API. Included mainly because it's popular and acts as a way to introduce the following two rather more interesting contributions.

  • Fit with FitLibrary/DoFixture – FitLibrary is an extension for Fit that aims to support testing in a domain language (using a meta-language called DoFixture) written by Rick Mugridge. While you could see ActionFixture (above) as a meta-language, it is very clumsy compared to DoFixture and consequently hard to believe that was its intention. FitLibrary contains many other more minor improvements and extensions to the core Fit.

  • Exactor – another API-driven tool focussed entirely on workflow testing. Written by Sean Hanly and Brian Swan at Exoftware. Also primarily focussed on the creation of a domain language for tests. Tests expressed in simple plain text files rather than tables: tables are great for calculations but seem to just get in the way for workflow. Additionally provides “standard commands” that define a language for driving the GUI directly. Some other interesting features lacked by both variants of Fit, notably a means of refactoring the tests at the customer level, pretty similar to our own GUI shortcuts

There is also a table comparing the features of these tools for workflow testing of GUIs.

Web Applications

Again there are many tools, reflecting the proliferation of this kind of application. Our own candidate is now TextTest with WebUseCase. What marks this out as unique in the field? It provides a mechanism for testing with a domain language and it allow you to record tests live, as with custom GUIs.

For comparison, both Fit and Exactor have web testing mechanisms, “HTMLFixture” and “WebCommand”. Neither of these provide support for use of a business domain language, though. There is also Canoo WebTest, which has been around for longest. What all of these share, and also share with WebUseCase, is the use of a browser simulator. WebUseCase, Canoo and Fit all use HtmlUnit, while Exactor uses JWebUnit.

In contrast to these four, two tools are included which have the not inconsiderable advantage of using a real browser. Watir drives Internet Explorer via the DOM, while Selenium has a testing engine written in javascript. Watir also has the advantage of being the only tool that generates a programming language (Ruby) - very useful for long-term refactoring, at least compared to the tool-defined languages in most of the others.

There is also a table comparing these tools.