Alex headshot

AlBlue’s Blog

Macs, Modularity and More

Push-button testing with auto test

2006 Test

Andreas Leitner has just given a talk on auto_test, a system for automatically generating (and running) test cases. The idea is that if you have a language (like Eiffel) that supports contracts, you can safely generate random objects and arguments to methods, and then execute them to see whether you have violated any contracts, and if so, to report on those failures. Other languages (like Java) are supported through extensions like OCL or jContract.

I think this approach is likely to work well for detecting edge cases that you've not catered for specifically enough in the preconditions of methods. For example, if you have a function that doesn't deal well with a Void input, but haven't declared it as a precondition, this type of approach is likely to work out well.

The strategies of what kind of test data to generate are pluggable; so whilst there's a RandomStrategy for generating method calls, it could be supplanted with other aspects. When a test (precondition) failure occurs, the failure is logged and the content of the test case is distilled into a replayable condition. A minimizer can try (unsoundly) to reduce the error case into an equivalent (but shorter) form -- but as it's unsound, it means that you might not get the same result. Fortunately, it's relatively easy to try running the test again, and it can automatically find out when you still have the same failure or not.

Given that it's random, you can weight the tests on a per method or class basis. You can also say how long to test; for example, if you're going away for lunch, you can get it to schedule it for half an hour. you could also schedule it to run over an evening.

In all, it sounds like a really neat idea; but it only works if your language has assertable conditions sprinkled liberally over your code. If you have no assertable conditions, then generating a large number of methods is possibly not going to be useful. Also, it is likely to be difficult to test if the methods have a lot of unconnected side effects; for example, class C1 and C2 might work OK separately, but not together. (The framework does handle this case, but you need to be aware that it will take longer and longer as the number of classes under test increase.)

It's available under an open-source license, so you can read up more and download it from the auto_test homepage. The video is now available.