Eiffeltest
This page specifies a tool that has not been written yet.
eiffeltest is a tool that runs a suite of tests. This tool allows to validate, with the provided test-suite, the compiler and the libraries. Users can use it to run a test-suite of their own, which is a usefull tool for their project robustness.
Synopsis
eiffeltest directory
eiffeltest source_file.e [source_file2.e...]
File handling
The tools recursively iterates over the directory given as a command-line parameter, looking for test files or over the given test file names.
Test files are Eiffel source files with special names:
- test_*.e: valid source file that should be compilable and runnable without causing an error
- bad_*.e: invalid source file that should trigger a given compiler error message
For each test file, there can be optional files that have the same name but a different extension. These optional files can be used to provide:
- output that the compiler is expected to provide when compiling bad_*.e files. This file mandatory file for invalid tests has the file name extension .msg
- input data to be fed to the program. This optional file (allowed only for valid tests) has the file name extension .in
- output that the program is expected to provide when run. This optional file (allowed only for valid tests) has the file name extension .out
Why not remove the 'test_' and 'bad_' substrings from the test names? That reduces clutter and leaves more space for readable, self-documenting filenames.
The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).
That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26
Open issues
- How can we use .in and .out files on platforms that do not allow redirecting the standard input and output of programs?
Easy: just finish the "exec" cluster ;-) --Cyril 08:16, 10 nov 2005 (CET)
Sounds good. Can you give a brief listing of what is to be done ? --pini 23:36, 25 nov 2005 (CET)
- Which compilation modes are used for testing? There are many options and running all possibilities is probably too much.
Flags I can remember from: -no_gc, -flat_check, -no_split, (-debug_check | -all_check | -loop_check | -invariant_check | -ensure_check | -require_check | -no_check | -boost) This means compiling and running about 64 times (minus incompatibilities between -flat_check and boost/no_check).
- How to test some specific capabilities: options -profile, -no_main, -c_mode, -sedb, -cecil, or testing with c2j?
- What about tests specific to some compiler mode? For example GC tests should not be run without GC, optimizer tests are invalid in modes other than -boost...
Perhaps we could use indexing/notes clauses with tags indicating which modes/options are relevant for the test? --200.45.81.188 22:43, 26 Nov 2005 (CET)