Eiffeltest

eiffeltest is a tool that runs a suite of tests.

Users can use it to run a test-suite of their own, which is a useful tool for their project robustness.

It is also used by the Liberty Eiffel development team itself to validate the compiler and the libraries, using the provided test-suite.

Synopsis

• se test directory

File handling

The tools recursively iterate over the directory given as a command-line parameter, looking for test files or over the given test file names.

Test files are Eiffel source files with special names:

• test_*.e: valid source file that should be compilable and runnable without causing an error
• bad_*.e: invalid source file that should trigger a given compiler error message

For each test file, there can be optional files that have the same name but a different extension. These optional files can be used to provide:

• output that the compiler is expected to provide when compiling bad_*.e files. This mandatory file for invalid tests has the extension .msg.
• input to be fed to the program. This optional file (allowed only for valid tests) has the extension .in.
• output that the program is expected to provide when run. This optional file (allowed only for valid tests) has the extension .out

• glu_*.c: ***
• cecil_*.se: ***
• extra_options_*.se: ***

In each test directory a file named loadpath.se may exist, which is passed to the compiler as loadpath.

Test implementation

A test case in general is a program - defined by an Eiffel class - returning 0 in case of success and 1 if the test failed. To ease writing such classes we recommend to insert library_class::EIFFELTEST_TOOLS and call label_assert("some descriptive message", assertion_condition) for any assertion condition that shall be tested. This class internally counts the assertions and provides an error message in case of failure - even with a sedb_breakpoint if compiled with -sedb. For more advanced, or dynamic labels the feature message_assert can be used, which takes an agent to generate the label. After telling the user about the failed testcase library_class::EIFFELTEST_TOOLS also ensures to terminate the program with exit code 1. One simple example for a test case is the following:

class TEST_EXAMPLE

insert
   EIFFELTEST_TOOLS

create {}
   make

feature {ANY}
   make
      -- the root creation procedure
      do
         label_assert("first test", 2 + 3 = 5)
         label_assert("second test", False)
      end
end -- class TEST_EXAMPLE

Modes that are checked

Eiffeltest checks the following modes (presuming they are not disabled by the excluded.lst file)

-boost -boost -bdw_gc -boost -debug -bdw_gc -no_check -no_check -bdw_gc	-flat_check -all_check -debug -flat_check -require_check -flat_check -ensure_check -flat_check -invariant_check -flat_check -loop_check -flat_check -all_check	-flat_check -no_check -bdw_gc -flat_check -require_check -bdw_gc -flat_check -ensure_check -bdw_gc -flat_check -invariant_check -bdw_gc -flat_check -loop_check -bdw_gc -flat_check -all_check -bdw_gc	-no_gc -boost -no_gc -boost -debug -no_gc -no_check -no_gc -flat_check -require_check -no_gc -ensure_check
-no_gc -flat_check -debug -no_gc -flat_check -invariant_check -no_gc -flat_check -loop_check -no_gc -flat_check -all_check	-all_check -debug -require_check -ensure_check -invariant_check -loop_check -all_check -debug	-no_gc -all_check -debug -no_gc -require_check -no_gc -ensure_check -no_gc -invariant_check -no_gc -loop_check -no_gc -all_check	-all_check -debug -bdw_gc -require_check -bdw_gc -ensure_check -bdw_gc -invariant_check -bdw_gc -loop_check -bdw_gc -all_check -bdw_gc -debug -bdw_gc
-profile -boost -profile -all_check -debug -profile -no_check -profile -require_check -profile -ensure_check -profile -invariant_check -profile -loop_check -profile -all_check	-profile -no_gc -boost -profile -no_gc -all_check -debug -profile -no_gc -no_check -profile -no_gc -require_check -profile -no_gc -ensure_check -profile -no_gc -invariant_check -profile -no_gc -loop_check -profile -no_gc -all_check	-profile -boost -bwd_gc -profile -all_check -debug -bdw_gc -profile -no_check -bdw_gc -profile -require_check -bdw_gc -profile -ensure_check -bdw_gc -profile -invariant_check -bdw_gc -profile -loop_check -bdw_gc -profile -all_check -bdw_gc	-profile -flat_check -all_check -debug -profile -flat_check -require_check -profile -flat_check -ensure_check -profile -flat_check -invariant_check -profile -flat_check -loop_check -profile -flat_check -all_check
-no_split -boost -no_split -no_gc -boost -no_split -boost -bdw_gc

Open issues

• How can we use .in and .out files on platforms that do not allow redirecting the standard input and output of programs?

Easy: just finish the "exec" cluster ;-) --Cyril 08:16, 10 nov 2005 (CET)

Sounds good. Can you give a brief listing of what is to be done ? --pini 23:36, 25 nov 2005 (CET)

The win32 port (at least) is to be written. Ah, and testing ;-) --Cyril 15:39, 28 Nov 2005 (CET)

• Which compilation modes are used for testing? There are many options and running all possibilities is probably too much.

Flags I can remember from: -no_gc, -flat_check, -no_split, (-debug_check | -all_check | -loop_check | -invariant_check | -ensure_check | -require_check | -no_check | -boost) This means compiling and running about 64 times (minus incompatibilities between -flat_check and boost/no_check).

• How to test some specific capabilities: options -profile, -no_main, -c_mode, -sedb, -cecil, or testing with c2j?

• What about tests specific to some compiler mode? For example GC tests should not be run without GC, optimizer tests are invalid in modes other than -boost...

Perhaps we could use indexing/notes clauses with tags indicating which modes/options are relevant for the test? --Dmoisset 22:44, 26 Nov 2005 (CET)

Nice idea. I would improve it by proposing the opposite: list the modes/options to exclude. This means that all modes will be checked by default. If the test is incompatible with some mode, then it has to be specified. If some new mode is added, this mode will be tested with all tests. If some tests are not compatible with this mode, they have to be excluded. Otherway, you would have to edit about every test to add this mode. -- Philippe Ribet

Why not remove the 'test_' and 'bad_' substrings from the test names? That reduces clutter and leaves more space for readable, self-documenting filenames.

The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).

That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26

That's a good point to raise, if only to mention that the current test suite (pre-eiffeltest) does a "fuzzy" matching of error messages (using SmartEiffel/tools/commands/same_msg.e).--FM 21:42, 19 Dec 2005 (CET)