Difference between revisions of "Eiffeltest"

From Liberty Eiffel Wiki
Jump to navigation Jump to search
m
 
(6 intermediate revisions by 2 users not shown)
Line 4: Line 4:
 
Users can use it to run a test-suite of their own, which is a useful tool for their project robustness.
 
Users can use it to run a test-suite of their own, which is a useful tool for their project robustness.
   
It is also used by LibertyEiffel itself to validate, with the provided test-suite, the compiler and the libraries.
+
It is also used by the Liberty Eiffel development team itself to validate the compiler and the libraries, using the provided test-suite.
   
 
== Synopsis ==
 
== Synopsis ==
   
 
* <code>se test directory </code>
 
* <code>se test directory </code>
 
* <code>se test source_file.e [source_file2.e...] </code>
 
   
 
== File handling ==
 
== File handling ==
   
The tools recursively iterates over the directory given as a command-line parameter, looking for test files or over the given test file names.
+
The tools recursively iterate over the directory given as a command-line parameter, looking for test files or over the given test file names.
   
 
Test files are Eiffel source files with special names:
 
Test files are Eiffel source files with special names:
Line 33: Line 31:
 
In each test directory a file named loadpath.se may exist, which is passed to the compiler as loadpath.
 
In each test directory a file named loadpath.se may exist, which is passed to the compiler as loadpath.
   
  +
== Test implementation ==
''Why not remove the 'test_' and 'bad_' substrings from the test names? That reduces clutter and leaves more space for readable, self-documenting filenames.''
 
  +
A test case in general is a program - defined by an Eiffel class - returning 0 in case of success and 1 if the test failed. To ease writing such classes we recommend to insert [[library_class::EIFFELTEST_TOOLS]] and call label_assert("some descriptive message", assertion_condition) for any assertion condition that shall be tested. This class internally counts the assertions and provides an error message in case of failure - even with a sedb_breakpoint if compiled with -sedb. For more advanced, or dynamic labels the feature message_assert can be used, which takes an agent to generate the label. After telling the user about the failed testcase [[library_class::EIFFELTEST_TOOLS]] also ensures to terminate the program with exit code 1. One simple example for a test case is the following:
   
  +
class TEST_EXAMPLE
''The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).''
 
  +
 
  +
insert
''That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26''
 
  +
EIFFELTEST_TOOLS
 
  +
That's a good point to raise, if only to mention that the current test suite (pre-eiffeltest) does a "fuzzy" matching of error messages (using SmartEiffel/tools/commands/same_msg.e).--[[User:FM|FM]] 21:42, 19 Dec 2005 (CET)
 
  +
create {}
  +
make
  +
  +
feature {ANY}
  +
make
  +
-- the root creation procedure
  +
do
  +
label_assert("first test", 2 + 3 = 5)
  +
label_assert("second test", False)
  +
end
  +
end -- class TEST_EXAMPLE
   
 
== Modes that are checked ==
 
== Modes that are checked ==
Line 47: Line 57:
 
|-
 
|-
 
|
 
|
* -boost
+
* -boost
  +
* -boost -bdw_gc
* -flat_check -debug_check
 
  +
* -boost -debug -bdw_gc
 
* -no_check
 
* -no_check
  +
* -no_check -bdw_gc
  +
|
  +
* -flat_check -all_check -debug
 
* -flat_check -require_check
 
* -flat_check -require_check
 
* -flat_check -ensure_check
 
* -flat_check -ensure_check
Line 55: Line 69:
 
* -flat_check -loop_check
 
* -flat_check -loop_check
 
* -flat_check -all_check
 
* -flat_check -all_check
  +
|
  +
* -flat_check -no_check -bdw_gc
  +
* -flat_check -require_check -bdw_gc
  +
* -flat_check -ensure_check -bdw_gc
  +
* -flat_check -invariant_check -bdw_gc
  +
* -flat_check -loop_check -bdw_gc
  +
* -flat_check -all_check -bdw_gc
 
|
 
|
 
* -no_gc -boost
 
* -no_gc -boost
* -no_gc -flat_check -debug_check
+
* -no_gc -boost -debug
 
* -no_gc -no_check
 
* -no_gc -no_check
 
* -no_gc -flat_check -require_check
 
* -no_gc -flat_check -require_check
 
* -no_gc -ensure_check
 
* -no_gc -ensure_check
  +
|-
  +
|
  +
* -no_gc -flat_check -debug
 
* -no_gc -flat_check -invariant_check
 
* -no_gc -flat_check -invariant_check
 
* -no_gc -flat_check -loop_check
 
* -no_gc -flat_check -loop_check
 
* -no_gc -flat_check -all_check
 
* -no_gc -flat_check -all_check
 
|
 
|
  +
* -all_check -debug
* -debug_check
 
 
* -require_check
 
* -require_check
 
* -ensure_check
 
* -ensure_check
Line 71: Line 95:
 
* -loop_check
 
* -loop_check
 
* -all_check
 
* -all_check
  +
* -debug
 
|
 
|
* -no_gc -debug_check
+
* -no_gc -all_check -debug
 
* -no_gc -require_check
 
* -no_gc -require_check
 
* -no_gc -ensure_check
 
* -no_gc -ensure_check
Line 78: Line 103:
 
* -no_gc -loop_check
 
* -no_gc -loop_check
 
* -no_gc -all_check
 
* -no_gc -all_check
  +
|
  +
* -all_check -debug -bdw_gc
  +
* -require_check -bdw_gc
  +
* -ensure_check -bdw_gc
  +
* -invariant_check -bdw_gc
  +
* -loop_check -bdw_gc
  +
* -all_check -bdw_gc
  +
* -debug -bdw_gc
 
|-
 
|-
 
|
 
|
 
* -profile -boost
 
* -profile -boost
* -profile -debug_check
+
* -profile -all_check -debug
 
* -profile -no_check
 
* -profile -no_check
 
* -profile -require_check
 
* -profile -require_check
Line 90: Line 123:
 
|
 
|
 
* -profile -no_gc -boost
 
* -profile -no_gc -boost
* -profile -no_gc -debug_check
+
* -profile -no_gc -all_check -debug
 
* -profile -no_gc -no_check
 
* -profile -no_gc -no_check
 
* -profile -no_gc -require_check
 
* -profile -no_gc -require_check
Line 98: Line 131:
 
* -profile -no_gc -all_check
 
* -profile -no_gc -all_check
 
|
 
|
* -profile -flat_check -debug_check
+
* -profile -boost -bwd_gc
  +
* -profile -all_check -debug -bdw_gc
  +
* -profile -no_check -bdw_gc
  +
* -profile -require_check -bdw_gc
  +
* -profile -ensure_check -bdw_gc
  +
* -profile -invariant_check -bdw_gc
  +
* -profile -loop_check -bdw_gc
  +
* -profile -all_check -bdw_gc
  +
|
  +
* -profile -flat_check -all_check -debug
 
* -profile -flat_check -require_check
 
* -profile -flat_check -require_check
 
* -profile -flat_check -ensure_check
 
* -profile -flat_check -ensure_check
Line 104: Line 146:
 
* -profile -flat_check -loop_check
 
* -profile -flat_check -loop_check
 
* -profile -flat_check -all_check
 
* -profile -flat_check -all_check
  +
|-
 
|
 
|
 
* -no_split -boost
 
* -no_split -boost
 
* -no_split -no_gc -boost
 
* -no_split -no_gc -boost
  +
* -no_split -boost -bdw_gc
 
|}
 
|}
 
   
 
== Open issues ==
 
== Open issues ==
Line 131: Line 174:
   
 
''Nice idea. I would improve it by proposing the opposite: list the modes/options to exclude. This means that all modes will be checked by default. If the test is incompatible with some mode, then it has to be specified. If some new mode is added, this mode will be tested with all tests. If some tests are not compatible with this mode, they have to be excluded. Otherway, you would have to edit about every test to add this mode. -- Philippe Ribet
 
''Nice idea. I would improve it by proposing the opposite: list the modes/options to exclude. This means that all modes will be checked by default. If the test is incompatible with some mode, then it has to be specified. If some new mode is added, this mode will be tested with all tests. If some tests are not compatible with this mode, they have to be excluded. Otherway, you would have to edit about every test to add this mode. -- Philippe Ribet
  +
  +
''Why not remove the 'test_' and 'bad_' substrings from the test names? That reduces clutter and leaves more space for readable, self-documenting filenames.''
  +
  +
''The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).''
  +
  +
''That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26''
  +
  +
That's a good point to raise, if only to mention that the current test suite (pre-eiffeltest) does a "fuzzy" matching of error messages (using SmartEiffel/tools/commands/same_msg.e).--[[User:FM|FM]] 21:42, 19 Dec 2005 (CET)

Latest revision as of 14:01, 9 September 2018

eiffeltest is a tool that runs a suite of tests.

Users can use it to run a test-suite of their own, which is a useful tool for their project robustness.

It is also used by the Liberty Eiffel development team itself to validate the compiler and the libraries, using the provided test-suite.

Synopsis

  • se test directory

File handling

The tools recursively iterate over the directory given as a command-line parameter, looking for test files or over the given test file names.

Test files are Eiffel source files with special names:

  • test_*.e: valid source file that should be compilable and runnable without causing an error
  • bad_*.e: invalid source file that should trigger a given compiler error message

For each test file, there can be optional files that have the same name but a different extension. These optional files can be used to provide:

  • output that the compiler is expected to provide when compiling bad_*.e files. This mandatory file for invalid tests has the extension .msg.
  • input to be fed to the program. This optional file (allowed only for valid tests) has the extension .in.
  • output that the program is expected to provide when run. This optional file (allowed only for valid tests) has the extension .out
  • glu_*.c: ***
  • cecil_*.se: ***
  • extra_options_*.se: ***

In each test directory a file named loadpath.se may exist, which is passed to the compiler as loadpath.

Test implementation

A test case in general is a program - defined by an Eiffel class - returning 0 in case of success and 1 if the test failed. To ease writing such classes we recommend to insert library_class::EIFFELTEST_TOOLS and call label_assert("some descriptive message", assertion_condition) for any assertion condition that shall be tested. This class internally counts the assertions and provides an error message in case of failure - even with a sedb_breakpoint if compiled with -sedb. For more advanced, or dynamic labels the feature message_assert can be used, which takes an agent to generate the label. After telling the user about the failed testcase library_class::EIFFELTEST_TOOLS also ensures to terminate the program with exit code 1. One simple example for a test case is the following:

class TEST_EXAMPLE

insert
   EIFFELTEST_TOOLS

create {}
   make

feature {ANY}
   make
      -- the root creation procedure
      do
         label_assert("first test", 2 + 3 = 5)
         label_assert("second test", False)
      end
end -- class TEST_EXAMPLE

Modes that are checked

Eiffeltest checks the following modes (presuming they are not disabled by the excluded.lst file)

  • -boost
  • -boost -bdw_gc
  • -boost -debug -bdw_gc
  • -no_check
  • -no_check -bdw_gc
  • -flat_check -all_check -debug
  • -flat_check -require_check
  • -flat_check -ensure_check
  • -flat_check -invariant_check
  • -flat_check -loop_check
  • -flat_check -all_check
  • -flat_check -no_check -bdw_gc
  • -flat_check -require_check -bdw_gc
  • -flat_check -ensure_check -bdw_gc
  • -flat_check -invariant_check -bdw_gc
  • -flat_check -loop_check -bdw_gc
  • -flat_check -all_check -bdw_gc
  • -no_gc -boost
  • -no_gc -boost -debug
  • -no_gc -no_check
  • -no_gc -flat_check -require_check
  • -no_gc -ensure_check
  • -no_gc -flat_check -debug
  • -no_gc -flat_check -invariant_check
  • -no_gc -flat_check -loop_check
  • -no_gc -flat_check -all_check
  • -all_check -debug
  • -require_check
  • -ensure_check
  • -invariant_check
  • -loop_check
  • -all_check
  • -debug
  • -no_gc -all_check -debug
  • -no_gc -require_check
  • -no_gc -ensure_check
  • -no_gc -invariant_check
  • -no_gc -loop_check
  • -no_gc -all_check
  • -all_check -debug -bdw_gc
  • -require_check -bdw_gc
  • -ensure_check -bdw_gc
  • -invariant_check -bdw_gc
  • -loop_check -bdw_gc
  • -all_check -bdw_gc
  • -debug -bdw_gc
  • -profile -boost
  • -profile -all_check -debug
  • -profile -no_check
  • -profile -require_check
  • -profile -ensure_check
  • -profile -invariant_check
  • -profile -loop_check
  • -profile -all_check
  • -profile -no_gc -boost
  • -profile -no_gc -all_check -debug
  • -profile -no_gc -no_check
  • -profile -no_gc -require_check
  • -profile -no_gc -ensure_check
  • -profile -no_gc -invariant_check
  • -profile -no_gc -loop_check
  • -profile -no_gc -all_check
  • -profile -boost -bwd_gc
  • -profile -all_check -debug -bdw_gc
  • -profile -no_check -bdw_gc
  • -profile -require_check -bdw_gc
  • -profile -ensure_check -bdw_gc
  • -profile -invariant_check -bdw_gc
  • -profile -loop_check -bdw_gc
  • -profile -all_check -bdw_gc
  • -profile -flat_check -all_check -debug
  • -profile -flat_check -require_check
  • -profile -flat_check -ensure_check
  • -profile -flat_check -invariant_check
  • -profile -flat_check -loop_check
  • -profile -flat_check -all_check
  • -no_split -boost
  • -no_split -no_gc -boost
  • -no_split -boost -bdw_gc

Open issues

  • How can we use .in and .out files on platforms that do not allow redirecting the standard input and output of programs?

Easy: just finish the "exec" cluster ;-) --Cyril 08:16, 10 nov 2005 (CET)

Sounds good. Can you give a brief listing of what is to be done ? --pini 23:36, 25 nov 2005 (CET)

The win32 port (at least) is to be written. Ah, and testing ;-) --Cyril 15:39, 28 Nov 2005 (CET)

  • Which compilation modes are used for testing? There are many options and running all possibilities is probably too much.

Flags I can remember from: -no_gc, -flat_check, -no_split, (-debug_check | -all_check | -loop_check | -invariant_check | -ensure_check | -require_check | -no_check | -boost) This means compiling and running about 64 times (minus incompatibilities between -flat_check and boost/no_check).

  • How to test some specific capabilities: options -profile, -no_main, -c_mode, -sedb, -cecil, or testing with c2j?
  • What about tests specific to some compiler mode? For example GC tests should not be run without GC, optimizer tests are invalid in modes other than -boost...

Perhaps we could use indexing/notes clauses with tags indicating which modes/options are relevant for the test? --Dmoisset 22:44, 26 Nov 2005 (CET)

Nice idea. I would improve it by proposing the opposite: list the modes/options to exclude. This means that all modes will be checked by default. If the test is incompatible with some mode, then it has to be specified. If some new mode is added, this mode will be tested with all tests. If some tests are not compatible with this mode, they have to be excluded. Otherway, you would have to edit about every test to add this mode. -- Philippe Ribet

Why not remove the 'test_' and 'bad_' substrings from the test names? That reduces clutter and leaves more space for readable, self-documenting filenames.

The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).

That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26

That's a good point to raise, if only to mention that the current test suite (pre-eiffeltest) does a "fuzzy" matching of error messages (using SmartEiffel/tools/commands/same_msg.e).--FM 21:42, 19 Dec 2005 (CET)