Difference between revisions of "Eiffeltest"

From Liberty Eiffel Wiki
Jump to navigation Jump to search
 
Line 30: Line 30:
 
''The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).''
 
''The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).''
   
''That is how I'm doing testing for the Amber project, and it seems to work well.''
+
''That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26''
   
 
== Open issues ==
 
== Open issues ==

Revision as of 16:36, 26 November 2005


This page specifies a tool that has not been written yet.

eiffeltest is a tool that runs a suite of tests. This tool allows to validate, with the provided test-suite, the compiler and the libraries. Users can use it to run a test-suite of their own, which is a usefull tool for their project robustness.

Synopsis

  • eiffeltest directory
  • eiffeltest source_file.e [source_file2.e...]

File handling

The tools recursively iterates over the directory given as a command-line parameter, looking for test files or over the given test file names.

Test files are Eiffel source files with special names:

  • test_*.e: valid source file that should be compilable and runnable without causing an error
  • bad_*.e: invalid source file that should trigger a given compiler error message

For each test file, there can be optional files that have the same name but a different extension. These optional files can be used to provide:

  • output that the compiler is expected to provide when compiling bad_*.e files. This file mandatory file for invalid tests has the file name extension .msg
  • input data to be fed to the program. This optional file (allowed only for valid tests) has the file name extension .in
  • output that the program is expected to provide when run. This optional file (allowed only for valid tests) has the file name extension .out

Why not remove the 'test_' and 'bad_' substrings from the test names? That reduces clutter and leaves more space for readable, self-documenting filenames.

The test harness can look for response files that match the test prefix. You already have two suffixes: *.out (for output) and *.msg (for a compiler error message). Another useful one is *.match (for a regular expression match on the output, because the output might contain things like date/time/version-number that are not relevant to the test).

That is how I'm doing testing for the Amber project, and it seems to work well. --Roger Browne 2005-11-26

Open issues

  • How can we use .in and .out files on platforms that do not allow redirecting the standard input and output of programs?

Easy: just finish the "exec" cluster ;-) --Cyril 08:16, 10 nov 2005 (CET)

Sounds good. Can you give a brief listing of what is to be done ? --pini 23:36, 25 nov 2005 (CET)

  • Which compilation modes are used for testing? There are many options and running all possibilities is probably too much.

Flags I can remember from: -no_gc, -flat_check, -no_split, (-debug_check | -all_check | -loop_check | -invariant_check | -ensure_check | -require_check | -no_check | -boost) This means compiling and running about 64 times (minus incompatibilities between -flat_check and boost/no_check).

  • How to test some specific capabilities: options -profile, -no_main, -c_mode, -sedb, -cecil, or testing with c2j?
  • What about tests specific to some compiler mode? For example GC tests should not be run without GC, optimizer tests are invalid in modes other than -boost...