Test Generator for ISO Modula-2

By Tom Breeden (tmb@virginia.edu)
May 20, 2004

Overview

tgM2 is a program to aide in unit testing of Modula-2 software. It takes as input a user-prepared test script (*.ts file) which specifies one or more tests of a procedure or code block to be executed and reported upon. tgM2 writes out a Modula-2 program for this test suite. The output program itself writes its results to standard output.

tgM2 owes its existence and most of its architecture to the Ada 95 test driver generator program tg for Ada written by Andre Spiegel, along with the version of tg for Modula-2 (still an Ada 95 program) adapted to generate Modula-2 scripts done by Ralf Reissing.

I produced tgM2 for two reasons. First, I wanted to remove the necessity of obtaining an Ada compiler in order to build the program. Second, I wanted to use ISO Modula-2 to restore the capabilities of the original Ada driver generation that had been lost in adaptation to a pre-ISO M2 language base. Specifically, these depended on the EXCEPT language construct in the ISO definition. Exception handing provides the possibility of generating more robust test scripts as well as the ability to test that situations which should cause exception execution actually do so.

The program was developed and tested on Win2K using Stony Brook Modula-2 and on an Amiga using a semi-ISO (unreleased) M2 compiler for Amiga (M68040).

The files in the download tgM2.zip include the complete sources of tgM2 and a Windows executable, plus some demo script files, and support module sources.

My own use of tgM2 has just begun, so it must be considered to be in a beta state. Your bug reports and other comments are appreciated.

Test Scripts	Test Script Layout	Test Script Syntax
Demo Test Script: ArithExpressions.ts	Test Driver Startup	Associated Modules
tgM2 Modules and Files	Notes

Test Scripts

The test script you prepare is a text file (with extension ".ts") containing all the information necessary for tgM2 to generate a "driver" program source file that can be compiled and linked to run the test suite.

The different sections of the test script are:

General Section - can occur once, at the beginning of the script.

Fail/Error Handling Specification
- "stop" at first failure/error or "continue"
Context
- the IMPORT statements necessary
Global Definition
- variables used by all the test cases
- CONST and TYPE statements for global use.
- Procedures called from the Test Code section.

Code Sections - can occur multiple times, and interspersed with Test Case Parts

Code Section
- code to be executed before/between Test Cases.
- Code Sections and Test Case Sections are executed in the order in which they appear in the script file.

Test Case Sections - can occur multiple times to create a test suite

Title
- descriptive title for the test case
Definitions
- variables used by this test case
- CONST and TYPE statements for this test case.
- Procedures called within this test case only.
Preparation Code
- statements executed prior to the test itself
Test Code
- statements that perform the test itself
Test Result Predicate(s)
- boolean expressions to be evaluated after the Test Code
- or within an EXCEPT block caused by the test code
- all must evaluate TRUE for the test to be marked "Pass"
CleanUp Code
- statements executed after to the test itself

Test Script Layout

The *.ts file is a "column 1 significant" text file; ie, if a non-blank character appears in column 1, it must be a "keyword" (such as "define", "test", "pass"). Other material is (mainly) Modula-2 statements or phrases, which can encompass more than one line, so long as column 1 is blank.

Comments in the *.ts file are in M2 comment format, but are never passed onto the driver program. (currently nested comments are not supported).

Most of the sections or subsections above are optional. However, there must be a Test Case Title, a Test Code subsection, and a Test Result Predicate. Subsections must appear in the order given above.

Keywords are not case sensitive

Test Script Syntax

General Section

Keyword: fail_handling stop | fail_handling continue
Keyword: error_handling stop | error_handling continue
Keyword: context <IMPORT statement> {<IMPORT statement>}
Keyword: define <variables list>:<Type>; {<M2 definitions and procedures >}
- Note: No VAR in front of the initial variables list.

Code Sections

Keyword: code <M2 definitions and code>
- Note: Will be wrapped into a procedure constructed by tgM2

Test Case Sections

Keyword: ***** <text>
- Note: The test case will be wrapped into a procedure constructed by tgM2
Keyword: define <variables list>:<Type>; {<M2 definitions and procedures >}
- Note: No VAR in front of the initial variables list.
Keyword: prepare <M2 code>
Keyword: test <M2 code>
Keyword: pass <boolean expression>
Keyword: pass exception <boolean expression>
- Note: the "pass exception" predicate is placed into the procedure's EXCEPT block.
- Note: Multiple instances of either of the pass keywords may appear.
Keyword: cleanup <M2 code>

Demo Test Script: ArithExpressions.ts

(* Test script for some Arithmetic expressions with a Modula-2 compiler *)
(*       051704     for Stony Brook  *)

fail_handling continue
error_handling stop

context FROM M2EXCEPTION IMPORT IsM2Exception;
        <*/NOOPTIMIZE*>  (* Otherwise, some of the simple tests below disappear! *)

***** INTEGER Overflow
define  Result, I, J, K  :INTEGER16;
prepare <*/PUSH*>
        <*/CHECK:O*>
        Result := 0;
        I := 30000; J := -30000;
test    Result := I-J;
pass    exception IsM2Exception()
cleanup <*/POP*>

***** INTEGER Overflow without checking
define  Result, I, J, K  :INTEGER16;
prepare <*/PUSH*>
        <*/NOCHECK:O*>
        Result := 0;
        I := 30000; J := -30000;
test    Result := I-J;
pass    TRUE
cleanup <*/POP*>

Test Driver Startup

tgM2: A Test Driver Generator for Modula-2 programs (v0.9-051804).

Usage:   tgM2 [options...] script_file [driver_file]

Options template is "-f=-fail/K,-p=-pass/K,-script/A,-driver,-overwrite/S,-h=-help/S"
                           /A = always required
                           /K = option keyword required
                           /S = boolean switch

-f    Specifies how failed test cases will be reported
               by the testdriver.  may be one of

               off      No output.
               numbers  Only test case numbers, followed by the string
                        "FAIL.".
               titles   Numbers and titles, followed by "...FAIL."
                        on the next line.
               full     Numbers and titles, "...FAIL" and a short
                        explanation on the next line.
                        This is the default.

-p    Same as "-f", but for passed test cases.
               Default is "numbers".

-overwrite     Overwrite any existing driver file of the same name

The script_file must have the extension ".ts".

If the driver_file is not specified, the name is constructed from the
script_file with extension changed to ".mod".

Associated Modules

	ISO Lib Support Modules		Modules to help in the use of ISO Lib procs for reading command line arguments and text files.
	Null delimited static strings		Modules for processing null char delimited static strings.
	Dynamic strings		A number of modules for processing dynamically allocated and unbounded length strings.
	List processing		Simple generalized list processing module
	Misc		Miscellaneous support modules

tgM2 Modules and Files

Test Scripts
Demo1.ts from Ada 95 tg package.
ArithExpressions.ts some arithmetic overflow tests with exceptions.
TsttgM2Parser.ts a more complex script doing a bit of testing of tgM2's parsing modules.

Program Modules
tgM2
tgM2TranslatorISO
tgM2Parser
tgM2Scanner
tgM2Exceptions
tgM2Utils
tgM2Version

tgM2.exe Win32 executable
tgM2.sbp Stony Brook Modula2 project file

ISO Lib Support Modules
ArgsSupport specification, reading, and initial parsing of command line arguments.
TextIOHelper eases the use of the ISO standard TextIO input routines.

Null delimited static strings
Str0 Basic procedures.
Str1 More procedures.

Dynamic strings
DynStr0 Basic procedures, size fixed at create time.
DynStr1 Common procedures.
DynStr1b some Strings module flavor procs.
DynStr1NC Compare procedures, case insensitive.
DynStr1C Common procedures, one character parameter.
DynStr3 As DynStr0 and DynStr1, but size will be reallocated on assignment.
TextBlocks In-memory, line-oriented, dynamically-extending blocks of text.

List processing
ListV2Processor Simple generalized list processing module.

Misc
MaxMin Maximum and Minimum functions for CARDs, INTs, REALs.
Debugging0 Assert and Debug procs.
TBChans used only by tsttgM2Parser.ts

Notes

"fail" vs. "error" - A test case is marked "fail" if any of the pass or pass exception predicates are false.
It is also marked "fail" if an exception occurred within the test subsection but there was no pass exception keyword, or, similarly, if there was a pass exception keyword but no exception occurred within the test subsection.
If there is an exception anywhere outside of the test subsection then the test case (and the overall run result) is marked "error".
The Windows executable was created in the Stony Brook M2 v4, build 29 system.
Using an optimizing compiler, for some very simple test and pass combinations it may be necessary to turn off optimization or take other steps to insure that the compiler does not remove some of your statements. (see ArithExpressions.ts)
The supplied module Debugging0's implementation is Windows specific.