Proposed Test Suite Design

Michael Hope (michaelh@juju.net.nz)



Abstract

This article describes the goals, requirements, and suggested
specification for a test suite for the output of the Small
Device C Compiler (sdcc). Also included is a short list
of existing works. 

 Goals

The main goals of a test suite for sdcc are 

 To allow developers to run regression tests to check that
  core changes do not break any of the many ports. 

 To verify the core. 

 To allow developers to verify individual ports. 

 To allow developers to test port changes. 

This design only covers the generated code. It does not cover
a test/unit test framework for the sdcc application itself,
which may be useful.

One side effect of (1) is that it requires that the individual
ports pass the tests originally. This may be too hard. See
the section on Exceptions below.

 Requirements

 Coverage

The suite is intended to cover language features only. Hardware
specific libraries are explicitly not covered.

 Permutations

The ports often generate different code for handling different
types (Byte, Word, DWord, and the signed forms). Meta information
could be used to permute the different test cases across
the different types.

 Exceptions

The different ports are all at different levels of development.
Test cases must be able to be disabled on a per port basis.
Permutations also must be able to be disabled on a port
level for unsupported cases. Disabling, as opposed to enabling,
on a per port basis seems more maintainable.

 Running

The tests must be able to run unaided. The test suite must
run on all platforms that sdcc runs on. A good minimum may
be a subset of Unix command set and common tools, provided
by default on a Unix host and provided through cygwin on
a Windows host.

The tests suits should be able to be sub-divided, so that
the failing or interesting tests may be run separately.

 Artifcats

The test code within the test cases should not generate artifacts.
An artifact occurs when the test code itself interferes
with the test and generates an erroneous result.

 Emulators

sdcc is a cross compiling compiler. As such, an emulator
is needed for each port to run the tests.

 Existing works

 DejaGnu

DejaGnu is a toolkit written in Expect designed to test an
interactive program. It provides a way of specifying an
interface to the program, and given that interface a way
of stimulating the program and interpreting the results.
It was originally written by Cygnus Solutions for running
against development boards. I believe the gcc test suite
is written against DejaGnu, perhaps partly to test the Cygnus
ports of gcc on target systems.

 gcc test suite

I don't know much about the gcc test suite. It was recently
removed from the gcc distribution due to issues with copyright
ownership. The code I saw from older distributions seemed
more concerned with esoteric features of the language.

 xUnit

The xUnit family, in particular JUnit, is a library of in
test assertions, test wrappers, and test suite wrappers
designed mainly for unit testing. PENDING: More.

 CoreLinux++ Assertion framework

While not a test suite system, the assertion framework is
an interesting model for the types of assertions that could
be used. They include pre-condition, post-condition, invariants,
conditional assertions, unconditional assertions, and methods
for checking conditions.

 Specification

This specification borrows from the JUnit style of unit testing
and the CoreLinux++ style of assertions. The emphasis is
on maintainability and ease of writing the test cases.

 Terms

PENDING: Align these terms with the rest of the world.

 An assertion is a statement of how things should be. PENDING:
  Better description, an example. 

 A test point is the smallest unit of a test suite, and
  consists of a single assertion that passes if the test
  passes. 

 A test case is a set of test points that test a certain
  feature. 

 A test suite is a set of test cases that test a certain
  set of features. 

 Test cases

Test cases shall be contained in their own C file, along
with the meta data on the test. Test cases shall be contained
within functions whose names start with 'test' and which
are descriptive of the test case. Any function that starts
with 'test' will be automatically run in the test suite.

To make the automatic code generation easier, the C code
shall have this format 

 Test functions shall start with 'test' to allow automatic
  detection. 

 Test functions shall follow the K&R intention style for
  ease of detection. i.e. the function name shall start
  in the left column on a new line below the return specification. 

 Assertions

All assertions shall log the line number, function name,
and test case file when they fail. Most assertions can have
a more descriptive message attached to them. Assertions
will be implemented through macros to get at the line information.
This may cause trouble with artifacts.

The following definitions use C++ style default arguments
where optional messages may be inserted. All assertions
use double opening and closing brackets in the macros to
allow them to be compiled out without any side effects.
While this is not required for a test suite, they are there
in case any of this code is incorporated into the main product.

Borrowing from JUnit, the assertions shall include 

 FAIL((String msg = "Failed")).
  Used when execution should not get here. 

 ASSERT((Boolean cond, String msg = "Assertion
  failed"). Fails if cond is false. Parent to REQUIRE
  and ENSURE. 

JUnit also includes may sub-cases of ASSERT, such as assertNotNull,
assertEquals, and assertSame.

CoreLinux++ includes the extra assertions 

 REQUIRE((Boolean cond, String msg = "Precondition
  failed"). Checks preconditions. 

 ENSURE((Boolean cond, String msg = "Postcondition
  failed"). Checks post conditions. 

 CHECK((Boolean cond, String msg = "Check
  failed")). Used to call a function and to check
  that the return value is as expected. i.e. CHECK((fread(in,
  buf, 10) != -1)). Very similar to ASSERT, but the function
  still gets called in a release build. 

 FORALL and EXISTS. Used to check conditions within part
  of the code. For example, can be used to check that a
  list is still sorted inside each loop of a sort routine. 

All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be
available.

 Meta data

PENDING: It's not really meta data.

Meta data includes permutation information, exception information,
and permutation exceptions.

Meta data shall be global to the file. Meta data names consist
of the lower case alphanumerics. Test case specific meta
data (fields) shall be stored in a comment block at the
start of the file. This is only due to style.

A field definition shall consist of 

 The field name 

 A colon. 

 A comma separated list of values. 

The values shall be stripped of leading and trailing white
space.

Permutation exceptions are by port only. Exceptions to a
field are specified by a modified field definition. An exception
definition consists of

 The field name. 

 An opening square bracket. 

 A comma separated list of ports the exception applies for. 

 A closing square bracket. 

 A colon. 

 The values to use for this field for these ports. 

An instance of the test case shall be generated for each
permutation of the test case specific meta data fields.

The runtime meta fields are 

 port - The port this test is running on. 

 testcase - The name of this test case. 

 function - The name of the current function. 

Most of the runtime fields are not very usable. They are
there for completeness.

Meta fields may be accessed inside the test case by enclosing
them in curly brackets. The curly brackets will be interpreted
anywhere inside the test case, including inside quoted strings.
Field names that are not recognised will be passed through
including the brackets. Note that it is therefore impossible
to use some strings within the test case.

Test case function names should include the permuted fields
in the name to reduce name collisions.

 An example

I don't know how to do pre-formatted text in LaTeX. Sigh.

The following code generates a simple increment test for
all combinations of the storage classes and all combinations
of the data sizes. This is a bad example as the optimiser
will often remove most of this code.

/** Test for increment. 

type: char, int, long

Z80 port does not fully support longs (4 byte)

type[z80]: char, int

class: "", register, static */

static void

testInc{class}{types}(void)

{

{class} {type} i = 0; 

i = i + 1;

ASSERT((i == 1));

}