% ``Test Suite Design''
% $Id$
\documentclass{widearticle}
\usepackage{url}

\begin{document}
\title{Proposed Test Suite Design}
\author{Michael Hope (michaelh@juju.net.nz)}
\date{\today}
\maketitle

\begin{abstract}
This article describes the goals, requirements, and suggested
specification for a test suite for the output of the Small Device C
Compiler (sdcc).  Also included is a short list of existing works.
\end{abstract}

\section{Goals}
The main goals of a test suite for sdcc are
\begin{enumerate}
    \item To allow developers to run regression tests to check that
core changes do not break any of the many ports.
    \item To verify the core.
    \item To allow developers to verify individual ports.
    \item To allow developers to test port changes.
\end{enumerate}

This design only covers the generated code.  It does not cover a
test/unit test framework for the sdcc application itself, which may be
useful.

One side effect of (1) is that it requires that the individual ports
pass the tests originally.  This may be too hard.  See the section on
Exceptions below.

\section{Requirements}
\subsection{Coverage}
The suite is intended to cover language features only.  Hardware
specific libraries are explicitly not covered.

\subsection{Permutations}
The ports often generate different code for handling different types
(Byte, Word, DWord, and the signed forms).  Meta information
could be used to permute the different test cases across the different
types.

\subsection{Exceptions}
The different ports are all at different levels of development.  Test
cases must be able to be disabled on a per port basis.  Permutations
also must be able to be disabled on a port level for unsupported
cases.  Disabling, as opposed to enabling, on a per port basis seems
more maintainable.

\subsection{Running}
The tests must be able to run unaided.  The test suite must run on all
platforms that sdcc runs on.  A good minimum may be a subset of Unix
command set and common tools, provided by default on a Unix host and
provided through cygwin on a Windows host.

The tests suits should be able to be sub-divided, so that the failing
or interesting tests may be run separately.

\subsection{Artifcats}
The test code within the test cases should not generate artifacts.  An
artifact occurs when the test code itself interferes with the test and
generates an erroneous result.

\subsection{Emulators}
sdcc is a cross compiling compiler.  As such, an emulator is needed
for each port to run the tests.

\section{Existing works}
\subsection{DejaGnu}
DejaGnu is a toolkit written in Expect designed to test an interactive
program.  It provides a way of specifying an interface to the program,
and given that interface a way of stimulating the program and
interpreting the results.  It was originally written by Cygnus
Solutions for running against development boards.  I believe the gcc
test suite is written against DejaGnu, perhaps partly to test the
Cygnus ports of gcc on target systems.

\subsection{gcc test suite}
I don't know much about the gcc test suite.  It was recently removed
from the gcc distribution due to issues with copyright ownership.  The
code I saw from older distributions seemed more concerned with
esoteric features of the language.

\subsection{xUnit}
The xUnit family, in particular JUnit, is a library of in test
assertions, test wrappers, and test suite wrappers designed mainly for
unit testing.  PENDING: More.

\subsection{CoreLinux++ Assertion framework}
While not a test suite system, the assertion framework is an
interesting model for the types of assertions that could be used.
They include pre-condition, post-condition, invariants, conditional
assertions, unconditional assertions, and methods for checking
conditions.

\section{Specification}
This specification borrows from the JUnit style of unit testing and
the CoreLinux++ style of assertions.  The emphasis is on
maintainability and ease of writing the test cases.

\subsection{Terms}
PENDING: Align these terms with the rest of the world.

\begin{itemize}
    \item An \emph{assertion} is a statement of how things should be.
PENDING: Better description, an example.
    \item A \emph{test point} is the smallest unit of a test suite,
and consists of a single assertion that passes if the test passes.
    \item A \emph{test case} is a set of test points that test a
certain feature.
    \item A \emph{test suite} is a set of test cases that test a
certain set of features.
\end{itemize}

\subsection{Test cases}
Test cases shall be contained in their own C file, along with the meta
data on the test.  Test cases shall be contained within functions
whose names start with 'test' and which are descriptive of the test
case.  Any function that starts with 'test' will be automatically run in
the test suite.

To make the automatic code generation easier, the C code shall have
this format
\begin{itemize}
    \item Test functions shall start with 'test' to allow
automatic detection.
    \item Test functions shall follow the K\&R intention style for ease
of detection.  i.e. the function name shall start in the left
column on a new line below the return specification.
\end{itemize}

\subsection{Assertions}
All assertions shall log the line number, function name, and test
case file when they fail.  Most assertions can have a more descriptive
message attached to them.  Assertions will be implemented through
macros to get at the line information.  This may cause trouble with
artifacts.

The following definitions use C++ style default arguments where
optional messages may be inserted.  All assertions use double opening
and closing brackets in the macros to allow them to be compiled out
without any side effects.  While this is not required for a test
suite, they are there in case any of this code is incorporated into the
main product.

Borrowing from JUnit, the assertions shall include
\begin{itemize}
    \item FAIL((String msg = ``Failed'')).  Used when execution should
not get here.
    \item ASSERT((Boolean cond, String msg = ``Assertion failed'').
Fails if cond is false.  Parent to REQUIRE and ENSURE.
\end{itemize}

JUnit also includes may sub-cases of ASSERT, such as assertNotNull,
assertEquals, and assertSame.

CoreLinux++ includes the extra assertions
\begin{itemize}
    \item REQUIRE((Boolean cond, String msg = ``Precondition
failed'').  Checks preconditions.
    \item ENSURE((Boolean cond, String msg = ``Postcondition
failed'').  Checks post conditions.
    \item CHECK((Boolean cond, String msg = ``Check failed'')).  Used
to call a function and to check that the return value is as expected.
i.e.  CHECK((fread(in, buf, 10) != -1)).  Very similar to ASSERT, but
the function still gets called in a release build.
    \item FORALL and EXISTS.  Used to check conditions within part of
the code.  For example, can be used to check that a list is still
sorted inside each loop of a sort routine.
\end{itemize}

All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.

\subsection{Meta data}
PENDING:  It's not really meta data.

Meta data includes permutation information, exception information, and
permutation exceptions.

Meta data shall be global to the file.  Meta data names consist of the
lower case alphanumerics.  Test case specific meta data (fields) shall
be stored in a comment block at the start of the file.  This is only
due to style.  

A field definition shall consist of
\begin{itemize}
    \item The field name
    \item A colon.
    \item A comma separated list of values.
\end{itemize}

The values shall be stripped of leading and trailing white space.

Permutation exceptions are by port only.  Exceptions to a field are
specified by a modified field definition.  An exception definition
consists of 

\begin{itemize}
    \item The field name.
    \item An opening square bracket.
    \item A comma separated list of ports the exception applies for.
    \item A closing square bracket.
    \item A colon.
    \item The values to use for this field for these ports.
\end{itemize}

An instance of the test case shall be generated for each permutation
of the test case specific meta data fields.

The runtime meta fields are
\begin{itemize}
    \item port - The port this test is running on.
    \item testcase - The name of this test case.
    \item function - The name of the current function.
\end{itemize}

Most of the runtime fields are not very usable.  They are there for
completeness.

Meta fields may be accessed inside the test case by enclosing them in
curly brackets.  The curly brackets will be interpreted anywhere
inside the test case, including inside quoted strings.  Field names that
are not recognised will be passed through including the brackets.
Note that it is therefore impossible to use some strings within the
test case.

Test case function names should include the permuted fields in the
name to reduce name collisions.

\subsection{An example}
I don't know how to do pre-formatted text in \LaTeX.  Sigh.

The following code generates a simple increment test for all combinations of the
storage classes and all combinations of the data sizes.  This is a
bad example as the optimiser will often remove most of this code.

\tt{
/** Test for increment. 
 
    type: char, int, long

    Z80 port does not fully support longs (4 byte)

    type[z80]: char, int   


    class: ``'', register, static
*/

static void

testInc\{class\}\{types\}(void)

\{

    \{class\} \{type\} i = 0;
    
    i = i + 1;

    ASSERT((i == 1));

\}

}

\end{document}
