1 #LyX 1.3 created this file. For more info see http://www.lyx.org/
11 \paperfontsize default
18 \use_numerical_citations 0
19 \paperorientation portrait
22 \paragraph_separation indent
24 \quotes_language english
28 \paperpagestyle default
32 Proposed Test Suite Design
35 Michael Hope (michaelh@juju.net.nz)
51 This article describes the goals, requirements, and suggested specification
52 for a test suite for the output of the Small Device C Compiler (sdcc).
53 Also included is a short list of existing works.
60 The main goals of a test suite for sdcc are
63 To allow developers to run regression tests to check that core changes do
64 not break any of the many ports.
72 To allow developers to verify individual ports.
76 To allow developers to test port changes.
80 This design only covers the generated code.
81 It does not cover a test/unit test framework for the sdcc application itself,
85 One side effect of (1) is that it requires that the individual ports pass
88 See the section on Exceptions below.
97 The suite is intended to cover language features only.
98 Hardware specific libraries are explicitly not covered.
104 The ports often generate different code for handling different types (Byte,
105 Word, DWord, and the signed forms).
106 Meta information could be used to permute the different test cases across
113 The different ports are all at different levels of development.
114 Test cases must be able to be disabled on a per port basis.
115 Permutations also must be able to be disabled on a port level for unsupported
117 Disabling, as opposed to enabling, on a per port basis seems more maintainable.
123 The tests must be able to run unaided.
124 The test suite must run on all platforms that sdcc runs on.
125 A good minimum may be a subset of Unix command set and common tools, provided
126 by default on a Unix host and provided through cygwin on a Windows host.
129 The tests suits should be able to be sub-divided, so that the failing or
130 interesting tests may be run separately.
136 The test code within the test cases should not generate artifacts.
137 An artifact occurs when the test code itself interferes with the test and
138 generates an erroneous result.
144 sdcc is a cross compiling compiler.
145 As such, an emulator is needed for each port to run the tests.
154 DejaGnu is a toolkit written in Expect designed to test an interactive program.
155 It provides a way of specifying an interface to the program, and given
156 that interface a way of stimulating the program and interpreting the results.
157 It was originally written by Cygnus Solutions for running against development
159 I believe the gcc test suite is written against DejaGnu, perhaps partly
160 to test the Cygnus ports of gcc on target systems.
166 I don't know much about the gcc test suite.
167 It was recently removed from the gcc distribution due to issues with copyright
169 The code I saw from older distributions seemed more concerned with esoteric
170 features of the language.
176 The xUnit family, in particular JUnit, is a library of in test assertions,
177 test wrappers, and test suite wrappers designed mainly for unit testing.
181 CoreLinux++ Assertion framework
184 While not a test suite system, the assertion framework is an interesting
185 model for the types of assertions that could be used.
186 They include pre-condition, post-condition, invariants, conditional assertions,
187 unconditional assertions, and methods for checking conditions.
193 This specification borrows from the JUnit style of unit testing and the
194 CoreLinux++ style of assertions.
195 The emphasis is on maintainability and ease of writing the test cases.
201 PENDING: Align these terms with the rest of the world.
208 is a statement of how things should be.
209 PENDING: Better description, an example.
217 is the smallest unit of a test suite, and consists of a single assertion
218 that passes if the test passes.
226 is a set of test points that test a certain feature.
234 is a set of test cases that test a certain set of features.
241 Test cases shall be contained in their own C file, along with the meta data
243 Test cases shall be contained within functions whose names start with 'test'
244 and which are descriptive of the test case.
245 Any function that starts with 'test' will be automatically run in the test
249 To make the automatic code generation easier, the C code shall have this
253 Test functions shall start with 'test' to allow automatic detection.
257 Test functions shall follow the K&R intention style for ease of detection.
259 the function name shall start in the left column on a new line below the
260 return specification.
267 All assertions shall log the line number, function name, and test case file
269 Most assertions can have a more descriptive message attached to them.
270 Assertions will be implemented through macros to get at the line information.
271 This may cause trouble with artifacts.
274 The following definitions use C++ style default arguments where optional
275 messages may be inserted.
276 All assertions use double opening and closing brackets in the macros to
277 allow them to be compiled out without any side effects.
278 While this is not required for a test suite, they are there in case any
279 of this code is incorporated into the main product.
282 Borrowing from JUnit, the assertions shall include
286 \begin_inset Quotes eld
290 \begin_inset Quotes erd
294 Used when execution should not get here.
298 ASSERT((Boolean cond, String msg =
299 \begin_inset Quotes eld
303 \begin_inset Quotes erd
307 Fails if cond is false.
308 Parent to REQUIRE and ENSURE.
312 JUnit also includes may sub-cases of ASSERT, such as assertNotNull, assertEquals
316 CoreLinux++ includes the extra assertions
319 REQUIRE((Boolean cond, String msg =
320 \begin_inset Quotes eld
324 \begin_inset Quotes erd
328 Checks preconditions.
332 ENSURE((Boolean cond, String msg =
333 \begin_inset Quotes eld
337 \begin_inset Quotes erd
341 Checks post conditions.
345 CHECK((Boolean cond, String msg =
346 \begin_inset Quotes eld
350 \begin_inset Quotes erd
354 Used to call a function and to check that the return value is as expected.
356 CHECK((fread(in, buf, 10) != -1)).
357 Very similar to ASSERT, but the function still gets called in a release
363 Used to check conditions within part of the code.
364 For example, can be used to check that a list is still sorted inside each
365 loop of a sort routine.
369 All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
375 PENDING: It's not really meta data.
378 Meta data includes permutation information, exception information, and permutati
382 Meta data shall be global to the file.
383 Meta data names consist of the lower case alphanumerics.
384 Test case specific meta data (fields) shall be stored in a comment block
385 at the start of the file.
386 This is only due to style.
389 A field definition shall consist of
399 A comma separated list of values.
403 The values shall be stripped of leading and trailing white space.
406 Permutation exceptions are by port only.
407 Exceptions to a field are specified by a modified field definition.
408 An exception definition consists of
415 An opening square bracket.
419 A comma separated list of ports the exception applies for.
423 A closing square bracket.
431 The values to use for this field for these ports.
435 An instance of the test case shall be generated for each permutation of
436 the test case specific meta data fields.
439 The runtime meta fields are
442 port - The port this test is running on.
446 testcase - The name of this test case.
450 function - The name of the current function.
454 Most of the runtime fields are not very usable.
455 They are there for completeness.
458 Meta fields may be accessed inside the test case by enclosing them in curly
460 The curly brackets will be interpreted anywhere inside the test case, including
461 inside quoted strings.
462 Field names that are not recognised will be passed through including the
464 Note that it is therefore impossible to use some strings within the test
468 Test case function names should include the permuted fields in the name
469 to reduce name collisions.
475 I don't know how to do pre-formatted text in LaTeX.
479 The following code generates a simple increment test for all combinations
480 of the storage classes and all combinations of the data sizes.
481 This is a bad example as the optimiser will often remove most of this code.
486 /** Test for increment.
492 type: char, int, long
497 Z80 port does not fully support longs (4 byte)
508 \begin_inset Quotes eld
512 \begin_inset Quotes erd
515 , register, static */
525 testInc{class}{types}(void)
535 {class} {type} i = 0;