1 #LyX 1.1 created this file. For more info see http://www.lyx.org/
11 \paperfontsize default
17 \paperorientation portrait
20 \paragraph_separation indent
22 \quotes_language english
26 \paperpagestyle default
30 Proposed Test Suite Design
33 Michael Hope (michaelh@juju.net.nz)
43 This article describes the goals, requirements, and suggested specification
44 for a test suite for the output of the Small Device C Compiler (sdcc).
45 Also included is a short list of existing works.
52 The main goals of a test suite for sdcc are
55 To allow developers to run regression tests to check that core changes do
56 not break any of the many ports.
64 To allow developers to verify individual ports.
68 To allow developers to test port changes.
72 This design only covers the generated code.
73 It does not cover a test/unit test framework for the sdcc application itself,
77 One side effect of (1) is that it requires that the individual ports pass
80 See the section on Exceptions below.
89 The suite is intended to cover language features only.
90 Hardware specific libraries are explicitly not covered.
96 The ports often generate different code for handling different types (Byte,
97 Word, DWord, and the signed forms).
98 Meta information could be used to permute the different test cases across
105 The different ports are all at different levels of development.
106 Test cases must be able to be disabled on a per port basis.
107 Permutations also must be able to be disabled on a port level for unsupported
109 Disabling, as opposed to enabling, on a per port basis seems more maintainable.
115 The tests must be able to run unaided.
116 The test suite must run on all platforms that sdcc runs on.
117 A good minimum may be a subset of Unix command set and common tools, provided
118 by default on a Unix host and provided through cygwin on a Windows host.
121 The tests suits should be able to be sub-divided, so that the failing or
122 interesting tests may be run separately.
128 The test code within the test cases should not generate artifacts.
129 An artifact occurs when the test code itself interferes with the test and
130 generates an erroneous result.
136 sdcc is a cross compiling compiler.
137 As such, an emulator is needed for each port to run the tests.
146 DejaGnu is a toolkit written in Expect designed to test an interactive program.
147 It provides a way of specifying an interface to the program, and given
148 that interface a way of stimulating the program and interpreting the results.
149 It was originally written by Cygnus Solutions for running against development
151 I believe the gcc test suite is written against DejaGnu, perhaps partly
152 to test the Cygnus ports of gcc on target systems.
158 I don't know much about the gcc test suite.
159 It was recently removed from the gcc distribution due to issues with copyright
161 The code I saw from older distributions seemed more concerned with esoteric
162 features of the language.
168 The xUnit family, in particular JUnit, is a library of in test assertions,
169 test wrappers, and test suite wrappers designed mainly for unit testing.
173 CoreLinux++ Assertion framework
176 While not a test suite system, the assertion framework is an interesting
177 model for the types of assertions that could be used.
178 They include pre-condition, post-condition, invariants, conditional assertions,
179 unconditional assertions, and methods for checking conditions.
185 This specification borrows from the JUnit style of unit testing and the
186 CoreLinux++ style of assertions.
187 The emphasis is on maintainability and ease of writing the test cases.
193 PENDING: Align these terms with the rest of the world.
200 is a statement of how things should be.
201 PENDING: Better description, an example.
209 is the smallest unit of a test suite, and consists of a single assertion
210 that passes if the test passes.
218 is a set of test points that test a certain feature.
226 is a set of test cases that test a certain set of features.
233 Test cases shall be contained in their own C file, along with the meta data
235 Test cases shall be contained within functions whose names start with 'test'
236 and which are descriptive of the test case.
237 Any function that starts with 'test' will be automatically run in the test
241 To make the automatic code generation easier, the C code shall have this
245 Test functions shall start with 'test' to allow automatic detection.
249 Test functions shall follow the K&R intention style for ease of detection.
251 the function name shall start in the left column on a new line below the
252 return specification.
259 All assertions shall log the line number, function name, and test case file
261 Most assertions can have a more descriptive message attached to them.
262 Assertions will be implemented through macros to get at the line information.
263 This may cause trouble with artifacts.
266 The following definitions use C++ style default arguments where optional
267 messages may be inserted.
268 All assertions use double opening and closing brackets in the macros to
269 allow them to be compiled out without any side effects.
270 While this is not required for a test suite, they are there in case any
271 of this code is incorporated into the main product.
274 Borrowing from JUnit, the assertions shall include
278 \begin_inset Quotes eld
282 \begin_inset Quotes erd
286 Used when execution should not get here.
290 ASSERT((Boolean cond, String msg =
291 \begin_inset Quotes eld
295 \begin_inset Quotes erd
299 Fails if cond is false.
300 Parent to REQUIRE and ENSURE.
304 JUnit also includes may sub-cases of ASSERT, such as assertNotNull, assertEquals
308 CoreLinux++ includes the extra assertions
311 REQUIRE((Boolean cond, String msg =
312 \begin_inset Quotes eld
316 \begin_inset Quotes erd
320 Checks preconditions.
324 ENSURE((Boolean cond, String msg =
325 \begin_inset Quotes eld
329 \begin_inset Quotes erd
333 Checks post conditions.
337 CHECK((Boolean cond, String msg =
338 \begin_inset Quotes eld
342 \begin_inset Quotes erd
346 Used to call a function and to check that the return value is as expected.
348 CHECK((fread(in, buf, 10) != -1)).
349 Very similar to ASSERT, but the function still gets called in a release
355 Used to check conditions within part of the code.
356 For example, can be used to check that a list is still sorted inside each
357 loop of a sort routine.
361 All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
367 PENDING: It's not really meta data.
370 Meta data includes permutation information, exception information, and permutati
374 Meta data shall be global to the file.
375 Meta data names consist of the lower case alphanumerics.
376 Test case specific meta data (fields) shall be stored in a comment block
377 at the start of the file.
378 This is only due to style.
381 A field definition shall consist of
391 A comma separated list of values.
395 The values shall be stripped of leading and trailing white space.
398 Permutation exceptions are by port only.
399 Exceptions to a field are specified by a modified field definition.
400 An exception definition consists of
407 An opening square bracket.
411 A comma separated list of ports the exception applies for.
415 A closing square bracket.
423 The values to use for this field for these ports.
427 An instance of the test case shall be generated for each permutation of
428 the test case specific meta data fields.
431 The runtime meta fields are
434 port - The port this test is running on.
438 testcase - The name of this test case.
442 function - The name of the current function.
446 Most of the runtime fields are not very usable.
447 They are there for completeness.
450 Meta fields may be accessed inside the test case by enclosing them in curly
452 The curly brackets will be interpreted anywhere inside the test case, including
453 inside quoted strings.
454 Field names that are not recognised will be passed through including the
456 Note that it is therefore impossible to use some strings within the test
460 Test case function names should include the permuted fields in the name
461 to reduce name collisions.
467 I don't know how to do pre-formatted text in LaTeX.
471 The following code generates a simple increment test for all combinations
472 of the storage classes and all combinations of the data sizes.
473 This is a bad example as the optimiser will often remove most of this code.
478 /** Test for increment.
484 type: char, int, long
489 Z80 port does not fully support longs (4 byte)
500 \begin_inset Quotes eld
504 \begin_inset Quotes erd
507 , register, static */
517 testInc{class}{types}(void)
527 {class} {type} i = 0;