1 #LyX 1.5.7 created this file. For more info see http://www.lyx.org/
7 \pdfoptionpdfminorversion=3
9 pdftitle={Proposed Test Suite Design},
10 pdfauthor={Michael Hope},
11 pdfkeywords={c case compiler framework GPL regression SDCC suite test},
13 linkcolor=blue] {hyperref}
19 \font_typewriter default
20 \font_default_family default
26 \paperfontsize default
34 \paperorientation portrait
37 \paragraph_separation indent
39 \quotes_language english
42 \paperpagestyle default
43 \tracking_changes false
51 Proposed Test Suite Design
55 \begin_layout Standard
68 Michael Hope (michaelh @ juju.net.nz)
71 \begin_layout Abstract
72 This article describes the goals, requirements, and suggested specification
73 for a test suite for the output of the Small Device C Compiler (sdcc).
74 Also included is a short list of existing works.
82 \begin_layout Standard
83 The main goals of a test suite for sdcc are
86 \begin_layout Enumerate
87 To allow developers to run regression tests to check that core changes do
88 not break any of the many ports.
92 \begin_layout Enumerate
97 \begin_layout Enumerate
98 To allow developers to verify individual ports.
102 \begin_layout Enumerate
103 To allow developers to test port changes.
107 \begin_layout Standard
108 This design only covers the generated code.
109 It does not cover a test/unit test framework for the sdcc application itself,
113 \begin_layout Standard
114 One side effect of (1) is that it requires that the individual ports pass
115 the tests originally.
116 This may be too hard.
117 See the section on Exceptions below.
120 \begin_layout Section
124 \begin_layout Subsection
128 \begin_layout Standard
129 The suite is intended to cover language features only.
130 Hardware specific libraries are explicitly not covered.
133 \begin_layout Subsection
137 \begin_layout Standard
138 The ports often generate different code for handling different types (Byte,
139 Word, DWord, and the signed forms).
140 Meta information could be used to permute the different test cases across
144 \begin_layout Subsection
148 \begin_layout Standard
149 The different ports are all at different levels of development.
150 Test cases must be able to be disabled on a per port basis.
151 Permutations also must be able to be disabled on a port level for unsupported
153 Disabling, as opposed to enabling, on a per port basis seems more maintainable.
156 \begin_layout Subsection
160 \begin_layout Standard
161 The tests must be able to run unaided.
162 The test suite must run on all platforms that sdcc runs on.
163 A good minimum may be a subset of Unix command set and common tools, provided
164 by default on a Unix host and provided through cygwin on a Windows host.
167 \begin_layout Standard
168 The tests suits should be able to be sub-divided, so that the failing or
169 interesting tests may be run separately.
172 \begin_layout Subsection
176 \begin_layout Standard
177 The test code within the test cases should not generate artifacts.
178 An artifact occurs when the test code itself interferes with the test and
179 generates an erroneous result.
182 \begin_layout Subsection
186 \begin_layout Standard
187 sdcc is a cross compiling compiler.
188 As such, an emulator is needed for each port to run the tests.
191 \begin_layout Section
195 \begin_layout Subsection
199 \begin_layout Standard
200 DejaGnu is a toolkit written in Expect designed to test an interactive program.
201 It provides a way of specifying an interface to the program, and given
202 that interface a way of stimulating the program and interpreting the results.
203 It was originally written by Cygnus Solutions for running against development
205 I believe the gcc test suite is written against DejaGnu, perhaps partly
206 to test the Cygnus ports of gcc on target systems.
209 \begin_layout Subsection
213 \begin_layout Standard
214 I don't know much about the gcc test suite.
215 It was recently removed from the gcc distribution due to issues with copyright
217 The code I saw from older distributions seemed more concerned with esoteric
218 features of the language.
221 \begin_layout Subsection
225 \begin_layout Standard
226 The xUnit family, in particular JUnit, is a library of in test assertions,
227 test wrappers, and test suite wrappers designed mainly for unit testing.
231 \begin_layout Subsection
232 CoreLinux++ Assertion framework
235 \begin_layout Standard
236 While not a test suite system, the assertion framework is an interesting
237 model for the types of assertions that could be used.
238 They include pre-condition, post-condition, invariants, conditional assertions,
239 unconditional assertions, and methods for checking conditions.
242 \begin_layout Section
246 \begin_layout Standard
247 This specification borrows from the JUnit style of unit testing and the
248 CoreLinux++ style of assertions.
249 The emphasis is on maintainability and ease of writing the test cases.
252 \begin_layout Subsection
256 \begin_layout Standard
257 PENDING: Align these terms with the rest of the world.
260 \begin_layout Itemize
265 is a statement of how things should be.
266 PENDING: Better description, an example.
270 \begin_layout Itemize
275 is the smallest unit of a test suite, and consists of a single assertion
276 that passes if the test passes.
280 \begin_layout Itemize
285 is a set of test points that test a certain feature.
289 \begin_layout Itemize
294 is a set of test cases that test a certain set of features.
298 \begin_layout Subsection
302 \begin_layout Standard
303 Test cases shall be contained in their own C file, along with the meta data
305 Test cases shall be contained within functions whose names start with 'test'
306 and which are descriptive of the test case.
307 Any function that starts with 'test' will be automatically run in the test
311 \begin_layout Standard
312 To make the automatic code generation easier, the C code shall have this
316 \begin_layout Itemize
317 Test functions shall start with 'test' to allow automatic detection.
321 \begin_layout Itemize
322 Test functions shall follow the K&R intention style for ease of detection.
324 the function name shall start in the left column on a new line below the
325 return specification.
329 \begin_layout Subsection
333 \begin_layout Standard
334 All assertions shall log the line number, function name, and test case file
336 Most assertions can have a more descriptive message attached to them.
337 Assertions will be implemented through macros to get at the line information.
338 This may cause trouble with artifacts.
341 \begin_layout Standard
342 The following definitions use C++ style default arguments where optional
343 messages may be inserted.
344 All assertions use double opening and closing brackets in the macros to
345 allow them to be compiled out without any side effects.
346 While this is not required for a test suite, they are there in case any
347 of this code is incorporated into the main product.
350 \begin_layout Standard
351 Borrowing from JUnit, the assertions shall include
354 \begin_layout Itemize
356 \begin_inset Quotes eld
360 \begin_inset Quotes erd
364 Used when execution should not get here.
368 \begin_layout Itemize
369 ASSERT((Boolean cond, String msg =
370 \begin_inset Quotes eld
374 \begin_inset Quotes erd
378 Fails if cond is false.
379 Parent to REQUIRE and ENSURE.
383 \begin_layout Standard
384 JUnit also includes may sub-cases of ASSERT, such as assertNotNull, assertEquals
388 \begin_layout Standard
389 CoreLinux++ includes the extra assertions
392 \begin_layout Itemize
393 REQUIRE((Boolean cond, String msg =
394 \begin_inset Quotes eld
398 \begin_inset Quotes erd
402 Checks preconditions.
406 \begin_layout Itemize
407 ENSURE((Boolean cond, String msg =
408 \begin_inset Quotes eld
412 \begin_inset Quotes erd
416 Checks post conditions.
420 \begin_layout Itemize
421 CHECK((Boolean cond, String msg =
422 \begin_inset Quotes eld
426 \begin_inset Quotes erd
430 Used to call a function and to check that the return value is as expected.
432 CHECK((fread(in, buf, 10) != -1)).
433 Very similar to ASSERT, but the function still gets called in a release
438 \begin_layout Itemize
440 Used to check conditions within part of the code.
441 For example, can be used to check that a list is still sorted inside each
442 loop of a sort routine.
446 \begin_layout Standard
447 All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
450 \begin_layout Subsection
454 \begin_layout Standard
455 PENDING: It's not really meta data.
458 \begin_layout Standard
459 Meta data includes permutation information, exception information, and permutati
463 \begin_layout Standard
464 Meta data shall be global to the file.
465 Meta data names consist of the lower case alphanumerics.
466 Test case specific meta data (fields) shall be stored in a comment block
467 at the start of the file.
468 This is only due to style.
471 \begin_layout Standard
472 A field definition shall consist of
475 \begin_layout Itemize
479 \begin_layout Itemize
484 \begin_layout Itemize
485 A comma separated list of values.
489 \begin_layout Standard
490 The values shall be stripped of leading and trailing white space.
493 \begin_layout Standard
494 Permutation exceptions are by port only.
495 Exceptions to a field are specified by a modified field definition.
496 An exception definition consists of
499 \begin_layout Itemize
504 \begin_layout Itemize
505 An opening square bracket.
509 \begin_layout Itemize
510 A comma separated list of ports the exception applies for.
514 \begin_layout Itemize
515 A closing square bracket.
519 \begin_layout Itemize
524 \begin_layout Itemize
525 The values to use for this field for these ports.
529 \begin_layout Standard
530 An instance of the test case shall be generated for each permutation of
531 the test case specific meta data fields.
534 \begin_layout Standard
535 The runtime meta fields are
538 \begin_layout Itemize
539 port - The port this test is running on.
543 \begin_layout Itemize
544 testcase - The name of this test case.
548 \begin_layout Itemize
549 function - The name of the current function.
553 \begin_layout Standard
554 Most of the runtime fields are not very usable.
555 They are there for completeness.
558 \begin_layout Standard
559 Meta fields may be accessed inside the test case by enclosing them in curly
561 The curly brackets will be interpreted anywhere inside the test case, including
562 inside quoted strings.
563 Field names that are not recognised will be passed through including the
565 Note that it is therefore impossible to use some strings within the test
569 \begin_layout Standard
570 Test case function names should include the permuted fields in the name
571 to reduce name collisions.
574 \begin_layout Subsection
578 \begin_layout Standard
579 I don't know how to do pre-formatted text in LaTeX.
583 \begin_layout Standard
584 The following code generates a simple increment test for all combinations
585 of the storage classes and all combinations of the data sizes.
586 This is a bad example as the optimiser will often remove most of this code.
594 /** Test for increment.
599 type: char, int, long
603 Z80 port does not fully support longs (4 byte)
613 \begin_inset Quotes eld
617 \begin_inset Quotes erd
620 , register, static */
626 testInc{class}{types}(void)