1 #LyX 1.4.4 created this file. For more info see http://www.lyx.org/
7 \pdfoptionpdfminorversion=3
9 pdftitle={Proposed Test Suite Design},
10 pdfauthor={Michael Hope},
11 pdfkeywords={c case compiler framework GPL regression SDCC suite test},
13 linkcolor=blue] {hyperref}
19 \paperfontsize default
26 \paperorientation portrait
29 \paragraph_separation indent
31 \quotes_language english
34 \paperpagestyle default
35 \tracking_changes false
42 Proposed Test Suite Design
46 \begin_layout Standard
59 Michael Hope (michaelh @ juju.net.nz)
62 \begin_layout Abstract
63 This article describes the goals, requirements, and suggested specification
64 for a test suite for the output of the Small Device C Compiler (sdcc).
65 Also included is a short list of existing works.
73 \begin_layout Standard
74 The main goals of a test suite for sdcc are
77 \begin_layout Enumerate
78 To allow developers to run regression tests to check that core changes do
79 not break any of the many ports.
83 \begin_layout Enumerate
88 \begin_layout Enumerate
89 To allow developers to verify individual ports.
93 \begin_layout Enumerate
94 To allow developers to test port changes.
98 \begin_layout Standard
99 This design only covers the generated code.
100 It does not cover a test/unit test framework for the sdcc application itself,
104 \begin_layout Standard
105 One side effect of (1) is that it requires that the individual ports pass
106 the tests originally.
107 This may be too hard.
108 See the section on Exceptions below.
111 \begin_layout Section
115 \begin_layout Subsection
119 \begin_layout Standard
120 The suite is intended to cover language features only.
121 Hardware specific libraries are explicitly not covered.
124 \begin_layout Subsection
128 \begin_layout Standard
129 The ports often generate different code for handling different types (Byte,
130 Word, DWord, and the signed forms).
131 Meta information could be used to permute the different test cases across
135 \begin_layout Subsection
139 \begin_layout Standard
140 The different ports are all at different levels of development.
141 Test cases must be able to be disabled on a per port basis.
142 Permutations also must be able to be disabled on a port level for unsupported
144 Disabling, as opposed to enabling, on a per port basis seems more maintainable.
147 \begin_layout Subsection
151 \begin_layout Standard
152 The tests must be able to run unaided.
153 The test suite must run on all platforms that sdcc runs on.
154 A good minimum may be a subset of Unix command set and common tools, provided
155 by default on a Unix host and provided through cygwin on a Windows host.
158 \begin_layout Standard
159 The tests suits should be able to be sub-divided, so that the failing or
160 interesting tests may be run separately.
163 \begin_layout Subsection
167 \begin_layout Standard
168 The test code within the test cases should not generate artifacts.
169 An artifact occurs when the test code itself interferes with the test and
170 generates an erroneous result.
173 \begin_layout Subsection
177 \begin_layout Standard
178 sdcc is a cross compiling compiler.
179 As such, an emulator is needed for each port to run the tests.
182 \begin_layout Section
186 \begin_layout Subsection
190 \begin_layout Standard
191 DejaGnu is a toolkit written in Expect designed to test an interactive program.
192 It provides a way of specifying an interface to the program, and given
193 that interface a way of stimulating the program and interpreting the results.
194 It was originally written by Cygnus Solutions for running against development
196 I believe the gcc test suite is written against DejaGnu, perhaps partly
197 to test the Cygnus ports of gcc on target systems.
200 \begin_layout Subsection
204 \begin_layout Standard
205 I don't know much about the gcc test suite.
206 It was recently removed from the gcc distribution due to issues with copyright
208 The code I saw from older distributions seemed more concerned with esoteric
209 features of the language.
212 \begin_layout Subsection
216 \begin_layout Standard
217 The xUnit family, in particular JUnit, is a library of in test assertions,
218 test wrappers, and test suite wrappers designed mainly for unit testing.
222 \begin_layout Subsection
223 CoreLinux++ Assertion framework
226 \begin_layout Standard
227 While not a test suite system, the assertion framework is an interesting
228 model for the types of assertions that could be used.
229 They include pre-condition, post-condition, invariants, conditional assertions,
230 unconditional assertions, and methods for checking conditions.
233 \begin_layout Section
237 \begin_layout Standard
238 This specification borrows from the JUnit style of unit testing and the
239 CoreLinux++ style of assertions.
240 The emphasis is on maintainability and ease of writing the test cases.
243 \begin_layout Subsection
247 \begin_layout Standard
248 PENDING: Align these terms with the rest of the world.
251 \begin_layout Itemize
256 is a statement of how things should be.
257 PENDING: Better description, an example.
261 \begin_layout Itemize
266 is the smallest unit of a test suite, and consists of a single assertion
267 that passes if the test passes.
271 \begin_layout Itemize
276 is a set of test points that test a certain feature.
280 \begin_layout Itemize
285 is a set of test cases that test a certain set of features.
289 \begin_layout Subsection
293 \begin_layout Standard
294 Test cases shall be contained in their own C file, along with the meta data
296 Test cases shall be contained within functions whose names start with 'test'
297 and which are descriptive of the test case.
298 Any function that starts with 'test' will be automatically run in the test
302 \begin_layout Standard
303 To make the automatic code generation easier, the C code shall have this
307 \begin_layout Itemize
308 Test functions shall start with 'test' to allow automatic detection.
312 \begin_layout Itemize
313 Test functions shall follow the K&R intention style for ease of detection.
315 the function name shall start in the left column on a new line below the
316 return specification.
320 \begin_layout Subsection
324 \begin_layout Standard
325 All assertions shall log the line number, function name, and test case file
327 Most assertions can have a more descriptive message attached to them.
328 Assertions will be implemented through macros to get at the line information.
329 This may cause trouble with artifacts.
332 \begin_layout Standard
333 The following definitions use C++ style default arguments where optional
334 messages may be inserted.
335 All assertions use double opening and closing brackets in the macros to
336 allow them to be compiled out without any side effects.
337 While this is not required for a test suite, they are there in case any
338 of this code is incorporated into the main product.
341 \begin_layout Standard
342 Borrowing from JUnit, the assertions shall include
345 \begin_layout Itemize
347 \begin_inset Quotes eld
351 \begin_inset Quotes erd
355 Used when execution should not get here.
359 \begin_layout Itemize
360 ASSERT((Boolean cond, String msg =
361 \begin_inset Quotes eld
365 \begin_inset Quotes erd
369 Fails if cond is false.
370 Parent to REQUIRE and ENSURE.
374 \begin_layout Standard
375 JUnit also includes may sub-cases of ASSERT, such as assertNotNull, assertEquals
379 \begin_layout Standard
380 CoreLinux++ includes the extra assertions
383 \begin_layout Itemize
384 REQUIRE((Boolean cond, String msg =
385 \begin_inset Quotes eld
389 \begin_inset Quotes erd
393 Checks preconditions.
397 \begin_layout Itemize
398 ENSURE((Boolean cond, String msg =
399 \begin_inset Quotes eld
403 \begin_inset Quotes erd
407 Checks post conditions.
411 \begin_layout Itemize
412 CHECK((Boolean cond, String msg =
413 \begin_inset Quotes eld
417 \begin_inset Quotes erd
421 Used to call a function and to check that the return value is as expected.
423 CHECK((fread(in, buf, 10) != -1)).
424 Very similar to ASSERT, but the function still gets called in a release
429 \begin_layout Itemize
431 Used to check conditions within part of the code.
432 For example, can be used to check that a list is still sorted inside each
433 loop of a sort routine.
437 \begin_layout Standard
438 All of FAIL, ASSERT, REQUIRE, ENSURE, and CHECK shall be available.
441 \begin_layout Subsection
445 \begin_layout Standard
446 PENDING: It's not really meta data.
449 \begin_layout Standard
450 Meta data includes permutation information, exception information, and permutati
454 \begin_layout Standard
455 Meta data shall be global to the file.
456 Meta data names consist of the lower case alphanumerics.
457 Test case specific meta data (fields) shall be stored in a comment block
458 at the start of the file.
459 This is only due to style.
462 \begin_layout Standard
463 A field definition shall consist of
466 \begin_layout Itemize
470 \begin_layout Itemize
475 \begin_layout Itemize
476 A comma separated list of values.
480 \begin_layout Standard
481 The values shall be stripped of leading and trailing white space.
484 \begin_layout Standard
485 Permutation exceptions are by port only.
486 Exceptions to a field are specified by a modified field definition.
487 An exception definition consists of
490 \begin_layout Itemize
495 \begin_layout Itemize
496 An opening square bracket.
500 \begin_layout Itemize
501 A comma separated list of ports the exception applies for.
505 \begin_layout Itemize
506 A closing square bracket.
510 \begin_layout Itemize
515 \begin_layout Itemize
516 The values to use for this field for these ports.
520 \begin_layout Standard
521 An instance of the test case shall be generated for each permutation of
522 the test case specific meta data fields.
525 \begin_layout Standard
526 The runtime meta fields are
529 \begin_layout Itemize
530 port - The port this test is running on.
534 \begin_layout Itemize
535 testcase - The name of this test case.
539 \begin_layout Itemize
540 function - The name of the current function.
544 \begin_layout Standard
545 Most of the runtime fields are not very usable.
546 They are there for completeness.
549 \begin_layout Standard
550 Meta fields may be accessed inside the test case by enclosing them in curly
552 The curly brackets will be interpreted anywhere inside the test case, including
553 inside quoted strings.
554 Field names that are not recognised will be passed through including the
556 Note that it is therefore impossible to use some strings within the test
560 \begin_layout Standard
561 Test case function names should include the permuted fields in the name
562 to reduce name collisions.
565 \begin_layout Subsection
569 \begin_layout Standard
570 I don't know how to do pre-formatted text in LaTeX.
574 \begin_layout Standard
575 The following code generates a simple increment test for all combinations
576 of the storage classes and all combinations of the data sizes.
577 This is a bad example as the optimiser will often remove most of this code.
585 /** Test for increment.
590 type: char, int, long
594 Z80 port does not fully support longs (4 byte)
604 \begin_inset Quotes eld
608 \begin_inset Quotes erd
611 , register, static */
617 testInc{class}{types}(void)