1Conformance Test: Definition and Structure 2=========================================== 3 4This document describes what conformance tests are, how they are created, 5and the structure they follow in CVS. 6 7Conformance Test Definition 8--------------------------- 9 10Conformance tests test conformance to the POSIX* specification. They test 11both that the header files are correct and that the behavior defined in 12the specification is implemented. 13 14Directory Structure 15------------------- 16 17The directory structure for conformance tests is as follows. 18 19All tests are located in the conformance/ directory off of posixtestsuite/. 20 21Within this directory, there are the following three subdirectories: 22 23definitions/ - This directory contains tests for each *.h file in the POSIX 24spec. Tests are taken from the Base Definitions document. 25 26interfaces/ - This directory contains tests for each function listed in the 27POSIX spec. Tests are taken from the System Interfaces document. 28 29behavior/ - This directory contains tests from any document that do not 30directly correlate to functions (as in interfaces/) or header files 31(as in definitions/). 32 33The breakdown of these subdirectories is as follows: 34 35definitions/ 36----------- 37The definitions/ directory contains one subdirectory per *.h file. Each 38subdirectory should have the name *_h (i.e., the *.h file name with "_" 39replacing "."). 40 41interfaces/ 42---------- 43The interfaces directory contains one subdirectory per function listed 44in the System Interfaces document. 45 46behavior/ 47-------- 48This directory is broken down by functional area in a manner similar to 49functional and stress tests (see HOWTO_<TBD> - for now, see email discussions 50on functional/stress test structure). 51 52Writing Tests 53------------- 54 55Within each lowest level directory (definitions/*_h for definitions, 56interfaces/<function> for interfaces, and behavior/<functional area> for 57behavior), an assertions.xml file is used to describe the tests, and tests 58have the structure M-N.c or M-N.sh. More details are below. 59 60Speculative Tests 61----------------- 62Tests which the POSIX spec uses the words "may" or "implementation-defined" 63to define are considered speculative by PTS and are created in a 64speculative/ directory off of the functional area directory. (For 65example, mq_timedsend/speculative.) 66These tests attempt to determine which of the implementation-defined 67alternatives the current implementation implements. PTS_PASS is always 68returned if the test is able to finish. 69 70assertions.xml 71-------------- 72This file contains a list of assertions to be tested, an ID for each 73assertion, and descriptive attributes to be used in categorizing 74assertions. For more information on creating and using these files, 75see HOWTO_Assertions. 76 77Test Cases 78---------- 79Test cases are numbered M-N.c or M-N.sh where M corresponds to the 80assertion ID of the assertion being tested, and N is an ID for the specific 81tests. (ex. assertion 8 may have test cases 8-1.c 8-2.c and 8-3.sh) 82Generally, *.sh scripts are only used when the test case is trivial. 83 84The following documents may be helpful in test case creation: 85 86- HOWTO_DefinitionsTest - describes how to create definitions tests 87- HOWTO_ResultCodes - describes the result codes test cases should return 88- HOWTO_BoundaryTest - describes how to create boundary tests for functions 89 90* POSIX (R) is a registered trademark of the IEEE 91 92Contributors: rusty.lynch REMOVE-THIS AT intel DOT com 93 julie.n.fleischer REMOVE-THIS AT intel DOT com 94