1[/ 2 / Copyright (c) 2003 Boost.Test contributors 3 / 4 / Distributed under the Boost Software License, Version 1.0. (See accompanying 5 / file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt) 6 /] 7 8[section:expected_failures Expected failures specification] 9 10While in a perfect world all test assertions should pass in order for a test module to pass, in some situations 11it is desirable to temporarily allow particular tests to fail. For example, where a particular feature is not 12implemented yet and one needs to prepare a library for the release or when particular test fails on some 13platforms. To avoid a nagging red box in regression tests table, you can use the expected failures feature. 14 15 16This feature allows specifying an expected number of failed assertions per test unit. The value is specified 17during test tree construction, and can't be updated during test execution. 18 19The feature is not intended to be used to check for expected functionality failures. To check that a particular 20input is causing an exception to be thrown use __BOOST_LEVEL_THROW__ family of testing 21tools. 22 23The usage of this feature should be limited and employed only after careful consideration. In general you should 24only use this feature when it is necessary to force a test module to pass without actually fixing the problem. 25Obviously, an excessive usage of expected failures defeats the purpose of the unit test. In most cases it only 26needs be applied temporarily. 27 28You also need to remember that the expected failure specification is per test case. This means that any failed 29assertion within that test case can satisfy the expected failures quota. Meaning it is possible for an 30unexpected failure to occur to satisfy this quota. 31 32 33[note If an assertion at fault is fixed and passed while an expected failures specification still present, 34 the number of failures becomes smaller than expected. The test is going to be reported as passed; instead, 35 a warning message will be issued. 36] 37 38 39[/-----------------------------------------------------------------] 40[#l_expected_failure][h3 Expected failure specification] 41 42The decorator __decorator_expected_failures__ defines the number of assertions that are expected to fail within the corresponding test 43unit. It is reported as failure when the number of failed assertions is greater than the declared expected number of 44failures. If the number of failed assertions is less than the number of expected failures a message is reported. The 45total number of expected failures for a given test suite `S` is the sum of the declared expected failures in `S` and the 46sum of expected failures in all nested test units: 47 48[bt_example decorator_10..decorator expected_failures..run-fail] 49 50In the above example, we first run all test cases with four failed assertions. The total number of expected failures 51is 3: 1 (for test `suite1`) + 2 (for `test1`). Because the expected failure count is exceeded, the error is reported. 52In the second case, we only run test case `suite1/test1`: two failures occur, two failures are expected, therefore no 53error is reported. 54 55 56 57 58[/-----------------------------------------------------------------] 59[h3 Usage with automatically registered test cases] 60 61[caution this usage is considered as deprecated. Please consider using the [link l_expected_failure `expected_failures`] 62 decorator instead.] 63 64For backwards compatibility, it is possible to indicate the expected failures with 65__BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES__ [footnote deprecated] before the test case definition. 66 67`` 68BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES(test_case_name, number_of_expected_failures); 69`` 70 71You can use this macro both on a file scope and inside a test suite. Moreover you can use it even if name of test 72units coincide in different test suites. Expected failures specification applies to the test unit belonging to the same 73test suite where __BOOST_AUTO_TEST_CASE_EXPECTED_FAILURES__ resides. 74 75[bt_example example17..Expected failures specification for automatically registered test case..run] 76 77 78[/-----------------------------------------------------------------] 79[h3 Usage with manually registered test cases] 80 81[caution this usage is considered as deprecated. Please consider using the [link l_expected_failure `expected_failures`] 82 decorator instead.] 83 84To set the value of expected failures for the manually registered test unit pass it as a second argument for the 85[link ref_test_case_registration `test_suite::add`] call during test unit registration. 86 87[bt_example example16..Expected failures specification for manually registered test case..run] 88 89[endsect] [/ expected failures] 90 91[/EOF] 92