Lines Matching +full:test +full:- +full:only

5 In Mbed TLS, we use black-box testing as much as possible: test the documented behavior of the prod…
7 The goal of this document is to identify areas where black-box testing is insufficient and to propo…
9 This is a test strategy document, not a test plan. A description of exactly what is tested is out o…
15 * [“Possible approaches”](#possible-approaches) discusses some general methods for non-black-box te…
26 See the section [“Possible approaches”](#possible-approaches) for a rationale.
30 …test-specific interfaces if there's a practical way of doing it another way. All public interfaces…
34 In unit tests and in test programs, it's ok to include internal header files from `library/`. Do no…
36 …test code. If the function should be `static` for optimization but can't be `static` for testing, …
38 If test code or test data depends on internal details of the library and not just on its documented…
41 > /* This test file is specific to the ITS implementation in PSA Crypto
48 > # This test assumes that PSA_MAX_KEY_BITS (currently 65536-8 bits = 8191 bytes
50 > # output from HKDF-SHA512 (255*64 = 16320 bytes).
53 ### Rules for compile-time options
55 …test something is to add code to the product that is only useful for testing, do so, but obey the …
57 * **Only use test-specific code when necessary.** Anything that can be tested through the documente…
58 * **Test-specific code must be guarded by `#if defined(MBEDTLS_TEST_HOOKS)`**. Do not create fine-g…
60 …ctures, but if so, make it very clear that these fields have no impact on non-test-specific fields.
61 …nction substitution.** See [“rules for function substitution”](#rules-for-function-substitution) f…
65 …how to replace a library function `mbedtls_foo()` by alternative code for test purposes. That is, …
67 … compile time to support alternative platforms, the test code should be compatible with this compi…
69 …identally skipping side effects in its parameters), to provide a hook for test code; such function…
73 In unit test code that needs to modify the internal behavior:
75 * The test function (or the whole test file) must depend on `MBEDTLS_TEST_HOOKS`.
76 * At the beginning of the test function, set the global function pointers to the desired value.
77 * In the test function's cleanup code, restore the global function pointers to their default value.
85 * Coverage: we need to test behaviors which are not easy to trigger by using the API or which canno…
86 * Correctness: we want to test the actual product, not a modified version, since conclusions drawn …
87 * Effacement: the product should not include features that are solely present for test purposes, si…
88 …ry platform. Skipping tests on certain platforms may hide errors that are only apparent on such pl…
89 …ty: tests should only enforce the documented behavior of the product, to avoid extra work when the…
97 Resource allocation can fail, but rarely does so in a typical test environment. How does the produc…
102 * Files in storage (PSA API only — in the Mbed TLS API, black-box unit tests are sufficient).
103 * Key slots (PSA API only).
105 * Communication handles (PSA crypto service only).
111 We also need to test resilience: if the system is reset during an operation, does it restart in a c…
121 * Deleting files in storage (PSA API only).
129 * Inject faults and test corruption checks inside the product.
141 ### Fine-grained public interfaces
143 …test in the public interface. Then the tests can be truly black-box. The limitation of this approa…
145 …only add public interfaces if they are also useful in production, at least sometimes. For example,…
147 Mbed TLS traditionally has very fine-grained public interfaces, with many platform functions that c…
150 | ----------- | -------- |
153 | Effacement | !! Requires adding many otherwise-useless interfaces |
156 |                 | ! Public interfaces must remain for backward compatibility even if the test arc…
158 ### Fine-grained undocumented interfaces
160 …test in undocumented interfaces. Undocumented interfaces are described in public headers for the s…
162 …ly intended for cross-module abstraction leakage rather than for testing. For the PSA API, we favo…
165 | ----------- | -------- |
168 | Effacement | !! Requires adding many otherwise-useless interfaces |
174 … in the public interfaces. This is nice when it works, because it lets us test the unchanged produ…
178 …he tests that are not necessarily always true (for example that are platform-specific). We may acc…
180 …portable since it only relies on C interfaces. A limitation is that the test-only interfaces must …
183 | ----------- | -------- |
187 | Effacement | ++ Fine as long as the internal interfaces aren't added solely for test purposes |
194 If we guarantee certain properties through static analysis, we don't need to test them. This puts s…
196 …n the specification (but we can gain this confidence by evaluating the specification on test data).
197 * This does not work for platform-dependent properties unless we have a formal model of the platfor…
200 | ----------- | -------- |
201 | Coverage | ~ Good for platform-independent properties, if we can guarantee them statically |
207 ### Compile-time options
209 …have in the product for testing, but not in production, we can add a compile-time option to enable…
212 | ----------- | -------- |
214 | Correctness | ! Difficult to ensure that what we test is what we run |
219 | Maintainability | + Test interfaces impact the product source code, but at least they're clearly …
221 #### Guidelines for compile-time options
223 * **Minimize the number of compile-time options.**<br>
224 …Either we're testing or we're not. Fine-grained options for testing would require more test builds…
225 * **Merely enabling the compile-time option should not change the behavior.**<br>
226 …When building in test mode, the code should have exactly the same behavior. Changing the behavior …
228 …We should not have test-specific conditional compilation littered through the code, as that makes …
238 | ----------- | -------- |
244 … properties come for free, but some require effort (e.g. the test code itself must be leak-free to…
246 ### Debugger-based testing
248 If we want to do something in a test that the product isn't capable of doing, we can use a debugger…
254 …ts is hard. We need to have confidence that we're testing what we mean to test, even in the face o…
255 * Debugger scripts are very much non-portable.
258 | ----------- | -------- |
260 | Correctness | ++ The code is unmodified, and tested as compiled (so we even detect compiler-induc…
276 Goal: test that `mbedtls_platform_zeroize` does wipe the memory buffer.
278 Solution ([debugger](#debugger-based-testing)): implemented in `tests/scripts/test_zeroize.gdb`.
284 Goal: test the absence of memory leaks.
286 Solution ([instrumentation](#runtime-instrumentation)): run tests with ASan. (We also use Valgrind,…
288 …test jobs with a memory leak detector, each test function or test program must clean up after itse…
292 Solution: TODO. We don't test this at all at this point.
296 Goal: test the absence of resource leaks in the PSA key store code, in particular that `psa_close_k…
298 …-interfaces)): in most tests involving PSA functions, the cleanup code explicitly calls `PSA_DONE(…
308 Goal: test that no stray files are left over in the key store after a test that succeeded.
310 Solution: TODO. Currently the various test suites do it differently.
314 Goal: ensure that no stray files are left over in the key store even if a test has failed (as that …
316 Solution: TODO. Currently the various test suites do it differently.
320 Goal: test the resilience of PSA storage against power failures.
324 See the [secure element driver interface test strategy](driver-interface-test-strategy.html) for mo…
328 Goal: test the robustness against corrupted storage.
330 Solution ([internal interface](#internal-interfaces)): call `psa_its` functions to modify the stora…
334 Goal: test the robustness against read errors.
340 Goal: test the robustness against write errors (`STORAGE_FAILURE` or `INSUFFICIENT_STORAGE`).
346 Goal: test that the storage format does not change between versions (or if it does, an upgrade path…
348 Solution ([internal interface](#internal-interfaces)): call internal functions to inspect the conte…
350 …only by the general layout, but also by the numerical values of encodings for key types and other …
356 Goal: test the failure of `psa_crypto_init`.
358 …tion ([compile-time option](#compile-time-options)): replace entropy initialization functions by f…
366 …tion structure, we can do it by looking inside the structure content, but only when running withou…