1# Porting to Other Implementations 2 3## Introduction 4 5This document provides an overview of the test runner and how to 6integrate it with other stacks. So far we have it working with 7BoringSSL and some incomplete integrations with NSS and OpenSSL. 8 9Note that supporting non-BoringSSL implementations is a work in 10progress and interfaces may change in the future. Consumers should pin 11to a particular revision rather than using BoringSSL’s master branch 12directly. As we gain experience with other implementations, we hope to 13make further improvements to portability, so please contact 14davidben@google.com and ekr@rtfm.com if implementing a new shim. 15 16 17## Integration Architecture 18 19The test runner integrates with the TLS stack under test through a 20“shim”: a command line program which encapsulates the stack. By 21default, the shim points to the BoringSSL shim in the same source 22tree, but any program can be supplied via the `-shim-path` flag. The 23runner opens up a server socket and provides the shim with a `-port` 24argument that points to that socket. The shim always connects to the 25runner as a TCP client even when acting as a TLS server. For DTLS, 26there is a small framing layer that gives packet boundaries over 27TCP. The shim can also pass a variety of command line arguments which 28are used to configure the stack under test. These can be found at 29`test_config.cc`. 30 31 32The shim reports success by exiting with a `0` error code and failure by 33reporting a non-zero error code and generally sending a textual error 34value to stderr. Many of the tests expect specific error string (such 35as `NO_SHARED_CIPHER`) that indicates what went wrong. 36 37 38## Compatibility Issues 39 40There are a number of situations in which the runner might succeed 41with some tests and not others: 42 43* Defects in the stack under test 44* Features which haven’t yet been implemented 45* Failure to implement one or more of the command line flags the runner uses with the shim 46* Disagreement about the right behavior/interpretation of the spec 47 48 49We have implemented several features which allow implementations to ease these compatibility issues. 50 51### Configuration File 52 53The runner can be supplied with a JSON configuration file which is 54intended to allow for a per-stack mapping. This file currently takes 55two directives: 56 57 58* `DisabledTests`: A JSON map consisting of the pattern matching the 59 tests to be disabled as the key and some sort of reason why it was 60 disabled as the value. The key is used as a match against the test 61 name. The value is ignored and is just used for documentation 62 purposes so you can remember why you disabled a 63 test. `-include-disabled` overrides this filter. 64 65* `ErrorMap`: A JSON map from the internal errors the runner expects to 66 the error strings that your implementation spits out. Generally 67 you’ll need to map every error, but if you also provide the 68 ` -loose-errors` flag, then every un-mapped error just gets mapped to 69 the empty string and treated as if it matched every error the runner 70 expects. 71 72 73The `-shim-config` flag is used to provide the config file. 74 75 76### Unimplemented Features 77If the shim encounters some request from the runner that it knows it 78can’t fulfill (e.g., a command line flag that it doesn’t recognize), 79then it can exit with the special code `89`. Shims are recommended to 80use this exit code on unknown command-line arguments. 81 82The test runner interprets this as “unimplemented” and skips the 83test. If run normally, this will cause the test runner to report that 84the entire test suite failed. The `-allow-unimplemented` flag suppresses 85this behavior and causes the test runner to ignore these tests for the 86purpose of evaluating the success or failure of the test suite. 87 88 89### Malloc Tests 90 91The test runner can also be used to stress malloc failure 92codepaths. If passed `-malloc-test=0`, the runner will run each test 93repeatedly with an incrementing `MALLOC_NUMBER_TO_FAIL` environment 94variable. The shim should then replace the malloc implementation with 95one which fails at the specified number of calls. If there are not 96enough calls to reach the number, the shim should fail with exit code 97`88`. This signals to the runner that the test has completed. 98 99See `crypto/test/malloc.cc` for an example malloc implementation. 100 101Note these tests are slow and will hit Go's test timeout. Pass `-timeout 72h` to 102avoid crashing after 10 minutes. 103 104 105## Example: Running Against NSS 106 107``` 108DYLD_LIBRARY_PATH=~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/lib go test -shim-path ~/dev/nss-dev/nss-sandbox/dist/Darwin15.6.0_64_DBG.OBJ/bin/nss_bogo_shim -loose-errors -allow-unimplemented -shim-config ~/dev/nss-dev/nss-sandbox/nss/external_tests/nss_bogo_shim/config.json 109``` 110