1Running external test suites with OpenSSL 2========================================= 3 4It is possible to integrate external test suites into OpenSSL's "make test". 5This capability is considered a developer option and does not work on all 6platforms. 7 8 9 10The BoringSSL test suite 11======================== 12 13In order to run the BoringSSL tests with OpenSSL, first checkout the BoringSSL 14source code into an appropriate directory. This can be done in two ways: 15 161) Separately from the OpenSSL checkout using: 17 18 $ git clone https://boringssl.googlesource.com/boringssl boringssl 19 20 The BoringSSL tests are only confirmed to work at a specific commit in the 21 BoringSSL repository. Later commits may or may not pass the test suite: 22 23 $ cd boringssl 24 $ git checkout 490469f850e 25 262) Using the already configured submodule settings in OpenSSL: 27 28 $ git submodule update --init 29 30Configure the OpenSSL source code to enable the external tests: 31 32$ cd ../openssl 33$ ./config enable-ssl3 enable-ssl3-method enable-weak-ssl-ciphers \ 34 enable-external-tests 35 36Note that using other config options than those given above may cause the tests 37to fail. 38 39Run the OpenSSL tests by providing the path to the BoringSSL test runner in the 40BORING_RUNNER_DIR environment variable: 41 42$ BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make test 43 44Note that the test suite may change directory while running so the path provided 45should be absolute and not relative to the current working directory. 46 47To see more detailed output you can run just the BoringSSL tests with the 48verbose option: 49 50$ VERBOSE=1 BORING_RUNNER_DIR=/path/to/boringssl/ssl/test/runner make \ 51 TESTS="test_external_boringssl" test 52 53 54Test failures and suppressions 55------------------------------ 56 57A large number of the BoringSSL tests are known to fail. A test could fail 58because of many possible reasons. For example: 59 60- A bug in OpenSSL 61- Different interpretations of standards 62- Assumptions about the way BoringSSL works that do not apply to OpenSSL 63- The test uses APIs added to BoringSSL that are not present in OpenSSL 64- etc 65 66In order to provide a "clean" baseline run with all the tests passing a config 67file has been provided that suppresses the running of tests that are known to 68fail. These suppressions are held in the file "test/ossl_shim/ossl_config.json" 69within the OpenSSL source code. 70 71The community is encouraged to contribute patches which reduce the number of 72suppressions that are currently present. 73 74 75Python PYCA/Cryptography test suite 76=================================== 77 78This python test suite runs cryptographic tests with a local OpenSSL build as 79the implementation. 80 81First checkout the PYCA/Cryptography module into ./pyca-cryptography using: 82 83$ git submodule update --init 84 85Then configure/build OpenSSL compatible with the python module: 86 87$ ./config shared enable-external-tests 88$ make 89 90The tests will run in a python virtual environment which requires virtualenv 91to be installed. 92 93$ make test VERBOSE=1 TESTS=test_external_pyca 94 95Test failures and suppressions 96------------------------------ 97 98Some tests target older (<=1.0.2) versions so will not run. Other tests target 99other crypto implementations so are not relevant. Currently no tests fail. 100 101 102krb5 test suite 103=============== 104 105Much like the PYCA/Cryptography test suite, this builds and runs the krb5 106tests against the local OpenSSL build. 107 108You will need a git checkout of krb5 at the top level: 109 110$ git clone https://github.com/krb5/krb5 111 112krb5's master has to pass this same CI, but a known-good version is 113krb5-1.15.1-final if you want to be sure. 114 115$ cd krb5 116$ git checkout krb5-1.15.1-final 117$ cd .. 118 119OpenSSL must be built with external tests enabled: 120 121$ ./config enable-external-tests 122$ make 123 124krb5's tests will then be run as part of the rest of the suite, or can be 125explicitly run (with more debugging): 126 127$ VERBOSE=1 make TESTS=test_external_krb5 test 128 129Test-failures suppressions 130-------------------------- 131 132krb5 will automatically adapt its test suite to account for the configuration 133of your system. Certain tests may require more installed packages to run. No 134tests are expected to fail. 135 136 137Updating test suites 138==================== 139 140To update the commit for any of the above test suites: 141 142- Make sure the submodules are cloned locally: 143 144 $ git submodule update --init --recursive 145 146- Enter subdirectory and pull from the repository (use a specific branch/tag if required): 147 148 $ cd <submodule-dir> 149 $ git pull origin master 150 151- Go to root directory, there should be a new git status: 152 153 $ cd ../ 154 $ git status 155 ... 156 # modified: <submodule-dir> (new commits) 157 ... 158 159- Add/commit/push the update 160 161 git add <submodule-dir> 162 git commit -m "Updated <submodule> to latest commit" 163 git push 164 165