1# Testing astcenc 2 3The repository contains a small suite of tests which can be used to sanity 4check source code changes to the compressor. It must be noted that this test 5suite is relatively limited in scope and does not cover every feature or 6bitrate of the standard. 7 8# Required software 9 10Running the tests requires Python 3.7 to be installed on the host machine, and 11an `astcenc-avx2` release build to have been previously compiled and installed 12into an directory called `astcenc` in the root of the git checkout. This 13can be achieved by configuring the CMake build using the install prefix 14`-DCMAKE_INSTALL_PREFIX=../` and then running a build with the `install` build 15target. 16 17# Running C++ unit tests 18 19We support a small (but growing) number of C++ unit tests, which are written 20using the `googletest` framework and integrated in the CMake "CTest" test 21framework. 22 23To build unit tests pull the `googletest` git submodule and add `-DUNITTEST=ON` 24to the CMake command line when configuring. 25 26To run unit tests use the CMake `ctest` utility from your build directory after 27you have built the tests. 28 29```shell 30cd build 31ctest --verbose 32``` 33 34# Running command line tests 35 36To run the command line tests, which aim to get coverage of the command line 37options and core codec stability without testing the compression quality 38itself, run the command line: 39 40 python3 -m unittest discover -s Test -p astc_test*.py -v 41 42# Running image tests 43 44To run the image test suite run the following command from the root directory 45of the repository: 46 47 python3 ./Test/astc_test_image.py 48 49This will run though a series of image compression tests, comparing the image 50PSNR against a set of reference results from the last stable baseline. The test 51will fail if any reduction in PSNR above a set threshold is detected. Note that 52performance information is reported, but regressions will not flag a failure. 53 54For debug purposes, all decompressed test output images and result CSV files 55are stored in the `TestOutput` directory, using the same test set structure as 56the `Test/Images` folder. 57 58## Test selection 59 60The runner supports a number of options to filter down what is run, enabling 61developers to focus local testing on the parts of the code they are working on. 62 63* `--encoder` selects which encoder to run. By default the `avx2` encoder is 64 selected. Note that some out-of-tree reference encoders (older encoders, and 65 some third-party encoders) are supported for comparison purposes. These will 66 not work without the binaries being manually provided; they are not 67 distributed here. 68* `--test-set` selects which image set to run. By default the `Small` image 69 test set is selected, which aims to provide basic coverage of many different 70 color formats and color profiles. 71* `--block-size` selects which block size to run. By default a range of 72 block sizes (2D and 3D) are used. 73* `--color-profile` selects which color profiles from the standard should be 74 used (LDR, LDR sRGB, or HDR) to select images. By default all are selected. 75* `--color-format` selects which color formats should be used (L, XY, RGB, 76 RGBA) to select images. By default all are selected. 77 78## Performance tests 79 80To provide less noisy performance results the test suite supports compressing 81each image multiple times and returning the best measured performance. To 82enable this mode use the following options: 83 84* `--repeats <M>` : Run M test compression passes which are timed. 85 86**Note:** The reference CSV contains performance results measured on an Intel 87Core i5 9600K running at 4.3GHz, running each test 5 times. 88 89## Updating reference data 90 91The reference PSNR and performance scores are stored in CSVs committed to the 92repository. This data is created by running the tests using the last stable 93release on a standard test machine we use for performance testing builds. 94 95It can be useful for developers to rebuild the reference results for their 96local machine, in particular for measuring performance improvements. To build 97new reference CSVs, download the current reference `astcenc` binary (1.7) from 98GitHub for your host OS and place it in to the `./Binaries/1.7/` directory. 99Once this is done, run the command: 100 101 python3 ./Test/astc_test_image.py --encoder 1.7 --test-set all --repeats 5 102 103... to regenerate the reference CSV files. 104 105**WARNING:** This can take some hours to complete, and it is best done when the 106test suite gets exclusive use of the machine to avoid other processing slowing 107down the compression and disturbing the performance data. It is recommended to 108shutdown or disable any background applications that are running. 109 110## Valgrind memcheck 111 112It is always worth running the Valgrind memcheck tool to validate that we have 113not introduced any obvious memory errors. Build a release build with symbols 114information with `-DCMAKE_BUILD_TYPE=RelWithDebInfo` and then run: 115 116 valgrind --tool=memcheck --track-origins=yes <command> 117 118- - - 119 120_Copyright © 2019-2022, Arm Limited and contributors. All rights reserved._ 121