1 2## Fruit benchmarks 3 4Fruit contains some code to benchmark various metrics (e.g. performance, compile time, executable size) in an automated 5way. 6 7### Benchmark suites 8 9The `suites` folder contains suites of Fruit (or Fruit-related) benchmarks that can be run using `run_benchmarks.py`. 10For example: 11 12```bash 13$ ~/projects/fruit/extras/benchmark/run_benchmarks.py \ 14 --continue-benchmark=true \ 15 --benchmark-definition ~/projects/fruit/extras/benchmark/suites/fruit_full.yml 16 --output-file ~/fruit_bench_results.txt \ 17 --fruit-sources-dir ~/projects/fruit \ 18 --fruit-benchmark-sources-dir ~/projects/fruit \ 19 --boost-di-sources-dir ~/projects/boost-di 20``` 21 22Once the benchmark run completes, you can format the results using some pre-defined tables, see the section below. 23 24The following benchmark suites are defined: 25 26* `fruit_full.yml`: full set of Fruit benchmarks (using the Fruit 3.x API). 27* `fruit_mostly_full.yml`: a subset of the tests in `fruit_full.yml`. 28* `fruit_quick.yml`: this is an even smaller subset, and the number of runs is capped at 10 so 29 the confidence intervals might be wider. It's useful as a quicker (around 10-15min) way to get a rough idea of the 30 performance (e.g. to evaluate the performance impact of a commit, during development). 31* `fruit_single.yml`: runs the Fruit runtime benchs under a single compiler and with just 1 combination of flags. This 32 also caps the number of runs at 8, so the resulting confidence intervals might be wider than they would be with 33 `fruit_full.yml`. This is a quick benchmark that can used during development of performance optimizations. 34* `fruit_debug.yml`: a suite used to debug Fruit's benchmarking code. This is very quick, but the actual results are 35 not meaningful. Run this after changing any benchmarking code, to check that it still works. 36* `boost_di`: unlike the others, this benchmark suite exercises the Boost.DI library (still in boost-experimental at the 37 time of writing) instead of Fruit. 38 39### Benchmark tables 40 41The `tables` folder contains some table definitions that can be used to get a human-readable representations of 42benchmark results generated using `run_benchmarks.py`. 43 44Note that there *isn't* a 1:1 mapping between benchmark suites and benchmark tables; the same table definition can be 45used with multiple benchmark suites (for example, a full suite and a quick variant that only has a subset of the 46dimensions) and multiple table definitions might be appropriate to display the results of a single suite (for example, 47there could be a table definition that displays only metrics meaningful to Fruit users and one that also displays 48more fine-grained metrics that are only meaningful to Fruit developers). 49 50Example usage of `format_bench_results.py` with one of these table definitions: 51 52```bash 53$ ~/projects/fruit/extras/benchmark/format_bench_results.py \ 54 --benchmark-results ~/fruit_bench_results.txt \ 55 --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml 56``` 57 58It's also possible to compare two benchmark results (for example, when running the same benchmarks before and after 59a Fruit commit): 60 61```bash 62$ ~/projects/fruit/extras/benchmark/format_bench_results.py \ 63 --benchmark-results ~/fruit_bench_results_after.txt \ 64 --benchmark-tables-definition ~/projects/fruit/extras/benchmark/tables/fruit_wiki.yml \ 65 --baseline-benchmark-results ~/fruit_bench_results_before.txt 66``` 67 68The following tables are defined: 69 70* `fruit_wiki.yml`: the "main" table definition, with the tables that are in Fruit's wiki. 71* `fruit_internal.yml`: a more detailed version of `fruit_wiki.yml`, also displaying metrics that are only meaningful 72 to Fruit developers (e.g. splitting the setup time into component creation time and normalization time). 73 74### Manual benchmarks 75 76In some cases, you might want to run the benchmarks manually (e.g. if you want to use `callgrind` to profile the 77benchmark run). This is how you can do that: 78 79```bash 80$ cd ~/projects/fruit 81$ mkdir build 82$ cd build 83$ CXX=g++-6 cmake .. -DCMAKE_BUILD_TYPE=RelWithDebInfo 84$ make -j 10 85$ cd .. 86$ mkdir generated-benchs 87$ extras/benchmark/generate_benchmark.py \ 88 --compiler g++-6 \ 89 --fruit-sources-dir ~/projects/fruit \ 90 --fruit-build-dir ~/projects/fruit/build \ 91 --num-components-with-no-deps 10 \ 92 --num-components-with-deps 90 \ 93 --num-deps 10 \ 94 --output-dir generated-benchs \ 95 --generate-debuginfo=true 96$ cd generated-benchs 97$ make -j 10 98$ valgrind \ 99 --tool=callgrind \ 100 --simulate-cache=yes \ 101 --dump-instr=yes \ 102 ./main 10000 103``` 104