1# Changelog 2All notable changes to this project will be documented in this file. 3 4The format is based on [Keep a Changelog](http://keepachangelog.com/en/1.0.0/) 5and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0.html). 6 7## [Unreleased] 8 9## [0.3.5] - 2021-07-26 10### Fixed 11- Corrected `Criterion.toml` in the book. 12- Corrected configuration typo in the book. 13 14### Changed 15- Bump plotters dependency to always include a bug-fix. 16- MSRV bumped to 1.46. 17 18## [0.3.4] - 2021-01-24 19### Added 20- Added support for benchmarking async functions 21- Added `with_output_color` for enabling or disabling CLI output coloring programmatically. 22 23### Fixed 24- Criterion.rs will now give a clear error message in case of benchmarks that take zero time. 25- Added some extra code to ensure that every sample has at least one iteration. 26- Added a notice to the `--help` output regarding "unrecognized option" errors. 27- Increased opacity on violin charts. 28- Fixed violin chart X axis not starting at zero in the plotters backend. 29- Criterion.rs will now automatically detect the right output directory. 30 31### Deprecated 32- `Criterion::can_plot` is no longer useful and is deprecated pending deletion in 0.4.0. 33- `Benchmark` and `ParameterizedBenchmark` were already hidden from documentation, but are now 34 formally deprecated pending deletion in 0.4.0. Callers should use `BenchmarkGroup` instead. 35- `Criterion::bench_function_over_inputs`, `Criterion::bench_functions`, and `Criterion::bench` were 36 already hidden from documentation, but are now formally deprecated pending deletion in 0.4.0. 37 Callers should use `BenchmarkGroup` instead. 38- Three new optional features have been added; "html_reports", "csv_output" and 39 "cargo_bench_support". These features currently do nothing except disable a warning message at 40 runtime, but in version 0.4.0 they will be used to enable HTML report generation, CSV file 41 generation, and the ability to run in cargo-bench (as opposed to [cargo-criterion]). 42 "cargo_bench_support" is enabled by default, but "html_reports" and "csv_output" 43 are not. If you use Criterion.rs' HTML reports, it is recommended to switch to [cargo-criterion]. 44 If you use CSV output, it is recommended to switch to [cargo-criterion] and use the 45 `--message-format=json` option for machine-readable output instead. A warning message will be 46 printed at the start of benchmark runs which do not have "html_reports" or "cargo_bench_support" 47 enabled, but because CSV output is not widely used it has no warning. 48 49[cargo-criterion]: https://github.com/bheisler/cargo-criterion 50 51## [0.3.3] - 2020-06-29 52### Added 53- Added `CRITERION_HOME` environment variable to set the directory for Criterion to store 54 its results and charts in. 55- Added support for [cargo-criterion]. The long-term goal here is to remove code from Criterion-rs 56 itself to improve compile times, as well as to add features to `cargo-criterion` that are 57 difficult to implement in Criterion-rs. 58- Add sampling mode option for benchmarks. This allows the user to change how Criterion.rs chooses 59 the iteration counts in each sample. By default, nothing will change for most benchmarks, but 60 very slow benchmarks will now run fewer iterations to fit in the desired number of samples. 61 This affects the statistics and plots generated. 62 63### Changed 64- The serialization format for some of the files has changed. This may cause your first benchmark 65 run after updating to produce errors, but they're harmless and will go away after running the 66 benchmarks once. 67 68### Fixed 69- Fixed a bug where the current measurement was not shown on the relative regression plot. 70- Fixed rare panic in the plotters backend. 71- Panic with a clear error message (rather than panicking messily later on) when the user sets the 72 group or function name to the empty string. 73- Escape single quotes in benchmark names when generating Gnuplot scripts. 74 75## [0.3.2] - 2020-04-26 76### Added 77- Added `?Sized` bound to benchmark parameter types, which allows dynamically sized types like 78 `&str` and `&[T]` to be used as benchmark parameters. 79- Added the `--output-format <format>` command-line option. If `--output-format bencher` is passed, 80 Criterion.rs will print its measurements in a format similar to that used by the `bencher` crate 81 or unstable `libtest` benchmarks, and using similar statistical measurements as well. Though this 82 provides less information than the default format, it may be useful for supporting tools which 83 parse this output format. 84- Added `--nocapture` argument. This argument does nothing, but prevents Criterion.rs from exiting 85 when running tests or benchmarks and allowing stdout output from other tests. 86 87### Fixed 88- Fixed panic when environment variables contains non-UTF8 characters. 89- Fixed panic when `CRITERION_DEBUG` or `CRITERION_TARGET_DIR` environment variables contain 90 non-UTF8 characters. 91 92## [0.3.1] - 2020-01-25 93### Added 94- Added new plotting backend using the `plotters` crate. Implementation generously provided by Hao 95 Hou, author of the `plotters` crate. 96- Added `--plotting-backend` command-line option to select the plotting backend. The existing 97 gnuplot backend will be used by default when available, and the plotters backend will be used when 98 gnuplot is not available or when requested. 99- Added `Criterion::plotting_backend()` function to configure the plotting backend in code. 100- Added `--load-baseline` command-line option to load a baseline for comparison 101 rather than measuring the current code 102- Benchmark filters can now be regular expressions. 103 104### Fixed 105- Fixed `fibonacci` functions. 106- Fixed `#[criterion]` benchmarks ignoring the command-line options. 107- Fixed incorrect scaling of the violin plots. 108- Don't print the recommended sample count if it's the same as the configured 109 sample count. 110- Fix potential panic when `nresamples` is set too low. Also added a warning 111 against setting `nresamples` too low. 112- Fixed issue where a slow outer closure would cause Criterion.rs to calculate 113 the wrong estimated time and number of iterations in the warm-up phase. 114 115## [0.3.0] - 2019-08-25 116### Added 117- Added support for plugging in custom measurements (eg. processor counters) 118 into Criterion.rs' measurement and analysis. 119- Added support for plugging in instrumentation for internal profilers such as 120 `cpuprofiler` which must be explicitly started and stopped within the profiled 121 process. 122- Added the `BenchmarkGroup` type, which supersedes `ParameterizedBenchmark`, `Benchmark`, 123 `Criterion::bench_functions`, `Criterion::bench_function_over_inputs`, and `Criterion::bench`. 124 `BenchmarkGroup` performs the same function as all of the above, but is cleaner to use and more 125 powerful and flexible. All of these types/functions are now soft-deprecated (meaning they're 126 hidden from the documentation and should not be used in new code). They will be fully deprecated 127 at some point in the 0.3.* series and removed in 0.4.0. 128- `iter_custom` - a "timing loop" that allows the caller to perform their own measurements. This is 129 useful for complex measurements that don't fit into the usual mode of calling a lambda in a loop. 130- If the benchmark cannot be completed in approximately the requested measurement time, 131 Criterion.rs will now print a suggested measurement time and sample size that would work. 132- Two new fields, `throughput_num` and `throughput_type` have been added to the `raw.csv` file. 133- Added command-line options to set the defaults for warm-up time, measurement-time, etc. 134 135### Changed 136- The `raw.csv` file format has been changed slightly. The `sample_time_nanos` field has been split 137 into `sample_measured_value` and `unit` fields to accommodate custom measurements. 138- Throughput has been expanded from u32 to u64 to accommodate very large input sizes. 139 140### Fixed 141- Fixed possible invalid file name error on Windows 142- Fixed potential case where data for two different benchmarks would be stored in the same directory. 143 144### Removed 145- Removed the `--measure-only` command-line argument; it was deprecated in favor of `--profile-time` 146 in 0.2.6. 147- External program benchmarks have been removed; they were deprecated in 0.2.6. The new 148 `iter_custom` timing loop can be used as a substitute; see `benches/external_process.rs` for an 149 example of this. 150 151### Deprecated 152- The `--test` argument is now deprecated. To test benchmarks, use `cargo test --benches`. 153 154## [0.2.11] - 2019-04-08 155### Added 156- Enabled automatic text-coloring on Windows. 157 158### Fixed 159- Fixed panic caused by outdated files after benchmark names or types were changed. 160- Reduced timing overhead of `Criterion::iter_batched/iter_batched_ref`. 161 162## [0.2.10] - 2019-02-09 163### Added 164- Added `iter_batched/iter_batched_ref` timing loops, which allow for setup (like 165 `iter_with_setup/iter_with_large_setup`) and exclude drop (like `iter_with_large_drop`) but 166 measure the runtime more accurately, use less memory and are more flexible. 167 168### Deprecated 169- `iter_with_setup/iter_with_large_setup` are now deprecated in favor of `iter_batched`. 170 171## [0.2.9] - 2019-01-24 172### Changed 173- Criterion.rs no longer depends on the default features of the `rand-core` crate. This fixes some 174 downstream crates which use `rand` in a `no_std` context. 175 176## [0.2.8] - 2019-01-20 177### Changed 178- Criterion.rs now uses `rayon` internally instead of manual `unsafe` code built with thread-scoped. 179- Replaced handlebars templates with [TinyTemplate](https://github.com/bheisler/TinyTemplate) 180- Merged `criterion-stats` crate into `criterion` crate. `criterion-stats` will no longer receive 181 updates. 182- Replaced or removed various other dependencies to reduce the size of Criterion.rs' dependency 183 tree. 184 185## [0.2.7] - 2018-12-29 186 187### Fixed 188- Fixed version numbers to prevent incompatibilities between `criterion` and `criterion-stats` 189 crates. 190 191## [0.2.6] - 2018-12-27 - Yanked 192### Added 193- Added `--list` command line option, which lists the benchmarks but does not run them, to match 194 `cargo test -- --list`. 195- Added README/CONTRIBUTING/LICENSE files to sub-crates. 196- Displays change in throughput in the command-line and HTML output as well as change in iteration 197 time. 198- Benchmarks with multiple functions and multiple values will now generate a per-value summary 199 report file in addition to the existing per-function one. 200- Added a `--profile-time` command-line argument which disables reporting and analysis and instead 201 simply iterates each benchmark for approximately the given number of seconds. This supersedes the 202 (now-deprecated) `--measure-only` argument. 203 204### Fixed 205- Functions passed to `Bencher::iter_with_large_setup` can now return output. This is necessary to 206 prevent the compiler from optimizing away the benchmark. This is technically a breaking change - 207 that function requires a new type parameter. It's so unlikely to break existing code that I 208 decided not to delay this for a breaking-change release. 209- Reduced measurement overhead for the `iter_with_large_setup` and `iter_with_drop` methods. 210- `criterion_group` and `criterion_main` macros no longer require the `Criterion` struct to be 211 explicitly imported. 212- Don't panic when `gnuplot --version` fails. 213- Criterion.rs macros no longer require user to `use criterion::Criterion;` 214- Criterion.rs no longer initializes a logger, meaning that it will no longer conflict with user 215 code which does. 216- Criterion.rs no longer fails to parse gnuplot version numbers like 217 `gnuplot 5.2 patchlevel 5a (Gentoo revision r0)` 218- Criterion.rs no longer prints an error message that gnuplot couldn't be found when chart 219 generation is disabled (either by `Criterion::without_plots`, `--noplot` or disabling the 220 HTML reports feature) 221- Benchmark names are now automatically truncated to 100 characters and a number may be added to 222 make them unique. This fixes a problem where gnuplot would crash if the title was extremely long, 223 and also improves the general usability of Criterion.rs. 224 225### Changed 226- Changed timing model of `iter_with_large_setup` to exclude time spent dropping values returned 227 by the routine. Time measurements taken with 0.2.6 using these methods may differ from those taken 228 with 0.2.5. 229- Benchmarks with multiple functions and multiple values will now appear as a table rather than a 230 tree in the benchmark index. This is to accommodate the new per-value summary reports. 231 232### Deprecated 233- Deprecated the `--measure-only` command-line-argument in favor of `--profile-time`. This will be 234 removed in 0.3.0. 235- External-program benchmarks are now deprecated. They will be removed in 0.3.0. 236- The `html_reports` cargo feature is now deprecated. This feature will become non-optional in 0.3.0. 237- Sample sizes less than 10 are deprecated and will be disallowed in 0.3.0. 238- This is not an exhaustive list - the full scope of changes in 0.3.0 is not yet determined. There 239 may be breaking changes that are not listed here. 240 241## [0.2.5] - 2018-08-27 242### Fixed 243- Fixed links from generated report files to documentation. 244- Fixed formatting for very large percentage changes (>1000%) 245- Sorted the benchmarks in the index report by name 246- Fixed case where benchmark ID with special characters would cause Criterion.rs to open the wrong 247 file and log an error message. 248- Fixed case where running `cargo clean; cargo bench -- <filter>` would cause Criterion.rs to log 249 an error message. 250- Fixed a GNUplot error message when sample size is very small. 251- Fixed several cases where Criterion.rs would generate invalid path names. 252- Fixed a bug where Criterion.rs would print an error if run with a filter that allowed no benchmarks and a clean target directory. 253- Fixed bug where some benchmarks didn't appear in the benchmark index report. 254- Criterion.rs now honors the `CARGO_TARGET_DIR` environment variable. 255 256### Added 257- Criterion.rs will generate a chart showing the effects of changes in input (or input size) for all 258 benchmarks with numeric inputs or throughput, not just for those which compare multiple functions. 259 260## [0.2.4] 2018-07-08 261### Added 262- Added a pair of flags, `--save-baseline` and `--baseline`, which change 263 how benchmark results are stored and compared. This is useful for 264 working against a fixed baseline(eg. comparing progress on an 265 optimization feature branch to the commit it forked from). 266 Default behavior of Criterion.rs is now `--save-baseline base` 267 which emulates the previous, user facing behavior. 268 - `--save-baseline` saves the benchmark results under the provided name. 269 - `--baseline` compares the results to a saved baseline. 270 If the baseline does not exist for a benchmark, an error is given. 271- Added user-guide documentation for baselines, throughput measurements and 272 plot configuration. 273- Added a flag, `--test`, which causes Criterion to execute the benchmarks once 274 without measuring or reporting the results. This is useful for checking that the 275 benchmarks run successfully in a CI setting. 276- Added a `raw.csv` file to the output which contains a stable, machine-readable 277 representation of the measurements taken by benchmarks. This enables users to 278 perform their own analysis or keep historical information without depending on 279 private implementation details. 280 281### Fixed 282- The `sample_size` method on the `Criterion`, `Benchmark` and 283 `ParameterizedBenchmark` structs has been changed to panic if the sample size 284 is less than 2. Other parts of the code require this and will panic if the 285 sample size is 1, so this is not considered to be a breaking change. 286- API documentation has been updated to show more-complete examples. 287- Certain characters will now be replaced with underscores when creating benchmark 288 directory paths, to avoid generating invalid or unexpected paths. 289 290## [0.2.3] - 2018-04-14 291### Fixed 292- Criterion.rs will now panic with a clear error message if the user attempts to run 293 a benchmark which doesn't call the `Bencher::iter` function or a related function, 294 rather than failing in an uncontrolled manner later. 295- Fixed broken links in some more summary reports. 296 297### Added 298- Added a `--measure-only` argument which causes the benchmark executable to run the 299 warmup and measurement and then move on to the next benchmark without analyzing or 300 saving data. This is useful to prevent Criterion.rs' analysis code from appearing 301 in profile data when profiling benchmarks. 302- Added an index report file at "target/criterion/report/index.html" which links to 303 the other reports for easy navigation. 304 305## [0.2.2] - 2018-03-25 306### Fixed 307- Fixed broken links in some summary reports. 308- Work around apparent rustc bug in >= 1.24.0. 309 310## [0.2.1] - 2018-02-24 311### Added 312- HTML reports are now a default Cargo feature. If you wish to disable HTML reports, 313 disable Criterion.rs' default features. Doing so will allow compatibility with 314 older Rust versions such as 1.20. If you wish to continue using HTML reports, you 315 don't need to do anything. 316- Added a summary report for benchmarks that compare multiple functions or different 317 inputs. 318 319### Changed 320- The plots and HTML reports are now generated in a `report` folder. 321 322### Fixed 323- Underscores in benchmark names will no longer cause subscripted characters to 324 appear in generated plots. 325 326## [0.2.0] - 2018-02-05 327### Added 328- Added `Criterion.bench` function, which accepts either a `Benchmark` or 329 `ParameterizedBenchmark`. These new structures allow for custom per-benchmark 330 configuration as well as more complex benchmark grouping (eg. comparing a Rust 331 function against an external program over a range of inputs) which was not 332 possible previously. 333- Criterion.rs can now report the throughput of the benchmarked code in units of 334 bytes or elements per second. See the `Benchmark.throughput` and 335 `ParameterizedBenchmark.throughput` functions for further details. 336- Criterion.rs now generates a basic HTML report for each benchmark. 337- Added `--noplot` command line option to disable plot generation. 338 339### Changed 340- The builder methods on the Criterion struct now take and return self by value 341 for easier chaining. Functions which configure a Criterion structure will need 342 to be updated accordingly, or will need to be changed to work with the 343 `Benchmark` or `ParameterizedBenchmark` types to do per-benchmark configuration 344 instead. 345- The closures taken by `Criterion.bench_*` must now have a `'static` lifetime. 346 This means that you may need to change your closures from `|bencher| {...}` 347 to `move |bencher| {...}`. 348- `Criterion.bench_functions` now takes `I` as an input parameter, not `&I`. 349- Input values must now implement `Debug` rather than `Display`. 350- The generated plots are stored in `target/criterion` rather than `.criterion`. 351 352### Removed 353- The hidden `criterion::ConfidenceInterval` and`criterion::Estimate` types are 354 no longer publicly accessible. 355- The `Criterion.summarize` function has been removed. 356 357### Fixed 358- Fixed the relative mean and median reports. 359- Fixed panic while summarizing benchmarks. 360 361## [0.1.2] - 2018-01-12 362### Changed 363- Criterion.rs is now stable-compatible! 364- Criterion.rs now includes its own stable-compatible `black_box` function. 365 Some benchmarks may now be affected by dead-code-elimination where they 366 previously weren't and may have to be updated. 367- Criterion.rs now uses `serde` to save results. Existing results files will 368 be automatically removed when benchmarks are run. 369- Redesigned the command-line output to highlight the important information 370 and reduce noise. 371 372### Added 373- Running benchmarks with the variable "CRITERION_DEBUG" in the environment will 374 cause Criterion.rs to generate extra debug output and save the gnuplot scripts 375 alongside the generated plots. 376 377### Fixed 378- Don't panic on IO errors or gnuplot failures 379- Fix generation of invalid gnuplot scripts when benchmarking over inputs and inputs include values <= 0. 380- Bug where benchmarks would run one sample fewer than was configured. 381 382### Removed 383- Generated plots will no longer use log-scale. 384 385## [0.1.1] - 2017-12-12 386### Added 387- A changelog file. 388- Added a chapter to the book on how Criterion.rs collects and analyzes data. 389- Added macro rules to generate a test harness for use with `cargo bench`. 390 Benchmarks defined without these macros should continue to work. 391- New contribution guidelines 392- Criterion.rs can selectively run benchmarks. See the Command-line page for 393more details 394 395## 0.1.0 - 2017-12-02 396### Added 397- Initial release on Crates.io. 398 399 400[Unreleased]: https://github.com/bheisler/criterion.rs/compare/0.3.4...HEAD 401[0.1.1]: https://github.com/bheisler/criterion.rs/compare/0.1.0...0.1.1 402[0.1.2]: https://github.com/bheisler/criterion.rs/compare/0.1.1...0.1.2 403[0.2.0]: https://github.com/bheisler/criterion.rs/compare/0.1.2...0.2.0 404[0.2.1]: https://github.com/bheisler/criterion.rs/compare/0.2.0...0.2.1 405[0.2.2]: https://github.com/bheisler/criterion.rs/compare/0.2.1...0.2.2 406[0.2.3]: https://github.com/bheisler/criterion.rs/compare/0.2.2...0.2.3 407[0.2.4]: https://github.com/bheisler/criterion.rs/compare/0.2.3...0.2.4 408[0.2.5]: https://github.com/bheisler/criterion.rs/compare/0.2.4...0.2.5 409[0.2.6]: https://github.com/bheisler/criterion.rs/compare/0.2.5...0.2.6 410[0.2.7]: https://github.com/bheisler/criterion.rs/compare/0.2.6...0.2.7 411[0.2.8]: https://github.com/bheisler/criterion.rs/compare/0.2.7...0.2.8 412[0.2.9]: https://github.com/bheisler/criterion.rs/compare/0.2.8...0.2.9 413[0.2.10]: https://github.com/bheisler/criterion.rs/compare/0.2.9...0.2.10 414[0.2.11]: https://github.com/bheisler/criterion.rs/compare/0.2.10...0.2.11 415[0.3.0]: https://github.com/bheisler/criterion.rs/compare/0.2.11...0.3.0 416[0.3.1]: https://github.com/bheisler/criterion.rs/compare/0.3.0...0.3.1 417[0.3.2]: https://github.com/bheisler/criterion.rs/compare/0.3.1...0.3.2 418[0.3.3]: https://github.com/bheisler/criterion.rs/compare/0.3.2...0.3.3 419[0.3.4]: https://github.com/bheisler/criterion.rs/compare/0.3.3...0.3.4