README.md
1# Run Spec Coverage Tool
2
3The tool reads ISA spec files and the given set of test files, then computes the coverage and outputs a bunch of coverage reports.
4
5```
6spectrac -r path_to_summary_report_yaml_file \
7 -d directory_where_tests_located \
8 -g glob_to_select_tests_files_in_tests_directory \
9 -s paths_to_isa_yaml_files \
10 -n path_to_nontestable_yaml_file \
11 -u path_to_output_notcovered_yaml_file \
12 -U path_to_output_notcovered_markdown_file \
13 -o path_to_output_yaml_file_with_irrelevant_tests \
14 -O path_to_output_markdown_file_with_irrelevant_tests \
15 -f path_to_output_yaml_file_with_coverage_data \
16 -F path_to_output_markdown_file_with_coverage_data
17```
18
19Options:
20```
21--testdir (-d) - directory with tests, required
22--testglob (-g) - glob for selecting test files in tests directory ("**/*.pa" for example), required
23--spec (-s) - ISA spec files, comma-separated, merged into one if multiple specified, required
24--non-testable (-n) - yaml file with non-testable assertions
25--full (-f) - output the full spec in yaml format with additional fields showing the coverage data (mark assertions as covered/not-covered/non-testable, provide coverage metric for each group)
26--full_md (-F) - same as --full, but in markdown format
27--uncovered (-u) - output yaml document listing the spec areas not covered by tests
28--uncovered_md (-U) - same as --uncovered, but in markdown format
29--orphaned (-o) - output list of test files that found not relevant to the current spec (the assertions in test file aren't found in the spec)
30--orphaned_md (-O) - same as --orphaned, but in markdown format
31--report (-r) - output the test coverage summary report in yaml
32```
33
34Example:
35```
36cd ~/panda
37tests/cts-coverage-tool/bin/spectrac.rb \
38 -g "**/*.pa" \
39 -d tests/cts-generator/cts-generated/ \
40 -s isa/isa.yaml,plugins/java/isa/isa.yaml \
41 -f full.yaml \
42 -u uncovered.yaml \
43 -o orphaned.yaml \
44 -n tests/cts-coverage-tool/config/non_testable.yaml
45```
46