README.md
1## Benchmark of Bytecode
2
3This folder contains scripts and documentaions for benchmarking the size
4of the bytecode.
5
6
7## `run_benchmark.py`
8
9### Help
10
11```sh
12usage: Run bytecode benchmarks [-h] [--testdir TESTDIR] [--bindir BINDIR]
13 [--input-type {class,pa}] [--compiler-options LIST]
14 [--compiler-options-file FILE] [--json FILE] [--verbose]
15
16optional arguments:
17 -h, --help show this help message and exit
18 --testdir TESTDIR Directory with tests (*.class or *.pa). Default:
19 './bytecode_optimizer/tests/benchmark/suite'
20 --bindir BINDIR Directory with compiled binaries (eg.: c2p). Default: './build/bin'
21 --input-type {class,pa}
22 Type of the test input. Default: 'class'
23 --compiler-options LIST
24 Comma separated list of compiler options for C2P (see 'build/bin/c2p
25 --help' for details
26 --compiler-options-file FILE
27 Input file containing compiler options for C2P (see 'build/bin/c2p
28 --help' for details
29 --json FILE JSON dump file name
30 --verbose, -v Enable verbose messages
31```
32
33### How to use
34
35```sh
36bytecode_optimizer/tests/benchmark/run_benchmark.py --testdir=../benchmark-input --compiler-options-file=compiler.config --json=benchmark-results.json -v
37```
38
39where `compiler.config` contains a list of compiler options one per line, e.g.:
40
41```sh
42--compiler-lowering=true
43--compiler-inlining=true
44--compiler-lowering=true
45--compiler-loop-peeling=true
46--compiler-lse=true
47--compiler-loop-unroll=true
48```
49
50### Example output
51
52```
53Average sizes (in bytes):
54 annotation_item section: 34
55 class_idx_item section: 32
56 class_item section: 223
57 code_item section: 158
58 debug_info_item section: 25
59 foreign_item section: 226
60 header_item section: 40
61 proto_item section: 61
62 string_item section: 239
63 total: 1042
64
65Minimum sizes (in bytes):
66 annotation_item section: 16
67 class_idx_item section: 24
68 class_item section: 143
69 code_item section: 11
70 debug_info_item section: 7
71 foreign_item section: 144
72 header_item section: 40
73 proto_item section: 48
74 string_item section: 149
75 total: 713
76
77Maximum sizes (in bytes):
78 annotation_item section: 72
79 class_idx_item section: 40
80 class_item section: 282
81 code_item section: 264
82 debug_info_item section: 43
83 foreign_item section: 307
84 header_item section: 40
85 proto_item section: 68
86 string_item section: 347
87 total: 1333
88Summary:
89========
90 Tests : 10
91 Passed: 7
92 Failed: 3
93```
94
95### JSON example
96```
97{
98 "StrictMath$RandomNumberGeneratorHolder.class": {
99 "annotation_item section": 43,
100 "class_idx_item section": 24,
101 "class_item section": 141,
102 "code_item section": 31,
103 "debug_info_item section": 12,
104 "foreign_item section": 158,
105 "header_item section": 40,
106 "line_number_program_item section": 8,
107 "proto_item section": 16,
108 "string_item section": 171,
109 "total": 644
110 },
111 "HttpDate.class": {
112 "error": "c2p: /panda/bytecode_optimizer/inst_builder.cpp:128:
113 void panda::bytecodeopt::InstBuilder::AddCatchPhi():
114 Assertion `catch_begin->IsCatchBegin()' failed.\n"
115 },
116 "SSLContextSpi.class": {
117 "error": "Unsupported instruction in alias analysis: 7.ref CatchPhi v10, v12, v12 -> (v19, v19, v18, v17, v17, v17)
118 c2p: /panda/compiler/optimizer/analysis/alias_analysis.cpp:395:
119 virtual void panda::compiler::AliasVisitor::VisitDefault(panda::compiler::Inst*):
120 Assertion inst->GetType() != DataType::REFERENCE && cond_val' failed."
121 }
122}
123```
124## `compare.py`
125
126This script can be used to compare the results of the `run_becnhmark.py` script.
127
128### Help
129
130```sh
131usage: Compare benchmark results [-h] --old JSON_FILE_PATH --new JSON_FILE_PATH --failed JSON_FILE_PATH
132
133optional arguments:
134 -h, --help show this help message and exit
135 --old JSON_FILE_PATH Base or reference benchmark result
136 --new JSON_FILE_PATH Benchmark result to be compared with the reference
137 --failed JSON_FILE_PATH
138 File to log error messages from c2p
139```
140
141### How to use
142
143```sh
144bytecode_optimizer/tests/benchmark/compare.py --old=reference.json --new=new.json --failed=error.json
145```
146
147### Example output
148
149```
150Classes that have been optimized:
151 Code_item section size:
152|Old: |New: |Diff:|Per: |File:
153| 1045| 1037| 8| 0.77%| ICULocaleService$LocaleKey.class
154| 302| 294| 8| 2.65%| Locale$LocaleKey.class
155| 5700| 5693| 7| 0.12%| LanguageTag.class
156| 957| 949| 8| 0.84%| LinkedList$ListItr.class
157| 780| 772| 8| 1.03%| FieldPosition.class
158
159Summary:
160=============
161 Total code_item section size of baseline files: 9201 bytes
162 Total code_item section size of compared files: 9162 bytes
163 Difference: 39 bytes [0.42%]
164 Number of optimized files: 5
165 Number of not optimized files : 1
166 Files with no code item section: 0
167 Files that are bigger than baseline: 0
168 Failed tests on baseline: 1
169 Failed tests compared to baseline: 0
170=============
171
172Statistics on optimized files:
173=============
174 Total code_item section size of baseline files: 8784 bytes
175 Total code_item section size of compared files: 8745 bytes
176 Difference: 39 bytes [0.44%]
177=============
178```
179