• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1## Benchmark of Bytecode
2
3This folder contains scripts and documentaions for benchmarking the size
4of the bytecode.
5
6
7## `run_benchmark.py`
8
9### Help
10
11```sh
12usage: Run bytecode benchmarks [-h] [--testdir TESTDIR] [--bindir BINDIR]
13                               [--input-type {class,pa}] [--compiler-options LIST]
14                               [--compiler-options-file FILE] [--json FILE] [--verbose]
15
16optional arguments:
17  -h, --help            show this help message and exit
18  --testdir TESTDIR     Directory with tests (*.class or *.pa). Default:
19                        './bytecode_optimizer/tests/benchmark/suite'
20  --bindir BINDIR       Directory with compiled binaries (eg.: c2p). Default: './build/bin'
21  --input-type {class,pa}
22                        Type of the test input. Default: 'class'
23  --compiler-options LIST
24                        Comma separated list of compiler options for C2P (see 'build/bin/c2p
25                        --help' for details
26  --compiler-options-file FILE
27                        Input file containing compiler options for C2P (see 'build/bin/c2p
28                        --help' for details
29  --json FILE           JSON dump file name
30  --verbose, -v         Enable verbose messages
31```
32
33### How to use
34
35```sh
36bytecode_optimizer/tests/benchmark/run_benchmark.py --testdir=../benchmark-input --compiler-options-file=compiler.config --json=benchmark-results.json -v
37```
38
39where `compiler.config` contains a list of compiler options one per line, e.g.:
40
41```sh
42--compiler-lowering=true
43--compiler-inlining=true
44--compiler-lowering=true
45--compiler-loop-peeling=true
46--compiler-lse=true
47--compiler-loop-unroll=true
48```
49
50### Example output
51
52```
53Average sizes (in bytes):
54     annotation_item section: 34
55      class_idx_item section: 32
56          class_item section: 223
57           code_item section: 158
58     debug_info_item section: 25
59        foreign_item section: 226
60         header_item section: 40
61          proto_item section: 61
62         string_item section: 239
63                       total: 1042
64
65Minimum sizes (in bytes):
66     annotation_item section: 16
67      class_idx_item section: 24
68          class_item section: 143
69           code_item section: 11
70     debug_info_item section: 7
71        foreign_item section: 144
72         header_item section: 40
73          proto_item section: 48
74         string_item section: 149
75                       total: 713
76
77Maximum sizes (in bytes):
78     annotation_item section: 72
79      class_idx_item section: 40
80          class_item section: 282
81           code_item section: 264
82     debug_info_item section: 43
83        foreign_item section: 307
84         header_item section: 40
85          proto_item section: 68
86         string_item section: 347
87                       total: 1333
88Summary:
89========
90  Tests : 10
91  Passed: 7
92  Failed: 3
93```
94
95### JSON example
96```
97{
98    "StrictMath$RandomNumberGeneratorHolder.class": {
99        "annotation_item section": 43,
100        "class_idx_item section": 24,
101        "class_item section": 141,
102        "code_item section": 31,
103        "debug_info_item section": 12,
104        "foreign_item section": 158,
105        "header_item section": 40,
106        "line_number_program_item section": 8,
107        "proto_item section": 16,
108        "string_item section": 171,
109        "total": 644
110    },
111    "HttpDate.class": {
112        "error": "c2p: /panda/bytecode_optimizer/inst_builder.cpp:128:
113        void panda::bytecodeopt::InstBuilder::AddCatchPhi():
114        Assertion `catch_begin->IsCatchBegin()' failed.\n"
115    },
116    "SSLContextSpi.class": {
117        "error": "Unsupported instruction in alias analysis: 7.ref CatchPhi v10, v12, v12 -> (v19, v19, v18, v17, v17, v17)
118        c2p: /panda/compiler/optimizer/analysis/alias_analysis.cpp:395:
119        virtual void panda::compiler::AliasVisitor::VisitDefault(panda::compiler::Inst*):
120        Assertion inst->GetType() != DataType::REFERENCE && cond_val' failed."
121    }
122}
123```
124## `compare.py`
125
126This script can be used to compare the results of the `run_becnhmark.py` script.
127
128### Help
129
130```sh
131usage: Compare benchmark results [-h] --old JSON_FILE_PATH --new JSON_FILE_PATH --failed JSON_FILE_PATH
132
133optional arguments:
134  -h, --help            show this help message and exit
135  --old JSON_FILE_PATH  Base or reference benchmark result
136  --new JSON_FILE_PATH  Benchmark result to be compared with the reference
137  --failed JSON_FILE_PATH
138                        File to log error messages from c2p
139```
140
141### How to use
142
143```sh
144bytecode_optimizer/tests/benchmark/compare.py --old=reference.json --new=new.json --failed=error.json
145```
146
147### Example output
148
149```
150Classes that have been optimized:
151  Code_item section size:
152|Old: |New: |Diff:|Per:  |File:
153| 1045| 1037|    8| 0.77%| ICULocaleService$LocaleKey.class
154|  302|  294|    8| 2.65%| Locale$LocaleKey.class
155| 5700| 5693|    7| 0.12%| LanguageTag.class
156|  957|  949|    8| 0.84%| LinkedList$ListItr.class
157|  780|  772|    8| 1.03%| FieldPosition.class
158
159Summary:
160=============
161 Total code_item section size of baseline files: 9201 bytes
162 Total code_item section size of compared files: 9162 bytes
163 Difference: 39 bytes [0.42%]
164 Number of optimized files: 5
165 Number of not optimized files : 1
166 Files with no code item section: 0
167 Files that are bigger than baseline: 0
168 Failed tests on baseline: 1
169 Failed tests compared to baseline: 0
170=============
171
172Statistics on optimized files:
173=============
174 Total code_item section size of baseline files: 8784 bytes
175 Total code_item section size of compared files: 8745 bytes
176 Difference: 39 bytes [0.44%]
177=============
178```
179