1 2# Protocol Buffers Benchmarks 3 4This directory contains benchmarking schemas and data sets that you 5can use to test a variety of performance scenarios against your 6protobuf language runtime. If you are looking for performance 7numbers of officially support languages, see [here]( 8https://github.com/protocolbuffers/protobuf/blob/master/docs/performance.md) 9 10## Prerequisite 11 12First, you need to follow the instruction in the root directory's README to 13build your language's protobuf, then: 14 15### CPP 16You need to install [cmake](https://cmake.org/) before building the benchmark. 17 18We are using [google/benchmark](https://github.com/google/benchmark) as the 19benchmark tool for testing cpp. This will be automaticly made during build the 20cpp benchmark. 21 22The cpp protobuf performance can be improved by linking with [tcmalloc library]( 23https://gperftools.github.io/gperftools/tcmalloc.html). For using tcmalloc, you 24need to build [gpertools](https://github.com/gperftools/gperftools) to generate 25libtcmallc.so library. 26 27### Java 28We're using maven to build the java benchmarks, which is the same as to build 29the Java protobuf. There're no other tools need to install. We're using 30[google/caliper](https://github.com/google/caliper) as benchmark tool, which 31can be automaticly included by maven. 32 33### Python 34We're using python C++ API for testing the generated 35CPP proto version of python protobuf, which is also a prerequisite for Python 36protobuf cpp implementation. You need to install the correct version of Python 37C++ extension package before run generated CPP proto version of Python 38protobuf's benchmark. e.g. under Ubuntu, you need to 39 40``` 41$ sudo apt-get install python-dev 42$ sudo apt-get install python3-dev 43``` 44And you also need to make sure `pkg-config` is installed. 45 46### Go 47Go protobufs are maintained at [github.com/golang/protobuf]( 48http://github.com/golang/protobuf). If not done already, you need to install the 49toolchain and the Go protoc-gen-go plugin for protoc. 50 51To install protoc-gen-go, run: 52 53``` 54$ go get -u github.com/golang/protobuf/protoc-gen-go 55$ export PATH=$PATH:$(go env GOPATH)/bin 56``` 57 58The first command installs `protoc-gen-go` into the `bin` directory in your local `GOPATH`. 59The second command adds the `bin` directory to your `PATH` so that `protoc` can locate the plugin later. 60 61### PHP 62PHP benchmark's requirement is the same as PHP protobuf's requirements. The benchmark will automaticly 63include PHP protobuf's src and build the c extension if required. 64 65### Node.js 66Node.js benchmark need [node](https://nodejs.org/en/)(higher than V6) and [npm](https://www.npmjs.com/) package manager installed. This benchmark is using the [benchmark](https://www.npmjs.com/package/benchmark) framework to test, which needn't to manually install. And another prerequisite is [protobuf js](https://github.com/protocolbuffers/protobuf/tree/master/js), which needn't to manually install either 67 68### C# 69The C# benchmark code is built as part of the main Google.Protobuf 70solution. It requires the .NET Core SDK, and depends on 71[BenchmarkDotNet](https://github.com/dotnet/BenchmarkDotNet), which 72will be downloaded automatically. 73 74### Big data 75 76There's some optional big testing data which is not included in the directory 77initially, you need to run the following command to download the testing data: 78 79``` 80$ ./download_data.sh 81``` 82 83After doing this the big data file will automatically generated in the 84benchmark directory. 85 86## Run instructions 87 88To run all the benchmark dataset: 89 90### Java: 91 92``` 93$ make java 94``` 95 96### CPP: 97 98``` 99$ make cpp 100``` 101 102For linking with tcmalloc: 103 104``` 105$ env LD_PRELOAD={directory to libtcmalloc.so} make cpp 106``` 107 108### Python: 109 110We have three versions of python protobuf implementation: pure python, cpp 111reflection and cpp generated code. To run these version benchmark, you need to: 112 113#### Pure Python: 114 115``` 116$ make python-pure-python 117``` 118 119#### CPP reflection: 120 121``` 122$ make python-cpp-reflection 123``` 124 125#### CPP generated code: 126 127``` 128$ make python-cpp-generated-code 129``` 130 131### Go 132``` 133$ make go 134``` 135 136 137### PHP 138We have two version of php protobuf implemention: pure php, php with c extension. To run these version benchmark, you need to: 139#### Pure PHP 140``` 141$ make php 142``` 143#### PHP with c extension 144``` 145$ make php_c 146``` 147 148### Node.js 149``` 150$ make js 151``` 152 153To run a specific dataset or run with specific options: 154 155### Java: 156 157``` 158$ make java-benchmark 159$ ./java-benchmark $(specific generated dataset file name) [$(caliper options)] 160``` 161 162### CPP: 163 164``` 165$ make cpp-benchmark 166$ ./cpp-benchmark $(specific generated dataset file name) [$(benchmark options)] 167``` 168 169### Python: 170 171For Python benchmark we have `--json` for outputing the json result 172 173#### Pure Python: 174 175``` 176$ make python-pure-python-benchmark 177$ ./python-pure-python-benchmark [--json] $(specific generated dataset file name) 178``` 179 180#### CPP reflection: 181 182``` 183$ make python-cpp-reflection-benchmark 184$ ./python-cpp-reflection-benchmark [--json] $(specific generated dataset file name) 185``` 186 187#### CPP generated code: 188 189``` 190$ make python-cpp-generated-code-benchmark 191$ ./python-cpp-generated-code-benchmark [--json] $(specific generated dataset file name) 192``` 193 194### Go: 195``` 196$ make go-benchmark 197$ ./go-benchmark $(specific generated dataset file name) [go testing options] 198``` 199 200### PHP 201#### Pure PHP 202``` 203$ make php-benchmark 204$ ./php-benchmark $(specific generated dataset file name) 205``` 206#### PHP with c extension 207``` 208$ make php-c-benchmark 209$ ./php-c-benchmark $(specific generated dataset file name) 210``` 211 212### Node.js 213``` 214$ make js-benchmark 215$ ./js-benchmark $(specific generated dataset file name) 216``` 217 218### C# 219From `csharp/src/Google.Protobuf.Benchmarks`, run: 220 221``` 222$ dotnet run -c Release 223``` 224 225We intend to add support for this within the makefile in due course. 226 227## Benchmark datasets 228 229Each data set is in the format of benchmarks.proto: 230 2311. name is the benchmark dataset's name. 2322. message_name is the benchmark's message type full name (including package and message name) 2333. payload is the list of raw data. 234 235The schema for the datasets is described in `benchmarks.proto`. 236 237Benchmark likely want to run several benchmarks against each data set (parse, 238serialize, possibly JSON, possibly using different APIs, etc). 239 240We would like to add more data sets. In general we will favor data sets 241that make the overall suite diverse without being too large or having 242too many similar tests. Ideally everyone can run through the entire 243suite without the test run getting too long. 244