1# Overview of performance test suite, with steps for manual runs: 2 3For design of the tests, see 4https://grpc.io/docs/guides/benchmarking. 5 6## Pre-reqs for running these manually: 7In general the benchmark workers and driver build scripts expect 8[linux_performance_worker_init.sh](../../gce/linux_performance_worker_init.sh) to have been ran already. 9 10### To run benchmarks locally: 11* From the grpc repo root, start the 12[run_performance_tests.py](../run_performance_tests.py) runner script. 13 14### On remote machines, to start the driver and workers manually: 15The [run_performance_test.py](../run_performance_tests.py) top-level runner script can also 16be used with remote machines, but for e.g., profiling the server, 17it might be useful to run workers manually. 18 191. You'll need a "driver" and separate "worker" machines. 20For example, you might use one GCE "driver" machine and 3 other 21GCE "worker" machines that are in the same zone. 22 232. Connect to each worker machine and start up a benchmark worker with a "driver_port". 24 * For example, to start the grpc-go benchmark worker: 25 [grpc-go worker main.go](https://github.com/grpc/grpc-go/blob/master/benchmark/worker/main.go) --driver_port <driver_port> 26 27#### Commands to start workers in different languages: 28 * Note that these commands are what the top-level 29 [run_performance_test.py](../run_performance_tests.py) script uses to 30 build and run different workers through the 31 [build_performance.sh](./build_performance.sh) script and "run worker" 32 scripts (such as the [run_worker_java.sh](./run_worker_java.sh)). 33 34##### Running benchmark workers for C-core wrapped languages (C++, Python, C#, Node, Ruby): 35 * These are more simple since they all live in the main grpc repo. 36 37``` 38$ cd <grpc_repo_root> 39$ tools/run_tests/performance/build_performance.sh 40$ tools/run_tests/performance/run_worker_<language>.sh 41``` 42 43 * Note that there is one "run_worker" script per language, e.g., 44 [run_worker_csharp.sh](./run_worker_csharp.sh) for c#. 45 46##### Running benchmark workers for gRPC-Java: 47 * You'll need the [grpc-java](https://github.com/grpc/grpc-java) repo. 48 49``` 50$ cd <grpc-java-repo> 51$ ./gradlew -PskipCodegen=true -PskipAndroid=true :grpc-benchmarks:installDist 52$ benchmarks/build/install/grpc-benchmarks/bin/benchmark_worker --driver_port <driver_port> 53``` 54 55##### Running benchmark workers for gRPC-Go: 56 * You'll need the [grpc-go repo](https://github.com/grpc/grpc-go) 57 58``` 59$ cd <grpc-go-repo>/benchmark/worker && go install 60$ # if profiling, it might be helpful to turn off inlining by building with "-gcflags=-l" 61$ $GOPATH/bin/worker --driver_port <driver_port> 62``` 63 64#### Build the driver: 65* Connect to the driver machine (if using a remote driver) and from the grpc repo root: 66``` 67$ tools/run_tests/performance/build_performance.sh 68``` 69 70#### Run the driver: 711. Get the 'scenario_json' relevant for the scenario to run. Note that "scenario 72 json" configs are generated from [scenario_config.py](./scenario_config.py). 73 The [driver](../../../test/cpp/qps/qps_json_driver.cc) takes a list of these configs as a json string of the form: `{scenario: <json_list_of_scenarios> }` 74 in its `--scenarios_json` command argument. 75 One quick way to get a valid json string to pass to the driver is by running 76 the [run_performance_tests.py](./run_performance_tests.py) locally and copying the logged scenario json command arg. 77 782. From the grpc repo root: 79 80* Set `QPS_WORKERS` environment variable to a comma separated list of worker 81machines. Note that the driver will start the "benchmark server" on the first 82entry in the list, and the rest will be told to run as clients against the 83benchmark server. 84 85Example running and profiling of go benchmark server: 86``` 87$ export QPS_WORKERS=<host1>:<10000>,<host2>,10000,<host3>:10000 88$ bins/opt/qps_json_driver --scenario_json='<scenario_json_scenario_config_string>' 89``` 90 91### Example profiling commands 92 93While running the benchmark, a profiler can be attached to the server. 94 95Example to count syscalls in grpc-go server during a benchmark: 96* Connect to server machine and run: 97``` 98$ netstat -tulpn | grep <driver_port> # to get pid of worker 99$ perf stat -p <worker_pid> -e syscalls:sys_enter_write # stop after test complete 100``` 101 102Example memory profile of grpc-go server, with `go tools pprof`: 103* After a run is done on the server, see its alloc profile with: 104``` 105$ go tool pprof --text --alloc_space http://localhost:<pprof_port>/debug/heap 106``` 107 108### Configuration environment variables: 109 110* QPS_WORKER_CHANNEL_CONNECT_TIMEOUT 111 112 Consuming process: qps_worker 113 114 Type: integer (number of seconds) 115 116 This can be used to configure the amount of time that benchmark 117 clients wait for channels to the benchmark server to become ready. 118 This is useful in certain benchmark environments in which the 119 server can take a long time to become ready. Note: if setting 120 this to a high value, then the scenario config under test should 121 probably also have a large "warmup_seconds". 122 123* QPS_WORKERS 124 125 Consuming process: qps_json_driver 126 127 Type: comma separated list of host:port 128 129 Set this to a comma separated list of QPS worker processes/machines. 130 Each scenario in a scenario config has specifies a certain number 131 of servers, `num_servers`, and the driver will start 132 "benchmark servers"'s on the first `num_server` `host:port` pairs in 133 the comma separated list. The rest will be told to run as clients 134 against the benchmark server. 135