1# Overview 2 3This directory contains scripts that facilitate building and running gRPC tests for combinations of language/runtimes (known as matrix). 4 5The setup builds gRPC docker images for each language/runtime and upload it to Google Container Registry (GCR). These images, encapsulating gRPC stack 6from specific releases/tag, are used to test version compatiblity between gRPC release versions. 7 8## Step-by-step instructions for adding a GCR image for a new release for compatibility test 9We have continuous nightly test setup to test gRPC backward compatibility between old clients and latest server. When a gRPC developer creates a new gRPC release, s/he is also responsible to add the just-released gRPC client to the nightly test. The steps are: 10- Add (or update) an entry in `./client_matrix.py` file to reference the github tag for the release. 11- Build new client docker image(s). For example, for C and wrapper languages release `v1.9.9`, do 12 - `tools/interop_matrix/create_matrix_images.py --git_checkout --release=v1.9.9 --upload_images --language cxx csharp python ruby php` 13- Verify that the new docker image was built successfully and uploaded to GCR. For example, 14 - `gcloud beta container images list --repository gcr.io/grpc-testing` shows image repos. 15 - `gcloud beta container images list-tags gcr.io/grpc-testing/grpc_interop_java_oracle8` should show an image entry with tag `v1.9.9`. 16- Verify the just-created docker client image would pass backward compatibility test (it should). For example, 17 - `gcloud docker -- pull gcr.io/grpc-testing/grpc_interop_java_oracle8:v1.9.9` followed by 18 - `docker_image=gcr.io/grpc-testing/grpc_interop_java_oracle8:v1.9.9 tools/interop_matrix/testcases/java__master` 19- git commit the change and merge it to upstream/master. 20- (Optional) clean up the tmp directory to where grpc source is cloned at `/export/hda3/tmp/grpc_matrix/`. 21For more details on each step, refer to sections below. 22 23## Instructions for adding new language/runtimes* 24- Create new `Dockerfile.template`, `build_interop.sh.template` for the language/runtime under `template/tools/dockerfile/`. 25- Run `tools/buildgen/generate_projects.sh` to create corresponding files under `tools/dockerfile/`. 26- Add language/runtimes to `client_matrix.py` following existing language/runtimes examples. 27- Run `tools/interop_matrix/create_matrix_images.py` which will build and upload images to GCR. Unless you are also building images for a gRPC release, make sure not to set `--release` (the default release 'master' is used for testing). 28 29*: Please delete your docker images at https://pantheon.corp.google.com/gcr/images/grpc-testing?project=grpc-testing afterwards. Permissions to access GrpcTesting project is required for this step. 30 31## Instructions for creating new test cases 32- Create test cases by running `LANG=<lang> [RELEASE=<release>] ./create_testcases.sh`. For example, 33 - `LANG=go ./create_testcases.sh` will generate `./testcases/go__master`, which is also a functional bash script. 34 - `LANG=go KEEP_IMAGE=1 ./create_testcases.sh` will generate `./testcases/go__master` and keep the local docker image so it can be invoked simply via `./testcases/go__master`. Note: remove local docker images manually afterwards with `docker rmi <image_id>`. 35- Stage and commit the generated test case file `./testcases/<lang>__<release>`. 36 37## Instructions for running test cases against GCR images 38- Run `tools/interop_matrix/run_interop_matrix_tests.py`. Useful options: 39 - `--release` specifies a git release tag. Defaults to `--release=all`. Make sure the GCR images with the tag have been created using `create_matrix_images.py` above. 40 - `--language` specifies a language. Defaults to `--language=all`. 41 For example, To test all languages for all gRPC releases across all runtimes, do `tools/interop_matrix/run_interop_matrix_test.py --release=all`. 42- The output for all the test cases is recorded in a junit style xml file (default to 'report.xml'). 43 44## Instructions for running test cases against a GCR image manually 45- Download docker image from GCR. For example: `gcloud docker -- pull gcr.io/grpc-testing/grpc_interop_go1.7:master`. 46- Run test cases by specifying `docker_image` variable inline with the test case script created above. 47For example: 48 - `docker_image=gcr.io/grpc-testing/grpc_interop_go1.7:master ./testcases/go__master` will run go__master test cases against `go1.7` with gRPC release `master` docker image in GCR. 49 50Note: 51- File path starting with `tools/` or `template/` are relative to the grpc repo root dir. File path starting with `./` are relative to current directory (`tools/interop_matrix`). 52- Creating and referencing images in GCR require read and write permission to Google Container Registry path gcr.io/grpc-testing. 53