README.md
1# Overview
2
3This directory contains scripts that facilitate building and running gRPC interoperability tests for combinations of language/runtimes (known as matrix).
4
5The setup builds gRPC docker images for each language/runtime and upload it to Artifact Registry (AR). These images, encapsulating gRPC stack
6from specific releases/tag, are used to test version compatibility between gRPC release versions.
7
8## Step-by-step instructions for adding a AR docker image for a new release for compatibility test
9
10We have continuous nightly test setup to test gRPC backward compatibility between old clients and latest server.
11When a gRPC developer creates a new gRPC release, s/he is also responsible to add the just-released gRPC client to the nightly test.
12The steps are:
13- Add (or update) an entry in `./client_matrix.py` file to reference the github tag for the release.
14- Build new client docker image(s). For example, for C and wrapper languages release `v1.9.9`, do
15 - `tools/interop_matrix/create_matrix_images.py --git_checkout --release=v1.9.9 --upload_images --language cxx python ruby php`
16- Verify that the new docker image was built successfully and uploaded to AR. For example,
17 - `gcloud artifacts docker images list us-docker.pkg.dev/grpc-testing/testing-images-public` lists available images.
18 - `gcloud artifacts docker images list us-docker.pkg.dev/grpc-testing/testing-images-public/grpc_interop_java --include-tags` should show an image entry with tag `v1.9.9`.
19 - images can also be viewed in https://pantheon.corp.google.com/artifacts/docker/grpc-testing/us/testing-images-public
20- Verify the just-created docker client image would pass backward compatibility test (it should). For example,
21 - `docker pull us-docker.pkg.dev/grpc-testing/testing-images-public/grpc_interop_java:v1.9.9` followed by
22 - `docker_image=us-docker.pkg.dev/grpc-testing/testing-images-public/grpc_interop_java:v1.9.9 tools/interop_matrix/testcases/java__master`
23- Commit the change and create a PR to upstream/master.
24- Trigger an adhoc run of interop matrix tests: https://fusion.corp.google.com/projectanalysis/summary/KOKORO/prod:grpc%2Fcore%2Fexperimental%2Flinux%2Fgrpc_interop_matrix_adhoc
25- Once tests pass, request a PR review.
26- (Optional) clean up the tmp directory to where grpc source is cloned at `/export/hda3/tmp/grpc_matrix/`.
27For more details on each step, refer to sections below.
28
29## Instructions for adding new language/runtimes
30
31- Create new `Dockerfile.template`, `build_interop.sh.template` for the language/runtime under `template/tools/dockerfile/`.
32- Run `tools/buildgen/generate_projects.sh` to create corresponding files under `tools/dockerfile/`.
33- Add language/runtimes to `client_matrix.py` following existing language/runtimes examples.
34- Run `tools/interop_matrix/create_matrix_images.py` which will build (and upload) images to AR.
35
36## Instructions for creating new test cases
37
38- Create test cases by running `LANG=<lang> [RELEASE=<release>] ./create_testcases.sh`. For example,
39 - `LANG=go ./create_testcases.sh` will generate `./testcases/go__master`, which is also a functional bash script.
40 - `LANG=go KEEP_IMAGE=1 ./create_testcases.sh` will generate `./testcases/go__master` and keep the local docker image so it can be invoked simply via `./testcases/go__master`. Note: remove local docker images manually afterwards with `docker rmi <image_id>`.
41- Stage and commit the generated test case file `./testcases/<lang>__<release>`.
42
43## Instructions for running test cases against AR docker images
44
45- Run `tools/interop_matrix/run_interop_matrix_tests.py`. Useful options:
46 - `--release` specifies a git release tag. Defaults to `--release=all`. Make sure the AR images with the tag have been created using `create_matrix_images.py` above.
47 - `--language` specifies a language. Defaults to `--language=all`.
48 For example, To test all languages for all gRPC releases across all runtimes, do `tools/interop_matrix/run_interop_matrix_test.py --release=all`.
49- The output for all the test cases is recorded in a junit style xml file (defaults to 'report.xml').
50
51## Instructions for running test cases against an AR image manually
52
53- Download a docker image from AR. For example: `docker pull us-docker.pkg.dev/grpc-testing/testing-images-public/grpc_interop_go1.8:v1.16.0`.
54- Run test cases by specifying `docker_image` variable inline with the test case script created above.
55For example:
56 - `docker_image=us-docker.pkg.dev/grpc-testing/testing-images-public/grpc_interop_go1.8:v1.16.0 ./testcases/go__master` will run go__master test cases against `go1.8` with gRPC release `v1.16.0` docker image in AR.
57
58Note:
59- File path starting with `tools/` or `template/` are relative to the grpc repo root dir. File path starting with `./` are relative to current directory (`tools/interop_matrix`).
60- Creating and referencing images in AR require read and write permission to AR path us-docker.pkg.dev/grpc-testing.
61