xref: /external/deqp/external/openglcts/README.md

OpenGL and OpenGL ES 2.0/3.X Conformance Test Instructions
=================

This document describes how to build, port, and run the OpenGL and OpenGL ES
2.0/3.X conformance tests, and how to verify and submit test results.

The Conformance Tests are built on the dEQP framework.
Up-to-date documentation for dEQP is available at:

* [The VK-GL-CTS wiki for Khronos members](https://gitlab.khronos.org/Tracker/vk-gl-cts/wikis/home)
* [The VK-GL-CTS wiki for non-Khronos members](https://github.com/KhronosGroup/VK-GL-CTS/wiki)


Contents
------------------------
 - [Test History](#test-history)
 - [Introduction](#introduction)
 - [Test Environment Requirements](#test-environment-requirements)
 - [Configuring and Building the Tests](#configuring-and-building-the-tests)
    - [Configuration](#configuration)
    - [Building the Tests](#building-the-tests)
       - [Windows](#windows)
       - [Linux](#linux)
       - [Android](#android)
 - [Porting](#porting)
    - [Common Porting Changes](#common-porting-changes)
    - [Other Allowable Porting Changes](#other-allowable-porting-changes)
 - [Running the Tests](#running-the-tests)
    - [Conformance runs](#conformance-runs)
       - [Linux and Windows](#linux-and-windows)
       - [Android](#android-1)
    - [Running Subsets](#running-subsets)
       - [Command line options](#command-line-options)
    - [Understanding the Results](#understanding-the-results)
    - [Test Logs](#test-logs)
 - [Debugging Test Failures](#debugging-test-failures)
 - [Waivers](#waivers)
 - [Creating a Submission Package](#creating-a-submission-package)
 - [Submission Update Package](#submission-update-package)
 - [Passing Criteria](#passing-criteria)
 - [Troubleshooting](#troubleshooting)
    - [Crashes early on in the run](#crashes-early-on-in-the-run)
    - [Build fails](#build-fails)
 - [Adding new tests](#adding-new-tests)
 - [Acknowledgments](#acknowledgments)
 - [Revision History](#revision-history)

Test History
------------------------
The OpenGL and OpenGL ES Conformance Tests are expanded versions of the
OpenGL ES 2.x Conformance Test. Much of the development was done by Symbio, Inc.
under a contract with The Khronos Group. drawElements donated a considerable
number of new tests and a new execution framework for version 1.1.
The tests are built from the same source code base, although some individual
feature tests are specific to OpenGL or OpenGL ES and their specification
versions, and compilation options differing between OpenGL and OpenGL ES affect
how the tests are compiled and executed in some cases.

Introduction
------------------------

This document contains instructions for certifying conformance of implementations
of the OpenGL and OpenGL ES APIs. The steps of the process are as follows:

1. Configure the conformance tests and port them to your platform.
2. Build a test executable and run it against your implementation to produce
result logs.
3. Debug any test failures and modify your implementation as needed until it
passes the test.
4. Create a Submission Package containing your final result logs and other
documents describing the tested platform.
5. Submit the results to the appropriate Review Committee via the
Khronos Adopters web page. The Committee will examine your submission and will
notify you within thirty days if they find any issues requiring action on your part.

This document describes each of these steps in detail. It also provides advice
on reproducing, understanding, and debugging test failures, and discusses how
to extend or modify the tests and the test framework.

The reader is assumed to be a fluent programmer experienced with command line
utilities and build tools, such as CMake or Make.

Test Environment Requirements
------------------------

The conformance tests require a file system. The file system requires support
for long file names (i.e. > 8.3 name format). Source files in the conformance
tests use mixed case file names. When the `--verbose` option is used, rendered
images and test case shaders are copied to the log files. This can lead to quite
large log files, up to hundreds of megabytes on disk.

Each execution of the conformance test writes a text-format results log to a disk.
You will need to include this log as part of your conformance submission package.

The conformance test executable can be large. Compiler options and CPU instruction
sets can cause substantial variation. The disk space required for the build
including all the temporary files can be up to 400MB.

The build environment is expected to support C++ with exceptions and
the Standard Template Library (STL).

Configuring and Building the Tests
------------------------
The CTS is built via CMake build system. The requirements for the build are as follows:
- CMake 3.0 (3.6 for Android NDK r17+ builds) or newer
- C++ compiler with STL and exceptions support
- Unix: Make + GCC / Clang
- Windows: Visual Studio or Windows SDK (available free-of-charge)
- Android: Android SDK and NDK for host platform

The build is controlled by the file CMakeLists.txt found at the root of
the CTS source.

If the platform and compiler tools you use are not supported, you may be able to
add support for that platform and tools to the build system. If you do this,
please submit your changes back to Khronos for inclusion in the official tests
going forward.

Otherwise, if you choose not to use the supplied Makefiles, you must construct
an equivalent build system for the chosen development environment(s).

### Configuration

The build is configured by using `CMakeLists.txt` files in the build target
directory (`targets/`).  They specify platform-specific configuration, including
include paths and link libraries.

The main `CMakeLists.txt` includes the target file based on the `DEQP_TARGET`
variable. For example `-DDEQP_TARGET=my_target` will use the target description
file `targets/my_target/my_target.cmake`.

See the main `CMakeLists.txt` file for the description of the variables that
the target file can set.

Porting to a new platform includes either creating a new target file, or
modifying an existing target description.

**NOTE**: All paths, except `TCUTIL_PLATFORM_SRCS` are relative to root source
directory. `TCUTIL_PLATFORM_SRCS` is relative to `framework/platform` directory.

Following target files are provided with the package:

| Name | Description  |
|:---------|-----------------|
|android | Used in Android build. Requires use of suitable toolchain file (see `cmake/` directory) |
|default| Checks for presence of GL, ES2, ES3, and EGL libraries and headers in default search paths and configures build accordingly|
|null | Null build target |
|nullws | NullWS build target |
|x11_egl| X11 build for platforms with native EGL support|
|x11_glx| X11 build for platforms with native GLX support|
|x11_egl_glx| X11 build for platforms with native EGL/GLX support|

**Example target file (targets/null/null.cmake):**
```
message("*** Using null context target")

set(DEQP_TARGET_NAME "Null")

set(TCUTIL_PLATFORM_SRCS
	null/tcuNullPlatform.cpp
	null/tcuNullPlatform.hpp
	null/tcuNullRenderContext.cpp
	null/tcuNullRenderContext.hpp
	null/tcuNullContextFactory.cpp
	null/tcuNullContextFactory.hpp
	)
```

**Common configuration variables and their default values in CMake syntax:**

- Target name
```
set(DEQP_TARGET_NAME "UNKNOWN")
```

- List of link libraries per API.  If no libraries are specified, entry points
are loaded at run-time by default for OpenGL ES APIs. EGL always requires link
libraries.  OpenGL always uses run-time loading.
```
set(DEQP_GLES2_LIBRARIES   )
set(DEQP_GLES3_LIBRARIES   )
set(DEQP_GLES31_LIBRARIES  )
set(DEQP_GLES32_LIBRARIES  )
set(DEQP_EGL_LIBRARIES     )
set(DEQP_OPENGL_LIBRARIES  )
```

- Generic platform libraries required to link a working OpenGL (ES) Application
(e.g. X11 libraries on Unix/X11)
```
set(DEQP_PLATFORM_LIBRARIES )
```

- Libraries / binaries that need to be copied to the build target dir
```
set(DEQP_PLATFORM_COPY_LIBRARIES )
```

- If running on Linux using X11 for creating windows etc., enable this.
```
set(DEQP_USE_X11 OFF)
```

- Embed the test files in the test Before building with this set (if GTF module is present), run these commands:
```
cd external/kc-cts/src/GTF_ES/glsl/GTF
perl mergeTestFilesToCSource.pl
```

 In your target `.cmake` file add
```
set(DEQP_EMBED_TESTS ON)
add_definitions(-DHKEMBEDDEDFILESYSTEM)
```

### Building the Tests

To build the framework, you need first to download sources for zlib, libpng, glslang,
spirv-headers, and spirv-tools.

To download sources, run:

	python external/fetch_sources.py

For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0
download Khronos Confidential Conformance Test Suite:

	python external/fetch_kc_cts.py

For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0
the results for the tests included in this suite must be included in a
conformance submission.

**NOTE**: You need to be a Khronos Adopter and have an active account
at [Khronos Gitlab](https://gitlab.khronos.org/) to be able to download
Khronos Confidential CTS.
It is possible to run and build the CTS without the Khronos Confidential CTS.
For OpenGL CTS releases, and OpenGL ES CTS releases prior to opengl-es-cts-3.2.4.0
Khronos Confidential CTS is mandatory if you plan to make a
conformance submission (see [Creating a Submission Package](#creating-a-submission-package)).
For opengl-es-cts-3.2.4.0 and later OpenGL ES CTS releases Khronos Confidential CTS
results must not be included in a submission package.


With CMake out-of-source builds are always recommended. Create a build directory
of your choosing, and in that directory generate Makefiles or IDE project
using Cmake.

#### Windows

Requirements:
- Visual Studio (2015 or newer recommended) or Windows SDK
- CMake 3.10.2 Windows native version (i.e. not Cygwin version)
- For GL/ES2/ES3.x tests: OpengGL, OpenGL ES 2 or ES 3.x libraries and headers

To choose the backend build system for CMake, choose one of the following Generator Names for the
command line examples in the next steps:
- VS2015: "Visual Studio 14"
- NMake (must be run in VS or SDK command prompt): "NMake Makefiles"

Building GL, ES2, or ES3.x conformance tests:

	cmake <path to VK-GL-CTS> -DDEQP_TARGET=default -G"<Generator Name>"
	cmake --build external/openglcts

Khronos Confidential CTS doesn't support run-time selection of API context.
If you intend to run it you need to additionally supply `GLCTS_GTF_TARGET`
option to you cmake command, e.g.:

	cmake <path to VK-GL-CTS> -DDEQP_TARGET=default -DGLCTS_GTF_TARGET=<target> -G"<Generator Name>"

Available `<target>`s are `gles2`, `gles3`, `gles31`, `gles32`, and `gl`.
The default `<target>` is `gles32`.

It's also possible to build `GL-CTS.sln` in Visual Studio instead of running
the `cmake --build external/openglcts` command.

**NOTE**: Do not create the build directory under the source directory
(i.e anywhere under `<path to VK-GL-CTS>`) on Windows, since it causes
random build failures when copying data files around.

**NOTE**: You can use the CMake for Windows GUI to do configuration and project
file generation.

**NOTE**: If using cygwin, you must install and ensure you use the Windows
version of cmake. The cygwin vesion does not contain the Visual Studio
generators. Here is a shell function you can put in your cygwin `.bash_profile`
to use it easily. With this you can simply type `wcmake` to run the Windows version.

```
function wcmake () {
    (TMP=$tmp TEMP=$temp; unset tmp; unset temp; "C:/Program Files (x86)/CMake 2.8/bin/cmake" "$@")
}
```

#### Linux

Required tools:
- Standard build utilities (make, gcc, etc.)
- CMake 3.10.2
- Necessary API libraries (OpenGL, GLES, EGL depending on configuration)

Building ES2 or ES3.x conformance tests:

	cmake <path to VK-GL-CTS> -DDEQP_TARGET=null -DGLCTS_GTF_TARGET=gles32
	cmake --build external/openglcts

Building OpenGL conformance tests:

	cmake <path to VK-GL-CTS> -DDEQP_TARGET=null -DGLCTS_GTF_TARGET=gl
	cmake --build external/openglcts

Khronos Confidential CTS doesn't support run-time selection of API context.
If you intend to run it then the `GLCTS_GTF_TARGET` option is necessary.

Available values for `GLCTS_GTF_TARGET` are `gles2`, `gles3`, `gles31`, `gles32`, and `gl`.
The default value is `gles32`.

CMake chooses to generate Makefiles by default. Other generators can be used
as well. See CMake help for more details.

#### Android

The conformance tests come with native Android support. The following packages
are needed in order to build an Android binary:
- Python 3.x (for the build related scripts, some other scripts still use Python 2.7.x)
- Android NDK r17c
- Android SDK with API 28 packages and tools installed
- Apache Ant

An Android binary (for ES 3.2) can be built using command:

	python scripts/android/build_apk.py --target=openglcts --sdk <path to Android SDK> --ndk <path to Android NDK>

By default the CTS package will be built for the Android API level 28.
Another API level may be supplied using --native-api command line option.

If Khronos Confidential CTS is present then the script will set `GLCTS_GTF_TARGET`
to `gles32` by default.
It is possible to specify a different `GLCTS_GTF_TARGET` target by invoking the script
with the `--kc-cts-target` option, e.g.:

	python scripts/android/build_apk.py --target=openglcts --kc-cts-target=gles31 --sdk <path to Android SDK> --ndk <path to Android NDK>

Available values for `--kc-cts-target` are `gles32`, `gles31`, `gles3`, `gles2` and `gl`.

The package can be installed by either running:

	python scripts/android/install_apk.py --target=openglcts

By default the CTS package will contain libdeqp.so built for `armeabi-v7a`, `arm64-v8a`,
`x86`, and `x86_64` ABIs, but that can be changed with `--abis` command line option.

To pick which ABI to use at install time, following commands must be used
instead:

	adb install -g --abi <ABI name> <build root>/Khronos-CTS.apk /data/local/tmp/Khronos-CTS.apk

Porting
------------------------
The Conformance Tests have been designed to be relatively platform-, OS-, and
compiler-independent. Adopters are responsible for final changes needed to allow
the Test to run on the platform they wish to
certify as conformant.

### Common Porting Changes

Porting the dEQP framework requires implementation of either `glu::Platform` or,
on platforms supporting EGL, the `tcu::EglPlatform` interface. The porting layer
API is described in detail in following files:

	framework/common/tcuPlatform.hpp
	framework/opengl/gluPlatform.hpp
	framework/egl/egluPlatform.hpp
	framework/platform/tcuMain.cpp

This version of the dEQP framework includes ports for Windows (both EGL and WGL),
X11 (EGL and XGL), and Android.

Base portability libraries in `framework/delibs` seldom need changes. However,
introducing support for a new compiler or a new processor family may require
some changes to correctly detect and parameterize the environment.

Porting typically involves three types of changes:
1. Changes to the make system used to generate the test executable.
2. Changes needed to adapt the test executable to the operating system used on the platform.
3. Changes to the platform specific GL and EGL header files.

Changes should normally be confined to build files (CMake or Python) or source
files (.c, .h, .cpp, and .h files) in the following directories or their
subdirectories:
- `framework/platform`
- `targets`

If you find that you must change other source (.c, .cpp, .h, or .hpp) files,
you will need to file a waiver as described below.

Note that the conformance tests assume that the implementation supports EGL.
However EGL is not required for OpenGL or OpenGL ES conformance.

Most of the tests require at least 256x256 pixels resolution in order to run properly
and produce stable results. It is, therefore, important to ensure that a port to a
new platform can support surfaces that fulfill width and height requirements.

### Other Allowable Changes

Changes to fix bugs in the conformance test are allowed. A bug in the conformance
test is a behavior which causes clearly incorrect execution (e.g., hanging, crashing,
or memory corruption), OR which requires behavior which contradicts or exceeds
the requirements of the relevant OpenGL or OpenGL ES Specification. Before
being used for a submission, bugfixes must be accepted and merged into
the CTS repository. `git cherry-pick` is strongly recommended as a method of
applying bug fixes.

Other changes must be accompanied by a [waiver](#waivers).

NOTE: When cherry-picking patches on top of release tag, please use `git cherry-pick -x`
to include original commit hash in the commit message.

Running the Tests
------------------------
All the following commands need to be run in the CTS build directory. If you
need to move the binaries from the build directory, remember to copy the
data directories named `gl_cts`, `gles2`, `gles3`, and `gles31` and its subdirectories
from the build directory to the test target in the same relative locations.

If the build instructions have been followed as-is, the correct path is:

	cd <builddir>/external/openglcts/modules

### Conformance runs
A conformance run can be launched either by running the `cts-runner` binary with
appropriate options on Linux/Windows or by running an Android application.

### Linux and Windows
Conformance run for OpenGL ES 3.2 on Windows:

	Debug/cts-runner.exe --type=es32
	  [For ES 3.1 use --type=es31; ES 3.0 use --type=es3; for ES 2.0, use --type=es2]

Conformance run for OpenGL 3.0 - 4.6 on Windows:

	Debug/cts-runner.exe --type=glxy
	  [x and y are the major and minor specifiction versions]

Full list of parameters for the `cts-runner` binary:
```
--type=[esN[M]|glNM] Conformance test run type. Choose from
					 ES: es2, es3, es31, es32
					 GL: gl30, gl31, gl32, gl33, gl40, gl41, gl42, gl43, gl44, gl45, gl46
--waivers=[path]     Path to xml file containing waived tests
--logdir=[path]      Destination directory for log files
--summary            Print summary without running the tests
--verbose            Print out and log more information
```

The conformance run will create one or more `.qpa` files per tested config, a
summary `.qpa` file containing run results and a summary `.xml` file containing
command line options for each run, all of which should be included in your
conformance submission package. The final verdict will be printed out at
the end of run.

Sometimes it is useful to know the command line options used for the conformance
before the run completed. Full conformance run configuration is written
to `cts-run-summary.xml` and this file can be generated by adding `--summary`
parameter.

By default the `cts-runner` does not include result images or shaders used in
the logs. Adding parameter `--verbose` will cause them to be included in
the logs. Images will be embedded as PNG data into the`.qpa` log files.
See Section [Test Logs](#test-logs) for instructions on how to view the images.

To direct logs to a directory, add `--logdir=[path]` parameter.

To specify waived tests, add `--waivers=[path]` parameter.

**NOTE**: Due to the lack of support for run-time selection of API context in the
Khronos Confidential CTS, a conformance run may fail if it is executed for an API
version that doesn't match the `GLCTS_GTF_TARGET` value used during the build step.

#### Android

Once the CTS binary is built and installed on the device, a new application
called `ES3.2 CTS`, `ES3.1 CTS`, `ES3 CTS`, `ES2 CTS`, `GL4.5 CTS`, or `GL4.6 CTS`
(depending on the test version you built) should appear in the launcher.
Conformance test runs can be done by launching the applications.

Alternatively it is possible to start a conformance run from the command line,
for example to launch a GLES 3.2 conformance run use:

	am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs"

For GLES 2.0, GLES 3.0, GLES 3.1, GL 4.5, or GL 4.6 conformance runs, substitute
the following activity name (respectively) ES2Activity, ES3Activity, ES31Activity,
GL45Activity, or GL46Activity.

Test logs will be written to `/sdcard` by default. The log path can be
customized by supplying a `logdir` string extra in launch intent. Verbose mode
can be enabled by supplying a `verbose` = `"true"` string extra. See
the following example:

	am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs" -e verbose "true"

Conformance run configuration can be generated by supplying a `summary` = `"true"`
string extra. See the following example:

	am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs" -e summary "true"

Waivers can be specified by supplying a `waivers` string extra. See the following example:

	am start -n org.khronos.gl_cts/org.khronos.cts.ES32Activity -e logdir "/sdcard/logs" -e waivers "/sdcard/waivers.xml"

**NOTE**: Supplying a `summary` = `"true"` string extra will result in the `cts-run-summary.xml` file
being written out but no tests will be executed.

Individual tests can be launched as well by targeting
`org.khronos.gl_cts/android.app.NativeActivity` activity. Command line
arguments must be supplied in a `cmdLine` string extra. See following example:

	am start -n org.khronos.gl_cts/android.app.NativeActivity -e cmdLine "cts --deqp-case=KHR-GLES32.info.version --deqp-gl-config-id=1 --deqp-log-filename=/sdcard/ES32-egl-config-1.qpa --deqp-surface-width=128 --deqp-surface-height=128"

In addition to the detailed `*.qpa` output files, the Android port of the CTS
logs a summary of the test run, including the pass/fail status of each test.
This summary can be viewed using the Android *logcat* utility.

See Section [Running Subsets](#running-subsets) below for details on command
line parameters.

### Running Subsets

Run shader compiler loop test cases from the OpenGL ES 3.0 CTS using EGL config with ID 3:

	Debug/glcts.exe --deqp-case=KHR-GLES3.shaders.loops.* --deqp-gl-config-id=3

Note that the GL context version is determined by the case name. `KHR-GLES3` in
the example above selects OpenGL ES 3.0. The command to run the same test
against OpenGL version 4.1 is:

	Debug/glcts.exe --deqp-case=GL41-CTS.shaders.loops.* --deqp-gl-config-id=3

To list available test cases (writes out `*-cases.txt` files per module), run:

	Debug/glcts.exe --deqp-runmode=txt-caselist

The type of the run for cts-runner chooses a specific list of test cases to
be run. The selected tests can be checked from the summary logs. To run
the same tests, just give equivalent test selection parameters to the `glcts`.

#### Command line options

Full list of parameters for the `glcts` binary:
```
  -h, --help
    Show this help

  -n, --deqp-case=<value>
    Test case(s) to run, supports wildcards (e.g. dEQP-GLES2.info.*)

  --deqp-caselist=<value>
    Case list to run in trie format (e.g. {dEQP-GLES2{info{version,renderer}}})

  --deqp-caselist-file=<value>
    Read case list (in trie format) from given file

  --deqp-caselist-resource=<value>
    Read case list (in trie format) from given file located application's assets

  --deqp-stdin-caselist
    Read case list (in trie format) from stdin

  --deqp-log-filename=<value>
    Write test results to given file
    default: 'TestResults.qpa'

  --deqp-runmode=[execute|xml-caselist|txt-caselist|stdout-caselist]
    Execute tests, or write list of test cases into a file
    default: 'execute'

  --deqp-caselist-export-file=<value>
    Set the target file name pattern for caselist export
    default: '${packageName}-cases.${typeExtension}'

  --deqp-watchdog=[enable|disable]
    Enable test watchdog
    default: 'disable'

  --deqp-crashhandler=[enable|disable]
    Enable crash handling
    default: 'disable'

  --deqp-base-seed=<value>
    Base seed for test cases that use randomization
    default: '0'

  --deqp-test-iteration-count=<value>
    Iteration count for cases that support variable number of iterations
    default: '0'

  --deqp-visibility=[windowed|fullscreen|hidden]
    Default test window visibility
    default: 'windowed'

  --deqp-surface-width=<value>
    Use given surface width if possible
    default: '-1'

  --deqp-surface-height=<value>
    Use given surface height if possible
    default: '-1'

  --deqp-surface-type=[window|pixmap|pbuffer|fbo]
    Use given surface type
    default: 'window'

  --deqp-screen-rotation=[unspecified|0|90|180|270]
    Screen rotation for platforms that support it
    default: '0'

  --deqp-gl-context-type=<value>
    OpenGL context type for platforms that support multiple

  --deqp-gl-config-id=<value>
    OpenGL (ES) render config ID (EGL config id on EGL platforms)
    default: '-1'

  --deqp-gl-config-name=<value>
    Symbolic OpenGL (ES) render config name

  --deqp-gl-context-flags=<value>
    OpenGL context flags (comma-separated, supports debug and robust)

  --deqp-cl-platform-id=<value>
    Execute tests on given OpenCL platform (IDs start from 1)
    default: '1'

  --deqp-cl-device-ids=<value>
    Execute tests on given CL devices (comma-separated, IDs start from 1)
    default: ''

  --deqp-cl-build-options=<value>
    Extra build options for OpenCL compiler

  --deqp-egl-display-type=<value>
    EGL native display type

  --deqp-egl-window-type=<value>
    EGL native window type

  --deqp-egl-pixmap-type=<value>
    EGL native pixmap type

  --deqp-log-images=[enable|disable]
    Enable or disable logging of result images
    default: 'enable'

  --deqp-log-shader-sources=[enable|disable]
    Enable or disable logging of shader sources
    default: 'enable'

  --deqp-test-oom=[enable|disable]
    Run tests that exhaust memory on purpose
    default: 'enable'

  --deqp-archive-dir=<value>
    Path to test resource files
    default: '.'

  --deqp-log-flush=[enable|disable]
    Enable or disable log file fflush
    default: 'enable'


  --deqp-renderdoc=[enable|disable]
    Enable RenderDoc frame markers
    default: 'disable'

  --deqp-fraction=<value>
    Run a fraction of the test cases (e.g. N,M means run group%M==N)
    default: ''

  --deqp-fraction-mandatory-caselist-file=<value>
    Case list file that must be run for each fraction
    default: ''

  --deqp-waiver-file=<value>
    Read waived tests from given file
    default: ''

  --deqp-runner-type=[any|none|amber]
    Filter test cases based on runner
    default: 'any'

  --deqp-terminate-on-fail=[enable|disable]
    Terminate the run on first failure
    default: 'disable'

  --deqp-egl-config-id=<value>
    Legacy name for --deqp-gl-config-id
    default: '-1'

  --deqp-egl-config-name=<value>
    Legacy name for --deqp-gl-config-name

  --deqp-waiver-file=<value>
    Path to xml file containing waived tests
```

### Understanding the Results

At the end of a completed test run, a file called `cts-run-summary.xml` is
generated. It will contain summaries per configuration and the full command
lines for the `glcts` application
(See Section [Running Subsets](#running-subsets)) for debugging purposes.
Additionally, a summary string similar to one below is printed:
```
4/4 sessions passed, conformance test PASSED
```

If the run fails, the message will say `FAILED` instead of `PASSED`. Under
Linux or Windows, this string is printed to stdout if available. Under Android,
it is emitted to the Android logging system for access via *logcat*.

Each test case will be logged into the `.qpa` files in XML. Below is a minimal
example of a test case log. The Result element contains the final verdict in
the `StatusCode` attribute. Passing cases will have `Pass` and failing cases
`Fail`. Other results such as `QualityWarning`, `CompatibilityWarning`,
`NotSupported` or `ResourceError` are possible. Only `Fail` status will count
as failure for conformance purposes.
```
<TestCaseResult Version="0.3.2" CasePath="ES2-CTS.info.vendor" CaseType="SelfValidate">
    <Text>Vendor A</Text>
    <Result StatusCode="Pass">Pass</Result>
</TestCaseResult>
```

If the failure count is zero for all config sequences, the implementation
passes the test. Note that in addition to a successful test result,
a Submission Package must satisfy the conditions specified below under
[Passing Criteria](#passing-criteria) in order to achieve conformance certification.

### Test Logs

The CTS writes test logs in XML encapsulated in a simple plain-text container
format. Each tested configuration listed in `cts-run-summary.xml`

To analyse and process the log files, run the following scripts
- `verify_submission.py` located in [VK-GL-CTS-Tools](https://github.com/KhronosGroup/VK-GL-CTS-Tools): Script that verifies logs based on `cts-run-summary.xml` file.
- `scripts/log/log_to_csv.py`: This utility converts `.qpa` log into CSV format. This is
useful for importing results into other systems.
- `scripts/log/log_to_xml.py`: Converts `.qpa` into well-formed XML document. The document
can be then viewed in browser using the testlog.{xsl,css} files.

Some browsers, like Chrome, limit local file access. In such case, the files
must be accessed over HTTP. Python comes with a simple HTTP server suitable
for the purpose. Run `python -m SimpleHTTPServer` in the directory containing
the generated XML files and point the browser to `127.0.0.1:8000`.

Parser for the `.qpa` log file format in python is provided in
`scripts/log/log_parser.py`.

Python scripts require python 2.7 or newer in 2.x series. They are not
compatible with python 3.x.

Debugging Test Failures
------------------------
The best first step is to run the failing test cases via `glcts` executable to
get the more verbose logs. Use, for example, the `log_to_xml.py` script
detailed in Section [Test Logs](#test-logs), to view the generated logs.
If the visual inspection of the logs does not give sufficient hints on the
nature of the issue, inspecting the test code and stepping through it in
debugger should help.

Waivers
------------------------
The procedure for requesting a waiver is to report the issue by filing a bug
report in the Gitlab VK GL CTS project
(https://gitlab.khronos.org/Tracker/vk-gl-cts). When you create your submission
package, include references to the waivers as described in the adopters' agreement.
[Fully-qualified links](https://en.wikipedia.org/wiki/Fully_qualified_domain_name)
to bug reports are highly recommended.
Including as much information as possible in your bug report will ensure the issue
can be progressed as speedily as possible. Such bug report must
include a link to suggested file changes. Issues must be labeled `Waiver` and `OpenGL-ES`
(for OpenGL ES submissions) or `Waiver` and `OpenGL` (for OpenGL submissions) and
identify the CTS release tag and affected tests.

Creating a Submission Package
------------------------
Please see the [Creating a Submission Package page](https://github.com/KhronosGroup/VK-GL-CTS/wiki/Creating-a-OpenGL-and-OpenGL-ES-Submission-Package).

Submission Update Package
------------------------
Please see the [Submission Update Package page](https://github.com/KhronosGroup/VK-GL-CTS/wiki/Submission-Update-Package).

Passing Criteria
------------------------
Please see the [Conformance Submission Passing Criteria page](https://github.com/KhronosGroup/VK-GL-CTS/wiki/OpenGL-and-OpenGL-ES-Conformance-Submission-Passing-Criteria).

Troubleshooting
------------------------
### Crashes early on in the run
If using run-time entry point loading, it is possible that not all required
entry points are available. This will result in `NULL` pointer dereferencing.

### Build fails
First try re-running the build. If that does not help and you have used the
same build directory with different version of the CTS, remove the build
directory and run the CMake again.

Adding new tests
------------------------

See the [Contribution Guide](CONTRIBUTING.md)

Acknowledgments
------------------------
The Khronos Group gratefully acknowledges the support of drawElements Oy,
who donated a large number of GLSL tests and a new test framework and build system.

The Khronos Group also gratefully acknowledges the support of 3DLabs Inc.,
who gave permission to use the 3DLabs Graphics Test Framework (GTF).

The first internal version of the test was created by Bruno Schwander of
Hooked Wireless, under a development contract with the Khronos Group.

Symbio added tests specific to OpenGL and OpenGL ES 3.0.

drawElements added their donated language tests and build system.

The CTS results from these efforts, together with additional hard work by
volunteers from the OpenGL ES Working Group, the OpenGL ARB Working Group,
and their member companies, including:

- Sumit Agarwal, Imagination Technologies
- Eric Anholt, Intel
- Oleksiy Avramchenko, Sony
- Anthony Berent, ARM
- Joseph Blankenship, AMD
- Jeff Bolz, NVIDIA
- Pierre Boudier, AMD
- Benji Bowman, Imagination Technologies
- Pat Brown, NVIDIA
- David Cairns, Apple
- Mark Callow, ArtSpark
- Antoine Chauveau, NVIDIA
- Aske Simon Christensen, ARM
- Lin Chen, Qualcomm
- Mathieu Comeau, QNX
- Graham Connor, Imagination Technologies
- Slawomir Cygan, Intel
- Piotr Czubak, Intel
- Piers Daniell, NVIDIA
- Matthias Dejaegher, ZiiLabs
- Chris Dodd, NVIDIA
- David Donohoe, Movidius
- Alex Eddy, Apple
- Sean Ellis, ARM
- Bryan Eyler, NVIDIA
- Erik Faye-Lund, ARM
- Nicholas FitzRoy-Dale, Broadcom
- Michael Frydrych, NVIDIA
- Toshiki Fujimori, Takumi
- David Garcia, Qualcomm
- Frido Garritsen, Vivante
- Klaus Gerlicher, NVIDIA
- Slawomir Grajewski, Intel
- Jonas Gustavsson, Sony
- Nick Haemel, NVIDIA
- Matthew Harrison, Imagination Technologies
- Pyry Haulos, drawElements
- Jim Hauxwell, Broadcom
- Valtteri Heikkil, Symbio
- Tsachi Herman, AMD
- Mathias Heyer, NVIDIA
- Atsuko Hirose, Fujitsu
- Ari Hirvonen, NVIDIA
- Rune Holm, ARM
- Jaakko Huovinen, Nokia
- James Jones, Imagination Technologies
- Norbert Juffa, NVIDIA
- Jordan Justen, Intel
- Sandeep Kakarlapudi, ARM
- Anssi Kalliolahti, NVIDIA
- Philip Kamenarsky, NVIDIA
- Krzysztof Kaminski, Intel
- Daniel Kartch, NVIDIA
- Maxim Kazakov, DMP
- Jon Kennedy, 3DLabs
- John Kessenich
- Daniel Koch, NVIDIA
- Benjamin Kohler-Crowe, NVIDIA
- Georg Kolling, Imagination Technologies
- Misa Komuro, DMP
- Boguslaw Kowalik, Intel
- Aleksandra Krstic, Qualcomm
- Karol Kurach, NVIDIA
- VP Kutti
- Sami Kyostila, Google
- Teemu Laakso, Symbio
- Antoine Labour, Sony
- Alexandre Laurent, Imagination Technologies
- Jon Leech, Khronos
- Graeme Leese, Broadcom
- I-Gene Leong, Intel
- Radoslava Leseva, Imagination Technologies
- Jake Lever, NVIDIA
- Fred Liao, MediaTek
- Bill Licea-Kane, Qualcomm
- Benj Lipchak, Apple
- Wayne Lister, Imagination Technologies
- Isaac Liu, NVIDIA
- Weiwan Liu, NVIDIA
- Zhifang Long, Marvell
- Toni L&#246;nnberg, AMD
- Erik Lovlie
- Christer Lunde, ARM
- Zong-Hong Lyu, DMP
- Daniel Mahashin, NVIDIA
- Rob Matthesen, NVIDIA
- Tom McReynolds, NVIDIA (CTS TSG Chair, ES 1.1)
- Bruce Merry, ARM
- Assif Mirza, Imagination Technologies
- Zhenyao Mo, Google
- Kazuhiro Mochizuki, Fujitsu
- Affie Munshi, Apple
- Yeshwant Muthusamy, Samsung
- Mirela Nicolescu, Broadcom
- Glenn Nissen, Broadcom
- Michael O'Hara, AMD
- Eisaku Ohbuchi, DMP
- Tom Olson, ARM
- Tapani Palli, Intel
- Brian Paul, VMWare
- Remi Pedersen, ARM
- Adrian Peirson, ARM
- Russell Pflughaupt, NVIDIA
- Anuj Phogat, Intel
- Tero Pihlajakoski, Nokia
- Peter Pipkorn, NVIDIA
- Acorn Pooley, NVIDIA
- Guillaume Portier, ArtSpark
- Greg Prisament, Lychee Software
- Jonathan Putsman, Imagination Technologies
- Mike Quinlan, AMD
- Tarik Rahman, CodePlay
- Kalle Raita, drawElements
- Daniel Rakos, AMD
- Manjunatha Ramachandra
- John Recker, NVIDIA
- Maurice Ribble, Qualcomm (CTS TSG Chair, ES 2.0)
- James Riordon, Khronos
- Lane Roberts, Samsung
- Ian Romanick, Intel
- Greg Roth, NVIDIA
- Kenneth Russell, Google
- Matteo Salardi, Imagination Technologies
- Jeremy Sandmel, Apple
- Shusaku Sawato, DMP
- Chris Scholtes, Fujitsu
- Mathias Schott, NVIDIA
- Bruno Schwander, Hooked Wireless
- Graham Sellers, AMD
- Shereef Shehata, Texas Instruments
- Benjamin Shen, Vivante
- Robert Simpson, Qualcomm
- Stuart Smith, Imagination Technologies
- Janusz Sobczak, Mobica
- Jacob Strom, Ericsson
- Timo Suoranta, Broadcom
- Jan Svarovsky, Ideaworks3D
- Anthony Tai, Apple
- Payal Talati, Imagination Technologies
- Gregg Tavares, Google
- Ross Thompson, NVIDIA
- Jeremy Thorne, Broadcom
- Jani Tikkanen, Symbio
- Antti Tirronen, Qualcomm (CTS TSG Chair, ES 3.0/3.1)
- Robert Tray, NVIDIA
- Matt Turner, Intel
- Eben Upton, Broadcom
- Jani Vaarala, Nokia
- Dmitriy Vasilev, NVIDIA
- Chad Versace, Intel
- Holger Waechtler, Broadcom
- Joerg Wagner, ARM
- Jun Wang, Imagination Technologies
- Yuan Wang, Imagination Technologies
- Hans-Martin Will
- Ewa Wisniewska, Mobica
- Dominik Witczak, Mobica
- Oliver Wohlmuth, Fujitsu
- Yanjun Zhang, Vivante
- Lefan Zhong, Vivante
- Jill Zhou
- Marek Zylak, NVIDIA
- Iliyan Dinev, Imagination Technologies
- James Glanville, Imagination Technologies
- Mark Adams, NVIDIA
- Alexander Galazin, ARM
- Riccardo Capra, ARM
- Lars-Ivar Simonsen, ARM
- Fei Yang, ARM

Revision History
------------------------
- 0.0 - Tom Olson

  Initial version cloned from `ES2_Readme`, plus feedback from Mark Callow.

- 0.2 - Tom Olson

  Modified to incorporate feedback in bug 8534.

- 0.3 - Jon Leech

  Added details for OpenGL Conformance.

- 0.4 - Jon Leech 2012/10/31

  Add configuration & build section, and table of contents

- 0.5 - Jon Leech 2012/10/31

  Fix typos noted by Mark Callow in bug 8534.

- 0.6 - Jon Leech 2012/11/13

  Discuss automatic version selection and document support for OpenGL 3.3-4.3.

- 0.7 - Jon Leech 2012/11/14

  Minor cleanup for GL version numbers per Bug 8534 comment #41.

- 0.8 - Tom Olson 2013/1/25

  Updated GL status in preparation for ES 3.0 release, removed display
  parameters from product description, and removed mention of sample submission.

- 0.9 - Jon Leech 2013/07/17

  Restore GL-specific details in preparation for initial GL CTS release.

- 1.0 - Jon Leech 2013/07/17

  Change references to Visual Studio 11 to Visual Studio 2012 per bug 9862.
  Reset change tracking to reduce clutter.

- 1.1 - Kalle Raita 2013/10/30

  Updated documentation after the integration of the drawElements framework and
  language tests.

- 1.2 - Kalle Raita 2013/12/03

  Removed TODOs, added some notes on further development, and notes on file
  dependencies. Exact list of directory sub-trees that can be modified during porting.

- 1.3 - Tom Olson 2014/05/27

  Updates for ES CTS 3.1.1.0 . Added Passing Criteria, updated examples to
  include 3.1 versioning, and updated Acknowledgements.

- 1.4 - Alexander Galazin 2016/05/12

  Updates for ES CTS 3.2.1.0.

- 2.0 - Alexander Galazin 2016/09/23

  Moved the contents to README.md.
  Updated to reflect new CTS structure and build instructions.

- 2.1 - Alexander Galazin 2016/12/15

  Updates in preparation for the new release.
  Document restructuring, more detailed process of creating a submission package.
  Incorporated OpenGL/CTS issue 39 and 40 in the Passing Criteria.