1# Debugger Testing 2 3This document describes Amber's shader debugger testing framework, which allows developers to write tests for Vulkan drivers that expose shader debugging functionality via the [Debug Adapter Protocol](https://microsoft.github.io/debug-adapter-protocol/). 4 5--- 6**Caveats** 7 8* Vulkan shader debugging currently does not have a formalized specification. A shader debugger implementation is being developed in [SwiftShader](https://swiftshader.googlesource.com/SwiftShader/), which one day may become a reference implementation for a formal specifiction. 9* Currently SwiftShader is the only Vulkan driver to implement a [DAP based shader debugger](https://swiftshader.googlesource.com/SwiftShader/+/refs/heads/master/docs/VulkanShaderDebugging.md). This implementation is also work-in-progress, and may significantly change. 10* Currently the debugger connection uses localhost sockets to connect Amber to the driver. The `VK_DEBUGGER_PORT` environment variable must be set to an unused localhost port number before attempting to run any Amber scripts that use the `DEBUG` command. 11* [`OpenCL.DebugInfo.100`](https://www.khronos.org/registry/spir-v/specs/unified1/OpenCL.DebugInfo.100.mobile.html) is a SPIR-V extended instruction set that adds rich debug information to the shader program, allowing for high-level shader source debugging. `OpenCL.DebugInfo.100` is not currently generated by [DXC](https://github.com/microsoft/DirectXShaderCompiler) or [glslang](https://github.com/KhronosGroup/glslang), but initial work has started to try and add `OpenCL.DebugInfo.100` support to DXC. 12* `OpenCL.DebugInfo.100` insstructions are not currently preserved by many of the [SPIR-V Tools](https://github.com/KhronosGroup/SPIRV-Tools) optimization passes, so these optimizations should not currently be used. 13* `OpenCL.DebugInfo.100` may be incorrectly interpreted by the [SPIR-V Tools](https://github.com/KhronosGroup/SPIRV-Tools) validator, so Amber should currently be invoked with the `--disable-spirv-val` flag. 14 15--- 16 17## Usage 18 19### Building 20 21The debugger testing functionality is disabled by default, and has to be enabled with the `AMBER_ENABLE_VK_DEBUGGING` CMake flag. 22 23As SwiftShader is currently the only Vulkan driver that supports DAP-based shader debugging, you will also likely want to build SwiftShader as part of Amber, using the `AMBER_ENABLE_SWIFTSHADER` flag. 24 25Both of these can be set by running CMake with: 26 27``` 28cmake <path-to-amber-root> -DAMBER_ENABLE_SWIFTSHADER=1 -DAMBER_ENABLE_VK_DEBUGGING=1 29``` 30 31### AmberScript 32 33Debugger tests must be written in AmberScript. 34 35The `DEBUG` command extends the [`RUN` command](amber_script.md#Run-a-pipeline) to execute a draw or compute shader. 36`DEBUG` expects exactly the same arguments to follow as `RUN` ([see: 'Run a pipeline'](amber_script.md#Run-a-pipeline)). However, unlike `RUN`, `DEBUG` begins a block that must be terminated with an `END`. 37 38Within this `RUN` block, you may declare any number of `THREAD` command blocks that list a sequence of debugger commands to perform on a single compute, vertex or fragment shader invocation: 39 40* `THREAD GLOBAL_INVOCATION_ID` \<x\> \<y\> \<z\> 41 42 Declares a sequence of debugger commands to run when the compute shader invocation with the given global invocation identifier is executed. 43 44* `THREAD VERTEX_INDEX` \<index\> 45 46 Declares a sequence of debugger commands to run when the shader invocation for the vertex with the given index is executed. 47 48* `THREAD FRAGMENT_WINDOW_SPACE_POSITION` \<x\> \<y\> 49 50 Defines a sequence of debugger commands to run when the shader invocation for the fragment with the given window space position is executed. 51 52Each of the `THREAD` commands begins a block that must be terminated with an `END`. 53 54Within each `THREAD` command block, you may use any of the following commands to control execution of the shader, and to verify the debugger's behaviour: 55 56* `STEP_IN` 57 58 Single line step execution of the thread, stepping into any function calls. 59 60* `STEP_OVER` 61 62 Single line step execution of the thread, stepping over any function calls. 63 64* `STEP_OUT` 65 66 Run, without stopping to the end of the currently executing function. If the current function is not the top most of the call stack, then the debugger will pause at the next line after the function call. 67 68* `EXPECT LOCATION` \<file name\> \<line number\> [\<line source\>] 69 70 Verifies that the debugger is currently paused at the given line location. 71 The [\<line source\>] is an additional, optional check that verifies the line of the file reported by the debugger is as expected. 72 73* `EXPECT CALLSTACK` 74 75 Verifies that the debugger is currently paused with the given complete stack frame. 76 Each frame must be declared on a separate line, starting with the most nested call, and has the form: 77 78 \<function name\> [\<file name\> [\<line number\>]] 79 80 The [\<file name\>] and [\<line number\>] fields are additional, optionals checks that verify the file and line numbers reported by the debugger for the frame are as expected. 81 82 The list of stack frames is terminated with `END`. 83 84* `EXPECT LOCAL` \<name\> `EQ` [\<value\>] 85 86 Verifies that the local variable with the given name holds the expected value. \<name\> may contain `.` delimiters to index structure or array types. 87 88Every shader invocation covered by a `THREAD` block must be executed. 89It is a test failure if the debugger does not break at all the `THREAD` shader invocations declared in the `DEBUG` block. 90 91#### Example: 92 93Given the following HLSL vertex shader: 94 95```hlsl 96/* 1 */ // simple_vs.hlsl 97/* 2 */ struct VS_OUTPUT { 98/* 3 */ float4 pos : SV_POSITION; 99/* 4 */ float4 color : COLOR; 100/* 5 */ }; 101/* 6 */ 102/* 7 */ VS_OUTPUT main(float4 pos : POSITION, 103/* 8 */ float4 color : COLOR) { 104/* 9 */ VS_OUTPUT vout; 105/* 10 */ vout.pos = pos; 106/* 11 */ vout.color = color; 107/* 12 */ return vout; 108/* 13 */ } 109``` 110 111The following performs a basic debugger test for the 3rd vertex in the triangle list: 112 113```groovy 114DEBUG pipeline DRAW_ARRAY AS TRIANGLE_LIST START_IDX 0 COUNT 6 115 THREAD VERTEX_INDEX 2 116 // Debugger starts at line 9. Inspect input variables. 117 EXPECT LOCATION "simple_vs.hlsl" 9 " VS_OUTPUT vout;" 118 EXPECT LOCAL "pos.x" EQ -1.007874 119 EXPECT LOCAL "pos.y" EQ 1.000000 120 EXPECT LOCAL "pos.z" EQ 0.000000 121 EXPECT LOCAL "color.x" EQ 1.000000 122 EXPECT LOCAL "color.y" EQ 0.000000 123 EXPECT LOCAL "color.z" EQ 0.000000 124 // Step to line 10. 125 STEP_IN 126 EXPECT LOCATION "simple_vs.hlsl" 10 " vout.pos = pos;" 127 // Step to line 11 and read result of line 10. 128 STEP_IN 129 EXPECT LOCAL "vout.pos.x" EQ -1.007874 130 EXPECT LOCAL "vout.pos.y" EQ 1.000000 131 EXPECT LOCAL "vout.pos.z" EQ 0.000000 132 EXPECT LOCATION "simple_vs.hlsl" 11 " vout.color = color;" 133 // Step to line 12 and read result of line 11. 134 STEP_IN 135 EXPECT LOCAL "vout.color.x" EQ 1.000000 136 EXPECT LOCAL "vout.color.y" EQ 0.000000 137 EXPECT LOCAL "vout.color.z" EQ 0.000000 138 EXPECT LOCATION "simple_vs.hlsl" 12 " return vout;" 139 CONTINUE 140 END 141END 142``` 143 144## Implementation 145 146This section covers the design of how the debugger testing is implemented in Amber. 147 148### Parsing 149 150`Parser::ParseDebug()` starts by immediately calling `ParseRun()`, which parses the tokens that normally immediately follow a `RUN` command. On successful parse, `ParseRun()` appends a command into the `command_list_` vector. \ 151`ParseDebug()` then constructs a new `amber::debug::Script`, and parses the debugger command block. The `amber::debug::Script` is then assigned to the command at the back of the `command_list_` (added by `ParseRun()`). 152 153### `amber::debug::Script` 154 155`amber::debug::Script` implements the `amber::debug::Events` interface, and records the calls made on it. 156These calls can be replayed to another `amber::debug::Events`, using the `amber::debug::Script::Run(Events*)` method. 157 158### `amber::debug::Events` 159 160The `amber::debug::Events` represents the `THREAD` commands in the Amber script, providing methods that set breakpoints for particular shader invocations. 161 162`amber::debug::Events` is the interface implemented by `amber::debug::Script` for recording the parsed debugger script, and extended by the `amber::Engine::Debugger` interface, used to actually drive an engine debugger. 163 164The `amber::debug::Events` interface has a number of `BreakOn`XXX`()` methods that have parameters for the shader invocation of interest (global invocation ID, vertex index, etc) and a `OnThread` function callback parameter. 165 166The `OnThread` callback has the signature `void(amber::debug::Thread*)` which is called to control the debugger thread of execution and perform test verifications. 167 168### `amber::debug::Thread` 169 170`amber::debug::Thread` is the interface used to control a debugger thread of execution, and represents the script commands within the `THREAD` script blocks. 171 172The following implementations of the `amber::debug::Thread` interface are returned by the `amber::debug::Script::BreakOn`XXX`()` methods: 173* `amber::debug::Script` implements this interface to record the `THREAD` commands in the Amber script, which will be replayed when calling `amber::debug::Script::Run(amber::debug::Events*)`. 174* `amber::Engine::Debugger` implementations will implement the `amber::debug::Thread` interface to actually control the debugger thread of execution and perform test verifications. 175 176### `amber::Engine::Debugger` 177 178The `amber::Engine::Debugger` interface extends the `amber::debug::Events` interface with a single `Flush()` method that ensures that all the previous event have been executed. `Flush()` returns a `amber::Result`, which holds the results of all the `amber::debug::Thread::Expect`XXX`()` calls. 179 180The `amber::Engine::Debugger` interface can be obtained from the `amber::Engine` using the `amber::Engine::GetDebugger()` method. 181 182### Debugger Execution 183 184`amber::Executor::Execute()` drives the `amber::Engine::Debugger`: 185 186* The before executing the first `amber::Command` that holds a `amber::debug::Script`, a `amber::Engine::Debugger` is obtained from the `amber::Engine`, and `amber::Engine::Debugger::Connect()` is called to create a connection to the Vulkan shader debugger. 187* If the `amber::Command` holds a `amber::debug::Script`, then this script is executed on the `amber::Engine::Debugger` using the `amber::debug::Script::Run(amber::debug::Events *)` method, before the Vulkan command is executed. 188* The command is then executed with `amber::Executor::ExecuteCommand()` 189* Once the command has completed, all debugger threads are synchronized and debugger test results are collected with a call to `amber::Engine::Debugger::Flush()`. 190* This process is repeated for all commands in the script. 191