1RADV 2==== 3 4RADV is a Vulkan driver for AMD GCN/RDNA GPUs. 5 6Introduction 7------------ 8 9RADV is a userspace driver that implements the Vulkan API on most modern AMD GPUs. 10 11Many Linux distributions include RADV in their default installation as part of their Mesa packages. 12It is also the Vulkan driver on the Steam Deck, the handheld console developed by Valve. 13 14Features 15~~~~~~~~ 16 17The easiest way to track the feature set of RADV (and other Vulkan drivers in Mesa) is to 18take a look at the 19`Mesa matrix <https://mesamatrix.net/#Vulkan>`__. 20 21Supported hardware 22~~~~~~~~~~~~~~~~~~ 23 24All GCN and RDNA GPUs that are supported by the Linux kernel (and capable of graphics) 25are also supported by RADV, starting from GCN 1. 26We are always working on supporting the very latest GPUs too. 27 28Vulkan API support: 29 30* GFX6-7 (GCN 1-2): Vulkan 1.3 31* GFX8 and newer (GCN 3-5 and RDNA): Vulkan 1.4 32 33`The exact list of Vulkan conformant products can be seen here. <https://www.khronos.org/conformance/adopters/conformant-products>`__ 34 35Each GPU chip can contain various hardware blocks (also known as IP blocks), and each of those are separately versioned. 36We usually refer to hardware generations by the main version number of the GFX (graphics) hardware block. 37 38Each hardware generation has different chips which usually only have minor differences between each other, 39such as number of compute units and different minor versions of some IP blocks. 40We refer to each chip by the code name of its first release, and we don't differentiate between 41refreshes of the same chip, because they are functionally exactly the same. 42 43For more information about which GPU chip name corresponds to which GPU product, 44`see the src/amd/common/amd_family.h file <https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/amd/common/amd_family.h>`__. 45 46Note that for GFX6-7 (GCN 1-2) GPUs, the ``amdgpu`` kernel driver is currently not the default in Linux 47(by default the old ``radeon`` KMD is used for these old GPUs, which is not supported by RADV), 48so users need to manually enable ``amdgpu`` by adding the following to the kernel command line: 49``radeon.si_support=0 radeon.cik_support=0 amdgpu.si_support=1 amdgpu.cik_support=1`` 50 51Basics 52~~~~~~ 53 54The RADV source code is located in ``src/amd/vulkan``. 55 56RADV is a userspace driver, compiled to a shared library file. 57On Linux, typically ``libvulkan_radeon.so`` (equivalent to a ``.dll`` on Windows). 58 59When you start a Vulkan application, the Vulkan loader (in userspace) 60will find a set of Vulkan drivers (also known as Vulkan implementations), 61all of which are technically shared libraries. 62If you are running on a system with a supported AMD GPU and have RADV installed, 63the loader will find ``libvulkan_radeon.so`` and load that. 64The Vulkan application can then choose which available Vulkan implementation to use. 65 66With RADV, when the application makes Vulkan API calls (aka. entry points), 67the ``vk*`` functions will end up calling ``radv_*`` functions. 68For example, ``vkCmdDrawMeshTasksEXT`` will actually call ``radv_CmdDrawMeshTasksEXT``. 69 70Responsibilities of RADV vs. the kernel driver 71~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 72 73Due to the complexity of how modern GPUs work, the graphics stack is split 74between kernel-mode drivers (KMD) and user-mode drivers (UMD). 75All Graphics APIs such as Vulkan, OpenGL, etc. are implemented in userspace. 76 77RADV is a UMD that currently works with the ``amdgpu`` KMD in the Linux kernel. 78Interacting with the KMD is done by RADV's winsys code. 79 80The KMD is responsible for: 81 82* Talking to the GPU through a PCIe port and power management (such as choosing voltage, frequency, sleep modes etc.) 83* Display functionality 84* Video memory management (and GTT) 85* Writing submitted commands to the GPU's ring buffers 86 87The UMD (in our case, RADV) is responsible for: 88 89* Recording commands in a binary format that the GPU can understand 90* Programming GPU registers for correct functionality 91* Compiling shaders to the GPU's native ISA (instruction set architecture) 92 and uploading them to memory accessible by the GPU 93* Submitting recorded commands to the kernel through a system call 94 95To communicate with ``amdgpu``, RADV relies on the DRM userspace API (uAPI) of the Linux kernel, 96which is a set of system calls. 97RADV depends on ``libdrm`` for some functionality and for others it uses the system calls directly. 98 99Command submission and PM4 packets 100~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 101 102Vulkan applications record a series of commands in command buffers and 103later submit these command buffers to one of the queues in the GPU. 104 105Command buffer recording in RADV is implemented by emitting packets in a buffer, 106which is called a command stream (CS). 107 108Command packets are more or less analogous to Vulkan API calls, 109which means that each Vulkan command (such as draw or dispatch) 110corresponds to one or more packets. 111However, depending on how closely the hardware follows Vulkan spec 112some commands will require a more complex implementation. 113For example, a simple ``vkCmdDraw`` call will result in just one packet, 114however emitting draw state (before the draw) may take dozens of packets. 115 116For the graphics queue (GFX) and async compute queue (ACE), the PM4 packet 117format is used; other queues such as SDMA and the various video queues 118have their own format. Commands typically vary between different GPU generations. 119 120When submitting to a queue, several CSs are submitted to the kernel at once. 121The kernel terminology calls these indirect buffers (IB) because the kernel 122typically uses the ``INDIRECT_BUFFER`` PM4 command to execute them. 123Modern AMD GPUs have several queues, which more or less map to the Vulkan 124queue types, though sometimes we need to submit to more than one HW queue 125at the same time. 126 127After command submission, the packets in the IBs will be executed 128by the command processor (CP) of the queue that the buffer was submitted to. 129The exact capabilities and supported commands of the CP depend on HW generation and queue type. 130 131Each CP has a collection of registers that control how the CP behaves. 132Registers in the CP are not to be confused with registers in shaders. 133For example, the addresses of shader programs, some details of shader execution, 134draw call information etc. have a corresponding register. 135These registers can be typically set by PM4 packets. 136 137Shader compilation 138~~~~~~~~~~~~~~~~~~ 139 140One of the main responsibilities of RADV is compiling shaders for Vulkan applications. 141 142RADV relies on the Mesa shader compiler stack. 143Here is a rough overview of how it works: 144 145* Vulkan applications pass shaders to RADV in the SPIR-V format 146* RADV calls ``spirv_to_nir`` which translates the shader into the NIR intermediate representation 147* We perform various optimizations and lowerings on the NIR shader 148* The shader is linked with the other shaders it is compiled with (if any) 149* Still using NIR, the shader is further lowered to be closer to the hardware 150* Finally, we pass the lowered NIR shader to ACO, our compiler backend, which compiles it into GPU specific ISA 151 152ACO is the default shader-compiler used in RADV. 153`Read its documentation here. <https://gitlab.freedesktop.org/mesa/mesa/-/blob/main/src/amd/compiler/README.md>`__ 154 155We still maintain an LLVM based compiler backend too, 156which is these days solely used for testing and hardware bringup. 157Users are recommended **NOT** to use the LLVM backend. 158 159Shader execution 160~~~~~~~~~~~~~~~~ 161 162Some commands (such as draws and dispatches) ask the CP to launch shaders. 163Shader launch is handled by the firmware, based on registers that control shader programs. 164Additionally, draw commands will also automatically use the appropriate fixed function 165units in the hardware. 166 167Shaders are executed by a so-called compute unit on the GPU, which is a SIMD machine. 168A shader invocation is a single SIMD lane (AMD also calls it thread, not to be confused by a CPU HW thread), 169and a subgroup is all 64 or 32 SIMD lanes together (also known as a wave). 170Each wave is a separate running instance of a shader program, 171but multiple waves can be grouped together into workgroups. 172 173Registers in shaders (not to be confused with registers in the CP): 174 175* VGPR - vector general purpose register: each SIMD lane has a different value for this register 176* SGPR - scalar general purpose register: same value within a wave 177 178For further reading, AMD has publised whitepapers and documentation for the GCN and RDNA GPU architectures. 179`These can be found on their GPUOpen site. <https://gpuopen.com/amd-gpu-architecture-programming-documentation/>`__ 180 181Debugging 182--------- 183 184For a list of environment variables to debug RADV, please see 185:ref:`radv env-vars` for a list. 186 187Instructions for debugging GPU hangs can be found :ref:`here <radv-debug-hang>`. 188 189Hardware Documentation 190---------------------- 191 192You can find a list of documentation for the various generations of 193AMD hardware on the `X.Org wiki 194<https://www.x.org/wiki/RadeonFeature/#documentation>`__. 195 196Additional community-written documentation is also available in Mesa: 197 198.. toctree:: 199 :glob: 200 201 amd/hw/* 202