# Unofficial GCN/RDNA ISA reference errata

## `v_sad_u32`

The Vega ISA reference writes its behaviour as:

```
D.u = abs(S0.i - S1.i) + S2.u.
```

This is incorrect. The actual behaviour is what is written in the GCN3 reference
guide:

```
ABS_DIFF (A,B) = (A>B) ? (A-B) : (B-A)
D.u = ABS_DIFF (S0.u,S1.u) + S2.u
```

The instruction doesn't subtract the S0 and S1 and use the absolute value (the
_signed_ distance), it uses the _unsigned_ distance between the operands. So
`v_sad_u32(-5, 0, 0)` would return `4294967291` (`-5` interpreted as unsigned),
not `5`.

## `s_bfe_*`

Both the RDNA, Vega and GCN3 ISA references write that these instructions don't write
SCC. They do.

## `v_bcnt_u32_b32`

The Vega ISA reference writes its behaviour as:

```
D.u = 0;
for i in 0 ... 31 do
D.u += (S0.u[i] == 1 ? 1 : 0);
endfor.
```

This is incorrect. The actual behaviour (and number of operands) is what
is written in the GCN3 reference guide:

```
D.u = CountOneBits(S0.u) + S1.u.
```

## `v_alignbyte_b32`

All versions of the ISA document are vague about it, but after some trial and
error we discovered that only 2 bits of the 3rd operand are used.
Therefore, this instruction can't shift more than 24 bits.

The correct description of `v_alignbyte_b32` is probably the following:

```
D.u = ({S0, S1} >> (8 * S2.u[1:0])) & 0xffffffff
```

## SMEM stores

The Vega ISA references doesn't say this (or doesn't make it clear), but
the offset for SMEM stores must be in m0 if IMM == 0.

The RDNA ISA doesn't mention SMEM stores at all, but they seem to be supported
by the chip and are present in LLVM. AMD devs however highly recommend avoiding
these instructions.

## SMEM atomics

RDNA ISA: same as the SMEM stores, the ISA pretends they don't exist, but they
are there in LLVM.

## VMEM stores

All reference guides say (under "Vector Memory Instruction Data Dependencies"):

> When a VM instruction is issued, the address is immediately read out of VGPRs
> and sent to the texture cache. Any texture or buffer resources and samplers
> are also sent immediately. However, write-data is not immediately sent to the
> texture cache.

Reading that, one might think that waitcnts need to be added when writing to
the registers used for a VMEM store's data. Experimentation has shown that this
does not seem to be the case on GFX8 and GFX9 (GFX6 and GFX7 are untested). It
also seems unlikely, since NOPs are apparently needed in a subset of these
situations.

## MIMG opcodes on GFX8/GCN3

The `image_atomic_{swap,cmpswap,add,sub}` opcodes in the GCN3 ISA reference
guide are incorrect. The Vega ISA reference guide has the correct ones.

## VINTRP encoding

VEGA ISA doc says the encoding should be `110010` but `110101` works.

## VOP1 instructions encoded as VOP3

RDNA ISA doc says that `0x140` should be added to the opcode, but that doesn't
work. What works is adding `0x180`, which LLVM also does.

## FLAT, Scratch, Global instructions

The NV bit was removed in RDNA, but some parts of the doc still mention it.

RDNA ISA doc 13.8.1 says that SADDR should be set to 0x7f when ADDR is used, but
9.3.1 says it should be set to NULL. We assume 9.3.1 is correct and set it to
SGPR_NULL.

## Legacy instructions

Some instructions have a `_LEGACY` variant which implements "DX9 rules", in which
the zero "wins" in multiplications, ie. `0.0*x` is always `0.0`. The VEGA ISA
mentions `V_MAC_LEGACY_F32` but this instruction is not really there on VEGA.

## LDS size and allocation granule

GFX7-8 ISA manuals are mistaken about the available LDS size.

* GFX7+ workgroups can use 64KB LDS.
  There is 64KB LDS per CU.
* GFX6 workgroups can use 32KB LDS.
  There is 64KB LDS per CU, but a single workgroup can only use half of it.

 Regarding the LDS allocation granule, Mesa has the correct details and
 the ISA manuals are mistaken.

## `m0` with LDS instructions on Vega and newer

The Vega ISA doc (both the old one and the "7nm" one) claims that LDS instructions
use the `m0` register for address clamping like older GPUs, but this is not the case.

In reality, only the `_addtid` variants of LDS instructions use `m0` on Vega and
newer GPUs, so the relevant section of the RDNA ISA doc seems to apply.
LLVM also doesn't emit any initialization of `m0` for LDS instructions, and this
was also confirmed by AMD devs.

## RDNA L0, L1 cache and DLC, GLC bits

The old L1 cache was renamed to L0, and a new L1 cache was added to RDNA. The
L1 cache is 1 cache per shader array. Some instruction encodings have DLC and
GLC bits that interact with the cache.

* DLC ("device level coherent") bit: controls the L1 cache
* GLC ("globally coherent") bit: controls the L0 cache

The recommendation from AMD devs is to always set these two bits at the same time,
as it doesn't make too much sense to set them independently, aside from some
circumstances (eg. we needn't set DLC when only one shader array is used).

Stores and atomics always bypass the L1 cache, so they don't support the DLC bit,
and it shouldn't be set in these cases. Setting the DLC for these cases can result
in graphical glitches or hangs.

## RDNA `s_dcache_wb`

The `s_dcache_wb` is not mentioned in the RDNA ISA doc, but it is needed in order
to achieve correct behavior in some SSBO CTS tests.

## RDNA subvector mode

The documentation of `s_subvector_loop_begin` and `s_subvector_mode_end` is not clear
on what sort of addressing should be used, but it says that it
"is equivalent to an `S_CBRANCH` with extra math", so the subvector loop handling
in ACO is done according to the `s_cbranch` doc.

## RDNA early rasterization

The ISA documentation says about `s_endpgm`:

> The hardware implicitly executes S_WAITCNT 0 and S_WAITCNT_VSCNT 0
> before executing this instruction.

What the doc doesn't say is that in case of NGG (and legacy VS) when there
are no param exports, the driver sets `NO_PC_EXPORT=1` for optimal performance,
and when this is set, the hardware will start clipping and rasterization
as soon as it encounters a position export with `DONE=1`, without waiting
for the NGG (or VS) to finish.

It can even launch PS waves before NGG (or VS) ends.

When this happens, any store performed by a VS is not guaranteed
to be complete when PS tries to load it, so we need to manually
make sure to insert wait instructions before the position exports.

## A16 and G16

On GFX9, the A16 field enables both 16 bit addresses and derivatives.
Since GFX10+ these are fully independent of each other, A16 controls 16 bit addresses
and G16 opcodes 16 bit derivatives. A16 without G16 uses 32 bit derivatives.

## POPS collision wave ID argument (GFX9-10.3)

The 2020 RDNA and RDNA 2 ISA references contain incorrect offsets and widths of
the fields of the "POPS collision wave ID" SGPR argument.

According to the code generated for Rasterizer Ordered View usage in Direct3D,
the correct layout is:

* [31]: Whether overlap has occurred.
* [29:28] (GFX10+) / [28] (GFX9): ID of the packer the wave should be associated
  with.
* [25:16]: Newest overlapped wave ID.
* [9:0]: Current wave ID.

## RDNA3 `v_pk_fmac_f16_dpp`

"Table 30. Which instructions support DPP" in the RDNA3 ISA documentation has no exception for
VOP2 `v_pk_fmac_f16`. But like all other packed math opcodes, DPP does not function in practice.
RDNA1 and RDNA2 support `v_pk_fmac_f16_dpp`.

## ds_swizzle_b32 rotate/fft modes

These are first mentioned in the GFX9 (Vega) ISA doc, information from the LLVM bug tracker
and testing show they were already present on GFX8.

# Hardware Bugs

## SMEM corrupts VCCZ on SI/CI

[See this LLVM source.](https://github.com/llvm/llvm-project/blob/acb089e12ae48b82c0b05c42326196a030df9b82/llvm/lib/Target/AMDGPU/SIInsertWaits.cpp#L580-L616)

After issuing a SMEM instructions, we need to wait for the SMEM instructions to
finish and then write to vcc (for example, `s_mov_b64 vcc, vcc`) to correct vccz

Currently, we don't do this.

## SGPR offset on MUBUF prevents addr clamping on SI/CI

[See this LLVM source.](https://github.com/llvm/llvm-project/blob/main/llvm/lib/Target/AMDGPU/Utils/AMDGPUBaseInfo.cpp#L1917-L1922)

This leads to wrong bounds checking, using a VGPR offset fixes it.

## unused VMEM/DS destination lanes can't be used without waiting

On GFX11, we can't safely read/write unused lanes of VMEM/DS destination
VGPRs without waiting for the load to finish.

## GCN / GFX6 hazards

### VINTRP followed by a read with `v_readfirstlane` or `v_readlane`

It's required to insert 1 wait state if the dst VGPR of any  `v_interp_*` is
followed by a read with `v_readfirstlane` or `v_readlane` to fix GPU hangs on GFX6.
Note that `v_writelane_*` is apparently not affected. This hazard isn't
documented anywhere but AMD confirmed it.

## RDNA / GFX10 hazards

### SMEM store followed by a load with the same address

We found that an `s_buffer_load` will produce incorrect results if it is preceded
by an `s_buffer_store` with the same address. Inserting an `s_nop` between them
does not mitigate the issue, so an `s_waitcnt lgkmcnt(0)` must be inserted.
This is not mentioned by LLVM among the other GFX10 bugs, but LLVM doesn't use
SMEM stores, so it's not surprising that they didn't notice it.

### VMEMtoScalarWriteHazard

Triggered by:
VMEM/FLAT/GLOBAL/SCRATCH/DS instruction reads an SGPR (or EXEC, or M0).
Then, a SALU/SMEM instruction writes the same SGPR.

Mitigated by:
A VALU instruction or an `s_waitcnt` between the two instructions.

### SMEMtoVectorWriteHazard

Triggered by:
An SMEM instruction reads an SGPR. Then, a VALU instruction writes that same SGPR.

Mitigated by:
Any non-SOPP SALU instruction (except `s_setvskip`, `s_version`, and any non-lgkmcnt `s_waitcnt`).

### Offset3fBug

Any branch that is located at offset 0x3f will be buggy. Just insert some NOPs to make sure no branch
is located at this offset.

### InstFwdPrefetchBug

According to LLVM, the `s_inst_prefetch` instruction can cause a hang on GFX10.
Seems to be resolved on GFX10.3+. There are no further details.

### LdsMisalignedBug

When there is a misaligned multi-dword FLAT load/store instruction in WGP mode,
it needs to be split into multiple single-dword FLAT instructions.

ACO doesn't use FLAT load/store on GFX10, so is unaffected.

### FlatSegmentOffsetBug

The 12-bit immediate OFFSET field of FLAT instructions must always be 0.
GLOBAL and SCRATCH are unaffected.

ACO doesn't use FLAT load/store on GFX10, so is unaffected.

### VcmpxPermlaneHazard

Triggered by:
Any permlane instruction that follows any VOPC instruction which writes exec.

Mitigated by: any VALU instruction except `v_nop`.

### VcmpxExecWARHazard

Triggered by:
Any non-VALU instruction reads the EXEC mask. Then, any VALU instruction writes the EXEC mask.

Mitigated by:
A VALU instruction that writes an SGPR (or has a valid SDST operand), or `s_waitcnt_depctr 0xfffe`.
Note: `s_waitcnt_depctr` is an internal instruction, so there is no further information
about what it does or what its operand means.

### LdsBranchVmemWARHazard

Triggered by:
VMEM/GLOBAL/SCRATCH instruction, then a branch, then a DS instruction,
or vice versa: DS instruction, then a branch, then a VMEM/GLOBAL/SCRATCH instruction.

Mitigated by:
Only `s_waitcnt_vscnt null, 0`. Needed even if the first instruction is a load.

### NSAClauseBug

"MIMG-NSA in a hard clause has unpredictable results on GFX10.1"

### NSAMaxSize5

NSA MIMG instructions should be limited to 3 dwords before GFX10.3 to avoid
stability issues: https://reviews.llvm.org/D103348

## RDNA3 / GFX11 hazards

### VcmpxPermlaneHazard

Same as GFX10.

### LdsDirectVALUHazard

Triggered by:
LDSDIR instruction writing a VGPR soon after it's used by a VALU instruction.

Mitigated by:
A vdst wait, preferably using the LDSDIR's field.

### LdsDirectVMEMHazard

Triggered by:
LDSDIR instruction writing a VGPR after it's used by a VMEM/DS instruction.

Mitigated by:
Waiting for the VMEM/DS instruction to finish, a VALU or export instruction, or
`s_waitcnt_depctr 0xffe3`.

### VALUTransUseHazard

Triggered by:
A VALU instruction reading a VGPR written by a transcendental VALU instruction without 6+ VALU or 2+
transcendental instructions in-between.

Mitigated by:
A va_vdst=0 wait: `s_waitcnt_deptr 0x0fff`

### VALUPartialForwardingHazard

Triggered by:
A VALU instruction reading two VGPRs: one written before an exec write by SALU and one after. To
trigger, there must be less than 3 VALU between the first and second VGPR writes and less than 5
VALU between the second VGPR write and the current instruction.

Mitigated by:
A va_vdst=0 wait: `s_waitcnt_deptr 0x0fff`

### VALUMaskWriteHazard

Triggered by:
SALU writing then SALU or VALU reading a SGPR that was previously used as a lane mask for a VALU.

Mitigated by:
A VALU instruction reading a non-exec SGPR before the SALU write, or a sa_sdst=0 wait after the
SALU write: `s_waitcnt_depctr 0xfffe`

# Welcome to ACO

ACO (short for *AMD compiler*) is a back-end compiler for AMD GCN / RDNA GPUs, based on the NIR compiler infrastructure.
Simply put, ACO translates shader programs from the NIR intermediate representation into a GCN / RDNA binary which the GPU can execute.

## Motivation

Why did we choose to develop a new compiler backend?

1. We'd like to give gamers a fluid, stutter-free experience, so we prioritize compilation speed.
2. Good divergence analysis allows us to better optimize runtime performance.
3. Issues can be fixed within mesa releases, independently of the schedule of other projects.

## Control flow

Modern GPUs are SIMD machines that execute the shader in parallel.
In case of GCN / RDNA the parallelism is achieved by executing the shader on several waves, and each wave has several lanes (32 or 64).
When every lane executes exactly the same instructions, and takes the same path, it's uniform control flow;
otherwise when some lanes take one path while other lanes take a different path, it's divergent.

Each hardware lane corresponds to a shader invocation from a software perspective.

The hardware doesn't directly support divergence,
so in case of divergent control flow, the GPU must execute both code paths, each with some lanes disabled.
This is why divergence is a performance concern in shader programming.

ACO deals with divergent control flow by maintaining two control flow graphs (CFG):

* logical CFG - directly translated from NIR and shows the intended control flow of the program.
* linear CFG - created according to Whole-Function Vectorization by Ralf Karrenberg and Sebastian Hack.
  The linear CFG represents how the program is physically executed on GPU and may contain additional blocks for control flow handling and to avoid critical edges.
  Note that all nodes of the logical CFG also participate in the linear CFG, but not vice versa.

## Compilation phases

#### Instruction Selection

The instruction selection is based around the divergence analysis and works in 3 passes on the NIR shader.

1. The divergence analysis pass calculates for each SSA definition if its value is guaranteed to be uniform across all threads of the wave (subgroup).
2. We determine the register class for each SSA definition.
3. Actual instruction selection. The advanced divergence analysis allows for better usage of the scalar unit, scalar memory loads and the scalar register file.

We have two types of instructions:

* Hardware instructions as specified by the GCN / RDNA instruction set architecture manuals.
* Pseudo instructions which are helpers that encapsulate more complex functionality.
  They eventually get lowered to real hardware instructions.

Each instruction can have operands (temporaries that it reads), and definitions (temporaries that it writes).
Temporaries can be fixed to a specific register, or just specify a register class (either a single register, or a vector of several registers).

#### Repair SSA

This repairs SSA in the case of mismatches between the logical and linear CFG, where the definition of a linear temporary logically dominate its users but not linearly. This is followed by lower_phis to lower the phis created by this pass.

Instruction selection might create mismatches between the logical CFG (the input NIR's CFG) and the linear CFG in the following situations:
- We add a break at the end of a loop in case it has no active invocations (an empty exec can prevent any logical breaks from being taken). This creates a linear edge but no logical edge, and SGPR uses outside the loop can require a phi.
- We add an empty exec skip over a block. This is a branch which skips most contents of a sequence of instructions if exec is empty. To avoid critical edges, the inside of the construct logically dominates the merge but not linearly.
- An SGPR is defined in one side of a divergent IF but it used in or after the merge block. If the other side of the IF ends in a branch, a phi is not necessary according to the logical CFG, but it is for the linear CFG. However, `sanitize_cf_list()` should already resolve this before translation from NIR for additional reasons.

#### Lower Phis

After instructions selection, some phi instructions need further lowering. This includes booleans which are represented as scalar values. Because the scalar ALU doesn't respect the execution mask, divergent boolean phis need to be lowered to SALU shuffle code. This pass also inserts the necessary code in order to fix phis with subdword access and repairs phis in case of mismatches between logical and linear CFG.

#### Lower Subdword

For GFX6 and GFX7, this pass already lowers subdword pseudo instructions.

#### Value Numbering

The value numbering pass is necessary for two reasons: the lack of descriptor load representation in NIR,
and every NIR instruction that gets emitted as multiple ACO instructions also has potential for CSE.
This pass does dominator-tree value numbering.

#### Optimization

In this phase, simpler instructions are combined into more complex instructions (like the different versions of multiply-add as well as neg, abs, clamp, and output modifiers) and constants are inlined, moves are eliminated, etc.
Exactly which optimizations are performed depends on the hardware for which the shader is being compiled.
After this, repair_ssa needs to be run again in case it moves a SGPR use to a different block.

#### Setup of reduction temporaries

This pass is responsible for making sure that register allocation is correct for reductions, by adding pseudo instructions that utilize linear VGPRs.
When a temporary has a linear VGPR register class, this means that the variable is considered *live* in the linear control flow graph.

#### Insert exec mask

In the GCN/RDNA architecture, there is a special register called `exec` which is used for manually controlling which VALU threads (aka. *lanes*) are active. The value of `exec` has to change in divergent branches, loops, etc. and it needs to be restored after the branch or loop is complete. This pass ensures that the correct lanes are active in every branch.

#### Live-Variable Analysis

A live-variable analysis is used to calculate the register need of the shader.
This information is used for spilling and scheduling before register allocation.

#### Spilling

First, we lower the shader program to CSSA form.
Then, if the register demand exceeds the global limit, this pass lowers register usage by temporarily storing excess scalar values in free vector registers, or excess vector values in scratch memory, and reloading them when needed. It is based on the paper "Register Spilling and Live-Range Splitting for SSA-Form Programs".

#### Instruction Scheduling

Scheduling is another NP-complete problem where basically all known heuristics suffer from unpredictable change in register pressure. For that reason, the implemented scheduler does not completely re-schedule all instructions, but only aims to move up memory loads as far as possible without exceeding the maximum register limit for the pre-calculated wave count. The reason this works is that ILP is very limited on GCN. This approach looks promising so far.

#### Register Allocation

The register allocator works on SSA (as opposed to LLVM's which works on virtual registers). The SSA properties guarantee that there are always as many registers available as needed. The problem is that some instructions require a vector of neighboring registers to be available, but the free regs might be scattered. In this case, the register allocator inserts shuffle code (moving some temporaries to other registers) to make space for the variable. The assumption is that it is (almost) always better to have a few more moves than to sacrifice a wave. The RA does SSA-reconstruction on the fly, which makes its runtime linear.

#### Optimization (post-RA)

Optimizations which depend on register assignment (like branching on VCCZ) are performed.

#### SSA Elimination

The next step is a pass out of SSA by inserting parallelcopies at the end of blocks to match the phi nodes' semantics.

#### Jump Threading

This pass aims to eliminate empty or unnecessary basic blocks. As this introduces critical edges, it can only be performed after SSA elimination.

#### Lower to HW instructions

Most pseudo instructions are lowered to actual machine instructions.
These are mostly parallel copy instructions created by instruction selection or register allocation and spill/reload code.

#### VOPD Scheduling

This pass makes use of the VOPD instruction encoding on GFX11+. When using wave32 mode, this pass works on a partial dependency graph in order to combine two VALU instructions each into one VOPD instruction.

#### ILP Scheduling

This second scheduler works on registers rather than SSA-values to determine dependencies. It implements a forward list scheduling algorithm using a partial dependency graph of few instructions at a time and aims to create larger memory clauses and improve ILP.

#### Insert wait states

GCN requires some wait states to be manually inserted in order to ensure correct behavior on memory instructions and some register dependencies.
This means that we need to insert `s_waitcnt` instructions (and its variants) so that the shader program waits until the eg. a memory operation is complete.

#### Resolve hazards and insert NOPs

Some instructions require wait states or other instructions to resolve hazards which are not handled by the hardware.
This pass makes sure that no known hazards occur.

#### Insert delay_alu and form clauses

These passes introduce optional instructions which provide performance hints to the hardware. `s_delay_alu` is available on GFX11+ and describes ALU dependencies in order to allow the hardware to execute instructions from a different wave in the meantime. `s_clause` is avilable on GFX10+ with the purpose to complete an entire set of memory instructions before switching to a different wave.

#### Emit program - Assembler

The assembler emits the actual binary that will be sent to the hardware for execution. ACO's assembler is straight-forward because all instructions have their format, opcode, registers and potential fields already available, so it only needs to cater to the some differences between each hardware generation.

## Supported shader stages

Hardware stages (as executed on the chip) don't exactly match software stages (as defined in OpenGL / Vulkan).
Which software stage gets executed on which hardware stage depends on what kind of software stages are present in the current pipeline.

An important difference is that VS is always the first stage to run in SW models,
whereas HW VS refers to the last HW stage before fragment shading in GCN/RDNA terminology.
That's why, among other things, the HW VS is no longer used to execute the SW VS when tessellation or geometry shading are used.

#### Glossary of software stages

* VS = Vertex Shader
* TCS = Tessellation Control Shader, equivalent to D3D HS = Hull Shader
* TES = Tessellation Evaluation Shader, equivalent to D3D DS = Domain Shader
* GS = Geometry Shader
* FS = Fragment Shader, equivalent to D3D PS = Pixel Shader
* CS = Compute Shader
* TS = Task Shader
* MS = Mesh Shader

#### Glossary of hardware stages

* LS = Local Shader (merged into HS on GFX9+), only runs SW VS when tessellation is used
* HS = Hull Shader, the HW equivalent of a Tessellation Control Shader, runs before the fixed function hardware performs tessellation
* ES = Export Shader (merged into GS on GFX9+), if there is a GS in the SW pipeline, the preceding stage (ie. SW VS or SW TES) always has to run on this HW stage
* GS = Geometry Shader, also known as legacy GS
* VS = Vertex Shader, **not equivalent to SW VS**: when there is a GS in the SW pipeline this stage runs a "GS copy" shader, otherwise it always runs the SW stage before FS
* NGG = Next Generation Geometry, a new hardware stage that replaces legacy HW GS and HW VS on RDNA GPUs
* PS = Pixel Shader, the HW equivalent to SW FS
* CS = Compute Shader

##### Notes about HW VS and the "GS copy" shader

HW PS reads its inputs from a special ring buffer called Parameter Cache (PC) that only HW VS can write to, using export instructions.
However, legacy GS store their output in VRAM (before GFX10/NGG).
So in order for HW PS to be able to read the GS outputs, we must run something on the VS stage which reads the GS outputs
from VRAM and exports them to the PC. This is what we call a "GS copy" shader.
From a HW perspective the "GS copy" shader is in fact VS (it runs on the HW VS stage),
but from a SW perspective it's not part of the traditional pipeline,
it's just some "glue code" that we need for outputs to play nicely.

On GFX10/NGG this limitation no longer exists, because NGG can export directly to the PC.

##### Notes about the attribute ring

Starting with GFX11, the parameter cache is replaced by the attribute ring,
which is a discardable ring buffer located in VRAM.
The outputs of the last pre-rasterization stage (VS, TES, GS or MS) are stored here.

The attribute ring is designed to utilize the Infinity Cache.
Store instructions are arranged so that each instruction writes a full cache line,
so the GPU will never actually have to write any of that to VRAM.

##### Notes about merged shaders

The merged stages on GFX9 (and GFX10/legacy) are: LSHS and ESGS. On GFX10/NGG the ESGS is merged with HW VS into NGG.

This might be confusing due to a mismatch between the number of invocations of these shaders.
For example, ES is per-vertex, but GS is per-primitive.
This is why merged shaders get an argument called `merged_wave_info` which tells how many invocations each part needs,
and there is some code at the beginning of each part to ensure the correct number of invocations by disabling some threads.
So, think about these as two independent shader programs slapped together.

### Which software stage runs on which hardware stage?

#### Graphics Pipeline

##### GFX6-8:

* Each SW stage has its own HW stage
* LS and HS share the same LDS space, so LS can store its output to LDS, where HS can read it
* HS, ES, GS outputs are stored in VRAM, next stage reads these from VRAM
* GS outputs got to VRAM, so they have to be copied by a GS copy shader running on the HW VS stage

| GFX6-8 HW stages:       | LS  | HS  | ES  | GS  | VS     | PS | ACO terminology |
| -----------------------:|:----|:----|:----|:----|:-------|:---|:----------------|
| SW stages: only VS+PS:  |     |     |     |     | VS     | FS | `vertex_vs`, `fragment_fs` |
|            with tess:   | VS  | TCS |     |     | TES    | FS | `vertex_ls`, `tess_control_hs`, `tess_eval_vs`, `fragment_fs` |
|            with GS:     |     |     | VS  | GS  | GS copy| FS | `vertex_es`, `geometry_gs`, `gs_copy_vs`, `fragment_fs` |
|            with both:   | VS  | TCS | TES | GS  | GS copy| FS | `vertex_ls`, `tess_control_hs`, `tess_eval_es`, `geometry_gs`, `gs_copy_vs`, `fragment_fs` |

##### GFX9+ (including GFX10/legacy):

* HW LS and HS stages are merged, and the merged shader still uses LDS in the same way as before
* HW ES and GS stages are merged, so ES outputs can go to LDS instead of VRAM
* LSHS outputs and ESGS outputs are still stored in VRAM, so a GS copy shader is still necessary

| GFX9+ HW stages:        | LSHS      | ESGS      | VS     | PS | ACO terminology |
| -----------------------:|:----------|:----------|:-------|:---|:----------------|
| SW stages: only VS+PS:  |           |           | VS     | FS | `vertex_vs`, `fragment_fs` |
|            with tess:   | VS + TCS  |           | TES    | FS | `vertex_tess_control_hs`, `tess_eval_vs`, `fragment_fs` |
|            with GS:     |           | VS + GS   | GS copy| FS | `vertex_geometry_gs`, `gs_copy_vs`, `fragment_fs` |
|            with both:   | VS + TCS  | TES + GS  | GS copy| FS | `vertex_tess_control_hs`, `tess_eval_geometry_gs`, `gs_copy_vs`, `fragment_fs` |

##### NGG (GFX10+ only):

 * HW GS and VS stages are now merged, and NGG can export directly to PC
 * GS copy shaders are no longer needed
 * On GFX10.3+, per-primitive attributes (parameters) are also supported
 * On GFX11+, parameter exports are replaced by attribute ring stores

| GFX10/NGG HW stages:    | LSHS      | NGG                | PS | ACO terminology |
| -----------------------:|:----------|:-------------------|:---|:----------------|
| SW stages: only VS+PS:  |           | VS                 | FS | `vertex_ngg`, `fragment_fs` |
|            with tess:   | VS + TCS  | TES                | FS | `vertex_tess_control_hs`, `tess_eval_ngg`, `fragment_fs` |
|            with GS:     |           | VS + GS            | FS | `vertex_geometry_ngg`, `fragment_fs` |
|            with both:   | VS + TCS  | TES + GS           | FS | `vertex_tess_control_hs`, `tess_eval_geometry_ngg`, `fragment_fs` |

#### Mesh Shading Graphics Pipeline

GFX10.3+:

* TS will run as a CS and stores its output payload to VRAM
* MS runs on NGG, loads its inputs from VRAM and stores outputs to LDS, then PC (or attribute ring)
* Pixel Shaders work the same way as before

| GFX10.3+ HW stages      | CS    | NGG   | PS | ACO terminology |
| -----------------------:|:------|:------|:---|:----------------|
| SW stages: only MS+PS:  |       | MS    | FS | `mesh_ngg`, `fragment_fs` |
|            with task:   | TS    | MS    | FS | `task_cs`, `mesh_ngg`, `fragment_fs` |

#### Compute pipeline

GFX6-10:

* Note that the SW CS always runs on the HW CS stage on all HW generations.

| GFX6-10 HW stage        | CS   | ACO terminology |
| -----------------------:|:-----|:----------------|
| SW stage                | CS   | `compute_cs`    |


## How to debug

Handy `RADV_DEBUG` options that help with ACO debugging:

* `nocache` - you always want to use this when debugging, otherwise you risk using a broken shader from the cache.
* `shaders` - makes ACO print the IR after register allocation, as well as the disassembled shader binary.
* `metashaders` - does the same thing as `shaders` but for built-in RADV shaders.
* `preoptir` - makes ACO print the final NIR shader before instruction selection, as well as the ACO IR after instruction selection.
* `nongg` - disables NGG support

We also have `ACO_DEBUG` options:

* `validateir` - Validate the ACO IR between compilation stages. By default, enabled in debug builds and disabled in release builds.
* `validatera` - Perform a RA (register allocation) validation.
* `force-waitcnt` - Forces ACO to emit a wait state after each instruction when there is something to wait for. Harms performance.
* `novn` - Disables the ACO value numbering stage.
* `noopt` - Disables the ACO optimizer.
* `nosched` - Disables the ACO pre-RA and post-RA scheduler.
* `nosched-ilp` - Disables the ACO post-RA ILP scheduler.

Note that you need to **combine these options into a comma-separated list**, for example: `RADV_DEBUG=nocache,shaders` otherwise only the last one will take effect. (This is how all environment variables work, yet this is an often made mistake.) Example:

```
RADV_DEBUG=nocache,shaders ACO_DEBUG=validateir,validatera vkcube
```

### Using GCC sanitizers

GCC has several sanitizers which can help figure out hard to diagnose issues. To use these, you need to pass
the `-Dbsanitize` flag to `meson` when building mesa. For example `-Dbsanitize=undefined` will add support for
the undefined behavior sanitizer.

### Hardened builds and glibc++ assertions

Several Linux distributions use "hardened" builds meaning several special compiler flags are added by
downstream packaging which are not used in mesa builds by default. These may be responsible for
some bug reports of inexplicable crashes with assertion failures you can't reproduce.

Most notable are the glibc++ debug flags, which you can use by adding the `-D_GLIBCXX_ASSERTIONS=1` and
`-D_GLIBCXX_DEBUG=1` flags.

To see the full list of downstream compiler flags, you can use eg. `rpm --eval "%optflags"`
on Red Hat based distros like Fedora.

### Good practices

Here are some good practices we learned while debugging visual corruption and hangs.

1. Bisecting shaders:
    * Use renderdoc when examining shaders. This is deterministic while real games often use multi-threading or change the order in which shaders get compiled.
    * Edit `radv_shader.c` or `radv_pipeline.c` to change if they are compiled with LLVM or ACO.
2. Things to check early:
    * Disable value_numbering, optimizer and/or scheduler.
      Note that if any of these change the output, it does not necessarily mean that the error is there, as register assignment does also change.
3. Finding the instruction causing a hang:
    * The ability to directly manipulate the binaries gives us an easy way to find the exact instruction which causes the hang.
      Use NULL exports (for FS and VS) and `s_endpgm` to end the shader early to find the problematic instruction.
4. Other faulty instructions:
    * Use print_asm and check for illegal instructions.
    * Compare to the ACO IR to see if the assembly matches what we want (this can take a while).
      Typical issues might be a wrong instruction format leading to a wrong opcode or an sgpr used for vgpr field.
5. Comparing to the LLVM backend:
   * If everything else didn't help, we probably just do something wrong. The LLVM backend is quite mature, so its output might help find differences, but this can be a long road.
6. Investigating regressions in shaders:
   * If you know that something used to work and are sure it's a shader problem,
     use `RADV_DEBUG=shaders` to print both the correct and incorrect version of the shader.
     You can further filter by shader stage and compilation stage, eg. `RADV_DEBUG=ps,nir,asm`
   * Copy the printed shaders into a diff viewer, eg. Meld, to quickly find what changed
     between the two versions.