1Buffer Sharing and Synchronization 2================================== 3 4The dma-buf subsystem provides the framework for sharing buffers for 5hardware (DMA) access across multiple device drivers and subsystems, and 6for synchronizing asynchronous hardware access. 7 8This is used, for example, by drm "prime" multi-GPU support, but is of 9course not limited to GPU use cases. 10 11The three main components of this are: (1) dma-buf, representing a 12sg_table and exposed to userspace as a file descriptor to allow passing 13between devices, (2) fence, which provides a mechanism to signal when 14one device has finished access, and (3) reservation, which manages the 15shared or exclusive fence(s) associated with the buffer. 16 17Shared DMA Buffers 18------------------ 19 20This document serves as a guide to device-driver writers on what is the dma-buf 21buffer sharing API, how to use it for exporting and using shared buffers. 22 23Any device driver which wishes to be a part of DMA buffer sharing, can do so as 24either the 'exporter' of buffers, or the 'user' or 'importer' of buffers. 25 26Say a driver A wants to use buffers created by driver B, then we call B as the 27exporter, and A as buffer-user/importer. 28 29The exporter 30 31 - implements and manages operations in :c:type:`struct dma_buf_ops 32 <dma_buf_ops>` for the buffer, 33 - allows other users to share the buffer by using dma_buf sharing APIs, 34 - manages the details of buffer allocation, wrapped in a :c:type:`struct 35 dma_buf <dma_buf>`, 36 - decides about the actual backing storage where this allocation happens, 37 - and takes care of any migration of scatterlist - for all (shared) users of 38 this buffer. 39 40The buffer-user 41 42 - is one of (many) sharing users of the buffer. 43 - doesn't need to worry about how the buffer is allocated, or where. 44 - and needs a mechanism to get access to the scatterlist that makes up this 45 buffer in memory, mapped into its own address space, so it can access the 46 same area of memory. This interface is provided by :c:type:`struct 47 dma_buf_attachment <dma_buf_attachment>`. 48 49Any exporters or users of the dma-buf buffer sharing framework must have a 50'select DMA_SHARED_BUFFER' in their respective Kconfigs. 51 52Userspace Interface Notes 53~~~~~~~~~~~~~~~~~~~~~~~~~ 54 55Mostly a DMA buffer file descriptor is simply an opaque object for userspace, 56and hence the generic interface exposed is very minimal. There's a few things to 57consider though: 58 59- Since kernel 3.12 the dma-buf FD supports the llseek system call, but only 60 with offset=0 and whence=SEEK_END|SEEK_SET. SEEK_SET is supported to allow 61 the usual size discover pattern size = SEEK_END(0); SEEK_SET(0). Every other 62 llseek operation will report -EINVAL. 63 64 If llseek on dma-buf FDs isn't support the kernel will report -ESPIPE for all 65 cases. Userspace can use this to detect support for discovering the dma-buf 66 size using llseek. 67 68- In order to avoid fd leaks on exec, the FD_CLOEXEC flag must be set 69 on the file descriptor. This is not just a resource leak, but a 70 potential security hole. It could give the newly exec'd application 71 access to buffers, via the leaked fd, to which it should otherwise 72 not be permitted access. 73 74 The problem with doing this via a separate fcntl() call, versus doing it 75 atomically when the fd is created, is that this is inherently racy in a 76 multi-threaded app[3]. The issue is made worse when it is library code 77 opening/creating the file descriptor, as the application may not even be 78 aware of the fd's. 79 80 To avoid this problem, userspace must have a way to request O_CLOEXEC 81 flag be set when the dma-buf fd is created. So any API provided by 82 the exporting driver to create a dmabuf fd must provide a way to let 83 userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd(). 84 85- Memory mapping the contents of the DMA buffer is also supported. See the 86 discussion below on `CPU Access to DMA Buffer Objects`_ for the full details. 87 88- The DMA buffer FD is also pollable, see `Implicit Fence Poll Support`_ below for 89 details. 90 91Basic Operation and Device DMA Access 92~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 93 94.. kernel-doc:: drivers/dma-buf/dma-buf.c 95 :doc: dma buf device access 96 97CPU Access to DMA Buffer Objects 98~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 99 100.. kernel-doc:: drivers/dma-buf/dma-buf.c 101 :doc: cpu access 102 103Implicit Fence Poll Support 104~~~~~~~~~~~~~~~~~~~~~~~~~~~ 105 106.. kernel-doc:: drivers/dma-buf/dma-buf.c 107 :doc: implicit fence polling 108 109DMA-BUF statistics 110~~~~~~~~~~~~~~~~~~ 111.. kernel-doc:: drivers/dma-buf/dma-buf-sysfs-stats.c 112 :doc: overview 113 114Kernel Functions and Structures Reference 115~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 116 117.. kernel-doc:: drivers/dma-buf/dma-buf.c 118 :export: 119 120.. kernel-doc:: include/linux/dma-buf.h 121 :internal: 122 123Reservation Objects 124------------------- 125 126.. kernel-doc:: drivers/dma-buf/dma-resv.c 127 :doc: Reservation Object Overview 128 129.. kernel-doc:: drivers/dma-buf/dma-resv.c 130 :export: 131 132.. kernel-doc:: include/linux/dma-resv.h 133 :internal: 134 135DMA Fences 136---------- 137 138.. kernel-doc:: drivers/dma-buf/dma-fence.c 139 :doc: DMA fences overview 140 141DMA Fence Cross-Driver Contract 142~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 143 144.. kernel-doc:: drivers/dma-buf/dma-fence.c 145 :doc: fence cross-driver contract 146 147DMA Fence Signalling Annotations 148~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 149 150.. kernel-doc:: drivers/dma-buf/dma-fence.c 151 :doc: fence signalling annotation 152 153DMA Fences Functions Reference 154~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 155 156.. kernel-doc:: drivers/dma-buf/dma-fence.c 157 :export: 158 159.. kernel-doc:: include/linux/dma-fence.h 160 :internal: 161 162Seqno Hardware Fences 163~~~~~~~~~~~~~~~~~~~~~ 164 165.. kernel-doc:: include/linux/seqno-fence.h 166 :internal: 167 168DMA Fence Array 169~~~~~~~~~~~~~~~ 170 171.. kernel-doc:: drivers/dma-buf/dma-fence-array.c 172 :export: 173 174.. kernel-doc:: include/linux/dma-fence-array.h 175 :internal: 176 177DMA Fence uABI/Sync File 178~~~~~~~~~~~~~~~~~~~~~~~~ 179 180.. kernel-doc:: drivers/dma-buf/sync_file.c 181 :export: 182 183.. kernel-doc:: include/linux/sync_file.h 184 :internal: 185 186Indefinite DMA Fences 187~~~~~~~~~~~~~~~~~~~~~ 188 189At various times &dma_fence with an indefinite time until dma_fence_wait() 190finishes have been proposed. Examples include: 191 192* Future fences, used in HWC1 to signal when a buffer isn't used by the display 193 any longer, and created with the screen update that makes the buffer visible. 194 The time this fence completes is entirely under userspace's control. 195 196* Proxy fences, proposed to handle &drm_syncobj for which the fence has not yet 197 been set. Used to asynchronously delay command submission. 198 199* Userspace fences or gpu futexes, fine-grained locking within a command buffer 200 that userspace uses for synchronization across engines or with the CPU, which 201 are then imported as a DMA fence for integration into existing winsys 202 protocols. 203 204* Long-running compute command buffers, while still using traditional end of 205 batch DMA fences for memory management instead of context preemption DMA 206 fences which get reattached when the compute job is rescheduled. 207 208Common to all these schemes is that userspace controls the dependencies of these 209fences and controls when they fire. Mixing indefinite fences with normal 210in-kernel DMA fences does not work, even when a fallback timeout is included to 211protect against malicious userspace: 212 213* Only the kernel knows about all DMA fence dependencies, userspace is not aware 214 of dependencies injected due to memory management or scheduler decisions. 215 216* Only userspace knows about all dependencies in indefinite fences and when 217 exactly they will complete, the kernel has no visibility. 218 219Furthermore the kernel has to be able to hold up userspace command submission 220for memory management needs, which means we must support indefinite fences being 221dependent upon DMA fences. If the kernel also support indefinite fences in the 222kernel like a DMA fence, like any of the above proposal would, there is the 223potential for deadlocks. 224 225.. kernel-render:: DOT 226 :alt: Indefinite Fencing Dependency Cycle 227 :caption: Indefinite Fencing Dependency Cycle 228 229 digraph "Fencing Cycle" { 230 node [shape=box bgcolor=grey style=filled] 231 kernel [label="Kernel DMA Fences"] 232 userspace [label="userspace controlled fences"] 233 kernel -> userspace [label="memory management"] 234 userspace -> kernel [label="Future fence, fence proxy, ..."] 235 236 { rank=same; kernel userspace } 237 } 238 239This means that the kernel might accidentally create deadlocks 240through memory management dependencies which userspace is unaware of, which 241randomly hangs workloads until the timeout kicks in. Workloads, which from 242userspace's perspective, do not contain a deadlock. In such a mixed fencing 243architecture there is no single entity with knowledge of all dependencies. 244Thefore preventing such deadlocks from within the kernel is not possible. 245 246The only solution to avoid dependencies loops is by not allowing indefinite 247fences in the kernel. This means: 248 249* No future fences, proxy fences or userspace fences imported as DMA fences, 250 with or without a timeout. 251 252* No DMA fences that signal end of batchbuffer for command submission where 253 userspace is allowed to use userspace fencing or long running compute 254 workloads. This also means no implicit fencing for shared buffers in these 255 cases. 256