1Buffer Sharing and Synchronization 2================================== 3 4The dma-buf subsystem provides the framework for sharing buffers for 5hardware (DMA) access across multiple device drivers and subsystems, and 6for synchronizing asynchronous hardware access. 7 8This is used, for example, by drm "prime" multi-GPU support, but is of 9course not limited to GPU use cases. 10 11The three main components of this are: (1) dma-buf, representing a 12sg_table and exposed to userspace as a file descriptor to allow passing 13between devices, (2) fence, which provides a mechanism to signal when 14one device has finished access, and (3) reservation, which manages the 15shared or exclusive fence(s) associated with the buffer. 16 17Shared DMA Buffers 18------------------ 19 20This document serves as a guide to device-driver writers on what is the dma-buf 21buffer sharing API, how to use it for exporting and using shared buffers. 22 23Any device driver which wishes to be a part of DMA buffer sharing, can do so as 24either the 'exporter' of buffers, or the 'user' or 'importer' of buffers. 25 26Say a driver A wants to use buffers created by driver B, then we call B as the 27exporter, and A as buffer-user/importer. 28 29The exporter 30 31 - implements and manages operations in :c:type:`struct dma_buf_ops 32 <dma_buf_ops>` for the buffer, 33 - allows other users to share the buffer by using dma_buf sharing APIs, 34 - manages the details of buffer allocation, wrapped in a :c:type:`struct 35 dma_buf <dma_buf>`, 36 - decides about the actual backing storage where this allocation happens, 37 - and takes care of any migration of scatterlist - for all (shared) users of 38 this buffer. 39 40The buffer-user 41 42 - is one of (many) sharing users of the buffer. 43 - doesn't need to worry about how the buffer is allocated, or where. 44 - and needs a mechanism to get access to the scatterlist that makes up this 45 buffer in memory, mapped into its own address space, so it can access the 46 same area of memory. This interface is provided by :c:type:`struct 47 dma_buf_attachment <dma_buf_attachment>`. 48 49Any exporters or users of the dma-buf buffer sharing framework must have a 50'select DMA_SHARED_BUFFER' in their respective Kconfigs. 51 52Userspace Interface Notes 53~~~~~~~~~~~~~~~~~~~~~~~~~ 54 55Mostly a DMA buffer file descriptor is simply an opaque object for userspace, 56and hence the generic interface exposed is very minimal. There's a few things to 57consider though: 58 59- Since kernel 3.12 the dma-buf FD supports the llseek system call, but only 60 with offset=0 and whence=SEEK_END|SEEK_SET. SEEK_SET is supported to allow 61 the usual size discover pattern size = SEEK_END(0); SEEK_SET(0). Every other 62 llseek operation will report -EINVAL. 63 64 If llseek on dma-buf FDs isn't support the kernel will report -ESPIPE for all 65 cases. Userspace can use this to detect support for discovering the dma-buf 66 size using llseek. 67 68- In order to avoid fd leaks on exec, the FD_CLOEXEC flag must be set 69 on the file descriptor. This is not just a resource leak, but a 70 potential security hole. It could give the newly exec'd application 71 access to buffers, via the leaked fd, to which it should otherwise 72 not be permitted access. 73 74 The problem with doing this via a separate fcntl() call, versus doing it 75 atomically when the fd is created, is that this is inherently racy in a 76 multi-threaded app[3]. The issue is made worse when it is library code 77 opening/creating the file descriptor, as the application may not even be 78 aware of the fd's. 79 80 To avoid this problem, userspace must have a way to request O_CLOEXEC 81 flag be set when the dma-buf fd is created. So any API provided by 82 the exporting driver to create a dmabuf fd must provide a way to let 83 userspace control setting of O_CLOEXEC flag passed in to dma_buf_fd(). 84 85- Memory mapping the contents of the DMA buffer is also supported. See the 86 discussion below on `CPU Access to DMA Buffer Objects`_ for the full details. 87 88- The DMA buffer FD is also pollable, see `Implicit Fence Poll Support`_ below for 89 details. 90 91Basic Operation and Device DMA Access 92~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 93 94.. kernel-doc:: drivers/dma-buf/dma-buf.c 95 :doc: dma buf device access 96 97CPU Access to DMA Buffer Objects 98~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 99 100.. kernel-doc:: drivers/dma-buf/dma-buf.c 101 :doc: cpu access 102 103Implicit Fence Poll Support 104~~~~~~~~~~~~~~~~~~~~~~~~~~~ 105 106.. kernel-doc:: drivers/dma-buf/dma-buf.c 107 :doc: implicit fence polling 108 109Kernel Functions and Structures Reference 110~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 111 112.. kernel-doc:: drivers/dma-buf/dma-buf.c 113 :export: 114 115.. kernel-doc:: include/linux/dma-buf.h 116 :internal: 117 118Reservation Objects 119------------------- 120 121.. kernel-doc:: drivers/dma-buf/dma-resv.c 122 :doc: Reservation Object Overview 123 124.. kernel-doc:: drivers/dma-buf/dma-resv.c 125 :export: 126 127.. kernel-doc:: include/linux/dma-resv.h 128 :internal: 129 130DMA Fences 131---------- 132 133.. kernel-doc:: drivers/dma-buf/dma-fence.c 134 :doc: DMA fences overview 135 136DMA Fence Cross-Driver Contract 137~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 138 139.. kernel-doc:: drivers/dma-buf/dma-fence.c 140 :doc: fence cross-driver contract 141 142DMA Fence Signalling Annotations 143~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 144 145.. kernel-doc:: drivers/dma-buf/dma-fence.c 146 :doc: fence signalling annotation 147 148DMA Fences Functions Reference 149~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 150 151.. kernel-doc:: drivers/dma-buf/dma-fence.c 152 :export: 153 154.. kernel-doc:: include/linux/dma-fence.h 155 :internal: 156 157Seqno Hardware Fences 158~~~~~~~~~~~~~~~~~~~~~ 159 160.. kernel-doc:: include/linux/seqno-fence.h 161 :internal: 162 163DMA Fence Array 164~~~~~~~~~~~~~~~ 165 166.. kernel-doc:: drivers/dma-buf/dma-fence-array.c 167 :export: 168 169.. kernel-doc:: include/linux/dma-fence-array.h 170 :internal: 171 172DMA Fence uABI/Sync File 173~~~~~~~~~~~~~~~~~~~~~~~~ 174 175.. kernel-doc:: drivers/dma-buf/sync_file.c 176 :export: 177 178.. kernel-doc:: include/linux/sync_file.h 179 :internal: 180 181Indefinite DMA Fences 182~~~~~~~~~~~~~~~~~~~~~ 183 184At various times &dma_fence with an indefinite time until dma_fence_wait() 185finishes have been proposed. Examples include: 186 187* Future fences, used in HWC1 to signal when a buffer isn't used by the display 188 any longer, and created with the screen update that makes the buffer visible. 189 The time this fence completes is entirely under userspace's control. 190 191* Proxy fences, proposed to handle &drm_syncobj for which the fence has not yet 192 been set. Used to asynchronously delay command submission. 193 194* Userspace fences or gpu futexes, fine-grained locking within a command buffer 195 that userspace uses for synchronization across engines or with the CPU, which 196 are then imported as a DMA fence for integration into existing winsys 197 protocols. 198 199* Long-running compute command buffers, while still using traditional end of 200 batch DMA fences for memory management instead of context preemption DMA 201 fences which get reattached when the compute job is rescheduled. 202 203Common to all these schemes is that userspace controls the dependencies of these 204fences and controls when they fire. Mixing indefinite fences with normal 205in-kernel DMA fences does not work, even when a fallback timeout is included to 206protect against malicious userspace: 207 208* Only the kernel knows about all DMA fence dependencies, userspace is not aware 209 of dependencies injected due to memory management or scheduler decisions. 210 211* Only userspace knows about all dependencies in indefinite fences and when 212 exactly they will complete, the kernel has no visibility. 213 214Furthermore the kernel has to be able to hold up userspace command submission 215for memory management needs, which means we must support indefinite fences being 216dependent upon DMA fences. If the kernel also support indefinite fences in the 217kernel like a DMA fence, like any of the above proposal would, there is the 218potential for deadlocks. 219 220.. kernel-render:: DOT 221 :alt: Indefinite Fencing Dependency Cycle 222 :caption: Indefinite Fencing Dependency Cycle 223 224 digraph "Fencing Cycle" { 225 node [shape=box bgcolor=grey style=filled] 226 kernel [label="Kernel DMA Fences"] 227 userspace [label="userspace controlled fences"] 228 kernel -> userspace [label="memory management"] 229 userspace -> kernel [label="Future fence, fence proxy, ..."] 230 231 { rank=same; kernel userspace } 232 } 233 234This means that the kernel might accidentally create deadlocks 235through memory management dependencies which userspace is unaware of, which 236randomly hangs workloads until the timeout kicks in. Workloads, which from 237userspace's perspective, do not contain a deadlock. In such a mixed fencing 238architecture there is no single entity with knowledge of all dependencies. 239Thefore preventing such deadlocks from within the kernel is not possible. 240 241The only solution to avoid dependencies loops is by not allowing indefinite 242fences in the kernel. This means: 243 244* No future fences, proxy fences or userspace fences imported as DMA fences, 245 with or without a timeout. 246 247* No DMA fences that signal end of batchbuffer for command submission where 248 userspace is allowed to use userspace fencing or long running compute 249 workloads. This also means no implicit fencing for shared buffers in these 250 cases. 251