1.. _vb_framework: 2 3Videobuf Framework 4================== 5 6Author: Jonathan Corbet <corbet@lwn.net> 7 8Current as of 2.6.33 9 10.. note:: 11 12 The videobuf framework was deprecated in favor of videobuf2. Shouldn't 13 be used on new drivers. 14 15Introduction 16------------ 17 18The videobuf layer functions as a sort of glue layer between a V4L2 driver 19and user space. It handles the allocation and management of buffers for 20the storage of video frames. There is a set of functions which can be used 21to implement many of the standard POSIX I/O system calls, including read(), 22poll(), and, happily, mmap(). Another set of functions can be used to 23implement the bulk of the V4L2 ioctl() calls related to streaming I/O, 24including buffer allocation, queueing and dequeueing, and streaming 25control. Using videobuf imposes a few design decisions on the driver 26author, but the payback comes in the form of reduced code in the driver and 27a consistent implementation of the V4L2 user-space API. 28 29Buffer types 30------------ 31 32Not all video devices use the same kind of buffers. In fact, there are (at 33least) three common variations: 34 35 - Buffers which are scattered in both the physical and (kernel) virtual 36 address spaces. (Almost) all user-space buffers are like this, but it 37 makes great sense to allocate kernel-space buffers this way as well when 38 it is possible. Unfortunately, it is not always possible; working with 39 this kind of buffer normally requires hardware which can do 40 scatter/gather DMA operations. 41 42 - Buffers which are physically scattered, but which are virtually 43 contiguous; buffers allocated with vmalloc(), in other words. These 44 buffers are just as hard to use for DMA operations, but they can be 45 useful in situations where DMA is not available but virtually-contiguous 46 buffers are convenient. 47 48 - Buffers which are physically contiguous. Allocation of this kind of 49 buffer can be unreliable on fragmented systems, but simpler DMA 50 controllers cannot deal with anything else. 51 52Videobuf can work with all three types of buffers, but the driver author 53must pick one at the outset and design the driver around that decision. 54 55[It's worth noting that there's a fourth kind of buffer: "overlay" buffers 56which are located within the system's video memory. The overlay 57functionality is considered to be deprecated for most use, but it still 58shows up occasionally in system-on-chip drivers where the performance 59benefits merit the use of this technique. Overlay buffers can be handled 60as a form of scattered buffer, but there are very few implementations in 61the kernel and a description of this technique is currently beyond the 62scope of this document.] 63 64Data structures, callbacks, and initialization 65---------------------------------------------- 66 67Depending on which type of buffers are being used, the driver should 68include one of the following files: 69 70.. code-block:: none 71 72 <media/videobuf-dma-sg.h> /* Physically scattered */ 73 <media/videobuf-vmalloc.h> /* vmalloc() buffers */ 74 <media/videobuf-dma-contig.h> /* Physically contiguous */ 75 76The driver's data structure describing a V4L2 device should include a 77struct videobuf_queue instance for the management of the buffer queue, 78along with a list_head for the queue of available buffers. There will also 79need to be an interrupt-safe spinlock which is used to protect (at least) 80the queue. 81 82The next step is to write four simple callbacks to help videobuf deal with 83the management of buffers: 84 85.. code-block:: none 86 87 struct videobuf_queue_ops { 88 int (*buf_setup)(struct videobuf_queue *q, 89 unsigned int *count, unsigned int *size); 90 int (*buf_prepare)(struct videobuf_queue *q, 91 struct videobuf_buffer *vb, 92 enum v4l2_field field); 93 void (*buf_queue)(struct videobuf_queue *q, 94 struct videobuf_buffer *vb); 95 void (*buf_release)(struct videobuf_queue *q, 96 struct videobuf_buffer *vb); 97 }; 98 99buf_setup() is called early in the I/O process, when streaming is being 100initiated; its purpose is to tell videobuf about the I/O stream. The count 101parameter will be a suggested number of buffers to use; the driver should 102check it for rationality and adjust it if need be. As a practical rule, a 103minimum of two buffers are needed for proper streaming, and there is 104usually a maximum (which cannot exceed 32) which makes sense for each 105device. The size parameter should be set to the expected (maximum) size 106for each frame of data. 107 108Each buffer (in the form of a struct videobuf_buffer pointer) will be 109passed to buf_prepare(), which should set the buffer's size, width, height, 110and field fields properly. If the buffer's state field is 111VIDEOBUF_NEEDS_INIT, the driver should pass it to: 112 113.. code-block:: none 114 115 int videobuf_iolock(struct videobuf_queue* q, struct videobuf_buffer *vb, 116 struct v4l2_framebuffer *fbuf); 117 118Among other things, this call will usually allocate memory for the buffer. 119Finally, the buf_prepare() function should set the buffer's state to 120VIDEOBUF_PREPARED. 121 122When a buffer is queued for I/O, it is passed to buf_queue(), which should 123put it onto the driver's list of available buffers and set its state to 124VIDEOBUF_QUEUED. Note that this function is called with the queue spinlock 125held; if it tries to acquire it as well things will come to a screeching 126halt. Yes, this is the voice of experience. Note also that videobuf may 127wait on the first buffer in the queue; placing other buffers in front of it 128could again gum up the works. So use list_add_tail() to enqueue buffers. 129 130Finally, buf_release() is called when a buffer is no longer intended to be 131used. The driver should ensure that there is no I/O active on the buffer, 132then pass it to the appropriate free routine(s): 133 134.. code-block:: none 135 136 /* Scatter/gather drivers */ 137 int videobuf_dma_unmap(struct videobuf_queue *q, 138 struct videobuf_dmabuf *dma); 139 int videobuf_dma_free(struct videobuf_dmabuf *dma); 140 141 /* vmalloc drivers */ 142 void videobuf_vmalloc_free (struct videobuf_buffer *buf); 143 144 /* Contiguous drivers */ 145 void videobuf_dma_contig_free(struct videobuf_queue *q, 146 struct videobuf_buffer *buf); 147 148One way to ensure that a buffer is no longer under I/O is to pass it to: 149 150.. code-block:: none 151 152 int videobuf_waiton(struct videobuf_buffer *vb, int non_blocking, int intr); 153 154Here, vb is the buffer, non_blocking indicates whether non-blocking I/O 155should be used (it should be zero in the buf_release() case), and intr 156controls whether an interruptible wait is used. 157 158File operations 159--------------- 160 161At this point, much of the work is done; much of the rest is slipping 162videobuf calls into the implementation of the other driver callbacks. The 163first step is in the open() function, which must initialize the 164videobuf queue. The function to use depends on the type of buffer used: 165 166.. code-block:: none 167 168 void videobuf_queue_sg_init(struct videobuf_queue *q, 169 struct videobuf_queue_ops *ops, 170 struct device *dev, 171 spinlock_t *irqlock, 172 enum v4l2_buf_type type, 173 enum v4l2_field field, 174 unsigned int msize, 175 void *priv); 176 177 void videobuf_queue_vmalloc_init(struct videobuf_queue *q, 178 struct videobuf_queue_ops *ops, 179 struct device *dev, 180 spinlock_t *irqlock, 181 enum v4l2_buf_type type, 182 enum v4l2_field field, 183 unsigned int msize, 184 void *priv); 185 186 void videobuf_queue_dma_contig_init(struct videobuf_queue *q, 187 struct videobuf_queue_ops *ops, 188 struct device *dev, 189 spinlock_t *irqlock, 190 enum v4l2_buf_type type, 191 enum v4l2_field field, 192 unsigned int msize, 193 void *priv); 194 195In each case, the parameters are the same: q is the queue structure for the 196device, ops is the set of callbacks as described above, dev is the device 197structure for this video device, irqlock is an interrupt-safe spinlock to 198protect access to the data structures, type is the buffer type used by the 199device (cameras will use V4L2_BUF_TYPE_VIDEO_CAPTURE, for example), field 200describes which field is being captured (often V4L2_FIELD_NONE for 201progressive devices), msize is the size of any containing structure used 202around struct videobuf_buffer, and priv is a private data pointer which 203shows up in the priv_data field of struct videobuf_queue. Note that these 204are void functions which, evidently, are immune to failure. 205 206V4L2 capture drivers can be written to support either of two APIs: the 207read() system call and the rather more complicated streaming mechanism. As 208a general rule, it is necessary to support both to ensure that all 209applications have a chance of working with the device. Videobuf makes it 210easy to do that with the same code. To implement read(), the driver need 211only make a call to one of: 212 213.. code-block:: none 214 215 ssize_t videobuf_read_one(struct videobuf_queue *q, 216 char __user *data, size_t count, 217 loff_t *ppos, int nonblocking); 218 219 ssize_t videobuf_read_stream(struct videobuf_queue *q, 220 char __user *data, size_t count, 221 loff_t *ppos, int vbihack, int nonblocking); 222 223Either one of these functions will read frame data into data, returning the 224amount actually read; the difference is that videobuf_read_one() will only 225read a single frame, while videobuf_read_stream() will read multiple frames 226if they are needed to satisfy the count requested by the application. A 227typical driver read() implementation will start the capture engine, call 228one of the above functions, then stop the engine before returning (though a 229smarter implementation might leave the engine running for a little while in 230anticipation of another read() call happening in the near future). 231 232The poll() function can usually be implemented with a direct call to: 233 234.. code-block:: none 235 236 unsigned int videobuf_poll_stream(struct file *file, 237 struct videobuf_queue *q, 238 poll_table *wait); 239 240Note that the actual wait queue eventually used will be the one associated 241with the first available buffer. 242 243When streaming I/O is done to kernel-space buffers, the driver must support 244the mmap() system call to enable user space to access the data. In many 245V4L2 drivers, the often-complex mmap() implementation simplifies to a 246single call to: 247 248.. code-block:: none 249 250 int videobuf_mmap_mapper(struct videobuf_queue *q, 251 struct vm_area_struct *vma); 252 253Everything else is handled by the videobuf code. 254 255The release() function requires two separate videobuf calls: 256 257.. code-block:: none 258 259 void videobuf_stop(struct videobuf_queue *q); 260 int videobuf_mmap_free(struct videobuf_queue *q); 261 262The call to videobuf_stop() terminates any I/O in progress - though it is 263still up to the driver to stop the capture engine. The call to 264videobuf_mmap_free() will ensure that all buffers have been unmapped; if 265so, they will all be passed to the buf_release() callback. If buffers 266remain mapped, videobuf_mmap_free() returns an error code instead. The 267purpose is clearly to cause the closing of the file descriptor to fail if 268buffers are still mapped, but every driver in the 2.6.32 kernel cheerfully 269ignores its return value. 270 271ioctl() operations 272------------------ 273 274The V4L2 API includes a very long list of driver callbacks to respond to 275the many ioctl() commands made available to user space. A number of these 276- those associated with streaming I/O - turn almost directly into videobuf 277calls. The relevant helper functions are: 278 279.. code-block:: none 280 281 int videobuf_reqbufs(struct videobuf_queue *q, 282 struct v4l2_requestbuffers *req); 283 int videobuf_querybuf(struct videobuf_queue *q, struct v4l2_buffer *b); 284 int videobuf_qbuf(struct videobuf_queue *q, struct v4l2_buffer *b); 285 int videobuf_dqbuf(struct videobuf_queue *q, struct v4l2_buffer *b, 286 int nonblocking); 287 int videobuf_streamon(struct videobuf_queue *q); 288 int videobuf_streamoff(struct videobuf_queue *q); 289 290So, for example, a VIDIOC_REQBUFS call turns into a call to the driver's 291vidioc_reqbufs() callback which, in turn, usually only needs to locate the 292proper struct videobuf_queue pointer and pass it to videobuf_reqbufs(). 293These support functions can replace a great deal of buffer management 294boilerplate in a lot of V4L2 drivers. 295 296The vidioc_streamon() and vidioc_streamoff() functions will be a bit more 297complex, of course, since they will also need to deal with starting and 298stopping the capture engine. 299 300Buffer allocation 301----------------- 302 303Thus far, we have talked about buffers, but have not looked at how they are 304allocated. The scatter/gather case is the most complex on this front. For 305allocation, the driver can leave buffer allocation entirely up to the 306videobuf layer; in this case, buffers will be allocated as anonymous 307user-space pages and will be very scattered indeed. If the application is 308using user-space buffers, no allocation is needed; the videobuf layer will 309take care of calling get_user_pages() and filling in the scatterlist array. 310 311If the driver needs to do its own memory allocation, it should be done in 312the vidioc_reqbufs() function, *after* calling videobuf_reqbufs(). The 313first step is a call to: 314 315.. code-block:: none 316 317 struct videobuf_dmabuf *videobuf_to_dma(struct videobuf_buffer *buf); 318 319The returned videobuf_dmabuf structure (defined in 320<media/videobuf-dma-sg.h>) includes a couple of relevant fields: 321 322.. code-block:: none 323 324 struct scatterlist *sglist; 325 int sglen; 326 327The driver must allocate an appropriately-sized scatterlist array and 328populate it with pointers to the pieces of the allocated buffer; sglen 329should be set to the length of the array. 330 331Drivers using the vmalloc() method need not (and cannot) concern themselves 332with buffer allocation at all; videobuf will handle those details. The 333same is normally true of contiguous-DMA drivers as well; videobuf will 334allocate the buffers (with dma_alloc_coherent()) when it sees fit. That 335means that these drivers may be trying to do high-order allocations at any 336time, an operation which is not always guaranteed to work. Some drivers 337play tricks by allocating DMA space at system boot time; videobuf does not 338currently play well with those drivers. 339 340As of 2.6.31, contiguous-DMA drivers can work with a user-supplied buffer, 341as long as that buffer is physically contiguous. Normal user-space 342allocations will not meet that criterion, but buffers obtained from other 343kernel drivers, or those contained within huge pages, will work with these 344drivers. 345 346Filling the buffers 347------------------- 348 349The final part of a videobuf implementation has no direct callback - it's 350the portion of the code which actually puts frame data into the buffers, 351usually in response to interrupts from the device. For all types of 352drivers, this process works approximately as follows: 353 354 - Obtain the next available buffer and make sure that somebody is actually 355 waiting for it. 356 357 - Get a pointer to the memory and put video data there. 358 359 - Mark the buffer as done and wake up the process waiting for it. 360 361Step (1) above is done by looking at the driver-managed list_head structure 362- the one which is filled in the buf_queue() callback. Because starting 363the engine and enqueueing buffers are done in separate steps, it's possible 364for the engine to be running without any buffers available - in the 365vmalloc() case especially. So the driver should be prepared for the list 366to be empty. It is equally possible that nobody is yet interested in the 367buffer; the driver should not remove it from the list or fill it until a 368process is waiting on it. That test can be done by examining the buffer's 369done field (a wait_queue_head_t structure) with waitqueue_active(). 370 371A buffer's state should be set to VIDEOBUF_ACTIVE before being mapped for 372DMA; that ensures that the videobuf layer will not try to do anything with 373it while the device is transferring data. 374 375For scatter/gather drivers, the needed memory pointers will be found in the 376scatterlist structure described above. Drivers using the vmalloc() method 377can get a memory pointer with: 378 379.. code-block:: none 380 381 void *videobuf_to_vmalloc(struct videobuf_buffer *buf); 382 383For contiguous DMA drivers, the function to use is: 384 385.. code-block:: none 386 387 dma_addr_t videobuf_to_dma_contig(struct videobuf_buffer *buf); 388 389The contiguous DMA API goes out of its way to hide the kernel-space address 390of the DMA buffer from drivers. 391 392The final step is to set the size field of the relevant videobuf_buffer 393structure to the actual size of the captured image, set state to 394VIDEOBUF_DONE, then call wake_up() on the done queue. At this point, the 395buffer is owned by the videobuf layer and the driver should not touch it 396again. 397 398Developers who are interested in more information can go into the relevant 399header files; there are a few low-level functions declared there which have 400not been talked about here. Also worthwhile is the vivi driver 401(drivers/media/platform/vivi.c), which is maintained as an example of how V4L2 402drivers should be written. Vivi only uses the vmalloc() API, but it's good 403enough to get started with. Note also that all of these calls are exported 404GPL-only, so they will not be available to non-GPL kernel modules. 405