1// Copyright 2018-2021 The Khronos Group Inc. 2// 3// SPDX-License-Identifier: CC-BY-4.0 4 5[[video-decode-operations]] 6== Video Decode Operations 7 8Before the application can start recording Vulkan command buffers for the 9Video Decode Operations, it must: do the following, beforehand: 10 11 . Ensure that the implementation can decode the Video Content by querying 12 the supported codec operations and profiles using 13 flink:vkGetPhysicalDeviceQueueFamilyProperties2. 14 . By using flink:vkGetPhysicalDeviceVideoFormatPropertiesKHR and providing 15 one or more video profiles, choose the Vulkan formats supported by the 16 implementation. 17 The formats for <<decoded-output-picture,output>> and 18 <<reference-picture,reference>> pictures must: be queried and chosen 19 separately. 20 Refer to the section on 21 <<supported-video-input-output-dpb-formats-enumeration,enumeration of 22 supported video formats>>. 23 . Before creating an image to be used as a video picture resource, obtain 24 the supported image creation parameters by querying with 25 flink:vkGetPhysicalDeviceFormatProperties2 and 26 flink:vkGetPhysicalDeviceImageFormatProperties2 using one of the 27 reported formats and adding slink:VkVideoProfilesKHR to the pname:pNext 28 chain of slink:VkFormatProperties2. 29 When querying the parameters with 30 flink:vkGetPhysicalDeviceImageFormatProperties2 for images targeting 31 decoded <<decoded-output-picture,output>> and 32 <<reference-picture,reference (DPB)>> pictures, the 33 slink:VkPhysicalDeviceImageFormatInfo2::pname:usage field should contain 34 ename:VK_IMAGE_USAGE_VIDEO_DECODE_DST_BIT_KHR and 35 ename:VK_IMAGE_USAGE_VIDEO_DECODE_DPB_BIT_KHR, respectively. 36 . Create none, some, or all of the required <<VkImage,images>> for the 37 <<decoded-output-picture,decoded output>> and 38 <<reference-picture,reference>> pictures. 39 More Video Picture Resources can: be created at some later point if 40 needed while processing the decoded content. 41 Also, if the decoded picture size is expected to change, the images can: 42 be created based on the maximum decoded content size required. 43 . Create the <<video-session,video session>> to be used for video decode 44 operations. 45 Before creating the Decode Video Session, the decode capabilities 46 should: be queried with flink:vkGetPhysicalDeviceVideoCapabilitiesKHR to 47 obtain the limits of the parameters allowed by the implementation for a 48 particular codec profile. 49 . Bind memory resources with the decode video session by calling 50 flink:vkBindVideoSessionMemoryKHR. 51 The video session cannot: be used until memory resources are allocated 52 and bound to it. 53 In order to determine the required memory sizes and heap types of the 54 device memory allocations, flink:vkGetVideoSessionMemoryRequirementsKHR 55 should: be called. 56 . Create one or more <<video-session-parameters-introduction,Session 57 Parameter objects>> for use across command buffer recording operations, 58 if required by the codec extension in use. 59 These objects must: be created against a <<video-session,video session>> 60 with the parameters required by the codec. 61 Each <<video-session-parameters-introduction,Session Parameter object>> 62 created is a child object of the associated <<video-session, Session 63 object>> and cannot: be bound in the command buffer with any other 64 <<video-session,Session Object>>. 65 66The recording of Video Decode Commands against a Vulkan command buffer 67consists of the following sequence: 68 69 . flink:vkCmdBeginVideoCodingKHR starts the recording of one or more Video 70 Decode operations in the command buffer. 71 For each Video Decode Command operation, a Video Session must: be bound 72 to the command buffer within this command. 73 This command establishes a Vulkan Video Decode Context that consists of 74 the bound Video Session Object, Session Parameters Object, and the 75 required Video Picture Resources. 76 The established Video Decode Context is in effect until the 77 flink:vkCmdEndVideoCodingKHR command is recorded. 78 If more Video Decode operations are to be required after the 79 flink:vkCmdEndVideoCodingKHR command, another Video Decode Context can: 80 be started with the flink:vkCmdBeginVideoCodingKHR command. 81 . flink:vkCmdDecodeVideoKHR specifies one or more compressed data buffers 82 to be decoded. 83 The slink:VkVideoDecodeInfoKHR parameters, and the codec extension 84 structures chained to this, specify the details of the decode operation. 85 . flink:vkCmdControlVideoCodingKHR records operations against the decoded 86 data, decoding device, or the Video Session state. 87 . flink:vkCmdEndVideoCodingKHR signals the end of the recording of the 88 Vulkan Video Decode Context, as established by 89 flink:vkCmdBeginVideoCodingKHR. 90 91In addition to the above, the following commands can: be recorded between 92flink:vkCmdBeginVideoCodingKHR and flink:vkCmdEndVideoCodingKHR: 93 94 * Query operations 95 * Global Memory Barriers 96 * Buffer Memory Barriers 97 * Image Memory Barriers (these must: be used to transition the Video 98 Picture Resources to the proper 99 ename:VK_IMAGE_LAYOUT_VIDEO_DECODE_DPB_KHR and 100 ename:VK_IMAGE_LAYOUT_VIDEO_DECODE_DST_KHR layouts). 101 * Pipeline Barriers 102 * Events 103 * Timestamps 104 * Device Groups (device mask) 105 106The following Video Decode related commands must: be recorded *outside* the 107Vulkan Video Decode Context established with the 108flink:vkCmdBeginVideoCodingKHR and flink:vkCmdEndVideoCodingKHR commands: 109 110 * Sparse Memory Binding 111 * Copy Commands 112 * Clear Commands 113 114 115[[video-picture-decode-modes]] 116=== Video Picture Decode Modes 117 118There are a few ways that the flink:vkCmdDecodeVideoKHR can be configured 119for the Video Picture Decode Operations, based on: 120 121 * if the <<decoded-output-picture, output resource>> would need to be used 122 as <<reference-picture,Reference Picture>> for subsequent decode 123 operations and; 124 * if <<dpb-slot,DPB Slots>> are required for use as 125 <<reference-picture,Reference Pictures>> indexes. 126 127[[video-picture-decode-no-reference-picture-no-slot-update]] 128The most basic Video Picture Decode operation with the 129flink:vkCmdDecodeVideoKHR command is to output the decoded pixel data 130without using any DPB <<reference-picture,Reference Pictures>> and without 131updating any <<dpb-slot,DPB Slot's>> indexes. 132 133In this case, the following slink:VkVideoDecodeInfoKHR parameters must: be 134set: 135 136 * slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->pPictureResource->imageViewBinding 137 must: be a valid slink:VkImageView. 138 This slink:VkImageView represents the <<decoded-output-picture, output 139 resource>> where the decoded pixels will be populated after a successful 140 decode operation. 141 * slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->slotIndex must: 142 be an invalid <<dpb-slot,DPB Slot>> index (-1) since the decoded picture 143 is not intended to be used as a reference picture with subsequent video 144 decode operations. 145 * The value of the slink:VkVideoDecodeInfoKHR::pname:referenceSlotCount 146 can: be `0` and slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots can: 147 be `NULL`. 148 * If slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots is not `NULL`, it 149 can: still have entries representing <<dpb-slot,DPB Slot>> indexes with 150 a <<video-session-dpb-slot-states, Valid Picture Reference>>. 151 The codec extension selects the actual use of the 152 <<reference-picture,Reference Pictures>> by referring to a 153 <<dpb-slot,DPB Slot>> index with a <<video-session-dpb-slot-states, 154 Valid Picture Reference>>. 155 156[[video-picture-decode-with-references-no-slot-update]] 157 158Video Picture Decode operations with the flink:vkCmdDecodeVideoKHR command, 159requiring one or more <<reference-picture,Reference Pictures>> for the 160predictions of the values of samples for the <<decoded-output-picture, 161decoded output picture>> would require <<dpb-slot,DPB Slots>> with 162<<video-session-dpb-slot-states, Valid Picture Reference>>. 163 164In this case, the following slink:VkVideoDecodeInfoKHR parameters must: be 165set: 166 167 * slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->pPictureResource->imageViewBinding 168 must: be a valid slink:VkImageView. 169 This slink:VkImageView represents the <<decoded-output-picture, output 170 resource>> where the decoded pixels will be populated after a successful 171 decode operation. 172 * slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->slotIndex must: 173 be an invalid <<dpb-slot,DPB Slot>> index (-1) since the decoded picture 174 is not intended to be used as a reference picture with subsequent video 175 decode operations. 176 * The value of the slink:VkVideoDecodeInfoKHR::pname:referenceSlotCount 177 must: not be `0` and slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots 178 should represent at least the number of the reference slots required for 179 the decode operation. 180 The codec extension selects the actual use of the 181 <<reference-picture,Reference Pictures>> by referring to a 182 <<dpb-slot,DPB Slot>> index with a <<video-session-dpb-slot-states, 183 Valid Picture Reference>>. 184 If the implementation does not use an opaque DPB, each <<dpb-slot,DPB 185 slot>> representing a <<reference-picture,reference picture>> must: 186 refer to a valid <<VkImageView,image view>>. 187 The <<VkImageView,image views>> must: represent the same 188 <<resources-images, image resources>> that were used to create the 189 <<reference-picture,reference picture>> for the corresponding 190 <<dpb-slot,DPB Slot>> index. 191 * slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots can: still have 192 entries representing <<dpb-slot,DPB Slot>> indexes with a 193 <<video-session-dpb-slot-states, Valid Picture Reference>>. 194 195After the flink:vkCmdDecodeVideoKHR operation is completed successfully, the 196slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->pPictureResource->imageViewBinding 197pixel data will be updated with the decoded content. 198The operation will not update any <<dpb-slot,DPB Slot>> with 199<<reference-picture,Reference Pictures>> data. 200However, any <<dpb-slot,DPB Slot>> activation, invalidation, or deactivation 201operations requested via slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots 202are still going to be performed. 203 204image::images/DecodeSessionDpbDecodeWithOutputToVkImageViewNoReferencePictureSlotUpdate.svg[align="center",title="Decoding a Frame to VkImageView without a slot update for a *Reference Picture*",opts="{imageopts}"] 205 206 207[[video-picture-decode-with-reference-picture-and-slot-update]] 208==== Video Picture Decode with a <<reference-picture,Reference Picture>> slot update and using optional <<reference-picture,Reference Pictures>> 209 210When it is known that the picture to be decoded will be used as a 211<<reference-picture,reference picture>> for subsequent decode operations, 212one of the available <<dpb-slot,DPB Slots>> needs to be selected for 213<<video-session-activating-dpb-slot, activation and update>> operations as 214part of the flink:vkCmdDecodeVideoKHR command. 215 216Based on whether a decode operation 217<<video-picture-decode-with-references-no-slot-update, with reference 218pictures>> or <<video-picture-decode-no-reference-picture-no-slot-update, 219without reference pictures>> is required, the flink:vkCmdDecodeVideoKHR 220should be configured with parameters as described in the previous sections. 221In addition, one of the available <<dpb-slot,DPB Slots>> must: be selected 222by the application, activated with resources and then set-up for an update 223with the decode operation. 224 225In this case, the following slink:VkVideoDecodeInfoKHR parameters must: be 226set: 227 228 * slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->pPictureResource->imageViewBinding 229 must: be a valid slink:VkImageView. 230 This slink:VkImageView represents the <<decoded-output-picture, output 231 resource>> where the decoded pixels will be populated after a successful 232 decode operation. 233 If the implementation does not use an opaque DPB, both the 234 <<decoded-output-picture, output>> and <<reference-picture,reference 235 picture>> resource coincide. 236 * slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->slotIndex must: 237 be a valid <<dpb-slot,DPB Slot>> index selected by the application, 238 based on the currently available slots. 239 * slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots can: still have 240 entries representing <<dpb-slot,DPB Slot>> indexes with a 241 <<video-session-dpb-slot-states, Valid Picture Reference>>. 242 243After the flink:vkCmdDecodeVideoKHR operation has completed successfully, 244the decoded content will be available in the resource provided for 245slink:VkVideoDecodeInfoKHR::pname:pSetupReferenceSlot->pPictureResource->imageViewBinding. 246In addition, this operation will update the selected <<dpb-slot,DPB Slot>> 247with <<reference-picture,Reference Pictures>> data. 248Any other <<dpb-slot,DPB Slot>> activation,invalidation, or deactivation 249operation requested via the 250slink:VkVideoDecodeInfoKHR::pname:pReferenceSlots are going to be performed 251as well. 252 253image::images/DecodeSessionDpbDecodeWithOutputToReferencePictureSlot.svg[align="center",title="Decoding a Frame to VkImageView with an update to a <<reference-picture,Reference Pictures>> DPB Slot",opts="{imageopts}"] 254 255 256=== Video Decode Command Buffer Commands 257 258[open,refpage='vkCmdDecodeVideoKHR',desc='Decode a frame',type='protos'] 259-- 260To decode a frame, call: 261 262include::{generated}/api/protos/vkCmdDecodeVideoKHR.txt[] 263 264 * pname:commandBuffer is the command buffer to be filled with this 265 function for decode frame command. 266 * pname:pFrameInfo is a pointer to a slink:VkVideoDecodeInfoKHR structure. 267 268include::{generated}/validity/protos/vkCmdDecodeVideoKHR.txt[] 269-- 270 271[open,refpage='VkVideoDecodeInfoKHR',desc='Structure specifying parameters of decoding a frame',type='structs'] 272-- 273The slink:VkVideoDecodeInfoKHR structure is defined as: 274 275include::{generated}/api/structs/VkVideoDecodeInfoKHR.txt[] 276 * pname:sType is the type of this structure. 277 * pname:pNext is `NULL` or a pointer to a structure extending this 278 structure. 279 All the codec specific structures related to each frame(picture 280 parameters, quantization matrix, etc.) must: be chained here and pass to 281 decode session with the function call flink:vkCmdDecodeVideoKHR. 282 * pname:flags is a bitmask of elink:VkVideoDecodeFlagBitsKHR specifying 283 decode flags, reserved for future versions of this specification. 284 * pname:codedOffset is the coded offset of the decode operations. 285 The purpose of this field is interpreted based on the codec extension. 286ifdef::VK_EXT_video_decode_h264[] 287 When decoding content in H.264 field mode, the pname:codedOffset 288 specifies the line or picture field's offset within the image. 289endif::VK_EXT_video_decode_h264[] 290 * pname:codedExtent is the coded size of the decode operations. 291 * pname:srcBuffer is the source buffer that holds the encoded bitstream. 292 * pname:srcBufferOffset is the buffer offset where the valid encoded 293 bitstream starts in srcBuffer. 294 It must: meet the alignment requirement 295 pname:minBitstreamBufferOffsetAlignment within 296 slink:VkVideoCapabilitiesKHR queried with the 297 flink:vkGetPhysicalDeviceVideoCapabilitiesKHR function. 298 * pname:srcBufferRange is the size of the srcBuffer with valid encoded 299 bitstream, starting from pname:srcBufferOffset. 300 It must: meet the alignment requirement 301 pname:minBitstreamBufferSizeAlignment within 302 slink:VkVideoCapabilitiesKHR queried with the 303 flink:vkGetPhysicalDeviceVideoCapabilitiesKHR function. 304 * pname:dstPictureResource is the destination 305 <<decoded-output-picture,Decoded Output Picture>> Resource. 306 * pname:pSetupReferenceSlot is `NULL` or a pointer to a 307 slink:VkVideoReferenceSlotKHR structure used for generating a DPB 308 reference slot and Picture Resource. 309 pname:pSetupReferenceSlot->slotIndex specifies the slot index number to 310 use as a target for producing the DPB data. 311 pname:slotIndex must: reference a valid entry as specified in 312 slink:VkVideoBeginCodingInfoKHR via the pname:pReferenceSlots within the 313 flink:vkCmdBeginVideoCodingKHR command that established the Vulkan Video 314 Decode Context for this command. 315 * pname:referenceSlotCount is the number of the DPB Reference Pictures 316 that will be used when this decoding operation is executing. 317 * pname:pReferenceSlots is a pointer to an array of 318 slink:VkVideoReferenceSlotKHR structures specifying the DPB Reference 319 pictures that will be used when this decoding operation is executing. 320 321include::{generated}/validity/structs/VkVideoDecodeInfoKHR.txt[] 322-- 323 324[open,refpage='VkVideoDecodeFlagBitsKHR',desc='Video Decode Command Flags',type='enums'] 325-- 326The flink:vkCmdDecodeVideoKHR flags are defined with the following 327enumeration: 328 329include::{generated}/api/enums/VkVideoDecodeFlagBitsKHR.txt[] 330 331 * ename:VK_VIDEO_DECODE_RESERVED_0_BIT_KHR The current version of the 332 specification has reserved this value for future use. 333-- 334 335[open,refpage='VkVideoDecodeFlagsKHR',desc='Bitmask specifying the video decode flink:vkCmdDecodeVideoKHR flags',type='flags'] 336-- 337include::{generated}/api/flags/VkVideoDecodeFlagsKHR.txt[] 338 339tlink:VkVideoDecodeFlagsKHR is a bitmask type for setting a mask of zero or 340more elink:VkVideoDecodeFlagBitsKHR. 341-- 342
