1// Copyright 2018-2021 The Khronos Group Inc. 2// 3// SPDX-License-Identifier: CC-BY-4.0 4 5[[video-encode-operations]] 6== Video Encode Operations 7 8Before the application can start recording Vulkan command buffers for the 9Video Encode Operations, it must: do the following, beforehand: 10 11 . Ensure that the implementation can encode the Video Content by querying 12 the supported codec operations and profiles using 13 flink:vkGetPhysicalDeviceQueueFamilyProperties2. 14 . By using flink:vkGetPhysicalDeviceVideoFormatPropertiesKHR and providing 15 one or more video profiles, choose the Vulkan formats supported by the 16 implementation. 17 The formats for <<input-encode-picture,input>> and 18 <<reference-picture,reference>> pictures must: be queried and chosen 19 separately. 20 Refer to the section on 21 <<supported-video-input-output-dpb-formats-enumeration,enumeration of 22 supported video formats>>. 23 . Before creating an image to be used as a video picture resource, obtain 24 the supported image creation parameters by querying with 25 flink:vkGetPhysicalDeviceFormatProperties2 and 26 flink:vkGetPhysicalDeviceImageFormatProperties2 using one of the 27 reported formats and adding slink:VkVideoProfilesKHR to the pname:pNext 28 chain of slink:VkFormatProperties2. 29 When querying the parameters with 30 flink:vkGetPhysicalDeviceImageFormatProperties2 for images targeting 31 <<input-encode-picture,input>> and <<reference-picture,reference (DPB)>> 32 pictures, the slink:VkPhysicalDeviceImageFormatInfo2::pname:usage field 33 should contain ename:VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR and 34 ename:VK_IMAGE_USAGE_VIDEO_ENCODE_DPB_BIT_KHR, respectively. 35 . Create none, some, or all of the required <<VkImage,images>> for the 36 <<input-encode-picture,input>> and <<reference-picture,reference>> 37 pictures. 38 More Video Picture Resources can: be created at some later point if 39 needed while processing the content to be encoded. 40 Also, if the size of the picture to be encoded is expected to change, 41 the images can: be created based on the maximum expected content size. 42 . Create the <<video-session,video session>> to be used for video encode 43 operations. 44 Before creating the Encode Video Session, the encode capabilities 45 should: be queried with flink:vkGetPhysicalDeviceVideoCapabilitiesKHR to 46 obtain the limits of the parameters allowed by the implementation for a 47 particular codec profile. 48 . Bind memory resources with the encode video session by calling 49 flink:vkBindVideoSessionMemoryKHR. 50 The video session cannot: be used until memory resources are allocated 51 and bound to it. 52 In order to determine the required memory sizes and heap types of the 53 device memory allocations, flink:vkGetVideoSessionMemoryRequirementsKHR 54 should: be called. 55 . Create one or more <<video-session-parameters-introduction,Session 56 Parameter objects>> for use across command buffer recording operations, 57 if required by the codec extension in use. 58 These objects must: be created against a <<video-session,video session>> 59 with the parameters required by the codec. 60 Each <<video-session-parameters-introduction,Session Parameter object>> 61 created is a child object of the associated <<video-session, Session 62 object>> and cannot: be bound in the command buffer with any other 63 <<video-session,Session Object>>. 64 65 66The recording of Video Encode Commands against a Vulkan Command Buffer 67consists of the following sequence: 68 69 . flink:vkCmdBeginVideoCodingKHR starts the recording of one or more Video 70 Encode operations in the command buffer. 71 For each Video Encode Command operation, a Video Session must: be bound 72 to the command buffer within this command. 73 This command establishes a Vulkan Video Encode Context that consists of 74 the bound Video Session Object, Session Parameters Object, and the 75 required Video Picture Resources. 76 The established Video Encode Context is in effect until the 77 flink:vkCmdEndVideoCodingKHR command is recorded. 78 If more Video Encode operations are to be required after the 79 flink:vkCmdEndVideoCodingKHR command, another Video Encode Context can: 80 be started with the flink:vkCmdBeginVideoCodingKHR command. 81 . flink:vkCmdEncodeVideoKHR specifies one or more frames to be encoded. 82 The slink:VkVideoEncodeInfoKHR parameters, and the codec extension 83 structures chained to this, specify the details of the encode operation. 84 . flink:vkCmdControlVideoCodingKHR records operations against the encoded 85 data, encoding device, or the Video Session state. 86 . flink:vkCmdEndVideoCodingKHR signals the end of the recording of the 87 Vulkan Video Encode Context, as established by 88 flink:vkCmdBeginVideoCodingKHR. 89 90In addition to the above, the following commands can: be recorded between 91flink:vkCmdBeginVideoCodingKHR and flink:vkCmdEndVideoCodingKHR: 92 93 * Query operations 94 * Global Memory Barriers 95 * Buffer Memory Barriers 96 * Image Memory Barriers (these must: be used to transition the Video 97 Picture Resources to the proper 98 ename:VK_IMAGE_LAYOUT_VIDEO_ENCODE_SRC_KHR and 99 ename:VK_IMAGE_LAYOUT_VIDEO_ENCODE_DPB_KHR layouts). 100 * Pipeline Barriers 101 * Events 102 * Timestamps 103 * Device Groups (device mask) 104 105The following Video Encode related commands must: be recorded *outside* the 106Vulkan Video Encode Context established with the 107flink:vkCmdBeginVideoCodingKHR and flink:vkCmdEndVideoCodingKHR commands: 108 109 * Sparse Memory Binding 110 * Copy Commands 111 * Clear Commands 112 113 114=== Video Encode Vulkan Command Buffer Commands 115 116[open,refpage='vkCmdEncodeVideoKHR',desc='Encode operation for bitstream generation',type='protos'] 117-- 118To launch an encode operation that results in bitstream generation, call: 119 120include::{generated}/api/protos/vkCmdEncodeVideoKHR.txt[] 121 122 * pname:commandBuffer is the command buffer to be filled with this 123 function for encoding to generate a bitstream. 124 * pname:pEncodeInfo is a pointer to a slink:VkVideoEncodeInfoKHR 125 structure. 126 127include::{generated}/validity/protos/vkCmdEncodeVideoKHR.txt[] 128-- 129 130[open,refpage='VkVideoEncodeInfoKHR',desc='Structure to chain codec-specific structures to',type='structs'] 131-- 132The slink:VkVideoEncodeInfoKHR structure is defined as: 133 134include::{generated}/api/structs/VkVideoEncodeInfoKHR.txt[] 135 * pname:sType is the type of this structure. 136 * pname:pNext is a pointer to a structure extending this structure. 137 A codec-specific extension structure must: be chained to specify what 138 bitstream unit to generate with this encode operation. 139 * pname:flags is a bitmask of elink:VkVideoEncodeFlagBitsKHR specifying 140 encode flags, and is reserved for future versions of this specification. 141 * pname:qualityLevel is the coding quality level of the encoding. 142 It is defined by the codec-specific extensions. 143 * pname:codedExtent is the coded size of the encode operations. 144 * pname:dstBitstreamBuffer is the buffer where the encoded bitstream 145 output will be produced. 146 * pname:dstBitstreamBufferOffset is the offset in the 147 pname:dstBitstreamBuffer where the encoded bitstream output will start. 148 pname:dstBitstreamBufferOffset's value must: be aligned to 149 slink:VkVideoCapabilitiesKHR::pname:minBitstreamBufferOffsetAlignment, 150 as reported by the implementation. 151 * pname:dstBitstreamBufferMaxRange is the maximum size of the 152 pname:dstBitstreamBuffer that can be used while the encoded bitstream 153 output is produced. 154 pname:dstBitstreamBufferMaxRange's value must: be aligned to 155 slink:VkVideoCapabilitiesKHR::pname:minBitstreamBufferSizeAlignment, as 156 reported by the implementation. 157 * pname:srcPictureResource is the Picture Resource of the 158 <<input-encode-picture,Input Picture>> to be encoded by the operation. 159 * pname:pSetupReferenceSlot is a pointer to a 160 slink:VkVideoReferenceSlotKHR structure used for generating a 161 reconstructed reference slot and Picture Resource. 162 pname:pSetupReferenceSlot->slotIndex specifies the slot index number to 163 use as a target for producing the Reconstructed (DPB) data. 164 pname:pSetupReferenceSlot must: be one of the entries provided in 165 slink:VkVideoBeginCodingInfoKHR via the pname:pReferenceSlots within the 166 flink:vkCmdBeginVideoCodingKHR command that established the Vulkan Video 167 Encode Context for this command. 168 * pname:referenceSlotCount is the number of Reconstructed Reference 169 Pictures that will be used when this encoding operation is executing. 170 * pname:pReferenceSlots is `NULL` or a pointer to an array of 171 slink:VkVideoReferenceSlotKHR structures that will be used when this 172 encoding operation is executing. 173 Each entry in pname:pReferenceSlots must: be one of the entries provided 174 in slink:VkVideoBeginCodingInfoKHR via the pname:pReferenceSlots within 175 the flink:vkCmdBeginVideoCodingKHR command that established the Vulkan 176 Video Encode Context for this command. 177 178Multiple flink:vkCmdEncodeVideoKHR commands may: be recorded within a Vulkan 179Video Encode Context. 180The execution of each flink:vkCmdEncodeVideoKHR command will result in 181generating codec-specific bitstream units. 182These bitstream units are generated consecutively into the bitstream buffer 183specified in pname:dstBitstreamBuffer of slink:VkVideoEncodeInfoKHR within 184the flink:vkCmdBeginVideoCodingKHR command. 185The produced bitstream is the sum of all these bitstream units, including 186any padding between the bitstream units. 187Any bitstream padding must: be filled with data compliant to the codec 188standard so as not to cause any syntax errors during decoding of the 189bitstream units with the padding included. 190The range of the bitstream buffer written can: be queried via 191<<queries-video-encode-bitstream-buffer-range, video encode bitstream buffer 192range queries>>. 193 194include::{generated}/validity/structs/VkVideoEncodeInfoKHR.txt[] 195-- 196 197[open,refpage='VkVideoEncodeFlagBitsKHR',desc='Video Encode Command Flags',type='enums'] 198-- 199The flink:vkCmdEncodeVideoKHR flags are defined with the following 200enumeration: 201 202include::{generated}/api/enums/VkVideoEncodeFlagBitsKHR.txt[] 203 204 * ename:VK_VIDEO_ENCODE_RESERVED_0_BIT_KHR The current version of the 205 specification has reserved this value for future use. 206-- 207 208[open,refpage='VkVideoEncodeFlagsKHR',desc='Bitmask specifying the video encode flink:vkCmdEncodeVideoKHR flags',type='flags'] 209-- 210include::{generated}/api/flags/VkVideoEncodeFlagsKHR.txt[] 211 212tlink:VkVideoEncodeFlagsKHR is a bitmask type for setting a mask of zero or 213more elink:VkVideoEncodeFlagBitsKHR. 214-- 215 216[open,refpage='VkVideoEncodeRateControlInfoKHR',desc='Structure to set encode rate control parameters',type='structs'] 217-- 218The slink:VkVideoEncodeRateControlInfoKHR structure is defined as: 219 220include::{generated}/api/structs/VkVideoEncodeRateControlInfoKHR.txt[] 221 * pname:sType is the type of this structure. 222 * pname:pNext is `NULL` or a pointer to a structure extending this 223 structure. 224 * pname:flags is a bitmask of 225 elink:VkVideoEncodeRateControlModeFlagBitsKHR specifying encode rate 226 control flags. 227 * pname:rateControlMode is a elink:VkVideoEncodeRateControlModeFlagBitsKHR 228 value specifying the encode rate control mode. 229 * pname:averageBitrate is the average bitrate in bits/second. 230 Valid when rate control is not 231 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 232 * pname:peakToAverageBitrateRatio is the peak bitrate to average bitrate 233 in percentage. 234 Valid when rate control is 235 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR. 236 * pname:frameRateNumerator is the numerator of the frame rate. 237 Valid when rate control is not 238 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 239 * pname:frameRateDenominator is the denominator of the frame rate. 240 Valid when rate control is not 241 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 242 * pname:virtualBufferSizeInMs is the leaky bucket model virtual buffer 243 size in milliseconds, with respect to peak bitrate. 244 Valid when rate control is not 245 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 246 For example, virtual buffer size is (pname:virtualBufferSizeInMs * 247 pname:peakToAverageBitrateRatio * pname:averageBitrate / 100000). 248 249A codec-specific extension structure for further rate control parameter 250settings may: be chained to slink:VkVideoEncodeRateControlInfoKHR. 251 252include::{generated}/validity/structs/VkVideoEncodeRateControlInfoKHR.txt[] 253-- 254 255[open,refpage='VkVideoEncodeRateControlModeFlagBitsKHR',desc='Video encode rate control modes',type='enums'] 256-- 257The rate control modes are defined with the following enums: 258 259include::{generated}/api/enums/VkVideoEncodeRateControlModeFlagBitsKHR.txt[] 260 261 * ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR for disabling rate 262 control. 263 * ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR for constant bitrate 264 rate control mode. 265 * ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR for variable bitrate 266 rate control mode. 267-- 268 269[open,refpage='VkVideoEncodeRateControlModeFlagsKHR',desc='Bitmask specifying the video encode flink:vkCmdControlVideoCodingKHR flags',type='flags'] 270-- 271include::{generated}/api/flags/VkVideoEncodeRateControlModeFlagsKHR.txt[] 272 273tlink:VkVideoEncodeRateControlModeFlagsKHR is a bitmask type for setting a 274mask of zero or more elink:VkVideoEncodeRateControlModeFlagBitsKHR. 275-- 276 277[open,refpage='VkVideoEncodeRateControlFlagBitsKHR',desc='Video Encode Rate Control Flags',type='enums'] 278-- 279The flink:vkCmdControlVideoCodingKHR flags are defined with the following 280enumeration: 281 282include::{generated}/api/enums/VkVideoEncodeRateControlFlagBitsKHR.txt[] 283 284 * ename:VK_VIDEO_ENCODE_RESERVED_0_BIT_KHR The current version of the 285 specification has reserved this value for future use. 286-- 287 288[open,refpage='VkVideoEncodeRateControlFlagsKHR',desc='Bitmask specifying the video encode flink:vkCmdControlVideoCodingKHR flags',type='flags'] 289-- 290include::{generated}/api/flags/VkVideoEncodeRateControlFlagsKHR.txt[] 291 292tlink:VkVideoEncodeRateControlFlagsKHR is a bitmask type for setting a mask 293of zero or more elink:VkVideoEncodeRateControlFlagBitsKHR. 294-- 295