1// Copyright 2018-2021 The Khronos Group Inc. 2// 3// SPDX-License-Identifier: CC-BY-4.0 4 5[[video-encode-operations]] 6== Video Encode Operations 7 8Before the application can start recording Vulkan command buffers for the 9Video Encode Operations, it must: do the following, beforehand: 10 11 . Ensure that the implementation can encode the Video Content by querying 12 the supported codec operations and profiles using 13 flink:vkGetPhysicalDeviceQueueFamilyProperties2. 14 . By using flink:vkGetPhysicalDeviceVideoFormatPropertiesKHR and providing 15 one or more video profiles, choose the Vulkan formats supported by the 16 implementation. 17 The formats for <<input-encode-picture,input>> and 18 <<reference-picture,reference>> pictures must: be queried and chosen 19 separately. 20 Refer to the section on 21 <<supported-video-input-output-dpb-formats-enumeration,enumeration of 22 supported video formats>>. 23 . Before creating an image to be used as a video picture resource, obtain 24 the supported image creation parameters by querying with 25 flink:vkGetPhysicalDeviceFormatProperties2 and 26 flink:vkGetPhysicalDeviceImageFormatProperties2 using one of the 27 reported formats and adding slink:VkVideoProfilesKHR to the pname:pNext 28 chain of slink:VkFormatProperties2. 29 When querying the parameters with 30 flink:vkGetPhysicalDeviceImageFormatProperties2 for images targeting 31 <<input-encode-picture,input>> and <<reference-picture,reference (DPB)>> 32 pictures, the slink:VkPhysicalDeviceImageFormatInfo2::pname:usage field 33 should contain ename:VK_IMAGE_USAGE_VIDEO_ENCODE_SRC_BIT_KHR and 34 ename:VK_IMAGE_USAGE_VIDEO_ENCODE_DPB_BIT_KHR, respectively. 35 . Create none, some, or all of the required <<VkImage,images>> for the 36 <<input-encode-picture,input>> and <<reference-picture,reference>> 37 pictures. 38 More Video Picture Resources can: be created at some later point if 39 needed while processing the content to be encoded. 40 Also, if the size of the picture to be encoded is expected to change, 41 the images can: be created based on the maximum expected content size. 42 . Create the <<video-session,video session>> to be used for video encode 43 operations. 44 Before creating the Encode Video Session, the encode capabilities 45 should: be queried with flink:vkGetPhysicalDeviceVideoCapabilitiesKHR to 46 obtain the limits of the parameters allowed by the implementation for a 47 particular codec profile. 48 . Bind memory resources with the encode video session by calling 49 flink:vkBindVideoSessionMemoryKHR. 50 The video session cannot: be used until memory resources are allocated 51 and bound to it. 52 In order to determine the required memory sizes and heap types of the 53 device memory allocations, flink:vkGetVideoSessionMemoryRequirementsKHR 54 should: be called. 55 . Create one or more <<video-session-parameters-introduction,Session 56 Parameter objects>> for use across command buffer recording operations, 57 if required by the codec extension in use. 58 These objects must: be created against a <<video-session,video session>> 59 with the parameters required by the codec. 60 Each <<video-session-parameters-introduction,Session Parameter object>> 61 created is a child object of the associated <<video-session, Session 62 object>> and cannot: be bound in the command buffer with any other 63 <<video-session,Session Object>>. 64 65 66The recording of Video Encode Commands against a Vulkan Command Buffer 67consists of the following sequence: 68 69 . flink:vkCmdBeginVideoCodingKHR starts the recording of one or more Video 70 Encode operations in the command buffer. 71 For each Video Encode Command operation, a Video Session must: be bound 72 to the command buffer within this command. 73 This command establishes a Vulkan Video Encode Context that consists of 74 the bound Video Session Object, Session Parameters Object, and the 75 required Video Picture Resources. 76 The established Video Encode Context is in effect until the 77 flink:vkCmdEndVideoCodingKHR command is recorded. 78 If more Video Encode operations are to be required after the 79 flink:vkCmdEndVideoCodingKHR command, another Video Encode Context can: 80 be started with the flink:vkCmdBeginVideoCodingKHR command. 81 . flink:vkCmdEncodeVideoKHR specifies one or more frames to be encoded. 82 The slink:VkVideoEncodeInfoKHR parameters, and the codec extension 83 structures chained to this, specify the details of the encode operation. 84 . flink:vkCmdControlVideoCodingKHR records operations against the encoded 85 data, encoding device, or the Video Session state. 86 . flink:vkCmdEndVideoCodingKHR signals the end of the recording of the 87 Vulkan Video Encode Context, as established by 88 flink:vkCmdBeginVideoCodingKHR. 89 90In addition to the above, the following commands can: be recorded between 91flink:vkCmdBeginVideoCodingKHR and flink:vkCmdEndVideoCodingKHR: 92 93 * Query operations 94 * Global Memory Barriers 95 * Buffer Memory Barriers 96 * Image Memory Barriers (these must: be used to transition the Video 97 Picture Resources to the proper 98 ename:VK_IMAGE_LAYOUT_VIDEO_ENCODE_SRC_KHR and 99 ename:VK_IMAGE_LAYOUT_VIDEO_ENCODE_DPB_KHR layouts). 100 * Pipeline Barriers 101 * Events 102 * Timestamps 103 * Device Groups (device mask) 104 105The following Video Encode related commands must: be recorded *outside* the 106Vulkan Video Encode Context established with the 107flink:vkCmdBeginVideoCodingKHR and flink:vkCmdEndVideoCodingKHR commands: 108 109 * Sparse Memory Binding 110 * Copy Commands 111 * Clear Commands 112 113 114=== Video Encode Vulkan Command Buffer Commands 115 116[open,refpage='vkCmdEncodeVideoKHR',desc='Encode operation for bitstream generation',type='protos'] 117-- 118To launch an encode operation that results in bitstream generation, call: 119 120include::{generated}/api/protos/vkCmdEncodeVideoKHR.txt[] 121 122 * pname:commandBuffer is the command buffer to be filled with this 123 function for encoding to generate a bitstream. 124 * pname:pEncodeInfo is a pointer to a slink:VkVideoEncodeInfoKHR 125 structure. 126 127include::{generated}/validity/protos/vkCmdEncodeVideoKHR.txt[] 128-- 129 130[open,refpage='VkVideoEncodeInfoKHR',desc='Structure to chain codec-specific structures to',type='structs'] 131-- 132The slink:VkVideoEncodeInfoKHR structure is defined as: 133 134include::{generated}/api/structs/VkVideoEncodeInfoKHR.txt[] 135 * pname:sType is the type of this structure. 136 * pname:pNext is a pointer to a structure extending this structure. 137 A codec-specific extension structure must: be chained to specify what 138 bitstream unit to generate with this encode operation. 139 * pname:flags is a bitmask of elink:VkVideoEncodeFlagBitsKHR specifying 140 encode flags, and is reserved for future versions of this specification. 141 * pname:qualityLevel is the coding quality level of the encoding. 142 It is defined by the codec-specific extensions. 143 * pname:codedExtent is the coded size of the encode operations. 144 * pname:dstBitstreamBuffer is the buffer where the encoded bitstream 145 output will be produced. 146 * pname:dstBitstreamBufferOffset is the offset in the 147 pname:dstBitstreamBuffer where the encoded bitstream output will start. 148 pname:dstBitstreamBufferOffset's value must: be aligned to 149 slink:VkVideoCapabilitiesKHR::pname:minBitstreamBufferOffsetAlignment, 150 as reported by the implementation. 151 * pname:dstBitstreamBufferMaxRange is the maximum size of the 152 pname:dstBitstreamBuffer that can be used while the encoded bitstream 153 output is produced. 154 pname:dstBitstreamBufferMaxRange's value must: be aligned to 155 slink:VkVideoCapabilitiesKHR::pname:minBitstreamBufferSizeAlignment, as 156 reported by the implementation. 157 * pname:srcPictureResource is the Picture Resource of the 158 <<input-encode-picture,Input Picture>> to be encoded by the operation. 159 * pname:pSetupReferenceSlot is a pointer to a 160 slink:VkVideoReferenceSlotKHR structure used for generating a 161 reconstructed reference slot and Picture Resource. 162 pname:pSetupReferenceSlot->slotIndex specifies the slot index number to 163 use as a target for producing the Reconstructed (DPB) data. 164 pname:pSetupReferenceSlot must: be one of the entries provided in 165 slink:VkVideoBeginCodingInfoKHR via the pname:pReferenceSlots within the 166 flink:vkCmdBeginVideoCodingKHR command that established the Vulkan Video 167 Encode Context for this command. 168 * pname:referenceSlotCount is the number of Reconstructed Reference 169 Pictures that will be used when this encoding operation is executing. 170 * pname:pReferenceSlots is `NULL` or a pointer to an array of 171 slink:VkVideoReferenceSlotKHR structures that will be used when this 172 encoding operation is executing. 173 Each entry in pname:pReferenceSlots must: be one of the entries provided 174 in slink:VkVideoBeginCodingInfoKHR via the pname:pReferenceSlots within 175 the flink:vkCmdBeginVideoCodingKHR command that established the Vulkan 176 Video Encode Context for this command. 177 * pname:precedingExternallyEncodedBytes is the number of bytes externally 178 encoded for insertion in the active video encode session overall 179 bitstream prior to the bitstream that will be generated by the 180 implementation for this instance of sname:VkVideoEncodeInfoKHR. 181 Valid when slink:VkVideoEncodeRateControlInfoKHR::pname:rateControlMode 182 is not ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 183 The value provided is used to update the implementation's rate control 184 algorithm for the rate control layer this instance of 185 sname:VkVideoEncodeInfoKHR belongs to, by accounting for the bitrate 186 budget consumed by these externally encoded bytes. 187 See slink:VkVideoEncodeRateControlInfoKHR for additional information 188 about encode rate control. 189 190Multiple flink:vkCmdEncodeVideoKHR commands may: be recorded within a Vulkan 191Video Encode Context. 192The execution of each flink:vkCmdEncodeVideoKHR command will result in 193generating codec-specific bitstream units. 194These bitstream units are generated consecutively into the bitstream buffer 195specified in pname:dstBitstreamBuffer of slink:VkVideoEncodeInfoKHR within 196the flink:vkCmdBeginVideoCodingKHR command. 197The produced bitstream is the sum of all these bitstream units, including 198any padding between the bitstream units. 199Any bitstream padding must: be filled with data compliant to the codec 200standard so as not to cause any syntax errors during decoding of the 201bitstream units with the padding included. 202The range of the bitstream buffer written can: be queried via 203<<queries-video-encode-bitstream-buffer-range, video encode bitstream buffer 204range queries>>. 205 206include::{generated}/validity/structs/VkVideoEncodeInfoKHR.txt[] 207-- 208 209[open,refpage='VkVideoEncodeFlagBitsKHR',desc='Video Encode Command Flags',type='enums'] 210-- 211The flink:vkCmdEncodeVideoKHR flags are defined with the following 212enumeration: 213 214include::{generated}/api/enums/VkVideoEncodeFlagBitsKHR.txt[] 215 216 * ename:VK_VIDEO_ENCODE_RESERVED_0_BIT_KHR The current version of the 217 specification has reserved this value for future use. 218-- 219 220[open,refpage='VkVideoEncodeFlagsKHR',desc='Bitmask specifying the video encode flink:vkCmdEncodeVideoKHR flags',type='flags'] 221-- 222include::{generated}/api/flags/VkVideoEncodeFlagsKHR.txt[] 223 224tlink:VkVideoEncodeFlagsKHR is a bitmask type for setting a mask of zero or 225more elink:VkVideoEncodeFlagBitsKHR. 226-- 227 228[open,refpage='VkVideoEncodeRateControlInfoKHR',desc='Structure to set encode stream rate control parameters',type='structs'] 229-- 230In order to provide video encode stream rate control settings, add a 231slink:VkVideoEncodeRateControlInfoKHR structure to the pname:pNext chain of 232the slink:VkVideoCodingControlInfoKHR structure passed to the 233flink:vkCmdControlVideoCodingKHR command. 234 235The slink:VkVideoEncodeRateControlInfoKHR structure is defined as: 236 237include::{generated}/api/structs/VkVideoEncodeRateControlInfoKHR.txt[] 238 * pname:sType is the type of this structure. 239 * pname:pNext is `NULL` or a pointer to a structure extending this 240 structure. 241 * pname:flags is a bitmask of elink:VkVideoEncodeRateControlFlagBitsKHR 242 specifying encode rate control flags. 243 * pname:rateControlMode is a elink:VkVideoEncodeRateControlModeFlagBitsKHR 244 value specifying the encode stream rate control mode. 245 * pname:layerCount specifies the number of rate control layers in the 246 video encode stream. 247 * pname:pLayerConfigs is a pointer to an array of 248 slink:VkVideoEncodeRateControlLayerInfoKHR structures specifying the 249 rate control configurations of pname:layerCount rate control layers. 250 251A codec-specific extension structure for further encode stream rate control 252parameter settings may: be chained to slink:VkVideoEncodeRateControlInfoKHR. 253 254To ensure that the video session is properly initalized with stream-level 255rate control settings, the application must: call 256flink:vkCmdControlVideoCodingKHR with stream-level rate control settings at 257least once in execution order before the first flink:vkCmdEncodeVideoKHR 258command that is executed after video session reset. 259If not provided, default implementation-specific stream rate control 260settings will be used. 261 262Stream rate control settings can: also be re-initialized during an active 263video encoding session. 264The re-initialization takes effect whenever the 265slink:VkVideoEncodeRateControlInfoKHR structure is included in the 266pname:pNext chain of the slink:VkVideoCodingControlInfoKHR structure in the 267call to flink:vkCmdControlVideoCodingKHR, and only impacts 268flink:vkCmdEncodeVideoKHR operations that follow in execution order. 269 270include::{generated}/validity/structs/VkVideoEncodeRateControlInfoKHR.txt[] 271-- 272 273[open,refpage='VkVideoEncodeRateControlFlagsKHR',desc='Reserved for future use',type='flags'] 274-- 275include::{generated}/api/flags/VkVideoEncodeRateControlFlagsKHR.txt[] 276 277tname:VkVideoEncodeRateControlFlagsKHR is a bitmask type for setting a mask, 278but currently reserved for future use. 279-- 280 281[open,refpage='VkVideoEncodeRateControlFlagBitsKHR',desc='Reserved for future use',type='enums'] 282-- 283include::{generated}/api/enums/VkVideoEncodeRateControlFlagBitsKHR.txt[] 284 285elink:VkVideoEncodeRateControlFlagBitsKHR defines bits which may be set in a 286tlink:VkVideoEncodeRateControlFlagsKHR value, but is currently unused. 287 288-- 289 290 291[open,refpage='VkVideoEncodeRateControlModeFlagBitsKHR',desc='Video encode rate control modes',type='enums'] 292-- 293The rate control modes are defined with the following enums: 294 295include::{generated}/api/enums/VkVideoEncodeRateControlModeFlagBitsKHR.txt[] 296 297 * ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR for disabling rate 298 control. 299 * ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_CBR_BIT_KHR for constant bitrate 300 rate control mode. 301 * ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR for variable bitrate 302 rate control mode. 303-- 304 305[open,refpage='VkVideoEncodeRateControlLayerInfoKHR',desc='Structure to set encode per-layer rate control parameters',type='structs'] 306-- 307The slink:VkVideoEncodeRateControlLayerInfoKHR structure is defined as: 308 309include::{generated}/api/structs/VkVideoEncodeRateControlLayerInfoKHR.txt[] 310 * pname:sType is the type of this structure. 311 * pname:pNext is a pointer to a structure extending this structure. 312 * pname:averageBitrate is the average bitrate in bits/second. 313 Valid when rate control mode is not 314 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 315 * pname:maxBitrate is the peak bitrate in bits/second. 316 Valid when rate control mode is 317 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_VBR_BIT_KHR. 318 * pname:frameRateNumerator is the numerator of the frame rate. 319 Valid when rate control mode is not 320 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 321 * pname:frameRateDenominator is the denominator of the frame rate. 322 Valid when rate control mode is not 323 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 324 * pname:virtualBufferSizeInMs is the leaky bucket model virtual buffer 325 size in milliseconds, with respect to peak bitrate. 326 Valid when rate control mode is not 327 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 328 For example, virtual buffer size is (pname:virtualBufferSizeInMs * 329 pname:maxBitrate / 1000). 330 * pname:initialVirtualBufferSizeInMs is the initial occupancy in 331 milliseconds of the virtual buffer in the leaky bucket model. 332 Valid when the rate control mode is not 333 ename:VK_VIDEO_ENCODE_RATE_CONTROL_MODE_NONE_BIT_KHR. 334 335A codec-specific structure specifying additional per-layer rate control 336settings must: be chained to sname:VkVideoEncodeRateControlLayerInfoKHR. 337If multiple rate control layers are enabled 338(slink:VkVideoEncodeRateControlInfoKHR::pname:layerCount is greater than 1), 339then the chained codec-specific extension structure also identifies the 340specific video coding layer its parent 341sname:VkVideoEncodeRateControlLayerInfoKHR applies to. 342If multiple rate control layers are enabled, the number of rate control 343layers must: match the number of video coding layers. 344The specification for an encode codec-specific extension would describe how 345multiple video coding layers are enabled for the corresponding codec. 346 347Per-layer rate control settings for all enabled rate control layers must: be 348initialized or re-initialized whenever stream rate control settings are 349provided via slink:VkVideoEncodeRateControlInfoKHR. 350This is done by specifying settings for all enabled rate control layers in 351slink:VkVideoEncodeRateControlInfoKHR::pname:pLayerConfigs. 352 353To adjust rate control settings for an individual layer at runtime, add a 354slink:VkVideoEncodeRateControlLayerInfoKHR structure to the pname:pNext 355chain of the slink:VkVideoCodingControlInfoKHR structure passed to the 356flink:vkCmdControlVideoCodingKHR command. 357This adjustment only impacts the specified layer without impacting the rate 358control settings or implementation rate control algorithm behavior for any 359other enabled rate control layers. 360The adjustment takes effect whenever the corresponding 361flink:vkCmdControlVideoCodingKHR is executed, and only impacts 362flink:vkCmdEncodeVideoKHR operations pertaining to the corresponding video 363coding layer that follow in execution order. 364 365It is possible for an application to enable multiple video coding layers 366(via codec-specific extensions to encoding operations) while only enabling a 367single layer of rate control for the entire video stream. 368To achieve this, pname:layerCount in slink:VkVideoEncodeRateControlInfoKHR 369must: be set to 1, and the single slink:VkVideoEncodeRateControlLayerInfoKHR 370provided in pname:pLayerConfigs would apply to all encoded segments of the 371video stream, regardless of which codec-defined video coding layer they 372belong to. 373In this case, the implementation decides bitrate distribution across video 374coding layers (if applicable to the specified stream rate control mode). 375 376include::{generated}/validity/structs/VkVideoEncodeRateControlLayerInfoKHR.txt[] 377-- 378 379