1## General Information 2 3### Scope of this document 4 5This document is aimed to provide information about the v4l2\_codec2 project. 6The target readers of this document are the developers and following maintainers 7of this project, and the partners who are willing to use the V4L2 components. 8 9### Introduction 10 11v4l2\_codec2 project provides a component implementation of Codec2 framework, 12the next-generation codec framework. The component implementation delegates the 13request to the driver via the V4L2 API. 14 15## Quick Start Guide 16 17### Prerequisites 18 19* Gralloc support for graphic buffer allocation 20* ION or Gralloc support for linear buffer allocation 21* Kernel v5.3+ or [this patch][1] backported for buffer identification 22* [V4L2 stateful decoding API](https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-decoder.html) 23* [V4L2 encoding API](https://www.kernel.org/doc/html/latest/userspace-api/media/v4l/dev-encoder.html) 24* Widevine DRM for secure playback 25 26### Enable V4L2 Components 27 28Install the build package and files, and set the parameters in device.mk 29 30```makefile 31# Add the folder to the namespace. 32PRODUCT_SOONG_NAMESPACES += external/v4l2_codec2 33 34# Add the build target. 35PRODUCT_PACKAGES += \ 36 android.hardware.media.c2@1.0-service-v4l2 \ 37 libc2plugin_store 38 39# If a customized allocator is needed, then add this package. 40# See more detail at "Customized allocator" section. 41PRODUCT_PACKAGES += \ 42 libv4l2_codec2_vendor_allocator 43 44# Install media_codecs_c2.xml. 45# The destination is: /vendor/etc/media_codecs_c2.xml 46PRODUCT_COPY_FILES += \ 47 <path_to_file>:$(TARGET_COPY_OUT_VENDOR)/etc/media_codecs_c2.xml 48 49# Set the customized property of v4l2_codec2, including: 50# - The maximum concurrent instances for decoder/encoder. 51# It should be the same as "concurrent-instances" at media_codec_c2.xml. 52PRODUCT_PROPERTY_OVERRIDES += \ 53 ro.vendor.v4l2_codec2.decode_concurrent_instances=8 \ 54 ro.vendor.v4l2_codec2.encode_concurrent_instances=8 55 56# Codec2.0 poolMask: 57# ION(16) 58# BUFFERQUEUE(18) 59# BLOB(19) 60# V4L2_BUFFERQUEUE(20) 61# V4L2_BUFFERPOOL(21) 62# SECURE_LINEAR(22) 63# SECURE_GRAPHIC(23) 64# 65# For linear buffer allocation: 66# If ION is chosen, then the mask should be 0xf50000 67# If BLOB is chosen, then the mask should be 0xfc0000 68PRODUCT_PROPERTY_OVERRIDES += \ 69 debug.stagefright.c2-poolmask=0xf50000 70 71# Install extended policy for codec2. 72# The destination is: /vendor/etc/seccomp_policy/codec2.vendor.ext.policy 73PRODUCT_COPY_FILES += \ 74 <path_to_policy>:$(TARGET_COPY_OUT_VENDOR)/etc/seccomp_policy/codec2.vendor.ext.policy 75``` 76 77Enable codec2 hidl in manifest.xml 78 79```xml 80<manifest version="1.0" type="device"> 81 <hal format="hidl"> 82 <name>android.hardware.media.c2</name> 83 <transport>hwbinder</transport> 84 <version>1.0</version> 85 <interface> 86 <name>IComponentStore</name> 87 <instance>default</instance> 88 </interface> 89 <interface> 90 <name>IConfigurable</name> 91 <instance>default</instance> 92 </interface> 93 </hal> 94</manifest> 95``` 96 97Add decode and encode components in media\_codecs\_c2.xml 98 99```xml 100<?xml version="1.0" encoding="utf-8" ?> 101<MediaCodecs> 102 <Encoders> 103 <MediaCodec name="c2.v4l2.avc.encoder" type="video/avc"> 104 <Limit name="size" min="32x32" max="1920x1088" /> 105 <Limit name="alignment" value="2x2" /> 106 <Limit name="block-size" value="16x16" /> 107 <Limit name="blocks-per-second" range="1-244800" /> 108 <Limit name="bitrate" range="1-12000000" /> 109 <Limit name="concurrent-instances" max="8" /> 110 <Limit name="performance-point-1280x720" range="30-30" /> 111 </MediaCodec> 112 113 <MediaCodec name="c2.v4l2.vp8.encoder" type="video/x-vnd.on2.vp8"> 114 <Limit name="size" min="32x32" max="1920x1088" /> 115 <Limit name="alignment" value="2x2" /> 116 <Limit name="block-size" value="16x16" /> 117 <Limit name="blocks-per-second" range="1-244800" /> 118 <Limit name="bitrate" range="1-12000000" /> 119 <Limit name="concurrent-instances" max="8" /> 120 <Limit name="performance-point-1280x720" range="30-30" /> 121 </MediaCodec> 122 123 <MediaCodec name="c2.v4l2.vp9.encoder" type="video/x-vnd.on2.vp9"> 124 <Limit name="size" min="32x32" max="1920x1088" /> 125 <Limit name="alignment" value="2x2" /> 126 <Limit name="block-size" value="16x16" /> 127 <Limit name="blocks-per-second" range="1-244800" /> 128 <Limit name="bitrate" range="1-12000000" /> 129 <Limit name="concurrent-instances" max="8" /> 130 <Limit name="performance-point-1280x720" range="30-30" /> 131 </MediaCodec> 132 </Encoders> 133 134 <Decoders> 135 <MediaCodec name="c2.v4l2.avc.decoder" type="video/avc" > 136 <Limit name="size" min="16x16" max="4096x4096" /> 137 <Limit name="alignment" value="2x2" /> 138 <Limit name="block-size" value="16x16" /> 139 <Limit name="blocks-per-second" min="1" max="1879200" /> 140 <Limit name="bitrate" range="1-62500000" /> 141 <Limit name="concurrent-instances" max="8" /> 142 <Limit name="performance-point-3840x2160" range="30-30" /> 143 <Feature name="adaptive-playback" /> 144 </MediaCodec> 145 146 <MediaCodec name="c2.v4l2.vp8.decoder" type="video/x-vnd.on2.vp8" > 147 <Limit name="size" min="16x16" max="4096x4096" /> 148 <Limit name="alignment" value="2x2" /> 149 <Limit name="block-size" value="16x16" /> 150 <Limit name="blocks-per-second" min="1" max="1984500" /> 151 <Limit name="bitrate" range="1-62500000" /> 152 <Limit name="concurrent-instances" max="8" /> 153 <Limit name="performance-point-3840x2160" range="30-30" /> 154 <Feature name="adaptive-playback" /> 155 </MediaCodec> 156 157 <MediaCodec name="c2.v4l2.vp9.decoder" type="video/x-vnd.on2.vp9" > 158 <Limit name="size" min="16x16" max="4096x4096" /> 159 <Limit name="alignment" value="2x2" /> 160 <Limit name="block-size" value="16x16" /> 161 <Limit name="blocks-per-second" min="1" max="2073600" /> 162 <Limit name="bitrate" range="1-62500000" /> 163 <Limit name="concurrent-instances" max="8" /> 164 <Limit name="performance-point-3840x2160" range="30-30" /> 165 <Feature name="adaptive-playback" /> 166 </MediaCodec> 167 168 <MediaCodec name="c2.v4l2.avc.decoder.secure" type="video/avc" > 169 <Limit name="size" min="16x16" max="4096x4096" /> 170 <Limit name="alignment" value="2x2" /> 171 <Limit name="block-size" value="16x16" /> 172 <Limit name="blocks-per-second" min="1" max="1879200" /> 173 <Limit name="bitrate" range="1-62500000" /> 174 <Limit name="concurrent-instances" max="8" /> 175 <Limit name="performance-point-3840x2160" range="30-30" /> 176 <Feature name="adaptive-playback" /> 177 <Feature name="secure-playback" required="true" /> 178 </MediaCodec> 179 180 <MediaCodec name="c2.v4l2.vp8.decoder.secure" type="video/x-vnd.on2.vp8" > 181 <Limit name="size" min="16x16" max="4096x4096" /> 182 <Limit name="alignment" value="2x2" /> 183 <Limit name="block-size" value="16x16" /> 184 <Limit name="blocks-per-second" min="1" max="1984500" /> 185 <Limit name="bitrate" range="1-62500000" /> 186 <Limit name="concurrent-instances" max="8" /> 187 <Limit name="performance-point-3840x2160" range="30-30" /> 188 <Feature name="adaptive-playback" /> 189 <Feature name="secure-playback" required="true" /> 190 </MediaCodec> 191 192 <MediaCodec name="c2.v4l2.vp9.decoder.secure" type="video/x-vnd.on2.vp9" > 193 <Limit name="size" min="16x16" max="4096x4096" /> 194 <Limit name="alignment" value="2x2" /> 195 <Limit name="block-size" value="16x16" /> 196 <Limit name="blocks-per-second" min="1" max="2073600" /> 197 <Limit name="bitrate" range="1-62500000" /> 198 <Limit name="concurrent-instances" max="8" /> 199 <Limit name="performance-point-3840x2160" range="30-30" /> 200 <Feature name="adaptive-playback" /> 201 <Feature name="secure-playback" required="true" /> 202 </MediaCodec> 203 </Decoders> 204</MediaCodecs> 205``` 206 207Set SELinux file policy in sepolicy/file\_contexts 208 209``` 210/vendor/bin/hw/android\.hardware\.media\.c2@1\.0-service-v4l2(.*)? u:object_r:mediacodec_exec:s0 211``` 212 213Add additional permission in codec2.vendor.ext.policy 214 215``` 216_llseek: 1 217epoll_create1: 1 218epoll_ctl: 1 219epoll_pwait: 1 220eventfd2: 1 221fstat64: 1 222fstatat64: 1 223fstatfs64: 1 224getcwd: 1 225getdents64: 1 226getuid32: 1 227mmap2: 1 228pselect6: 1 229statfs64: 1 230sysinfo: 1 231ugetrlimit: 1 232``` 233 234Set file permission in ueventd.rc 235 236``` 237/dev/video* 0600 media media 238``` 239 240### Customized Allocator 241 242The default allocator of the decoder's output buffer is C2AllocatorGralloc. 243However, C2AllocatorGralloc might not fit the requirement for secure playback. 244In this case, we can implement a customized C2Allocator, and load it at 245run-time. Please create a library `libv4l2_codec2_vendor_allocator`, and export 246a function `CreateVendorAllocator() `for creating the customized C2Allocator. 247Here is an example below. 248 249#### Example of Android.bp 250 251```json 252cc_library_shared { 253 name: "libv4l2_codec2_vendor_allocator", 254 vendor: true, 255 256 defaults: [ 257 "libcodec2-impl-defaults", 258 ], 259 260 srcs: [ 261 "C2VendorAllocatorFactory.cpp", 262 ], 263 264 shared_libs: [ 265 "libc2plugin_store", 266 ], 267} 268``` 269 270#### Example of C2VendorAllocatorFactory.cpp 271 272```cpp 273//#define LOG_NDEBUG 0 274#define LOG_TAG "C2VendorAllocatorFactory" 275 276#include <C2AllocatorGralloc.h> 277#include <utils/Log.h> 278#include <v4l2_codec2/plugin_store/V4L2AllocatorId.h> 279 280namespace android { 281 282::C2Allocator* CreateVendorAllocator(::C2Allocator::id_t allocatorId) { 283 ALOGV("%s(%d)", __func__, allocatorId); 284 285 // Change to create vendor-customized implementation. 286 switch (allocatorId) { 287 case V4L2AllocatorId::V4L2_BUFFERQUEUE: 288 return new C2AllocatorGralloc(V4L2AllocatorId::V4L2_BUFFERQUEUE, true); 289 case V4L2AllocatorId::V4L2_BUFFERPOOL: 290 return new C2AllocatorGralloc(V4L2AllocatorId::V4L2_BUFFERPOOL, true); 291 case V4L2AllocatorId::SECURE_LINEAR: 292 return new C2AllocatorGralloc(V4L2AllocatorId::SECURE_LINEAR, true); 293 case V4L2AllocatorId::SECURE_GRAPHIC: 294 return new C2AllocatorGralloc(V4L2AllocatorId::SECURE_GRAPHIC, true); 295 default: 296 ALOGE("%s(): Unknown allocator ID: %d", __func__, allocatorId); 297 } 298 return nullptr; 299} 300 301} // namespace android 302 303extern "C" ::C2Allocator* CreateAllocator(::C2Allocator::id_t allocatorId) { 304 return ::android::CreateVendorAllocator(allocatorId); 305} 306``` 307 308## V4L2 Encoder 309 310### Supported Codecs 311 312Currently the V4L2 encoder has support for the H.264, VP8 and VP9 codecs. Codec 313selection can be done by selecting the encoder with the appropriate name. 314 315- H26: *c2.v4l2.avc.encoder* 316- VP8: *c2.v4l2.vp8.encoder* 317- VP9: *c2.v4l2.vp9.encoder* 318 319### Supported Parameters: 320 321The following parameters are static and can not be changed at run-time: 322 323- *C2_PARAMKEY_PICTURE_SIZE*: This parameter can be used to configure the 324resolution of the encoded video stream. 325- *C2_PARAMKEY_PROFILE_LEVEL*: This parameter can be used to adjust the desired 326profile level. When using the H.264 codec the profile level might be adjusted to 327conform to the minimum requirements for the specified bitrate and framerate. 328- *C2_PARAMKEY_SYNC_FRAME_INTERVAL*: This parameter can be used to configure the 329desired time between subsequent key frames in microseconds. 330- *C2_PARAMKEY_BITRATE_MODE*: This parameter can be used to switch between 331constant (*C2Config::BITRATE_CONST*) and variable (*C2Config::BITRATE_VARIABLE*) 332bitrate modes. When using CBR the encoder will try to maintain a constant 333bitrate. This mode is preferable for video conferencing where maintaining a 334stable bitrate is more important than quality. When using VBR the encoder will 335be allowed to dynamically adjust the bitrate to maintain a constant quality. As 336the mediacodec framework does not provide facilities to configure the peak 337bitrate when using VBR, it is currently always set to 2x the target bitrate. 338 339The following parameters are dynamic, and can be freely adjusted at run-time: 340 341- *C2_PARAMKEY_BITRATE*: Use this parameter to specify the desired bitrate. 342- *C2_PARAMKEY_FRAME_RATE*: This parameter can be used to configure the desired 343framerate. Note that the encoder will automatically try to adjust the framerate 344if the timestamps on the input video frames don't match the configured 345framerate. 346- *C2_PARAMKEY_REQUEST_SYNC_FRAME*: This parameter can be used to request 347additional key frames in addition to the periodic ones requested through the 348*C2_PARAMKEY_SYNC_FRAME_INTERVAL* parameter. 349 350### Supported Input Pixel Formats: 351 352The V4L2 encoder supports various input pixel formats, however frames are 353currently always passed to the V4L2 encoder in the NV12 format. If a video frame 354using a different pixel format is passed to the encoder, format conversion will 355be performed to convert the frame to the NV12 format. 356 357### Additional Features: 358 359To improve the resilience of H.264 video streams when data is missing, SPS and 360PPS NAL units are prepended to IDR frames by enabling the 361*V4L2_CID_MPEG_VIDEO_PREPEND_SPSPPS_TO_IDR* control. If the V4L2 driver does not 362support this control the encoder will manually cache and prepend SPS and PPS NAL 363units. 364 365[1]: https://android.googlesource.com/kernel/common/+/ed63bb1d1f8469586006a9ca63c42344401aa2ab 366