1# Codec 2 3## Overview 4### Function 5 6The OpenHarmony codec Hardware Device Interface (HDI) driver framework implements the video hardware codec driver based on OpenMAX. It provides APIs for the upper-layer media services to obtain component encoding and decoding capabilities, create a component, set parameters, transfer data, and destroy a component. The codec driver can encode video data in YUV or RGB format to H.264 or H.265 format, and decode raw stream data from H.264 or H.265 format to YUV or RGB format. This document describes the codec functionality developed based on the OpenHarmony Hardware Driver Foundation (HDF). 7 8The codec HDI driver framework is implemented based on the HDF. The figure below shows the codec HDI driver framework. 9 10**Figure 1** Codec HDI driver framework 11 12 13 14- Codec HDI Callback Remote Service: an anonymous callback service used to process callbacks. 15- Codec HDI: provides standard APIs based on OpenMAX. The upper layer services call the APIs to implement hardware encoding and decoding. 16- Codec HDI Adapter: HDI implementation layer, which implements HDI APIs and interacts with OpenMAX Integration layer (IL). 17- OpenMAX IL interface: provides OpenMAX IL APIs to directly interact with the codec HDI driver. 18- Vendor Impl: vendor adaptation layer, which is the OpenMAX implementation layer adapted by each vendor. 19- Codec Hardware: hardware decoding device. 20 21### Basic Concepts 22Before you get started, understand the following concepts: 23 24- Sampling rate 25 26 The number of samples taken from continuous signals every second to form discrete signals, in Hz. 27 28- OpenMAX IL 29 30 A standardized media component interface to enable applications and media frameworks to interact with multimedia codecs and supported components in a unified manner. 31 32- Frame rate 33 34 Number of frames of images transmitted per second, or the number of times that a GPU can refresh images per second. A higher frame rate indicates smoother motion, while a lower frame rate means choppier motion and blurry footage. 35 36- Bit rate 37 38 Number of bits transmitted or processed per unit of time, generally in kbit/s. A higher bit rate indicates clearer image, while a lower bit rate means blurry image with artifacts. 39 40- Component 41 42 An OpenMAX IL component, which is an abstraction of modules in video streams. The components in this document refer to codec components for video encoding and decoding. 43 44### Constraints 45 46The codec HDI applies only to the standard system. 47 48For more details, see [OpenMAX IL](https://www.khronos.org/api/openmax/il). 49 50## Development Guidelines 51 52### When to Use 53The codec module implements hardware encoding and decoding of video data. It converts raw stream data such as H.264 data into YUV or RGB data, and converts YUV or RGB data into data formats such as H.264. 54 55### Available APIs 56 57- codec_component_manager.h 58 59 | API | Description | 60 | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------| 61 | int32_t (*CreateComponent)(struct CodecComponentType **component, uint32_t *componentId, char *compName, int64_t appData, struct CodecCallbackType *callbacks) | Creates a codec component instance. | 62 | int32_t (*DestroyComponent)(uint32_t componentId) | Destroys a component instance. | 63 64- codec_component _if.h 65 66 | API | Description | 67 | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------- | 68 | int32_t (*SendCommand)(struct CodecComponentType *self, enum OMX_COMMANDTYPE cmd, uint32_t param, int8_t *cmdData, uint32_t cmdDataLen) | Sends commands to a component. | 69 | int32_t (*GetParameter)(struct CodecComponentType *self, uint32_t paramIndex, int8_t *paramStruct, uint32_t paramStructLen) | Obtains component parameter settings. | 70 | int32_t (*SetParameter)(struct CodecComponentType *self, uint32_t index, int8_t *paramStruct, uint32_t paramStructLen) | Sets component parameters. | 71 | int32_t (*GetState)(struct CodecComponentType *self, enum OMX_STATETYPE *state) | Obtains the component status. | 72 | int32_t (*UseBuffer)(struct CodecComponentType *self, uint32_t portIndex, struct OmxCodecBuffer *buffer) | Specifies the buffer of a component port. | 73 | int32_t (*FreeBuffer)(struct CodecComponentType *self, uint32_t portIndex, const struct OmxCodecBuffer *buffer) | Releases the buffer. | 74 | int32_t (*EmptyThisBuffer)(struct CodecComponentType *self, const struct OmxCodecBuffer *buffer) | Empties this buffer. | 75 | int32_t (*FillThisBuffer)(struct CodecComponentType *self, const struct OmxCodecBuffer *buffer) | Fills this buffer. | 76 77- codec_callback_if.h 78 79 | API | Description | 80 | ---------------------------------------------------------------------------------------------------------------- |----------------------------------- | 81 | int32_t (*EventHandler)(struct CodecCallbackType *self, enum OMX_EVENTTYPE event, struct EventInfo *info) | Reports an event. | 82 | int32_t (*EmptyBufferDone)(struct CodecCallbackType *self, int64_t appData, const struct OmxCodecBuffer *buffer) | Reports an event indicating that the encoding or decoding in the input buffer is complete.| 83 | int32_t (*FillBufferDone)(struct CodecCallbackType *self, int64_t appData, const struct OmxCodecBuffer *buffer) | Reports an event indicating that the output buffer is filled. | 84 85For more information, see [codec](https://gitee.com/openharmony/drivers_peripheral/tree/master/codec). 86 87### How to Develop 88The codec HDI driver development procedure is as follows: 89 90#### Registering and Initializing the Driver 91Define the **HdfDriverEntry** structure (which defines the driver initialization method) and fill in the **g_codecComponentDriverEntry** structure to implement the **Bind()**, **Init()**, and **Release()** pointers. 92 93```c 94struct HdfDriverEntry g_codecComponentDriverEntry = { 95 .moduleVersion = 1, 96 .moduleName = "codec_hdi_omx_server", 97 .Bind = HdfCodecComponentTypeDriverBind, 98 .Init = HdfCodecComponentTypeDriverInit, 99 .Release = HdfCodecComponentTypeDriverRelease, 100}; 101HDF_INIT(g_codecComponentDriverEntry); // Register HdfDriverEntry of the codec HDI with the HDF. 102``` 103 104- **HdfCodecComponentTypeDriverBind**: binds the device in the HDF to **CodecComponentTypeHost** and registers the codec service with the HDF. 105 106 ```c 107 int32_t HdfCodecComponentTypeDriverBind(struct HdfDeviceObject *deviceObject) 108 { 109 HDF_LOGI("HdfCodecComponentTypeDriverBind enter."); 110 struct HdfCodecComponentTypeHost *omxcomponenttypeHost = 111 (struct HdfCodecComponentTypeHost *)OsalMemAlloc(sizeof(struct HdfCodecComponentTypeHost)); 112 if (omxcomponenttypeHost == NULL) { 113 HDF_LOGE("HdfCodecComponentTypeDriverBind OsalMemAlloc HdfCodecComponentTypeHost failed!"); 114 return HDF_FAILURE; 115 } 116 int ret = HdfDeviceObjectSetInterfaceDesc(deviceObject, COMPONENT_MANAGER_SERVICE_DESC); 117 if (ret != HDF_SUCCESS) { 118 HDF_LOGE("Failed to set interface desc"); 119 return ret; 120 } 121 122 omxcomponenttypeHost->ioservice.Dispatch = CodecComponentTypeDriverDispatch; 123 omxcomponenttypeHost->ioservice.Open = NULL; 124 omxcomponenttypeHost->ioservice.Release = NULL; 125 omxcomponenttypeHost->service = CodecComponentManagerSerivceGet(); 126 if (omxcomponenttypeHost->service == NULL) { 127 OsalMemFree(omxcomponenttypeHost); 128 return HDF_FAILURE; 129 } 130 131 deviceObject->service = &omxcomponenttypeHost->ioservice; 132 return HDF_SUCCESS; 133 } 134 ``` 135 136- **HdfCodecComponentTypeDriverInit**: loads the attribute configuration from the HDF configuration source (HCS). 137 138 ```c 139 int32_t HdfCodecComponentTypeDriverInit(struct HdfDeviceObject *deviceObject) 140 { 141 HDF_LOGI("HdfCodecComponentTypeDriverInit enter."); 142 if (deviceObject == NULL) { 143 return HDF_FAILURE; 144 } 145 InitDataNode(deviceObject->property); 146 if (LoadCapabilityData() != HDF_SUCCESS) { 147 ClearCapabilityData(); 148 } 149 return HDF_SUCCESS; 150 } 151 ``` 152 153- **HdfCodecComponentTypeDriverRelease**: releases the driver instance. 154 155 ```c 156 void HdfCodecComponentTypeDriverRelease(struct HdfDeviceObject *deviceObject) 157 { 158 HDF_LOGI("HdfCodecComponentTypeDriverRelease enter."); 159 struct HdfCodecComponentTypeHost *omxcomponenttypeHost = 160 CONTAINER_OF(deviceObject->service, struct HdfCodecComponentTypeHost, ioservice); 161 OmxComponentManagerSeriveRelease(omxcomponenttypeHost->service); 162 OsalMemFree(omxcomponenttypeHost); 163 ClearCapabilityData(); 164 } 165 ``` 166 167#### Driver HCS 168The HCS consists of the following: 169 170- Device configuration 171- Configuration of the supported components 172 173The HCS includes the driver node, loading sequence, and service name. For details about the HCS syntax, see [Configuration Management](driver-hdf-manage.md). 174 175Configuration file Path of the standard system: 176vendor/hihope/rk3568/hdf_config/uhdf/ 177 1781. Device configuration 179 180 Add the **codec_omx_service** configuration to **codec_host** in **device_info.hcs**. The following is an example: 181 ```c 182 codec :: host { 183 hostName = "codec_host"; 184 priority = 50; 185 gid = ["codec_host", "uhdf_driver", "vendor_mpp_driver"]; 186 codec_omx_device :: device { 187 device0 :: deviceNode { 188 policy = 2; // Automatic loading, not lazy loading. 189 priority = 100; // Priority. 190 moduleName = "libcodec_hdi_omx_server.z.so"; // Dynamic library of the driver. 191 serviceName = "codec_hdi_omx_service"; // Service name of the driver. 192 deviceMatchAttr = "codec_component_capabilities"; //Attribute configuration. 193 } 194 } 195 } 196 ``` 197 1982. Configuration of supported components 199 200 Add the component configuration to the **media_codec\codec_component_capabilities.hcs file**. The following is an example: 201 ```c 202 /* node name explanation -- HDF_video_hw_enc_avc_rk: 203 ** 204 ** HDF____________video__________________hw____________________enc____________avc_______rk 205 ** | | | | | | 206 ** HDF or OMX video or audio hardware or software encoder or decoder mime vendor 207 */ 208 HDF_video_hw_enc_avc_rk { 209 role = 1; // Role of the AvCodec. 210 type = 1; // Codec type. 211 name = "OMX.rk.video_encoder.avc"; // Component name. 212 supportProfiles = [1, 32768, 2, 32768, 8, 32768]; // Supported profiles. 213 maxInst = 4; // Maximum number of instances. 214 isSoftwareCodec = false; // Whether it is software codec. 215 processModeMask = []; // Codec processing mode. 216 capsMask = [0x01]; // Codec playback capabilities. 217 minBitRate = 1; // Minimum bit rate. 218 maxBitRate = 40000000; // Maximum bit rate. 219 minWidth = 176; // Minimum video width. 220 minHeight = 144;; // Minimum video height. 221 maxWidth = 1920; // Maximum video width. 222 maxHeight = 1088; // Maximum video height. 223 widthAlignment = 16; // Horizontal alignment. 224 heightAlignment = 8; // Vertical alignment. 225 minBlockCount = 0xFFFFFFFF; 226 maxBlockCount = 0xFFFFFFFF; 227 minBlocksPerSecond = 0xFFFFFFFF; 228 maxBlocksPerSecond = 0xFFFFFFFF; 229 blockSizeWidth = 0xFFFFFFFF; 230 blockSizeHeight = 0xFFFFFFFF; 231 supportPixelFmts = [28, 24, 30, 22, 7, 3, 14, 13, 20, 26, 27, 12]; // List of supported colors. 232 measuredFrameRate = [320, 240, 165, 165, 720, 480, 149, 149, 1280, 720, 73, 73, 1920, 1080, 18, 18]; 233 bitRateMode = [1, 2]; // Bit rate mode. 234 minFrameRate = 0; // Frame rate. 235 maxFrameRate = 0; 236 } 237 ``` 238 239### Development Example 240After completing codec module driver adaptation, use the HDI APIs provided by the codec module for further development. The codec HDI provides the following features: 241 2421. Provides codec HDI APIs for video services to implement encoding and decoding of video services. 2432. Provides standard interfaces for device developers to ensure that the OEM vendors comply with the HDI adapter standard. This promises a healthy evolution of the ecosystem. 244 245The development procedure is as follows: 246 2471. Initialize the driver, including initializing the instances, callbacks, and component. 2482. Set codec parameters and information such as the video width, height, and bit rate. 2493. Apply for input and output buffers. 2504. Flip codec buffers, enable the component to enter the **OMX_Executing** state, and process the callbacks. 2515. Deinitialize the interface instance, destroy the buffers, close the component, and releases all interface objects. 252 253#### Initializing the Driver 254Initialize the interface instance and callbacks, and create a component. 255```cpp 256// Initialize the codec HDI ComponentManager instance. 257omxMgr_ = GetCodecComponentManager(); 258 259// Initialize the callback. 260callback_ = CodecCallbackTypeStubGetInstance(); 261if (!omxMgr_ || !callback_) { 262 FUNC_EXIT_ERR(); 263 return false; 264} 265// Set the callback pointers. 266callback_->EventHandler = &OMXCore::OnEvent; 267callback_->EmptyBufferDone = &OMXCore::OnEmptyBufferDone; 268callback_->FillBufferDone = &OMXCore::OnFillBufferDone; 269 270// Create a component instance. 271uint32_t err = HDF_SUCCESS; 272if (codec == codecMime::AVC) { 273 err = omxMgr_->CreateComponent(&client_, &componentId_, const_cast<char *>(DECODER_AVC), (int64_t)this, 274 callback_); 275} else { 276 err = omxMgr_->CreateComponent(&client_, &componentId_, const_cast<char *>(DECODER_HEVC), (int64_t)this, 277 callback_); 278} 279``` 280 281#### Setting Codec Parameters and Configuration 282Set the width and height of the input and output data, input data format, and output data format. 283```cpp 284// Set the width and height of the input image. 285OMX_PARAM_PORTDEFINITIONTYPE param; 286InitParam(param); 287param.nPortIndex = (uint32_t)PortIndex::PORT_INDEX_INPUT; 288auto err = client_->GetParameter(client_, OMX_IndexParamPortDefinition, (int8_t *)¶m, sizeof(param)); 289if (err != HDF_SUCCESS) { 290 HDF_LOGE("%{public}s failed PortIndex::PORT_INDEX_INPUT, index is OMX_IndexParamPortDefinition", __func__); 291 return false; 292} 293HDF_LOGI("PortIndex::PORT_INDEX_INPUT: eCompressionFormat = %{public}d, eColorFormat = %{public}d ", 294 param.format.video.eCompressionFormat, param.format.video.eColorFormat); 295param.format.video.nFrameWidth = width_; 296param.format.video.nFrameHeight = height_; 297param.format.video.nStride = width_; 298param.format.video.nSliceHeight = height_; 299err = client_->SetParameter(client_, OMX_IndexParamPortDefinition, (int8_t *)¶m, sizeof(param)); 300if (err != HDF_SUCCESS) { 301 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_INPUT, index is OMX_IndexParamPortDefinition", __func__); 302 return false; 303} 304// Set the output width, height, and format. 305InitParam(param); 306param.nPortIndex = (uint32_t)PortIndex::PORT_INDEX_OUTPUT; 307err = client_->GetParameter(client_, OMX_IndexParamPortDefinition, (int8_t *)¶m, sizeof(param)); 308if (err != HDF_SUCCESS) { 309 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_OUTPUT, index is OMX_IndexParamPortDefinition", __func__); 310 return false; 311} 312HDF_LOGI("PortIndex::PORT_INDEX_OUTPUT eCompressionFormat = %{public}d, eColorFormat=%{public}d", 313 param.format.video.eCompressionFormat, param.format.video.eColorFormat); 314param.format.video.nFrameWidth = width_; 315param.format.video.nFrameHeight = height_; 316param.format.video.nStride = width_; 317param.format.video.nSliceHeight = height_; 318param.format.video.eColorFormat = AV_COLOR_FORMAT; // Set the output data format to YUV420SP. 319err = client_->SetParameter(client_, OMX_IndexParamPortDefinition, (int8_t *)¶m, sizeof(param)); 320if (err != HDF_SUCCESS) { 321 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_OUTPUT, index is OMX_IndexParamPortDefinition", 322 __func__); 323 return false; 324} 325// Set the input data format to H.264/H.265. 326OMX_VIDEO_PARAM_PORTFORMATTYPE param; 327InitParam(param); 328param.nPortIndex = (uint32_t)PortIndex::PORT_INDEX_INPUT; 329auto err = client_->GetParameter(client_, OMX_IndexParamVideoPortFormat, (int8_t *)¶m, sizeof(param)); 330if (err != HDF_SUCCESS) { 331 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_INPUT", __func__); 332 return false; 333} 334HDF_LOGI("set Format PortIndex::PORT_INDEX_INPUT eCompressionFormat = %{public}d, eColorFormat=%{public}d", 335 param.eCompressionFormat, param.eColorFormat); 336param.xFramerate = FRAME; // Set the frame rate to 30. 337if (codecMime_ == codecMime::AVC) { 338 param.eCompressionFormat = OMX_VIDEO_CodingAVC; // H264 339} else { 340 param.eCompressionFormat = (OMX_VIDEO_CODINGTYPE)CODEC_OMX_VIDEO_CodingHEVC; // H265 341} 342 343err = client_->SetParameter(client_, OMX_IndexParamVideoPortFormat, (int8_t *)¶m, sizeof(param)); 344if (err != HDF_SUCCESS) { 345 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_INPUT", __func__); 346 return false; 347} 348``` 349 350#### Applying for Input and Output Buffers 351Perform the following steps: 352 3531. Use **UseBuffer()** to apply for input and output buffers and save the buffer IDs. The buffer IDs can be used for subsequent buffer flipping. 3542. Check whether the corresponding port is enabled. If not, enable the port first. 3553. Use **SendCommand()** to change the component status to OMX_StateIdle, and wait until the operation result is obtained. 356```cpp 357// Apply for the input buffer. 358auto ret = UseBufferOnPort(PortIndex::PORT_INDEX_INPUT); 359if (!ret) { 360 HDF_LOGE("%{public}s UseBufferOnPort PortIndex::PORT_INDEX_INPUT error", __func__); 361 return false; 362} 363// Apply for the output buffer. 364ret = UseBufferOnPort(PortIndex::PORT_INDEX_OUTPUT); 365if (!ret) { 366 HDF_LOGE("%{public}s UseBufferOnPort PortIndex::PORT_INDEX_OUTPUT error", __func__); 367 return false; 368} 369// Enable the component to enter the OMX_StateIdle state. 370auto err = client_->SendCommand(client_, OMX_CommandStateSet, OMX_StateIdle, NULL, 0); 371if (err != HDF_SUCCESS) { 372 HDF_LOGE("%{public}s failed to SendCommand with OMX_CommandStateSet:OMX_StateIdle", __func__); 373 return false; 374} 375HDF_LOGI("Wait for OMX_StateIdle status"); 376this->WaitForStatusChanged(); 377``` 378 379Implement **UseBufferOnPort** as follows: 380 381```cpp 382bool CodecHdiDecode::UseBufferOnPort(enum PortIndex portIndex) 383{ 384 HDF_LOGI("%{public}s enter, portIndex = %{public}d", __func__, portIndex); 385 int bufferSize = 0; 386 int bufferCount = 0; 387 bool bPortEnable = false; 388 // Obtain parameters of the port buffer. 389 OMX_PARAM_PORTDEFINITIONTYPE param; 390 InitParam(param); 391 param.nPortIndex = (OMX_U32)portIndex; 392 auto err = client_->GetParameter(client_, OMX_IndexParamPortDefinition, (int8_t *)¶m, sizeof(param)); 393 if (err != HDF_SUCCESS) { 394 HDF_LOGE("%{public}s failed to GetParameter with OMX_IndexParamPortDefinition : portIndex[%{public}d]", 395 __func__, portIndex); 396 return false; 397 } 398 bufferSize = param.nBufferSize; 399 bufferCount = param.nBufferCountActual; 400 bPortEnable = param.bEnabled; 401 HDF_LOGI("buffer index [%{public}d], buffer size [%{public}d], " 402 "buffer count [%{public}d], portEnable[%{public}d], err [%{public}d]", 403 portIndex, bufferSize, bufferCount, bPortEnable, err); 404 { 405 OMX_PARAM_BUFFERSUPPLIERTYPE param; 406 InitParam(param); 407 param.nPortIndex = (uint32_t)portIndex; 408 auto err = client_->GetParameter(client_, OMX_IndexParamCompBufferSupplier, (int8_t *)¶m, sizeof(param)); 409 HDF_LOGI("param.eBufferSupplier[%{public}d] isSupply [%{public}d], err [%{public}d]", param.eBufferSupplier, 410 this->isSupply_, err); 411 } 412 // Set the port buffer. 413 UseBufferOnPort(portIndex, bufferCount, bufferSize); 414 // Check whether the port is available. 415 if (!bPortEnable) { 416 auto err = client_->SendCommand(client_, OMX_CommandPortEnable, (uint32_t)portIndex, NULL, 0); 417 if (err != HDF_SUCCESS) { 418 HDF_LOGE("%{public}s SendCommand OMX_CommandPortEnable::PortIndex::PORT_INDEX_INPUT error", __func__); 419 return false; 420 } 421 } 422 return true; 423} 424 425bool CodecHdiDecode::UseBufferOnPort(enum PortIndex portIndex, int bufferCount, int bufferSize) 426{ 427 for (int i = 0; i < bufferCount; i++) { 428 OmxCodecBuffer *omxBuffer = new OmxCodecBuffer(); 429 memset_s(omxBuffer, sizeof(OmxCodecBuffer), 0, sizeof(OmxCodecBuffer)); 430 omxBuffer->size = sizeof(OmxCodecBuffer); 431 omxBuffer->version.s.nVersionMajor = 1; 432 omxBuffer->bufferType = BUFFER_TYPE_AVSHARE_MEM_FD; 433 int fd = AshmemCreate(0, bufferSize); 434 shared_ptr<Ashmem> sharedMem = make_shared<Ashmem>(fd, bufferSize); 435 omxBuffer->bufferLen = FD_SIZE; 436 omxBuffer->buffer = (uint8_t *)(unsigned long)fd; 437 omxBuffer->allocLen = bufferSize; 438 omxBuffer->fenceFd = -1; 439 440 if (portIndex == PortIndex::PORT_INDEX_INPUT) { 441 omxBuffer->type = READ_ONLY_TYPE; // ReadOnly 442 sharedMem->MapReadAndWriteAshmem(); 443 } else { 444 omxBuffer->type = READ_WRITE_TYPE; 445 sharedMem->MapReadOnlyAshmem(); 446 } 447 auto err = client_->UseBuffer(client_, (uint32_t)portIndex, omxBuffer); 448 if (err != HDF_SUCCESS) { 449 HDF_LOGE("%{public}s failed to UseBuffer with portIndex[%{public}d]", __func__, portIndex); 450 sharedMem->UnmapAshmem(); 451 sharedMem->CloseAshmem(); 452 sharedMem = nullptr; 453 return false; 454 } 455 omxBuffer->bufferLen = 0; 456 HDF_LOGI("UseBuffer returned bufferID [%{public}d]", omxBuffer->bufferId); 457 458 BufferInfo *bufferInfo = new BufferInfo; 459 bufferInfo->omxBuffer = omxBuffer; 460 bufferInfo->avSharedPtr = sharedMem; 461 bufferInfo->portIndex = portIndex; 462 omxBuffers_.insert(std::make_pair<int, BufferInfo *>(omxBuffer->bufferId, std::move(bufferInfo))); 463 if (portIndex == PortIndex::PORT_INDEX_INPUT) { 464 unUsedInBuffers_.push_back(omxBuffer->bufferId); 465 } else { 466 unUsedOutBuffers_.push_back(omxBuffer->bufferId); 467 } 468 int fdret = (int)omxBuffer->buffer; 469 HDF_LOGI("{bufferID = %{public}d, srcfd = %{public}d, retfd = %{public}d}", omxBuffer->bufferId, fd, fdret); 470 } 471 return true; 472} 473``` 474 475#### Codec Buffer Flipping 476Set the component to the **OMX_StateExecuting** state, fill the input buffer, read data from the output buffer, and flip the buffers. 477 478```cpp 479// Set the component to the OMX_StateExecuting state and start buffer flipping. 480HDF_LOGI("...command to OMX_StateExecuting...."); 481auto err = client_->SendCommand(client_, OMX_CommandStateSet, OMX_StateExecuting, NULL, 0); 482if (err != HDF_SUCCESS) { 483 HDF_LOGE("%{public}s failed to SendCommand with OMX_CommandStateSet:OMX_StateIdle", __func__); 484 return; 485} 486// Set the output buffer. 487for (auto bufferId : unUsedOutBuffers_) { 488 HDF_LOGI("fill bufferid [%{public}d]", bufferId); 489 auto iter = omxBuffers_.find(bufferId); 490 if (iter != omxBuffers_.end()) { 491 BufferInfo *bufferInfo = iter->second; 492 auto err = client_->FillThisBuffer(client_, bufferInfo->pOmxBuffer); 493 if (err != HDF_SUCCESS) { 494 HDF_LOGE("FillThisBuffer error"); 495 FUNC_EXIT_ERR(); 496 return; 497 } 498 } 499} 500// Fill the input buffer. 501bool bEndOfFile = false; 502while (!bEndOfFile) { 503 int bufferID = GetFreeBufferId(); 504 if (this->exit_) { 505 break; 506 } 507 if (bufferID < 0) { 508 usleep(10000); 509 continue; 510 } 511 auto iter = omxBuffers_.find(bufferID); 512 if (iter == omxBuffers_.end()) { 513 continue; 514 } 515 BufferInfo *bufferInfo = iter->second; 516 void *sharedAddr = (void *)bufferInfo->avSharedPtr->ReadFromAshmem(0, 0); 517 bool bEOS = (size_t)this->ReadOnePacket(fpIn_, (char *)sharedAddr, bufferInfo->omxBuffer->filledLen); 518 HDF_LOGI("read data size is %{public}d", bufferInfo->omxBuffer->filledLen); 519 bufferInfo->omxBuffer->offset = 0; 520 if (bEOS) { 521 bufferInfo->omxBuffer->flag = OMX_BUFFERFLAG_EOS; 522 bEndOfFile = true; 523 } 524 auto err = client_->EmptyThisBuffer(client_, bufferInfo->omxBuffer); 525 if (err != HDF_SUCCESS) { 526 HDF_LOGE("%{public}s EmptyThisBuffer error", __func__); 527 return; 528 } 529} 530// Wait. 531while (!this->exit_) { 532 usleep(10000); 533 continue; 534} 535// Enable the component to enter the OMX_StateIdle state after decoding. 536client_->SendCommand(client_, OMX_CommandStateSet, OMX_StateIdle, NULL, 0); 537``` 538 539Automatic framing is not supported in rk OMX decoding. Therefore, you need to manually divide data into frames. Currently, data is divided into frames from code 0x000001 or 0x00000001 and sent to the server for processing. The sample code is as follows: 540 541```cpp 542// Read a file by frame. 543bool OMXCore::ReadOnePacket(FILE* fp, char* buf, uint32_t& nFilled) 544{ 545 // Read four bytes first. 546 size_t t = fread(buf, 1, 4, fp); 547 if (t < 4) { 548 // The file reading ends. 549 return true; 550 } 551 size_t filled = 0; 552 filled = 4; 553 554 bool bRet = true; 555 while (!feof(fp)) { 556 fread(buf + filled, 1, 1, fp); 557 if (buf[filled] == 1) { 558 // Check the start code. 559 if ((buf[filled - 1] == 0) && 560 (buf[filled - 2] == 0) && 561 (buf[filled - 3] == 0)) { 562 fseek(fp, -4, SEEK_CUR); 563 filled -= 3; 564 bRet = false; 565 break; 566 } else if ((buf[filled - 1] == 0) && 567 (buf[filled - 2] == 0)) { 568 fseek(fp, -3, SEEK_CUR); 569 filled -= 2; 570 bRet = false; 571 break; 572 } 573 } 574 filled++; 575 } 576 nFilled = filled; 577 return bRet; 578} 579``` 580 581The codec HDI provides the following callbacks: 582 583- **EventHandler**: Called when a command is executed. For example, when the command for changing the component state from **OMX_StateIdle** to **OMX_StateExecuting** is executed, this callback is invoked to return the result. 584- **EmptyBufferDone**: Called when the input data is consumed. If the client needs to fill in data to encode or decode, call **EmptyThisBuffer()**. 585- **FillBufferDone**: Called when the output data is filled. If the client needs to read the encoded or decoded data, call **FillThisBuffer()**. 586 587```cpp 588// EmptyBufferDone example 589int32_t OMXCore::OnEmptyBufferDone(struct CodecCallbackType *self, int8_t *pAppData, uint32_t pAppDataLen, 590 const struct OmxCodecBuffer *pBuffer) 591{ 592 HDF_LOGI("onEmptyBufferDone: pBuffer.bufferID [%{public}d]", pBuffer->bufferId); 593 g_core->OnEmptyBufferDone(pBuffer); 594 return HDF_SUCCESS; 595} 596int32_t OMXCore::OnEmptyBufferDone(const struct OmxCodecBuffer *pBuffer) 597{ 598 unique_lock<mutex> ulk(mLockInputBuffers_); 599 unUsedInBuffers_.push_back(pBuffer->bufferId); 600 return HDF_SUCCESS; 601} 602// FillBufferDone example 603int32_t OMXCore::OnFillBufferDone(struct CodecCallbackType *self, int8_t *pAppData, uint32_t pAppDataLen, 604 struct OmxCodecBuffer *pBuffer) 605{ 606 HDF_LOGI("onFillBufferDone: pBuffer.bufferID [%{public}d]", pBuffer->bufferId); 607 g_core->OnFillBufferDone(pBuffer); 608 return HDF_SUCCESS; 609} 610int32_t OMXCore::onFillBufferDone(struct OmxCodecBuffer* pBuffer) 611{ 612 // Locate the buffer based on the buffer ID. 613 if (bExit_) { 614 return HDF_SUCCESS; 615 } 616 617 auto iter = omxBuffers_.find(pBuffer->bufferId); 618 if (iter == omxBuffers_.end() || !iter->second) { 619 return HDF_SUCCESS; 620 } 621 // Obtain the output data. 622 BufferInfo *pBufferInfo = iter->second; 623 const void *addr = pBufferInfo->avSharedPtr->ReadFromAshmem(pBuffer->filledLen, pBuffer->offset); 624 // Decode the data and save it to a file. 625 fwrite(addr, 1, pBuffer->filledLen, fpOut_.get()); 626 fflush(fpOut_.get()); 627 // Reset the buffer data. 628 pBuffer->offset = 0; 629 pBuffer->filledLen = 0; 630 if (pBuffer->flag == OMX_BUFFERFLAG_EOS) { 631 // End 632 bExit_ = true; 633 HDF_LOGI("OnFillBufferDone the END coming"); 634 return HDF_SUCCESS; 635 } 636 // Call FillThisBuffer() again. 637 auto err = client_->FillThisBuffer(client_, pBufferInfo->pOmxBuffer); 638 if (err != HDF_SUCCESS) { 639 HDF_LOGE("FillThisBuffer error"); 640 return HDF_SUCCESS; 641 } 642 return HDF_SUCCESS; 643} 644 645// EventHandler example 646int32_t CodecHdiDecode::OnEvent(struct CodecCallbackType *self, enum OMX_EVENTTYPE event, struct EventInfo *info) 647{ 648 HDF_LOGI("onEvent: appData[0x%{public}p], eEvent [%{public}d], " 649 "nData1[%{public}d]", 650 info->appData, event, info->data1); 651 switch (event) { 652 case OMX_EventCmdComplete: { 653 OMX_COMMANDTYPE cmd = (OMX_COMMANDTYPE)info->data1; 654 if (OMX_CommandStateSet == cmd) { 655 HDF_LOGI("OMX_CommandStateSet reached, status is %{public}d", info->data2); 656 g_core->onStatusChanged(); 657 } 658 break; 659 } 660 default: 661 break; 662 } 663 return HDF_SUCCESS; 664} 665``` 666 667#### Destroying a Component 668Change the component state to IDLE, release the input and output buffers, change the component state to **OMX_StateLoaded**, and call **DestoryComponent** to destroy the component. 669 670##### Example of Releasing Buffers 671 672```cpp 673// Change the component state to OMX_StateLoaded. 674client_->SendCommand(client_, OMX_CommandStateSet, OMX_StateLoaded, nullptr, 0); 675 676// Release all buffers in use. 677auto iter = omxBuffers_.begin(); 678while (iter != omxBuffers_.end()) { 679 BufferInfo *bufferInfo = iter->second; 680 client_->FreeBuffer(client_, (uint32_t)bufferInfo->portIndex, bufferInfo->omxBuffer); 681 delete bufferInfo; 682 iter++; 683} 684omxBuffers_.clear(); 685unUsedInBuffers_.clear(); 686unUsedOutBuffers_.clear(); 687 688enum OMX_STATETYPE status; 689client_->GetState(client_, &status); 690// After the buffers are released, the component enters the OMX_StateLoaded state. 691if (status != OMX_StateLoaded) { 692 HDF_LOGI("Wait for OMX_StateLoaded status"); 693 this->WaitForStatusChanged(); 694} else { 695 HDF_LOGI(" status is %{public}d", status); 696} 697``` 698 699##### Example of Destroying a Component Instance 700 701```cpp 702// Destroy a component instance. 703void OMXCore::Release() { 704 omxMgr_->DestoryComponent(client_); 705 client_ = nullptr; 706 CodecComponentManagerRelease(); 707} 708``` 709 710# FAQs 711 712## Green Screens Displayed During the Decoding Process 713 714**Symptom** 715 716Green screens are displayed during the decoding process. 717 718**Possible Causes** 719 720OpenMAX does not support framing. 721 722**Solution** 723 724Transfer data frame by frame when **EmptyThisBuffer** is called. 725 726## Only Green Screen Displayed During the Decoding Process 727 728**Symptom** 729 730Decoding fails, and all the frames decoded cannot be played. 731 732**Possible Causes** 733 734For the data in AVCC format, the first frame to be processed must be extra_data. 735 736**Solution** 737 738Write sps and pps to the buffer in extra_data format, and set the buffer flag to **OMX_BUFFERFLAG_EXTRADATA**. 739 740## Failed to Play the Encoded Video 741 742**Symptom** 743 744After the generated video stream (H.264 stream) is written to a file, the video stream cannot be played by FFplay. 745 746**Possible Causes** 747 748- The **xFramerate** parameter of the output port is incorrectly set. 749- The **OMX_VIDEO_PARAM_AVCTYPE** parameter is correctly set. 750 751 752**Solution** 753 754View the **codec_host** log generated during encoding, search for "encode params init settings", and check for incorrect parameters. If **framerate** is **0**, **xFramerate** is incorrectly set. In this case, move the framerate leftwards by 16 bits. 755 756Check the value of **OMX_VIDEO_PARAM_AVCTYPE**, and set it correctly. 757 758 759# Reference 760 761For more information, see [Codec](https://gitee.com/openharmony/drivers_peripheral/tree/master/codec). 762
