1# Codec 2 3## Overview 4### Function 5 6The OpenHarmony codec Hardware Device Interface (HDI) driver framework implements the video hardware codec driver based on OpenMAX. It provides APIs for the upper-layer media services to obtain component encoding and decoding capabilities, create a component, set parameters, transfer data, and destroy a component. The codec driver can encode video data in YUV or RGB format to H.264 or H.265 format, and decode raw stream data from H.264 or H.265 format to YUV or RGB format. This document describes the codec functionality developed based on the OpenHarmony Hardware Driver Foundation (HDF). 7 8The codec HDI driver framework is implemented based on the HDF. The figure below shows the codec HDI driver framework. 9 10**Figure 1** Codec HDI driver framework 11 12 13 14- Codec HDI Callback Remote Service: an anonymous callback service used to process callbacks. 15- Codec HDI: provides standard APIs based on OpenMAX. The upper layer services call the APIs to implement hardware encoding and decoding. 16- Codec HDI Adapter: HDI implementation layer, which implements HDI APIs and interacts with OpenMAX Integration layer (IL). 17- OpenMAX IL interface: provides OpenMAX IL APIs to directly interact with the codec HDI driver. 18- Vendor Impl: vendor adaptation layer, which is the OpenMAX implementation layer adapted by each vendor. 19- Codec Hardware: hardware decoding device. 20 21### Basic Concepts 22Before you get started, understand the following concepts: 23 24- Sampling rate 25 26 The number of samples taken from continuous signals every second to form discrete signals, in Hz. 27 28- OpenMAX IL 29 30 A standardized media component interface to enable applications and media frameworks to interact with multimedia codecs and supported components in a unified manner. 31 32- Frame rate 33 34 Number of frames of images transmitted per second, or the number of times that a GPU can refresh images per second. A higher frame rate indicates smoother motion, while a lower frame rate means choppier motion and blurry footage. 35 36- Bit rate 37 38 Number of bits transmitted or processed per unit of time, generally in kbit/s. A higher bit rate indicates clearer image, while a lower bit rate means blurry image with artifacts. 39 40- Component 41 42 An OpenMAX IL component, which is an abstraction of modules in video streams. The components in this document refer to codec components used for video encoding and decoding. 43 44### Constraints 45 46The codec HDI applies only to the standard system. 47 48For more details, see [OpenMAX IL](https://www.khronos.org/api/openmax/il). 49 50## Development Guidelines 51 52### When to Use 53The codec module implements hardware encoding and decoding of video data. It converts raw stream data such as H.264 data into YUV or RGB data, and converts YUV or RGB data into data formats such as H.264. 54 55### Available APIs 56 57- icodec_component_manager.h 58 59 | API | Description | 60 | -------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------| 61 | int32_t CreateComponent(sptr<ICodecComponent>& component, uint32_t& componentId,<br>const std::string& compName, int64_t appData, const sptr<ICodecCallback>& callbacks) | Creates a codec component instance. | 62 | int32_t DestoryComponent(uint32_t componentId) | Destroys a codec component instance. | 63 64- icodec_component.h 65 66 | API | Description | 67 | ------------------------------------------------------------ | ---------------------- | 68 | int32_t SendCommand(CodecCommandType cmd, uint32_t param, const std::vector<int8_t>& cmdData) | Sends commands to a component. | 69 | int32_t GetParameter(uint32_t index, const std::vector<int8_t>& inParamStruct, std::vector<int8_t>& outParamStruct) | Obtains component parameter settings. | 70 | int32_t SetParameter(uint32_t index, const std::vector<int8_t>& paramStruct) | Sets component parameters. | 71 | int32_t GetState(CodecStateType& state) | Obtains the component status. | 72 | int32_t UseBuffer(uint32_t portIndex, const OmxCodecBuffer& inBuffer, OmxCodecBuffer& outBuffer) | Requests a port buffer for the component. | 73 | int32_t FreeBuffer(uint32_t portIndex, const OmxCodecBuffer& buffer) | Releases the buffer. | 74 | int32_t EmptyThisBuffer(const OmxCodecBuffer& buffer) | Empties this buffer.| 75 | int32_t FillThisBuffer(const OmxCodecBuffer& buffer) | Fills this buffer. | 76 77- icodec_callback.h 78 79 | API | Description | 80 | ------------------------------------------------------------ | ---------------------------------- | 81 | int32_t EventHandler(CodecEventType event, const EventInfo& info) | Called to report an event. | 82 | int32_t EmptyBufferDone(int64_t appData, const OmxCodecBuffer& buffer) | Called to report an event indicating that the encoding or decoding in the input buffer is complete.| 83 | int32_t FillBufferDone(int64_t appData, const OmxCodecBuffer& buffer) | Called to report an event indicating that the output buffer is filled. | 84 85For more information, see [codec](https://gitee.com/openharmony/drivers_peripheral/tree/master/codec). 86 87### How to Develop 88The codec HDI driver development procedure is as follows: 89 90#### Registering and Initializing the Driver 91Define the **HdfDriverEntry** struct (which defines the driver initialization method) and fill in the **g_codeccomponentmanagerDriverEntry** struct to implement the **Bind()**, **Init()**, and **Release()** pointers. 92 93```c 94static struct HdfDriverEntry g_codeccomponentmanagerDriverEntry = { 95 .moduleVersion = 1, 96 .moduleName = "codec_component_manager_service", 97 .Bind = HdfCodecComponentManagerDriverBind, 98 .Init = HdfCodecComponentManagerDriverInit, 99 .Release = HdfCodecComponentManagerDriverRelease, 100}; // Register the HdfDriverEntry struct of the codec HDI with the HDF. 101``` 102 103- **HdfCodecComponentManagerDriverBind**: binds the device in the HDF to the **HdfCodecComponentManagerHost** and registers the codec service with the HDF. 104 105 ```c 106 static int HdfCodecComponentManagerDriverBind(struct HdfDeviceObject *deviceObject) 107 { 108 CODEC_LOGI("HdfCodecComponentManagerDriverBind enter"); 109 110 auto *hdfCodecComponentManagerHost = new (std::nothrow) HdfCodecComponentManagerHost; 111 if (hdfCodecComponentManagerHost == nullptr) { 112 CODEC_LOGE("failed to create create HdfCodecComponentManagerHost object"); 113 return HDF_FAILURE; 114 } 115 116 hdfCodecComponentManagerHost->ioService.Dispatch = CodecComponentManagerDriverDispatch; 117 hdfCodecComponentManagerHost->ioService.Open = NULL; 118 hdfCodecComponentManagerHost->ioService.Release = NULL; 119 120 auto serviceImpl = ICodecComponentManager::Get(true); 121 if (serviceImpl == nullptr) { 122 CODEC_LOGE("failed to get of implement service"); 123 delete hdfCodecComponentManagerHost; 124 return HDF_FAILURE; 125 } 126 127 hdfCodecComponentManagerHost->stub = 128 OHOS::HDI::ObjectCollector::GetInstance().GetOrNewObject(serviceImpl, ICodecComponentManager::GetDescriptor()); 129 if (hdfCodecComponentManagerHost->stub == nullptr) { 130 CODEC_LOGE("failed to get stub object"); 131 delete hdfCodecComponentManagerHost; 132 return HDF_FAILURE; 133 } 134 135 deviceObject->service = &hdfCodecComponentManagerHost->ioService; 136 return HDF_SUCCESS; 137 } 138 ``` 139 140- **HdfCodecComponentManagerDriverInit**: loads the attribute configuration from the HDF Configuration Source (HCS). 141 142 ```c 143 static int HdfCodecComponentManagerDriverInit(struct HdfDeviceObject *deviceObject) 144 { 145 CODEC_LOGI("HdfCodecComponentManagerDriverInit enter"); 146 if (DevHostRegisterDumpHost(CodecDfxService::DevCodecHostDump) != HDF_SUCCESS) { 147 CODEC_LOGE("DevHostRegisterDumpHost error!"); 148 } 149 return HDF_SUCCESS; 150 } 151 ``` 152 153- **HdfCodecComponentTypeDriverRelease**: releases the driver instance. 154 155 ```c 156 static void HdfCodecComponentManagerDriverRelease(struct HdfDeviceObject *deviceObject) 157 { 158 CODEC_LOGI("HdfCodecComponentManagerDriverRelease enter"); 159 if (deviceObject->service == nullptr) { 160 CODEC_LOGE("HdfCodecComponentManagerDriverRelease not initted"); 161 return; 162 } 163 164 auto *hdfCodecComponentManagerHost = 165 CONTAINER_OF(deviceObject->service, struct HdfCodecComponentManagerHost, ioService); 166 delete hdfCodecComponentManagerHost; 167 } 168 ``` 169 170#### Configuring the Driver HCS 171The HCS consists of the following: 172 173- Device configuration 174- Configuration of the supported components 175 176The HCS includes the driver node, loading sequence, and service name. For details about the HCS syntax, see [Configuration Management](driver-hdf-manage.md). 177 178The following uses the RK3568 development board as an example. The configuration files of the standard system are in the **vendor/hihope/rk3568/hdf_config/uhdf/** directory. 179 1801. Configure the device. 181 182 Add the **codec_component_manager_service** configuration to **codec_host** in **device_info.hcs**. 183 184 Example: 185 186 ```c 187 codec :: host { 188 hostName = "codec_host"; 189 priority = 50; 190 gid = ["codec_host", "uhdf_driver", "vendor_mpp_driver"]; 191 codec_omx_idl_device :: device { 192 device0 :: deviceNode { 193 policy = 2; // Automatic loading, not lazy loading. 194 priority = 100; // Priority. 195 moduleName = "libcodec_driver.z.so"; // Dynamic library of the driver. 196 serviceName = "codec_component_manager_service"; // Service name of the driver. 197 deviceMatchAttr = "media_codec_capabilities"; // Attribute configuration. 198 } 199 } 200 } 201 ``` 202 2032. Configure supported components. 204 205 Add the component configuration to the **media_codec\media_codec_capabilities.hcs** file. 206 207 Example: 208 209 ```c 210 /* node name explanation -- HDF_video_hw_enc_avc_rk: 211 ** 212 ** HDF____________video__________________hw____________________enc____________avc_______rk 213 ** | | | | | | 214 ** HDF or OMX video or audio hardware or software encoder or decoder MIME vendor 215 */ 216 HDF_video_hw_enc_avc_rk { 217 role = 1; // Role of the audio and video codec. 218 type = 1; // Codec type. 219 name = "OMX.rk.video_encoder.avc"; // Component name. 220 supportProfiles = [1, 32768, 2, 32768, 8, 32768]; // Supported profiles. 221 maxInst = 4; // Maximum number of instances. 222 isSoftwareCodec = false; // Whether it is software codec. 223 processModeMask = []; // Codec processing mode. 224 capsMask = [0x01]; // CodecCapsMask configuration. 225 minBitRate = 1; // Minimum bit rate. 226 maxBitRate = 40000000; // Maximum bit rate. 227 minWidth = 176; // Minimum video width. 228 minHeight = 144;; // Minimum video height. 229 maxWidth = 1920; // Maximum video width. 230 maxHeight = 1088; // Maximum video height. 231 widthAlignment = 16; // Horizontal alignment. 232 heightAlignment = 8; // Vertical alignment. 233 minBlockCount = 0xFFFFFFFF; 234 maxBlockCount = 0xFFFFFFFF; 235 minBlocksPerSecond = 0xFFFFFFFF; 236 maxBlocksPerSecond = 0xFFFFFFFF; 237 blockSizeWidth = 0xFFFFFFFF; 238 blockSizeHeight = 0xFFFFFFFF; 239 supportPixelFmts = [28, 24, 20, 12]; // List of colors supported by the display. 240 measuredFrameRate = [320, 240, 165, 165, 720, 480, 149, 149, 1280, 720, 73, 73, 1920, 1080, 18, 18]; 241 bitRateMode = [1, 2]; // Bit rate mode. 242 minFrameRate = 0; // Frame rate. 243 maxFrameRate = 0; 244 } 245 ``` 246 247### Development Example 248After completing codec module driver adaptation, use the HDI APIs provided by the codec module for further development. The codec HDI provides the following features: 249 250- Provides codec HDI APIs for video services to implement encoding and decoding for video services. 251- Provides standard interfaces for device developers to ensure that the OEM vendors comply with the HDI adapter standard. This promises a healthy evolution of the ecosystem. 252 253The development procedure is as follows: 254 2551. Initialize the driver, including initializing the instances, callbacks, and component. 2562. Set codec parameters and information such as the video width, height, and bit rate. 2573. Apply for input and output buffers. 2584. Flip codec buffers, enable the component to enter the **CODEC_STATE_EXECUTING** state, and process the callbacks. 2595. Deinitialize the interface instance, destroy the buffers, close the component, and releases all interface instances. 260 261#### Initializing the Driver 262Initialize the interface instance and callbacks, and create a component. 263```cpp 264// Initialize the codec HDI ComponentManager instance. 265omxMgr_ = ICodecComponentManager::Get(false); 266if ((omxMgr_ == nullptr)) { 267 HDF_LOGE("%{public}s omxMgr_ is null", __func__); 268 return false; 269} 270// Initialize the callback. 271callback_ = new CodecHdiCallback(shared_from_this()); 272if ((callback_ == nullptr)) { 273 HDF_LOGE("%{public}s callback_ is null", __func__); 274 return false; 275} 276// Create a component instance. 277err = omxMgr_->CreateComponent(client_, componentId_, compName, reinterpret_cast<int64_t>(this), callback_); 278if (err != HDF_SUCCESS) { 279 HDF_LOGE("%{public}s failed to CreateComponent", __func__); 280 return false; 281} 282``` 283 284#### Setting Codec Parameters and Configuration 285Set the width and height of the input and output data, input data format, and output data format. 286```cpp 287// Set the width and height of the input image. 288OMX_PARAM_PORTDEFINITIONTYPE param; 289if (util_->InitParam(param) != HDF_SUCCESS) { 290 return HDF_FAILURE; 291} 292param.nPortIndex = static_cast<uint32_t>(PortIndex::PORT_INDEX_INPUT); 293 294std::vector<int8_t> inVec, outVec; 295util_->ObjectToVector(param, inVec); 296auto err = client_->GetParameter(OMX_IndexParamPortDefinition, inVec, outVec); 297if (err != HDF_SUCCESS) { 298 HDF_LOGE("%{public}s failed PortIndex::PORT_INDEX_INPUT, index is OMX_IndexParamPortDefinition", __func__); 299 return err; 300} 301util_->VectorToObject(outVec, param); 302 303HDF_LOGI("PortIndex::PORT_INDEX_INPUT: eCompressionFormat = %{public}d, eColorFormat = %{public}d ", 304 param.format.video.eCompressionFormat, param.format.video.eColorFormat); 305util_->setParmValue(param, width_, height_, stride_); 306util_->ObjectToVector(param, inVec); 307err = client_->SetParameter(OMX_IndexParamPortDefinition, inVec); 308if (err != HDF_SUCCESS) { 309 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_INPUT, index is OMX_IndexParamPortDefinition", __func__); 310 return err; 311} 312// Set the output width, height, and format. 313if (util_->InitParam(param) != HDF_SUCCESS) { 314 return HDF_FAILURE; 315} 316param.nPortIndex = static_cast<uint32_t>(PortIndex::PORT_INDEX_OUTPUT); 317util_->ObjectToVector(param, inVec); 318err = client_->GetParameter(OMX_IndexParamPortDefinition, inVec, outVec); 319if (err != HDF_SUCCESS) { 320 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_OUTPUT, index is OMX_IndexParamPortDefinition", 321 __func__); 322 return err; 323} 324util_->VectorToObject(outVec, param); 325 326HDF_LOGI("PortIndex::PORT_INDEX_OUTPUT eCompressionFormat = %{public}d, eColorFormat=%{public}d", 327 param.format.video.eCompressionFormat, param.format.video.eColorFormat); 328util_->setParmValue(param, width_, height_, stride_); 329param.format.video.eColorFormat = AV_COLOR_FORMAT; // Set the output data format to YUV420SP. 330err = client_->SetParameter(OMX_IndexParamPortDefinition, inVec); 331if (err != HDF_SUCCESS) { 332 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_OUTPUT, index is OMX_IndexParamPortDefinition", 333 __func__); 334 return err; 335} 336// Set the input data format to H.264/H.265. 337OMX_VIDEO_PARAM_PORTFORMATTYPE param; 338if (util_->InitParam(param) != HDF_SUCCESS) { 339 return false; 340} 341param.nPortIndex = (uint32_t)PortIndex::PORT_INDEX_INPUT; 342std::vector<int8_t> inVec, outVec; 343util_->ObjectToVector(param, inVec); 344auto err = client_->GetParameter(OMX_IndexParamVideoPortFormat, inVec, outVec); 345if (err != HDF_SUCCESS) { 346 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_INPUT", __func__); 347 return false; 348} 349util_->VectorToObject(outVec, param); 350 351HDF_LOGI("set Format PortIndex::PORT_INDEX_INPUT eCompressionFormat = %{public}d, eColorFormat=%{public}d", 352 param.eCompressionFormat, param.eColorFormat); 353param.xFramerate = FRAME // Set the frame rate to 30. 354if (codecMime_ == codecMime::AVC) { 355 param.eCompressionFormat = OMX_VIDEO_CodingAVC; // H264 356} else { 357 param.eCompressionFormat = (OMX_VIDEO_CODINGTYPE)CODEC_OMX_VIDEO_CodingHEVC; // H265 358} 359 360util_->ObjectToVector(param, inVec); 361err = client_->SetParameter(OMX_IndexParamVideoPortFormat, inVec); 362if (err != HDF_SUCCESS) { 363 HDF_LOGE("%{public}s failed with PortIndex::PORT_INDEX_INPUT", __func__); 364 return false; 365} 366``` 367 368#### Applying for Input and Output Buffers 369Perform the following steps: 370 3711. Use **UseBuffer()** to apply for input and output buffers and save the buffer IDs. The buffer IDs can be used for subsequent buffer flipping. 3722. Check whether the corresponding port is enabled. If not, enable the port first. 3733. Use **SendCommand()** to change the component status to **CODEC_STATE_IDLE**, and wait until the operation result is obtained. 374```cpp 375// Apply for the input buffer. 376auto err = UseBufferOnPort(PortIndex::PORT_INDEX_INPUT); 377if (err != HDF_SUCCESS) { 378 HDF_LOGE("%{public}s UseBufferOnPort PortIndex::PORT_INDEX_INPUT error", __func__); 379 return false; 380} 381// Apply for the output buffer. 382err = UseBufferOnPort(PortIndex::PORT_INDEX_OUTPUT); 383if (err != HDF_SUCCESS) { 384 HDF_LOGE("%{public}s UseBufferOnPort PortIndex::PORT_INDEX_OUTPUT error", __func__); 385 return false; 386} 387// Enable the component to enter the OMX_StateIdle state. 388std::vector<int8_t> cmdData; 389auto err = client_->SendCommand(CODEC_COMMAND_STATE_SET, CODEC_STATE_IDLE, cmdData); 390if (err != HDF_SUCCESS) { 391 HDF_LOGE("%{public}s failed to SendCommand with CODEC_COMMAND_STATE_SET:CODEC_STATE_IDLE", __func__); 392 return false; 393} 394``` 395 396Implement **UseBufferOnPort()** as follows: 397 398```cpp 399int32_t CodecHdiDecode::UseBufferOnPort(PortIndex portIndex) 400{ 401 HDF_LOGI("%{public}s enter, portIndex = %{public}d", __func__, portIndex); 402 int bufferSize = 0; 403 int bufferCount = 0; 404 bool PortEnable = false; 405 // Obtain parameters of the port buffer. 406 OMX_PARAM_PORTDEFINITIONTYPE param; 407 if (util_->InitParam(param) != HDF_SUCCESS) { 408 return HDF_FAILURE; 409 } 410 param.nPortIndex = static_cast<OMX_U32>(portIndex); 411 412 std::vector<int8_t> inVec, outVec; 413 util_->ObjectToVector(param, inVec); 414 auto err = client_->GetParameter(OMX_IndexParamPortDefinition, inVec, outVec); 415 if (err != HDF_SUCCESS) { 416 HDF_LOGE("%{public}s failed to GetParameter with OMX_IndexParamPortDefinition : portIndex[%{public}d]", 417 __func__, portIndex); 418 return err; 419 } 420 util_->VectorToObject(outVec, param); 421 422 bufferSize = param.nBufferSize; 423 bufferCount = param.nBufferCountActual; 424 portEnable = param.bEnabled; 425 HDF_LOGI("buffer index [%{public}d], buffer size [%{public}d], " 426 "buffer count [%{public}d], portEnable[%{public}d], ret [%{public}d]", 427 portIndex, bufferSize, bufferCount, portEnable, err); 428 // Set the port buffer. 429 if (useBufferHandle_ && portIndex == PortIndex::PORT_INDEX_OUTPUT) { 430 err = UseBufferHandle(bufferCount, bufferSize); 431 } else { 432 err = UseBufferOnPort(portIndex, bufferCount, bufferSize); 433 } 434 // Check whether the port is available. 435 if (!portEnable) { 436 err = client_->SendCommand(CODEC_COMMAND_PORT_ENABLE, static_cast<uint32_t>(portIndex), {}); 437 if (err != HDF_SUCCESS) { 438 HDF_LOGE("%{public}s SendCommand OMX_CommandPortEnable::PortIndex::PORT_INDEX_INPUT error", __func__); 439 return err; 440 } 441 } 442 return HDF_SUCCESS; 443} 444 445int32_t CodecHdiDecode::UseBufferOnPort(PortIndex portIndex, int bufferCount, int bufferSize) 446{ 447 if (bufferCount <= 0 || bufferSize <= 0) { 448 HDF_LOGE("UseBufferOnPort bufferCount <= 0 or bufferSize <= 0"); 449 return HDF_ERR_INVALID_PARAM; 450 } 451 for (int i = 0; i < bufferCount; i++) { 452 std::shared_ptr<OmxCodecBuffer> omxBuffer = std::make_shared<OmxCodecBuffer>(); 453 omxBuffer->size = sizeof(OmxCodecBuffer); 454 omxBuffer->version.s.nVersionMajor = 1; 455 omxBuffer->bufferType = CODEC_BUFFER_TYPE_AVSHARE_MEM_FD; 456 int fd = AshmemCreate(0, bufferSize); 457 shared_ptr<Ashmem> sharedMem = make_shared<Ashmem>(fd, bufferSize); 458 omxBuffer->fd = fd; 459 omxBuffer->bufferhandle = nullptr; 460 omxBuffer->allocLen = bufferSize; 461 omxBuffer->fenceFd = -1; 462 omxBuffer->pts = 0; 463 omxBuffer->flag = 0; 464 465 if (portIndex == PortIndex::PORT_INDEX_INPUT) { 466 omxBuffer->type = READ_ONLY_TYPE; // ReadOnly 467 sharedMem->MapReadAndWriteAshmem(); 468 } else { 469 omxBuffer->type = READ_WRITE_TYPE; 470 sharedMem->MapReadOnlyAshmem(); 471 } 472 OmxCodecBuffer outBuffer; 473 auto err = client_->UseBuffer((uint32_t)portIndex, *omxBuffer.get(), outBuffer); 474 if (err != HDF_SUCCESS) { 475 HDF_LOGE("%{public}s failed to UseBuffer with portIndex[%{public}d]", __func__, portIndex); 476 sharedMem->UnmapAshmem(); 477 sharedMem->CloseAshmem(); 478 sharedMem = nullptr; 479 return err; 480 } 481 omxBuffer->bufferId = outBuffer.bufferId; 482 omxBuffer->fd = -1; 483 HDF_LOGI("UseBuffer returned bufferID [%{public}d]", omxBuffer->bufferId); 484 485 std::shared_ptr<BufferInfo> bufferInfo = std::make_shared<BufferInfo>(); 486 bufferInfo->omxBuffer = omxBuffer; 487 bufferInfo->avSharedPtr = sharedMem; 488 bufferInfo->portIndex = portIndex; 489 omxBuffers_.emplace(std::make_pair(omxBuffer->bufferId, bufferInfo)); 490 if (portIndex == PortIndex::PORT_INDEX_INPUT) { 491 unUsedInBuffers_.push_back(omxBuffer->bufferId); 492 } else { 493 unUsedOutBuffers_.push_back(omxBuffer->bufferId); 494 } 495 } 496 497 return HDF_SUCCESS; 498} 499``` 500 501#### Flipping Codec Buffers 502Set the component to the **CODEC_STATE_EXECUTING** state, fill the input buffer, read data from the output buffer, and flip the buffers. 503 504```cpp 505// Set the component to the OMX_StateExecuting state and start buffer flipping. 506HDF_LOGI("...command to CODEC_STATE_EXECUTING...."); 507auto err = client_->SendCommand(CODEC_COMMAND_STATE_SET, CODEC_STATE_EXECUTING, {}); 508if (err != HDF_SUCCESS) { 509 HDF_LOGE("%{public}s failed to SendCommand with CODEC_COMMAND_STATE_SET:CODEC_STATE_IDLE", __func__); 510 return; 511} 512// Set the output buffer to fill. 513if (!FillAllTheBuffer()) { 514 HDF_LOGE("%{public}s FillAllTheBuffer error", __func__); 515 return; 516} 517// Fill the input buffer. 518auto t1 = std::chrono::system_clock::now(); 519bool eosFlag = false; 520while (!eosFlag) { 521 if (this->exit_) { 522 break; 523 } 524 int bufferID = GetFreeBufferId(); 525 if (bufferID < 0) { 526 usleep(10000); // 10000 for wait 10ms 527 continue; 528 } 529 auto iter = omxBuffers_.find(bufferID); 530 if (iter == omxBuffers_.end()) { 531 continue; 532 } 533 auto bufferInfo = iter->second; 534 void *sharedAddr = const_cast<void *>(bufferInfo->avSharedPtr->ReadFromAshmem(0, 0)); 535 eosFlag = this->ReadOnePacket(fpIn_, static_cast<char *>(sharedAddr), bufferInfo->omxBuffer->filledLen); 536 bufferInfo->omxBuffer->offset = 0; 537 if (eosFlag) { 538 bufferInfo->omxBuffer->flag = OMX_BUFFERFLAG_EOS; 539 } 540 err = client_->EmptyThisBuffer(*bufferInfo->omxBuffer.get()); 541 if (err != HDF_SUCCESS) { 542 HDF_LOGE("%{public}s EmptyThisBuffer error", __func__); 543 return; 544 } 545} 546// Wait. 547while (!this->exit_) { 548 usleep(10000); // 10000 for wait 10ms 549} 550// Enable the component to enter the OMX_StateIdle state after decoding. 551auto t2 = std::chrono::system_clock::now(); 552std::chrono::duration<double> diff = t2 - t1; 553HDF_LOGI("cost %{public}f, count=%{public}d", diff.count(), count_); 554(void)client_->SendCommand(CODEC_COMMAND_STATE_SET, CODEC_STATE_IDLE, {}); 555return; 556} 557``` 558 559Automatic framing is not supported in rk OMX decoding. Therefore, you need to manually divide data into frames. Currently, data is divided into frames from code 0x000001 or 0x00000001 and sent to the server for processing. The sample code is as follows: 560 561```cpp 562// Read a file by frame. 563bool CodecHdiDecode::ReadOnePacket(FILE *fp, char *buf, uint32_t &filledCount) 564{ 565 // Read the start code. 566 size_t t = fread(buf, 1, START_CODE_SIZE_FRAME, fp); 567 if (t < START_CODE_SIZE_FRAME) { 568 return true; 569 } 570 char *temp = buf; 571 temp += START_CODE_SIZE_FRAME; 572 bool ret = true; 573 while (!feof(fp)) { 574 (void)fread(temp, 1, 1, fp); 575 if (*temp != START_CODE) { 576 temp++; 577 continue; 578 } 579 // Check the start code. 580 if ((temp[START_CODE_OFFSET_ONE] == 0) && (temp[START_CODE_OFFSET_SEC] == 0) && 581 (temp[START_CODE_OFFSET_THIRD] == 0)) { 582 fseek(fp, -START_CODE_SIZE_FRAME, SEEK_CUR); 583 temp -= (START_CODE_SIZE_FRAME - 1); 584 ret = false; 585 break; 586 } 587 if ((temp[START_CODE_OFFSET_ONE] == 0) && (temp[START_CODE_OFFSET_SEC] == 0)) { 588 fseek(fp, -START_CODE_SIZE_SLICE, SEEK_CUR); 589 temp -= (START_CODE_SIZE_SLICE - 1); 590 ret = false; 591 break; 592 } 593 temp++; 594 } 595 filledCount = (temp - buf); 596 return ret; 597} 598``` 599 600The codec HDI provides the following callbacks: 601 602- **EventHandler**: Called when a command is executed. For example, when the command for changing the component state from **CODEC_STATE_IDLE** to **CODEC_STATE_EXECUTING** is executed, this callback is invoked to return the result. 603- **EmptyBufferDone**: Called when the input data is consumed. If the client needs to fill data to encode or decode, it must call **EmptyThisBuffer()** again. 604- **FillBufferDone**: Called when the output data is filled. If the client needs to read the encoded or decoded data, it must call **FillThisBuffer()** again. 605 606```cpp 607// EmptyBufferDone example 608int32_t CodecHdiDecode::OnEmptyBufferDone(const struct OmxCodecBuffer &buffer) 609{ 610 HDF_LOGI("OnEmptyBufferDone, bufferId [%{public}d]", buffer.bufferId); 611 unique_lock<mutex> ulk(lockInputBuffers_); 612 unUsedInBuffers_.push_back(buffer.bufferId); 613 return HDF_SUCCESS; 614} 615// FillBufferDone example 616int32_t CodecHdiDecode::OnFillBufferDone(const struct OmxCodecBuffer &buffer) 617{ 618 HDF_LOGI("OnFillBufferDone, bufferId [%{public}d]", buffer.bufferId); 619 if (exit_) { 620 return HDF_SUCCESS; 621 } 622 623 auto iter = omxBuffers_.find(buffer.bufferId); 624 if ((iter == omxBuffers_.end()) || (iter->second == nullptr)) { 625 return HDF_SUCCESS; 626 } 627 count_++; 628 // read buffer 629 auto bufferInfo = iter->second; 630 if (bufferInfo->avSharedPtr != nullptr) { 631 const void *addr = bufferInfo->avSharedPtr->ReadFromAshmem(buffer.filledLen, buffer.offset); 632 (void)fwrite(addr, 1, buffer.filledLen, fpOut_); 633 } else if (bufferInfo->bufferHandle != nullptr && gralloc_ != nullptr) { 634 gralloc_->Mmap(*bufferInfo->bufferHandle); 635 (void)fwrite(bufferInfo->bufferHandle->virAddr, 1, buffer.filledLen, fpOut_); 636 gralloc_->Unmap(*bufferInfo->bufferHandle); 637 } 638 639 (void)fflush(fpOut_); 640 if (buffer.flag == OMX_BUFFERFLAG_EOS) { 641 // end 642 exit_ = true; 643 HDF_LOGI("OnFillBufferDone the END coming"); 644 return HDF_SUCCESS; 645 } 646 // call fillthisbuffer again 647 auto err = client_->FillThisBuffer(*bufferInfo->omxBuffer.get()); 648 if (err != HDF_SUCCESS) { 649 HDF_LOGE("%{public}s FillThisBuffer error", __func__); 650 return HDF_SUCCESS; 651 } 652 return HDF_SUCCESS; 653} 654// EventHandler example 655int32_t CodecHdiDecode::EventHandler(CodecEventType event, const EventInfo &info) 656{ 657 switch (event) { 658 case CODEC_EVENT_CMD_COMPLETE: { 659 CodecCommandType cmd = (CodecCommandType)info.data1; 660 if (CODEC_COMMAND_STATE_SET == cmd) { 661 HDF_LOGI("CODEC_COMMAND_STATE_SET reached, status is %{public}d", info.data2); 662 this->OnStatusChanged(); 663 } 664 break; 665 } 666 case OMX_EventPortSettingsChanged: { 667 HDF_LOGI("OMX_EventPortSeetingsChanged reached"); 668 this->HandleEventPortSettingsChanged(info.data1, info.data2); 669 } 670 671 default: 672 break; 673 } 674 675 return HDF_SUCCESS; 676} 677``` 678 679#### Destroying a Component 680Change the component state to **CODEC_STATE_IDLE**, release the input and output buffers, change the component state to **CODEC_STATE_LOADED**, and call **DestoryComponent** to destroy the component. 681 682##### Releasing Buffers 683 684```cpp 685// Change the component state to OMX_StateLoaded. 686client_->SendCommand(CODEC_COMMAND_STATE_SET, CODEC_STATE_LOADED, {}); 687 688// Release all buffers in use. 689auto iter = omxBuffers_.begin(); 690while (iter != omxBuffers_.end()) { 691 auto bufferInfo = iter->second; 692 iter = omxBuffers_.erase(iter); 693 (void)client_->FreeBuffer((uint32_t)bufferInfo->portIndex, *bufferInfo->omxBuffer.get()); 694 bufferInfo = nullptr; 695} 696 697unUsedInBuffers_.clear(); 698unUsedOutBuffers_.clear(); 699 700// After the buffers are released, the component enters the OMX_StateLoaded state. 701CodecStateType status = CODEC_STATE_INVALID; 702int32_t err = HDF_SUCCESS; 703int32_t tryCount = 3; 704do { 705 err = client_->GetState(status); 706 if (err != HDF_SUCCESS) { 707 HDF_LOGE("%s GetState error [%{public}x]", __func__, err); 708 break; 709 } 710 if (status != CODEC_STATE_LOADED) { 711 HDF_LOGI("Wait for OMX_StateLoaded status"); 712 this->WaitForStatusChanged(); 713 } 714 tryCount--; 715} while ((status != CODEC_STATE_LOADED) && (tryCount > 0)); 716``` 717 718##### Destroying a Component Instance 719 720```cpp 721// Destroy a component instance. 722void CodecHdiDecode::Release() 723{ 724 omxMgr_->DestoryComponent(componentId_); 725 client_ = nullptr; 726 callback_ = nullptr; 727 omxMgr_ = nullptr; 728} 729``` 730 731# FAQs 732 733## Green Screens Displayed During the Decoding Process 734 735**Symptom** 736 737Green screens are displayed during the decoding process. 738 739**Possible Causes** 740 741OpenMAX does not support framing. 742 743**Solution** 744 745When **EmptyThisBuffer** is call, only one frame can be passed in at a time. 746 747## Only Green Screen Displayed During the Decoding Process 748 749**Symptom** 750 751Decoding fails, and all the frames decoded cannot be played. 752 753**Possible Causes** 754 755For the data in AVCC format, the first frame to be processed must be extra_data. 756 757**Solution** 758 759Write sps and pps to the buffer in extra_data format, and set the buffer flag to **OMX_BUFFERFLAG_EXTRADATA**. 760 761## Failed to Play the Encoded Video 762 763**Symptom** 764 765After the generated video stream (H.264 stream) is written to a file, the video stream cannot be played by FFplay. 766 767**Possible Causes** 768 7691. The **xFramerate** parameter of the output port is incorrectly set. 7702. The **OMX_VIDEO_PARAM_AVCTYPE** parameter is correctly set. 771 772**Solution** 773 774View the **codec_host** log generated during encoding, search for "encode params init settings", and check for incorrect parameters. If **framerate** is **0**, **xFramerate** is incorrectly set. In this case, move the frame rate leftwards by 16 bits. <br>In other cases, correct the setting of **OMX_VIDEO_PARAM_AVCTYPE**. 775 776 777# Reference 778 779For more information, see [Codec](https://gitee.com/openharmony/drivers_peripheral/tree/master/codec). 780
