1# Synchronous Audio Decoding 2 3<!--Kit: AVCodec Kit--> 4<!--Subsystem: Multimedia--> 5<!--Owner: @mr-chencxy--> 6<!--Designer: @dpy2650---> 7<!--Tester: @baotianhao--> 8<!--Adviser: @zengyawen--> 9 10Starting from API version 20, audio decoding in synchronous mode is supported. 11 12You can call native APIs to perform audio decoding in synchronous mode, which decodes media data into PCM streams. 13 14For details about the supported decoding capabilities, see [AVCodec Supported Formats](avcodec-support-formats.md#audio-decoding). 15 16**When to Use** 17 18Asynchronous mode is generally recommended for most use cases. For details, see [Audio Decoding](audio-decoding.md). Synchronous mode can be used if you need to actively request buffers for frame delivery. 19Decoding an audio/video file into a PCM stream typically involves the following steps: [media data demultiplexing](audio-video-demuxer.md) -> audio decoding. 20 21This guide outlines the audio decoding process: inputting audio frames and decoding them into PCM streams. 22 23## Development Guidelines 24 25Read [AudioCodec](../../reference/apis-avcodec-kit/_audio_codec.md) for the API reference. 26 27Refer to the code snippet below to complete the entire audio decoding process, including creating a decoder, setting decoding parameters (such as the sample rate, bit rate, and audio channel count), and starting, refreshing, resetting, and destroying the decoder. 28 29During application development, you must call the APIs in the defined sequence. Otherwise, an exception or undefined behavior may occur. 30 31The figure below shows the call relationship of audio decoding in synchronous mode. 32 33- The dotted line indicates an optional operation. 34 35- The solid line indicates a mandatory operation. 36 37 38 39### Linking the Dynamic Libraries in the CMake Script 40 41```cmake 42target_link_libraries(sample PUBLIC libnative_media_codecbase.so) 43target_link_libraries(sample PUBLIC libnative_media_core.so) 44target_link_libraries(sample PUBLIC libnative_media_acodec.so) 45``` 46> **NOTE** 47> 48> The word **sample** in the preceding code snippet is only an example. Use the actual project directory name. 49 50### How to Develop 51 521. Add the header files and namespace. 53 54 ```cpp 55 #include <multimedia/player_framework/native_avcodec_audiocodec.h> 56 #include <multimedia/native_audio_channel_layout.h> 57 #include <multimedia/player_framework/native_avcapability.h> 58 #include <multimedia/player_framework/native_avcodec_base.h> 59 #include <multimedia/player_framework/native_avformat.h> 60 #include <multimedia/player_framework/native_avbuffer.h> 61 62 // Namespace of the C++ standard library. 63 using namespace std; 64 ``` 65 662. Create a decoder instance. In the code snippet below, **OH_AVCodec *** is the pointer to the decoder instance created. 67 68 You can create a decoder by MIME type or codec name. 69 70 Method 1: Create a decoder by MIME type. 71 ```c++ 72 // Create a decoder by MIME type. The following example creates an AAC decoder. The second parameter is set to false to indicate decoding. 73 OH_AVCodec *audioDec_ = OH_AudioCodec_CreateByMime(OH_AVCODEC_MIMETYPE_AUDIO_AAC, false); 74 ``` 75 Method 2: Create a decoder by codec name. 76 ```c++ 77 // Create a decoder by name. 78 OH_AVCapability *capability = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_AUDIO_AAC, false); 79 const char *name = OH_AVCapability_GetName(capability); 80 OH_AVCodec *audioDec_ = OH_AudioCodec_CreateByName(name); 81 ``` 82 833. (Optional) Call **OH_AudioCodec_SetDecryptionConfig** to set the decryption configuration. 84 85 Call this API after the media key system information is obtained but before **Prepare()** is called. For details about how to obtain such information, see step 4 in [Media Data Demultiplexing](audio-video-demuxer.md). 86 87 For details about DRM APIs, see [DRM](../../reference/apis-drm-kit/capi-drm.md). 88 89 Add the header files. 90 91 ```c++ 92#include <multimedia/drm_framework/native_mediakeysystem.h> 93 #include <multimedia/drm_framework/native_mediakeysession.h> 94 #include <multimedia/drm_framework/native_drm_err.h> 95 #include <multimedia/drm_framework/native_drm_common.h> 96 ``` 97 Link the dynamic libraries in the CMake script. 98 99 ``` cmake 100target_link_libraries(sample PUBLIC libnative_drm.so) 101 ``` 102 103 The sample code is as follows: 104```c++ 105 // Create a media key system based on the media key system information. The following uses com.clearplay.drm as an example. 106 MediaKeySystem *system = nullptr; 107 int32_t ret = OH_MediaKeySystem_Create("com.clearplay.drm", &system); 108 if (system == nullptr) { 109 printf("create media key system failed"); 110 return; 111 } 112 113 // Create a media key session. 114MediaKeySession *session = nullptr; 115 DRM_ContentProtectionLevel contentProtectionLevel = CONTENT_PROTECTION_LEVEL_SW_CRYPTO; 116 ret = OH_MediaKeySystem_CreateMediaKeySession(system, &contentProtectionLevel, &session); 117 if (ret != DRM_OK) { 118 // If the creation fails, refer to the DRM interface document and check logs. 119 printf("create media key session failed."); 120 return; 121 } 122 if (session == nullptr) { 123 printf("media key session is nullptr."); 124 return; 125 } 126 // Generate a media key request and set the response to the media key request. 127 // Set the decryption configuration, that is, assign the decryption session to the decoder. Currently, audio decryption does not support secure decoders. Therefore, set the secureAudio parameter to false. 128 bool secureAudio = false; 129 ret = OH_AudioCodec_SetDecryptionConfig(audioDec_, session, secureAudio); 130 ``` 131 1324. Call **OH_AudioCodec_Configure()** to configure the decoder. 133 134 Key values of configuration options are described as follows: 135 136 | Key | Description | AAC | FLAC| Vorbis | MPEG | G711mu | AMR (AMR-NB and AMR-WB) | APE | G711a | 137 | ---------------------------- | :--------------: | :--------------------------------: | :--: | :--------------------------------: | :--: | :-----------------: | :-------------------------------: | :--: | :----------------------: | 138 | OH_MD_KEY_ENABLE_SYNC_MODE | Synchronous mode. It must be set to **1** if synchronization mode is enabled. | Mandatory for synchronous mode| Mandatory for synchronous mode| Mandatory for synchronous mode| Mandatory for synchronous mode| Mandatory for synchronous mode| Mandatory for synchronous mode| Mandatory for synchronous mode| Mandatory for synchronous mode| 139 | OH_MD_KEY_AUD_SAMPLE_RATE | Sample rate. | Mandatory | Mandatory| Mandatory | Mandatory| Mandatory | Mandatory | Mandatory| Mandatory | 140 | OH_MD_KEY_AUD_CHANNEL_COUNT | Audio channel count. | Mandatory | Mandatory| Mandatory | Mandatory| Mandatory | Mandatory | Mandatory| Mandatory | 141 | OH_MD_KEY_MAX_INPUT_SIZE | Maximum input size. | Optional | Optional| Optional | Optional| Optional | Optional | Optional| Optional | 142 | OH_MD_KEY_AAC_IS_ADTS | ADTS or not. | Optional (defaults to **1**) | - | - | - | - | - | - | - | 143 | OH_MD_KEY_AUDIO_SAMPLE_FORMAT | Output audio stream format. | Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional| Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional| Optional (default: SAMPLE_S16LE)| Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional| Optional (default: SAMPLE_S16LE)| 144 | OH_MD_KEY_BITRATE | Bit rate. | Optional | Optional| Optional | Optional| Optional | Optional | Optional| Optional | 145 | OH_MD_KEY_IDENTIFICATION_HEADER | ID header. | - | - | Mandatory (Either this parameter or **MD_KEY_CODEC_CONFIG** must be set.) | - | - | - | - | - | 146 | OH_MD_KEY_SETUP_HEADER | Setup header. | - | - | Mandatory (Either this parameter or **MD_KEY_CODEC_CONFIG** must be set.) | - | - | - | - | - | 147 | OH_MD_KEY_CODEC_CONFIG | Codec-specific data.| Optional | - | Mandatory (Either this parameter or the combination of **MD_KEY_IDENTIFICATION_HEADER** and **MD_KEY_SETUP_HEADER** must be selected.) | - | - | - | Optional| - | 148 149 The sample below lists the value range of each audio decoding type. 150 151 | Audio Decoding Type| Sample Rate (Hz) | Audio Channel Count| 152 | ----------- | ---------------------------------------------------------------------------------------------- | :----: | 153 | AAC | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000 | 1–8 | 154 | FLAC | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 192000 | 1–8 | 155 | Vorbis | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 176400, 192000| 1–8 | 156 | MPEG (MP3) | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 | 1–2 | 157 | G711mu | 8000 | 1 | 158 | AMR (amrnb) | 8000 | 1 | 159 | AMR (amrwb) | 16000 | 1 | 160 | APE | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 176400, 192000| 1–2 | 161 | G711a | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 | 1–6 | 162 <!--RP4--> 163 <!--RP4End--> 164 165 ```c++ 166 // (Mandatory) Configure the audio sample rate. 167 constexpr uint32_t DEFAULT_SAMPLERATE = 44100; 168 // (Mandatory) Configure the audio channel count. 169 constexpr uint32_t DEFAULT_CHANNEL_COUNT = 2; 170 // Configure whether to use ADTS decoding (optional for AAC decoding). 171 constexpr uint32_t DEFAULT_AAC_TYPE = 1; 172 OH_AVFormat *format = OH_AVFormat_Create(); 173 // Set the format. 174 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_SAMPLE_RATE, DEFAULT_SAMPLERATE); 175 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_CHANNEL_COUNT, DEFAULT_CHANNEL_COUNT); 176 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AAC_IS_ADTS, DEFAULT_AAC_TYPE); 177 OH_AVFormat_SetIntValue(format, OH_MD_KEY_ENABLE_SYNC_MODE, 1); // Set synchronous mode. 178 // Configure the decoder. 179 OH_AVErrCode ret = OH_AudioCodec_Configure(audioDec_, format); 180 if (ret != AV_ERR_OK) { 181 // Handle exceptions. 182 } 183 ``` 184 1855. Call **OH_AudioCodec_Prepare()** to prepare internal resources for the decoder. 186 187 ```c++ 188 OH_AVErrCode ret = OH_AudioCodec_Prepare(audioDec_); 189 if (ret != AV_ERR_OK) { 190 // Handle exceptions. 191 } 192 ``` 193 1946. Call **OH_AudioCodec_Start()** to start the decoder. 195 196 Add the header file. 197 ```c++ 198 #include <fstream> 199 ``` 200 201 The sample code is as follows: 202 ```c++ 203 ifstream inputFile_; 204 ofstream outFile_; 205 206 // Set the input file path based on the actual situation. To demonstrate audio decoding, the input file here contains audio frames and related information (not a directly playable audio file). 207 const char* inputFilePath = "/"; 208 // Set the output file path based on the actual situation. This guide only demonstrates audio decoding; decoded PCM data is saved to a file without further processing. 209 const char* outputFilePath = "/"; 210 // Open the path of the binary file to be decoded. 211 inputFile_.open(inputFilePath, ios::in | ios::binary); 212 // Set the path of the output file. 213 outFile_.open(outputFilePath, ios::out | ios::binary); 214 // Start decoding. 215 OH_AVErrCode ret = OH_AudioCodec_Start(audioDec_); 216 if (ret != AV_ERR_OK) { 217 // Handle exceptions. 218 } 219 ``` 220 2217. (Optional) Call **OH_AVCencInfo_SetAVBuffer()** to set the Common Encryption Scheme (CENC) information. 222 223 If the content being played is DRM encrypted and [demultiplexing](audio-video-demuxer.md) is performed by the upper-layer application, call **OH_AVCencInfo_SetAVBuffer()** to set the CENC information to the AVBuffer for media data decryption. 224 225 Add the header file. 226 227 ```c++ 228 #include <multimedia/player_framework/native_cencinfo.h> 229 ``` 230 Link the dynamic libraries in the CMake script. 231 232 ``` cmake 233 target_link_libraries(sample PUBLIC libnative_media_avcencinfo.so) 234 ``` 235 236 The sample code is as follows: 237 ```c++ 238 auto buffer = signal_->inBufferQueue_.front(); 239 uint32_t keyIdLen = DRM_KEY_ID_SIZE; 240 uint8_t keyId[] = { 241 0xd4, 0xb2, 0x01, 0xe4, 0x61, 0xc8, 0x98, 0x96, 242 0xcf, 0x05, 0x22, 0x39, 0x8d, 0x09, 0xe6, 0x28}; 243 uint32_t ivLen = DRM_KEY_IV_SIZE; 244 uint8_t iv[] = { 245 0xbf, 0x77, 0xed, 0x51, 0x81, 0xde, 0x36, 0x3e, 246 0x52, 0xf7, 0x20, 0x4f, 0x72, 0x14, 0xa3, 0x95}; 247 uint32_t encryptedBlockCount = 0; 248 uint32_t skippedBlockCount = 0; 249 uint32_t firstEncryptedOffset = 0; 250 uint32_t subsampleCount = 1; 251 DrmSubsample subsamples[1] = { {0x10, 0x16} }; 252 // Create a CencInfo instance. 253 OH_AVCencInfo *cencInfo = OH_AVCencInfo_Create(); 254 if (cencInfo == nullptr) { 255 // Handle exceptions. 256 } 257 // Set the decryption algorithm. 258 OH_AVErrCode errNo = OH_AVCencInfo_SetAlgorithm(cencInfo, DRM_ALG_CENC_AES_CTR); 259 if (errNo != AV_ERR_OK) { 260 // Handle exceptions. 261 } 262 // Set KeyId and Iv. 263 errNo = OH_AVCencInfo_SetKeyIdAndIv(cencInfo, keyId, keyIdLen, iv, ivLen); 264 if (errNo != AV_ERR_OK) { 265 // Handle exceptions. 266 } 267 // Set the sample information. 268 errNo = OH_AVCencInfo_SetSubsampleInfo(cencInfo, encryptedBlockCount, skippedBlockCount, firstEncryptedOffset, 269 subsampleCount, subsamples); 270 if (errNo != AV_ERR_OK) { 271 // Handle exceptions. 272 } 273 // Set the mode. KeyId, Iv, and SubSamples have been set. 274 errNo = OH_AVCencInfo_SetMode(cencInfo, DRM_CENC_INFO_KEY_IV_SUBSAMPLES_SET); 275 if (errNo != AV_ERR_OK) { 276 // Handle exceptions. 277 } 278 // Set CencInfo to the AVBuffer. 279 errNo = OH_AVCencInfo_SetAVBuffer(cencInfo, buffer); 280 if (errNo != AV_ERR_OK) { 281 // Handle exceptions. 282 } 283 // Destroy the CencInfo instance. 284 errNo = OH_AVCencInfo_Destroy(cencInfo); 285 if (errNo != AV_ERR_OK) { 286 // Handle exceptions. 287 } 288 ``` 289 2908. Invoke synchronous mode to write the data to decode and obtain the decoded output. 291 292 Read data: 293 ```c++ 294 // size is the length of each frame of the data to decode. pts is the timestamp of each frame and is used to indicate when the audio should be played. 295 // The values of size and pts are obtained from an audio and video resource file or data stream to decode. 296 // In the case of an audio and video resource file, the values are obtained from the buffer in the demultiplexed OH_AVDemuxer_ReadSampleBuffer. 297 // In the case of a data stream, the values are obtained from the data stream provider. 298 // In this example, the values of size and pts are obtained from the test file. 299 bool DecoderFillInputBuffer(OH_AVBuffer *buffer, ifstream &inputFile) 300 { 301 OH_AVCodecBufferAttr attr; 302 memset(&attr, 0, sizeof(attr)); 303 attr.flags = AVCODEC_BUFFER_FLAGS_EOS; 304 bool finished = true; 305 do { 306 int64_t size; 307 inputFile.read(reinterpret_cast<char *>(&size), sizeof(size)); 308 if (inputFile.eof() || inputFile.gcount() != sizeof(size)) { 309 break; 310 } 311 inputFile.read(reinterpret_cast<char *>(&attr.pts), sizeof(attr.pts)); 312 if (inputFile.gcount() != sizeof(attr.pts)) { 313 break; 314 } 315 attr.size = static_cast<int32_t>(size); 316 if (attr.size > 0) { 317 inputFile.read((char *)OH_AVBuffer_GetAddr(buffer), attr.size); 318 attr.flags = AVCODEC_BUFFER_FLAGS_NONE; 319 finished = false; 320 } 321 } while (0); 322 OH_AVBuffer_SetBufferAttr(buffer, &attr); 323 return finished; 324 } 325 bool InputOneFrame(OH_AVCodec *codec, ifstream &inputFile) 326 { 327 uint32_t index = 0; 328 bool finished = true; 329 OH_AVErrCode ret = OH_AudioCodec_QueryInputBuffer(codec, &index, 20000); // 20000us 330 if (ret == AV_ERR_TRY_AGAIN_LATER) { 331 // Timeout: exception handling. The timeout period is too short, or input/output buffers are not consumed/released (causing decoding blocking). 332 return finished; 333 } 334 if (ret != AV_ERR_OK) { 335 // Handle exceptions. 336 return finished; 337 } 338 OH_AVBuffer *inputBuf = OH_AudioCodec_GetInputBuffer(codec, index); 339 if (inputBuf == nullptr) { 340 // Handle exceptions. 341 return finished; 342 } 343 finished = DecoderFillInputBuffer(inputBuf, inputFile); 344 OH_AudioCodec_PushInputBuffer(codec, index); 345 return finished; 346 } 347 348 ``` 349 350 You should fill in complete input data before calling this API. 351 352 When finished, set the flags to **AVCODEC_BUFFER_FLAGS_EOS**. 353 354 ```c++ 355 bool inputFinished = false; 356 for (;;) { 357 if (!inputFinished) { 358 inputFinished = InputOneFrame(audioDec_, inputFile_); 359 } 360 uint32_t index; 361 OH_AVErrCode ret = OH_AudioCodec_QueryOutputBuffer(audioDec_, &index, 20000); // 20000us 362 if (ret == AV_ERR_TRY_AGAIN_LATER) { 363 // Timeout: exception handling. The timeout period is too short, or input/output buffers are not consumed/released (causing decoding blocking). 364 continue; 365 } else if (ret == AV_ERR_STREAM_CHANGED) { 366 // Callback processing after the decoding output parameters are changed. The application performs processing as required. 367 OH_AVFormat *outFormat = OH_AudioCodec_GetOutputDescription(audioDec_); 368 int32_t sampleRate; 369 int32_t channelCount; 370 int32_t sampleFormat; 371 if (OH_AVFormat_GetIntValue(outFormat, OH_MD_KEY_AUD_SAMPLE_RATE, &sampleRate)) { 372 // Check whether the sample rate changes and perform processing as required. 373 } 374 if (OH_AVFormat_GetIntValue(outFormat, OH_MD_KEY_AUD_CHANNEL_COUNT, &channelCount)) { 375 // Check whether the audio channel count changes and perform processing as required. 376 } 377 if (OH_AVFormat_GetIntValue(outFormat, OH_MD_KEY_AUDIO_SAMPLE_FORMAT, &sampleFormat)) { 378 // Check whether the audio sample format changes and perform processing as required. 379 } 380 continue; 381 } 382 if (ret != AV_ERR_OK) { 383 // Handle exceptions. 384 break; 385 } 386 OH_AVBuffer *outputBuf = OH_AudioCodec_GetOutputBuffer(audioDec_, index); 387 if (outputBuf == nullptr) { 388 // Handle exceptions. 389 break; 390 } 391 OH_AVCodecBufferAttr attr; 392 if (OH_AVBuffer_GetBufferAttr(outputBuf, &attr) != AV_ERR_OK) { 393 // Handle exceptions. 394 break; 395 } 396 if (attr.flags & AVCODEC_BUFFER_FLAGS_EOS) { 397 OH_AudioCodec_FreeOutputBuffer(audioDec_, index); 398 // Decoding output completed. 399 break; 400 } 401 outFile_.write(reinterpret_cast<char *>(OH_AVBuffer_GetAddr(outputBuf)), attr.size); 402 OH_AudioCodec_FreeOutputBuffer(audioDec_, index); 403 } 404 ``` 405 4069. (Optional) Call **OH_AudioCodec_Flush()** to refresh the decoder. 407 408 After **OH_AudioCodec_Flush()** is called, the decoder remains in the running state, but the current queue is cleared and the buffer storing the decoded data is freed. The input and output buffers obtained before flushing cannot be reused. 409 410 To continue decoding, you must call **OH_AudioCodec_Start()** again. 411 412 You need to call **OH_AudioCodec_Start()** in the following cases: 413 * Previously input data is no longer needed, for example, after seeking during demultiplexing. 414 * The EOS of the file is reached. 415 * An error with **OH_AudioCodec_IsValid** set to **true** (indicating that the execution can continue) occurs. 416 417 ```c++ 418 // Refresh the decoder. 419 OH_AVErrCode ret = OH_AudioCodec_Flush(audioDec_); 420 if (ret != AV_ERR_OK) { 421 // Handle exceptions. 422 } 423 // Start decoding again. 424 ret = OH_AudioCodec_Start(audioDec_); 425 if (ret != AV_ERR_OK) { 426 // Handle exceptions. 427 } 428 ``` 429 43010. (Optional) Call **OH_AudioCodec_Reset()** to reset the decoder. 431 432 After **OH_AudioCodec_Reset()** is called, the decoder returns to the initialized state. To continue decoding, you must call **OH_AudioCodec_Configure()** and then **OH_AudioCodec_Start()**. 433 434 ```c++ 435 // Reset the decoder. 436 OH_AVErrCode ret = OH_AudioCodec_Reset(audioDec_); 437 if (ret != AV_ERR_OK) { 438 // Handle exceptions. 439 } 440 // Reconfigure the decoder. 441 ret = OH_AudioCodec_Configure(audioDec_, format); 442 if (ret != AV_ERR_OK) { 443 // Handle exceptions. 444 } 445 ``` 446 44711. Call **OH_AudioCodec_Stop()** to stop the decoder. 448 449 After the decoder is stopped, you can call **OH_AudioCodec_Start()** to start it again. The input and output buffers obtained before the decoder is stopped cannot be reused. You must obtain them again after the decoder is started. 450 451 ```c++ 452 // Stop the decoder. 453 OH_AVErrCode ret = OH_AudioCodec_Stop(audioDec_); 454 if (ret != AV_ERR_OK) { 455 // Handle exceptions. 456 } 457 ``` 458 45912. Call **OH_AudioCodec_Destroy()** to destroy the decoder instance and release resources. 460 461 > **NOTE** 462 > 463 > You only need to call this API once. 464 465 ```c++ 466 // Call OH_AudioCodec_Destroy to destroy the decoder. 467 OH_AVErrCode ret = OH_AudioCodec_Destroy(audioDec_); 468 if (ret != AV_ERR_OK) { 469 // Handle exceptions. 470 } else { 471 audioDec_ = NULL; // The decoder cannot be destroyed repeatedly. 472 } 473 ``` 474
