1# Audio Decoding 2 3You can call the native APIs provided by the AudioCodec module to decode audio, that is, to decode media data into PCM streams. 4 5Currently, the following decoding capabilities are supported: 6 7| Container Format| Audio Decoding Type | 8| -------- | :--------------------------- | 9| mp4 | AAC, MPEG (MP3), FLAC, Vorbis<!--RP1--><!--RP1End--> | 10| m4a | AAC | 11| flac | FLAC | 12| ogg | Vorbis<!--RP2--><!--RP2End--> | 13| aac | AAC | 14| mp3 | MPEG (MP3) | 15| amr | AMR (AMR-NB and AMR-WB) | 16| raw | G711mu | 17| ape | APE | 18 19**Usage Scenario** 20 21- Audio playback 22 23 Decode audio and transmit the data to the speaker for playing. 24- Audio rendering 25 26 Decode audio and transmit the data to the audio processing module for audio rendering. 27- Audio editing 28 29 Decode audio and transmit the data for audio editing (for example, adjusting the playback speed of a channel). Audio editing is performed based on PCM streams. 30> **NOTE** 31> 32> Streams generated in the MP3 audio encoding process cannot be directly decoded through the MP3 audio decoding process. The following process is recommended: PCM stream -> MP3 audio encoding -> muxing -> demuxing -> MP3 audio decoding. 33 34## How to Develop 35 36Read [AudioCodec](../../reference/apis-avcodec-kit/_audio_codec.md) for the API reference. 37 38Refer to the code snippet below to complete the entire audio decoding process, including creating a decoder, setting decoding parameters (such as the sampling rate, bit rate, and number of audio channels), and starting, refreshing, resetting, and destroying the decoder. 39 40During application development, you must call the APIs in the defined sequence. Otherwise, an exception or undefined behavior may occur. 41 42The figure below shows the call relationship of audio decoding. 43 44- The dotted line indicates an optional operation. 45 46- The solid line indicates a mandatory operation. 47 48 49 50### Linking the Dynamic Libraries in the CMake Script 51 52```cmake 53target_link_libraries(sample PUBLIC libnative_media_codecbase.so) 54target_link_libraries(sample PUBLIC libnative_media_core.so) 55target_link_libraries(sample PUBLIC libnative_media_acodec.so) 56``` 57 58### How to Develop 59 601. Add the header files. 61 62 ```cpp 63 #include <multimedia/player_framework/native_avcodec_audiocodec.h> 64 #include <multimedia/native_audio_channel_layout.h> 65 #include <multimedia/player_framework/native_avcapability.h> 66 #include <multimedia/player_framework/native_avcodec_base.h> 67 #include <multimedia/player_framework/native_avformat.h> 68 #include <multimedia/player_framework/native_avbuffer.h> 69 ``` 70 712. Create a decoder instance. In the code snippet below, **OH_AVCodec *** is the pointer to the decoder instance created. 72 73 ```cpp 74 // Namespace of the C++ standard library. 75 using namespace std; 76 // Create a decoder by name. 77 OH_AVCapability *capability = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_AUDIO_MPEG, false); 78 const char *name = OH_AVCapability_GetName(capability); 79 OH_AVCodec *audioDec_ = OH_AudioCodec_CreateByName(name); 80 ``` 81 82 ```cpp 83 // Specify whether encoding is used. The value false means decoding. 84 bool isEncoder = false; 85 // Create a decoder by MIME type. 86 OH_AVCodec *audioDec_ = OH_AudioCodec_CreateByMime(OH_AVCODEC_MIMETYPE_AUDIO_MPEG, isEncoder); 87 ``` 88 89 ```cpp 90 // Initialize the queues. 91 class ADecBufferSignal { 92 public: 93 std::mutex inMutex_; 94 std::mutex outMutex_; 95 std::mutex startMutex_; 96 std::condition_variable inCond_; 97 std::condition_variable outCond_; 98 std::condition_variable startCond_; 99 std::queue<uint32_t> inQueue_; 100 std::queue<uint32_t> outQueue_; 101 std::queue<OH_AVBuffer *> inBufferQueue_; 102 std::queue<OH_AVBuffer *> outBufferQueue_; 103 }; 104 ADecBufferSignal *signal_; 105 ``` 106 1073. Call **OH_AudioCodec_RegisterCallback()** to register callback functions. 108 109 Register the **OH_AVCodecCallback** struct that defines the following callback function pointers: 110 111 - **OH_AVCodecOnError**, a callback used to report a codec operation error 112 - **OH_AVCodecOnStreamChanged**, a callback used to report a codec stream change, for example, audio channel change 113 - **OH_AVCodecOnNeedInputBuffer**, a callback used to report input data required, which means that the decoder is ready for receiving data 114 - **OH_AVCodecOnNewOutputBuffer**, a callback used to report output data generated, which means that decoding is complete 115 116 You need to process the callback functions to ensure that the decoder runs properly. 117 118 ```cpp 119 // Implement the OH_AVCodecOnError callback function. 120 static void OnError(OH_AVCodec *codec, int32_t errorCode, void *userData) 121 { 122 (void)codec; 123 (void)errorCode; 124 (void)userData; 125 } 126 // Implement the OH_AVCodecOnStreamChanged callback function. 127 static void OnOutputFormatChanged(OH_AVCodec *codec, OH_AVFormat *format, void *userData) 128 { 129 (void)codec; 130 (void)format; 131 (void)userData; 132 } 133 // Implement the OH_AVCodecOnNeedInputBuffer callback function. 134 static void OnInputBufferAvailable(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *data, void *userData) 135 { 136 (void)codec; 137 ADecBufferSignal *signal = static_cast<ADecBufferSignal *>(userData); 138 unique_lock<mutex> lock(signal->inMutex_); 139 signal->inQueue_.push(index); 140 signal->inBufferQueue_.push(data); 141 signal->inCond_.notify_all(); 142 // The input stream is sent to inBufferQueue_. 143 } 144 // Implement the OH_AVCodecOnNewOutputBuffer callback function. 145 static void OnOutputBufferAvailable(OH_AVCodec *codec, uint32_t index, OH_AVBuffer *data, void *userData) 146 { 147 (void)codec; 148 ADecBufferSignal *signal = static_cast<ADecBufferSignal *>(userData); 149 unique_lock<mutex> lock(signal->outMutex_); 150 signal->outQueue_.push(index); 151 signal->outBufferQueue_.push(data); 152 signal->outCond_.notify_all(); 153 // The index of the output buffer is sent to outQueue_. 154 // The decoded data is sent to outBufferQueue_. 155 } 156 signal_ = new ADecBufferSignal(); 157 OH_AVCodecCallback cb_ = {&OnError, &OnOutputFormatChanged, &OnInputBufferAvailable, &OnOutputBufferAvailable}; 158 int32_t ret = OH_AudioCodec_RegisterCallback(audioDec_, cb_, signal_); 159 if (ret != AVCS_ERR_OK) { 160 // Exception handling. 161 } 162 ``` 163 1644. (Optional) Call **OH_AudioCodec_SetDecryptionConfig** to set the decryption configuration. Call this API after the media key system information is obtained but before **Prepare()** is called. For details about how to obtain such information, see step 3 in [Media Data Demuxing](audio-video-demuxer.md). For details about DRM APIs, see [DRM](../../reference/apis-drm-kit/_drm.md). 165 166 Add the header files. 167 168 ```c++ 169 #include <multimedia/drm_framework/native_mediakeysystem.h> 170 #include <multimedia/drm_framework/native_mediakeysession.h> 171 #include <multimedia/drm_framework/native_drm_err.h> 172 #include <multimedia/drm_framework/native_drm_common.h> 173 ``` 174 Link the dynamic library in the CMake script. 175 176 ``` cmake 177 target_link_libraries(sample PUBLIC libnative_drm.so) 178 ``` 179 180 The following is the sample code: 181 ```c++ 182 // Create a media key system based on the media key system information. The following uses com.clearplay.drm as an example. 183 MediaKeySystem *system = nullptr; 184 int32_t ret = OH_MediaKeySystem_Create("com.clearplay.drm", &system); 185 if (system == nullptr) { 186 printf("create media key system failed"); 187 return; 188 } 189 190 // Create a media key session. 191 MediaKeySession *session = nullptr; 192 DRM_ContentProtectionLevel contentProtectionLevel = CONTENT_PROTECTION_LEVEL_SW_CRYPTO; 193 ret = OH_MediaKeySystem_CreateMediaKeySession(system, &contentProtectionLevel, &session); 194 if (ret != DRM_OK) { 195 // If the creation fails, refer to the DRM interface document and check logs. 196 printf("create media key session failed."); 197 return; 198 } 199 if (session == nullptr) { 200 printf("media key session is nullptr."); 201 return; 202 } 203 // Generate a media key request and set the response to the media key request. 204 // Set the decryption configuration, that is, set the decryption session and secure channel flag to the decoder. (Currently, the secure channel is not supported for audio decryption and therefore the secure channel flag should be set to false.) 205 bool secureAudio = false; 206 ret = OH_AudioCodec_SetDecryptionConfig(audioDec_, session, secureAudio); 207 ``` 208 2095. Call **OH_AudioCodec_Configure()** to configure the decoder. 210 211 Key values of configuration options are described as follows: 212 213 | | Description | AAC | FLAC| Vorbis | MPEG | G711mu | AMR (AMR-NB and AMR-WB) | APE | 214 | ---------------------------- | :----------------------------------------------------------: | :--------------------------------: | :--: | :--------------------------------: | :--: | :-----------------: | :-------------------------------: | :-------------------------------: | 215 | OH_MD_KEY_AUD_SAMPLE_RATE | Sampling rate | Mandatory | Mandatory| Mandatory | Mandatory| Mandatory | Mandatory | Mandatory | 216 | OH_MD_KEY_AUD_CHANNEL_COUNT | Number of audio channels | Mandatory | Mandatory| Mandatory | Mandatory| Mandatory | Mandatory | Mandatory | 217 | OH_MD_KEY_MAX_INPUT_SIZE | Maximum input size | Optional | Optional| Optional | Optional| Optional | Optional | Optional | 218 | OH_MD_KEY_AAC_IS_ADTS | ADTS or not | Optional (Default value: 1 latm) | - | - | - | - | - | - | 219 | MD_KEY_AUDIO_SAMPLE_FORMAT | Output audio stream format | Optional (SAMPLE_S16LE, SAMPLE_F32LE)| - | Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional| Optional (default: SAMPLE_S16LE)| Optional (SAMPLE_S16LE, SAMPLE_F32LE)| Optional | 220 | MD_KEY_BITRATE | Optional | Optional | Optional| Optional | Optional| Optional | Optional | Optional | 221 | MD_KEY_IDENTIFICATION_HEADER | ID Header | - | - | Mandatory (Either this parameter or **MD_KEY_CODEC_CONFIG** must be set.) | - | - | - | - | 222 | MD_KEY_SETUP_HEADER | Setup Header | - | - | Mandatory (Either this parameter or **MD_KEY_CODEC_CONFIG** must be set.) | - | - | - | - | 223 | MD_KEY_CODEC_CONFIG | MD_KEY_SETUP_HEADERID Header+Common Header+Setup Header stitching| - | | Mandatory (Either this parameter or the combination of **MD_KEY_IDENTIFICATION_HEADER** and **MD_KEY_SETUP_HEADER** must be selected.) | - | - | - | - | 224 225 The sample below lists the value range of each audio decoding type. 226 | Audio Decoding Type| Sampling Rate (Hz) | Audio Channel Count| 227 | ----------- | ---------------------------------------------------------------------------------------------- | :----: | 228 | AAC | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000 | 1–8 | 229 | FLAC | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 192000 | 1–8 | 230 | Vorbis | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 176400, 192000| 1–8 | 231 | MPEG (MP3) | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000 | 1–2 | 232 | G711mu | 8000 | 1 | 233 | AMR (amrnb) | 8000 | 1 | 234 | AMR (amrwb) | 16000 | 1 | 235 | APE | 8000, 11025, 12000, 16000, 22050, 24000, 32000, 44100, 48000, 64000, 88200, 96000, 176400, 192000| 1–2 | 236 237 ```cpp 238 // Set the decoding resolution. 239 int32_t ret; 240 // (Mandatory) Configure the audio sampling rate. 241 constexpr uint32_t DEFAULT_SAMPLERATE = 44100; 242 // (Mandatory) Configure the audio bit rate. 243 constexpr uint32_t DEFAULT_BITRATE = 32000; 244 // (Mandatory) Configure the number of audio channels. 245 constexpr uint32_t DEFAULT_CHANNEL_COUNT = 2; 246 // (Optional) Configure the maximum input length. 247 constexpr uint32_t DEFAULT_MAX_INPUT_SIZE = 1152; 248 // Configure whether to use ADTS decoding (ACC). 249 constexpr uint32_t DEFAULT_AAC_TYPE = 1; 250 OH_AVFormat *format = OH_AVFormat_Create(); 251 // Set the format. 252 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_SAMPLE_RATE, DEFAULT_SAMPLERATE); 253 OH_AVFormat_SetIntValue(format, OH_MD_KEY_BITRATE, DEFAULT_BITRATE); 254 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_CHANNEL_COUNT, DEFAULT_CHANNEL_COUNT); 255 OH_AVFormat_SetIntValue(format, OH_MD_KEY_MAX_INPUT_SIZE, DEFAULT_MAX_INPUT_SIZE); 256 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AAC_IS_ADTS, DEFAULT_AAC_TYPE); 257 // Configure the decoder. 258 ret = OH_AudioCodec_Configure(audioDec_, format); 259 if (ret != AV_ERR_OK) { 260 // Exception handling. 261 } 262 ``` 263 2646. Call **OH_AudioCodec_Prepare()** to prepare internal resources for the decoder. 265 266 ```cpp 267 ret = OH_AudioCodec_Prepare(audioDec_); 268 if (ret != AV_ERR_OK) { 269 // Exception handling. 270 } 271 ``` 272 2737. Call **OH_AudioCodec_Start()** to start the decoder. 274 275 ```c++ 276 unique_ptr<ifstream> inputFile_ = make_unique<ifstream>(); 277 unique_ptr<ofstream> outFile_ = make_unique<ofstream>(); 278 // Open the path of the binary file to be decoded. 279 inputFile_->open(inputFilePath.data(), ios::in | ios::binary); 280 // Configure the path of the output file. 281 outFile_->open(outputFilePath.data(), ios::out | ios::binary); 282 // Start decoding. 283 ret = OH_AudioCodec_Start(audioDec_); 284 if (ret != AV_ERR_OK) { 285 // Exception handling. 286 } 287 ``` 288 2898. (Optional) Call **OH_AVCencInfo_SetAVBuffer()** to set the Common Encryption Scheme (CENC) information. 290 291 If the content being played is DRM encrypted and demuxing is performed by the upper-layer application, call **OH_AVCencInfo_SetAVBuffer()** to set the CENC information to the AVBuffer so that the media data can be decrypted in the AVBuffer. 292 293 Add the header file. 294 295 ```c++ 296 #include <multimedia/player_framework/native_cencinfo.h> 297 ``` 298 Link the dynamic library in the CMake script. 299 300 ``` cmake 301 target_link_libraries(sample PUBLIC libnative_media_avcencinfo.so) 302 ``` 303 304 The following is the sample code: 305 ```c++ 306 auto buffer = signal_->inBufferQueue_.front(); 307 int64_t size; 308 int64_t pts; 309 uint32_t keyIdLen = DRM_KEY_ID_SIZE; 310 uint8_t keyId[] = { 311 0xd4, 0xb2, 0x01, 0xe4, 0x61, 0xc8, 0x98, 0x96, 312 0xcf, 0x05, 0x22, 0x39, 0x8d, 0x09, 0xe6, 0x28}; 313 uint32_t ivLen = DRM_KEY_IV_SIZE; 314 uint8_t iv[] = { 315 0xbf, 0x77, 0xed, 0x51, 0x81, 0xde, 0x36, 0x3e, 316 0x52, 0xf7, 0x20, 0x4f, 0x72, 0x14, 0xa3, 0x95}; 317 uint32_t encryptedBlockCount = 0; 318 uint32_t skippedBlockCount = 0; 319 uint32_t firstEncryptedOffset = 0; 320 uint32_t subsampleCount = 1; 321 DrmSubsample subsamples[1] = { {0x10, 0x16} }; 322 inputFile_.read(reinterpret_cast<char *>(&size), sizeof(size)); 323 inputFile_.read(reinterpret_cast<char *>(&pts), sizeof(pts)); 324 inputFile_.read((char *)OH_AVMemory_GetAddr(buffer), size); 325 OH_AVCencInfo *cencInfo = OH_AVCencInfo_Create(); 326 if (cencInfo == nullptr) { 327 // Exception handling. 328 } 329 OH_AVErrCode errNo = OH_AVCencInfo_SetAlgorithm(cencInfo, DRM_ALG_CENC_AES_CTR); 330 if (errNo != AV_ERR_OK) { 331 // Exception handling. 332 } 333 errNo = OH_AVCencInfo_SetKeyIdAndIv(cencInfo, keyId, keyIdLen, iv, ivLen); 334 if (errNo != AV_ERR_OK) { 335 // Exception handling. 336 } 337 errNo = OH_AVCencInfo_SetSubsampleInfo(cencInfo, encryptedBlockCount, skippedBlockCount, firstEncryptedOffset, 338 subsampleCount, subsamples); 339 if (errNo != AV_ERR_OK) { 340 // Exception handling. 341 } 342 errNo = OH_AVCencInfo_SetMode(cencInfo, DRM_CENC_INFO_KEY_IV_SUBSAMPLES_SET); 343 if (errNo != AV_ERR_OK) { 344 // Exception handling. 345 } 346 errNo = OH_AVCencInfo_SetAVBuffer(cencInfo, buffer); 347 if (errNo != AV_ERR_OK) { 348 // Exception handling. 349 } 350 errNo = OH_AVCencInfo_Destroy(cencInfo); 351 if (errNo != AV_ERR_OK) { 352 // Exception handling. 353 } 354 ``` 355 3569. Call **OH_AudioCodec_PushInputBuffer()** to write the data to decode. 357 358 To indicate the End of Stream (EOS), pass in the **AVCODEC_BUFFER_FLAGS_EOS** flag. 359 360 ```c++ 361 uint32_t index = signal_->inQueue_.front(); 362 auto buffer = signal_->inBufferQueue_.front(); 363 int64_t size; 364 int64_t pts; 365 // size is the length of each frame of the data to decode. pts is the timestamp of each frame and is used to indicate when the audio should be played. 366 // The values of size and pts are obtained from an audio and video resource file or data stream to decode. 367 // In the case of an audio and video resource file, the values are obtained from the buffer in the decapsulated OH_AVDemuxer_ReadSampleBuffer. 368 // In the case of a data stream, the values are obtained from the data stream provider. 369 // In this example, the values of size and pts are obtained from the test file. 370 inputFile_.read(reinterpret_cast<char *>(&size), sizeof(size)); 371 inputFile_.read(reinterpret_cast<char *>(&pts), sizeof(pts)); 372 inputFile_.read((char *)OH_AVBuffer_GetAddr(buffer), size); 373 OH_AVCodecBufferAttr attr = {0}; 374 if (inputFile_->eof()) { 375 attr.size = 0; 376 attr.flags = AVCODEC_BUFFER_FLAGS_EOS; 377 } else { 378 attr.size = size; 379 attr.flags = AVCODEC_BUFFER_FLAGS_NONE; 380 } 381 attr.pts = pts; 382 OH_AVBuffer_SetBufferAttr(buffer, &attr); 383 int32_t ret = OH_AudioCodec_PushInputBuffer(audioDec_, index); 384 if (ret != AV_ERR_OK) { 385 // Exception handling. 386 } 387 ``` 388 38910. Call **OH_AudioCodec_FreeOutputBuffer()** to output decoded PCM streams. 390 391 <!--RP3--> 392 ```c++ 393 uint32_t index = signal_->outQueue_.front(); 394 OH_AVBuffer *data = signal_->outBufferQueue_.front(); 395 // Obtain the buffer attributes. 396 OH_AVCodecBufferAttr attr = {0}; 397 ret = OH_AVBuffer_GetBufferAttr(data, &attr); 398 if (ret != AV_ERR_OK) { 399 // Exception handling. 400 } 401 // Write the decoded data (specified by data) to the output file. 402 pcmOutputFile_.write(reinterpret_cast<char *>(OH_AVBuffer_GetAddr(data)), attr.size); 403 ret = OH_AudioCodec_FreeOutputBuffer(audioDec_, index); 404 if (ret != AV_ERR_OK) { 405 // Exception handling. 406 } 407 if (attr.flags == AVCODEC_BUFFER_FLAGS_EOS) { 408 // End 409 } 410 ``` 411 <!--RP3End--> 412 41311. (Optional) Call **OH_AudioCodec_Flush()** to refresh the decoder. 414 415 After **OH_AudioCodec_Flush()** is called, the decoder remains in the running state, but the current queue is cleared and the buffer storing the decoded data is freed. To continue decoding, you must call **OH_AudioCodec_Start()** again. 416 417 You need to call **OH_AudioCodec_Start()** in the following cases: 418 419 * The EOS of the file is reached. 420 * An error with **OH_AudioCodec_IsValid** set to **true** (indicating that the execution can continue) occurs. 421 422 ```c++ 423 // Refresh the decoder. 424 ret = OH_AudioCodec_Flush(audioDec_); 425 if (ret != AV_ERR_OK) { 426 // Exception handling. 427 } 428 // Start decoding again. 429 ret = OH_AudioCodec_Start(audioDec_); 430 if (ret != AV_ERR_OK) { 431 // Exception handling. 432 } 433 ``` 434 43512. (Optional) Call **OH_AudioCodec_Reset()** to reset the decoder. 436 437 After **OH_AudioCodec_Reset()** is called, the decoder returns to the initialized state. To continue decoding, you must call **OH_AudioCodec_Configure()** and then **OH_AudioCodec_Start()**. 438 439 ```c++ 440 // Reset the decoder. 441 ret = OH_AudioCodec_Reset(audioDec_); 442 if (ret != AV_ERR_OK) { 443 // Exception handling. 444 } 445 // Reconfigure the decoder. 446 ret = OH_AudioCodec_Configure(audioDec_, format); 447 if (ret != AV_ERR_OK) { 448 // Exception handling. 449 } 450 ``` 451 45213. Call **OH_AudioCodec_Stop()** to stop the decoder. 453 454 ```c++ 455 // Stop the decoder. 456 ret = OH_AudioCodec_Stop(audioDec_); 457 if (ret != AV_ERR_OK) { 458 // Exception handling. 459 } 460 ``` 461 46214. Call **OH_AudioCodec_Destroy()** to destroy the decoder instance and release resources. 463 464 > **NOTE** 465 > 466 > You only need to call this API once. 467 468 ```c++ 469 // Call OH_AudioCodec_Destroy to destroy the decoder. 470 ret = OH_AudioCodec_Destroy(audioDec_); 471 if (ret != AV_ERR_OK) { 472 // Exception handling. 473 } else { 474 audioDec_ = NULL; // The decoder cannot be destroyed repeatedly. 475 } 476 ``` 477 478
