1# Audio Decoding 2 3You can call the native APIs provided by the **AudioDecoder** module to decode audio, that is, to decode media data into PCM streams. 4 5Currently, the following decoding capabilities are supported: 6 7| Container Specification| Audio Decoding Type | 8| -------- | :--------------------------- | 9| mp4 | AAC, MPEG (MP3), FLAC, Vorbis| 10| m4a | AAC | 11| flac | FLAC | 12| ogg | Vorbis | 13| aac | AAC | 14| mp3 | MPEG (MP3) | 15 16**Usage Scenario** 17 18- Audio playback 19 20 Decode audio and transmit the data to the speaker for playing. 21- Audio rendering 22 23 Decode audio and transmit the data to the audio processing module for audio rendering. 24- Audio editing 25 26 Decode audio and transmit the data for audio editing (for example, adjusting the playback speed of a channel). Audio editing is performed based on PCM streams. 27 28## How to Develop 29 30Read [AudioDecoder](../reference/native-apis/_audio_decoder.md) for the API reference. 31 32Refer to the code snippet below to complete the entire audio decoding process, including creating a decoder, setting decoding parameters (such as the sampling rate, bit rate, and number of audio channels), and starting, refreshing, resetting, and destroying the decoder. 33 34During application development, you must call the APIs in the defined sequence. Otherwise, an exception or undefined behavior may occur. 35 36For details about the complete code, see [Sample](https://gitee.com/openharmony/multimedia_av_codec/blob/master/test/nativedemo/audio_demo/avcodec_audio_decoder_demo.cpp). 37 38The figure below shows the call relationship of audio decoding. 39 40 41 42### Linking the Dynamic Library in the CMake Script 43``` cmake 44target_link_libraries(sample PUBLIC libnative_media_codecbase.so) 45target_link_libraries(sample PUBLIC libnative_media_core.so) 46target_link_libraries(sample PUBLIC libnative_media_adec.so) 47``` 48 49### How to Develop 50 511. Create a decoder instance. 52 53 ```cpp 54 // Namespace of the C++ standard library. 55 using namespace std; 56 // Create a decoder by name. 57 OH_AVCapability *capability = OH_AVCodec_GetCapability(OH_AVCODEC_MIMETYPE_AUDIO_MPEG, false); 58 const char *name = OH_AVCapability_GetName(capability); 59 OH_AVCodec *audioDec = OH_AudioDecoder_CreateByName(name); 60 ``` 61 62 ```cpp 63 // Create a decoder by MIME type. 64 OH_AVCodec *audioDec = OH_AudioDecoder_CreateByMime(OH_AVCODEC_MIMETYPE_AUDIO_MPEG); 65 ``` 66 67 ```cpp 68 // Initialize the queues. 69 class ADecSignal { 70 public: 71 mutex inMutex_; 72 mutex outMutex_; 73 mutex startMutex_; 74 condition_variable inCond_; 75 condition_variable outCond_; 76 condition_variable startCond_; 77 queue<uint32_t> inQueue_; 78 queue<uint32_t> outQueue_; 79 queue<OH_AVMemory *> inBufferQueue_; 80 queue<OH_AVMemory *> outBufferQueue_; 81 queue<OH_AVCodecBufferAttr> attrQueue_; 82 }; 83 ADecSignal *signal_; 84 ``` 85 862. Call **OH_AudioDecoder_SetCallback()** to set callback functions. 87 88 Register the **OH_AVCodecAsyncCallback** struct that defines the following callback function pointers: 89 90 - **OH_AVCodecOnError**, a callback used to report a codec operation error 91 - **OH_AVCodecOnStreamChanged**, a callback used to report a codec stream change, for example, audio channel change 92 - **OH_AVCodecOnNeedInputData**, a callback used to report input data required, which means that the decoder is ready for receiving data 93 - **OH_AVCodecOnNewOutputData**, a callback used to report output data generated, which means that decoding is complete 94 95 You need to process the callback functions to ensure that the decoder runs properly. 96 97 ```cpp 98 // Implement the OH_AVCodecOnError callback function. 99 static void OnError(OH_AVCodec *codec, int32_t errorCode, void *userData) 100 { 101 (void)codec; 102 (void)errorCode; 103 (void)userData; 104 } 105 // Implement the OH_AVCodecOnStreamChanged callback function. 106 static void OnStreamChanged(OH_AVCodec *codec, OH_AVFormat *format, void*userData) 107 { 108 (void)codec; 109 (void)format; 110 (void)userData; 111 } 112 // Implement the OH_AVCodecOnNeedInputData callback function. 113 static void onNeedInputData(OH_AVCodec *codec, uint32_t index, OH_AVMemory*data, void *userData) 114 { 115 (void)codec; 116 ADecSignal *signal = static_cast<ADecSignal *>(userData); 117 unique_lock<mutex> lock(signal->inMutex_); 118 signal->inQueue_.push(index); 119 signal->inBufferQueue_.push(data); 120 signal->inCond_.notify_all(); 121 // The input stream is sent to the InputBuffer queue. 122 } 123 // Implement the OH_AVCodecOnNewOutputData callback function. 124 static void onNeedOutputData(OH_AVCodec *codec, uint32_t index, OH_AVMemory*data, OH_AVCodecBufferAttr *attr, 125 void *userData) 126 { 127 (void)codec; 128 ADecSignal *signal = static_cast<ADecSignal *>(userData); 129 unique_lock<mutex> lock(signal->outMutex_); 130 signal->outQueue_.push(index); 131 signal->outBufferQueue_.push(data); 132 if (attr) { 133 signal->attrQueue_.push(*attr); 134 } 135 signal->outCond_.notify_all(); 136 // The index of the output buffer is sent to OutputQueue_. 137 // The decoded data is sent to the OutputBuffer queue. 138 } 139 signal_ = new ADecSignal(); 140 OH_AVCodecAsyncCallback cb = {&OnError, &OnStreamChanged, &onNeedInputData, &onNeedOutputData}; 141 // Set the asynchronous callbacks. 142 int32_t ret = OH_AudioDecoder_SetCallback(audioDec, cb, signal_); 143 if (ret != AV_ERR_OK) { 144 // Exception handling. 145 } 146 ``` 1473. Call **OH_AudioDecoder_Configure()** to configure the decoder. 148 149 The following options are mandatory: sampling rate, bit rate, and number of audio channels. The maximum input length is optional. 150 151 - For AAC decoding, the parameter that specifies whether the data type is Audio Data Transport Stream (ADTS) must be specified. If this parameter is not specified, the data type is considered as Low Overhead Audio Transport Multiplex (LATM). 152 153 ```cpp 154 // Set the decoding resolution. 155 int32_t ret; 156 // (Mandatory) Configure the audio sampling rate. 157 constexpr uint32_t DEFAULT_SMAPLERATE = 44100; 158 // (Mandatory) Configure the audio bit rate. 159 constexpr uint32_t DEFAULT_BITRATE = 32000; 160 // (Mandatory) Configure the number of audio channels. 161 constexpr uint32_t DEFAULT_CHANNEL_COUNT = 2; 162 // (Optional) Configure the maximum input length. 163 constexpr uint32_t DEFAULT_MAX_INPUT_SIZE = 1152; 164 // Configure whether to use ADTS decoding (ACC). 165 constexpr uint32_t DEFAULT_AAC_TYPE = 1; 166 OH_AVFormat *format = OH_AVFormat_Create(); 167 // Set the format. 168 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_SAMPLE_RATE,DEFAULT_SMAPLERATE); 169 OH_AVFormat_SetIntValue(format, OH_MD_KEY_BITRATE,DEFAULT_BITRATE); 170 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AUD_CHANNEL_COUNT,DEFAULT_CHANNEL_COUNT); 171 OH_AVFormat_SetIntValue(format, OH_MD_KEY_MAX_INPUT_SIZE,DEFAULT_MAX_INPUT_SIZE); 172 OH_AVFormat_SetIntValue(format, OH_MD_KEY_AAC_IS_ADTS, DEFAULT_AAC_TYPE); 173 // Configure the decoder. 174 ret = OH_AudioDecoder_Configure(audioDec, format); 175 if (ret != AV_ERR_OK) { 176 // Exception handling. 177 } 178 ``` 179 1804. Call **OH_AudioDecoder_Prepare()** to prepare internal resources for the decoder. 181 182 ```cpp 183 ret = OH_AudioDecoder_Prepare(audioDec); 184 if (ret != AV_ERR_OK) { 185 // Exception handling. 186 } 187 ``` 1885. Call **OH_AudioDecoder_Start()** to start the decoder. 189 190 ```c++ 191 unique_ptr<ifstream> inputFile_ = make_unique<ifstream>(); 192 unique_ptr<ofstream> outFile_ = make_unique<ofstream>(); 193 // Open the path of the binary file to be decoded. 194 inputFile_->open(inputFilePath.data(), ios::in | ios::binary); 195 // Configure the path of the output file. 196 outFile_->open(outputFilePath.data(), ios::out | ios::binary); 197 // Start decoding. 198 ret = OH_AudioDecoder_Start(audioDec); 199 if (ret != AV_ERR_OK) { 200 // Exception handling. 201 } 202 ``` 2036. Call **OH_AudioDecoder_PushInputData()** to write the data to decode. 204 205 To indicate the End of Stream (EOS), pass in the **AVCODEC_BUFFER_FLAGS_EOS** flag. 206 ```c++ 207 // Configure the buffer information. 208 OH_AVCodecBufferAttr info; 209 // Set the size, offset, and timestamp. 210 auto buffer = signal_->inBufferQueue_.front(); 211 int64_t size; 212 int64_t pts; 213 inputFile_.read(reinterpret_cast<char *>(&size), sizeof(size)); 214 if (inputFile_->eof()) { 215 size = 0; 216 info.flags = AVCODEC_BUFFER_FLAGS_EOS; 217 } else { 218 inputFile_.read(reinterpret_cast<char *>(&pts), sizeof(pts)); 219 inputFile_.read((char *)OH_AVMemory_GetAddr(buffer), size); 220 info.flags = AVCODEC_BUFFER_FLAGS_CODEC_DATA; 221 } 222 info.size = size; 223 info.offset = 0; 224 info.pts = pts; 225 uint32_t index = signal_->inQueue_.front(); 226 // Send the data to the input queue for decoding. The index is the subscript of the queue. 227 int32_t ret = OH_AudioDecoder_PushInputData(audioDec, index, info); 228 if (ret != AV_ERR_OK) { 229 // Exception handling. 230 } 231 ``` 2327. Call **OH_AudioDecoder_FreeOutputData()** to output decoded PCM streams. 233 234 ```c++ 235 OH_AVCodecBufferAttr attr = signal_->attrQueue_.front(); 236 OH_AVMemory *data = signal_->outBufferQueue_.front(); 237 uint32_t index = signal_->outQueue_.front(); 238 // Write the decoded data (specified by data) to the output file. 239 outFile_->write(reinterpret_cast<char *>(OH_AVMemory_GetAddr(data)), attr.size); 240 // Free the buffer that stores the data. 241 ret = OH_AudioDecoder_FreeOutputData(audioDec, index); 242 if (ret != AV_ERR_OK) { 243 // Exception handling. 244 } 245 ``` 2468. (Optional) Call **OH_AudioDecoder_Flush()** to refresh the decoder. 247 248 After **OH_AudioDecoder_Flush()** is called, the decoder remains in the running state, but the current queue is cleared and the buffer storing the decoded data is freed. To continue decoding, you must call **OH_AudioDecoder_Start()** again. 249 250 You need to call **OH_AudioDecoder_Start()** in the following cases: 251 * The EOS of the file is reached. 252 * An error with **OH_AudioDecoder_IsValid** set to **true** (indicating that the execution can continue) occurs. 253 ```c++ 254 // Refresh the decoder. 255 ret = OH_AudioDecoder_Flush(audioDec); 256 if (ret != AV_ERR_OK) { 257 // Exception handling. 258 } 259 // Start decoding again. 260 ret = OH_AudioDecoder_Start(audioDec); 261 if (ret != AV_ERR_OK) { 262 // Exception handling. 263 } 264 ``` 2659. (Optional) Call **OH_AudioDecoder_Reset()** to reset the decoder. 266 267 After **OH_AudioDecoder_Reset()** is called, the decoder returns to the initialized state. To continue decoding, you must call **OH_AudioDecoder_Configure()** and then **OH_AudioDecoder_Start()**. 268 269 ```c++ 270 // Reset the decoder. 271 ret = OH_AudioDecoder_Reset(audioDec); 272 if (ret != AV_ERR_OK) { 273 // Exception handling. 274 } 275 // Reconfigure the decoder. 276 ret = OH_AudioDecoder_Configure(audioDec, format); 277 if (ret != AV_ERR_OK) { 278 // Exception handling. 279 } 280 ``` 28110. Call **OH_AudioDecoder_Stop()** to stop the decoder. 282 283 ```c++ 284 // Stop the decoder. 285 ret = OH_AudioDecoder_Stop(audioDec); 286 if (ret != AV_ERR_OK) { 287 // Exception handling. 288 } 289 ``` 29011. Call **OH_AudioDecoder_Destroy()** to destroy the decoder instance and release resources. 291 292 **NOTE**: You only need to call this API once. 293 294 ```c++ 295 // Call OH_AudioDecoder_Destroy to destroy the decoder. 296 ret = OH_AudioDecoder_Destroy(audioDec); 297 if (ret != AV_ERR_OK) { 298 // Exception handling. 299 } else { 300 audioDec = NULL; // The decoder cannot be destroyed repeatedly. 301 } 302 ``` 303
