1# Media Data Multiplexing 2 3You can call native APIs to multiplex audio and video streams, that is, to store encoded audio and video data to a file in a certain format. 4 5For details about the supported multiplexing formats, see [AVCodec Supported Formats](avcodec-support-formats.md#media-data-multiplexing). 6 7<!--RP2--><!--RP2End--> 8 9**When to Use** 10 11- Video and audio recording 12 13 After you encode audio and video streams, multiplex them into files. 14 15- Audio and video editing 16 17 After you edit audio and video, multiplex them into files. 18 19- Audio and video transcoding 20 21 After you transcode audio and video, multiplex them into files. 22 23## Development Guidelines 24 25Read [AVMuxer](../../reference/apis-avcodec-kit/_a_v_muxer.md) for the API reference. 26 27> **NOTE** 28> 29> To call the AVMuxer module to write a local file, request the ohos.permission.READ_MEDIA and ohos.permission.WRITE_MEDIA permissions by following the instructions provided in [Requesting User Authorization](../../security/AccessToken/request-user-authorization.md). 30 31### Linking the Dynamic Libraries in the CMake Script 32 33``` cmake 34target_link_libraries(sample PUBLIC libnative_media_avmuxer.so) 35target_link_libraries(sample PUBLIC libnative_media_core.so) 36``` 37 38### How to Develop 39 40The following walks you through how to implement the entire process of audio and video multiplexing. It uses the MP4 format as an example. 41 42For details about the keys to be configured for different container formats, see [AVCodec Supported Formats](avcodec-support-formats.md#media-data-multiplexing). 43 441. Add the header files. 45 46 ```c++ 47 #include <multimedia/player_framework/native_avmuxer.h> 48 #include <multimedia/player_framework/native_avcodec_base.h> 49 #include <multimedia/player_framework/native_avformat.h> 50 #include <multimedia/player_framework/native_avbuffer.h> 51 #include <fcntl.h> 52 ``` 53 542. Call **OH_AVMuxer_Create()** to create an OH_AVMuxer instance. 55 56 ```c++ 57 // Set the container format to MP4. 58 OH_AVOutputFormat format = AV_OUTPUT_FORMAT_MPEG_4; 59 // Create an FD in read/write mode. 60 int32_t fd = open("test.mp4", O_CREAT | O_RDWR | O_TRUNC, S_IRUSR | S_IWUSR); 61 OH_AVMuxer *muxer = OH_AVMuxer_Create(fd, format); 62 ``` 63 643. (Optional) Call **OH_AVMuxer_SetRotation()** to set the rotation angle. 65 66 ```c++ 67 // Set the rotation angle when a video image needs to be rotated. 68 OH_AVMuxer_SetRotation(muxer, 0); 69 ``` 70 714. Add file-level data. 72 ```c++ 73 OH_AVFormat *format = OH_AVFormat_Create(); // Call OH_AVFormat_Create to create a format. 74 OH_AVFormat_SetStringValue(format, OH_MD_KEY_CREATION_TIME, "2024-12-28T00:00:00:000000Z"); // Set the creation time (UTC time in ISO 8601 format). 75 int ret = OH_AVMuxer_SetFormat(muxer, format); // Set format data to the muxer. 76 if (ret != AV_ERR_OK) { 77 // Failed to set the format because no valid key data to be written is found. 78 } 79 OH_AVFormat_Destroy(format); // Destroy the format. 80 ``` 81 825. Add an audio track. 83 84 **Method 1: Use OH_AVFormat_Create to create the format.** 85 86 ```c++ 87 int audioTrackId = -1; 88 uint8_t *buffer = ...; // Encoding configuration data. If there is no configuration data, leave the parameter unspecified. 89 size_t size = ...; // Length of the encoding configuration data. Set this parameter based on project requirements. 90 OH_AVFormat *formatAudio = OH_AVFormat_Create(); // Call OH_AVFormat_Create to create a format. The following showcases how to multiplex an AAC-LC audio with the sample rate of 44100 Hz and two audio channels. 91 OH_AVFormat_SetStringValue(formatAudio, OH_MD_KEY_CODEC_MIME, OH_AVCODEC_MIMETYPE_AUDIO_AAC); // Mandatory. 92 OH_AVFormat_SetIntValue(formatAudio, OH_MD_KEY_AUD_SAMPLE_RATE, 44100); // Mandatory. 93 OH_AVFormat_SetIntValue(formatAudio, OH_MD_KEY_AUD_CHANNEL_COUNT, 2); // Mandatory. 94 OH_AVFormat_SetIntValue(formatAudio, OH_MD_KEY_PROFILE, AAC_PROFILE_LC); // Optional. 95 OH_AVFormat_SetBuffer(formatAudio, OH_MD_KEY_CODEC_CONFIG, buffer, size); // Optional. 96 97 int ret = OH_AVMuxer_AddTrack(muxer, &audioTrackId, formatAudio); 98 if (ret != AV_ERR_OK || audioTrackId < 0) { 99 // Failure to add the audio track. 100 } 101 OH_AVFormat_Destroy (formatAudio); // Destroy the format. 102 ``` 103 104 **Method 2: Use OH_AVFormat_CreateAudioFormat to create the format.** 105 106 ```c++ 107 int audioTrackId = -1; 108 uint8_t *buffer = ...; // Encoding configuration data. If there is no configuration data, leave the parameter unspecified. 109 size_t size = ...; // Length of the encoding configuration data. Set this parameter based on project requirements. 110 OH_AVFormat *formatAudio = OH_AVFormat_CreateAudioFormat(OH_AVCODEC_MIMETYPE_AUDIO_AAC, 44100, 2); 111 OH_AVFormat_SetIntValue(formatAudio, OH_MD_KEY_PROFILE, AAC_PROFILE_LC); // Optional. 112 OH_AVFormat_SetBuffer(formatAudio, OH_MD_KEY_CODEC_CONFIG, buffer, size); // Optional. 113 114 int ret = OH_AVMuxer_AddTrack(muxer, &audioTrackId, formatAudio); 115 if (ret != AV_ERR_OK || audioTrackId < 0) { 116 // Failure to add the audio track. 117 } 118 OH_AVFormat_Destroy (formatAudio); // Destroy the format. 119 ``` 120 1216. Add a video track. 122 123 **Method 1: Use OH_AVFormat_Create to create the format.** 124 125 ```c++ 126 int videoTrackId = -1; 127 uint8_t *buffer = ...; // Encoding configuration data. If there is no configuration data, leave the parameter unspecified. 128 size_t size = ...; // Length of the encoding configuration data. Set this parameter based on project requirements. 129 OH_AVFormat *formatVideo = OH_AVFormat_Create(); 130 OH_AVFormat_SetStringValue(formatVideo, OH_MD_KEY_CODEC_MIME, OH_AVCODEC_MIMETYPE_VIDEO_AVC); // Mandatory. 131 OH_AVFormat_SetIntValue(formatVideo, OH_MD_KEY_WIDTH, 1280); // Mandatory. 132 OH_AVFormat_SetIntValue(formatVideo, OH_MD_KEY_HEIGHT, 720); // Mandatory. 133 OH_AVFormat_SetBuffer(formatVideo, OH_MD_KEY_CODEC_CONFIG, buffer, size); // Optional. 134 135 int ret = OH_AVMuxer_AddTrack(muxer, &videoTrackId, formatVideo); 136 if (ret != AV_ERR_OK || videoTrackId < 0) { 137 // Failure to add the video track. 138 } 139 OH_AVFormat_Destroy(formatVideo); // Destroy the format. 140 ``` 141 142 **Method 2: Use OH_AVFormat_CreateVideoFormat to create the format.** 143 144 ```c++ 145 int videoTrackId = -1; 146 uint8_t *buffer = ...; // Encoding configuration data. If there is no configuration data, leave the parameter unspecified. 147 size_t size = ...; // Length of the encoding configuration data. Set this parameter based on project requirements. 148 OH_AVFormat *formatVideo = OH_AVFormat_CreateVideoFormat(OH_AVCODEC_MIMETYPE_VIDEO_AVC, 1280, 720); 149 OH_AVFormat_SetBuffer(formatVideo, OH_MD_KEY_CODEC_CONFIG, buffer, size); // Optional. 150 151 int ret = OH_AVMuxer_AddTrack(muxer, &videoTrackId, formatVideo); 152 if (ret != AV_ERR_OK || videoTrackId < 0) { 153 // Failure to add the video track. 154 } 155 OH_AVFormat_Destroy(formatVideo); // Destroy the format. 156 ``` 157 1587. Add an auxiliary track. 159 > **NOTE** 160 > 161 > To add an auxiliary track, set **OH_MD_KEY_TRACK_TYPE** to **MEDIA_TYPE_AUXILIARY**.<br> 162 > When setting **OH_MD_KEY_TRACK_REFERENCE_TYPE**, choose from **hint**, **cdsc**, **font**, **hind**, **vdep**, **vplx**, **subt**, **thmb**, **auxl**, **cdtg**, **shsc**, or **aest**.<br> 163 > When setting **OH_MD_KEY_TRACK_DESCRIPTION**, use a string that starts with **"com.openharmony."** and contains no more than 256 characters.<br> 164 > When setting **OH_MD_KEY_REFERENCE_TRACK_IDS**, ensure that the track ID is greater than or equal to 0 and refers to an existing track. 165 166 Add an audio auxiliary track. 167 168 ```c++ 169 int32_t audioAuxlTrackId = -1; 170 std::vector<int32_t> audioTrackIDsVector = {0}; // Set the IDs of the audio auxiliary tracks based on the actual situation. Multiple values can be entered. 171 int32_t *audioAuxlTrackIDs = audioTrackIDsVector.data(); 172 OH_AVFormat *audioAuxlFormat = OH_AVFormat_Create(); 173 OH_AVFormat_SetStringValue(audioAuxlFormat, OH_MD_KEY_CODEC_MIME, OH_AVCODEC_MIMETYPE_AUDIO_AAC); // Mandatory. 174 OH_AVFormat_SetIntValue(audioAuxlFormat, OH_MD_KEY_AUD_SAMPLE_RATE, 44100); // Mandatory. 175 OH_AVFormat_SetIntValue(audioAuxlFormat, OH_MD_KEY_AUD_CHANNEL_COUNT, 2); // Mandatory. 176 OH_AVFormat_SetIntValue(audioAuxlFormat, OH_MD_KEY_PROFILE, AAC_PROFILE_LC); // Optional. 177 OH_AVFormat_SetIntValue(audioAuxlFormat, OH_MD_KEY_TRACK_TYPE, static_cast<int32_t>(OH_MediaType::MEDIA_TYPE_AUXILIARY)); // Mandatory. 178 OH_AVFormat_SetStringValue(audioAuxlFormat, OH_MD_KEY_TRACK_REFERENCE_TYPE, "auxl"); // Mandatory. 179 OH_AVFormat_SetStringValue(audioAuxlFormat, OH_MD_KEY_TRACK_DESCRIPTION, "com.openharmony.audiomode.auxiliary"); // Mandatory. 180 OH_AVFormat_SetIntBuffer(audioAuxlFormat, OH_MD_KEY_REFERENCE_TRACK_IDS, audioAuxlTrackIDs, audioTrackIDsVector.size()); // Mandatory. 181 182 int ret = OH_AVMuxer_AddTrack(muxer, &audioAuxlTrackId, audioAuxlFormat); 183 if (ret != AV_ERR_OK || audioAuxlTrackId < 0) { 184 // Failed to add the audio auxiliary track. 185 } 186 ``` 187 188 Add a video auxiliary track. 189 190 ```c++ 191 int32_t videoAuxlTrackId = -1; 192 std::vector<int32_t> videoTrackIDsVector = {0}; // Set the IDs of the video auxiliary tracks based on the actual situation. Multiple values can be entered. 193 int32_t *videoAuxlTrackIDs = videoTrackIDsVector.data(); 194 OH_AVFormat *videoAuxlFormat = OH_AVFormat_Create(); 195 OH_AVFormat_SetStringValue(videoAuxlFormat, OH_MD_KEY_CODEC_MIME, OH_AVCODEC_MIMETYPE_VIDEO_AVC); // Mandatory. 196 OH_AVFormat_SetIntValue(videoAuxlFormat, OH_MD_KEY_WIDTH, 1280); // Mandatory. 197 OH_AVFormat_SetIntValue(videoAuxlFormat, OH_MD_KEY_HEIGHT, 720); // Mandatory. 198 OH_AVFormat_SetIntValue(videoAuxlFormat, OH_MD_KEY_TRACK_TYPE, static_cast<int32_t>(OH_MediaType::MEDIA_TYPE_AUXILIARY)); // Mandatory. 199 OH_AVFormat_SetStringValue(videoAuxlFormat, OH_MD_KEY_TRACK_REFERENCE_TYPE, "vdep"); // Mandatory. 200 OH_AVFormat_SetStringValue(videoAuxlFormat, OH_MD_KEY_TRACK_DESCRIPTION, "com.openharmony.moviemode.depth"); // Mandatory. 201 OH_AVFormat_SetIntBuffer(videoAuxlFormat, OH_MD_KEY_REFERENCE_TRACK_IDS, videoAuxlTrackIDs, videoTrackIDsVector.size()); // Mandatory. 202 203 int ret = OH_AVMuxer_AddTrack(muxer, &videoAuxlTrackId, videoAuxlFormat); 204 if (ret != AV_ERR_OK || videoAuxlTrackId < 0) { 205 // Failed to add the video auxiliary track. 206 } 207 ``` 208 2098. Add a cover track. 210 211 **Method 1: Use OH_AVFormat_Create to create the format.** 212 213 ```c++ 214 int coverTrackId = -1; 215 OH_AVFormat *formatCover = OH_AVFormat_Create(); 216 OH_AVFormat_SetStringValue(formatCover, OH_MD_KEY_CODEC_MIME, OH_AVCODEC_MIMETYPE_IMAGE_JPG); 217 OH_AVFormat_SetIntValue(formatCover, OH_MD_KEY_WIDTH, 1280); 218 OH_AVFormat_SetIntValue(formatCover, OH_MD_KEY_HEIGHT, 720); 219 220 int ret = OH_AVMuxer_AddTrack(muxer, &coverTrackId, formatCover); 221 if (ret != AV_ERR_OK || coverTrackId < 0) { 222 // Failure to add the cover track. 223 } 224 OH_AVFormat_Destroy(formatCover); // Destroy the format. 225 ``` 226 227 **Method 2: Use OH_AVFormat_CreateVideoFormat to create the format.** 228 229 ```c++ 230 int coverTrackId = -1; 231 OH_AVFormat *formatCover = OH_AVFormat_CreateVideoFormat(OH_AVCODEC_MIMETYPE_IMAGE_JPG, 1280, 720); 232 233 int ret = OH_AVMuxer_AddTrack(muxer, &coverTrackId, formatCover); 234 if (ret != AV_ERR_OK || coverTrackId < 0) { 235 // Failure to add the cover track. 236 } 237 OH_AVFormat_Destroy(formatCover); // Destroy the format. 238 ``` 239 2409. Call **OH_AVMuxer_Start()** to start multiplexing. 241 242 ```c++ 243 // Call Start() to write the file header. After this API is called, you cannot set media parameters or add tracks. 244 if (OH_AVMuxer_Start(muxer) != AV_ERR_OK) { 245 // Handle exceptions. 246 } 247 ``` 248 24910. Call **OH_AVMuxer_WriteSampleBuffer()** to write data. 250 251 The encapsulated data includes video, audio, and cover data. 252 253 ```c++ 254 // Data can be written only after Start() is called. 255 int size = ...; 256 OH_AVBuffer *sample = OH_AVBuffer_Create(size); // Create an AVBuffer instance. 257 // Write data to the sample buffer by using OH_AVBuffer_GetAddr(sample). For details, see the usage of OH_AVBuffer. 258 // Multiplex the cover. One image must be written at a time. 259 260 // Set buffer information. 261 OH_AVCodecBufferAttr info = {0}; 262 info.pts =...; // Playback start time of the current data, in microseconds. The relative time is used. 263 info.size = size; // Length of the current data. 264 info.offset = 0; // Offset. Generally, the value is 0. 265 info.flags |= AVCODEC_BUFFER_FLAGS_SYNC_FRAME; // Flag of the current data. For details, see OH_AVCodecBufferFlags. 266 info.flags |= AVCODEC_BUFFER_FLAGS_CODEC_DATA; // The AVC in annex-b format contains the codec configuration flag. 267 OH_AVBuffer_SetBufferAttr(sample, &info); // Set the buffer attribute. 268 int trackId = audioTrackId; // Select the audio or video track to be written. 269 270 int ret = OH_AVMuxer_WriteSampleBuffer(muxer, trackId, sample); 271 if (ret != AV_ERR_OK) { 272 // Handle exceptions. 273 } 274 ``` 275 27611. Call **OH_AVMuxer_Stop()** to stop multiplexing. 277 278 ```c++ 279 // Call Stop() to write the file trailer. After this API is called, you cannot write media data. 280 if (OH_AVMuxer_Stop(muxer) != AV_ERR_OK) { 281 // Handle exceptions. 282 } 283 ``` 284 28512. Call **OH_AVMuxer_Destroy()** to release the instance. 286 287 Do not repeatedly destroy the instance. Otherwise, the program may crash. 288 289 ```c++ 290 if (OH_AVMuxer_Destroy(muxer) != AV_ERR_OK) { 291 // Handle exceptions. 292 } 293 muxer = NULL; 294 close(fd); // Close the FD. 295 ``` 296