1# Introduction to Media Kit 2 3Media Kit provides [AVPlayer](#avplayer) and [AVRecorder](#avrecorder) for playing and recording audio and video. 4 5You will learn how to use the APIs provided by Media Kit to develop a wealth of audio and video playback or recording scenarios. For example, you can use **SoundPool** to implement simple prompt tones so that a drip sound is played upon the receipt of a new message, or use **AVPlayer** to develop a music player, which can loop a piece of music. 6 7## Highlights 8 9- Lightweight media engine 10 11 Less system resources (threads and memory) are required. Audio and video playback and recording, flexible pipeline assembly, and source, demuxer, and codec plugins are supported. 12 13- HDR video 14 15 The native data structures and interfaces are provided to support the capture and playback of HDR VIVID. Third-party applications can deliver a more immersive experience by leveraging the HDR capability of the system. 16 17- Sound pool 18 19 Short sound effects (such as the camera shutter sound effect and system notification sound effect) are often required during application development. You can call the APIs provided by **SoundPool** to implement one-time loading of short sounds and multiple times of low-latency playback. 20 21## Development Description 22 23This development guide applies only to audio and video playback or recording, which are implemented by Media Kit. The UI, image processing, media storage, or other related capabilities are not covered. 24 25Before developing features related to audio and video playback or recording, you are advised to understand the following concepts: 26 27- Playback process: network protocol > container format > audio and video codec > graphics/audio rendering 28 29- Network protocols: HLS, HTTP, HTTPS, and more 30 31- Container formats: mp4, mkv, mpeg-ts, and more 32 33- Encoding format: H.264, H.265, and more 34 35## AVPlayer 36 37The AVPlayer transcodes audio and video media assets (such as MP4, MP3, MKV, and MPEG-TS) into renderable images and hearable audio analog signals, and plays the audio and video through output devices. 38 39The AVPlayer provides the integrated playback capability. This means that your application only needs to provide streaming media sources to implement media playback. It does not need to parse or decode data. 40 41### Audio Playback 42 43The figure below shows the interaction when the **AVPlayer** class is used to develop a music application. 44 45**Figure 1** Interaction with external modules for audio playback 46 47 48 49When a music application calls the **AVPlayer** APIs at the JS interface layer to implement audio playback, the player framework at the framework layer parses the media asset into audio data streams (in PCM format). The audio data streams are then decoded by software and output to the audio framework. The audio framework outputs the audio data streams to the audio HDI for rendering. A complete audio playback process requires the cooperation of the application, player framework, audio framework, and audio HDI. 50 51In Figure 1, the numbers indicate the process where data is transferred to external modules. 52 531. The music application transfers the media asset to the **AVPlayer** instance. 54 552. The player framework outputs the audio PCM data streams to the audio framework, which then outputs the data streams to the audio HDI. 56 57### Video Playback 58 59The figure below shows the interaction when the **AVPlayer** class is used to develop a video application. 60 61**Figure 2** Interaction with external modules for video playback 62 63 64 65When the video application calls the **AVPlayer** APIs at the JS interface layer to implement audio and video playback, the player framework at the framework layer parses the media asset into separate audio data streams and video data streams. The audio data streams are then decoded by software and output to the audio framework. The audio framework outputs the audio data streams to the audio HDI at the hardware interface layer to implement audio playback. The video data streams are then decoded by hardware (recommended) or software and output to the graphic framework. The graphic framework outputs the video data streams to the display HDI at the hardware interface layer to implement graphics rendering. 66 67A complete video playback process requires the cooperation of the application, XComponent, player framework, graphic framework, audio framework, display HDI, and audio HDI. 68 69In Figure 2, the numbers indicate the process where data is transferred to external modules. 70 711. The application obtains a window surface ID from the XComponent. For details about how to obtain the window surface ID, see [XComponent](../reference/apis-arkui/arkui-ts/ts-basic-components-xcomponent.md). 72 732. The application transfers the media asset and surface ID to the **AVPlayer** instance. 74 753. The player framework outputs the video elementary streams (ESs) to the decoding HDI to obtain video frames (NV12/NV21/RGBA). 76 774. The player framework outputs the audio PCM data streams to the audio framework, which then outputs the data streams to the audio HDI. 78 795. The player framework outputs the video frames (NV12/NV21/RGBA) to the graphic framework, which then outputs the video frames to the display HDI. 80 81### Supported Formats and Protocols 82 83Audio and video containers and codecs are domains specific to content creators. You are advised to use the mainstream playback formats, rather than custom ones to avoid playback failures, frame freezing, and artifacts. The system will not be affected by incompatibility issues. If such an issue occurs, you can exit playback. 84 85The table below lists the supported protocols. 86 87| Scenario| Description| 88| -------- | -------- | 89| Local VOD| The file descriptor is supported, but the file path is not.| 90| Network VoD| HTTP, HTTPS, and HLS are supported.| 91| Live webcasting| HLS is supported.| 92 93The table below lists the supported audio playback formats. 94 95| Audio Container Format| Description| 96| -------- | -------- | 97| m4a | Audio format: AAC| 98| aac | Audio format: AAC| 99| mp3 | Audio format: MP3| 100| ogg | Audio format: VORBIS | 101| wav | Audio format: PCM| 102 103> **NOTE** 104> 105> The supported video formats are further classified into mandatory and optional ones. All vendors must support mandatory ones and can determine whether to implement optional ones based on their service requirements. You are advised to perform compatibility processing to ensure that all the application functions are compatible on different platforms. 106 107| Video Format| Mandatory or Not| 108| -------- | -------- | 109| H265<sup>10+</sup> | Yes| 110| H264 | Yes| 111 112The table below lists the supported playback formats and mainstream resolutions. 113 114| Video Container Format| Description| Resolution| 115| -------- | -------- | -------- | 116| mp4 | Video formats: H.265<sup>10+</sup>, H.264, MPEG2, and MPEG4<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p| 117| mkv | Video formats: H.265<sup>10+</sup>, and H.264<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p| 118| ts | Video formats: H.265<sup>10+</sup>, and H.264<br>Audio formats: AAC and MP3| Mainstream resolutions, such as 4K, 1080p, 720p, 480p, and 270p| 119 120## AVRecorder 121 122The AVRecorder captures audio signals, receives video signals, encodes the audio and video signals, and saves them to files. With the AVRecorder, you can easily implement audio and video recording, including starting, pausing, resuming, and stopping recording, and releasing resources. You can also specify parameters such as the encoding format, encapsulation format, and file path for recording. 123 124**Figure 3** Interaction with external modules for video recording 125 126 127 128- Audio recording: When an application calls the **AVRecorder** APIs at the JS interface layer to implement audio recording, the player framework at the framework layer invokes the audio framework to capture audio data through the audio HDI. The audio data is then encoded by software and saved into a file. 129 130- Video recording: When an application calls the **AVRecorder** APIs at the JS interface layer to implement video recording, the camera framework is first invoked to capture image data. Through the video encoding HDI, the camera framework sends the data to the player framework at the framework layer. The player framework encodes the image data through the video HDI and saves the encoded image data into a file. 131 132With the AVRecorder, you can implement pure audio recording, pure video recording, and audio and video recording. 133 134In Figure 3, the numbers indicate the process where data is transferred to external modules. 135 1361. The application obtains a surface ID from the player framework through the **AVRecorder** instance. 137 1382. The application sets the surface ID for the camera framework, which obtains the surface corresponding to the surface ID. The camera framework captures image data through the video HDI and sends the data to the player framework at the framework layer. 139 1403. The camera framework transfers the video data to the player framework through the surface. 141 1424. The player framework encodes video data through the video HDI. 143 1445. The player framework sets the audio parameters for the audio framework and obtains the audio data from the audio framework. 145 146### Supported Formats 147 148The table below lists the supported audio sources. 149 150| Type| Description| 151| -------- | -------- | 152| mic | The system microphone is used as the audio source input.| 153 154The table below lists the supported video sources. 155 156| Type| Description| 157| -------- | -------- | 158| surface_yuv | The input surface carries raw data.| 159| surface_es | The input surface carries ES data.| 160 161The table below lists the supported audio and video encoding formats. 162 163| Encoding Format| Description| 164| -------- | -------- | 165| audio/mp4a-latm | Audio encoding format MP4A-LATM.| 166| video/hevc | Video encoding format HEVC.| 167| video/avc | Video encoding format AVC.| 168 169The table below lists the supported output file formats. 170 171| Format| Description| 172| -------- | -------- | 173| mp4 | Video container format MP4.| 174| m4a | Audio container format M4A.| 175
