• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1# AI业务子系统<a name="ZH-CN_TOPIC_0000001083278044"></a>
2
3-   [简介](#section187321516154516)
4-   [目录](#section571610913453)
5-   [约束](#section5748426453)
6-   [使用](#section7981135212144)
7-   [相关仓](#section10492183517430)
8-   [AI引擎开发导航](#section6808423133718)
9
10## 简介<a name="section187321516154516"></a>
11
12AI业务子系统是OpenHarmony提供原生的分布式AI能力的子系统。本次开源范围是提供了统一的AI引擎框架，实现算法能力快速插件化集成。框架中主要包含插件管理、模块管理和通信管理等模块，对AI算法能力进行生命周期管理和按需部署。后续，会逐步定义统一的AI能力接口，便于AI能力的分布式调用。同时，提供适配不同推理框架层级的统一推理接口。
13
14**图 1**  AI引擎框架<a name="fig17296164711526"></a>
15![](figures/AI引擎框架.png "AI引擎框架")
16
17## 目录<a name="section571610913453"></a>
18
19```
20/foundation/ai/engine                        # AI子系统主目录
21├── interfaces
22│  └── kits                                  # AI子系统对外接口
23└── services
24│  ├── client                                # AI子系统Client模块
25│  │  ├── client_executor                    # Client模块执行主体
26│  │  └── communication_adapter              # Client模块通信适配层，支持拓展
27│  ├── common                                # AI子系统公共工具、协议模块
28│  │  ├── platform
29│  │  ├── protocol
30│  │  └── utils
31│  └── server                                # AI子系统服务端模块
32│  │  ├── communication_adapter              # Server模块通信适配层，支持拓展
33│  │  ├── plugin
34│  │     ├── asr
35│  │        └── keyword_spotting             # ASR算法插件参考：唤醒词识别
36│  │     └── cv
37│  │        └── image_classification         # CV算法插件参考：图片分类
38│  │  ├── plugin_manager
39│  │  └── server_executor                    # Server模块执行主体
40```
41
42## 约束<a name="section5748426453"></a>
43
44* **语言限制**：C/C++语言
45
46* **操作系统限制**：OpenHarmony操作系统
47
48* **AI服务启动的约束与限制**：SAMGR（System Ability Manager）启动且运行正常
49
50## 使用<a name="section7981135212144"></a>
51
521.  **AI业务子系统编译**
53
54    轻量级AI引擎框架模块，代码所在路径：//foundation/ai/engine/services
55
56    编译指令如下：
57
58    1. **设置编译路径**
59
60    ```
61    hb set -root dir(项目代码根目录)
62    ```
63
64    2. **设置编译产品**（执行后用方向键和回车进行选择）：
65
66    ```
67    hb set -p
68    ```
69
70    3. **执行编译**：
71
72    ```
73    hb build -f（编译全仓）
74    或者 hb build ai_engine（只编译ai_engine组件）
75    ```
76
77   >**注意**：hb相关配置请参考编译构建子系统**build\_lite**
78
792.  **插件开发**（以唤醒词识别为例）
80
81    位置：//foundation/ai/engine/services/server/plugin/asr/keyword\_spotting
82
83    >**注意**：插件需要实现server提供的IPlugin接口和IPluginCallback接口
84
85    ```
86    #include "plugin/i_plugin.h
87    class KWSPlugin : public IPlugin {       // Keywords Spotting Plugin（KWSPlugin）继承IPlugin算法插件基类public:
88        KWSPlugin();
89        ~KWSPlugin();
90
91        const long long GetVersion() const override;
92        const char* GetName() const override;
93        const char* GetInferMode() const override;
94
95        int32_t Prepare(long long transactionId, const DataInfo &amp;amp;inputInfo, DataInfo &amp;amp;outputInfo) override;
96        int32_t SetOption(int optionType, const DataInfo &amp;amp;inputInfo) override;
97        int32_t GetOption(int optionType, const DataInfo &amp;amp;inputInfo, DataInfo &amp;amp;outputInfo) override;
98        int32_t SyncProcess(IRequest *request, IResponse *&amp;amp;response) override;
99        int32_t AsyncProcess(IRequest *request, IPluginCallback*callback) override;
100        int32_t Release(bool isFullUnload, long long transactionId, const DataInfo &amp;amp;inputInfo) override;
101        .
102        .
103        .
104    };
105    ```
106
107    >**注意**：SyncProcess和AsyncProcess接口只需要根据算法实际情况实现一个接口即可，另一个用空方法占位（这里KWS插件为同步算法，故异步接口为空实现）。
108
109    ```
110    #include "aie_log.h"
111    #include "aie_retcode_inner.h"
112    .
113    .
114    .
115
116    const long long KWSPlugin::GetVersion() const
117    {
118        return ALGOTYPE_VERSION_KWS;
119    }
120
121    const char *KWSPlugin::GetName() const
122    {
123        return ALGORITHM_NAME_KWS.c_str();
124    }
125
126    const char *KWSPlugin::GetInferMode() const
127    {
128        return DEFAULT_INFER_MODE.c_str();
129    }
130    .
131    .
132    .
133    int32_t KWSPlugin::AsyncProcess(IRequest *request, IPluginCallback *callback)
134    {
135        return RETCODE_SUCCESS;
136    }
137    ```
138
1393.  **插件SDK开发**（以唤醒词识别kws\_sdk为例）
140
141    位置：//foundation/ai/engine/services/client/algorithm\_sdk/asr/keyword\_spotting
142
143    唤醒词识别SDK：
144
145    ```
146    class KWSSdk {
147    public:
148        KWSSdk();
149        virtual ~KWSSdk();
150
151        /**
152         * @brief Create a new session with KWS Plugin
153         *
154         * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful,
155         *         returns a non-zero value otherwise.
156         */
157        int32_t Create();
158
159        /**
160         * @brief Synchronously execute keyword spotting once
161         *
162         * @param audioInput pcm data.
163         * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful,
164         *         returns a non-zero value otherwise.
165         */
166        int32_t SyncExecute(const Array<int16_t> &audioInput);
167
168        /**
169         * @brief Asynchronously execute keyword spotting once
170         *
171         * @param audioInput pcm data.
172         * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful,
173         *         returns a non-zero value otherwise.
174         */
175        int32_t AsyncExecute(const Array<int16_t> &audioInput);
176
177        /**
178         * @brief Set callback
179         *
180         * @param callback Callback function that will be called during the process.
181         * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful,
182         *         returns a non-zero value otherwise.
183         */
184        int32_t SetCallback(const std::shared_ptr<KWSCallback> &callback);
185
186        /**
187         * @brief Destroy the created session with KWS Plugin
188         *
189         * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful,
190         *         returns a non-zero value otherwise.
191         */
192        int32_t Destroy();
193    ```
194
195   >**注意**：SDK调用AI引擎客户端接口顺序应遵循AieClientInit-\>AieClientPrepare-\>AieClientSyncProcess/AieClientAsyncProcess-\>AieClientRelease-\>AieClientDestroy，否则调用接口会返回错误码；同时应保证各个接口都有调用到，要不然会引起内存泄漏。
196
197
198 ```
199    int32_t KWSSdk::KWSSdkImpl::Create()
200    {
201        if (kwsHandle_ != INVALID_KWS_HANDLE) {
202            HILOGE("[KWSSdkImpl]The SDK has been created");
203            return KWS_RETCODE_FAILURE;
204        }
205        if (InitComponents() != RETCODE_SUCCESS) {
206            HILOGE("[KWSSdkImpl]Fail to init sdk components");
207            return KWS_RETCODE_FAILURE;
208        }
209        int32_t retCode = AieClientInit(configInfo_, clientInfo_, algorithmInfo_, nullptr);
210        if (retCode != RETCODE_SUCCESS) {
211            HILOGE("[KWSSdkImpl]AieClientInit failed. Error code[%d]", retCode);
212            return KWS_RETCODE_FAILURE;
213        }
214        if (clientInfo_.clientId == INVALID_CLIENT_ID) {
215            HILOGE("[KWSSdkImpl]Fail to allocate client id");
216            return KWS_RETCODE_FAILURE;
217        }
218        DataInfo inputInfo = {
219            .data = nullptr,
220            .length = 0,
221        };
222        DataInfo outputInfo = {
223            .data = nullptr,
224            .length = 0,
225        };
226        retCode = AieClientPrepare(clientInfo_, algorithmInfo_, inputInfo, outputInfo, nullptr);
227        if (retCode != RETCODE_SUCCESS) {
228            HILOGE("[KWSSdkImpl]AieclientPrepare failed. Error code[%d]", retCode);
229            return KWS_RETCODE_FAILURE;
230        }
231        if (outputInfo.data == nullptr || outputInfo.length <= 0) {
232            HILOGE("[KWSSdkImpl]The data or length of output info is invalid");
233            return KWS_RETCODE_FAILURE;
234        }
235        MallocPointerGuard<unsigned char> pointerGuard(outputInfo.data);
236        retCode = PluginHelper::UnSerializeHandle(outputInfo, kwsHandle_);
237        if (retCode != RETCODE_SUCCESS) {
238            HILOGE("[KWSSdkImpl]Get handle from inputInfo failed");
239            return KWS_RETCODE_FAILURE;
240        }
241        return KWS_RETCODE_SUCCESS;
242    }
243
244    int32_t KWSSdk::KWSSdkImpl::SyncExecute(const Array<uint16_t> &audioInput)
245    {
246        intptr_t newHandle = 0;
247        Array<int32_t> kwsResult = {
248            .data = nullptr,
249            .size = 0
250        };
251        DataInfo inputInfo = {
252            .data = nullptr,
253            .length = 0
254        };
255        DataInfo outputInfo = {
256            .data = nullptr,
257            .length = 0
258        };
259        int32_t retCode = PluginHelper::SerializeInputData(kwsHandle_, audioInput, inputInfo);
260        if (retCode != RETCODE_SUCCESS) {
261            HILOGE("[KWSSdkImpl]Fail to serialize input data");
262            callback_->OnError(KWS_RETCODE_SERIALIZATION_ERROR);
263            return RETCODE_FAILURE;
264        }
265        retCode = AieClientSyncProcess(clientInfo_, algorithmInfo_, inputInfo, outputInfo);
266        if (retCode != RETCODE_SUCCESS) {
267            HILOGE("[KWSSdkImpl]AieClientSyncProcess failed. Error code[%d]", retCode);
268            callback_->OnError(KWS_RETCODE_PLUGIN_EXECUTION_ERROR);
269            return RETCODE_FAILURE;
270        }
271        if (outputInfo.data == nullptr || outputInfo.length <= 0) {
272            HILOGE("[KWSSdkImpl] The data or length of outputInfo is invalid. Error code[%d]", retCode);
273            callback_->OnError(KWS_RETCODE_NULL_PARAM);
274            return RETCODE_FAILURE;
275        }
276        MallocPointerGuard<unsigned char> pointerGuard(outputInfo.data);
277        retCode = PluginHelper::UnSerializeOutputData(outputInfo, newHandle, kwsResult);
278        if (retCode != RETCODE_SUCCESS) {
279            HILOGE("[KWSSdkImpl]UnSerializeOutputData failed. Error code[%d]", retCode);
280            callback_->OnError(KWS_RETCODE_UNSERIALIZATION_ERROR);
281            return retCode;
282        }
283        if (kwsHandle_ != newHandle) {
284            HILOGE("[KWSSdkImpl]The handle[%lld] of output data is not equal to the current handle[%lld]",
285                (long long)newHandle, (long long)kwsHandle_);
286            callback_->OnError(KWS_RETCODE_PLUGIN_SESSION_ERROR);
287            return RETCODE_FAILURE;
288        }
289        callback_->OnResult(kwsResult);
290        return RETCODE_SUCCESS;
291    }
292
293    int32_t KWSSdk::KWSSdkImpl::Destroy()
294    {
295        if (kwsHandle_ == INVALID_KWS_HANDLE) {
296            return KWS_RETCODE_SUCCESS;
297        }
298        DataInfo inputInfo = {
299            .data = nullptr,
300            .length = 0
301        };
302        int32_t retCode = PluginHelper::SerializeHandle(kwsHandle_, inputInfo);
303        if (retCode != RETCODE_SUCCESS) {
304            HILOGE("[KWSSdkImpl]SerializeHandle failed. Error code[%d]", retCode);
305            return KWS_RETCODE_FAILURE;
306        }
307        retCode = AieClientRelease(clientInfo_, algorithmInfo_, inputInfo);
308        if (retCode != RETCODE_SUCCESS) {
309            HILOGE("[KWSSdkImpl]AieClientRelease failed. Error code[%d]", retCode);
310            return KWS_RETCODE_FAILURE;
311        }
312        retCode = AieClientDestroy(clientInfo_);
313        if (retCode != RETCODE_SUCCESS) {
314            HILOGE("[KWSSdkImpl]AieClientDestroy failed. Error code[%d]", retCode);
315            return KWS_RETCODE_FAILURE;
316        }
317        mfccProcessor_ = nullptr;
318        pcmIterator_ = nullptr;
319        callback_ = nullptr;
320        kwsHandle_ = INVALID_KWS_HANDLE;
321        return KWS_RETCODE_SUCCESS;
322    }
323 ```
324
3254.  **sample开发** [（参考唤醒词识别demo）](https://gitee.com/openharmony/applications_sample_camera/tree/master/ai)
326
327    位置：//applications/sample/camera/ai/asr/keyword\_spotting
328
329    **调用Create**
330
331    ```
332    bool KwsManager::PreparedInference()
333    {
334        if (capturer_ == nullptr) {
335            printf("[KwsManager] only load plugin after AudioCapturer ready\n");
336            return false;
337        }
338        if (plugin_ != nullptr) {
339            printf("[KwsManager] stop created InferencePlugin at first\n");
340            StopInference();
341        }
342        plugin_ = std::make_shared<KWSSdk>();
343        if (plugin_ == nullptr) {
344            printf("[KwsManager] fail to create inferencePlugin\n");
345            return false;
346        }
347        if (plugin_->Create() != SUCCESS) {
348            printf("[KwsManager] KWSSdk fail to create.\n");
349            return false;
350        }
351        std::shared_ptr<KWSCallback> callback = std::make_shared<MyKwsCallback>();
352        if (callback == nullptr) {
353            printf("[KwsManager] new Callback failed.\n");
354            return false;
355        }
356        plugin_->SetCallback(callback);
357        return true;
358    }
359    ```
360
361    **调用SyncExecute**
362
363    ```
364    void KwsManager::ConsumeSamples()
365    {
366        uintptr_t sampleAddr = 0;
367        size_t sampleSize = 0;
368        int32_t retCode = SUCCESS;
369        while (status_ == RUNNING) {
370            {
371                std::lock_guard<std::mutex> lock(mutex_);
372                if (cache_ == nullptr) {
373                    printf("[KwsManager] cache_ is nullptr.\n");
374                    break;
375                }
376                sampleSize = cache_->GetCapturedBuffer(sampleAddr);
377            }
378            if (sampleSize == 0 || sampleAddr == 0) {
379                continue;
380            }
381            Array<int16_t> input = {
382                .data = (int16_t *)(sampleAddr),
383                .size = sampleSize >> 1
384            };
385            {
386                std::lock_guard<std::mutex> lock(mutex_);
387                if (plugin_ == nullptr) {
388                    printf("[KwsManager] cache_ is nullptr.\n");
389                    break;
390                }
391                if ((retCode = plugin_->SyncExecute(input)) != SUCCESS) {
392                    printf("[KwsManager] SyncExecute KWS failed with retCode = [%d]\n", retCode);
393                    continue;
394                }
395            }
396        }
397    }
398    ```
399
400    **调用Destroy**
401
402    ```
403    void KwsManager::StopInference()
404    {
405        printf("[KwsManager] StopInference\n");
406        if (plugin_ != nullptr) {
407            int ret = plugin_->Destroy();
408            if (ret != SUCCESS) {
409                printf("[KwsManager] plugin_ destroy failed.\n");
410            }
411            plugin_ = nullptr;
412        }
413    }
414    ```
415
416
417## 相关仓<a name="section10492183517430"></a>
418
419[AI子系统](https://gitee.com/openharmony/docs/blob/master/zh-cn/readme/AI%E4%B8%9A%E5%8A%A1%E5%AD%90%E7%B3%BB%E7%BB%9F.md)
420
421[ai_engine](https://gitee.com/openharmony/ai_engine)
422
423## 依赖仓：
424
425[build\_lite](https://gitee.com/openharmony/build_lite/blob/master/README_zh.md)
426
427[systemabilitymgr\_samgr\_lite](https://gitee.com/openharmony/systemabilitymgr_samgr_lite/blob/master/README_zh.md)
428
429[startup\_init\_lite](https://gitee.com/openharmony/startup_init_lite/blob/master/README_zh.md)
430
431## AI引擎开发导航<a name="section6808423133718"></a>
432
433[《AI插件开发指南》](https://gitee.com/openharmony/docs/blob/master/zh-cn/device-dev/subsystems/subsys-ai-aiframework-devguide.md)