1# AI Subsystem<a name="ZH-CN_TOPIC_0000001083278044"></a> 2 3- [Introduction](#section187321516154516) 4- [Directory Structure](#section571610913453) 5- [Constraints](#section5748426453) 6- [Usage](#section7981135212144) 7- [Repositories Involved](#section10492183517430) 8- [AI Engine Development](#section6808423133718) 9 10## Introduction<a name="section187321516154516"></a> 11 12The AI subsystem is the part of OpenHarmony that provides native distributed AI capabilities. At the heart of the subsystem is a unified AI engine framework, which implements quick integration of AI algorithm plug-ins. The framework consists of the plug-in management, module management, and communication management modules, fulfilling lifecycle management and on-demand deployment of AI algorithms. Under this framework, AI algorithm APIs will be standardized to facilitate distributed calling of AI capabilities. In addition, unified inference APIs will be provided to adapt to different inference framework hierarchies. 13 14**Figure 1** AI engine framework<a name="fig17296164711526"></a> 15 16 17 18## Directory Structure<a name="section571610913453"></a> 19 20``` 21/foundation/ai/engine # Home directory of the AI subsystem 22├── interfaces 23│ └── kits # External APIs of the AI subsystem 24└── services 25│ ├── client # Client module of the AI subsystem 26│ │ ├── client_executor # Executor of the client module 27│ │ └── communication_adapter # Communication adaptation layer for the client module, with extension supported 28│ ├── common # Common tools and protocol modules of the AI subsystem 29│ │ ├── platform 30│ │ ├── protocol 31│ │ └── utils 32│ └── server # Server module of the AI subsystem 33│ │ ├── communication_adapter # Communication adaptation layer for the server module, with extension supported 34│ │ ├── plugin 35│ │ ├── asr 36│ │ └── keyword_spotting # ASR algorithm plug-in reference: keyword spotting 37│ │ └── cv 38│ │ └── image_classification # CV algorithm plug-in reference: image classification 39│ │ ├── plugin_manager 40│ │ └── server_executor # Executor of the server module 41``` 42 43## Constraints<a name="section5748426453"></a> 44 45* **Programming language**: C/C++ 46 47* **Operating system**: OpenHarmony mini- and small-system 48 49* **Others**: The System Ability Manager (Samgr) has been started and is running properly. 50 51## Usage<a name="section7981135212144"></a> 52 531. Compile the AI subsystem. 54 55 The source code for lightweight AI framework is available at **//foundation/ai/engine/services**. 56 57 The compilation procedure is as follows: 58 59 1. Set the build path. 60 61 ``` 62 hb set -root dir (**root dir** is the root directory of the project code.) 63 ``` 64 65 2. Specify the build product. \(After the product list is displayed using the following command, move to the desired product with arrow keys and press **Enter**.\) 66 67 ``` 68 hb set -p 69 ``` 70 71 3. Start build. 72 73 ``` 74 hb build -f (Use this command if you want to compile the entire repository.) 75 hb build ai_engine (Use this command if you want to compile only the ai_engine module.) 76 ``` 77 78 >**NOTE**<br>For details about the hb configuration, see the readme of the build_lite subsystem. 79 802. Develop the plug-in, with keyword spotting as an example. 81 82 Directory: //foundation/ai/engine/services/server/plugin/asr/keyword\_spotting 83 84 >**NOTE**<br>The plug-in must implement the **IPlugin** and **IPluginCallback** APIs provided by the server. 85 86 ``` 87 #include "plugin/i_plugin.h 88 class KWSPlugin : public IPlugin { // Inherit the public base class of the IPlugin API for Keywords Spotting Plugin (KWSPlugin). 89 KWSPlugin(); 90 ~KWSPlugin(); 91 92 const long long GetVersion() const override; 93 const char* GetName() const override; 94 const char* GetInferMode() const override; 95 96 int32_t Prepare(long long transactionId, const DataInfo &amp;inputInfo, DataInfo &amp;outputInfo) override; 97 int32_t SetOption(int optionType, const DataInfo &amp;inputInfo) override; 98 int32_t GetOption(int optionType, const DataInfo &amp;inputInfo, DataInfo &amp;outputInfo) override; 99 int32_t SyncProcess(IRequest *request, IResponse *&amp;response) override; 100 int32_t AsyncProcess(IRequest *request, IPluginCallback*callback) override; 101 int32_t Release(bool isFullUnload, long long transactionId, const DataInfo &amp;inputInfo) override; 102 . 103 . 104 . 105 }; 106 ``` 107 108 >**NOTE**<br>Depending on the algorithm in use, you only need to implement the **SyncProcess** or **AsyncProcess** API. Use an empty function as a placeholder for the other API. In this example, the KWS plug-in uses the synchronous algorithm. Therefore, you need to implement **SyncProcess** API and use an empty function as a placeholder for the **SyncProcess** API. 109 110 ``` 111 #include "aie_log.h" 112 #include "aie_retcode_inner.h" 113 . 114 . 115 . 116 117 const long long KWSPlugin::GetVersion() const 118 { 119 return ALGOTYPE_VERSION_KWS; 120 } 121 122 const char *KWSPlugin::GetName() const 123 { 124 return ALGORITHM_NAME_KWS.c_str(); 125 } 126 127 const char *KWSPlugin::GetInferMode() const 128 { 129 return DEFAULT_INFER_MODE.c_str(); 130 } 131 . 132 . 133 . 134 int32_t KWSPlugin::AsyncProcess(IRequest *request, IPluginCallback *callback) 135 { 136 return RETCODE_SUCCESS; 137 } 138 ``` 139 1403. Develop the SDK, with keyword spotting as an example. 141 142 Directory: //foundation/ai/engine/services/client/algorithm\_sdk/asr/keyword\_spotting 143 144 Keyword spotting SDK: 145 146 ``` 147 class KWSSdk { 148 public: 149 KWSSdk(); 150 virtual ~KWSSdk(); 151 152 /** 153 * @brief Create a new session with KWS Plugin 154 * 155 * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful, 156 * returns a non-zero value otherwise. 157 */ 158 int32_t Create(); 159 160 /** 161 * @brief Synchronously execute keyword spotting once 162 * 163 * @param audioInput pcm data. 164 * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful, 165 * returns a non-zero value otherwise. 166 */ 167 int32_t SyncExecute(const Array<int16_t> &audioInput); 168 169 /** 170 * @brief Asynchronously execute keyword spotting once 171 * 172 * @param audioInput pcm data. 173 * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful, 174 * returns a non-zero value otherwise. 175 */ 176 int32_t AsyncExecute(const Array<int16_t> &audioInput); 177 178 /** 179 * @brief Set callback 180 * 181 * @param callback Callback function that will be called during the process. 182 * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful, 183 * returns a non-zero value otherwise. 184 */ 185 int32_t SetCallback(const std::shared_ptr<KWSCallback> &callback); 186 187 /** 188 * @brief Destroy the created session with KWS Plugin 189 * 190 * @return Returns KWS_RETCODE_SUCCESS(0) if the operation is successful, 191 * returns a non-zero value otherwise. 192 */ 193 int32_t Destroy(); 194 } 195 ``` 196 197 >**NOTE**<br>The sequence for the SDK to call client APIs of the AI engine is as follows: AieClientInit -\> AieClientPrepare -\> AieClientSyncProcess/AieClientAsyncProcess -\> AieClientRelease -\> AieClientDestroy. An exception will be thrown if the call sequence is violated. In addition, all these APIs must be called. Otherwise, a memory leakage may occur. 198 199 200 ``` 201 int32_t KWSSdk::KWSSdkImpl::Create() 202 { 203 if (kwsHandle_ != INVALID_KWS_HANDLE) { 204 HILOGE("[KWSSdkImpl]The SDK has been created"); 205 return KWS_RETCODE_FAILURE; 206 } 207 if (InitComponents() != RETCODE_SUCCESS) { 208 HILOGE("[KWSSdkImpl]Fail to init sdk components"); 209 return KWS_RETCODE_FAILURE; 210 } 211 int32_t retCode = AieClientInit(configInfo_, clientInfo_, algorithmInfo_, nullptr); 212 if (retCode != RETCODE_SUCCESS) { 213 HILOGE("[KWSSdkImpl]AieClientInit failed. Error code[%d]", retCode); 214 return KWS_RETCODE_FAILURE; 215 } 216 if (clientInfo_.clientId == INVALID_CLIENT_ID) { 217 HILOGE("[KWSSdkImpl]Fail to allocate client id"); 218 return KWS_RETCODE_FAILURE; 219 } 220 DataInfo inputInfo = { 221 .data = nullptr, 222 .length = 0, 223 }; 224 DataInfo outputInfo = { 225 .data = nullptr, 226 .length = 0, 227 }; 228 retCode = AieClientPrepare(clientInfo_, algorithmInfo_, inputInfo, outputInfo, nullptr); 229 if (retCode != RETCODE_SUCCESS) { 230 HILOGE("[KWSSdkImpl]AieclientPrepare failed. Error code[%d]", retCode); 231 return KWS_RETCODE_FAILURE; 232 } 233 if (outputInfo.data == nullptr || outputInfo.length <= 0) { 234 HILOGE("[KWSSdkImpl]The data or length of output info is invalid"); 235 return KWS_RETCODE_FAILURE; 236 } 237 MallocPointerGuard<unsigned char> pointerGuard(outputInfo.data); 238 retCode = PluginHelper::UnSerializeHandle(outputInfo, kwsHandle_); 239 if (retCode != RETCODE_SUCCESS) { 240 HILOGE("[KWSSdkImpl]Get handle from inputInfo failed"); 241 return KWS_RETCODE_FAILURE; 242 } 243 return KWS_RETCODE_SUCCESS; 244 } 245 246 int32_t KWSSdk::KWSSdkImpl::SyncExecute(const Array<uint16_t> &audioInput) 247 { 248 intptr_t newHandle = 0; 249 Array<int32_t> kwsResult = { 250 .data = nullptr, 251 .size = 0 252 }; 253 DataInfo inputInfo = { 254 .data = nullptr, 255 .length = 0 256 }; 257 DataInfo outputInfo = { 258 .data = nullptr, 259 .length = 0 260 }; 261 int32_t retCode = PluginHelper::SerializeInputData(kwsHandle_, audioInput, inputInfo); 262 if (retCode != RETCODE_SUCCESS) { 263 HILOGE("[KWSSdkImpl]Fail to serialize input data"); 264 callback_->OnError(KWS_RETCODE_SERIALIZATION_ERROR); 265 return RETCODE_FAILURE; 266 } 267 retCode = AieClientSyncProcess(clientInfo_, algorithmInfo_, inputInfo, outputInfo); 268 if (retCode != RETCODE_SUCCESS) { 269 HILOGE("[KWSSdkImpl]AieClientSyncProcess failed. Error code[%d]", retCode); 270 callback_->OnError(KWS_RETCODE_PLUGIN_EXECUTION_ERROR); 271 return RETCODE_FAILURE; 272 } 273 if (outputInfo.data == nullptr || outputInfo.length <= 0) { 274 HILOGE("[KWSSdkImpl] The data or length of outputInfo is invalid. Error code[%d]", retCode); 275 callback_->OnError(KWS_RETCODE_NULL_PARAM); 276 return RETCODE_FAILURE; 277 } 278 MallocPointerGuard<unsigned char> pointerGuard(outputInfo.data); 279 retCode = PluginHelper::UnSerializeOutputData(outputInfo, newHandle, kwsResult); 280 if (retCode != RETCODE_SUCCESS) { 281 HILOGE("[KWSSdkImpl]UnSerializeOutputData failed. Error code[%d]", retCode); 282 callback_->OnError(KWS_RETCODE_UNSERIALIZATION_ERROR); 283 return retCode; 284 } 285 if (kwsHandle_ != newHandle) { 286 HILOGE("[KWSSdkImpl]The handle[%lld] of output data is not equal to the current handle[%lld]", 287 (long long)newHandle, (long long)kwsHandle_); 288 callback_->OnError(KWS_RETCODE_PLUGIN_SESSION_ERROR); 289 return RETCODE_FAILURE; 290 } 291 callback_->OnResult(kwsResult); 292 return RETCODE_SUCCESS; 293 } 294 295 int32_t KWSSdk::KWSSdkImpl::Destroy() 296 { 297 if (kwsHandle_ == INVALID_KWS_HANDLE) { 298 return KWS_RETCODE_SUCCESS; 299 } 300 DataInfo inputInfo = { 301 .data = nullptr, 302 .length = 0 303 }; 304 int32_t retCode = PluginHelper::SerializeHandle(kwsHandle_, inputInfo); 305 if (retCode != RETCODE_SUCCESS) { 306 HILOGE("[KWSSdkImpl]SerializeHandle failed. Error code[%d]", retCode); 307 return KWS_RETCODE_FAILURE; 308 } 309 retCode = AieClientRelease(clientInfo_, algorithmInfo_, inputInfo); 310 if (retCode != RETCODE_SUCCESS) { 311 HILOGE("[KWSSdkImpl]AieClientRelease failed. Error code[%d]", retCode); 312 return KWS_RETCODE_FAILURE; 313 } 314 retCode = AieClientDestroy(clientInfo_); 315 if (retCode != RETCODE_SUCCESS) { 316 HILOGE("[KWSSdkImpl]AieClientDestroy failed. Error code[%d]", retCode); 317 return KWS_RETCODE_FAILURE; 318 } 319 mfccProcessor_ = nullptr; 320 pcmIterator_ = nullptr; 321 callback_ = nullptr; 322 kwsHandle_ = INVALID_KWS_HANDLE; 323 return KWS_RETCODE_SUCCESS; 324 } 325 ``` 326 3274. Develop a sample application. 328 329 Call the **Create** API. 330 331 ``` 332 bool KwsManager::PreparedInference() 333 { 334 if (capturer_ == nullptr) { 335 printf("[KwsManager] only load plugin after AudioCapturer ready\n"); 336 return false; 337 } 338 if (plugin_ != nullptr) { 339 printf("[KwsManager] stop created InferencePlugin at first\n"); 340 StopInference(); 341 } 342 plugin_ = std::make_shared<KWSSdk>(); 343 if (plugin_ == nullptr) { 344 printf("[KwsManager] fail to create inferencePlugin\n"); 345 return false; 346 } 347 if (plugin_->Create() != SUCCESS) { 348 printf("[KwsManager] KWSSdk fail to create.\n"); 349 return false; 350 } 351 std::shared_ptr<KWSCallback> callback = std::make_shared<MyKwsCallback>(); 352 if (callback == nullptr) { 353 printf("[KwsManager] new Callback failed.\n"); 354 return false; 355 } 356 plugin_->SetCallback(callback); 357 return true; 358 } 359 ``` 360 361 Call the **SyncExecute** API. 362 363 ``` 364 void KwsManager::ConsumeSamples() 365 { 366 uintptr_t sampleAddr = 0; 367 size_t sampleSize = 0; 368 int32_t retCode = SUCCESS; 369 while (status_ == RUNNING) { 370 { 371 std::lock_guard<std::mutex> lock(mutex_); 372 if (cache_ == nullptr) { 373 printf("[KwsManager] cache_ is nullptr.\n"); 374 break; 375 } 376 sampleSize = cache_->GetCapturedBuffer(sampleAddr); 377 } 378 if (sampleSize == 0 || sampleAddr == 0) { 379 continue; 380 } 381 Array<int16_t> input = { 382 .data = (int16_t *)(sampleAddr), 383 .size = sampleSize >> 1 384 }; 385 { 386 std::lock_guard<std::mutex> lock(mutex_); 387 if (plugin_ == nullptr) { 388 printf("[KwsManager] cache_ is nullptr.\n"); 389 break; 390 } 391 if ((retCode = plugin_->SyncExecute(input)) != SUCCESS) { 392 printf("[KwsManager] SyncExecute KWS failed with retCode = [%d]\n", retCode); 393 continue; 394 } 395 } 396 } 397 } 398 ``` 399 400 Call the **Destroy** API. 401 402 ``` 403 void KwsManager::StopInference() 404 { 405 printf("[KwsManager] StopInference\n"); 406 if (plugin_ != nullptr) { 407 int ret = plugin_->Destroy(); 408 if (ret != SUCCESS) { 409 printf("[KwsManager] plugin_ destroy failed.\n"); 410 } 411 plugin_ = nullptr; 412 } 413 } 414 ``` 415 416 417## Repositories Involved<a name="section10492183517430"></a> 418 419[AI subsystem](https://gitee.com/openharmony/docs/blob/master/en/readme/ai.md) 420 421[ai_engine](https://gitee.com/openharmony/ai_engine) 422 423## Dependency Repositories 424 425[build\_lite](https://gitee.com/openharmony/build_lite/blob/master/README.md) 426 427[systemabilitymgr\_samgr\_lite](https://gitee.com/openharmony/systemabilitymgr_samgr_lite/blob/master/README.md) 428 429[startup\_init\_lite](https://gitee.com/openharmony/startup_init/blob/master/README.md) 430 431## Reference<a name="section6808423133718"></a> 432 433[AI Framework Development](https://gitee.com/openharmony/docs/blob/master/en/device-dev/subsystems/subsys-ai-aiframework-devguide.md) 434 435<!--no_check-->
