1# AI Framework Development 2 3## **Overview** 4 5### Introduction 6 7The AI subsystem is the part of OpenHarmony that provides native distributed AI capabilities. At the heart of the subsystem is a unified AI engine framework, which implements quick integration of AI algorithm plug-ins. 8 9The framework consists of the plug-in management, module management, and communication management modules, fulfilling lifecycle management and on-demand deployment of AI algorithms. Specifically, plug-in management implements lifecycle management, on-demand deployment, and quick integration of AI algorithm plug-ins; module management implements task scheduling and client instance management; communication management manages inter-process communication (IPC) between the client and server and data transmission between the AI engine and plug-ins. 10 11Under this framework, AI algorithm APIs will be standardized to facilitate distributed calling of AI capabilities. In addition, unified inference APIs will be provided to adapt to different inference framework hierarchies. 12 13The following figure shows the AI engine framework. 14 15 **Figure 1** AI engine framework 16  17 18### Setting Up the Environment 19 20 211. Prepare a development board, which can be Hi3516D V300 or Hi3518E V300. 22 232. [Download the source code.](../get-code/sourcecode-acquire.md) 24 25## Technical specifications 26 27### Code Management 28 29The AI engine framework consists of three modules: **client**, **server**, and **common**. The client module provides the server connection management function. The SDK needs to encapsulate and call the public APIs provided by the client in the algorithm's external APIs. The server module provides functions such as plug-in loading and task management. Plug-ins are integrated using the plug-in APIs provided by the server. The common module provides platform-related operation methods, engine protocols, and tool classes for other modules. 30 31 32The following figure shows the code dependency between modules of the AI engine framework. 33 34 35 **Figure 2** Code dependency 36 37  38 39 40#### Recommendation: Develop plug-ins and SDKs in the directories specified by the AI engine. 41 42In the overall planning of the AI engine framework, SDKs are a part of the client, and plug-ins are called by the server and are considered a part of the server. Therefore, the following directories have been planned for plug-in and SDK development in the AI engine framework: 43 44- SDK code directory: //foundation/ai/engine/services/client/algorithm_sdk 45 46 Example 1: //foundation/ai/engine/services/client/algorithm_sdk/cv 47 48 Example 2: //foundation/ai/engine/services/client/algorithm_sdk/nlu 49 50- Plug-in code directory: //foundation/ai/engine/services/server/plugin 51 52 Example 1: //foundation/ai/engine/services/server/plugin/cv 53 54 Example 2: //foundation/ai/engine/services/server/plugin/nlu 55 56 57#### Rule: Store all external APIs provided by plug-ins in the **interfaces/kits** directory of the AI subsystem. 58 59The AI subsystem exposes its capabilities through external APIs of SDKs. According to API management requirements of OpenHarmony, store all external APIs of SDKs in the **interfaces/kits** directory of the subsystem. Currently, the external APIs of plug-ins of the AI subsystem are stored in the **//foundation/ai/engine/interfaces/kits** directory. You can add a sub-directory for each newly added plug-in in this directory. For example, if you add a CV plug-in, then store its external APIs in the **//foundation/ai/engine/interfaces/kits/cv** directory. 60 61 62#### Rule: Make sure that plug-in build results are stored in the **/usr/lib** directory. 63 64Plug-in loading on the server uses the dlopen mode and can only be performed in the** /usr/lib** directory. Therefore, when compiling the **.so** file of a plug-in, set the output directory as **/usr/lib** in the build configuration file. 65 66### Naming rule 67 68#### Rule: Name an SDK in the format of **domain\_keyword<*other information 1*\_*other information 2*\_…>\_sdk.so**. 69 70You are advised to use the commonly known abbreviations for domains. For example, use **cv** for image and video, **asr** for voice recognition, and **translation** for text translation. Add one if there is no available abbreviation for a domain. Use keywords that accurately describe the algorithm capability of the plug-in. For example, use **keyword\_spotting** for wakeup keyword spotting (KWS). Add other information, such as the supported chip type and applicable region, between **keyword** and **sdk**, with each of them separated by an underscore (\_). Note that the name of a SDK must end with **\_sdk**. 71 72For example, if the SDK for the KWS plug-in supports only the Kirin 9000 chipset and is applicable only in China, then name the SDK as follows: **asr\_keyword\_spotting\_kirin9000\_china\_sdk.so**. 73 74 75#### Rule: Name a plug-in in the format of **domain\_keyword<*other information 1*\_*other information 2*\_…>.so**. 76 77There is a one-to-one mapping between plug-ins and SDKs. Therefore, the definitions and requirements of terms such as the domain, keyword, and other information in plug-in names are the same as those in SDK names. The only difference is that the name of the SDK ends with **\_sdk** additionally. For example, if the plug-in is named **asr\_keyword\_spotting.so**, the corresponding SDK is named **asr\_keyword\_spotting\_sdk.so**. 78 79For example, if the SDK for the KWS plug-in supports only the Kirin 9000 chipset and is applicable only in China, then name the plug-in as follows: **asr\_keyword\_spotting\_kirin9000\_china.so**. 80 81### API Development 82 83#### Rule: Encapsulate the external APIs provided by the client based on the algorithm call sequence. For the SDK of an asynchronous plug-in, implement the **IClientCb** API provided by the client. 84 85The external APIs provided by the client of the AI engine include **AieClientInit**, **AieClientPrepare**, **AieClientSyncProcess**, **AieClientAsyncProcess**, **AieClientRelease**, **AieClientDestroy**, **AieClientSetOption**, and **AieClientGetOption**. The SDK needs to encapsulate at least the following five APIs in sequence: **AieClientInit**, **AieClientPrepare**, **AieClientSyncProcess** (or **AieClientAsyncProcess**), **AieClientRelease**, and **AieClientDestroy**. Otherwise, a call failure or memory leakage may occur. For example, if the **AieClientPrepare** API is omitted during encapsulation, the server cannot load the plug-in. As a result, APIs that follow it cannot be called. 86 87For an asynchronous plug-in, the SDK needs to implement the **IClientCb** API to receive the algorithm inference result from the client and return the result to the third-party caller. 88 89 90#### Rule: Save all common data related to client interaction in the SDK during API implementation. 91 92The client of the AI engine uses the singleton pattern for API implementation. If the client is connecting to multiple SDKs, each SDK needs to store all common data exchanged with the client so that they can connect to the server to perform operations such as task inference and return the result. Common data usually includes **clientInfo**, **algorithmInfo**, and **configInfo**, which are defined in the SDK's member variables. 93 94 95#### Recommendation: Enable the SDK to implement the **IServiceDeadCb** API defined by the client. 96 97The processes running on the server are system resident processes. The server provides services for clients by way of system capabilities. The **IServiceDeadCb** API is called if a server process is abnormally killed. The SDK can implement related operations in this API, for example, stopping process call or restarting the server. 98 99The following is an example of **IServiceDeadCb** API implementation: 100 101 102``` 103class ServiceDeadCb : public IServiceDeadCb { 104public: 105ServiceDeadCb() = default; 106~ServiceDeadCb() override = default; 107void OnServiceDead() override 108{ 109printf("[ServiceDeadCb]OnServiceDead Callback happens"); 110} 111}; 112``` 113 114As shown above, the SDK can implement its own operations in the **OnServiceDead()** function, for example, stopping API call. 115 116 117#### Rule: Convert dedicated algorithm data into common data of the AI engine if the SDK and plug-ins need to use the codec module. 118 119For plug-ins, inference data is transmitted by the third-party caller to them through the client and server. The required data type varies according to algorithms. For example, the CV algorithm requires image data, and the ASR algorithm requires audio data. To address this issue, the AI engine provides the codec capabilities to convert different types of data into common data that can be used by it. 120 121The encoded data is as follows: 122 123 124``` 125struct DataInfo { 126unsigned char *data; 127int length; 128} DataInfo; 129 130``` 131 132As shown above, **DataInfo** consists of two variables: a pointer to the data memory, and the data length. 133 134To use the APIs of the AI engine framework, you need to: 135 1361. Add the dependency header file utils/encdec/include/encdec.h. 137 1382. Add the dependency items in the **build.gn** file. 139 140Add **//foundation/ai/engine/services/common** to **include\_dirs**. 141 142Add **//foundation/ai/engine/services/common/utils/encdec:encdec** to **deps**. 143 1443. Convert different types of data through codec. The following is an example: 145 146 147``` 148// Example function for encoding: arg1, arg2, and arg3 are variables to be encoded, and dataInfo is the encoding result. 149retCode = ProcessEncode(dataInfo, arg1, arg2, arg3) // The number of parameters can be flexible. 150// Example function for decoding: dataInfo is the data to be decoded, and arg1, arg2, and arg3 are the decoding result. 151retCode = ProcessDecode(dataInfo, arg1, arg2, arg3) // The number of parameters can be flexible. 152``` 153 154> **NOTE** 155> 156> - The sequence of parameters must be the same during encoding and decoding. 157> - After encoding, the memory used by **dataInfo** needs to be manually released by the caller. 158> - The memory is managed and released separately on the server and the client. 159> - If a pointer contains the shared memory, no extra processing is required. 160> - If other types of pointers are used, you need to dereference them before using **ProcessEncode** or **ProcessDecode**. 161> - The codec module has not been adapted to the **class** data type and therefore it is not recommended. 162 163 164#### Rule: Release the memory used by the encoded or decoded parameters in the SDK. Otherwise, a memory leakage occurs. 165 166Encoding is essentially a process of encapsulating different types of data in the same memory space and then encapsulating the start address and length of the memory into the body. A plug-in is unable to release the memory that has been allocated to output parameter data, which is returned to the SDK through encoding. To obtain the data, the SDK first needs to release the memory. 167 168The following is an example of releasing the memory: 169 170 171``` 172DataInfo outputInfo = { 173.data = nullptr, 174.length = 0, 175}; 176AieClientPrepare(clientInfo_, algorithmInfo_, inputInfo, outputInfo, nullptr); 177if (outputInfo.data != nullptr) { 178free(outputInfo.data); 179outputInfo.data = nullptr; 180outputInfo.length = 0; 181} 182``` 183 184 185#### Rule: Enable plug-ins to implement the **IPlugin** API defined by the server and use the **PLUGIN\_INTERFACE\_IMPL** statement to provide the function pointer for external systems. 186 187The server manages a variety of plug-ins, and the API implementation logic varies according to plug-ins. To unify the plug-in loading process, the AI engine provides the **IPlugin** API. In the runtime environment, a plug-in is loaded as a dynamic link library (DLL) by the AI engine framework in dlopen mode. Therefore, the plug-in needs to use the **PLUGIN\_INTERFACE\_IMPL** statement to expose the function pointer. Otherwise, the plug-in cannot be properly loaded. 188 189 190#### Rule: Use the unified data channel provided by the AI engine for plug-ins. 191 192The AI engine provides a unified data channel between the server and plug-ins to send inference requests from the SDK and returned results from plug-ins. Plug-ins need to obtain the request data and encapsulate the inference result over the data channel when calling the inference API. 193 194The following is an example of using the data channel: 195 196 197``` 198int SyncProcess(IRequest *request, IResponse *&response) 199{ 200HILOGI("[IvpPlugin]Begin SyncProcess"); 201if (request == nullptr) { 202HILOGE("[IvpPlugin]SyncProcess request is nullptr"); 203return RETCODE_NULL_PARAM; 204} 205DataInfo inputInfo = request->GetMsg(); 206if (inputInfo.data == nullptr) { 207HILOGE("[IvpPlugin]InputInfo data is nullptr"); 208return RETCODE_NULL_PARAM; 209} 210 211... 212 213response = IResponse::Create(request); 214response->SetResult(outputInfo); 215return RETCODE_SUCCESS; 216} 217``` 218 219In the example, the request and response are the data body sent over the data channel. The server encapsulates data in the request and sends it to the plug-in. After completing algorithm processing, the plug-in encapsulates the result into the response and returns it to the server over the data channel. 220 221## Development Guidelines 222 223### Developing the SDK 224 225The function of the SDK header file is implemented by mapping SDK API calls to client API calls. The following table lists the APIs provided by the client. 226 227 228 **Table 1** Client APIs 229 230| API| Description| Parameters| 231| -------- | -------- | -------- | 232| int **AieClientInit**(const ConfigInfo &configInfo,<br> ClientInfo &clientInfo, const AlgorithmInfo<br> &algorithmInfo, IServiceDeadCb \*cb) | **Function**: Links and initializes the engine service and activates IPC call.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **configInfo**: engine-related initial configuration data. This parameter must not be null.<br>**clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**cb**: death callback object. This parameter can be null. | 233| int **AieClientPrepare**(const ClientInfo &clientInfo<br>, const AlgorithmInfo &algorithmInfo, const DataInfo<br> &inputInfo, DataInfo &outputInfo, IClientCb \*cb) | **Function**: Loads an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in loading. This parameter can be null.<br>**outputInfo**: information returned after algorithm plug-in loading, if any. This parameter can be null.<br>**cb**: return result of the asynchronous algorithm. This parameter must not be null for the asynchronous algorithm. For the synchronous algorithm, this parameter must be null. | 234| int **AieClientAsyncProcess**(const ClientInfo &clientInfo,<br> const AlgorithmInfo &algorithmInfo, const DataInfo<br> &inputInfo) | **Function**: Executes an asynchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm operations. This parameter can be null. | 235| int **AieClientSyncProcess**(const ClientInfo &clientInfo,<br> const AlgorithmInfo &algorithmInfo, const<br> DataInfo &inputInfo, DataInfo &outputInfo) | **Function**: Executes a synchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the called algorithm. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm operations. This parameter can be null.<br>**outputInfo**: output information in the return result of the synchronous algorithm. This parameter can be null. | 236| int **AieClientRelease**(const ClientInfo &clientInfo,<br> const AlgorithmInfo &algorithmInfo, const<br> DataInfo &inputInfo) | **Function**: Uninstalls an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**algorithmInfo**: information about the algorithm plug-in to be uninstalled. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in uninstallation. This parameter can be null. | 237| int **AieClientDestroy**(ClientInfo &clientInfo) | **Function**: Disconnects from the server and releases the cache.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null. | 238| int **AieClientSetOption**(const ClientInfo &clientInfo,<br> int optionType, const DataInfo &inputInfo) | **Function**: Sets configuration items. You can use this API to pass algorithm's extended information to plug-ins.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**optionType**: algorithm used for obtaining the configuration item information. An algorithm plug-in can use this parameter as needed. This parameter must not be null.<br>**inputInfo**: Indicates algorithm parameter information. An algorithm plug-in can use this parameter as needed. This parameter can be null. | 239| int **AieClientGetOption**(const ClientInfo &clientInfo,<br> int optionType, const DataInfo &inputInfo,<br> DataInfo &outputInfo) | **Function**: Obtains configuration item information based on the specified **optionType** and **inputInfo**.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **clientInfo**: engine client information. This parameter must not be null.<br>**optionType**: algorithm used for obtaining the configuration item information. This parameter must not be null.<br>**inputInfo**: input information specified for obtaining configuration item information of the algorithm. This parameter can be null.<br>**outputInfo**: configuration item information in the return result. This parameter can be null. | 240 241 242The following table describes the data structure of **ConfigInfo**, **ClientInfo**, **AlgorithmInfo**, and **DataInfo**. 243 244 245 **Table 2** Data structure of ConfigInfo, ClientInfo, AlgorithmInfo, and DataInfo 246 247| Data Structure| Description| Attributes| 248| -------- | -------- | -------- | 249| ConfigInfo | Algorithm configuration item information| **const char \*description**: body of configuration item information. | 250| ClientInfo | Client information.| **long long clientVersion**: client version number. This parameter is not used currently.<br>**int clientId**: client ID.<br>**int sessionId**: session ID.<br>**uid\_t serverUid**: server UID.<br>**uid\_t clientUid**: client UID.<br>**int extendLen**: length of the extended information (**extendMsg**).<br>**unsigned char \*extendMsg**: body of the extended information. | 251| AlgorithmInfo | Algorithm information| **long long clientVersion**: client version number. This parameter is not used currently.<br>**bool isAsync**: whether asynchronous execution is used.<br>**int algorithmType**: algorithm type ID allocated by the AI engine framework based on the plug-in loading sequence.<br>**long long algorithmVersion**: algorithm version number.<br>**bool isCloud**: whether to migrate data to the cloud. This parameter is not used currently.<br>**int operateId**: execution ID. This parameter is not used currently.<br>**int requestId**: request ID, which identifies each request and corresponds to the execution result.<br>**int extendLen**: length of the extended information (**extendMsg**).<br>**unsigned char \*extendMsg**: body of the extended information. | 252| DataInfo | Algorithm input parameter configuration information (**inputInfo**) and output parameter configuration information (**outputInfo**)| **unsigned char \*data**: data subject.<br>**int length**: data length. | 253 254 255For details about the development process, see the development example of the [KWS SDK](#kws-sdk). 256 257### Developing a Plug-in 258The AI engine framework has defined a set of algorithm plug-in access specifications. Each plug-in needs to implement specified APIs to implement functions such as obtaining the plug-in version information and algorithm inference type, executing synchronous and asynchronous algorithms, loading algorithm plug-ins, uninstalling algorithm plug-ins, setting algorithm configuration information, and obtaining specified algorithm configuration information. Specifically, implement the **SyncProcess** API for the synchronous algorithm and the **AsyncProcess** API for the asynchronous algorithm. 259 260 261The following table describes the **IPlugin** APIs. 262 263 264 **Table 3** IPlugin APIs 265 266| API| Description| Parameters| 267| -------- | -------- | -------- | 268| const long long GetVersion() const; | **Function**: Obtains the plug-in version information.<br>**Return value**: version number (in the **long long** format)| - | 269| const char \*GetInferMode() const; | **Function**: Obtains the algorithm inference type.<br>**Return value**: "SYNC" or "ASYNC"| - | 270| int SyncProcess(IRequest \*request,<br>IResponse \*&response); | **Function**: Executes a synchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **request**: used to pass the request content to the algorithm plug-in over the data channel between the engine server and the plug-in. This parameter must not be null.<br>**response**: used to receive the synchronous algorithm execution result returned by the algorithm plug-in over the data channel between the engine server and the plug-in. This parameter must not be null.| 271| int AsyncProcess(IRequest \*request,<br>IPluginAlgorithmCallback \*callback); | **Function**: Executes an asynchronous algorithm.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **request**: used to pass the request content to the algorithm plug-in over the data channel between the engine server and the plug-in. This parameter must not be null.<br>**callback**: used to return the asynchronous algorithm execution result to the engine server. This parameter must not be null.| 272| int Prepare(long long transactionId,<br>const DataInfo &inputInfo, DataInfo<br>&outputInfo); | **Function**: Loads an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **transactionId**: transaction ID, which is used to identify the client and session. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in loading. This parameter can be null.<br>**outputInfo**: output information in the return result of algorithm plug-in loading. This parameter can be null.| 273| int Release(bool isFullUnload, long long<br>transactionId, const DataInfo &inputInfo); | **Function**: Uninstalls an algorithm plug-in.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **isFullUnload**: whether a plug-in is called by only one client. A plug-in can be uninstalled only when it is called by only one client. This parameter must not be null.<br>**transactionId**: transaction ID, which is used to identify the client and session. This parameter must not be null.<br>**inputInfo**: input information specified for algorithm plug-in uninstallation. This parameter can be null.| 274| int SetOption(int optionType, const<br>DataInfo &inputInfo); | **Function**: Sets configuration items. You can use this API to pass algorithm's extended information to plug-ins.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **optionType**: algorithm used for obtaining the configuration item information. An algorithm plug-in can use this parameter as needed. This parameter must not be null.<br>**inputInfo**: algorithm parameter information. An algorithm plug-in can use this parameter as needed. This parameter can be null.| 275| int GetOption(int optionType, const<br>DataInfo &inputInfo, DataInfo<br>&outputInfo); | **Function**: Obtains configuration item information based on the specified **optionType** and **inputInfo**.<br>**Return value**: **0** if the operation is successful; a non-zero value otherwise.| **optionType**: algorithm used for obtaining the configuration item information. This parameter must not be null.<br>**inputInfo**: input information specified for obtaining configuration item information of the algorithm. This parameter can be null.<br>**outputInfo**: configuration item information in the return result. This parameter can be null.| 276 277 278Algorithm plug-in APIs including **Prepare**, **SyncProcess**, **AsyncProcess**, **Release**, **SetOption**, and **GetOption** are in 1:1 mapping with the client APIs including **AieClientPrepare**, **AieClientSyncProcess**, **AieClientAsyncProcess**, **AieClientRelease**, **AieClientSetOption**, and **AieClientGetOption**. The **GetInferMode** API is used to return the algorithm execution type, which can be synchronous or asynchronous. 279 280 281The following table describes the **IPluginCallback** APIs. 282 283 284 **Table 4** IPluginCallback APIs 285 286| API| Description| Parameters| 287| -------- | -------- | -------- | 288| void OnEvent(PluginEvent event,<br>IResponse \*response); | **Function**: Returns the asynchronous algorithm execution result.| **event**: used to enumerate the algorithm execution result. The value can be **ON\_PLUGIN\_SUCCEED** or **ON\_PLUGIN\_FAIL**.<br>**response**: used to return the algorithm execution result.| 289 290 291The **Request** and **Response** classes define the requests and responses used for communication between the AI engine server and algorithm plug-ins. A request encapsulates the request content and input data of the caller. The plug-in returns the calculation result to the AI engine server through a response. 292 293 294The following table describes the attributes of the **Request** class. 295 296 297 **Table 5** Attributes of the Request class 298 299| Attribute| Description| Default Value| 300| -------- | -------- | -------- | 301| innerSequenceId_ | **Type**: long long<br>**Function**: reserved| 0 | 302| requestId_ | **Type**: int<br>**Function**: Indicates the request sequence, which is used to bind the return result.| 0 | 303| operationId_ | **Type**: int<br>**Function**: reserved| 0 | 304| transactionId_ | Type: long long<br>**Function**: Indicates the transaction ID, which is the combination of **clientId** and **sessionId**.| 0 | 305| algoPluginType_ | **Type**: int<br>**Function**: Indicates the algorithm type ID allocated by the AI engine framework based on the plug-in loading sequence.| 0 | 306| msg_ | **Type**: DataInfo<br>**Function**: Stores the input parameters for calling the algorithm API.| .data = nullptr<br>.length = 0 | 307 308 309The following table describes the attributes of the **Response** class. 310 311 312 **Table 6** Attributes of the Response class 313 314| Attribute| Description| Default Value| 315| -------- | -------- | -------- | 316| innerSequenceId_ | Type: long long<br>**Function**: reserved| 0 | 317| requestId_ | **Type**: int<br>**Function**: Indicates the request sequence, which is used to bind the return result.| 0 | 318| retCode__ | **Type**: int<br>**Function**: Indicates the inference result code of the asynchronous algorithm.| 0 | 319| retDesc_ | **Type**: string<br>**Function**: reserved| - | 320| transactionId_ | Type: long long<br>**Function**: Indicates the transaction ID, which is the combination of **clientId** and **sessionId**.| 0 | 321| algoPluginType_ | **Type**: int<br>**Function**: Indicates the algorithm type ID allocated by the AI engine framework based on the plug-in loading sequence.| INVALID_ALGO_PLUGIN_TYPE(-1) | 322| result_ | **Type**: DataInfo<br>**Function**: Stores the inference result of the asynchronous algorithm.| .data = nullptr<br>.length = 0 | 323 324 325For details about the development process, see the development example of the [KWS plug-in](#kws-plug-in). 326 327### Preparing the Configuration File 328 329The SDK identifies the plug-in type based on **algorithmVersion** and **algorithmType** in the **AlgorithmInfo** structure so it can call the plug-in capabilities. Therefore, you need to develop the plug-in configuration file as follows: 330 331 3321. Add the plug-in configuration file to the **//foundation/ai/engine/services/common/protocol/plugin\_config/plugin\_config\_ini/** directory. 333 3342. Add the algorithm type to the **aie\_algorithm_type.h** file in the **//foundation/ai/engine/services/common/protocol/plugin\_config/** directory. 335 3363. Add the name of the KWS algorithm and its sequence number in **ALGORITHM\_TYPE\_ID\_LIST** to the **aie\_plugin\_info.h** file in the **//foundation/ai/engine/services/server/plugin\_manager/include/** directory. 337 338For details about the development process, see the development example of the [configuration file](#kws-configuration-file). 339 340## Development Example 341 342### KWS SDK 343 344 3451. Add the API definition of the KWS SDK to the **//foundation/ai/engine /interfaces/kits** directory. This API can be called by third-party applications. The following code snippet is an example API for the KWS SDK. The reference code is available at the **//foundation/ai/engine /interfaces/kits/asr/keyword\_spotting** directory. 346 347 ``` 348 class KWSSdk { 349 public: 350 KWSSdk(); 351 virtual ~KWSSdk(); 352 353 // Create a KWS SDK instance. 354 int32_t Create(); 355 356 // Synchronously execute the KWS task. 357 int32_t SyncExecute(const Array<int16_t> &audioInput); 358 359 // Set the KWS callback. 360 int32_t SetCallback(const std::shared_ptr<KWSCallback> &callback); 361 362 // Destroy the KWS SDK instance to release the session engaged with the plug-in. 363 int32_t Destroy(); 364 }; 365 ``` 366 3672. Add the API implementation of the SDK to the **//foundation/ai/engine/services/client/algorithm\_sdk** directory and call the APIs provided by the client to use the algorithm plug-in capabilities. The following code snippet is an example implementation of the **create** method in the API of the KWS SDK. For more details, see the reference code at **//foundation/ai/engine/services/client/algorithm\_sdk/asr/keyword\_spotting**. 368 369 ``` 370 int32_t KWSSdk::KWSSdkImpl::Create() 371 { 372 if (kwsHandle_ != INVALID_KWS_HANDLE) { 373 HILOGE("[KWSSdkImpl]The SDK has been created"); 374 return KWS_RETCODE_FAILURE; 375 } 376 if (InitComponents() != RETCODE_SUCCESS) { 377 HILOGE("[KWSSdkImpl]Fail to init sdk components"); 378 return KWS_RETCODE_FAILURE; 379 } 380 // Call the AieClientInit API provided by the client to initialize the engine service and activate IPC call. 381 int32_t retCode = AieClientInit(configInfo_, clientInfo_, algorithmInfo_, nullptr); 382 if (retCode != RETCODE_SUCCESS) { 383 HILOGE("[KWSSdkImpl]AieClientInit failed. Error code[%d]", retCode); 384 return KWS_RETCODE_FAILURE; 385 } 386 if (clientInfo_.clientId == INVALID_CLIENT_ID) { 387 HILOGE("[KWSSdkImpl]Fail to allocate client id"); 388 return KWS_RETCODE_FAILURE; 389 } 390 DataInfo inputInfo = { 391 .data = nullptr, 392 .length = 0, 393 }; 394 DataInfo outputInfo = { 395 .data = nullptr, 396 .length = 0, 397 }; 398 // Call the AieClientPrepare API provided by the client to load the algorithm plug-in. 399 retCode = AieClientPrepare(clientInfo_, algorithmInfo_, inputInfo, outputInfo, nullptr); 400 if (retCode != RETCODE_SUCCESS) { 401 HILOGE("[KWSSdkImpl]AieclientPrepare failed. Error code[%d]", retCode); 402 return KWS_RETCODE_FAILURE; 403 } 404 if (outputInfo.data == nullptr || outputInfo.length <= 0) { 405 HILOGE("[KWSSdkImpl]The data or length of output info is invalid"); 406 return KWS_RETCODE_FAILURE; 407 } 408 MallocPointerGuard<unsigned char> pointerGuard(outputInfo.data); 409 retCode = PluginHelper::UnSerializeHandle(outputInfo, kwsHandle_); 410 if (retCode != RETCODE_SUCCESS) { 411 HILOGE("[KWSSdkImpl]Get handle from inputInfo failed"); 412 return KWS_RETCODE_FAILURE; 413 } 414 return KWS_RETCODE_SUCCESS; 415 } 416 ``` 417 418 The preceding code is the specific API implementation. The **create** function in the API of the KWS SDK calls the open **AieClientInit** and **AieClientPrepare** APIs provided by the client to connect to the server and load the algorithm model. 419 420 > **NOTE** 421 > 422 > The sequence for the SDK to call client APIs of the AI engine is as follows: AieClientInit -> AieClientPrepare -> AieClientSyncProcess/AieClientAsyncProcess -> AieClientRelease -> AieClientDestroy. An exception will be thrown if the call sequence is violated. 423 424### KWS Plug-in 425Add the API definition (that is, **IPlugin**) of the KWS plug-in to the **//foundation/ai/engine/services/server/plugin** directory. This API is used to call AI capabilities. The following code snippet is an example API implementation of the KWS plug-in. The reference code is available at the **//foundation/ai/engine/services/server/plugin/asr/keyword\_spotting** directory. 426 427 ``` 428 #include "plugin/i_plugin.h 429 class KWSPlugin : public IPlugin { 430 public: 431 KWSPlugin(); 432 ~KWSPlugin(); 433 const long long GetVersion() const override; 434 const char* GetName() const override; 435 const char* GetInferMode() const override; 436 int32_t Prepare(long long transactionId, const DataInfo &amp;inputInfo, DataInfo &amp;outputInfo) override; 437 int32_t SetOption(int optionType, const DataInfo &amp;inputInfo) override; 438 int32_t GetOption(int optionType, const DataInfo &amp;inputInfo, DataInfo &amp;outputInfo) override; 439 int32_t SyncProcess(IRequest *request, IResponse *&amp;response) override; 440 int32_t AsyncProcess(IRequest *request, IPluginCallback*callback) override; 441 int32_t Release(bool isFullUnload, long long transactionId, const DataInfo &amp;inputInfo) override; 442 }; 443 ``` 444The preceding code implements the **IPlugin** APIs provided by the server. The following table shows the mapping between the client APIs and the plug-in APIs. 445 446**Table 7** Mapping between the client APIs and the plug-in APIs 447 448| Client API| Plug-in API| Description| 449| -------- | -------- | -------- | 450| AieClientPrepare | Prepare | Initializes the inference algorithm plug-in. For KWS, this API loads the KWS model from the fixed location (**/sdcard/wenwen\_inst.wk**) to the memory.| 451| AieClientSyncProcess | SyncProcess | Executes the inference algorithm synchronously. For KWS, this API synchronously executes the audio inference algorithm to determine whether the specified wakeup keyword exists in the audio.| 452| AieClientAsyncProcess | AsyncProcess | Executes the inference algorithm asynchronously. Currently, this API is not used in KWS. However, you can implement the API based on your use case.| 453| AieClientSetOption | SetOption | Sets algorithm-related configuration items, such as the confidence threshold and delay. Currently, this API is not used in KWS. However, you can implement the API based on your use case.| 454| AieClientGetOption | GetOption | Obtains algorithm-related configuration items. For KWS, this API can obtain the input and output scale of the KWS model. The input scale is the MFCC feature (fixed value: **4000**) required by the KWS model, and the output scale is the confidence (fixed value: **2**) of the result.| 455| AieClientRelease | Release | Releases the algorithm model. For KWS, this API releases the specified algorithm model and clears the dynamic memory in the feature processor.| 456 457> **NOTE** 458> 459> - The **AieClientInit** and **AieClientDestroy** APIs are used to connect to and disconnect from the server, respectively. They are not called in the plug-in algorithm and therefore do not need to be defined in the plug-in. 460> - The KWS plug-in needs to use the **PLUGIN\_INTERFACE\_IMPL** statement to expose the function pointer. Otherwise, the plug-in cannot be properly loaded. 461 462 463 ``` 464 PLUGIN_INTERFACE_IMPL(KWSPlugin); 465 ``` 466 467### KWS Configuration File 468 4691. Add the KWS configuration file to the **//foundation/ai/engine/services/common/protocol/plugin\_config/plugin\_config\_ini/** directory. 470 471 ``` 472 [base] 473 supported_boards = hi3516dv300 474 related_sessions = asr_keyword_spotting+20001002 475 476 // Naming rule: [algorithm name+algorithm version], for example, [asr\_keyword\_spotting+20001002] 477 [asr_keyword_spotting+20001002] 478 AID = asr_keyword_spotting 479 VersionCode = 20001002 480 VersionName = 2.00.01.002 481 XPU = NNIE 482 District = China 483 // Location of the complied **.so** file of the plug-in 484 FullPath = /usr/lib/libasr_keyword_spotting.so 485 Chipset = ALL 486 ChkSum = '' 487 Key = '' 488 ``` 489 4902. Add the type ID of the KWS algorithm to the **aie\_algorithm_type.h** file in the **//foundation/ai/engine/services/common/protocol/plugin\_config/** directory. 491 492 ``` 493 // Make sure that the type ID maps to the sequence number in **ALGORITHM\_TYPE\_ID\_LIST**. 494 const int ALGORITHM_TYPE_KWS = 3; 495 ``` 496 4973. Add the name of the KWS algorithm and its sequence number in **ALGORITHM\_TYPE\_ID\_LIST** to the **aie\_plugin\_info.h** file in the **//foundation/ai/engine/services/server/plugin\_manager/include/** directory. 498 499 ``` 500 const std::string ALGORITHM_ID_SAMPLE_1 = "sample_plugin_1"; 501 const std::string ALGORITHM_ID_SAMPLE_2 = "sample_plugin_2"; 502 const std::string ALGORITHM_ID_IVP = "cv_human_detect"; 503 // Add the name of the KWS algorithm: asr\_keyword\_spotting. 504 // Name the algorithm variable in the same way as the algorithm type ID in ALGORITHM\_TYPE\_ID\_LIST, for example, ALGORITHM_ID_KWS. 505 const std::string ALGORITHM_ID_KWS = "asr_keyword_spotting"; 506 const std::string ALGORITHM_ID_IC = "cv_image_classification"; 507 const std::string ALGORITHM_ID_INVALID = "invalid algorithm id"; 508 509 const std::vector<std::string> ALGORITHM_TYPE_ID_LIST = { 510 ALGORITHM_ID_SAMPLE_1, 511 ALGORITHM_ID_SAMPLE_2, 512 ALGORITHM_ID_IVP, 513 // Add the sequence number of the KWS algorithm to ALGORITHM\_TYPE\_ID\_LIST, so that the name of the KWS algorithm can be obtained based on the sequence number. 514 // Make sure that the algorithm name maps to the sequence number in ALGORITHM\_TYPE\_ID\_LIST. 515 ALGORITHM_ID_KWS, 516 ALGORITHM_ID_IC, 517 }; 518 ``` 519
