1# xDevice User Guide 2 3 4## Overview 5 6As an open-source OS, OpenHarmony supports product development in many chip scenarios. To ensure compatibility of the OpenHarmony ecosystem, OpenHarmony provides the [compatibility test service](https://www.openharmony.cn/certification/document/guid). For related products, API tests are required for verification. However, executing a large number of automated test cases requires a scheduling and execution framework that supports capabilities such as generating visual test reports. Therefore, we designed and developed the xDevice test scheduling and execution framework. 7 8### Introduction 9 10The xDevice test scheduling and execution framework is a core component of the test infrastructure of OpenHarmony. It provides related services required for scheduling and executing automated test cases, supports scheduling and execution of a large number of automated test cases, as well as supports the generation of visual test reports. The binary package of xDevice will be compiled together with the XTS suite of OpenHarmony. You can obtain the xDevice tool from the XTS archiving path. 11 12Based on the device type, xDevice mainly tests tasks in the following scenarios: 13 14- Perform XTS tests for mini-system devices (such as the Hi3861 development board). 15- Perform XTS tests for small-system devices (such as the Hi3516 development board). 16- Perform XTS tests for standard-system devices (such as the RK3568 development board). 17 18### Implementation Principles 19 20The xDevice tool includes the following functional modules: 21 22- **command**: enables command-based interactions between users and the test platform. It parses and processes user commands. 23- **config**: sets test framework configurations and provides different configuration options for the serial port connection and USB connection modes. 24- **driver**: functions as a test case executor, which defines main test steps, such as test case distribution, execution, and result collection. 25- **report**: parses test results and generates test reports. 26- **scheduler**: schedules various test case executors in the test framework. 27- **environment**: configures the test framework environment, enabling device discovery and device management. 28- **testkit**: provides test tools to implement JSON parsing, network file mounting, etc. 29- **log**: records task logs and device logs. 30 31In addition to the preceding functional modules, the framework depends on user-defined configuration files, which are classified into two types: 32 33**Test Task Configuration File** 34 35The test task configuration file provided by the framework is **user_config.xml**. You can modify the configuration file based on your environment information. 36 37The environment configuration is as follows: 38 39```xml 40<environment> 41 42 <!-- Large-system device configuration --> 43<device type="usb-hdc"> <!-- type: device connection mode. usb-hdc (default) indicates to control devices using the HDC. Currently, the framework supports only one USB device. --> 44 <ip></ip> <!-- ip: remote device IP address. The local device is used if ip and port are empty, and the remote device is used if ip and port are not empty. --> 45 <port></port> <!-- port: remote device port number. --> 46 <sn></sn> <!-- sn: device SN list. SNs are separated using semicolons (;). All local devices are used when the SN is empty. The device with the specified SN will be used if the SN is not empty. --> 47</device> 48 49<!-- L0 device configuration --> 50<device type="com" label="wifiiot"> <!-- type: device connection mode. com indicates the serial port connection mode. label indicates the device type, for example, wifiot. --> 51 <serial> <!-- serial: serial port definition. --> 52 <com></com> <!-- com: serial port of the local connection, for example, COM20. --> 53 <type>cmd</type> <!-- type indicates the serial port type. cmd is the command serial port. --> 54 <baud_rate>115200</baud_rate> <!-- baud_rate, data_bits, stop_bits, timeout: serial port parameters. Generally, the default values are used. --> 55 <data_bits>8</data_bits> 56 <stop_bits>1</stop_bits> 57 <timeout>20</timeout> 58 </serial> 59 <serial> 60 <com></com> 61 <type>deploy</type> <!-- type indicates the serial port type. deploy indicates the deployment serial port. --> 62 <baud_rate>115200</baud_rate> 63 </serial> 64</device> 65 66<!-- L1 device local connection configuration --> 67<device type="com" label="ipcamera"> 68 <serial> 69 <com></com> 70 <type>cmd</type> 71 <baud_rate>115200</baud_rate> 72 <data_bits>8</data_bits> 73 <stop_bits>1</stop_bits> 74 <timeout>1</timeout> 75 </serial> 76</device> 77 78<!-- L1 device remote connection configuration. Multiple connections can be configured. --> 79<device type="com" label="ipcamera"> 80 <ip></ip> 81 <port></port> 82</device> 83</environment> 84``` 85 86Set the test case directory. 87 88```xml 89<testcases> 90 <!-- If both dir and server are configured, only either of them takes effect. --> 91 <!-- Specify the test case directory. If this parameter is empty, the testcase directory in the current project will be used. --> 92 <dir></dir> 93 94 <!-- NFS mounting configuration. Set the value to NfsServer. --> 95 <server label="NfsServer"> 96 <ip></ip> <!-- Mounting environment IP address. --> 97 <port></port> <!-- Mounting environment port. --> 98 <dir></dir> <!-- Mounted external path. --> 99 <username></username> <!-- Login user name. --> 100 <password></password> <!-- Login user password. --> 101 <remote></remote> <!-- whether the NFS server and the XDevice executor are deployed on different devices. If yes, set this parameter to true. Otherwise, set it to false. --> 102 </server> 103</testcases> 104``` 105 106Set the resource directory. 107 108```xml 109<resource> 110 <!-- Specify the resource directory. If this parameter is empty, the resource directory in the current project will be used. --> 111 <dir></dir> 112</resource> 113``` 114 115Set the log level. 116 117```xml 118<!-- The default level is INFO. For more detailed information, change the level to DEBUG. --> 119<loglevel>INFO</loglevel> 120``` 121 122**Test Suite Configuration File** 123 124The test support suite executed by the device is specified by the test configuration file. 125 126Each test suite has a test configuration file, which mainly specifies the test support suites (kits) that need to be used. In addition, the setup and teardown operations are supported. 127 128The following is a configuration file example: 129 130```json 131{ 132 // Description of the test support suite. 133 "description": "Configuration for aceceshi Tests", 134 135 // Specify the device for executing the current test support suite. 136 "environment": { 137 "type": "device", 138 "label": "wifiiot" 139 }, 140 141 // Specify the driver executed by the device. 142 "driver": { 143 "type": "OHJSUnitTest", 144 "test-timeout": "700000", 145 "bundle-name": "com.open.harmony.acetestfive", 146 "package-name": "com.open.harmony.acetestfive", 147 "shell-timeout": "700000" 148 }, 149 // The kit is mainly used to support test execution activities, including the setup operation before the test and the teardown operation after the test. 150 "kits": [ 151 { 152 "type": "ShellKit", 153 "run-command": [ 154 "remount", 155 "mkdir /data/data/resource", 156 "chmod -R 777 /data/data/resource", 157 "settings put secure adb_install_need_confirm 0" 158 ], 159 "teardown-command": [ 160 "remount", 161 "rm -rf /data/data/resource" 162 ] 163 }, 164 ] 165} 166``` 167 168### Test Commands 169 170Test commands can be classified into three groups: **help**, **list**, and **run**. Among them, **run** commands are most commonly used in the instruction sequence. 171 172------ 173 174You can run **help** commands to obtain help information about the test framework commands. 175 176```text 177help: 178 use help to get information. 179usage: 180 run: Display a list of supported run command. 181 list: Display a list of supported device and task record. 182Examples: 183 help run 184 help list 185``` 186 187**NOTE** 188 189**help run**: displays the description of **run** commands. 190 191**help list**: displays the description of **list** commands. 192 193------ 194 195You can run **list** commands to display device information and related task information. 196 197```text 198list: 199 This command is used to display device list and task record. 200usage: 201 list 202 list history 203 list <id> 204Introduction: 205 list: display device list 206 list history: display history record of a serial of tasks 207 list <id>: display history record about task what contains specific id 208Examples: 209 list 210 list history 211 list 6e****90 212``` 213 214 **NOTE** 215 216 **list**: displays device information. 217 218 **list history**: displays historical task information. 219 220 **list *\<id>***: displays historical information about tasks with specified IDs. 221 222------ 223 224Run the **run** commands to execute test tasks. 225 226```text 227run: 228 This command is used to execute the selected testcases. 229 It includes a series of processes such as use case compilation, execution, and result collection. 230usage: run [-l TESTLIST [TESTLIST ...] | -tf TESTFILE 231 [TESTFILE ...]] [-tc TESTCASE] [-c CONFIG] [-sn DEVICE_SN] 232 [-rp REPORT_PATH [REPORT_PATH ...]] 233 [-respath RESOURCE_PATH [RESOURCE_PATH ...]] 234 [-tcpath TESTCASES_PATH [TESTCASES_PATH ...]] 235 [-ta TESTARGS [TESTARGS ...]] [-pt] 236 [-env TEST_ENVIRONMENT [TEST_ENVIRONMENT ...]] 237 [-e EXECTYPE] [-t [TESTTYPE [TESTTYPE ...]]] 238 [-td TESTDRIVER] [-tl TESTLEVEL] [-bv BUILD_VARIANT] 239 [-cov COVERAGE] [--retry RETRY] [--session SESSION] 240 [--dryrun] [--reboot-per-module] [--check-device] 241 [--repeat REPEAT] 242 action task 243Specify tests to run. 244 positional arguments: 245 action Specify action 246 task Specify task name,such as "ssts", "acts", "hits" 247``` 248 249The table below describes how to use **run** commands. 250 251| xDevice Command | Function | Example | 252| :----------: | :----------------------------------------------------------: | :----------------------------------------------------------: | 253| run xts | Runs all XTS modules of the specified type, for example, **acts**, **hits**, and **ssys**. | run acts | 254| run -l xxx | Runs specified module test suites. Multiple module test suites are separated using semicolons (;). | run -l ActsWifiServiceTest;ActsLwipTest | 255| run -sn | Specifies the SNs of the executed devices. Multiple SNs are separated using semicolons (;). | run acts -sn 10.117.183.37:17001<br>run acts -sn 88Y02******57723;VEG02******16642 | 256| run -rp | Specifies the report generation path. By default, subdirectories will be created under **reports** of the work directory using the timestamp or task ID.| run acts -rp /suites/hits/xdevice_reports/2020.09.28-14.21.26 | 257| run -respath | Specifies the test resource path. The default value is **resource**. | run acts -respath /cloud/zidane/xts/release/suites/resource | 258| run -ta | Specifies module running parameters. You can specify test cases in the running module. Multiple cases are separated using commas (,). Currently, the JS driver test suite is supported.| run acts -ta class:ohos.hardware.soundtrigger.SoundTriggerTest#testKeyphraseParcelUnparcel_noUsers | 259| run --retry | Executes failed test cases of the previous task to generate a new test report. | run –retryrun --retry --session 2020-10-30-17-15-11 (task directory name)| 260 261### Test Report 262 263After the test framework executes the **run** commands, the console outputs the corresponding logs, and the execution report is generated. The report is generated in the path specified by the **-rp** parameter if set. If the parameter is not set, the report will be generated in the default directory. 264 265```text 266Structure of the report directory (the default or the specified one) 267 ├── result (test case execution results of the module) 268 │ ├── *Module name*.xml 269 │ ├── ... ... 270 │ 271 ├── log (running logs of devices and tasks) 272 │ ├── *Device 1*.log 273 │ ├── ... ... 274 │ ├── *<Task>*.log 275 ├── summary_report.xml (task summary report) 276 ├── summary_report.html (task summary visual report) 277 ├── details_report.html (case execution visual report) 278 ├── failures_report.html (failed case visual report, which will not be generated if no case fails) 279 ├── summary.ini (Records information such as the used device, start time, and end time.) 280 ├── task_info.record (Records executed commands and list of failed cases.) 281 ├── xxxx.zip (compression file generated by compressing the preceding files) 282 ├── summary_report.hash (file obtained by encrypting the compression file using SHA-256) 283 └── ... ... 284``` 285 286 287## Environment Preparations 288 289 290### Environment Requirements 291 292- Python version: 3.7 or later 293- pyserial: 3.3 or later 294- paramiko: 2.7.1 or later 295- rsa: 4.0 or later 296 297 298### Installing the xDevice 299 300- Install the basic framework of xDevice. 301 302 1. Go to the root directory of xDevice. 303 304 ```bash 305 cd testfwk_xdevice 306 ``` 307 308 2. Open the console and run the following command: 309 310 ```bash 311 python setup.py install 312 ``` 313 314- Install the OpenHarmony driver plugin **ohos**. 315 316 1. Go to the **plugin/ohos** directory. 317 318 ```bash 319 cd testfwk_xdevice/plugin/ohos 320 ``` 321 322 2. Open the console and run the following command as the current user: 323 324 ```bash 325 python setup.py install 326 ``` 327 328### Verifying the Environment 329 330Check whether xDevice is installed successfully. 331 3321. Go to the root directory of xDevice. 333 334 ```bash 335 cd testfwk_xdevice 336 ``` 337 3382. Open the console and run the following command: 339 340 ```bash 341 python -m pip list 342 ``` 343 3443. Check whether the **xdevice** and **xdevice-ohos** libraries are successfully installed. 345 346 ```text 347 xdevice 0.0.0 348 xdevice-ohos 0.0.0 349 ``` 350 351Check whether xDevice runs properly. 352 3531. Go to the root directory of xDevice. 354 355 ```bash 356 cd testfwk_xdevice 357 ``` 358 3592. Open the console and run the following command: 360 361 ```bash 362 python -m xdevice 363 ``` 364 3653. Check whether the following information is displayed on the console: 366 367 ```text 368 [2022-10-13 15:43:31,284] [30076] [Main] [INFO] [*************** xDevice Test Framework 2.11.0.1091 Starting ***************] 369 [2022-10-13 15:43:31,286] [30076] [ManagerLite] [WARNING] [wifiiot local com cannot be empty, please check] 370 [2022-10-13 15:43:31,286] [30076] [ManagerLite] [WARNING] [ipcamera local com cannot be empty, please check] 371 [2022-10-13 15:43:31,287] [30076] [ManagerLite] [WARNING] [device com or ip cannot be empty, please check] 372 >>> 373 ``` 374 375 376## Mini-System Device XTS Test Guide (wifiiot) 377 3781. Identify the serial port usage and modify the **user_config.xml** file in the root directory. 379 380 The COM port whose **type** is **cmd** corresponds to the AT command serial port on the board. The port is used to send commands to the device. In the example, the **ChA(COM20)** serial port is used. 381 382 The COM port whose **type** is **deploy** corresponds to the log output serial port on the board. The port is used to burn the image and print logs. In the example, the **ChB(COM18)** serial port is used. 383 384 If the AT command serial port is the same as the log output serial port, the serial ports can be set to the same port. That is, in the **user_config** file, the COM port whose **type** is **cmd** and the COM port whose **type** is **deploy** can be set to the same port, for example, **COM18**. 385 386  387 388 The following is an example of the modified **user_config.xml** file: 389 390 ```xml 391 <user_config> 392 <environment> 393 <device type="com" label="wifiiot"> 394 <serial> 395 <com>com20</com> 396 <type>cmd</type> 397 <baud_rate>115200</baud_rate> 398 <data_bits>8</data_bits> 399 <stop_bits>1</stop_bits> 400 <timeout>20</timeout> 401 </serial> 402 <serial> 403 <com>com18</com> 404 <type>deploy</type> 405 <baud_rate>115200</baud_rate> 406 </serial> 407 </device> 408 </environment> 409 <testcases> 410 <dir></dir> 411 <server label="NfsServer"> 412 <ip></ip> 413 <port></port> 414 <dir></dir> 415 <username></username> 416 <password></password> 417 <remote></remote> 418 </server> 419 </testcases> 420 <resource> 421 <dir></dir> 422 </resource> 423 <loglevel>DEBUG</loglevel> 424 </user_config> 425 ``` 426 4272. Create a **testcase** directory in the root directory of xDevice to store test suite files. XTS test suites are obtained from the daily builds of the system. 428 429 Daily builds: http://ci.openharmony.cn/dailys/dailybuilds 430 431 The following is an example of the test suite configuration file in JSON format: 432 433 ```json 434 { 435 "description": "Config for ActsAllTest test cases", 436 "environment": [ 437 { 438 "type": "device", 439 "label": "wifiiot" 440 } 441 ], 442 "kits": [ 443 { 444 "type": "DeployKit", 445 "timeout": "20000", 446 "burn_file": "acts/Hi3861_wifiiot_app_allinone.bin" 447 } 448 ], 449 "driver": { 450 "type": "CTestLite" 451 } 452 } 453 ``` 454 4553. Execute test cases. 456 457 Go to the root directory of xDevice, open the xDevice console, and run the following command: 458 459 ```bash 460 python -m xdevice 461 ``` 462 463 Run the test suite command. 464 465 ```text 466 run -l ActsAllTest 467 ``` 468 469 The command output is as follows. 470 471  472 473## Small-System Device XTS Test Guide (ipcamera) 474 4751. Identify the serial port usage. 476 477 The COM port whose **type** is **cmd** corresponds to the AT command serial port on the board. The port is used to send commands to the device. In the example, the **ChA(COM20)** serial port is used. 478 479  480 481 IP camera devices have two connection modes. One is to connect through the local serial port, and the other is to connect through the IP address of the local area network. 482 4832. Configure the NFS server. 484 485 There are two NFS mounting modes. One is to mount through the remote PC, and the other is to mount through the local area network. 486 487 To configure the NFS service on the local area network, perform the following steps: 488 489 1. Download and install the NFS server. Download address: https://www.hanewin.net/nfs-e.htm 490 491 2. Configure output and edit the output table file. 492 493 Add an NFS sharing path, for example, **D:\HS\NFS_Share_File -public –alldirs**. Note that the FTP IP address 192.168.1.10 is the IP address of the development board. 494 495  496 497 3. Stop the NFS server and restart the NFS server to make the added sharing path take effect. 498 499 4. Find the mapped network port of the IP camera device on the PC, and manually set the IP address to 192.168.1.11 on the PC. 500 5013. Modify the **user_config.xml** file in the root directory. The following is an example: 502 503 ```xml 504 <user_config> 505 <environment> 506 <device type="com" label="ipcamera"> <!--local connection mode--> 507 <serial> 508 <com>com20</com> 509 <type>cmd</type> 510 <baud_rate>115200</baud_rate> 511 <data_bits>8</data_bits> 512 <stop_bits>1</stop_bits> 513 <timeout>1</timeout> 514 </serial> 515 </device> 516 <device type="com" label="ipcamera"> <!--local area network connection mode--> 517 <ip>10.176.49.47</ip> 518 <port>10003</port> 519 </device> 520 </environment> 521 <testcases> 522 <dir></dir> 523 <server label="NfsServer"> <!--remote mounting mode--> 524 <ip>10.176.48.202</ip> 525 <port>1022</port> 526 <dir>/data/data/local/</dir> 527 <username>root</username> 528 <password>xxx</password> 529 <remote>true</remote> 530 </server> 531 <server label="NfsServer"> <!--local area network mounting mode--> 532 <ip>192.168.1.11</ip> 533 <port>2049</port> 534 <dir>D:\test</dir> 535 <remote>false</remote> 536 </server> 537 </testcases> 538 <resource> 539 <dir></dir> 540 </resource> 541 <loglevel>DEBUG</loglevel> 542 </user_config> 543 ``` 544 5454. Create a **testcase** directory in the root directory of xDevice to store test suite files. XTS test suites are obtained from the daily builds of the system. 546 547 Daily builds: http://ci.openharmony.cn/dailys/dailybuilds 548 549 The following is an example of the test suite configuration file in JSON format: 550 551 ```json 552 { 553 "description": "Config for kernel test cases", 554 "environment": [ 555 { 556 "type": "device", 557 "label": "ipcamera" 558 } 559 ], 560 "kits": [ 561 { 562 "type": "MountKit", 563 "server": "NfsServer", 564 "mount": [ 565 { 566 "source": "testcases/kernel", 567 "target": "/test_root/kernel" 568 } 569 ] 570 } 571 ], 572 "driver": { 573 "type": "CppTestLite", 574 "excute": "/test_root/kernel/ActsKernelIPCTest.bin" 575 } 576 } 577 ``` 578 5795. Execute test cases. 580 581 Go to the root directory of xDevice, open the xDevice console, and run the following command: 582 583 ```bash 584 python -m xdevice 585 ``` 586 587 Run the test suite command. 588 589 ```text 590 run -l kernel 591 ``` 592 593 The command output is as follows. 594 595  596 597## Standard-System Device XTS Test Guide (RK3568) 598 5991. Configure the HDC tool, and download the latest **ohos_sdk** from daily builds. 600 601 Daily builds: http://ci.openharmony.cn/dailys/dailybuilds 602 603 After downloading the tool, configure HDC into the environment variables on the PC. To do so, right-click the **Computer** or **My Computer** desktop icon and select **Properties**. Choose **Advanced system settings**. Select the **Advanced** tab, and click **Environment Variables**. In the **Environment Variables** dialog box, select the path of the environment variable. 604 6052. Run the following command to check whether the device is properly connected: 606 607 ```bash 608 hdc_std list targets 609 ``` 610 6113. Modify the **user_config.xml** file. The following is an example: 612 613 ```xml 614 <user_config> 615 <environment> 616 <device type="usb-hdc"> 617 <ip></ip> 618 <port></port> 619 <sn>xxx;xxx</sn> <!--SNs of multiple connected devices are separated using semicolons.--> 620 </device> 621 </environment> 622 <testcases> 623 <dir></dir> 624 </testcases> 625 <resource> 626 <dir></dir> 627 </resource> 628 <loglevel>DEBUG</loglevel> 629 </user_config> 630 ``` 631 6324. Create a **testcase** directory in the root directory of xDevice to store test suite files. XTS test suites are obtained from the daily builds of the system. 633 634 Daily builds: http://ci.openharmony.cn/dailys/dailybuilds 635 636 The following is an example of the test suite configuration file in JSON format: 637 638 ```json 639 { 640 "description": "Configuration for hjunit demo Tests", 641 "driver": { 642 "type": "OHJSUnitTest", 643 "test-timeout": "180000", 644 "bundle-name": "ohos.acts.bundle.stage.test", 645 "module-name": "phone", 646 "shell-timeout": "600000", 647 "testcase-timeout": 70000 648 }, 649 "kits": [ 650 { 651 "test-file-name": [ 652 "ActBmsStageEtsTest.hap" 653 ], 654 "type": "AppInstallKit", 655 "cleanup-apps": true 656 }, 657 { 658 "type": "ShellKit", 659 "teardown-command":[ 660 "bm uninstall -n ohos.acts.bundle.stage.test" 661 ] 662 } 663 ] 664 } 665 ``` 666 6675. Execute test cases. 668 669 Go to the root directory of xDevice, open the xDevice console, and run the following command: 670 671 ```bash 672 python -m xdevice 673 ``` 674 675 Run the test suite command. 676 677 ```text 678 run -l ActBmsStageEtsTest 679 ``` 680 681 The command output is as follows. 682 683  684 685## FAQs 686 687### The **hdc list targets** command can find a device, but xDevice cannot identify the device. 688 689**Issue Description** 690 691The following error information is displayed. 692 693 694 695**Possible Causes** 696 697The **HDC_SERVER_PORT** variable has been set and the HDC port has been modified. By default, xDevice uses port **8710**. If the port has been modified, the xDevice framework cannot identify the device. 698 699**Solution** 700 701Check whether the **HDC_SERVER_PROT** variable is set. If yes, change the port number to **8710** and restart xDevice. 702
