1# OpenHarmony Build Specifications 2 3## Overview 4 5This topic aims to guide OpenHarmony developers to enhance the build reproducibility, maintainability, and quality. The build standards team has analyzed a variety of typical build issues and summarized the build guidelines and recommendations in this topic. 6 7## General Build Principles 8 9**P01 Automate the entire build process.** 10 11Manual operations are error-prone and time-consuming. Automate all the build operations to make the build process more efficient and reliable. 12 13**P02 Make build projects and build environments code-based.** 14 15Use a high-level build framework such as CMake, Maven, and Gradle to describe build projects, and use Ansible or Dockerfile to describe build environments. 16 17This principle helps hide the complexity of the build system from developers. 18 19**P03 Make the build process reproducible and traceable.** 20 21Manage build dependencies and always explicitly specify fixed dependency versions to ensure version consistency. Incorporate build environments and build projects into the configuration file as configuration options to ensure that the build projects are traceable. 22 23**P04 Ensure that the build scripts are simple, clear, and easy to maintain.** 24 25Build scripts are also code and should be easy to read. 26 27**P05 Standardize the build process.** 28 29The build directory structure, dependencies, initialization, entry, and naming must be standardized to ensure consistency among all OpenHarmony products, platforms, and components for easier management and maintenance. 30 31## Build Projects 32 33### General Guidelines 34 35#### One-Click Build 36 37##### G.COM.01 Use build scripts to implement one-click, automated build by delivery unit. 38 39One-click, automated build means that no manual interventions are allowed throughout the entire build process (until the final delivery package is generated) in a build environment. 40 41Manual interventions include but are not limited to: manual parameter setting on the IDE; file directory creation or deletion; file creation, copying, moving, deletion, or renaming; manual setting of file attributes; file compression/decompression. 42 43Delivery unit refers to a product, platform, or component that can be independently compiled, loaded, deployed, and run. 44 45[Type] Requirement 46 47[Description] One-click build improves operation efficiency while greatly reducing the possibility of operation errors. 48 49[Negative Example] The one-click build of a component can be triggered only by the CI system but cannot be triggered locally. 50 51[Negative Example] Manually set the memory mapping address on the Xplorer IDE UI for a component, and then perform manual build. 52 53[Negative Example] Manually create the **r6c03_view\r6c03_client_view** directory for a component. 54 55[Positive Example] Use the python script to automatically create a directory. 56 57```python 58dir_src = os.getcwd() 59dir_client_view = r"r6c03_client_view" 60# os.path is used to shield system differences. 61dir_mk = os.path.join(dir_src, dir_client_view) 62 63cmd = "{0} {1}".format("mkdir", dir_mk) 64cmd_re = subprocess.run(cmd) 65``` 66 67#### Build Directory 68 69##### G.COM.02 DO NOT delete or modify source code files and their directory structures during build. 70 71[Type] Forbidden 72 73[Description] 74 75- If the source code directory structure is deleted or modified during the build, the build process will be non-reproducible. 76 77- During the build, the build output (including the target files, temporary files, and build logs) should not pollute the source code directory. 78 79- During the build, do not modify source files, including but not limited to copying, moving, and running **dos2unix** to convert the source code format. Any modification to the source files must be completed in the code preparation phase before the build. 80 81- The tool-triggered, automatic generation of source code must also be completed in the preparation phase before the build. If tools are used to automatically generate source code during the build, the source code generated (low-value and reproducible) must be isolated from the existing source code directory for better distinguishing, so as to reduce the complexity of the build system. 82 83[Exception] Some source code may be added or adjusted during patch build. 84 85##### G.COM.12 Provide appropriate permissions for files and directories created during the build. 86 87[Type] Requirement 88 89[Description] Directories or files of the target system that are created during the build must comply with the design of least privilege. For example, do not create directories or files with the **777** permission in the Linux system during the build. 90 91For details about common directory files and permissions in the Linux system, see the *Linux Security Configuration Standard*. 92 93#### Build Initialization 94 95##### G.COM.03 Provide the clean command for each component. 96 97- If the **clean** command is executed with no arguments specified, all target files, temporary files, and build logs in the build project of the current level will be cleared, and the **clean** commands of the lower-level build project are recursively called to restore the build projects to the initial state. 98- If the **clean** command is executed with certain arguments specified, only the corresponding target files, temporary files, and build logs generated in the specified build will be cleared. 99 100[Type] Requirement 101 102[Description] The **clean** command prevents a build from being affected by historical build files and build logs, ensuring build reproducibility. The **clean** command without any arguments must be supported. The **clean** command with arguments applies only to internal delivery and local build. 103 104[Positive Example] 105 106``` 107base_dir 108 |---build.suffix 109 |---logs 110 |---component_depository_1 111 |---build.suffix 112 |---logs 113 |---component_depository_2 114 |---build.suffix 115 |---logs 116 117# No arguments 118base_dir/build.suffix clean 119#....Call clean of component_depository_1 and component_depository_2. 120 121# With arguments: component name 122base_dir/build.suffix clean component_depository_1 123#....Call clean of component_depository_1 only. 124 125# With arguments 126component_depository_1/build.suffix clean makebin hert umpt 127#....Call clean of the umpt board of component_depository_1. 128``` 129 130##### G.COM.04 Clear legacy build files in the build environment before the build of a component. 131 132[Type] Requirement 133 134[Description] If code is downloaded for the first time and the build environment has been initialized, the build environment does not have legacy build files. In this case, you do not need to run the **clean** command. If build has been performed, run the **clean** command to clear legacy build files. 135 136#### Full Build 137 138##### G.COM.05 For a version release build, recompile all the archived deliverables (including all dependent platforms and components). DO NOT use incremental build. DO NOT change the installation disk by manually replacing files. 139 140A version release build refers to the build of a formally released product version (including all dependent platforms and components). 141 142[Type] Requirement 143 144[Description] Conducting incremental build after file modification may cause failures in updating some binary files and integrating new security build options into the version, causing inconsistent build results. Manually replacing files may make the build unreproducible and inconsistent. 145 146 147#### Build Configurations 148 149Separate the build configuration data from the build script to prevent the build project architecture from decaying. Store configuration data, such as the source code path, build options, and target file path, in a file different from the file that stores the build script to minimize the maintenance cost of build scripts. 150 151##### G.COM.06 DO NOT use a file that is strongly bound to the operating system (for example, an Excel file) as a build configuration file. Use a file format recognized by different platforms, such as an XML file, as the build configuration file. 152 153[Type] Requirement 154 155[Description] Using an Excel configuration file brings the following problems: 156 157- The file calls Microsoft Office APIs during the build. Each time the Excel file is accessed, the Excel program is started in the background, causing slow responding. 158 159- A large number of Excel configurations require manual operations on the GUI, resulting in poor manageability. 160 161#### Build Logs 162 163##### G.COM.07 Generate simple and clear build logs, and use the format "timestamp + [module name] (optional) + log level + log content" for each log record. 164[Type] Requirement 165 166You are advised to set the timestamp in the format of date and time, for example, *MM/dd/yyyy HH:mm:ss*. 167 168Log levels are classified into error, warning (warn), and information (info), either in lowercase or uppercase. 169 170You are advised to use square brackets ([]) to enclose each part of a log record. 171 172[Positive Example] 173 174[05/21/2020 00:12:40] [ERROR] mkdir: cannot create directory Permission denied. 175 176[Exception] If the entire log is automatically generated by a tool, you can skip the log file in the following way: Output "This project is built using + *tool name*" at the beginning of the log, for example, "This project is built using CMake." 177 178##### G.COM.08 Stop the build if error is displayed in the log. 179 180[Type] Requirement 181 182[Description] An error log indicates a build error that requires manual intervention, for example, an incorrect environment variable, tool version error, operating system error, or incorrect software source code. For a version release build, all errors generated during the build must be eliminated. DO NOT shield build errors. 183 184[Negative Example] A component is successfully built, but the build log contains a large amount of exception information, such as "fail", "critical", "cannot", "not found", "missing" and "no input files". 185 186##### G.COM.09 Retain only the logs of the current build in the build log file. 187 188[Type] Requirement 189 190[Description] Retaining historical build logs in the build log file may cause confusing. For example, when a new build fails, users may confuse the logs of historical successful builds with those of the new build and mistakenly consider the new build successful. 191 192##### G.COM.10 Add the corresponding module name to each log for quick fault locating. 193 194[Type] Recommendation 195 196[Description] If there are a large number of logs, it is difficult to quickly locate the module involved in a specific fault. Adding the corresponding module name facilitates quick fault locating. 197 198[Exception] The native logs of tools such as CMake contain the module paths, which can be used to locate the modules. For such logs, you do not need to add the module name information. 199 200#### Build Users 201 202##### G.COM.11 DO NOT use the super administrator **root** or system user for build. Instead, use a common user account. 203 204[Type] Requirement 205 206[Description] The super administrator **root** and system user have high system permissions. Using such an account for the build may cause the build environment to be tampered with. 207 208In the installation state, the **root** user account can be used. In the running state, use a common user account. If you need to run the **sudo** command for privilege escalation, comply with *IAM Security Design Specifications*. 209 210#### Build Output Files 211 212##### G.COM.12 Follow the industry conventions for build output file name extensions. 213 214[Type] Requirement 215 216[Description] An incorrect file name extension is misleading. 217 218The file name extensions of the output files, such as .lib and .obj files, must comply with the default naming rules of the build tools. 219 220[Negative Example] A text file is named ***Example*.lib**. 221 222[Negative Example] An object file is named ***Example*.a**. 223 224[Negative Example] A static library is named **lib*Example***, without a file name extension. 225 226[Positive Example] Query the common file name extension conventions at http://www.fileextension.org/, https://fileinfo.com/, https://www.file-extensions.org/, and http://file-extension.net/. 227 228Conventions about common file name extensions are as follows: 229 230| File Name Extension| Type Convention | File Name Extension| Type Convention | 231| ---------- | -------------------- | ---------- | --------------- | 232| .a | Static library | .so | Dynamic library | 233| .o | Object file | .7z | 7-zip compressed file | 234| .tar | TAR archive file | .gz/.gzip | GNU zip archive file| 235| .pack | Java pack200 compressed file| .rar/.rar5 | RAR compressed file | 236 237### C/C++ Build Projects 238 239#### Build Directory 240 241##### G.C&C++.01 Standardize the build directory structure. 242 243Build directories are classified into Source Tree, Build Tree, and Install Tree by functionality. 244 245- Source Tree is the directory for storing source code and build scripts. 246- Build Tree is the directory for storing build middleware. Generally, the directory name is **build**. 247- Install Tree is the directory for storing build deliverables. The directory name is fixed at **output**. 248 249The Source Tree, Build Tree, and Install Tree directories should be isolated from each other and should not overlap. One directory cannot be used for two or more purposes. For example, it is prohibited that a directory is used as both Source Tree to store source code and Build Tree to store build middleware. 250 251Source Tree contains the following files and directories: 252 253- Build tool entry file, such as **CMakeLists.txt**. After the **add_subdirectory()** command is executed in the upper-level **CMakeLists.txt** file, the CMake automatically invokes the **CMakeLists.txt** files in the subdirectory and lower-level subdirectories. 254- **build.*suffix*** script file, which is the one-click build entry. You only need to invoke this script to complete the build. In the build entry file name, *.suffix* (such as .bat, .sh, or .py) indicates the programming language used by the build script. 255- **config.*suffix*** configuration file, which is used to store build options and is the unique entry for configuration. 256- Build script directory, which is optional. For example, the **cmake** directory is used to store CMake scripts. CMake script files include macros, functions, and the toolchain. The **CMakeLists.txt** script contains CMake script files by executing the **include()** command, and invokes the macros and functions in the script files. 257- Component code directory, which is used to store source code and build scripts of each component. 258 259Among the preceding files and directories, only the **CMakeLists.txt**, **build.*suffix***, and **config.*suffix*** files are mandatory. Other files and directories are optional and provided as examples for your better understanding. 260 261Build Tree contains the following directories: 262 263- **build** directory, which is used to store the build middleware. This directory may be created during the build process and may not exist in the Git repository. If the **build** directory is used to store build scripts, you can create another directory as Build Tree. 264 265Install Tree contains the following directories: 266 267- **output** directory, which is used to store deliverables. This directory may be created during the build process and may not exist in the Git repository. 268 269[Type] Requirement 270 271[Description] 272 273A typical directory structure is as follows: 274 275``` 276base_dir 277 |---CMakeLists.txt ---| 278 |---build.suffix | 279 |---config.suffix | 280 |---cmake |--> Source Tree 281 |---component_1 | 282 |---component_2 | 283 |---...... | 284 |---component_n ---| 285 |---build ------> Build Tree 286 |---output ------> Install Tree 287``` 288 289The directory structure of each component is similar to the top directory structure. Example: 290 291``` 292component_1 293 |---CMakeLists.txt ---| 294 |---build.suffix | 295 |---config.suffix | 296 |---cmake |--> Source Tree 297 |---module_1 | 298 |---module_2 | 299 |---...... | 300 |---module_n ---| 301 |---build ------> Build Tree 302 |---output ------> Install Tree 303``` 304 305##### G.C&C++.02 DO NOT modify Source Tree in any form during the build process. 306 307[Type] Recommendation 308 309[Description] If Source Tree is modified during the build, the build process will be non-reproducible. 310 311The common operations for modifying Source Tree are as follows: 312 313- Patch installation 314- Dotting 315- Tailoring 316- Automatic source code generation 317- Modifying source code and then restoring it 318- Adding, modifying, or deleting temporary files or directories 319- Modifying the attribute or format of a file or directory (For example, modify the execute permission of a file or **dos2unix**.) 320 321Recommended operations are as follows: 322 323(1) Copy code to Build Tree, install a patch, and build the code. 324 325(2) As the dotting tool modifies source code and makes the build process untrustworthy, do not use the dotting tool during the build. Instead, upload the dotted code to the code repository and use the dotted code for the build. 326 327(3) Tailoring is an independent source code delivery requirement and can be considered as the code preparation phase. Source Tree in the versions before and after tailoring must not be modified during the build. 328 329(4) The automatically generated source code must be stored in Build Tree. 330 331(5) Modifying and then restoring source code is prohibited because the source code has changed during the build. 332 333(6) Temporary files or directories must be stored in Build Tree. 334 335(7) Ensure that the attributes and formats of files in the code repository are correct to prevent modifying them during the build. 336 337To check for Source Tree changes, run the **git status** command in the source code directory after the build. Source Tree should not have any change. The **git status** command may fail to detect changes to Source Tree where source code is modified and then restored. 338 339[Exception] 340 341(1) Changes in the Build Tree and Install Tree directories detected by the **git status** command are allowed. 342 343(2) Tailoring-caused changes detected by the **git status** command are allowed. 344 345##### G.C&C++.03 Use D:\*deliveryUnitName + versionNumber (optional)* and /usr1/*deliveryUnitName + versionNumber (optional)* as the build root directory in Windows and Linux, respectively. 346 347[Type] Recommendation 348 349[Description] Name the build root directory in the format of *deliveryUnitName + versionNumber*, where *versionNumber* is optional. DO NOT use the directory name that cannot identify a specific delivery unit, such as **build** or **code**. 350 351A clear build directory structure helps test personnel set build parameters, execute one-click build entries, and compare build results. 352 353The following are two root directory examples: 354 355``` 356D:\Offering [Version (optional)] or /usr1/Offering [Version (optional)] 357``` 358 359##### G.C&C++.04 Store all middleware generated during the build in Build Tree. 360 361[Type] Requirement 362 363[Description] The middleware generated during the build includes the makefile automatically generated by the build tool CMake, the source code automatically generated by the build script, the source code and patches copied by the build script, as well as the object files, repository files, executable programs, and build logs generated during the build. Store the middleware in Build Tree only, to avoid pollution to Source Tree or Install Tree. Do not store anything else in Build Tree. 364Create the **logs** subdirectory under Build Tree and name build log files in .log format. 365 366##### G.C&C++.05 You can specify any directory except Source Tree and Install Tree as Build Tree. 367 368[Type] Requirement 369 370[Description] You can specify any directory other than Source Tree and Install Tree as Build Tree. In this way, the build process is irrelevant to the directory. The directory where the build is executed is Build Tree, and the build middleware is stored in this directory. Generally, the Build Tree directory name is **build**. You can also use another name. 371 372[Positive Example] Use CMake system variables **CMAKE_BINARY_DIR** and **CMAKE_CURRENT_BINARY_DIR** to access Build Tree to prevent coupling between Build Tree and Source Tree. 373 374##### G.C&C++.06 Store all deliverables in Install Tree. 375 376[Type] Requirement 377 378[Description] In the local build scenario, deliverables are directly installed on the host computer and run. In the cross-build scenario, deliverables run on the target computer. 379 380The deliverables include library files, executable programs, package files, and header files. They are binary interfaces provided by components. All the deliverables must be stored in Install Tree. 381 382Do not place any file other than deliverables (such as build middleware) in Install Tree. 383 384##### G.C&C++.07 You can specify any directory except Source Tree and Build Tree as Install Tree. 385 386[Type] Requirement 387 388[Description] You can specify any directory other than Source Tree and Build Tree as Install Tree. In this way, the build process is irrelevant to the directory. The Install Tree directory name is fixed at **output**. 389 390[Positive Example] The CMake build project supports the function of specifying the Install Tree directory through the system variable **CMAKE_INSTALL_PREFIX**. 391 392#### Build Entry 393 394##### G.C&C++.08 Provide a unique build entry for each delivery unit. Name all the build entry scripts in the **build.*suffix*** format, and store them in the build root directory. 395 396[Type] Requirement 397 398[Description] A unique build entry allows for a more efficient and automated build process. Each delivery unit has a unique build entry, making one-click, automated build possible. 399 400[Negative Example] The following build has multiple entry points. If no description document is provided, it is difficult to determine which entry point is correct. 401 402build.bat 403 404build_all.sh 405 406build_v6.sh 407 408[Positive Example] A typical one-click build script **build.sh** is as follows: 409 410```bash 411#!/bin/bash 412 413if [ -d "build" ]; then 414 rm -fr build/* 415else 416 mkdir build 417fi 418 419if [ -d "output" ]; then 420 rm -fr output/* 421else 422 mkdir output 423fi 424 425cd build 426cmake .. 427 428cpu_processor_num=$(grep processor /proc/cpuinfo | wc -l) 429job_num=$(expr "$cpu_processor_num" \* 2) 430echo Parallel job num is "$job_num" 431make -j"$job_num" 432``` 433 434##### G.C&C++.09 You can specify the build target. 435 436[Type] Requirement 437 438[Description] In routine development scenarios, you can specify the target to build modified code. A project can be built by specifying the target to meet flexible build and debugging requirements. 439 440[Positive Example] A typical command is as follows: 441 442``` 443base_dir # cd build 444base_dir/build # cmake .. 445# Build all targets. 446base_dir/build # make 447# Build a specific target. 448base_dir/build # make target_name 449``` 450 451##### G.C&C++.10 Reproducible build is supported. 452 453[Type] Requirement 454 455[Description] If you do not modify the source code, clear the middleware and deliverables, or modify the build environment after the last successful build, you can perform a new round of build and obtain the same result. 456 457##### G.C&C++.11 Incremental build is supported. 458 459[Type] Recommendation 460 461[Description] In routine development scenarios, incremental build can improve development efficiency. Therefore, it is advised to support incremental build. 462 463##### G.C&C++.12 Parallel build is supported. 464 465[Type] Requirement 466 467[Description] You can run the **make -jN** command for quicker parallel build. This guideline applies only to projects that use the make tool. 468 469Support unified scheduling in jobserver mode to optimize the project load to the best level. The following alarms are not allowed: 470 471``` 472warning: jobserver unavailable: using -j1. Add '+' to parent make rule. 473warning: -jN forced in submake: disabling jobserver mode. 474``` 475 476To support the jobserver mode, perform any of the following operations: 477 4781. Use **$(MAKE)** to directly invoke the **make** command. 479 480 ```cmake 481 ExternalProject_Add(foo 482 SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/foo 483 CONFIGURE_COMMAND sh configure_ext.sh 484 BUILD_COMMAND $(MAKE) 485 ) 486 ``` 487 4882. Use the shell script to invoke the **make** command. 489 490 ```cmake 491 ExternalProject_Add(foo 492 SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/foo 493 CONFIGURE_COMMAND sh configure_ext.sh 494 BUILD_COMMAND sh build_ext.sh $(MAKE) 495 ) 496 ``` 497 498 The content of **build_ext.sh** is as follows: 499 500 ```bash 501 #!/bin/bash 502 503 make 504 ``` 505 506 Note: **build_ext.sh** does not need to parse or use the **$(MAKE)** parameter. 507 5083. Use the python script to invoke the **make** command. 509 510 ```cmake 511 ExternalProject_Add(foo 512 SOURCE_DIR ${CMAKE_CURRENT_SOURCE_DIR}/foo 513 CONFIGURE_COMMAND sh configure_ext.sh 514 BUILD_COMMAND python build_ext.py $(MAKE) 515 ) 516 ``` 517 518 The content of **build_ext.py** is as follows: 519 520 ```bash 521 #!/usr/bin/python 522 # -*- coding: UTF-8 -*- 523 524 import subprocess 525 526 def main(): 527 child = subprocess.Popen("make", close_fds=False) 528 ret = child.wait() 529 return 530 531 if __name__ == '__main__': 532 main() 533 ``` 534 535 Note: **build_ext.py** does not need to parse or use the **$(MAKE)** parameter. 536 537 538#### Build Dependencies 539 540##### G.C&C++.13 Define a build dependency file **dependence.xml** to describe all components on which the build depends. The build script automatically reads the dependency file to produce the final software package. 541 542[Type] Recommendation 543 544[Description] Make software packages based on the dependency file, so that you do not need to define dependency components in the build scripts. This improves the build process maintainability. 545 546#### Build Configurations 547 548##### G.C&C++.14 Use the configuration file **config.*suffix*** in the build root directory as the unique configuration entry for the entire delivery project. 549 550[Type] Requirement 551 552[Description] Expose the least configuration options in the top-level **config.*suffix*** file. Configure only the information about the build environment and build tool in this file. 553 554[Exception] If there are a small number of build options and key-value pairs are used, the configuration file can be named **config.conf**. 555 556### GN Build Specifications 557 558#### Build Rules 559 560##### Rule 1.1 DO NOT use GN to invoke external build tools to build software modules. 561 562[Type] Forbidden 563 564[Description] Port external components to the GN build mode to avoid unnecessary dependencies on the environment during the build and obtain common capabilities, such as compiler security options and AddressSanitizer (ASan), provided by the build framework. 565 566[Negative Example] In GN, use **action** to invoke **automake** and **Make** to build third-party components. 567 568[Exception] The Linux kernel build framework builds user-mode programs. The kernel can be independently built outside the build framework. It is acceptable that some platforms use GN to include the kernel build in the build process to deliver one-click builds. 569 570##### Rule 1.2 DO NOT add compiler security options that have been added to the build system to the GN file of the module. 571 572[Type] Forbidden 573 574[Description] The default options that have been added globally should not be added again to meet internal and external rules. 575 576| Option| Parameter | Default Value | 577|---------|------------|------------| 578| Stack protection | -fstack-protector-strong| Enabled| 579| Fortify Source | -D_FORTIFY_SOURCE=2 -O2 | Enabled| 580 581[Negative Example] Add **-fstack-protector-strong** to the GN file of the module. 582 583##### Rule 1.3 DO NOT add build options that are opposite to the default build options to GN. 584 585[Type] Forbidden 586 587[Description] The default build options represent the default capabilities of the build system. If your module needs to remove some default build options, there must be sufficient reasons. 588 589[Negative Example] Add **-wno-unused** to a module to clear build alarms. 590 591[Exception] When porting or using a third-party component, you can overwrite the default build options based on the component requirements. 592 593##### Rule 2.1 Use **gn format** to format GN files to meet the format and typesetting requirements. 594 595[Type] Requirement 596 597##### Rule 2.2 Use Python instead of shell to compile **action**. 598 599[Type] Recommendation 600 601[Description] The Python environment is easier to keep code unified and can run on multiple operating systems. It also provides better scalability, readability, and testability. 602 603##### Rule 2.3 DO NOT modify the content in the source code directory during the execution of GN and Ninja. 604 605[Type] Forbidden 606 607[Description] The forbidden operations include but are not limited to installing patches for, copying files to, performing build tasks in, and generating intermediate files in the source code directory. 608 609##### Rule 2.4 Set the encoding format of the build script to UTF-8 and the newline character to UNIX format. 610 611[Type] Requirement 612 613[Negative Example] After a script is compiled on Windows, Chinese comments are used and saved as local codes.