1# Android Build System Concepts 2 3This document provides high level explanations and mapping of the internal 4build system components and concepts of the Android build system and Bazel, 5and how components communicate with each other. 6 7For implementation concepts, see: 8https://android.googlesource.com/platform/build/bazel/+/refs/heads/master/docs/internal_concepts.md. 9 10## High level components 11 12This table provides a high level overview of the components in the current 13Android Platform build system, and how each component maps to a concept in 14Bazel. 15 16|Android build system component|Description|Mapping to Bazel concepts| 17|---|---|---| 18|Kati|Make-compatible front-end. Encodes build logic in `.mk` scripts. Declares buildable units in `Android.mk`. Generates Ninja file directly.|Loading and analysis phase. Conceptually similar to `bazel build --nobuild`.| 19|Blueprint|Build definition syntax. Build syntax parser. Internal data structures like Modules/Variations/Context/Scope. Ninja file generator.|Starlark.| 20|Soong|Bazel-like front-end. Encodes build logic in Go. Declares build units in `Android.bp`, parsed by Blueprint. Uses Blueprint to generate Ninja file. Generates a `.mk` file with prebuilt module stubs to Kati.|Loading and analysis phase. Conceptually similar to `bazel build --nobuild command`.| 21|Ninja|Serialized command line action graph executor. Executes Ninja graph generated from Kati and Soong.|Bazel's execution phase.| 22|atest|Test executor and orchestrator.|Conceptually similar to `bazel test` command.| 23|Blueprint + Kati + Soong + Ninja + atest|The entire build pipeline for Android.|Conceptually similar to `bazel build` or `bazel test` commands.| 24|`<script>.sh`|Running arbitrary scripts in AOSP.|Conceptually similar to `bazel run` command.| 25|Make (replaced in-place by Kati)|No longer in use. Entire build system, replaced by the tools above.|Loading, analysis, execution phases. Conceptually similar to `bazel build` command.| 26|`Android.bp`|Build definition file for Soong.|`BUILD.bazel` or `BUILD`.| 27|`Android.mk`|Build definition file for Kati.|`BUILD.bazel` or `BUILD`.| 28 29## Communication between components 30 31* Kati product configuration component to generate config variables (config.mk, AndroidProducts.mk) 32 * **“Kati-config” for the purpose of this document** 33* Kati component to generate build actions in Ninja files (main.mk, Android.mk files) 34 * **“Kati-build” for the purpose of this document** 35* Kati component to generate packaging actions in Ninja files (packaging.mk file) 36 * **“Kati-package” for the purpose of this document** 37* Kati component to generate cleaning actions in Ninja files (cleanbuild.mk, CleanSpec.mk files) 38 * **"Kati-cleanspec" for the purpose of this document** 39* **soong\_build** (and **Blueprint**) component to generate build actions (Android.bp, Blueprints files) 40* **Ninja** component to execute actions from Kati-build, Kati-package and soong\_build 41* **Bazel** as the next generation of the entire build system, starting as a Ninja executor drop-in replacement 42* **soong\_ui** as the tool to orchestrate all of the above, and with auxiliary tools like finder, path\_interposer, soong\_env, minibp and bpglob. 43 44The current build system architecture primarily uses **files** as the medium 45for inter-process communication (IPC), with one known case of unix socket 46communication (e.g. 47[path\_interposer](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/paths/logs.go;l=112-133;drc=184901135cda8bdcc51cab4f16c401a28a510593)), 48and a fifo between Ninja and soong_ui for the Protobuf stream for build 49status reporting. 50 51## Component order 52 53The build system components run in the following order, orchestrated by soong\_ui: 54 551. soong\_ui bootstraps itself with microfactory (`go build` replacement) and launches. 561. soong\_ui runs auxiliary tools to aggregate files into filelists, for 57Android.mk, Android.bp, AndroidProducts.mk and several others. 581. soong\_ui runs Kati-config with 59 [the config.mk entry point](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/dumpvars.go;l=89;drc=9f43597ff7349c4facd9e338e5b4b277e625e518). 601. soong\_ui orchestrates 3 Blueprint/Soong phases to generate the main out/soong/build.ninja file: 61 minibootstrap, bootstrap, and primary. 62 1. Minibootstrap phase uses Blueprint/Microfactory to build itself 63 (minibp) so that Android.bp and Blueprint files can be used to define 64 Soong. 65 1. Bootstrap phase runs Ninja on a build.ninja file that runs minibp to 66 read all Android.bp files across the source tree that describes Soong and 67 plugins, and builds soong\_build. 68 1. Primary phase runs Ninja on a build.ninja file that runs soong_build 69 to generate the final out/soong/build.ninja file. 70 1. soong\_build also runs its own tests alongside generating 71 out/soong/build.ninja, which can be skipped with the `--skip-soong-tests` 72 argument. 731. soong\_ui runs Kati-cleanspec with 74 [the cleanbuild.mk entry point](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/kati.go;l=362;drc=b1d30d63c5d1b818ea38e77cd155da2016fe8b6c). 751. soong\_ui runs Kati-build to generate a Ninja file, with 76 [the main.mk entry point.](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/kati.go;l=202;drc=b1d30d63c5d1b818ea38e77cd155da2016fe8b6c) 771. soong\_ui runs Kati-package to generate a Ninja file, with 78 [the packaging/main.mk](https://cs.android.com/android/platform/superproject/+/master:build/soong/ui/build/kati.go;l=314;drc=b1d30d63c5d1b818ea38e77cd155da2016fe8b6c) 79 entry point. 801. soong\_ui generates a Ninja file to combine above Ninja files. 811. soong\_ui runs either Ninja or Bazel to execute the build, with the 82combined Ninja file as entry point. 83 84soong\_ui has a --skip-make flag that will skip Kati-config, Kati cleanspec, 85Kati-build and Kati-package, used for Soong-only builds in NDK and some 86Mainline projects. 87 88### soong\_ui 89 90soong\_ui is primarily responsible for orchestrating the build, cleaning the 91build environment, and running auxiliary tools. These tools (minibp, 92microfactory) can bootstrap other tools (soong\_build), aggregate file lists 93(finder.go), improve hermeticity (path\_interposer, nsjail) or perform checks 94against the environment (soong\_env). 95 96soong\_ui uses finder.go to generate <filename>.list files for other 97tools. For example, it generates Android.mk.list for Kati-build, 98AndroidProducts.mk.list for Kati-config, and Android.bp.list for 99soong\_build. 100 101soong\_ui uses path\_interposer to prepare an hermetic $PATH with runtime 102checks against allowlisted system tools. The $PATH contains these system 103tools with checked-in prebuilts, and uses path\_interposer to intercept calls 104and error out whenever non-allowlisted tools are used (see out/.path for 105directory of intercepted tool symlinks). 106 107soong\_ui generates a Kati suffix to ensure that Kati-generated files are 108regenerated if inputs to Kati have changed between builds. 109 110soong\_ui calls Soong and Kati to generate Ninja files, and eventually 111creates another Ninja file (out/combined-<product>.ninja) to combine the 112others, and executes either Ninja or Bazel to complete the build. 113 114soong\_ui sets up the sandbox and environment for the Ninja/Bazel process. 115 116## Kati-config 117 118As a product configuration tool, soong\_ui runs Kati-config in 119**[--dumpvars-mode](https://cs.android.com/android/platform/superproject/+/master:build/soong/cmd/soong_ui/main.go;l=298-305;drc=master)** 120to dump the values of specified Make variables at the end of an evaluation, 121with build/make/core/config.mk as the entry point. During this phase, 122Kati-config eventually evaluates[ 123soong\_config.mk](https://cs.android.com/android/platform/superproject/+/master:build/make/core/soong_config.mk;l=2?q=soong.variables) 124to generate the **[soong.variables JSON 125file](https://cs.android.com/android/platform/superproject/+/master:build/make/core/soong_config.mk;l=16-222;drc=341928ecc5da205401bcfd86f098662b0cee7857)**. 126This way, Kati-config can communicate product configuration to soong\_build, 127as soong\_build parses the dumped variables from the JSON on startup, and 128stores them into an in-memory Config object. 129 130To communicate 131[dexpreopt](https://cs.android.com/android/platform/superproject/+/master:build/soong/java/dexpreopt.go;l=115;drc=8cbc5d269b20b5743679bfb8684ed174dcf58a30) 132variables to soong\_build, [dexpreopt.config is also 133generated](https://cs.android.com/android/platform/superproject/+/master:build/make/core/dex_preopt_config.mk;l=71-142;drc=f26015449f0747b9fdeceb5ce70e30ecd76e20e8) 134as a $(shell) action and [read by 135soong\_build](https://cs.android.com/android/platform/superproject/+/master:build/soong/dexpreopt/config.go;l=175-196;drc=1af783fae74715bcf1a94733bd75b2e6cc688e8c) 136in a similar way as Kati-config evaluates dex\_preopt\_config.mk included in 137soong\_config.mk. 138 139soong\_ui sets up a **KatiReader** to monitor Kati-config’s stdout/err for UI 140reporting and error handling purposes. 141 142## soong\_build 143 144soong\_build’s primary role is to evaluate all Android.bp files, run a series 145of mutators, and generate out/soong/build.ninja file. 146 147soong\_build communicates with Kati-build by generating Make Vars and running 148the AndroidMk singleton to generate .mk files in the output directory 149(out/soong/{Android, late, make\_vars}-<product>.mk). 150 151* Android-<product>.mk contains Soong modules as Make modules so Make 152 modules can depend on Soong modules. 153* make\_vars-<product>.mk contains Make variables for Kati-build, exported 154 from Soong modules. There are also checks built into this .mk file to ensure that 155 if a duplicate Make variable of the same name comes from another source, the Soong 156 and Make variable values are identical. 157* late-<product>.mk contains Make variables that are not read while Kati-build 158 parses the Android.mk file. (Late variables) 159 * soong\_ui invokes Kati to parse make\_vars .mk file earlier than the Android.mk 160 files,and late.mk after parsing the Android.mk files. 161 * late.mk is used to define phony rules to take advantage of Make’s ability to 162 add extra dependencies to an existing rule. late.mk is not strictly necessary to 163 make this happen at this moment, since late.mk rules don’t currently depend on any 164 variables defined during Android.mk processing (e.g. ALL\_MODULES$(module).INSTALLED). 165 166## Kati-build / Kati-package 167 168Kati-build’s primary role is to evaluate all Android.mk files with 169build/make/core/main.mk as entry point, and generate 170out/build-<product>.ninja. It also generates cleanspec.ninja for the 171product, containing statements on how to remove stale output files. 172 173Kati-build’s primary role is to evaluate all packaging .mk files with 174build/make/packaging/main.mk as entry point, including 175build/make/packaging/distdir.mk for dist-for-goals calls, and generate 176out/package-<product>.ninja. 177 178Kati-build/Kati-package’s stdout/stderr is monitored by soong\_ui’s 179KatiReader to UI and error handling. 180 181As Kati-build/Kati-package generates Ninja files, they also generate 182out/ninja-<product>.sh and out/env-<product>.sh. These scripts are 183wrappers for soong\_ui to execute Ninja with the correct Ninja files, in a 184controlled environment. 185 186## Ninja 187 188As Ninja executes files from Kati-build, Kati-package, soong\_build and other 189bootstrapping tools like Blueprint, it writes to a fifo in a proto front end 190that soong\_ui monitors with NinjaReader. NinjaReader ensures that the user 191interface for Ninja progress is consistent with the rest of the build. 192 193## Bazel 194 195As more Soong modules are converted to BUILD files, soong\_build serializes 196information about converted modules to BUILD/bzl files on disk. soong\_build 197then consumes information about these targets from Bazel by directly calling 198the Bazel client to issue `cquery` calls about these targets. 199