1This directory contains a number of shell scripts, which we will 2call the "dev-scripts", that are only used to develop the NDK 3itself, i.e. they are un-needed when using ndk-build to build 4applicative native code. 5 6Their purpose is to handle various sophisticated issues: 7 8 - rebuilding host cross-toolchains for our supported CPU ABIs 9 10 - rebuilding other required host tools (e.g. ndk-stack) from sources 11 12 - rebuilding all target-specific prebuilt binaries from sources 13 (this requires working host cross-toolchains) 14 15 - packaging final NDK release tarballs, including adding samples 16 and documentation which normally live in $NDK/../development/ndk 17 18This document is here to explain how to use these dev-scripts and how 19everything is architected / designed, in case you want to maintain it. 20 21 22I. Organization: 23---------------- 24 25First, a small description of the NDK's overall directory structure: 26 27 $NDK/build/core 28 Contains the main NDK build system used when 'ndk-build'. 29 Relies heavily on GNU Make 3.81+ but isn't used by any of the 30 scripts described here. 31 32 $NDK/build/tools 33 Contains all the dev-scripts that are described in this document. 34 More on this later. 35 36 $NDK/sources/host-tools/ 37 Contains sources of various libraries or programs that will be 38 compiled to generate useful host programs for the final NDK 39 installation. For example, $NDK/sources/host-tools/ndk-stack/ 40 contains the sources of the 'ndk-stack' program. 41 42 $NDK/sources/cxx-stl/ 43 Contains the sources of various C++ runtime and libraries that 44 can be used with 'ndk-build'. See docs/CPLUSPLUS-SUPPORT.html for 45 more details. 46 47 $NDK/sources/cxx-stl/gabi++/ 48 Contains the sources of the GAbi++ C++ runtime library. Note that 49 the dev-script 'build-cxx-stl.sh' can be used to generate prebuilt 50 libraries from these sources, that will be copied under this 51 directory. 52 53 $NDK/sources/cxx-stl/stlport/ 54 Contains the sources of a port of STLport that can be used with ndk-build. 55 The dev-script 'build-cxx-stl.sh' can be used to generate prebuilt 56 libraries from these sources, that will be copied under this directory. 57 58 $NDK/sources/cxx-stl/llvm-libc++/ 59 Contains the sources of a port of LLVM's libc++ that can be used with 60 ndk-build. The dev-script 'build-cxx-stl.sh' can be used to generate 61 prebuilt libraries from these sources, that will be copied under this 62 directory. 63 64 $NDK/sources/cxx-stl/gnu-libstdc++/ 65 This directory doesn't contain sources at all, only an Android.mk. 66 The dev-script 'build-gnu-libstdc++.sh' is used to generate prebuilt 67 libraries from the sources that are located in the toolchain source 68 tree instead. 69 70 $NDK/sources/cxx-stl/system/ 71 This directory contains a few headers used to use the native system 72 Android C++ runtime (with _very_ limited capabilites), a.k.a. 73 /system/lib/libstdc++.so. The prebuilt version of this library is 74 generated by the 'gen-platform.sh' dev-script described later, but 75 it never placed in this directory. 76 77 $NDK/sources/android/libthread_db/ 78 This directory contains the sources of the libthread_db implementation 79 that is linked into the prebuilt target gdbserver binary. Note that two 80 versions are provided, corresponding to different versions of gdb. 81 82 The sources are customized versions of the ones found under 83 $ANDROID/bionic/libthread_db. They contain various changes used to 84 deal with platform bugs. 85 86 $NDK/sources/ 87 The rest of $NDK/sources/ is used to store the sources of helper 88 libraries used with 'ndk-build'. For example, the 'cpu-features' 89 helper library is under $NDK/sources/android/cpu-features/ 90 91 $DEVNDK a.k.a $NDK/../development/ndk 92 This directory contains platform-specific files. The reason why it 93 it is separate from $NDK is because it is not primarly developed in 94 the open. 95 96 More specifically: 97 98 - all $NDK development happens in the public AOSP repository ndk.git 99 100 - any $DEVNDK development that happens in the public AOSP development.git 101 repository is auto-merged to the internal tree maintained by Google 102 103 - $DEVNDK developments that are specific to an yet-unreleased version of 104 the system happen only in the internal tree. They get back-ported to 105 the public tree only when the corresponding system release is 106 open-sourced. 107 108 Having two distinct git repositories makes things easier. Also, you 109 generally don't want to see the churn that goes in the internal tree 110 during development. It's usually not very pretty :) 111 112 $DEVNDK/platforms/android-$PLATFORM, where $PLATFORM is a decimal number 113 Contains all files that are specific to a given API level $PLATFORM, 114 that were not already defined for the previous API level. 115 116 For example, android-3 corresponds to Android 1.5, and android-4 117 corresponds to Android 1.6. The platforms/android-4 directory only 118 contains files that are either new or modified, compared to android-3. 119 120 $DEVNDK/platforms/android-$PLATFORM/include/ 121 Contains all system headers exposed by the NDK for a given platform. 122 All these headers are independent from the CPU architecture of target devices. 123 124 $DEVNDK/platforms/android-$PLATFORM/arch-$ARCH/ 125 Contains all files that are specific to a given $PLATFORM level and a 126 specific CPU architecture. $ARCH is typically 'arm' or 'x86' 127 128 $DEVNDK/platforms/android-$PLATFORM/arch-$ARCH/include/ 129 Contains all the architecture-specific headers for a given API level. 130 131 $DEVNDK/platforms/android-$PLATFORM/arch-$ARCH/lib/ 132 Contains several CPU-specific object files and static libraries that 133 are required to build the host cross-toolchains properly. 134 135 Before NDK r7, this also contains prebuilt system shared libraries that 136 had been hand-picked from various platform builds. These have been 137 replaced by symbol list files instead (see below). 138 139 $DEVNDK/platforms/android-$PLATFORM/arch-$ARCH/symbols/ 140 Contains, for each system shared library exposed by the NDK, two 141 files describing the dynamic symbols it exports, for example, for 142 the C library: 143 144 libc.so.functions.txt -> list of exported function names 145 libc.so.variables.txt -> list of exported variable names 146 147 These files were introduced in NDK r7 and are used to generate "shell" 148 shared libraries that can be used by ndk-build at link time. The shell 149 libraries only provide dynamic symbols to 150 151 These files can be generated from a given platform build using the 152 'dev-platform-import.sh' dev-script, described later in this document. 153 154 This is handy to compare which symbols were added between platform 155 releases (and check that nothing disappeared). 156 157 $DEVNDK/platforms/android-$PLATFORM/samples/ 158 Contains samples that are specific to a given API level. These are 159 usually copied into $INSTALLED_NDK/samples/ by the 'gen-platforms.sh' 160 script. 161 162 163 $NDK/platforms/ 164 Note to be confused with $DEVNDK/platforms/, this directory is not 165 part of the NDK git directory (and is spefically listed in $NDK/.gitignore) 166 but of its final installation. 167 168 Its purpose is to hold the fully expanded platform-specific files. 169 This means that, unlike $DEVNDK/platforms/android-$PLATFORM, the 170 $NDK/platforms/android-$PLATFORM will contain _all_ the files that 171 are specific to API level $PLATFORM. 172 173 Moreover, the directory is organized slightly differently, i.e. as 174 toolchain sysroot, i.e. for each supported $PLATFORM and $ARCH values, 175 it provides two directories: 176 177 $NDK/platforms/android-$PLATFORM/arch-$ARCH/usr/include 178 $NDK/platforms/android-$PLATFORM/arch-$ARCH/usr/lib 179 180 Notice the 'usr' subdirectory here. It is required by GCC to be able 181 to use the directories with --with-sysroot. For example, to generate 182 binaries that target API level 5 for the arm architecture, one would 183 use: 184 185 $TOOLCHAIN_PREFIX-gcc --with-sysroot=$NDK/platforms/android-5/arch-arm 186 187 Where $TOOLCHAIN_PREFIX depends on the exact toolchain being used. 188 189 The dev-script 'gen-platforms.sh' is used to populate $NDK/platforms. 190 Note that by default, the script does more, see its detailed description 191 below. 192 193Generally, everything dev-script supports the --help option to display a 194description of the program and the list of all supported options. Also, 195debug traces can be activated by using the --verbose option. Use it several 196times to increase the level of verbosity. 197 198Note that all Windows host programs can be built on Linux if you have the 199'mingw32' cross-toolchain installed ('apt-get install mingw32' on Debian or 200Ubuntu). You will need to add the '--mingw' option when invoking the script. 201 202All dev-scripts rebuilding host programs on Linux and Darwin will only 203generate 32-bit programs by default. You can experiment with 64-bit binary 204generation by adding the '--try-64' option. Note that as of now, 64-bit 205binaries are never distributed as part of official NDK releases. 206 207When building 32-bit Linux host programs, the dev-scripts will look for 208$NDK/../prebuilts/gcc/linux-x86/host/i686-linux-glibc2.7-4.6/, which is 209part of the Android platform source tree. It is a special toolchain that 210ensures that the generated programs can run on old systems like Ubuntu 8.04 211that only have GLibc 2.7. Otherwise, the corresponding binaries may not run 212due to ABI changes in mor recent versions of GLibc. 213 214This shall always be used to official NDK Linux releases. 215 216 217II. Host toolchains: 218-------------------- 219 220The host toolchains are the compiler, linker, debugger and other crucial 221programs used to generate machine code for the target Android 222system supported by the NDK. 223 224 225II.1 Getting the toolchain sources: 226- - - - - - - - - - - - - - - - - - 227 228The NDK toolchain sources are located under the toolchain/ directly 229from the top of the public AOSP repositories. There are actually several 230git repositories of interest here: 231 232 binutils/ 233 build/ 234 gcc/ 235 gdb/ 236 gold/ 237 gmp/ 238 mpfr/ 239 240The AOSP toolchain/ repository contains sources corresponding to toolchains 241that are used to build the Android platform tree. For various reasons, they 242need to be slightly patched for use with the NDK. 243 244All patches are located under $NDK/build/tools/toolchain-patches/ 245 246The script 'download-toolchain-sources.sh' can be used to download and patch 247the toolchain sources from the server. You simply need to give it the name of 248a destination directory, for example: 249 250 $NDK/build/tools/download-toolchain-sources.sh /tmp/ndk-$USER/src 251 252Note that this process can take several minutes. If you happen to already 253have a clone of the repo toolchain/ tree, you can speed this up considerably 254using the --git-reference or --git-base option. See --help for more options 255and details. 256 257Note, to create a local toolchain repo: 258 259 mkdir -p /tmp/ndk-$USER/toolchain 260 cd /tmp/ndk-$USER/toolchain 261 repo init https://android.googlesource.com/toolchain/manifest.git 262 repo sync 263 264then use with: 265 266 $NDK/build/tools/download-toolchain-sources.sh \ 267 --git-reference=/tmp/ndk-$USER/toolchain \ 268 /tmp/ndk-$USER/src 269 270 271The script doesn't simply copy the current HEAD of the toolchain 272repositories. Instead, it will checkout the sources as they were 273at a given known date, defined as TOOLCHAIN_GIT_DATE in 'prebuilt-common.sh'. 274 275If you adjust TOOLCHAIN_GIT_DATE, verify that all patches still apply 276correctly, you might need to remove patches if they have been integrated 277into the toolchain/ repositories. 278 279Note: you can avoid the patching with the --no-patches option to 280'download-toolchain-patches.sh', and later use 'patch-sources.sh' manually 281to verify that the patches apply cleanly. 282 283The toolchains binaries are typically placed under the directory 284$NDK/toolchains/$NAME/prebuilt/$SYSTEM, where $NAME is the toolchain name's 285full name (e.g. arm-linux-androideabi-4.6), and $SYSTEM is the name of the 286host system it is meant to run on (e.g. 'linux-x86', 'windows' or 'darwin-x86') 287 288 289I.2. Building the toolchains: 290- - - - - - - - - - - - - - - 291 292After downloading and patching the toolchain sources, you will need to build 293a proper "sysroot" directory before being able to configure/build them. 294 295A sysroot is a directory containing system headers and libraries that the 296compiler will use to build a few required target-specific binaries (e.g. libgcc.a) 297 298To do that, use: 299 300 $NDK/build/tools/gen-platforms.sh --minimal 301 302This will populate $NDK/platforms/ with just the files necessary to rebuild the 303toolchains. Note that without the --minimal option, the script will fail without 304prebuilt toolchain binaries. 305 306Once the sysroots are in place, use 'build-gcc.sh' by providing the path 307to the toolchain sources root directory, a destination NDK installation 308directory to build, and the full toolchain name. 309 310For example, to rebuild the arm and x86 prebuilt toolchain binaries in the 311current NDK directory (which can be handy if you want to later use them to 312rebuild other target prebuilts or run tests), do: 313 314 $NDK/build/tools/build-gcc.sh /tmp/ndk-$USER/src $NDK arm-linux-androideabi-4.6 315 $NDK/build/tools/build-gcc.sh /tmp/ndk-$USER/src $NDK x86-4.6 316 317Here, we assume you downloaded the toolchain sources in /tmp/ndk-$USER/src 318as described in the previous section. 319 320This operation can take some time. The script automatically performs a parallel 321build to speed up the build on multi-core machine (use the -j<number> option to 322control this), but the GCC sources are very large, so expect to wait a few 323minutes. 324 325For the record, on a 2.4 GHz Xeon with 16 Hyper-threaded cores and 12GB of 326memory, rebuilding each toolchain takes between 2 and 4 minutes. 327 328You need to be on Linux to build the Windows binaries, using the "mingw32" 329cross-toolchain (install it with "apt-get install mingw32" on Ubuntu). To do so 330use the "--mingw" option, as in: 331 332 $NDK/build/tools/build-gcc.sh --mingw \ 333 /tmp/ndk-$USER/src $NDK arm-linux-androideabi-4.6 334 335 $NDK/build/tools/build-gcc.sh --mingw \ 336 /tmp/ndk-$USER/src $NDK x86-4.6 337 338The corresponding binaries are installed under $NDK/toolchains/$NAME/prebuilt/windows 339Note that these are native Windows programs, not Cygwin ones. 340 341Building the Windows toolchains under MSys and Cygwin is completely unsupported 342and highly un-recommended: even if it works, it will probably take several 343hours, even on a powerful machine :-( 344 345The Darwin binaries must be generated on a Darwin machine. Note that the script 346will try to use the 10.5 XCode SDK if it is installed on your system. This 347ensures that the generated binaries run on Leopard, even if you're building 348on a more recent version of the system. 349 350Once you've completed your builds, you should be able to generate the other 351target-specific prebuilts. 352 353 354III. Target-specific prebuilt binaries: 355--------------------------------------- 356 357A final NDK installation comes with a lot of various target-specific prebuilt 358binaries that must be generated from sources once you have working host toolchains. 359 360III.1.: Preparation of platform sysroots: 361- - - - - - - - - - - - - - - - - - - - - 362 363Each target prebuilt is handled by a specific dev-script. HOWEVER, all these script 364require that you generate a fully populated $NDK/platforms/ directory first. To do 365that, simply run: 366 367 $NDK/gen-platforms.sh 368 369Note that we used this script with the --minimal option to generate the host 370toolchains. That's because without this flag, the script will also auto-generate 371tiny versions of the system shared libraries that will be used at link-time when 372building our target prebuilts. 373 374III.2.: Generation of gdbserver: 375- - - - - - - - - - - - - - - - - 376 377A target-specific 'gdbserver' binary is required. This is a small program 378that is run on the device through 'ndk-gdb' during debugging. For a variety 379of technical reasons, it must be copied into a debuggable project's output 380directory when 'ndk-build' is called. 381 382The prebuilt binary is placed under $NDK/toolchains/$NAME/prebuilt/gdbserver 383in the final NDK installation. You can generate with 'build-gdbserver.sh' and 384takes the same parameters than 'build-gcc.sh'. So one can do: 385 386 $NDK/build/tools/build-gcc.sh /tmp/ndk-$USER/src $NDK arm-linux-androideabi-4.6 387 $NDK/build/tools/build-gcc.sh /tmp/ndk-$USER/src $NDK x86-4.6 388 389 390III.3. Generating C++ runtime prebuilt binaries: 391- - - - - - - - - - - - - - - - - - - - - - - - 392 393Sources and support files for several C++ runtimes / standard libraries are 394provided under $NDK/sources/cxx-stl/. Several dev-scripts are provided to 395rebuild their binaries. The scripts place them to their respective location 396(e.g. the GAbi++ binaries will go to $NDK/sources/cxx-stl/gabi++/libs/) 397unless you use the --out-dir=<path> option. 398 399Note that: 400 401 - each script will generate the binaries for all the CPU ABIs supported by 402 the NDK, e.g. armeabi, armeabi-v7a, x86 and mips. You can restrict them using 403 the --abis=<list> option though. 404 405 - the GNU libstdc++ dev-script requires the path to the toolchain sources, 406 since this is where the library's sources are located. 407 408An example usage would be: 409 410 $NDK/build/tools/build-cxx-stl.sh --stl=gabi++ 411 $NDK/build/tools/build-cxx-stl.sh --stl=stlport 412 $NDK/build/tools/build-cxx-stl.sh --stl=libc++ 413 $NDK/build/tools/build-gnu-libstdc++.sh /tmp/ndk-$USER/src 414 415Note that generating the STLport and GNU libstdc++ binaries can take a 416few minutes. You can follow the build by using the --verbose option to display 417what's going on. 418 419 420IV. Other host prebuilt binaries: 421--------------------------------- 422 423There are a few other host prebuilt binaries that are needed for a full 424NDK installation. Their sources are typically installed under 425$NDK/sources/host-tools/ 426 427Note that the corresponding dev-script recognize the --mingw and --try-64 428options described at the end of section I above. 429 430 431IV.1.: Building 'ndk-stack': 432- - - - - - - - - - - - - - 433 434The 'build-ndk-stack.sh' script can be used to rebuild the 'ndk-stack' 435helper host program. See docs/NDK-STACK.html for a usage description. 436To build it, just do: 437 438 $NDK/build/tools/build-ndk-stack.sh 439 440IV.2.: Building 'ndk-depends': 441- - - - - - - - - - - - - - - 442 443Similar to 'ndk-stack', see the 'build-ndk-depends.sh' script. 444 445 446V. Packaging all prebuilts: 447--------------------------- 448 449Generating all the prebuilt binaries takes a lot of time and is no fun. 450To avoid doing it again and again, it is useful to place all the generated 451files aside in special tarballs. 452 453Most dev-scripts generating them typically support a --package-dir=<path> 454option to do this, where <path> points to a directory that will store 455compressed tarballs of the generated binaries. 456 457For example, to build and package the GAbi++ binaries, use: 458 459 $NDK/build/tools/build-cxx-stl.sh --stl=gabi++ --package-dir=/tmp/ndk-$USER/prebuilt/ 460 461In NDK r7, this will actually create three tarballs (one per supported ABI), 462under the directory /tmp/ndk-$USER/prebuilt/, i.e.: 463 464 gabixx-libs-armeabi.tar.bz2 465 gabixx-libs-armeabi-v7a.tar.bz2 466 gabixx-libs-x86.tar.bz2 467 ... 468 469Note that these tarballs are built to be uncompressed from the top-level 470of an existing NDK install tree. 471 472Similarly, to rebuild the STLport binaries and package them: 473 474 $NDK/build/tools/build-cxx-stl.sh --stl=stlport --package-dir=/tmp/ndk-$USER/prebuilt 475 476A dev-script is provided to rebuild _and_ package all prebuilts. It is called 477'rebuild-all-prebuilt.sh'. Note that by default, it will automatically 478invoke 'download-toolchain-sources.sh', and place the prebuilt tarballs 479under /tmp/ndk-$USER/prebuilt-$DATE, where $DATE is the current date in 480ISO format (e.g. 20110915 for the 15th of September of 2011). 481 482If you have already downloaded the toolchain sources, use the 483--toolchain-src-dir=<path> option to save a few minutes to your rebuild, 484as in: 485 486 $NDK/build/tools/rebuild-all-prebuilt.sh \ 487 --toolchain-src-dir=/tmp/ndk-$USER/src 488 489By default, this only rebuilds the host prebuilds for the current host 490system. You can use --mingw to force the generation of Windows binaries on 491Linux. 492 493Additionally, you can use the --darwin-ssh=<hostname> option to launch the 494build of the Darwin binaries from a Linux machine, by using ssh to access a 495remote Darwin machine. The script will package all required sources into a 496temporary tarball, copy it to the remote machine, launch the build there, 497then copy back all binaries to your own machine. 498 499This means that it is possible to generate the host binaries for all supported 500host systems from Linux (provided you have ssh access to a Darwin machine). 501 502Alternatively, you can run 'rebuild-all-prebuilt.sh' on a Darwin machine. 503 504Once you have used the script three times (once per supported host systems), 505you should have plenty of files under /tmp/ndk-$USER/prebuilt-$DATE. 506For the record, with NDK r7, the list was: 507 508 509VI. Packaging NDK releases: 510--------------------------- 511 512Use the 'package-release.sh' dev-script to generate full NDK release packages. 513These contain everything needed by a typical NDK user, including: 514 515 - all prebuilt binaries (host toolchains, host tools, target libs, etc...) 516 - all samples (including those collected from $DEVNDK/platforms/) 517 - all documentation 518 519You need to have a directory containing prebuilt tarballs, as described in 520the previous section. You can use it as: 521 522 $NDK/build/tools/package-release.sh \ 523 --release=<name> \ 524 --systems=<list> \ 525 --arch=<list> \ 526 --prebuilt-dir=<path> 527 528The --release option is optional and allows you to provide a name for your 529generated NDK archive. More specifically, the archive file name will be 530something like android-ndk-$RELEASE-$SYSTEM.tar.bz2, where $RELEASE is 531the release name, and $SYSTEM the supported host system (e.g. linux-x86). 532 533By default, i.e. without the option, $RELEASE will be set to the current $DATE. 534 535The --systems=<list> is optional, but can be used to limit the number of host 536systems you want to generate for. <list> must be a comma-separated list of 537system names (from 'linux-x86', 'windows' and 'darwin-x86'). This is useful 538if you're working on a experimental feature and don't have the time to 539regenerate the host toolchains for all systems. It allows you to generate 540an experimental package that you can distribute to third-party for 541experimentation. 542 543By default, i.e. without the option, the scripts tries to build NDK archives 544for all supported host systems. 545 546The --arch=<list> is also optional, but can be used ot limit the number of 547target architectures you want to generate for. <list> must be a comma-separated 548list of CPU architectures (e.g. from 'arm' and 'x86'). Without the option, 549this will try to build packages that support all architectures. 550 551Finally, --prebuilt-dir=<path> must point to the directory that contains the 552prebuilt tarballs described in section V. Following our previous example, one 553could use --prebuilt-dir=/tmp/ndk-$USER/prebuilt here. 554 555 556VI. Testing: 557------------ 558 559The $NDK/tests directory contains a number of NDK unit-tests that can be 560used to verify that the generated NDK packages or the working NDK tree still 561behave correctly. 562 563If you have an NDK package archive, you can run the following to run the 564test suite against it: 565 566 $NDK/tests/run-tests.sh --package=<ndk-archive> 567 568This will uncompress the NDK archive in a temporary directory, then run all 569the tests with it. When all tests have run, the temporary directory is removed 570automatically. 571 572You can also point to an existing NDK installation with --ndk=<path>, as in: 573 574 $NDK/tests/run-tests.sh --ndk=<path> 575 576Where <path> points to another NDK installation. The script will run the 577test suite present under $NDK/tests/, not the one in the remote NDK directory. 578 579If you don't use any option, the test suite will be run with the current NDK 580directory. This can only work if you have generated or unpacked all prebuilt 581archives into it before that. 582 583You can get more traces from the tests by using --verbose. Use it twice to see 584even more traces. 585 586There are several kinds of tests: 587 588 - 'build tests' are used to test the building capabilities of the NDK. 589 I.e. the tests will only use them to check that the NDK build system 590 didn't regress. The corresponding generated binaries are never used 591 otherwise. 592 593 - 'device tests' are used to test both the build and the behaviour of 594 the generated code. If the 'adb' program is in your path, and have 595 one device or emulator connected to your host machine, 'run-tests.sh' 596 will automatically upload, run and cleanup these tests for you. 597 598 If adb is not in your path, or no device is connected, run-tests.sh 599 will simply print a warning and carry on. 600 601 602Whenever you add a feature to the NDK, or fix a bug, it is recommended to 603add a unit test to check the feature or the fix. Use $NDK/tests/build for 604build tests, and $NDK/tests/device for device tests. 605 606