1#!/usr/bin/env python3 2 3# Copyright 2017 The Glslang Authors. All rights reserved. 4# Copyright (c) 2018-2023 Valve Corporation 5# Copyright (c) 2018-2023 LunarG, Inc. 6# Copyright (c) 2023-2023 RasterGrid Kft. 7# 8# Licensed under the Apache License, Version 2.0 (the "License"); 9# you may not use this file except in compliance with the License. 10# You may obtain a copy of the License at 11# 12# http://www.apache.org/licenses/LICENSE-2.0 13# 14# Unless required by applicable law or agreed to in writing, software 15# distributed under the License is distributed on an "AS IS" BASIS, 16# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. 17# See the License for the specific language governing permissions and 18# limitations under the License. 19 20# This script was heavily leveraged from KhronosGroup/glslang 21# update_glslang_sources.py. 22"""update_deps.py 23 24Get and build dependent repositories using known-good commits. 25 26Purpose 27------- 28 29This program is intended to assist a developer of this repository 30(the "home" repository) by gathering and building the repositories that 31this home repository depend on. It also checks out each dependent 32repository at a "known-good" commit in order to provide stability in 33the dependent repositories. 34 35Known-Good JSON Database 36------------------------ 37 38This program expects to find a file named "known-good.json" in the 39same directory as the program file. This JSON file is tailored for 40the needs of the home repository by including its dependent repositories. 41 42Program Options 43--------------- 44 45See the help text (update_deps.py --help) for a complete list of options. 46 47Program Operation 48----------------- 49 50The program uses the user's current directory at the time of program 51invocation as the location for fetching and building the dependent 52repositories. The user can override this by using the "--dir" option. 53 54For example, a directory named "build" in the repository's root directory 55is a good place to put the dependent repositories because that directory 56is not tracked by Git. (See the .gitignore file.) The "external" directory 57may also be a suitable location. 58A user can issue: 59 60$ cd My-Repo 61$ mkdir build 62$ cd build 63$ ../scripts/update_deps.py 64 65or, to do the same thing, but using the --dir option: 66 67$ cd My-Repo 68$ mkdir build 69$ scripts/update_deps.py --dir=build 70 71With these commands, the "build" directory is considered the "top" 72directory where the program clones the dependent repositories. The 73JSON file configures the build and install working directories to be 74within this "top" directory. 75 76Note that the "dir" option can also specify an absolute path: 77 78$ cd My-Repo 79$ scripts/update_deps.py --dir=/tmp/deps 80 81The "top" dir is then /tmp/deps (Linux filesystem example) and is 82where this program will clone and build the dependent repositories. 83 84Helper CMake Config File 85------------------------ 86 87When the program finishes building the dependencies, it writes a file 88named "helper.cmake" to the "top" directory that contains CMake commands 89for setting CMake variables for locating the dependent repositories. 90This helper file can be used to set up the CMake build files for this 91"home" repository. 92 93A complete sequence might look like: 94 95$ git clone git@github.com:My-Group/My-Repo.git 96$ cd My-Repo 97$ mkdir build 98$ cd build 99$ ../scripts/update_deps.py 100$ cmake -C helper.cmake .. 101$ cmake --build . 102 103JSON File Schema 104---------------- 105 106There's no formal schema for the "known-good" JSON file, but here is 107a description of its elements. All elements are required except those 108marked as optional. Please see the "known_good.json" file for 109examples of all of these elements. 110 111- name 112 113The name of the dependent repository. This field can be referenced 114by the "deps.repo_name" structure to record a dependency. 115 116- api 117 118The name of the API the dependency is specific to (e.g. "vulkan"). 119 120- url 121 122Specifies the URL of the repository. 123Example: https://github.com/KhronosGroup/Vulkan-Loader.git 124 125- sub_dir 126 127The directory where the program clones the repository, relative to 128the "top" directory. 129 130- build_dir 131 132The directory used to build the repository, relative to the "top" 133directory. 134 135- install_dir 136 137The directory used to store the installed build artifacts, relative 138to the "top" directory. 139 140- commit 141 142The commit used to checkout the repository. This can be a SHA-1 143object name or a refname used with the remote name "origin". 144 145- deps (optional) 146 147An array of pairs consisting of a CMake variable name and a 148repository name to specify a dependent repo and a "link" to 149that repo's install artifacts. For example: 150 151"deps" : [ 152 { 153 "var_name" : "VULKAN_HEADERS_INSTALL_DIR", 154 "repo_name" : "Vulkan-Headers" 155 } 156] 157 158which represents that this repository depends on the Vulkan-Headers 159repository and uses the VULKAN_HEADERS_INSTALL_DIR CMake variable to 160specify the location where it expects to find the Vulkan-Headers install 161directory. 162Note that the "repo_name" element must match the "name" element of some 163other repository in the JSON file. 164 165- prebuild (optional) 166- prebuild_linux (optional) (For Linux and MacOS) 167- prebuild_windows (optional) 168 169A list of commands to execute before building a dependent repository. 170This is useful for repositories that require the execution of some 171sort of "update" script or need to clone an auxillary repository like 172googletest. 173 174The commands listed in "prebuild" are executed first, and then the 175commands for the specific platform are executed. 176 177- custom_build (optional) 178 179A list of commands to execute as a custom build instead of using 180the built in CMake way of building. Requires "build_step" to be 181set to "custom" 182 183You can insert the following keywords into the commands listed in 184"custom_build" if they require runtime information (like whether the 185build config is "Debug" or "Release"). 186 187Keywords: 188{0} reference to a dictionary of repos and their attributes 189{1} reference to the command line arguments set before start 190{2} reference to the CONFIG_MAP value of config. 191 192Example: 193{2} returns the CONFIG_MAP value of config e.g. debug -> Debug 194{1}.config returns the config variable set when you ran update_dep.py 195{0}[Vulkan-Headers][repo_root] returns the repo_root variable from 196 the Vulkan-Headers GoodRepo object. 197 198- cmake_options (optional) 199 200A list of options to pass to CMake during the generation phase. 201 202- ci_only (optional) 203 204A list of environment variables where one must be set to "true" 205(case-insensitive) in order for this repo to be fetched and built. 206This list can be used to specify repos that should be built only in CI. 207 208- build_step (optional) 209 210Specifies if the dependent repository should be built or not. This can 211have a value of 'build', 'custom', or 'skip'. The dependent repositories are 212built by default. 213 214- build_platforms (optional) 215 216A list of platforms the repository will be built on. 217Legal options include: 218"windows" 219"linux" 220"darwin" 221"android" 222 223Builds on all platforms by default. 224 225Note 226---- 227 228The "sub_dir", "build_dir", and "install_dir" elements are all relative 229to the effective "top" directory. Specifying absolute paths is not 230supported. However, the "top" directory specified with the "--dir" 231option can be a relative or absolute path. 232 233""" 234 235import argparse 236import json 237import os 238import os.path 239import subprocess 240import sys 241import platform 242import multiprocessing 243import shlex 244import shutil 245import stat 246import time 247 248KNOWN_GOOD_FILE_NAME = 'known_good.json' 249 250CONFIG_MAP = { 251 'debug': 'Debug', 252 'release': 'Release', 253 'relwithdebinfo': 'RelWithDebInfo', 254 'minsizerel': 'MinSizeRel' 255} 256 257# NOTE: CMake also uses the VERBOSE environment variable. This is intentional. 258VERBOSE = os.getenv("VERBOSE") 259 260DEVNULL = open(os.devnull, 'wb') 261 262 263def on_rm_error( func, path, exc_info): 264 """Error handler for recursively removing a directory. The 265 shutil.rmtree function can fail on Windows due to read-only files. 266 This handler will change the permissions for the file and continue. 267 """ 268 os.chmod( path, stat.S_IWRITE ) 269 os.unlink( path ) 270 271def make_or_exist_dirs(path): 272 "Wrapper for os.makedirs that tolerates the directory already existing" 273 # Could use os.makedirs(path, exist_ok=True) if we drop python2 274 if not os.path.isdir(path): 275 os.makedirs(path) 276 277def command_output(cmd, directory): 278 # Runs a command in a directory and returns its standard output stream. 279 # Captures the standard error stream and prints it an error occurs. 280 # Raises a RuntimeError if the command fails to launch or otherwise fails. 281 if VERBOSE: 282 print('In {d}: {cmd}'.format(d=directory, cmd=cmd)) 283 284 result = subprocess.run(cmd, cwd=directory, capture_output=True, text=True) 285 286 if result.returncode != 0: 287 print(f'{result.stderr}', file=sys.stderr) 288 raise RuntimeError(f'Failed to run {cmd} in {directory}') 289 290 if VERBOSE: 291 print(result.stdout) 292 return result.stdout 293 294def run_cmake_command(cmake_cmd): 295 # NOTE: Because CMake is an exectuable that runs executables 296 # stdout/stderr are mixed together. So this combines the outputs 297 # and prints them properly in case there is a non-zero exit code. 298 result = subprocess.run(cmake_cmd, 299 stdout = subprocess.PIPE, 300 stderr = subprocess.STDOUT, 301 text = True 302 ) 303 304 if VERBOSE: 305 print(result.stdout) 306 print(f"CMake command: {cmake_cmd} ", flush=True) 307 308 if result.returncode != 0: 309 print(result.stdout, file=sys.stderr) 310 sys.exit(result.returncode) 311 312def escape(path): 313 return path.replace('\\', '/') 314 315class GoodRepo(object): 316 """Represents a repository at a known-good commit.""" 317 318 def __init__(self, json, args): 319 """Initializes this good repo object. 320 321 Args: 322 'json': A fully populated JSON object describing the repo. 323 'args': Results from ArgumentParser 324 """ 325 self._json = json 326 self._args = args 327 # Required JSON elements 328 self.name = json['name'] 329 self.url = json['url'] 330 self.sub_dir = json['sub_dir'] 331 self.commit = json['commit'] 332 # Optional JSON elements 333 self.build_dir = None 334 self.install_dir = None 335 if json.get('build_dir'): 336 self.build_dir = os.path.normpath(json['build_dir']) 337 if json.get('install_dir'): 338 self.install_dir = os.path.normpath(json['install_dir']) 339 self.deps = json['deps'] if ('deps' in json) else [] 340 self.prebuild = json['prebuild'] if ('prebuild' in json) else [] 341 self.prebuild_linux = json['prebuild_linux'] if ( 342 'prebuild_linux' in json) else [] 343 self.prebuild_windows = json['prebuild_windows'] if ( 344 'prebuild_windows' in json) else [] 345 self.custom_build = json['custom_build'] if ('custom_build' in json) else [] 346 self.cmake_options = json['cmake_options'] if ( 347 'cmake_options' in json) else [] 348 self.ci_only = json['ci_only'] if ('ci_only' in json) else [] 349 self.build_step = json['build_step'] if ('build_step' in json) else 'build' 350 self.build_platforms = json['build_platforms'] if ('build_platforms' in json) else [] 351 self.optional = set(json.get('optional', [])) 352 self.api = json['api'] if ('api' in json) else None 353 # Absolute paths for a repo's directories 354 dir_top = os.path.abspath(args.dir) 355 self.repo_dir = os.path.join(dir_top, self.sub_dir) 356 if self.build_dir: 357 self.build_dir = os.path.join(dir_top, self.build_dir) 358 if self.install_dir: 359 self.install_dir = os.path.join(dir_top, self.install_dir) 360 361 # By default the target platform is the host platform. 362 target_platform = platform.system().lower() 363 # However, we need to account for cross-compiling. 364 for cmake_var in self._args.cmake_var: 365 if "android.toolchain.cmake" in cmake_var: 366 target_platform = 'android' 367 368 self.on_build_platform = False 369 if self.build_platforms == [] or target_platform in self.build_platforms: 370 self.on_build_platform = True 371 372 def Clone(self, retries=10, retry_seconds=60): 373 if VERBOSE: 374 print('Cloning {n} into {d}'.format(n=self.name, d=self.repo_dir)) 375 for retry in range(retries): 376 make_or_exist_dirs(self.repo_dir) 377 try: 378 command_output(['git', 'clone', self.url, '.'], self.repo_dir) 379 # If we get here, we didn't raise an error 380 return 381 except RuntimeError as e: 382 print("Error cloning on iteration {}/{}: {}".format(retry + 1, retries, e)) 383 if retry + 1 < retries: 384 if retry_seconds > 0: 385 print("Waiting {} seconds before trying again".format(retry_seconds)) 386 time.sleep(retry_seconds) 387 if os.path.isdir(self.repo_dir): 388 print("Removing old tree {}".format(self.repo_dir)) 389 shutil.rmtree(self.repo_dir, onerror=on_rm_error) 390 continue 391 392 # If we get here, we've exhausted our retries. 393 print("Failed to clone {} on all retries.".format(self.url)) 394 raise e 395 396 def Fetch(self, retries=10, retry_seconds=60): 397 for retry in range(retries): 398 try: 399 command_output(['git', 'fetch', 'origin'], self.repo_dir) 400 # if we get here, we didn't raise an error, and we're done 401 return 402 except RuntimeError as e: 403 print("Error fetching on iteration {}/{}: {}".format(retry + 1, retries, e)) 404 if retry + 1 < retries: 405 if retry_seconds > 0: 406 print("Waiting {} seconds before trying again".format(retry_seconds)) 407 time.sleep(retry_seconds) 408 continue 409 410 # If we get here, we've exhausted our retries. 411 print("Failed to fetch {} on all retries.".format(self.url)) 412 raise e 413 414 def Checkout(self): 415 if VERBOSE: 416 print('Checking out {n} in {d}'.format(n=self.name, d=self.repo_dir)) 417 418 if os.path.exists(os.path.join(self.repo_dir, '.git')): 419 url_changed = command_output(['git', 'config', '--get', 'remote.origin.url'], self.repo_dir).strip() != self.url 420 else: 421 url_changed = False 422 423 if self._args.do_clean_repo or url_changed: 424 if os.path.isdir(self.repo_dir): 425 if VERBOSE: 426 print('Clearing directory {d}'.format(d=self.repo_dir)) 427 shutil.rmtree(self.repo_dir, onerror = on_rm_error) 428 if not os.path.exists(os.path.join(self.repo_dir, '.git')): 429 self.Clone() 430 self.Fetch() 431 if len(self._args.ref): 432 command_output(['git', 'checkout', self._args.ref], self.repo_dir) 433 else: 434 command_output(['git', 'checkout', self.commit], self.repo_dir) 435 436 if VERBOSE: 437 print(command_output(['git', 'status'], self.repo_dir)) 438 439 def CustomPreProcess(self, cmd_str, repo_dict): 440 return cmd_str.format(repo_dict, self._args, CONFIG_MAP[self._args.config]) 441 442 def PreBuild(self): 443 """Execute any prebuild steps from the repo root""" 444 for p in self.prebuild: 445 command_output(shlex.split(p), self.repo_dir) 446 if platform.system() == 'Linux' or platform.system() == 'Darwin': 447 for p in self.prebuild_linux: 448 command_output(shlex.split(p), self.repo_dir) 449 if platform.system() == 'Windows': 450 for p in self.prebuild_windows: 451 command_output(shlex.split(p), self.repo_dir) 452 453 def CustomBuild(self, repo_dict): 454 """Execute any custom_build steps from the repo root""" 455 456 # It's not uncommon for builds to not support universal binaries 457 if self._args.OSX_ARCHITECTURES: 458 print("Universal Binaries not supported for custom builds", file=sys.stderr) 459 exit(-1) 460 461 for p in self.custom_build: 462 cmd = self.CustomPreProcess(p, repo_dict) 463 command_output(shlex.split(cmd), self.repo_dir) 464 465 def CMakeConfig(self, repos): 466 """Build CMake command for the configuration phase and execute it""" 467 if self._args.do_clean_build: 468 if os.path.isdir(self.build_dir): 469 shutil.rmtree(self.build_dir, onerror=on_rm_error) 470 if self._args.do_clean_install: 471 if os.path.isdir(self.install_dir): 472 shutil.rmtree(self.install_dir, onerror=on_rm_error) 473 474 # Create and change to build directory 475 make_or_exist_dirs(self.build_dir) 476 os.chdir(self.build_dir) 477 478 cmake_cmd = [ 479 'cmake', self.repo_dir, 480 '-DCMAKE_INSTALL_PREFIX=' + self.install_dir 481 ] 482 483 # Allow users to pass in arbitrary cache variables 484 for cmake_var in self._args.cmake_var: 485 pieces = cmake_var.split('=', 1) 486 cmake_cmd.append('-D{}={}'.format(pieces[0], pieces[1])) 487 488 # For each repo this repo depends on, generate a CMake variable 489 # definitions for "...INSTALL_DIR" that points to that dependent 490 # repo's install dir. 491 for d in self.deps: 492 dep_commit = [r for r in repos if r.name == d['repo_name']] 493 if len(dep_commit) and dep_commit[0].on_build_platform: 494 cmake_cmd.append('-D{var_name}={install_dir}'.format( 495 var_name=d['var_name'], 496 install_dir=dep_commit[0].install_dir)) 497 498 # Add any CMake options 499 for option in self.cmake_options: 500 cmake_cmd.append(escape(option.format(**self.__dict__))) 501 502 # Set build config for single-configuration generators (this is a no-op on multi-config generators) 503 cmake_cmd.append(f'-D CMAKE_BUILD_TYPE={CONFIG_MAP[self._args.config]}') 504 505 if self._args.OSX_ARCHITECTURES: 506 # CMAKE_OSX_ARCHITECTURES must be a semi-colon seperated list 507 cmake_osx_archs = self._args.OSX_ARCHITECTURES.replace(':', ';') 508 cmake_cmd.append(f'-D CMAKE_OSX_ARCHITECTURES={cmake_osx_archs}') 509 510 # Use the CMake -A option to select the platform architecture 511 # without needing a Visual Studio generator. 512 if platform.system() == 'Windows' and self._args.generator != "Ninja": 513 cmake_cmd.append('-A') 514 if self._args.arch.lower() == '64' or self._args.arch == 'x64' or self._args.arch == 'win64': 515 cmake_cmd.append('x64') 516 elif self._args.arch == 'arm64': 517 cmake_cmd.append('arm64') 518 elif self._args.arch == 'arm': 519 cmake_cmd.append('arm') 520 else: 521 cmake_cmd.append('Win32') 522 523 # Apply a generator, if one is specified. This can be used to supply 524 # a specific generator for the dependent repositories to match 525 # that of the main repository. 526 if self._args.generator is not None: 527 cmake_cmd.extend(['-G', self._args.generator]) 528 529 # Removes warnings related to unused CLI 530 # EX: Setting CMAKE_CXX_COMPILER for a C project 531 if not VERBOSE: 532 cmake_cmd.append("--no-warn-unused-cli") 533 534 run_cmake_command(cmake_cmd) 535 536 def CMakeBuild(self): 537 """Build CMake command for the build phase and execute it""" 538 cmake_cmd = ['cmake', '--build', self.build_dir, '--target', 'install', '--config', CONFIG_MAP[self._args.config]] 539 if self._args.do_clean: 540 cmake_cmd.append('--clean-first') 541 542 # Xcode / Ninja are parallel by default. 543 if self._args.generator != "Ninja" or self._args.generator != "Xcode": 544 cmake_cmd.append('--parallel') 545 cmake_cmd.append(format(multiprocessing.cpu_count())) 546 547 run_cmake_command(cmake_cmd) 548 549 def Build(self, repos, repo_dict): 550 """Build the dependent repo and time how long it took""" 551 if VERBOSE: 552 print('Building {n} in {d}'.format(n=self.name, d=self.repo_dir)) 553 print('Build dir = {b}'.format(b=self.build_dir)) 554 print('Install dir = {i}\n'.format(i=self.install_dir)) 555 556 start = time.time() 557 558 self.PreBuild() 559 560 if self.build_step == 'custom': 561 self.CustomBuild(repo_dict) 562 else: 563 self.CMakeConfig(repos) 564 self.CMakeBuild() 565 566 total_time = time.time() - start 567 568 print(f"Installed {self.name} ({self.commit}) in {total_time} seconds", flush=True) 569 570 def IsOptional(self, opts): 571 return len(self.optional.intersection(opts)) > 0 572 573def GetGoodRepos(args): 574 """Returns the latest list of GoodRepo objects. 575 576 The known-good file is expected to be in the same 577 directory as this script unless overridden by the 'known_good_dir' 578 parameter. 579 """ 580 if args.known_good_dir: 581 known_good_file = os.path.join( os.path.abspath(args.known_good_dir), 582 KNOWN_GOOD_FILE_NAME) 583 else: 584 known_good_file = os.path.join( 585 os.path.dirname(os.path.abspath(__file__)), KNOWN_GOOD_FILE_NAME) 586 with open(known_good_file) as known_good: 587 return [ 588 GoodRepo(repo, args) 589 for repo in json.loads(known_good.read())['repos'] 590 ] 591 592 593def GetInstallNames(args): 594 """Returns the install names list. 595 596 The known-good file is expected to be in the same 597 directory as this script unless overridden by the 'known_good_dir' 598 parameter. 599 """ 600 if args.known_good_dir: 601 known_good_file = os.path.join(os.path.abspath(args.known_good_dir), 602 KNOWN_GOOD_FILE_NAME) 603 else: 604 known_good_file = os.path.join( 605 os.path.dirname(os.path.abspath(__file__)), KNOWN_GOOD_FILE_NAME) 606 with open(known_good_file) as known_good: 607 install_info = json.loads(known_good.read()) 608 if install_info.get('install_names'): 609 return install_info['install_names'] 610 else: 611 return None 612 613 614def CreateHelper(args, repos, filename): 615 """Create a CMake config helper file. 616 617 The helper file is intended to be used with 'cmake -C <file>' 618 to build this home repo using the dependencies built by this script. 619 620 The install_names dictionary represents the CMake variables used by the 621 home repo to locate the install dirs of the dependent repos. 622 This information is baked into the CMake files of the home repo and so 623 this dictionary is kept with the repo via the json file. 624 """ 625 install_names = GetInstallNames(args) 626 with open(filename, 'w') as helper_file: 627 for repo in repos: 628 # If the repo has an API tag and that does not match 629 # the target API then skip it 630 if repo.api is not None and repo.api != args.api: 631 continue 632 if install_names and repo.name in install_names and repo.on_build_platform: 633 helper_file.write('set({var} "{dir}" CACHE STRING "")\n' 634 .format( 635 var=install_names[repo.name], 636 dir=escape(repo.install_dir))) 637 638 639def main(): 640 parser = argparse.ArgumentParser( 641 description='Get and build dependent repos at known-good commits') 642 parser.add_argument( 643 '--known_good_dir', 644 dest='known_good_dir', 645 help="Specify directory for known_good.json file.") 646 parser.add_argument( 647 '--dir', 648 dest='dir', 649 default='.', 650 help="Set target directory for repository roots. Default is \'.\'.") 651 parser.add_argument( 652 '--ref', 653 dest='ref', 654 default='', 655 help="Override 'commit' with git reference. E.g., 'origin/main'") 656 parser.add_argument( 657 '--no-build', 658 dest='do_build', 659 action='store_false', 660 help= 661 "Clone/update repositories and generate build files without performing compilation", 662 default=True) 663 parser.add_argument( 664 '--clean', 665 dest='do_clean', 666 action='store_true', 667 help="Clean files generated by compiler and linker before building", 668 default=False) 669 parser.add_argument( 670 '--clean-repo', 671 dest='do_clean_repo', 672 action='store_true', 673 help="Delete repository directory before building", 674 default=False) 675 parser.add_argument( 676 '--clean-build', 677 dest='do_clean_build', 678 action='store_true', 679 help="Delete build directory before building", 680 default=False) 681 parser.add_argument( 682 '--clean-install', 683 dest='do_clean_install', 684 action='store_true', 685 help="Delete install directory before building", 686 default=False) 687 parser.add_argument( 688 '--skip-existing-install', 689 dest='skip_existing_install', 690 action='store_true', 691 help="Skip build if install directory exists", 692 default=False) 693 parser.add_argument( 694 '--arch', 695 dest='arch', 696 choices=['32', '64', 'x86', 'x64', 'win32', 'win64', 'arm', 'arm64'], 697 type=str.lower, 698 help="Set build files architecture (Visual Studio Generator Only)", 699 default='64') 700 parser.add_argument( 701 '--config', 702 dest='config', 703 choices=['debug', 'release', 'relwithdebinfo', 'minsizerel'], 704 type=str.lower, 705 help="Set build files configuration", 706 default='debug') 707 parser.add_argument( 708 '--api', 709 dest='api', 710 default='vulkan', 711 choices=['vulkan'], 712 help="Target API") 713 parser.add_argument( 714 '--generator', 715 dest='generator', 716 help="Set the CMake generator", 717 default=None) 718 parser.add_argument( 719 '--optional', 720 dest='optional', 721 type=lambda a: set(a.lower().split(',')), 722 help="Comma-separated list of 'optional' resources that may be skipped. Only 'tests' is currently supported as 'optional'", 723 default=set()) 724 parser.add_argument( 725 '--cmake_var', 726 dest='cmake_var', 727 action='append', 728 metavar='VAR[=VALUE]', 729 help="Add CMake command line option -D'VAR'='VALUE' to the CMake generation command line; may be used multiple times", 730 default=[]) 731 parser.add_argument( 732 '--osx-archs', 733 dest='OSX_ARCHITECTURES', 734 help="Architectures when building a universal binary. Takes a colon seperated list. Ex: arm64:x86_64", 735 type=str, 736 default=None) 737 738 args = parser.parse_args() 739 save_cwd = os.getcwd() 740 741 if args.OSX_ARCHITECTURES: 742 print(f"Building dependencies as universal binaries targeting {args.OSX_ARCHITECTURES}") 743 744 # Create working "top" directory if needed 745 make_or_exist_dirs(args.dir) 746 abs_top_dir = os.path.abspath(args.dir) 747 748 repos = GetGoodRepos(args) 749 repo_dict = {} 750 751 print('Starting builds in {d}'.format(d=abs_top_dir)) 752 for repo in repos: 753 # If the repo has an API tag and that does not match 754 # the target API then skip it 755 if repo.api is not None and repo.api != args.api: 756 continue 757 758 # If the repo has a platform whitelist, skip the repo 759 # unless we are building on a whitelisted platform. 760 if not repo.on_build_platform: 761 continue 762 763 # Skip building the repo if its install directory already exists 764 # and requested via an option. This is useful for cases where the 765 # install directory is restored from a cache that is known to be up 766 # to date. 767 if args.skip_existing_install and os.path.isdir(repo.install_dir): 768 print('Skipping build for repo {n} due to existing install directory'.format(n=repo.name)) 769 continue 770 771 # Skip test-only repos if the --tests option was not passed in 772 if repo.IsOptional(args.optional): 773 continue 774 775 field_list = ('url', 776 'sub_dir', 777 'commit', 778 'build_dir', 779 'install_dir', 780 'deps', 781 'prebuild', 782 'prebuild_linux', 783 'prebuild_windows', 784 'custom_build', 785 'cmake_options', 786 'ci_only', 787 'build_step', 788 'build_platforms', 789 'repo_dir', 790 'on_build_platform') 791 repo_dict[repo.name] = {field: getattr(repo, field) for field in field_list} 792 793 # If the repo has a CI whitelist, skip the repo unless 794 # one of the CI's environment variable is set to true. 795 if len(repo.ci_only): 796 do_build = False 797 for env in repo.ci_only: 798 if env not in os.environ: 799 continue 800 if os.environ[env].lower() == 'true': 801 do_build = True 802 break 803 if not do_build: 804 continue 805 806 # Clone/update the repository 807 repo.Checkout() 808 809 # Build the repository 810 if args.do_build and repo.build_step != 'skip': 811 repo.Build(repos, repo_dict) 812 813 # Need to restore original cwd in order for CreateHelper to find json file 814 os.chdir(save_cwd) 815 CreateHelper(args, repos, os.path.join(abs_top_dir, 'helper.cmake')) 816 817 sys.exit(0) 818 819 820if __name__ == '__main__': 821 main() 822 823