1#!/usr/bin/env python3 2# SPDX-License-Identifier: GPL-2.0+ 3# 4# Author: Masahiro Yamada <yamada.masahiro@socionext.com> 5# 6 7""" 8Move config options from headers to defconfig files. 9 10Since Kconfig was introduced to U-Boot, we have worked on moving 11config options from headers to Kconfig (defconfig). 12 13This tool intends to help this tremendous work. 14 15 16Usage 17----- 18 19First, you must edit the Kconfig to add the menu entries for the configs 20you are moving. 21 22And then run this tool giving CONFIG names you want to move. 23For example, if you want to move CONFIG_CMD_USB and CONFIG_SYS_TEXT_BASE, 24simply type as follows: 25 26 $ tools/moveconfig.py CONFIG_CMD_USB CONFIG_SYS_TEXT_BASE 27 28The tool walks through all the defconfig files and move the given CONFIGs. 29 30The log is also displayed on the terminal. 31 32The log is printed for each defconfig as follows: 33 34<defconfig_name> 35 <action1> 36 <action2> 37 <action3> 38 ... 39 40<defconfig_name> is the name of the defconfig. 41 42<action*> shows what the tool did for that defconfig. 43It looks like one of the following: 44 45 - Move 'CONFIG_... ' 46 This config option was moved to the defconfig 47 48 - CONFIG_... is not defined in Kconfig. Do nothing. 49 The entry for this CONFIG was not found in Kconfig. The option is not 50 defined in the config header, either. So, this case can be just skipped. 51 52 - CONFIG_... is not defined in Kconfig (suspicious). Do nothing. 53 This option is defined in the config header, but its entry was not found 54 in Kconfig. 55 There are two common cases: 56 - You forgot to create an entry for the CONFIG before running 57 this tool, or made a typo in a CONFIG passed to this tool. 58 - The entry was hidden due to unmet 'depends on'. 59 The tool does not know if the result is reasonable, so please check it 60 manually. 61 62 - 'CONFIG_...' is the same as the define in Kconfig. Do nothing. 63 The define in the config header matched the one in Kconfig. 64 We do not need to touch it. 65 66 - Compiler is missing. Do nothing. 67 The compiler specified for this architecture was not found 68 in your PATH environment. 69 (If -e option is passed, the tool exits immediately.) 70 71 - Failed to process. 72 An error occurred during processing this defconfig. Skipped. 73 (If -e option is passed, the tool exits immediately on error.) 74 75Finally, you will be asked, Clean up headers? [y/n]: 76 77If you say 'y' here, the unnecessary config defines are removed 78from the config headers (include/configs/*.h). 79It just uses the regex method, so you should not rely on it. 80Just in case, please do 'git diff' to see what happened. 81 82 83How does it work? 84----------------- 85 86This tool runs configuration and builds include/autoconf.mk for every 87defconfig. The config options defined in Kconfig appear in the .config 88file (unless they are hidden because of unmet dependency.) 89On the other hand, the config options defined by board headers are seen 90in include/autoconf.mk. The tool looks for the specified options in both 91of them to decide the appropriate action for the options. If the given 92config option is found in the .config, but its value does not match the 93one from the board header, the config option in the .config is replaced 94with the define in the board header. Then, the .config is synced by 95"make savedefconfig" and the defconfig is updated with it. 96 97For faster processing, this tool handles multi-threading. It creates 98separate build directories where the out-of-tree build is run. The 99temporary build directories are automatically created and deleted as 100needed. The number of threads are chosen based on the number of the CPU 101cores of your system although you can change it via -j (--jobs) option. 102 103 104Toolchains 105---------- 106 107Appropriate toolchain are necessary to generate include/autoconf.mk 108for all the architectures supported by U-Boot. Most of them are available 109at the kernel.org site, some are not provided by kernel.org. This tool uses 110the same tools as buildman, so see that tool for setup (e.g. --fetch-arch). 111 112 113Tips and trips 114-------------- 115 116To sync only X86 defconfigs: 117 118 ./tools/moveconfig.py -s -d <(grep -l X86 configs/*) 119 120or: 121 122 grep -l X86 configs/* | ./tools/moveconfig.py -s -d - 123 124To process CONFIG_CMD_FPGAD only for a subset of configs based on path match: 125 126 ls configs/{hrcon*,iocon*,strider*} | \ 127 ./tools/moveconfig.py -Cy CONFIG_CMD_FPGAD -d - 128 129 130Finding implied CONFIGs 131----------------------- 132 133Some CONFIG options can be implied by others and this can help to reduce 134the size of the defconfig files. For example, CONFIG_X86 implies 135CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and 136all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to 137each of the x86 defconfig files. 138 139This tool can help find such configs. To use it, first build a database: 140 141 ./tools/moveconfig.py -b 142 143Then try to query it: 144 145 ./tools/moveconfig.py -i CONFIG_CMD_IRQ 146 CONFIG_CMD_IRQ found in 311/2384 defconfigs 147 44 : CONFIG_SYS_FSL_ERRATUM_IFC_A002769 148 41 : CONFIG_SYS_FSL_ERRATUM_A007075 149 31 : CONFIG_SYS_FSL_DDR_VER_44 150 28 : CONFIG_ARCH_P1010 151 28 : CONFIG_SYS_FSL_ERRATUM_P1010_A003549 152 28 : CONFIG_SYS_FSL_ERRATUM_SEC_A003571 153 28 : CONFIG_SYS_FSL_ERRATUM_IFC_A003399 154 25 : CONFIG_SYS_FSL_ERRATUM_A008044 155 22 : CONFIG_ARCH_P1020 156 21 : CONFIG_SYS_FSL_DDR_VER_46 157 20 : CONFIG_MAX_PIRQ_LINKS 158 20 : CONFIG_HPET_ADDRESS 159 20 : CONFIG_X86 160 20 : CONFIG_PCIE_ECAM_SIZE 161 20 : CONFIG_IRQ_SLOT_COUNT 162 20 : CONFIG_I8259_PIC 163 20 : CONFIG_CPU_ADDR_BITS 164 20 : CONFIG_RAMBASE 165 20 : CONFIG_SYS_FSL_ERRATUM_A005871 166 20 : CONFIG_PCIE_ECAM_BASE 167 20 : CONFIG_X86_TSC_TIMER 168 20 : CONFIG_I8254_TIMER 169 20 : CONFIG_CMD_GETTIME 170 19 : CONFIG_SYS_FSL_ERRATUM_A005812 171 18 : CONFIG_X86_RUN_32BIT 172 17 : CONFIG_CMD_CHIP_CONFIG 173 ... 174 175This shows a list of config options which might imply CONFIG_CMD_EEPROM along 176with how many defconfigs they cover. From this you can see that CONFIG_X86 177implies CONFIG_CMD_EEPROM. Therefore, instead of adding CONFIG_CMD_EEPROM to 178the defconfig of every x86 board, you could add a single imply line to the 179Kconfig file: 180 181 config X86 182 bool "x86 architecture" 183 ... 184 imply CMD_EEPROM 185 186That will cover 20 defconfigs. Many of the options listed are not suitable as 187they are not related. E.g. it would be odd for CONFIG_CMD_GETTIME to imply 188CMD_EEPROM. 189 190Using this search you can reduce the size of moveconfig patches. 191 192You can automatically add 'imply' statements in the Kconfig with the -a 193option: 194 195 ./tools/moveconfig.py -s -i CONFIG_SCSI \ 196 -a CONFIG_ARCH_LS1021A,CONFIG_ARCH_LS1043A 197 198This will add 'imply SCSI' to the two CONFIG options mentioned, assuming that 199the database indicates that they do actually imply CONFIG_SCSI and do not 200already have an 'imply SCSI'. 201 202The output shows where the imply is added: 203 204 18 : CONFIG_ARCH_LS1021A arch/arm/cpu/armv7/ls102xa/Kconfig:1 205 13 : CONFIG_ARCH_LS1043A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:11 206 12 : CONFIG_ARCH_LS1046A arch/arm/cpu/armv8/fsl-layerscape/Kconfig:31 207 208The first number is the number of boards which can avoid having a special 209CONFIG_SCSI option in their defconfig file if this 'imply' is added. 210The location at the right is the Kconfig file and line number where the config 211appears. For example, adding 'imply CONFIG_SCSI' to the 'config ARCH_LS1021A' 212in arch/arm/cpu/armv7/ls102xa/Kconfig at line 1 will help 18 boards to reduce 213the size of their defconfig files. 214 215If you want to add an 'imply' to every imply config in the list, you can use 216 217 ./tools/moveconfig.py -s -i CONFIG_SCSI -a all 218 219To control which ones are displayed, use -I <list> where list is a list of 220options (use '-I help' to see possible options and their meaning). 221 222To skip showing you options that already have an 'imply' attached, use -A. 223 224When you have finished adding 'imply' options you can regenerate the 225defconfig files for affected boards with something like: 226 227 git show --stat | ./tools/moveconfig.py -s -d - 228 229This will regenerate only those defconfigs changed in the current commit. 230If you start with (say) 100 defconfigs being changed in the commit, and add 231a few 'imply' options as above, then regenerate, hopefully you can reduce the 232number of defconfigs changed in the commit. 233 234 235Available options 236----------------- 237 238 -c, --color 239 Surround each portion of the log with escape sequences to display it 240 in color on the terminal. 241 242 -C, --commit 243 Create a git commit with the changes when the operation is complete. A 244 standard commit message is used which may need to be edited. 245 246 -d, --defconfigs 247 Specify a file containing a list of defconfigs to move. The defconfig 248 files can be given with shell-style wildcards. Use '-' to read from stdin. 249 250 -n, --dry-run 251 Perform a trial run that does not make any changes. It is useful to 252 see what is going to happen before one actually runs it. 253 254 -e, --exit-on-error 255 Exit immediately if Make exits with a non-zero status while processing 256 a defconfig file. 257 258 -s, --force-sync 259 Do "make savedefconfig" forcibly for all the defconfig files. 260 If not specified, "make savedefconfig" only occurs for cases 261 where at least one CONFIG was moved. 262 263 -S, --spl 264 Look for moved config options in spl/include/autoconf.mk instead of 265 include/autoconf.mk. This is useful for moving options for SPL build 266 because SPL related options (mostly prefixed with CONFIG_SPL_) are 267 sometimes blocked by CONFIG_SPL_BUILD ifdef conditionals. 268 269 -H, --headers-only 270 Only cleanup the headers; skip the defconfig processing 271 272 -j, --jobs 273 Specify the number of threads to run simultaneously. If not specified, 274 the number of threads is the same as the number of CPU cores. 275 276 -r, --git-ref 277 Specify the git ref to clone for building the autoconf.mk. If unspecified 278 use the CWD. This is useful for when changes to the Kconfig affect the 279 default values and you want to capture the state of the defconfig from 280 before that change was in effect. If in doubt, specify a ref pre-Kconfig 281 changes (use HEAD if Kconfig changes are not committed). Worst case it will 282 take a bit longer to run, but will always do the right thing. 283 284 -v, --verbose 285 Show any build errors as boards are built 286 287 -y, --yes 288 Instead of prompting, automatically go ahead with all operations. This 289 includes cleaning up headers, CONFIG_SYS_EXTRA_OPTIONS, the config whitelist 290 and the README. 291 292To see the complete list of supported options, run 293 294 $ tools/moveconfig.py -h 295 296""" 297 298import collections 299import copy 300import difflib 301import filecmp 302import fnmatch 303import glob 304import multiprocessing 305import optparse 306import os 307import queue 308import re 309import shutil 310import subprocess 311import sys 312import tempfile 313import threading 314import time 315 316sys.path.append(os.path.join(os.path.dirname(__file__), 'buildman')) 317sys.path.append(os.path.join(os.path.dirname(__file__), 'patman')) 318import bsettings 319import kconfiglib 320import toolchain 321 322SHOW_GNU_MAKE = 'scripts/show-gnu-make' 323SLEEP_TIME=0.03 324 325STATE_IDLE = 0 326STATE_DEFCONFIG = 1 327STATE_AUTOCONF = 2 328STATE_SAVEDEFCONFIG = 3 329 330ACTION_MOVE = 0 331ACTION_NO_ENTRY = 1 332ACTION_NO_ENTRY_WARN = 2 333ACTION_NO_CHANGE = 3 334 335COLOR_BLACK = '0;30' 336COLOR_RED = '0;31' 337COLOR_GREEN = '0;32' 338COLOR_BROWN = '0;33' 339COLOR_BLUE = '0;34' 340COLOR_PURPLE = '0;35' 341COLOR_CYAN = '0;36' 342COLOR_LIGHT_GRAY = '0;37' 343COLOR_DARK_GRAY = '1;30' 344COLOR_LIGHT_RED = '1;31' 345COLOR_LIGHT_GREEN = '1;32' 346COLOR_YELLOW = '1;33' 347COLOR_LIGHT_BLUE = '1;34' 348COLOR_LIGHT_PURPLE = '1;35' 349COLOR_LIGHT_CYAN = '1;36' 350COLOR_WHITE = '1;37' 351 352AUTO_CONF_PATH = 'include/config/auto.conf' 353CONFIG_DATABASE = 'moveconfig.db' 354 355CONFIG_LEN = len('CONFIG_') 356 357SIZES = { 358 "SZ_1": 0x00000001, "SZ_2": 0x00000002, 359 "SZ_4": 0x00000004, "SZ_8": 0x00000008, 360 "SZ_16": 0x00000010, "SZ_32": 0x00000020, 361 "SZ_64": 0x00000040, "SZ_128": 0x00000080, 362 "SZ_256": 0x00000100, "SZ_512": 0x00000200, 363 "SZ_1K": 0x00000400, "SZ_2K": 0x00000800, 364 "SZ_4K": 0x00001000, "SZ_8K": 0x00002000, 365 "SZ_16K": 0x00004000, "SZ_32K": 0x00008000, 366 "SZ_64K": 0x00010000, "SZ_128K": 0x00020000, 367 "SZ_256K": 0x00040000, "SZ_512K": 0x00080000, 368 "SZ_1M": 0x00100000, "SZ_2M": 0x00200000, 369 "SZ_4M": 0x00400000, "SZ_8M": 0x00800000, 370 "SZ_16M": 0x01000000, "SZ_32M": 0x02000000, 371 "SZ_64M": 0x04000000, "SZ_128M": 0x08000000, 372 "SZ_256M": 0x10000000, "SZ_512M": 0x20000000, 373 "SZ_1G": 0x40000000, "SZ_2G": 0x80000000, 374 "SZ_4G": 0x100000000 375} 376 377### helper functions ### 378def get_devnull(): 379 """Get the file object of '/dev/null' device.""" 380 try: 381 devnull = subprocess.DEVNULL # py3k 382 except AttributeError: 383 devnull = open(os.devnull, 'wb') 384 return devnull 385 386def check_top_directory(): 387 """Exit if we are not at the top of source directory.""" 388 for f in ('README', 'Licenses'): 389 if not os.path.exists(f): 390 sys.exit('Please run at the top of source directory.') 391 392def check_clean_directory(): 393 """Exit if the source tree is not clean.""" 394 for f in ('.config', 'include/config'): 395 if os.path.exists(f): 396 sys.exit("source tree is not clean, please run 'make mrproper'") 397 398def get_make_cmd(): 399 """Get the command name of GNU Make. 400 401 U-Boot needs GNU Make for building, but the command name is not 402 necessarily "make". (for example, "gmake" on FreeBSD). 403 Returns the most appropriate command name on your system. 404 """ 405 process = subprocess.Popen([SHOW_GNU_MAKE], stdout=subprocess.PIPE) 406 ret = process.communicate() 407 if process.returncode: 408 sys.exit('GNU Make not found') 409 return ret[0].rstrip() 410 411def get_matched_defconfig(line): 412 """Get the defconfig files that match a pattern 413 414 Args: 415 line: Path or filename to match, e.g. 'configs/snow_defconfig' or 416 'k2*_defconfig'. If no directory is provided, 'configs/' is 417 prepended 418 419 Returns: 420 a list of matching defconfig files 421 """ 422 dirname = os.path.dirname(line) 423 if dirname: 424 pattern = line 425 else: 426 pattern = os.path.join('configs', line) 427 return glob.glob(pattern) + glob.glob(pattern + '_defconfig') 428 429def get_matched_defconfigs(defconfigs_file): 430 """Get all the defconfig files that match the patterns in a file. 431 432 Args: 433 defconfigs_file: File containing a list of defconfigs to process, or 434 '-' to read the list from stdin 435 436 Returns: 437 A list of paths to defconfig files, with no duplicates 438 """ 439 defconfigs = [] 440 if defconfigs_file == '-': 441 fd = sys.stdin 442 defconfigs_file = 'stdin' 443 else: 444 fd = open(defconfigs_file) 445 for i, line in enumerate(fd): 446 line = line.strip() 447 if not line: 448 continue # skip blank lines silently 449 if ' ' in line: 450 line = line.split(' ')[0] # handle 'git log' input 451 matched = get_matched_defconfig(line) 452 if not matched: 453 print("warning: %s:%d: no defconfig matched '%s'" % \ 454 (defconfigs_file, i + 1, line), file=sys.stderr) 455 456 defconfigs += matched 457 458 # use set() to drop multiple matching 459 return [ defconfig[len('configs') + 1:] for defconfig in set(defconfigs) ] 460 461def get_all_defconfigs(): 462 """Get all the defconfig files under the configs/ directory.""" 463 defconfigs = [] 464 for (dirpath, dirnames, filenames) in os.walk('configs'): 465 dirpath = dirpath[len('configs') + 1:] 466 for filename in fnmatch.filter(filenames, '*_defconfig'): 467 defconfigs.append(os.path.join(dirpath, filename)) 468 469 return defconfigs 470 471def color_text(color_enabled, color, string): 472 """Return colored string.""" 473 if color_enabled: 474 # LF should not be surrounded by the escape sequence. 475 # Otherwise, additional whitespace or line-feed might be printed. 476 return '\n'.join([ '\033[' + color + 'm' + s + '\033[0m' if s else '' 477 for s in string.split('\n') ]) 478 else: 479 return string 480 481def show_diff(a, b, file_path, color_enabled): 482 """Show unidified diff. 483 484 Arguments: 485 a: A list of lines (before) 486 b: A list of lines (after) 487 file_path: Path to the file 488 color_enabled: Display the diff in color 489 """ 490 491 diff = difflib.unified_diff(a, b, 492 fromfile=os.path.join('a', file_path), 493 tofile=os.path.join('b', file_path)) 494 495 for line in diff: 496 if line[0] == '-' and line[1] != '-': 497 print(color_text(color_enabled, COLOR_RED, line), end=' ') 498 elif line[0] == '+' and line[1] != '+': 499 print(color_text(color_enabled, COLOR_GREEN, line), end=' ') 500 else: 501 print(line, end=' ') 502 503def extend_matched_lines(lines, matched, pre_patterns, post_patterns, extend_pre, 504 extend_post): 505 """Extend matched lines if desired patterns are found before/after already 506 matched lines. 507 508 Arguments: 509 lines: A list of lines handled. 510 matched: A list of line numbers that have been already matched. 511 (will be updated by this function) 512 pre_patterns: A list of regular expression that should be matched as 513 preamble. 514 post_patterns: A list of regular expression that should be matched as 515 postamble. 516 extend_pre: Add the line number of matched preamble to the matched list. 517 extend_post: Add the line number of matched postamble to the matched list. 518 """ 519 extended_matched = [] 520 521 j = matched[0] 522 523 for i in matched: 524 if i == 0 or i < j: 525 continue 526 j = i 527 while j in matched: 528 j += 1 529 if j >= len(lines): 530 break 531 532 for p in pre_patterns: 533 if p.search(lines[i - 1]): 534 break 535 else: 536 # not matched 537 continue 538 539 for p in post_patterns: 540 if p.search(lines[j]): 541 break 542 else: 543 # not matched 544 continue 545 546 if extend_pre: 547 extended_matched.append(i - 1) 548 if extend_post: 549 extended_matched.append(j) 550 551 matched += extended_matched 552 matched.sort() 553 554def confirm(options, prompt): 555 if not options.yes: 556 while True: 557 choice = input('{} [y/n]: '.format(prompt)) 558 choice = choice.lower() 559 print(choice) 560 if choice == 'y' or choice == 'n': 561 break 562 563 if choice == 'n': 564 return False 565 566 return True 567 568def cleanup_empty_blocks(header_path, options): 569 """Clean up empty conditional blocks 570 571 Arguments: 572 header_path: path to the cleaned file. 573 options: option flags. 574 """ 575 pattern = re.compile(r'^\s*#\s*if.*$\n^\s*#\s*endif.*$\n*', flags=re.M) 576 with open(header_path) as f: 577 data = f.read() 578 579 new_data = pattern.sub('\n', data) 580 581 show_diff(data.splitlines(True), new_data.splitlines(True), header_path, 582 options.color) 583 584 if options.dry_run: 585 return 586 587 with open(header_path, 'w') as f: 588 f.write(new_data) 589 590def cleanup_one_header(header_path, patterns, options): 591 """Clean regex-matched lines away from a file. 592 593 Arguments: 594 header_path: path to the cleaned file. 595 patterns: list of regex patterns. Any lines matching to these 596 patterns are deleted. 597 options: option flags. 598 """ 599 with open(header_path) as f: 600 lines = f.readlines() 601 602 matched = [] 603 for i, line in enumerate(lines): 604 if i - 1 in matched and lines[i - 1][-2:] == '\\\n': 605 matched.append(i) 606 continue 607 for pattern in patterns: 608 if pattern.search(line): 609 matched.append(i) 610 break 611 612 if not matched: 613 return 614 615 # remove empty #ifdef ... #endif, successive blank lines 616 pattern_if = re.compile(r'#\s*if(def|ndef)?\W') # #if, #ifdef, #ifndef 617 pattern_elif = re.compile(r'#\s*el(if|se)\W') # #elif, #else 618 pattern_endif = re.compile(r'#\s*endif\W') # #endif 619 pattern_blank = re.compile(r'^\s*$') # empty line 620 621 while True: 622 old_matched = copy.copy(matched) 623 extend_matched_lines(lines, matched, [pattern_if], 624 [pattern_endif], True, True) 625 extend_matched_lines(lines, matched, [pattern_elif], 626 [pattern_elif, pattern_endif], True, False) 627 extend_matched_lines(lines, matched, [pattern_if, pattern_elif], 628 [pattern_blank], False, True) 629 extend_matched_lines(lines, matched, [pattern_blank], 630 [pattern_elif, pattern_endif], True, False) 631 extend_matched_lines(lines, matched, [pattern_blank], 632 [pattern_blank], True, False) 633 if matched == old_matched: 634 break 635 636 tolines = copy.copy(lines) 637 638 for i in reversed(matched): 639 tolines.pop(i) 640 641 show_diff(lines, tolines, header_path, options.color) 642 643 if options.dry_run: 644 return 645 646 with open(header_path, 'w') as f: 647 for line in tolines: 648 f.write(line) 649 650def cleanup_headers(configs, options): 651 """Delete config defines from board headers. 652 653 Arguments: 654 configs: A list of CONFIGs to remove. 655 options: option flags. 656 """ 657 if not confirm(options, 'Clean up headers?'): 658 return 659 660 patterns = [] 661 for config in configs: 662 patterns.append(re.compile(r'#\s*define\s+%s\W' % config)) 663 patterns.append(re.compile(r'#\s*undef\s+%s\W' % config)) 664 665 for dir in 'include', 'arch', 'board': 666 for (dirpath, dirnames, filenames) in os.walk(dir): 667 if dirpath == os.path.join('include', 'generated'): 668 continue 669 for filename in filenames: 670 if not filename.endswith(('~', '.dts', '.dtsi')): 671 header_path = os.path.join(dirpath, filename) 672 # This file contains UTF-16 data and no CONFIG symbols 673 if header_path == 'include/video_font_data.h': 674 continue 675 cleanup_one_header(header_path, patterns, options) 676 cleanup_empty_blocks(header_path, options) 677 678def cleanup_one_extra_option(defconfig_path, configs, options): 679 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in one defconfig file. 680 681 Arguments: 682 defconfig_path: path to the cleaned defconfig file. 683 configs: A list of CONFIGs to remove. 684 options: option flags. 685 """ 686 687 start = 'CONFIG_SYS_EXTRA_OPTIONS="' 688 end = '"\n' 689 690 with open(defconfig_path) as f: 691 lines = f.readlines() 692 693 for i, line in enumerate(lines): 694 if line.startswith(start) and line.endswith(end): 695 break 696 else: 697 # CONFIG_SYS_EXTRA_OPTIONS was not found in this defconfig 698 return 699 700 old_tokens = line[len(start):-len(end)].split(',') 701 new_tokens = [] 702 703 for token in old_tokens: 704 pos = token.find('=') 705 if not (token[:pos] if pos >= 0 else token) in configs: 706 new_tokens.append(token) 707 708 if new_tokens == old_tokens: 709 return 710 711 tolines = copy.copy(lines) 712 713 if new_tokens: 714 tolines[i] = start + ','.join(new_tokens) + end 715 else: 716 tolines.pop(i) 717 718 show_diff(lines, tolines, defconfig_path, options.color) 719 720 if options.dry_run: 721 return 722 723 with open(defconfig_path, 'w') as f: 724 for line in tolines: 725 f.write(line) 726 727def cleanup_extra_options(configs, options): 728 """Delete config defines in CONFIG_SYS_EXTRA_OPTIONS in defconfig files. 729 730 Arguments: 731 configs: A list of CONFIGs to remove. 732 options: option flags. 733 """ 734 if not confirm(options, 'Clean up CONFIG_SYS_EXTRA_OPTIONS?'): 735 return 736 737 configs = [ config[len('CONFIG_'):] for config in configs ] 738 739 defconfigs = get_all_defconfigs() 740 741 for defconfig in defconfigs: 742 cleanup_one_extra_option(os.path.join('configs', defconfig), configs, 743 options) 744 745def cleanup_whitelist(configs, options): 746 """Delete config whitelist entries 747 748 Arguments: 749 configs: A list of CONFIGs to remove. 750 options: option flags. 751 """ 752 if not confirm(options, 'Clean up whitelist entries?'): 753 return 754 755 with open(os.path.join('scripts', 'config_whitelist.txt')) as f: 756 lines = f.readlines() 757 758 lines = [x for x in lines if x.strip() not in configs] 759 760 with open(os.path.join('scripts', 'config_whitelist.txt'), 'w') as f: 761 f.write(''.join(lines)) 762 763def find_matching(patterns, line): 764 for pat in patterns: 765 if pat.search(line): 766 return True 767 return False 768 769def cleanup_readme(configs, options): 770 """Delete config description in README 771 772 Arguments: 773 configs: A list of CONFIGs to remove. 774 options: option flags. 775 """ 776 if not confirm(options, 'Clean up README?'): 777 return 778 779 patterns = [] 780 for config in configs: 781 patterns.append(re.compile(r'^\s+%s' % config)) 782 783 with open('README') as f: 784 lines = f.readlines() 785 786 found = False 787 newlines = [] 788 for line in lines: 789 if not found: 790 found = find_matching(patterns, line) 791 if found: 792 continue 793 794 if found and re.search(r'^\s+CONFIG', line): 795 found = False 796 797 if not found: 798 newlines.append(line) 799 800 with open('README', 'w') as f: 801 f.write(''.join(newlines)) 802 803def try_expand(line): 804 """If value looks like an expression, try expanding it 805 Otherwise just return the existing value 806 """ 807 if line.find('=') == -1: 808 return line 809 810 try: 811 cfg, val = re.split("=", line) 812 val= val.strip('\"') 813 if re.search("[*+-/]|<<|SZ_+|\(([^\)]+)\)", val): 814 newval = hex(eval(val, SIZES)) 815 print("\tExpanded expression %s to %s" % (val, newval)) 816 return cfg+'='+newval 817 except: 818 print("\tFailed to expand expression in %s" % line) 819 820 return line 821 822 823### classes ### 824class Progress: 825 826 """Progress Indicator""" 827 828 def __init__(self, total): 829 """Create a new progress indicator. 830 831 Arguments: 832 total: A number of defconfig files to process. 833 """ 834 self.current = 0 835 self.total = total 836 837 def inc(self): 838 """Increment the number of processed defconfig files.""" 839 840 self.current += 1 841 842 def show(self): 843 """Display the progress.""" 844 print(' %d defconfigs out of %d\r' % (self.current, self.total), end=' ') 845 sys.stdout.flush() 846 847 848class KconfigScanner: 849 """Kconfig scanner.""" 850 851 def __init__(self): 852 """Scan all the Kconfig files and create a Config object.""" 853 # Define environment variables referenced from Kconfig 854 os.environ['srctree'] = os.getcwd() 855 os.environ['UBOOTVERSION'] = 'dummy' 856 os.environ['KCONFIG_OBJDIR'] = '' 857 self.conf = kconfiglib.Kconfig() 858 859 860class KconfigParser: 861 862 """A parser of .config and include/autoconf.mk.""" 863 864 re_arch = re.compile(r'CONFIG_SYS_ARCH="(.*)"') 865 re_cpu = re.compile(r'CONFIG_SYS_CPU="(.*)"') 866 867 def __init__(self, configs, options, build_dir): 868 """Create a new parser. 869 870 Arguments: 871 configs: A list of CONFIGs to move. 872 options: option flags. 873 build_dir: Build directory. 874 """ 875 self.configs = configs 876 self.options = options 877 self.dotconfig = os.path.join(build_dir, '.config') 878 self.autoconf = os.path.join(build_dir, 'include', 'autoconf.mk') 879 self.spl_autoconf = os.path.join(build_dir, 'spl', 'include', 880 'autoconf.mk') 881 self.config_autoconf = os.path.join(build_dir, AUTO_CONF_PATH) 882 self.defconfig = os.path.join(build_dir, 'defconfig') 883 884 def get_arch(self): 885 """Parse .config file and return the architecture. 886 887 Returns: 888 Architecture name (e.g. 'arm'). 889 """ 890 arch = '' 891 cpu = '' 892 for line in open(self.dotconfig): 893 m = self.re_arch.match(line) 894 if m: 895 arch = m.group(1) 896 continue 897 m = self.re_cpu.match(line) 898 if m: 899 cpu = m.group(1) 900 901 if not arch: 902 return None 903 904 # fix-up for aarch64 905 if arch == 'arm' and cpu == 'armv8': 906 arch = 'aarch64' 907 908 return arch 909 910 def parse_one_config(self, config, dotconfig_lines, autoconf_lines): 911 """Parse .config, defconfig, include/autoconf.mk for one config. 912 913 This function looks for the config options in the lines from 914 defconfig, .config, and include/autoconf.mk in order to decide 915 which action should be taken for this defconfig. 916 917 Arguments: 918 config: CONFIG name to parse. 919 dotconfig_lines: lines from the .config file. 920 autoconf_lines: lines from the include/autoconf.mk file. 921 922 Returns: 923 A tupple of the action for this defconfig and the line 924 matched for the config. 925 """ 926 not_set = '# %s is not set' % config 927 928 for line in autoconf_lines: 929 line = line.rstrip() 930 if line.startswith(config + '='): 931 new_val = line 932 break 933 else: 934 new_val = not_set 935 936 new_val = try_expand(new_val) 937 938 for line in dotconfig_lines: 939 line = line.rstrip() 940 if line.startswith(config + '=') or line == not_set: 941 old_val = line 942 break 943 else: 944 if new_val == not_set: 945 return (ACTION_NO_ENTRY, config) 946 else: 947 return (ACTION_NO_ENTRY_WARN, config) 948 949 # If this CONFIG is neither bool nor trisate 950 if old_val[-2:] != '=y' and old_val[-2:] != '=m' and old_val != not_set: 951 # tools/scripts/define2mk.sed changes '1' to 'y'. 952 # This is a problem if the CONFIG is int type. 953 # Check the type in Kconfig and handle it correctly. 954 if new_val[-2:] == '=y': 955 new_val = new_val[:-1] + '1' 956 957 return (ACTION_NO_CHANGE if old_val == new_val else ACTION_MOVE, 958 new_val) 959 960 def update_dotconfig(self): 961 """Parse files for the config options and update the .config. 962 963 This function parses the generated .config and include/autoconf.mk 964 searching the target options. 965 Move the config option(s) to the .config as needed. 966 967 Arguments: 968 defconfig: defconfig name. 969 970 Returns: 971 Return a tuple of (updated flag, log string). 972 The "updated flag" is True if the .config was updated, False 973 otherwise. The "log string" shows what happend to the .config. 974 """ 975 976 results = [] 977 updated = False 978 suspicious = False 979 rm_files = [self.config_autoconf, self.autoconf] 980 981 if self.options.spl: 982 if os.path.exists(self.spl_autoconf): 983 autoconf_path = self.spl_autoconf 984 rm_files.append(self.spl_autoconf) 985 else: 986 for f in rm_files: 987 os.remove(f) 988 return (updated, suspicious, 989 color_text(self.options.color, COLOR_BROWN, 990 "SPL is not enabled. Skipped.") + '\n') 991 else: 992 autoconf_path = self.autoconf 993 994 with open(self.dotconfig) as f: 995 dotconfig_lines = f.readlines() 996 997 with open(autoconf_path) as f: 998 autoconf_lines = f.readlines() 999 1000 for config in self.configs: 1001 result = self.parse_one_config(config, dotconfig_lines, 1002 autoconf_lines) 1003 results.append(result) 1004 1005 log = '' 1006 1007 for (action, value) in results: 1008 if action == ACTION_MOVE: 1009 actlog = "Move '%s'" % value 1010 log_color = COLOR_LIGHT_GREEN 1011 elif action == ACTION_NO_ENTRY: 1012 actlog = "%s is not defined in Kconfig. Do nothing." % value 1013 log_color = COLOR_LIGHT_BLUE 1014 elif action == ACTION_NO_ENTRY_WARN: 1015 actlog = "%s is not defined in Kconfig (suspicious). Do nothing." % value 1016 log_color = COLOR_YELLOW 1017 suspicious = True 1018 elif action == ACTION_NO_CHANGE: 1019 actlog = "'%s' is the same as the define in Kconfig. Do nothing." \ 1020 % value 1021 log_color = COLOR_LIGHT_PURPLE 1022 elif action == ACTION_SPL_NOT_EXIST: 1023 actlog = "SPL is not enabled for this defconfig. Skip." 1024 log_color = COLOR_PURPLE 1025 else: 1026 sys.exit("Internal Error. This should not happen.") 1027 1028 log += color_text(self.options.color, log_color, actlog) + '\n' 1029 1030 with open(self.dotconfig, 'a') as f: 1031 for (action, value) in results: 1032 if action == ACTION_MOVE: 1033 f.write(value + '\n') 1034 updated = True 1035 1036 self.results = results 1037 for f in rm_files: 1038 os.remove(f) 1039 1040 return (updated, suspicious, log) 1041 1042 def check_defconfig(self): 1043 """Check the defconfig after savedefconfig 1044 1045 Returns: 1046 Return additional log if moved CONFIGs were removed again by 1047 'make savedefconfig'. 1048 """ 1049 1050 log = '' 1051 1052 with open(self.defconfig) as f: 1053 defconfig_lines = f.readlines() 1054 1055 for (action, value) in self.results: 1056 if action != ACTION_MOVE: 1057 continue 1058 if not value + '\n' in defconfig_lines: 1059 log += color_text(self.options.color, COLOR_YELLOW, 1060 "'%s' was removed by savedefconfig.\n" % 1061 value) 1062 1063 return log 1064 1065 1066class DatabaseThread(threading.Thread): 1067 """This thread processes results from Slot threads. 1068 1069 It collects the data in the master config directary. There is only one 1070 result thread, and this helps to serialise the build output. 1071 """ 1072 def __init__(self, config_db, db_queue): 1073 """Set up a new result thread 1074 1075 Args: 1076 builder: Builder which will be sent each result 1077 """ 1078 threading.Thread.__init__(self) 1079 self.config_db = config_db 1080 self.db_queue= db_queue 1081 1082 def run(self): 1083 """Called to start up the result thread. 1084 1085 We collect the next result job and pass it on to the build. 1086 """ 1087 while True: 1088 defconfig, configs = self.db_queue.get() 1089 self.config_db[defconfig] = configs 1090 self.db_queue.task_done() 1091 1092 1093class Slot: 1094 1095 """A slot to store a subprocess. 1096 1097 Each instance of this class handles one subprocess. 1098 This class is useful to control multiple threads 1099 for faster processing. 1100 """ 1101 1102 def __init__(self, toolchains, configs, options, progress, devnull, 1103 make_cmd, reference_src_dir, db_queue): 1104 """Create a new process slot. 1105 1106 Arguments: 1107 toolchains: Toolchains object containing toolchains. 1108 configs: A list of CONFIGs to move. 1109 options: option flags. 1110 progress: A progress indicator. 1111 devnull: A file object of '/dev/null'. 1112 make_cmd: command name of GNU Make. 1113 reference_src_dir: Determine the true starting config state from this 1114 source tree. 1115 db_queue: output queue to write config info for the database 1116 """ 1117 self.toolchains = toolchains 1118 self.options = options 1119 self.progress = progress 1120 self.build_dir = tempfile.mkdtemp() 1121 self.devnull = devnull 1122 self.make_cmd = (make_cmd, 'O=' + self.build_dir) 1123 self.reference_src_dir = reference_src_dir 1124 self.db_queue = db_queue 1125 self.parser = KconfigParser(configs, options, self.build_dir) 1126 self.state = STATE_IDLE 1127 self.failed_boards = set() 1128 self.suspicious_boards = set() 1129 1130 def __del__(self): 1131 """Delete the working directory 1132 1133 This function makes sure the temporary directory is cleaned away 1134 even if Python suddenly dies due to error. It should be done in here 1135 because it is guaranteed the destructor is always invoked when the 1136 instance of the class gets unreferenced. 1137 1138 If the subprocess is still running, wait until it finishes. 1139 """ 1140 if self.state != STATE_IDLE: 1141 while self.ps.poll() == None: 1142 pass 1143 shutil.rmtree(self.build_dir) 1144 1145 def add(self, defconfig): 1146 """Assign a new subprocess for defconfig and add it to the slot. 1147 1148 If the slot is vacant, create a new subprocess for processing the 1149 given defconfig and add it to the slot. Just returns False if 1150 the slot is occupied (i.e. the current subprocess is still running). 1151 1152 Arguments: 1153 defconfig: defconfig name. 1154 1155 Returns: 1156 Return True on success or False on failure 1157 """ 1158 if self.state != STATE_IDLE: 1159 return False 1160 1161 self.defconfig = defconfig 1162 self.log = '' 1163 self.current_src_dir = self.reference_src_dir 1164 self.do_defconfig() 1165 return True 1166 1167 def poll(self): 1168 """Check the status of the subprocess and handle it as needed. 1169 1170 Returns True if the slot is vacant (i.e. in idle state). 1171 If the configuration is successfully finished, assign a new 1172 subprocess to build include/autoconf.mk. 1173 If include/autoconf.mk is generated, invoke the parser to 1174 parse the .config and the include/autoconf.mk, moving 1175 config options to the .config as needed. 1176 If the .config was updated, run "make savedefconfig" to sync 1177 it, update the original defconfig, and then set the slot back 1178 to the idle state. 1179 1180 Returns: 1181 Return True if the subprocess is terminated, False otherwise 1182 """ 1183 if self.state == STATE_IDLE: 1184 return True 1185 1186 if self.ps.poll() == None: 1187 return False 1188 1189 if self.ps.poll() != 0: 1190 self.handle_error() 1191 elif self.state == STATE_DEFCONFIG: 1192 if self.reference_src_dir and not self.current_src_dir: 1193 self.do_savedefconfig() 1194 else: 1195 self.do_autoconf() 1196 elif self.state == STATE_AUTOCONF: 1197 if self.current_src_dir: 1198 self.current_src_dir = None 1199 self.do_defconfig() 1200 elif self.options.build_db: 1201 self.do_build_db() 1202 else: 1203 self.do_savedefconfig() 1204 elif self.state == STATE_SAVEDEFCONFIG: 1205 self.update_defconfig() 1206 else: 1207 sys.exit("Internal Error. This should not happen.") 1208 1209 return True if self.state == STATE_IDLE else False 1210 1211 def handle_error(self): 1212 """Handle error cases.""" 1213 1214 self.log += color_text(self.options.color, COLOR_LIGHT_RED, 1215 "Failed to process.\n") 1216 if self.options.verbose: 1217 self.log += color_text(self.options.color, COLOR_LIGHT_CYAN, 1218 self.ps.stderr.read()) 1219 self.finish(False) 1220 1221 def do_defconfig(self): 1222 """Run 'make <board>_defconfig' to create the .config file.""" 1223 1224 cmd = list(self.make_cmd) 1225 cmd.append(self.defconfig) 1226 self.ps = subprocess.Popen(cmd, stdout=self.devnull, 1227 stderr=subprocess.PIPE, 1228 cwd=self.current_src_dir) 1229 self.state = STATE_DEFCONFIG 1230 1231 def do_autoconf(self): 1232 """Run 'make AUTO_CONF_PATH'.""" 1233 1234 arch = self.parser.get_arch() 1235 try: 1236 toolchain = self.toolchains.Select(arch) 1237 except ValueError: 1238 self.log += color_text(self.options.color, COLOR_YELLOW, 1239 "Tool chain for '%s' is missing. Do nothing.\n" % arch) 1240 self.finish(False) 1241 return 1242 env = toolchain.MakeEnvironment(False) 1243 1244 cmd = list(self.make_cmd) 1245 cmd.append('KCONFIG_IGNORE_DUPLICATES=1') 1246 cmd.append(AUTO_CONF_PATH) 1247 self.ps = subprocess.Popen(cmd, stdout=self.devnull, env=env, 1248 stderr=subprocess.PIPE, 1249 cwd=self.current_src_dir) 1250 self.state = STATE_AUTOCONF 1251 1252 def do_build_db(self): 1253 """Add the board to the database""" 1254 configs = {} 1255 with open(os.path.join(self.build_dir, AUTO_CONF_PATH)) as fd: 1256 for line in fd.readlines(): 1257 if line.startswith('CONFIG'): 1258 config, value = line.split('=', 1) 1259 configs[config] = value.rstrip() 1260 self.db_queue.put([self.defconfig, configs]) 1261 self.finish(True) 1262 1263 def do_savedefconfig(self): 1264 """Update the .config and run 'make savedefconfig'.""" 1265 1266 (updated, suspicious, log) = self.parser.update_dotconfig() 1267 if suspicious: 1268 self.suspicious_boards.add(self.defconfig) 1269 self.log += log 1270 1271 if not self.options.force_sync and not updated: 1272 self.finish(True) 1273 return 1274 if updated: 1275 self.log += color_text(self.options.color, COLOR_LIGHT_GREEN, 1276 "Syncing by savedefconfig...\n") 1277 else: 1278 self.log += "Syncing by savedefconfig (forced by option)...\n" 1279 1280 cmd = list(self.make_cmd) 1281 cmd.append('savedefconfig') 1282 self.ps = subprocess.Popen(cmd, stdout=self.devnull, 1283 stderr=subprocess.PIPE) 1284 self.state = STATE_SAVEDEFCONFIG 1285 1286 def update_defconfig(self): 1287 """Update the input defconfig and go back to the idle state.""" 1288 1289 log = self.parser.check_defconfig() 1290 if log: 1291 self.suspicious_boards.add(self.defconfig) 1292 self.log += log 1293 orig_defconfig = os.path.join('configs', self.defconfig) 1294 new_defconfig = os.path.join(self.build_dir, 'defconfig') 1295 updated = not filecmp.cmp(orig_defconfig, new_defconfig) 1296 1297 if updated: 1298 self.log += color_text(self.options.color, COLOR_LIGHT_BLUE, 1299 "defconfig was updated.\n") 1300 1301 if not self.options.dry_run and updated: 1302 shutil.move(new_defconfig, orig_defconfig) 1303 self.finish(True) 1304 1305 def finish(self, success): 1306 """Display log along with progress and go to the idle state. 1307 1308 Arguments: 1309 success: Should be True when the defconfig was processed 1310 successfully, or False when it fails. 1311 """ 1312 # output at least 30 characters to hide the "* defconfigs out of *". 1313 log = self.defconfig.ljust(30) + '\n' 1314 1315 log += '\n'.join([ ' ' + s for s in self.log.split('\n') ]) 1316 # Some threads are running in parallel. 1317 # Print log atomically to not mix up logs from different threads. 1318 print(log, file=(sys.stdout if success else sys.stderr)) 1319 1320 if not success: 1321 if self.options.exit_on_error: 1322 sys.exit("Exit on error.") 1323 # If --exit-on-error flag is not set, skip this board and continue. 1324 # Record the failed board. 1325 self.failed_boards.add(self.defconfig) 1326 1327 self.progress.inc() 1328 self.progress.show() 1329 self.state = STATE_IDLE 1330 1331 def get_failed_boards(self): 1332 """Returns a set of failed boards (defconfigs) in this slot. 1333 """ 1334 return self.failed_boards 1335 1336 def get_suspicious_boards(self): 1337 """Returns a set of boards (defconfigs) with possible misconversion. 1338 """ 1339 return self.suspicious_boards - self.failed_boards 1340 1341class Slots: 1342 1343 """Controller of the array of subprocess slots.""" 1344 1345 def __init__(self, toolchains, configs, options, progress, 1346 reference_src_dir, db_queue): 1347 """Create a new slots controller. 1348 1349 Arguments: 1350 toolchains: Toolchains object containing toolchains. 1351 configs: A list of CONFIGs to move. 1352 options: option flags. 1353 progress: A progress indicator. 1354 reference_src_dir: Determine the true starting config state from this 1355 source tree. 1356 db_queue: output queue to write config info for the database 1357 """ 1358 self.options = options 1359 self.slots = [] 1360 devnull = get_devnull() 1361 make_cmd = get_make_cmd() 1362 for i in range(options.jobs): 1363 self.slots.append(Slot(toolchains, configs, options, progress, 1364 devnull, make_cmd, reference_src_dir, 1365 db_queue)) 1366 1367 def add(self, defconfig): 1368 """Add a new subprocess if a vacant slot is found. 1369 1370 Arguments: 1371 defconfig: defconfig name to be put into. 1372 1373 Returns: 1374 Return True on success or False on failure 1375 """ 1376 for slot in self.slots: 1377 if slot.add(defconfig): 1378 return True 1379 return False 1380 1381 def available(self): 1382 """Check if there is a vacant slot. 1383 1384 Returns: 1385 Return True if at lease one vacant slot is found, False otherwise. 1386 """ 1387 for slot in self.slots: 1388 if slot.poll(): 1389 return True 1390 return False 1391 1392 def empty(self): 1393 """Check if all slots are vacant. 1394 1395 Returns: 1396 Return True if all the slots are vacant, False otherwise. 1397 """ 1398 ret = True 1399 for slot in self.slots: 1400 if not slot.poll(): 1401 ret = False 1402 return ret 1403 1404 def show_failed_boards(self): 1405 """Display all of the failed boards (defconfigs).""" 1406 boards = set() 1407 output_file = 'moveconfig.failed' 1408 1409 for slot in self.slots: 1410 boards |= slot.get_failed_boards() 1411 1412 if boards: 1413 boards = '\n'.join(boards) + '\n' 1414 msg = "The following boards were not processed due to error:\n" 1415 msg += boards 1416 msg += "(the list has been saved in %s)\n" % output_file 1417 print(color_text(self.options.color, COLOR_LIGHT_RED, 1418 msg), file=sys.stderr) 1419 1420 with open(output_file, 'w') as f: 1421 f.write(boards) 1422 1423 def show_suspicious_boards(self): 1424 """Display all boards (defconfigs) with possible misconversion.""" 1425 boards = set() 1426 output_file = 'moveconfig.suspicious' 1427 1428 for slot in self.slots: 1429 boards |= slot.get_suspicious_boards() 1430 1431 if boards: 1432 boards = '\n'.join(boards) + '\n' 1433 msg = "The following boards might have been converted incorrectly.\n" 1434 msg += "It is highly recommended to check them manually:\n" 1435 msg += boards 1436 msg += "(the list has been saved in %s)\n" % output_file 1437 print(color_text(self.options.color, COLOR_YELLOW, 1438 msg), file=sys.stderr) 1439 1440 with open(output_file, 'w') as f: 1441 f.write(boards) 1442 1443class ReferenceSource: 1444 1445 """Reference source against which original configs should be parsed.""" 1446 1447 def __init__(self, commit): 1448 """Create a reference source directory based on a specified commit. 1449 1450 Arguments: 1451 commit: commit to git-clone 1452 """ 1453 self.src_dir = tempfile.mkdtemp() 1454 print("Cloning git repo to a separate work directory...") 1455 subprocess.check_output(['git', 'clone', os.getcwd(), '.'], 1456 cwd=self.src_dir) 1457 print("Checkout '%s' to build the original autoconf.mk." % \ 1458 subprocess.check_output(['git', 'rev-parse', '--short', commit]).strip()) 1459 subprocess.check_output(['git', 'checkout', commit], 1460 stderr=subprocess.STDOUT, cwd=self.src_dir) 1461 1462 def __del__(self): 1463 """Delete the reference source directory 1464 1465 This function makes sure the temporary directory is cleaned away 1466 even if Python suddenly dies due to error. It should be done in here 1467 because it is guaranteed the destructor is always invoked when the 1468 instance of the class gets unreferenced. 1469 """ 1470 shutil.rmtree(self.src_dir) 1471 1472 def get_dir(self): 1473 """Return the absolute path to the reference source directory.""" 1474 1475 return self.src_dir 1476 1477def move_config(toolchains, configs, options, db_queue): 1478 """Move config options to defconfig files. 1479 1480 Arguments: 1481 configs: A list of CONFIGs to move. 1482 options: option flags 1483 """ 1484 if len(configs) == 0: 1485 if options.force_sync: 1486 print('No CONFIG is specified. You are probably syncing defconfigs.', end=' ') 1487 elif options.build_db: 1488 print('Building %s database' % CONFIG_DATABASE) 1489 else: 1490 print('Neither CONFIG nor --force-sync is specified. Nothing will happen.', end=' ') 1491 else: 1492 print('Move ' + ', '.join(configs), end=' ') 1493 print('(jobs: %d)\n' % options.jobs) 1494 1495 if options.git_ref: 1496 reference_src = ReferenceSource(options.git_ref) 1497 reference_src_dir = reference_src.get_dir() 1498 else: 1499 reference_src_dir = None 1500 1501 if options.defconfigs: 1502 defconfigs = get_matched_defconfigs(options.defconfigs) 1503 else: 1504 defconfigs = get_all_defconfigs() 1505 1506 progress = Progress(len(defconfigs)) 1507 slots = Slots(toolchains, configs, options, progress, reference_src_dir, 1508 db_queue) 1509 1510 # Main loop to process defconfig files: 1511 # Add a new subprocess into a vacant slot. 1512 # Sleep if there is no available slot. 1513 for defconfig in defconfigs: 1514 while not slots.add(defconfig): 1515 while not slots.available(): 1516 # No available slot: sleep for a while 1517 time.sleep(SLEEP_TIME) 1518 1519 # wait until all the subprocesses finish 1520 while not slots.empty(): 1521 time.sleep(SLEEP_TIME) 1522 1523 print('') 1524 slots.show_failed_boards() 1525 slots.show_suspicious_boards() 1526 1527def find_kconfig_rules(kconf, config, imply_config): 1528 """Check whether a config has a 'select' or 'imply' keyword 1529 1530 Args: 1531 kconf: Kconfiglib.Kconfig object 1532 config: Name of config to check (without CONFIG_ prefix) 1533 imply_config: Implying config (without CONFIG_ prefix) which may or 1534 may not have an 'imply' for 'config') 1535 1536 Returns: 1537 Symbol object for 'config' if found, else None 1538 """ 1539 sym = kconf.syms.get(imply_config) 1540 if sym: 1541 for sel in sym.get_selected_symbols() | sym.get_implied_symbols(): 1542 if sel.get_name() == config: 1543 return sym 1544 return None 1545 1546def check_imply_rule(kconf, config, imply_config): 1547 """Check if we can add an 'imply' option 1548 1549 This finds imply_config in the Kconfig and looks to see if it is possible 1550 to add an 'imply' for 'config' to that part of the Kconfig. 1551 1552 Args: 1553 kconf: Kconfiglib.Kconfig object 1554 config: Name of config to check (without CONFIG_ prefix) 1555 imply_config: Implying config (without CONFIG_ prefix) which may or 1556 may not have an 'imply' for 'config') 1557 1558 Returns: 1559 tuple: 1560 filename of Kconfig file containing imply_config, or None if none 1561 line number within the Kconfig file, or 0 if none 1562 message indicating the result 1563 """ 1564 sym = kconf.syms.get(imply_config) 1565 if not sym: 1566 return 'cannot find sym' 1567 locs = sym.get_def_locations() 1568 if len(locs) != 1: 1569 return '%d locations' % len(locs) 1570 fname, linenum = locs[0] 1571 cwd = os.getcwd() 1572 if cwd and fname.startswith(cwd): 1573 fname = fname[len(cwd) + 1:] 1574 file_line = ' at %s:%d' % (fname, linenum) 1575 with open(fname) as fd: 1576 data = fd.read().splitlines() 1577 if data[linenum - 1] != 'config %s' % imply_config: 1578 return None, 0, 'bad sym format %s%s' % (data[linenum], file_line) 1579 return fname, linenum, 'adding%s' % file_line 1580 1581def add_imply_rule(config, fname, linenum): 1582 """Add a new 'imply' option to a Kconfig 1583 1584 Args: 1585 config: config option to add an imply for (without CONFIG_ prefix) 1586 fname: Kconfig filename to update 1587 linenum: Line number to place the 'imply' before 1588 1589 Returns: 1590 Message indicating the result 1591 """ 1592 file_line = ' at %s:%d' % (fname, linenum) 1593 data = open(fname).read().splitlines() 1594 linenum -= 1 1595 1596 for offset, line in enumerate(data[linenum:]): 1597 if line.strip().startswith('help') or not line: 1598 data.insert(linenum + offset, '\timply %s' % config) 1599 with open(fname, 'w') as fd: 1600 fd.write('\n'.join(data) + '\n') 1601 return 'added%s' % file_line 1602 1603 return 'could not insert%s' 1604 1605(IMPLY_MIN_2, IMPLY_TARGET, IMPLY_CMD, IMPLY_NON_ARCH_BOARD) = ( 1606 1, 2, 4, 8) 1607 1608IMPLY_FLAGS = { 1609 'min2': [IMPLY_MIN_2, 'Show options which imply >2 boards (normally >5)'], 1610 'target': [IMPLY_TARGET, 'Allow CONFIG_TARGET_... options to imply'], 1611 'cmd': [IMPLY_CMD, 'Allow CONFIG_CMD_... to imply'], 1612 'non-arch-board': [ 1613 IMPLY_NON_ARCH_BOARD, 1614 'Allow Kconfig options outside arch/ and /board/ to imply'], 1615}; 1616 1617def do_imply_config(config_list, add_imply, imply_flags, skip_added, 1618 check_kconfig=True, find_superset=False): 1619 """Find CONFIG options which imply those in the list 1620 1621 Some CONFIG options can be implied by others and this can help to reduce 1622 the size of the defconfig files. For example, CONFIG_X86 implies 1623 CONFIG_CMD_IRQ, so we can put 'imply CMD_IRQ' under 'config X86' and 1624 all x86 boards will have that option, avoiding adding CONFIG_CMD_IRQ to 1625 each of the x86 defconfig files. 1626 1627 This function uses the moveconfig database to find such options. It 1628 displays a list of things that could possibly imply those in the list. 1629 The algorithm ignores any that start with CONFIG_TARGET since these 1630 typically refer to only a few defconfigs (often one). It also does not 1631 display a config with less than 5 defconfigs. 1632 1633 The algorithm works using sets. For each target config in config_list: 1634 - Get the set 'defconfigs' which use that target config 1635 - For each config (from a list of all configs): 1636 - Get the set 'imply_defconfig' of defconfigs which use that config 1637 - 1638 - If imply_defconfigs contains anything not in defconfigs then 1639 this config does not imply the target config 1640 1641 Params: 1642 config_list: List of CONFIG options to check (each a string) 1643 add_imply: Automatically add an 'imply' for each config. 1644 imply_flags: Flags which control which implying configs are allowed 1645 (IMPLY_...) 1646 skip_added: Don't show options which already have an imply added. 1647 check_kconfig: Check if implied symbols already have an 'imply' or 1648 'select' for the target config, and show this information if so. 1649 find_superset: True to look for configs which are a superset of those 1650 already found. So for example if CONFIG_EXYNOS5 implies an option, 1651 but CONFIG_EXYNOS covers a larger set of defconfigs and also 1652 implies that option, this will drop the former in favour of the 1653 latter. In practice this option has not proved very used. 1654 1655 Note the terminoloy: 1656 config - a CONFIG_XXX options (a string, e.g. 'CONFIG_CMD_EEPROM') 1657 defconfig - a defconfig file (a string, e.g. 'configs/snow_defconfig') 1658 """ 1659 kconf = KconfigScanner().conf if check_kconfig else None 1660 if add_imply and add_imply != 'all': 1661 add_imply = add_imply.split() 1662 1663 # key is defconfig name, value is dict of (CONFIG_xxx, value) 1664 config_db = {} 1665 1666 # Holds a dict containing the set of defconfigs that contain each config 1667 # key is config, value is set of defconfigs using that config 1668 defconfig_db = collections.defaultdict(set) 1669 1670 # Set of all config options we have seen 1671 all_configs = set() 1672 1673 # Set of all defconfigs we have seen 1674 all_defconfigs = set() 1675 1676 # Read in the database 1677 configs = {} 1678 with open(CONFIG_DATABASE) as fd: 1679 for line in fd.readlines(): 1680 line = line.rstrip() 1681 if not line: # Separator between defconfigs 1682 config_db[defconfig] = configs 1683 all_defconfigs.add(defconfig) 1684 configs = {} 1685 elif line[0] == ' ': # CONFIG line 1686 config, value = line.strip().split('=', 1) 1687 configs[config] = value 1688 defconfig_db[config].add(defconfig) 1689 all_configs.add(config) 1690 else: # New defconfig 1691 defconfig = line 1692 1693 # Work through each target config option in tern, independently 1694 for config in config_list: 1695 defconfigs = defconfig_db.get(config) 1696 if not defconfigs: 1697 print('%s not found in any defconfig' % config) 1698 continue 1699 1700 # Get the set of defconfigs without this one (since a config cannot 1701 # imply itself) 1702 non_defconfigs = all_defconfigs - defconfigs 1703 num_defconfigs = len(defconfigs) 1704 print('%s found in %d/%d defconfigs' % (config, num_defconfigs, 1705 len(all_configs))) 1706 1707 # This will hold the results: key=config, value=defconfigs containing it 1708 imply_configs = {} 1709 rest_configs = all_configs - set([config]) 1710 1711 # Look at every possible config, except the target one 1712 for imply_config in rest_configs: 1713 if 'ERRATUM' in imply_config: 1714 continue 1715 if not (imply_flags & IMPLY_CMD): 1716 if 'CONFIG_CMD' in imply_config: 1717 continue 1718 if not (imply_flags & IMPLY_TARGET): 1719 if 'CONFIG_TARGET' in imply_config: 1720 continue 1721 1722 # Find set of defconfigs that have this config 1723 imply_defconfig = defconfig_db[imply_config] 1724 1725 # Get the intersection of this with defconfigs containing the 1726 # target config 1727 common_defconfigs = imply_defconfig & defconfigs 1728 1729 # Get the set of defconfigs containing this config which DO NOT 1730 # also contain the taret config. If this set is non-empty it means 1731 # that this config affects other defconfigs as well as (possibly) 1732 # the ones affected by the target config. This means it implies 1733 # things we don't want to imply. 1734 not_common_defconfigs = imply_defconfig & non_defconfigs 1735 if not_common_defconfigs: 1736 continue 1737 1738 # If there are common defconfigs, imply_config may be useful 1739 if common_defconfigs: 1740 skip = False 1741 if find_superset: 1742 for prev in list(imply_configs.keys()): 1743 prev_count = len(imply_configs[prev]) 1744 count = len(common_defconfigs) 1745 if (prev_count > count and 1746 (imply_configs[prev] & common_defconfigs == 1747 common_defconfigs)): 1748 # skip imply_config because prev is a superset 1749 skip = True 1750 break 1751 elif count > prev_count: 1752 # delete prev because imply_config is a superset 1753 del imply_configs[prev] 1754 if not skip: 1755 imply_configs[imply_config] = common_defconfigs 1756 1757 # Now we have a dict imply_configs of configs which imply each config 1758 # The value of each dict item is the set of defconfigs containing that 1759 # config. Rank them so that we print the configs that imply the largest 1760 # number of defconfigs first. 1761 ranked_iconfigs = sorted(imply_configs, 1762 key=lambda k: len(imply_configs[k]), reverse=True) 1763 kconfig_info = '' 1764 cwd = os.getcwd() 1765 add_list = collections.defaultdict(list) 1766 for iconfig in ranked_iconfigs: 1767 num_common = len(imply_configs[iconfig]) 1768 1769 # Don't bother if there are less than 5 defconfigs affected. 1770 if num_common < (2 if imply_flags & IMPLY_MIN_2 else 5): 1771 continue 1772 missing = defconfigs - imply_configs[iconfig] 1773 missing_str = ', '.join(missing) if missing else 'all' 1774 missing_str = '' 1775 show = True 1776 if kconf: 1777 sym = find_kconfig_rules(kconf, config[CONFIG_LEN:], 1778 iconfig[CONFIG_LEN:]) 1779 kconfig_info = '' 1780 if sym: 1781 locs = sym.get_def_locations() 1782 if len(locs) == 1: 1783 fname, linenum = locs[0] 1784 if cwd and fname.startswith(cwd): 1785 fname = fname[len(cwd) + 1:] 1786 kconfig_info = '%s:%d' % (fname, linenum) 1787 if skip_added: 1788 show = False 1789 else: 1790 sym = kconf.syms.get(iconfig[CONFIG_LEN:]) 1791 fname = '' 1792 if sym: 1793 locs = sym.get_def_locations() 1794 if len(locs) == 1: 1795 fname, linenum = locs[0] 1796 if cwd and fname.startswith(cwd): 1797 fname = fname[len(cwd) + 1:] 1798 in_arch_board = not sym or (fname.startswith('arch') or 1799 fname.startswith('board')) 1800 if (not in_arch_board and 1801 not (imply_flags & IMPLY_NON_ARCH_BOARD)): 1802 continue 1803 1804 if add_imply and (add_imply == 'all' or 1805 iconfig in add_imply): 1806 fname, linenum, kconfig_info = (check_imply_rule(kconf, 1807 config[CONFIG_LEN:], iconfig[CONFIG_LEN:])) 1808 if fname: 1809 add_list[fname].append(linenum) 1810 1811 if show and kconfig_info != 'skip': 1812 print('%5d : %-30s%-25s %s' % (num_common, iconfig.ljust(30), 1813 kconfig_info, missing_str)) 1814 1815 # Having collected a list of things to add, now we add them. We process 1816 # each file from the largest line number to the smallest so that 1817 # earlier additions do not affect our line numbers. E.g. if we added an 1818 # imply at line 20 it would change the position of each line after 1819 # that. 1820 for fname, linenums in add_list.items(): 1821 for linenum in sorted(linenums, reverse=True): 1822 add_imply_rule(config[CONFIG_LEN:], fname, linenum) 1823 1824 1825def main(): 1826 try: 1827 cpu_count = multiprocessing.cpu_count() 1828 except NotImplementedError: 1829 cpu_count = 1 1830 1831 parser = optparse.OptionParser() 1832 # Add options here 1833 parser.add_option('-a', '--add-imply', type='string', default='', 1834 help='comma-separated list of CONFIG options to add ' 1835 "an 'imply' statement to for the CONFIG in -i") 1836 parser.add_option('-A', '--skip-added', action='store_true', default=False, 1837 help="don't show options which are already marked as " 1838 'implying others') 1839 parser.add_option('-b', '--build-db', action='store_true', default=False, 1840 help='build a CONFIG database') 1841 parser.add_option('-c', '--color', action='store_true', default=False, 1842 help='display the log in color') 1843 parser.add_option('-C', '--commit', action='store_true', default=False, 1844 help='Create a git commit for the operation') 1845 parser.add_option('-d', '--defconfigs', type='string', 1846 help='a file containing a list of defconfigs to move, ' 1847 "one per line (for example 'snow_defconfig') " 1848 "or '-' to read from stdin") 1849 parser.add_option('-i', '--imply', action='store_true', default=False, 1850 help='find options which imply others') 1851 parser.add_option('-I', '--imply-flags', type='string', default='', 1852 help="control the -i option ('help' for help") 1853 parser.add_option('-n', '--dry-run', action='store_true', default=False, 1854 help='perform a trial run (show log with no changes)') 1855 parser.add_option('-e', '--exit-on-error', action='store_true', 1856 default=False, 1857 help='exit immediately on any error') 1858 parser.add_option('-s', '--force-sync', action='store_true', default=False, 1859 help='force sync by savedefconfig') 1860 parser.add_option('-S', '--spl', action='store_true', default=False, 1861 help='parse config options defined for SPL build') 1862 parser.add_option('-H', '--headers-only', dest='cleanup_headers_only', 1863 action='store_true', default=False, 1864 help='only cleanup the headers') 1865 parser.add_option('-j', '--jobs', type='int', default=cpu_count, 1866 help='the number of jobs to run simultaneously') 1867 parser.add_option('-r', '--git-ref', type='string', 1868 help='the git ref to clone for building the autoconf.mk') 1869 parser.add_option('-y', '--yes', action='store_true', default=False, 1870 help="respond 'yes' to any prompts") 1871 parser.add_option('-v', '--verbose', action='store_true', default=False, 1872 help='show any build errors as boards are built') 1873 parser.usage += ' CONFIG ...' 1874 1875 (options, configs) = parser.parse_args() 1876 1877 if len(configs) == 0 and not any((options.force_sync, options.build_db, 1878 options.imply)): 1879 parser.print_usage() 1880 sys.exit(1) 1881 1882 # prefix the option name with CONFIG_ if missing 1883 configs = [ config if config.startswith('CONFIG_') else 'CONFIG_' + config 1884 for config in configs ] 1885 1886 check_top_directory() 1887 1888 if options.imply: 1889 imply_flags = 0 1890 if options.imply_flags == 'all': 1891 imply_flags = -1 1892 1893 elif options.imply_flags: 1894 for flag in options.imply_flags.split(','): 1895 bad = flag not in IMPLY_FLAGS 1896 if bad: 1897 print("Invalid flag '%s'" % flag) 1898 if flag == 'help' or bad: 1899 print("Imply flags: (separate with ',')") 1900 for name, info in IMPLY_FLAGS.items(): 1901 print(' %-15s: %s' % (name, info[1])) 1902 parser.print_usage() 1903 sys.exit(1) 1904 imply_flags |= IMPLY_FLAGS[flag][0] 1905 1906 do_imply_config(configs, options.add_imply, imply_flags, 1907 options.skip_added) 1908 return 1909 1910 config_db = {} 1911 db_queue = queue.Queue() 1912 t = DatabaseThread(config_db, db_queue) 1913 t.setDaemon(True) 1914 t.start() 1915 1916 if not options.cleanup_headers_only: 1917 check_clean_directory() 1918 bsettings.Setup('') 1919 toolchains = toolchain.Toolchains() 1920 toolchains.GetSettings() 1921 toolchains.Scan(verbose=False) 1922 move_config(toolchains, configs, options, db_queue) 1923 db_queue.join() 1924 1925 if configs: 1926 cleanup_headers(configs, options) 1927 cleanup_extra_options(configs, options) 1928 cleanup_whitelist(configs, options) 1929 cleanup_readme(configs, options) 1930 1931 if options.commit: 1932 subprocess.call(['git', 'add', '-u']) 1933 if configs: 1934 msg = 'Convert %s %sto Kconfig' % (configs[0], 1935 'et al ' if len(configs) > 1 else '') 1936 msg += ('\n\nThis converts the following to Kconfig:\n %s\n' % 1937 '\n '.join(configs)) 1938 else: 1939 msg = 'configs: Resync with savedefconfig' 1940 msg += '\n\nRsync all defconfig files using moveconfig.py' 1941 subprocess.call(['git', 'commit', '-s', '-m', msg]) 1942 1943 if options.build_db: 1944 with open(CONFIG_DATABASE, 'w') as fd: 1945 for defconfig, configs in config_db.items(): 1946 fd.write('%s\n' % defconfig) 1947 for config in sorted(configs.keys()): 1948 fd.write(' %s=%s\n' % (config, configs[config])) 1949 fd.write('\n') 1950 1951if __name__ == '__main__': 1952 main() 1953