1#! /usr/bin/env python 2 3""" 4Usage: 5 6python -m test.regrtest [options] [test_name1 [test_name2 ...]] 7python path/to/Lib/test/regrtest.py [options] [test_name1 [test_name2 ...]] 8 9 10If no arguments or options are provided, finds all files matching 11the pattern "test_*" in the Lib/test subdirectory and runs 12them in alphabetical order (but see -M and -u, below, for exceptions). 13 14For more rigorous testing, it is useful to use the following 15command line: 16 17python -E -tt -Wd -3 -m test.regrtest [options] [test_name1 ...] 18 19 20Options: 21 22-h/--help -- print this text and exit 23 24Verbosity 25 26-v/--verbose -- run tests in verbose mode with output to stdout 27-w/--verbose2 -- re-run failed tests in verbose mode 28-W/--verbose3 -- re-run failed tests in verbose mode immediately 29-q/--quiet -- no output unless one or more tests fail 30-S/--slowest -- print the slowest 10 tests 31 --header -- print header with interpreter info 32 33Selecting tests 34 35-r/--randomize -- randomize test execution order (see below) 36 --randseed -- pass a random seed to reproduce a previous random run 37-f/--fromfile -- read names of tests to run from a file (see below) 38-x/--exclude -- arguments are tests to *exclude* 39-s/--single -- single step through a set of tests (see below) 40-m/--match PAT -- match test cases and methods with glob pattern PAT 41--matchfile FILENAME -- filters tests using a text file, one pattern per line 42-G/--failfast -- fail as soon as a test fails (only with -v or -W) 43-u/--use RES1,RES2,... 44 -- specify which special resource intensive tests to run 45-M/--memlimit LIMIT 46 -- run very large memory-consuming tests 47 48Special runs 49 50-l/--findleaks -- if GC is available detect tests that leak memory 51-L/--runleaks -- run the leaks(1) command just before exit 52-R/--huntrleaks RUNCOUNTS 53 -- search for reference leaks (needs debug build, v. slow) 54-j/--multiprocess PROCESSES 55 -- run PROCESSES processes at once 56-T/--coverage -- turn on code coverage tracing using the trace module 57-D/--coverdir DIRECTORY 58 -- Directory where coverage files are put 59-N/--nocoverdir -- Put coverage files alongside modules 60-t/--threshold THRESHOLD 61 -- call gc.set_threshold(THRESHOLD) 62-F/--forever -- run the specified tests in a loop, until an error happens 63-P/--pgo -- enable Profile Guided Optimization training 64--testdir -- execute test files in the specified directory 65 (instead of the Python stdlib test suite) 66--list-tests -- only write the name of tests that will be run, 67 don't execute them 68--list-cases -- only write the name of test cases that will be run, 69 don't execute them 70--fail-env-changed -- if a test file alters the environment, mark the test 71 as failed 72 73 74Additional Option Details: 75 76-r randomizes test execution order. You can use --randseed=int to provide an 77int seed value for the randomizer; this is useful for reproducing troublesome 78test orders. 79 80-s On the first invocation of regrtest using -s, the first test file found 81or the first test file given on the command line is run, and the name of 82the next test is recorded in a file named pynexttest. If run from the 83Python build directory, pynexttest is located in the 'build' subdirectory, 84otherwise it is located in tempfile.gettempdir(). On subsequent runs, 85the test in pynexttest is run, and the next test is written to pynexttest. 86When the last test has been run, pynexttest is deleted. In this way it 87is possible to single step through the test files. This is useful when 88doing memory analysis on the Python interpreter, which process tends to 89consume too many resources to run the full regression test non-stop. 90 91-f reads the names of tests from the file given as f's argument, one 92or more test names per line. Whitespace is ignored. Blank lines and 93lines beginning with '#' are ignored. This is especially useful for 94whittling down failures involving interactions among tests. 95 96-L causes the leaks(1) command to be run just before exit if it exists. 97leaks(1) is available on Mac OS X and presumably on some other 98FreeBSD-derived systems. 99 100-R runs each test several times and examines sys.gettotalrefcount() to 101see if the test appears to be leaking references. The argument should 102be of the form stab:run:fname where 'stab' is the number of times the 103test is run to let gettotalrefcount settle down, 'run' is the number 104of times further it is run and 'fname' is the name of the file the 105reports are written to. These parameters all have defaults (5, 4 and 106"reflog.txt" respectively), and the minimal invocation is '-R :'. 107 108-M runs tests that require an exorbitant amount of memory. These tests 109typically try to ascertain containers keep working when containing more than 1102 billion objects, which only works on 64-bit systems. There are also some 111tests that try to exhaust the address space of the process, which only makes 112sense on 32-bit systems with at least 2Gb of memory. The passed-in memlimit, 113which is a string in the form of '2.5Gb', determines howmuch memory the 114tests will limit themselves to (but they may go slightly over.) The number 115shouldn't be more memory than the machine has (including swap memory). You 116should also keep in mind that swap memory is generally much, much slower 117than RAM, and setting memlimit to all available RAM or higher will heavily 118tax the machine. On the other hand, it is no use running these tests with a 119limit of less than 2.5Gb, and many require more than 20Gb. Tests that expect 120to use more than memlimit memory will be skipped. The big-memory tests 121generally run very, very long. 122 123-u is used to specify which special resource intensive tests to run, 124such as those requiring large file support or network connectivity. 125The argument is a comma-separated list of words indicating the 126resources to test. Currently only the following are defined: 127 128 all - Enable all special resources. 129 130 audio - Tests that use the audio device. (There are known 131 cases of broken audio drivers that can crash Python or 132 even the Linux kernel.) 133 134 curses - Tests that use curses and will modify the terminal's 135 state and output modes. 136 137 largefile - It is okay to run some test that may create huge 138 files. These tests can take a long time and may 139 consume >2GB of disk space temporarily. 140 141 network - It is okay to run tests that use external network 142 resource, e.g. testing SSL support for sockets. 143 144 bsddb - It is okay to run the bsddb testsuite, which takes 145 a long time to complete. 146 147 decimal - Test the decimal module against a large suite that 148 verifies compliance with standards. 149 150 cpu - Used for certain CPU-heavy tests. 151 152 subprocess Run all tests for the subprocess module. 153 154 urlfetch - It is okay to download files required on testing. 155 156 gui - Run tests that require a running GUI. 157 158 xpickle - Test pickle and cPickle against Python 2.4, 2.5 and 2.6 to 159 test backwards compatibility. These tests take a long time 160 to run. 161 162To enable all resources except one, use '-uall,-<resource>'. For 163example, to run all the tests except for the bsddb tests, give the 164option '-uall,-bsddb'. 165 166--matchfile filters tests using a text file, one pattern per line. 167Pattern examples: 168 169- test method: test_stat_attributes 170- test class: FileTests 171- test identifier: test_os.FileTests.test_stat_attributes 172""" 173 174import StringIO 175import datetime 176import getopt 177import imp 178import json 179import math 180import os 181import platform 182import random 183import re 184import shutil 185import sys 186import sysconfig 187import tempfile 188import time 189import traceback 190import unittest 191import warnings 192 193 194# Some times __path__ and __file__ are not absolute (e.g. while running from 195# Lib/) and, if we change the CWD to run the tests in a temporary dir, some 196# imports might fail. This affects only the modules imported before os.chdir(). 197# These modules are searched first in sys.path[0] (so '' -- the CWD) and if 198# they are found in the CWD their __file__ and __path__ will be relative (this 199# happens before the chdir). All the modules imported after the chdir, are 200# not found in the CWD, and since the other paths in sys.path[1:] are absolute 201# (site.py absolutize them), the __file__ and __path__ will be absolute too. 202# Therefore it is necessary to absolutize manually the __file__ and __path__ of 203# the packages to prevent later imports to fail when the CWD is different. 204for module in sys.modules.itervalues(): 205 if hasattr(module, '__path__'): 206 module.__path__ = [os.path.abspath(path) for path in module.__path__] 207 if hasattr(module, '__file__'): 208 module.__file__ = os.path.abspath(module.__file__) 209 210 211# MacOSX (a.k.a. Darwin) has a default stack size that is too small 212# for deeply recursive regular expressions. We see this as crashes in 213# the Python test suite when running test_re.py and test_sre.py. The 214# fix is to set the stack limit to 2048. 215# This approach may also be useful for other Unixy platforms that 216# suffer from small default stack limits. 217if sys.platform == 'darwin': 218 try: 219 import resource 220 except ImportError: 221 pass 222 else: 223 soft, hard = resource.getrlimit(resource.RLIMIT_STACK) 224 newsoft = min(hard, max(soft, 1024*2048)) 225 resource.setrlimit(resource.RLIMIT_STACK, (newsoft, hard)) 226 227# Windows, Tkinter, and resetting the environment after each test don't 228# mix well. To alleviate test failures due to Tcl/Tk not being able to 229# find its library, get the necessary environment massage done once early. 230if sys.platform == 'win32': 231 try: 232 import FixTk 233 except Exception: 234 pass 235 236# Test result constants. 237PASSED = 1 238FAILED = 0 239ENV_CHANGED = -1 240SKIPPED = -2 241RESOURCE_DENIED = -3 242INTERRUPTED = -4 243CHILD_ERROR = -5 # error in a child process 244 245# Minimum duration of a test to display its duration or to mention that 246# the test is running in background 247PROGRESS_MIN_TIME = 30.0 # seconds 248 249# Display the running tests if nothing happened last N seconds 250PROGRESS_UPDATE = 30.0 # seconds 251 252from test import support 253 254ALL_RESOURCES = ('audio', 'curses', 'largefile', 'network', 'bsddb', 255 'decimal', 'cpu', 'subprocess', 'urlfetch', 'gui', 256 'xpickle') 257 258# Other resources excluded from --use=all: 259# 260# - extralagefile (ex: test_zipfile64): really too slow to be enabled 261# "by default" 262RESOURCE_NAMES = ALL_RESOURCES + ('extralargefile',) 263 264TEMPDIR = os.path.abspath(tempfile.gettempdir()) 265 266 267def usage(code, msg=''): 268 print __doc__ 269 if msg: print msg 270 sys.exit(code) 271 272 273def format_duration(seconds): 274 ms = int(math.ceil(seconds * 1e3)) 275 seconds, ms = divmod(ms, 1000) 276 minutes, seconds = divmod(seconds, 60) 277 hours, minutes = divmod(minutes, 60) 278 279 parts = [] 280 if hours: 281 parts.append('%s hour' % hours) 282 if minutes: 283 parts.append('%s min' % minutes) 284 if seconds: 285 parts.append('%s sec' % seconds) 286 if ms: 287 parts.append('%s ms' % ms) 288 if not parts: 289 return '0 ms' 290 291 parts = parts[:2] 292 return ' '.join(parts) 293 294 295_FORMAT_TEST_RESULT = { 296 PASSED: '%s passed', 297 FAILED: '%s failed', 298 ENV_CHANGED: '%s failed (env changed)', 299 SKIPPED: '%s skipped', 300 RESOURCE_DENIED: '%s skipped (resource denied)', 301 INTERRUPTED: '%s interrupted', 302 CHILD_ERROR: '%s crashed', 303} 304 305 306def format_test_result(test_name, result): 307 fmt = _FORMAT_TEST_RESULT.get(result, "%s") 308 return fmt % test_name 309 310 311def cpu_count(): 312 # first try os.sysconf() to prevent loading the big multiprocessing module 313 try: 314 return os.sysconf('SC_NPROCESSORS_ONLN') 315 except (AttributeError, ValueError): 316 pass 317 318 # try multiprocessing.cpu_count() 319 try: 320 import multiprocessing 321 except ImportError: 322 pass 323 else: 324 return multiprocessing.cpu_count() 325 326 return None 327 328 329def unload_test_modules(save_modules): 330 # Unload the newly imported modules (best effort finalization) 331 for module in sys.modules.keys(): 332 if module not in save_modules and module.startswith("test."): 333 support.unload(module) 334 335 336def main(tests=None, testdir=None, verbose=0, quiet=False, 337 exclude=False, single=False, randomize=False, fromfile=None, 338 findleaks=False, use_resources=None, trace=False, coverdir='coverage', 339 runleaks=False, huntrleaks=False, verbose2=False, print_slow=False, 340 random_seed=None, use_mp=None, verbose3=False, forever=False, 341 header=False, pgo=False, failfast=False, match_tests=None): 342 """Execute a test suite. 343 344 This also parses command-line options and modifies its behavior 345 accordingly. 346 347 tests -- a list of strings containing test names (optional) 348 testdir -- the directory in which to look for tests (optional) 349 350 Users other than the Python test suite will certainly want to 351 specify testdir; if it's omitted, the directory containing the 352 Python test suite is searched for. 353 354 If the tests argument is omitted, the tests listed on the 355 command-line will be used. If that's empty, too, then all *.py 356 files beginning with test_ will be used. 357 358 The other default arguments (verbose, quiet, exclude, 359 single, randomize, findleaks, use_resources, trace, coverdir, 360 print_slow, and random_seed) allow programmers calling main() 361 directly to set the values that would normally be set by flags 362 on the command line. 363 """ 364 regrtest_start_time = time.time() 365 366 support.record_original_stdout(sys.stdout) 367 try: 368 opts, args = getopt.getopt(sys.argv[1:], 'hvqxsSrf:lu:t:TD:NLR:FwWM:j:PGm:', 369 ['help', 'verbose', 'verbose2', 'verbose3', 'quiet', 370 'exclude', 'single', 'slow', 'slowest', 'randomize', 'fromfile=', 371 'findleaks', 372 'use=', 'threshold=', 'trace', 'coverdir=', 'nocoverdir', 373 'runleaks', 'huntrleaks=', 'memlimit=', 'randseed=', 374 'multiprocess=', 'slaveargs=', 'forever', 'header', 'pgo', 375 'failfast', 'match=', 'testdir=', 'list-tests', 'list-cases', 376 'coverage', 'matchfile=', 'fail-env-changed']) 377 except getopt.error, msg: 378 usage(2, msg) 379 380 # Defaults 381 if random_seed is None: 382 random_seed = random.randrange(10000000) 383 if use_resources is None: 384 use_resources = [] 385 slaveargs = None 386 list_tests = False 387 list_cases_opt = False 388 fail_env_changed = False 389 for o, a in opts: 390 if o in ('-h', '--help'): 391 usage(0) 392 elif o in ('-v', '--verbose'): 393 verbose += 1 394 elif o in ('-w', '--verbose2'): 395 verbose2 = True 396 elif o in ('-W', '--verbose3'): 397 verbose3 = True 398 elif o in ('-G', '--failfast'): 399 failfast = True 400 elif o in ('-q', '--quiet'): 401 quiet = True; 402 verbose = 0 403 elif o in ('-x', '--exclude'): 404 exclude = True 405 elif o in ('-s', '--single'): 406 single = True 407 elif o in ('-S', '--slow', '--slowest'): 408 print_slow = True 409 elif o in ('-r', '--randomize'): 410 randomize = True 411 elif o == '--randseed': 412 random_seed = int(a) 413 elif o in ('-f', '--fromfile'): 414 fromfile = a 415 elif o in ('-m', '--match'): 416 if match_tests is None: 417 match_tests = [] 418 match_tests.append(a) 419 elif o == '--matchfile': 420 if match_tests is None: 421 match_tests = [] 422 filename = os.path.join(support.SAVEDCWD, a) 423 with open(filename) as fp: 424 for line in fp: 425 match_tests.append(line.strip()) 426 elif o in ('-l', '--findleaks'): 427 findleaks = True 428 elif o in ('-L', '--runleaks'): 429 runleaks = True 430 elif o in ('-t', '--threshold'): 431 import gc 432 gc.set_threshold(int(a)) 433 elif o in ('-T', '--coverage'): 434 trace = True 435 elif o in ('-D', '--coverdir'): 436 coverdir = os.path.join(os.getcwd(), a) 437 elif o in ('-N', '--nocoverdir'): 438 coverdir = None 439 elif o in ('-R', '--huntrleaks'): 440 huntrleaks = a.split(':') 441 if len(huntrleaks) not in (2, 3): 442 print a, huntrleaks 443 usage(2, '-R takes 2 or 3 colon-separated arguments') 444 if not huntrleaks[0]: 445 huntrleaks[0] = 5 446 else: 447 huntrleaks[0] = int(huntrleaks[0]) 448 if not huntrleaks[1]: 449 huntrleaks[1] = 4 450 else: 451 huntrleaks[1] = int(huntrleaks[1]) 452 if len(huntrleaks) == 2 or not huntrleaks[2]: 453 huntrleaks[2:] = ["reflog.txt"] 454 elif o in ('-M', '--memlimit'): 455 support.set_memlimit(a) 456 elif o in ('-u', '--use'): 457 u = [x.lower() for x in a.split(',')] 458 for r in u: 459 if r == 'all': 460 use_resources[:] = ALL_RESOURCES 461 continue 462 remove = False 463 if r[0] == '-': 464 remove = True 465 r = r[1:] 466 if r not in RESOURCE_NAMES: 467 usage(1, 'Invalid -u/--use option: ' + a) 468 if remove: 469 if r in use_resources: 470 use_resources.remove(r) 471 elif r not in use_resources: 472 use_resources.append(r) 473 elif o in ('-F', '--forever'): 474 forever = True 475 elif o in ('-j', '--multiprocess'): 476 use_mp = int(a) 477 elif o == '--header': 478 header = True 479 elif o == '--slaveargs': 480 slaveargs = a 481 elif o in ('-P', '--pgo'): 482 pgo = True 483 elif o == '--testdir': 484 testdir = a 485 elif o == '--list-tests': 486 list_tests = True 487 elif o == '--list-cases': 488 list_cases_opt = True 489 elif o == '--fail-env-changed': 490 fail_env_changed = True 491 else: 492 print >>sys.stderr, ("No handler for option {}. Please " 493 "report this as a bug at http://bugs.python.org.").format(o) 494 sys.exit(1) 495 if single and fromfile: 496 usage(2, "-s and -f don't go together!") 497 if use_mp and trace: 498 usage(2, "-T and -j don't go together!") 499 if use_mp and findleaks: 500 usage(2, "-l and -j don't go together!") 501 if failfast and not (verbose or verbose3): 502 usage("-G/--failfast needs either -v or -W") 503 504 if testdir: 505 testdir = os.path.abspath(testdir) 506 507 # Prepend test directory to sys.path, so runtest() will be able 508 # to locate tests 509 sys.path.insert(0, testdir) 510 511 # Make sure that '' and Lib/test/ are not in sys.path 512 regrtest_dir = os.path.abspath(os.path.dirname(__file__)) 513 for path in ('', regrtest_dir): 514 try: 515 sys.path.remove(path) 516 except ValueError: 517 pass 518 519 if huntrleaks: 520 warmup, repetitions, _ = huntrleaks 521 if warmup < 1 or repetitions < 1: 522 msg = ("Invalid values for the --huntrleaks/-R parameters. The " 523 "number of warmups and repetitions must be at least 1 " 524 "each (1:1).") 525 print >>sys.stderr, msg 526 sys.exit(2) 527 528 if slaveargs is not None: 529 args, kwargs = json.loads(slaveargs) 530 if kwargs['huntrleaks']: 531 warm_caches() 532 if testdir: 533 kwargs['testdir'] = testdir 534 try: 535 result = runtest(*args, **kwargs) 536 except BaseException, e: 537 result = INTERRUPTED, e.__class__.__name__ 538 print # Force a newline (just in case) 539 print json.dumps(result) 540 sys.exit(0) 541 542 if huntrleaks: 543 warm_caches() 544 545 good = [] 546 bad = [] 547 skipped = [] 548 resource_denieds = [] 549 environment_changed = [] 550 rerun = [] 551 first_result = None 552 interrupted = False 553 554 if findleaks: 555 try: 556 import gc 557 except ImportError: 558 print 'No GC available, disabling findleaks.' 559 findleaks = False 560 else: 561 # Uncomment the line below to report garbage that is not 562 # freeable by reference counting alone. By default only 563 # garbage that is not collectable by the GC is reported. 564 #gc.set_debug(gc.DEBUG_SAVEALL) 565 found_garbage = [] 566 567 if single: 568 filename = os.path.join(TEMPDIR, 'pynexttest') 569 try: 570 fp = open(filename, 'r') 571 next_test = fp.read().strip() 572 tests = [next_test] 573 fp.close() 574 except IOError: 575 pass 576 577 if fromfile: 578 tests = [] 579 fp = open(os.path.join(support.SAVEDCWD, fromfile)) 580 for line in fp: 581 guts = line.split() # assuming no test has whitespace in its name 582 if guts and not guts[0].startswith('#'): 583 tests.extend(guts) 584 fp.close() 585 586 # Strip .py extensions. 587 removepy(args) 588 removepy(tests) 589 590 stdtests = STDTESTS[:] 591 nottests = NOTTESTS.copy() 592 if exclude: 593 for arg in args: 594 if arg in stdtests: 595 stdtests.remove(arg) 596 nottests.add(arg) 597 args = [] 598 599 if huntrleaks: 600 # FIXME: bpo-31731: test_io hangs with --huntrleaks 601 print("Warning: bpo-31731: test_io hangs with --huntrleaks: " 602 "exclude the test") 603 nottests.add('test_io') 604 605 display_header = (verbose or header or not (quiet or single or tests or args)) and (not pgo) 606 alltests = findtests(testdir, stdtests, nottests) 607 selected = tests or args or alltests 608 if single: 609 selected = selected[:1] 610 try: 611 next_single_test = alltests[alltests.index(selected[0])+1] 612 except IndexError: 613 next_single_test = None 614 615 if list_tests: 616 for name in selected: 617 print(name) 618 sys.exit(0) 619 620 if list_cases_opt: 621 list_cases(testdir, selected, match_tests) 622 sys.exit(0) 623 624 if trace: 625 import trace 626 tracer = trace.Trace(trace=False, count=True) 627 628 test_times = [] 629 support.use_resources = use_resources 630 save_modules = set(sys.modules) 631 632 def accumulate_result(test, result): 633 ok, test_time = result 634 if ok not in (CHILD_ERROR, INTERRUPTED): 635 test_times.append((test_time, test)) 636 if ok == PASSED: 637 good.append(test) 638 elif ok in (FAILED, CHILD_ERROR): 639 bad.append(test) 640 elif ok == ENV_CHANGED: 641 environment_changed.append(test) 642 elif ok == SKIPPED: 643 skipped.append(test) 644 elif ok == RESOURCE_DENIED: 645 skipped.append(test) 646 resource_denieds.append(test) 647 elif ok != INTERRUPTED: 648 raise ValueError("invalid test result: %r" % ok) 649 650 if forever: 651 def test_forever(tests=list(selected)): 652 while True: 653 for test in tests: 654 yield test 655 if bad: 656 return 657 if fail_env_changed and environment_changed: 658 return 659 tests = test_forever() 660 test_count = '' 661 test_count_width = 3 662 else: 663 tests = iter(selected) 664 test_count = '/{}'.format(len(selected)) 665 test_count_width = len(test_count) - 1 666 667 def display_progress(test_index, test): 668 # "[ 51/405/1] test_tcl" 669 line = "{1:{0}}{2}".format(test_count_width, test_index, test_count) 670 fails = len(bad) + len(environment_changed) 671 if fails and not pgo: 672 line = '{}/{}'.format(line, fails) 673 line = '[{}]'.format(line) 674 675 # add the system load prefix: "load avg: 1.80 " 676 if hasattr(os, 'getloadavg'): 677 load_avg_1min = os.getloadavg()[0] 678 line = "load avg: {:.2f} {}".format(load_avg_1min, line) 679 680 # add the timestamp prefix: "0:01:05 " 681 test_time = time.time() - regrtest_start_time 682 test_time = datetime.timedelta(seconds=int(test_time)) 683 line = "%s %s" % (test_time, line) 684 685 # add the test name 686 line = "{} {}".format(line, test) 687 688 print(line) 689 sys.stdout.flush() 690 691 # For a partial run, we do not need to clutter the output. 692 if display_header: 693 # Print basic platform information 694 print "==", platform.python_implementation(), \ 695 " ".join(sys.version.split()) 696 print "== ", platform.platform(aliased=True), \ 697 "%s-endian" % sys.byteorder 698 print "== ", os.getcwd() 699 ncpu = cpu_count() 700 if ncpu: 701 print "== CPU count:", ncpu 702 703 if huntrleaks: 704 warmup, repetitions, _ = huntrleaks 705 if warmup < 3: 706 print("WARNING: Running tests with --huntrleaks/-R and less than " 707 "3 warmup repetitions can give false positives!") 708 709 if randomize: 710 random.seed(random_seed) 711 print "Using random seed", random_seed 712 random.shuffle(selected) 713 714 if use_mp: 715 try: 716 from threading import Thread 717 except ImportError: 718 print "Multiprocess option requires thread support" 719 sys.exit(2) 720 from Queue import Queue, Empty 721 from subprocess import Popen, PIPE 722 debug_output_pat = re.compile(r"\[\d+ refs\]$") 723 output = Queue() 724 def tests_and_args(): 725 for test in tests: 726 args_tuple = ( 727 (test, verbose, quiet), 728 dict(huntrleaks=huntrleaks, use_resources=use_resources, 729 failfast=failfast, 730 match_tests=match_tests, 731 pgo=pgo) 732 ) 733 yield (test, args_tuple) 734 pending = tests_and_args() 735 opt_args = support.args_from_interpreter_flags() 736 base_cmd = [sys.executable] + opt_args + ['-m', 'test.regrtest'] 737 # required to spawn a new process with PGO flag on/off 738 if pgo: 739 base_cmd = base_cmd + ['--pgo'] 740 741 class MultiprocessThread(Thread): 742 current_test = None 743 start_time = None 744 745 def runtest(self): 746 try: 747 test, args_tuple = next(pending) 748 except StopIteration: 749 output.put((None, None, None, None)) 750 return True 751 752 # -E is needed by some tests, e.g. test_import 753 args = base_cmd + ['--slaveargs', json.dumps(args_tuple)] 754 if testdir: 755 args.extend(('--testdir', testdir)) 756 try: 757 self.start_time = time.time() 758 self.current_test = test 759 popen = Popen(args, 760 stdout=PIPE, stderr=PIPE, 761 universal_newlines=True, 762 close_fds=(os.name != 'nt')) 763 stdout, stderr = popen.communicate() 764 retcode = popen.wait() 765 finally: 766 self.current_test = None 767 768 # Strip last refcount output line if it exists, since it 769 # comes from the shutdown of the interpreter in the subcommand. 770 stderr = debug_output_pat.sub("", stderr) 771 772 if retcode == 0: 773 stdout, _, result = stdout.strip().rpartition("\n") 774 if not result: 775 output.put((None, None, None, None)) 776 return True 777 778 result = json.loads(result) 779 else: 780 result = (CHILD_ERROR, "Exit code %s" % retcode) 781 782 output.put((test, stdout.rstrip(), stderr.rstrip(), result)) 783 return False 784 785 def run(self): 786 try: 787 stop = False 788 while not stop: 789 stop = self.runtest() 790 except BaseException: 791 output.put((None, None, None, None)) 792 raise 793 794 workers = [MultiprocessThread() for i in range(use_mp)] 795 print("Run tests in parallel using %s child processes" 796 % len(workers)) 797 for worker in workers: 798 worker.start() 799 800 def get_running(workers): 801 running = [] 802 for worker in workers: 803 current_test = worker.current_test 804 if not current_test: 805 continue 806 dt = time.time() - worker.start_time 807 if dt >= PROGRESS_MIN_TIME: 808 running.append('%s (%s)' % (current_test, format_duration(dt))) 809 return running 810 811 finished = 0 812 test_index = 1 813 get_timeout = max(PROGRESS_UPDATE, PROGRESS_MIN_TIME) 814 try: 815 while finished < use_mp: 816 try: 817 item = output.get(timeout=get_timeout) 818 except Empty: 819 running = get_running(workers) 820 if running and not pgo: 821 print('running: %s' % ', '.join(running)) 822 sys.stdout.flush() 823 continue 824 825 test, stdout, stderr, result = item 826 if test is None: 827 finished += 1 828 continue 829 accumulate_result(test, result) 830 if not quiet: 831 ok, test_time = result 832 text = format_test_result(test, ok) 833 if (ok not in (CHILD_ERROR, INTERRUPTED) 834 and test_time >= PROGRESS_MIN_TIME 835 and not pgo): 836 text += ' (%s)' % format_duration(test_time) 837 running = get_running(workers) 838 if running and not pgo: 839 text += ' -- running: %s' % ', '.join(running) 840 display_progress(test_index, text) 841 842 if stdout: 843 print(stdout) 844 sys.stdout.flush() 845 if stderr and not pgo: 846 print >>sys.stderr, stderr 847 sys.stderr.flush() 848 849 if result[0] == INTERRUPTED: 850 assert result[1] == 'KeyboardInterrupt' 851 raise KeyboardInterrupt # What else? 852 853 test_index += 1 854 except KeyboardInterrupt: 855 interrupted = True 856 pending.close() 857 for worker in workers: 858 worker.join() 859 else: 860 print("Run tests sequentially") 861 862 previous_test = None 863 for test_index, test in enumerate(tests, 1): 864 if not quiet: 865 text = test 866 if previous_test: 867 text = '%s -- %s' % (text, previous_test) 868 display_progress(test_index, text) 869 870 def local_runtest(): 871 result = runtest(test, verbose, quiet, huntrleaks, None, pgo, 872 failfast=failfast, 873 match_tests=match_tests, 874 testdir=testdir) 875 accumulate_result(test, result) 876 return result 877 878 start_time = time.time() 879 if trace: 880 # If we're tracing code coverage, then we don't exit with status 881 # if on a false return value from main. 882 ns = dict(locals()) 883 tracer.runctx('result = local_runtest()', 884 globals=globals(), locals=ns) 885 result = ns['result'] 886 else: 887 try: 888 result = local_runtest() 889 if verbose3 and result[0] == FAILED: 890 if not pgo: 891 print "Re-running test %r in verbose mode" % test 892 runtest(test, True, quiet, huntrleaks, None, pgo, 893 testdir=testdir) 894 except KeyboardInterrupt: 895 interrupted = True 896 break 897 except: 898 raise 899 900 test_time = time.time() - start_time 901 previous_test = format_test_result(test, result[0]) 902 if test_time >= PROGRESS_MIN_TIME: 903 previous_test = "%s in %s" % (previous_test, 904 format_duration(test_time)) 905 elif result[0] == PASSED: 906 # be quiet: say nothing if the test passed shortly 907 previous_test = None 908 909 if findleaks: 910 gc.collect() 911 if gc.garbage: 912 print "Warning: test created", len(gc.garbage), 913 print "uncollectable object(s)." 914 # move the uncollectable objects somewhere so we don't see 915 # them again 916 found_garbage.extend(gc.garbage) 917 del gc.garbage[:] 918 919 unload_test_modules(save_modules) 920 921 922 def get_tests_result(): 923 result = [] 924 if bad: 925 result.append("FAILURE") 926 elif fail_env_changed and environment_changed: 927 result.append("ENV CHANGED") 928 929 if interrupted: 930 result.append("INTERRUPTED") 931 932 if not result: 933 result.append("SUCCESS") 934 935 result = ', '.join(result) 936 if first_result: 937 result = '%s then %s' % (first_result, result) 938 return result 939 940 941 def display_result(): 942 # print a newline after ^C 943 print 944 print("== Tests result: %s ==" % get_tests_result()) 945 946 if interrupted and not pgo: 947 print 948 print "Test suite interrupted by signal SIGINT." 949 omitted = set(selected) - set(good) - set(bad) - set(skipped) 950 print count(len(omitted), "test"), "omitted:" 951 printlist(omitted) 952 953 if good and not quiet and not pgo: 954 print 955 if not bad and not skipped and not interrupted and len(good) > 1: 956 print "All", 957 print count(len(good), "test"), "OK." 958 959 if print_slow: 960 test_times.sort(reverse=True) 961 print 962 print "10 slowest tests:" 963 for test_time, test in test_times[:10]: 964 print("- %s: %.1fs" % (test, test_time)) 965 966 if bad and not pgo: 967 print 968 print count(len(bad), "test"), "failed:" 969 printlist(bad) 970 971 if environment_changed and not pgo: 972 print 973 print "{} altered the execution environment:".format( 974 count(len(environment_changed), "test")) 975 printlist(environment_changed) 976 977 if skipped and not quiet and not pgo: 978 print 979 print count(len(skipped), "test"), "skipped:" 980 printlist(skipped) 981 982 e = _ExpectedSkips() 983 plat = sys.platform 984 if e.isvalid(): 985 surprise = set(skipped) - e.getexpected() - set(resource_denieds) 986 if surprise: 987 print count(len(surprise), "skip"), \ 988 "unexpected on", plat + ":" 989 printlist(surprise) 990 else: 991 print "Those skips are all expected on", plat + "." 992 else: 993 print "Ask someone to teach regrtest.py about which tests are" 994 print "expected to get skipped on", plat + "." 995 996 if rerun: 997 print 998 print("%s:" % count(len(rerun), "re-run test")) 999 printlist(rerun) 1000 1001 1002 display_result() 1003 1004 if verbose2 and bad: 1005 first_result = get_tests_result() 1006 1007 print 1008 print "Re-running failed tests in verbose mode" 1009 rerun = bad[:] 1010 for test in rerun: 1011 print "Re-running test %r in verbose mode" % test 1012 sys.stdout.flush() 1013 try: 1014 support.verbose = True 1015 ok = runtest(test, True, quiet, huntrleaks, None, pgo, 1016 match_tests=match_tests, testdir=testdir) 1017 except KeyboardInterrupt: 1018 # print a newline separate from the ^C 1019 print 1020 break 1021 else: 1022 if ok[0] in {PASSED, ENV_CHANGED, SKIPPED, RESOURCE_DENIED}: 1023 bad.remove(test) 1024 else: 1025 if bad: 1026 print count(len(bad), "test"), "failed again:" 1027 printlist(bad) 1028 1029 display_result() 1030 1031 if single: 1032 if next_single_test: 1033 with open(filename, 'w') as fp: 1034 fp.write(next_single_test + '\n') 1035 else: 1036 os.unlink(filename) 1037 1038 if trace: 1039 r = tracer.results() 1040 r.write_results(show_missing=True, summary=True, coverdir=coverdir) 1041 1042 if runleaks: 1043 os.system("leaks %d" % os.getpid()) 1044 1045 print 1046 duration = time.time() - regrtest_start_time 1047 print("Total duration: %s" % format_duration(duration)) 1048 1049 print("Tests result: %s" % get_tests_result()) 1050 1051 if bad: 1052 sys.exit(2) 1053 if interrupted: 1054 sys.exit(130) 1055 if fail_env_changed and environment_changed: 1056 sys.exit(3) 1057 sys.exit(0) 1058 1059 1060STDTESTS = [ 1061 'test_grammar', 1062 'test_opcodes', 1063 'test_dict', 1064 'test_builtin', 1065 'test_exceptions', 1066 'test_types', 1067 'test_unittest', 1068 'test_doctest', 1069 'test_doctest2', 1070] 1071 1072NOTTESTS = { 1073 'test_support', 1074 'test_future1', 1075 'test_future2', 1076} 1077 1078def findtests(testdir=None, stdtests=STDTESTS, nottests=NOTTESTS): 1079 """Return a list of all applicable test modules.""" 1080 testdir = findtestdir(testdir) 1081 names = os.listdir(testdir) 1082 tests = [] 1083 others = set(stdtests) | nottests 1084 for name in names: 1085 modname, ext = os.path.splitext(name) 1086 if modname[:5] == "test_" and ext == ".py" and modname not in others: 1087 tests.append(modname) 1088 return stdtests + sorted(tests) 1089 1090def runtest(test, verbose, quiet, 1091 huntrleaks=False, use_resources=None, pgo=False, 1092 failfast=False, match_tests=None, testdir=None): 1093 """Run a single test. 1094 1095 test -- the name of the test 1096 verbose -- if true, print more messages 1097 quiet -- if true, don't print 'skipped' messages (probably redundant) 1098 test_times -- a list of (time, test_name) pairs 1099 huntrleaks -- run multiple times to test for leaks; requires a debug 1100 build; a triple corresponding to -R's three arguments 1101 pgo -- if true, do not print unnecessary info when running the test 1102 for Profile Guided Optimization build 1103 1104 Returns one of the test result constants: 1105 CHILD_ERROR Child process crashed 1106 INTERRUPTED KeyboardInterrupt when run under -j 1107 RESOURCE_DENIED test skipped because resource denied 1108 SKIPPED test skipped for some other reason 1109 ENV_CHANGED test failed because it changed the execution environment 1110 FAILED test failed 1111 PASSED test passed 1112 """ 1113 1114 support.verbose = verbose # Tell tests to be moderately quiet 1115 if use_resources is not None: 1116 support.use_resources = use_resources 1117 try: 1118 support.set_match_tests(match_tests) 1119 # reset the environment_altered flag to detect if a test altered 1120 # the environment 1121 support.environment_altered = False 1122 if failfast: 1123 support.failfast = True 1124 1125 return runtest_inner(test, verbose, quiet, huntrleaks, pgo, testdir) 1126 finally: 1127 cleanup_test_droppings(test, verbose) 1128 1129 1130# Unit tests are supposed to leave the execution environment unchanged 1131# once they complete. But sometimes tests have bugs, especially when 1132# tests fail, and the changes to environment go on to mess up other 1133# tests. This can cause issues with buildbot stability, since tests 1134# are run in random order and so problems may appear to come and go. 1135# There are a few things we can save and restore to mitigate this, and 1136# the following context manager handles this task. 1137 1138class saved_test_environment: 1139 """Save bits of the test environment and restore them at block exit. 1140 1141 with saved_test_environment(testname, verbose, quiet): 1142 #stuff 1143 1144 Unless quiet is True, a warning is printed to stderr if any of 1145 the saved items was changed by the test. The attribute 'changed' 1146 is initially False, but is set to True if a change is detected. 1147 1148 If verbose is more than 1, the before and after state of changed 1149 items is also printed. 1150 """ 1151 1152 changed = False 1153 1154 def __init__(self, testname, verbose=0, quiet=False, pgo=False): 1155 self.testname = testname 1156 self.verbose = verbose 1157 self.quiet = quiet 1158 self.pgo = pgo 1159 1160 # To add things to save and restore, add a name XXX to the resources list 1161 # and add corresponding get_XXX/restore_XXX functions. get_XXX should 1162 # return the value to be saved and compared against a second call to the 1163 # get function when test execution completes. restore_XXX should accept 1164 # the saved value and restore the resource using it. It will be called if 1165 # and only if a change in the value is detected. 1166 # 1167 # Note: XXX will have any '.' replaced with '_' characters when determining 1168 # the corresponding method names. 1169 1170 resources = ('sys.argv', 'cwd', 'sys.stdin', 'sys.stdout', 'sys.stderr', 1171 'os.environ', 'sys.path', 'asyncore.socket_map', 1172 'files', 1173 ) 1174 1175 def get_sys_argv(self): 1176 return id(sys.argv), sys.argv, sys.argv[:] 1177 def restore_sys_argv(self, saved_argv): 1178 sys.argv = saved_argv[1] 1179 sys.argv[:] = saved_argv[2] 1180 1181 def get_cwd(self): 1182 return os.getcwd() 1183 def restore_cwd(self, saved_cwd): 1184 os.chdir(saved_cwd) 1185 1186 def get_sys_stdout(self): 1187 return sys.stdout 1188 def restore_sys_stdout(self, saved_stdout): 1189 sys.stdout = saved_stdout 1190 1191 def get_sys_stderr(self): 1192 return sys.stderr 1193 def restore_sys_stderr(self, saved_stderr): 1194 sys.stderr = saved_stderr 1195 1196 def get_sys_stdin(self): 1197 return sys.stdin 1198 def restore_sys_stdin(self, saved_stdin): 1199 sys.stdin = saved_stdin 1200 1201 def get_os_environ(self): 1202 return id(os.environ), os.environ, dict(os.environ) 1203 def restore_os_environ(self, saved_environ): 1204 os.environ = saved_environ[1] 1205 os.environ.clear() 1206 os.environ.update(saved_environ[2]) 1207 1208 def get_sys_path(self): 1209 return id(sys.path), sys.path, sys.path[:] 1210 def restore_sys_path(self, saved_path): 1211 sys.path = saved_path[1] 1212 sys.path[:] = saved_path[2] 1213 1214 def get_asyncore_socket_map(self): 1215 asyncore = sys.modules.get('asyncore') 1216 # XXX Making a copy keeps objects alive until __exit__ gets called. 1217 return asyncore and asyncore.socket_map.copy() or {} 1218 def restore_asyncore_socket_map(self, saved_map): 1219 asyncore = sys.modules.get('asyncore') 1220 if asyncore is not None: 1221 asyncore.close_all(ignore_all=True) 1222 asyncore.socket_map.update(saved_map) 1223 1224 def get_support_TESTFN(self): 1225 if os.path.isfile(support.TESTFN): 1226 result = 'f' 1227 elif os.path.isdir(support.TESTFN): 1228 result = 'd' 1229 else: 1230 result = None 1231 return result 1232 def restore_support_TESTFN(self, saved_value): 1233 if saved_value is None: 1234 if os.path.isfile(support.TESTFN): 1235 os.unlink(support.TESTFN) 1236 elif os.path.isdir(support.TESTFN): 1237 shutil.rmtree(support.TESTFN) 1238 1239 def get_files(self): 1240 return sorted(fn + ('/' if os.path.isdir(fn) else '') 1241 for fn in os.listdir(os.curdir)) 1242 def restore_files(self, saved_value): 1243 fn = support.TESTFN 1244 if fn not in saved_value and (fn + '/') not in saved_value: 1245 if os.path.isfile(fn): 1246 support.unlink(fn) 1247 elif os.path.isdir(fn): 1248 support.rmtree(fn) 1249 1250 def resource_info(self): 1251 for name in self.resources: 1252 method_suffix = name.replace('.', '_') 1253 get_name = 'get_' + method_suffix 1254 restore_name = 'restore_' + method_suffix 1255 yield name, getattr(self, get_name), getattr(self, restore_name) 1256 1257 def __enter__(self): 1258 self.saved_values = dict((name, get()) for name, get, restore 1259 in self.resource_info()) 1260 return self 1261 1262 def __exit__(self, exc_type, exc_val, exc_tb): 1263 saved_values = self.saved_values 1264 del self.saved_values 1265 1266 # Read support.environment_altered, set by support helper functions 1267 self.changed |= support.environment_altered 1268 1269 for name, get, restore in self.resource_info(): 1270 current = get() 1271 original = saved_values.pop(name) 1272 # Check for changes to the resource's value 1273 if current != original: 1274 self.changed = True 1275 restore(original) 1276 if not self.quiet and not self.pgo: 1277 print >>sys.stderr, ( 1278 "Warning -- {} was modified by {}".format( 1279 name, self.testname)) 1280 print >>sys.stderr, ( 1281 " Before: {}\n After: {} ".format( 1282 original, current)) 1283 # XXX (ncoghlan): for most resources (e.g. sys.path) identity 1284 # matters at least as much as value. For others (e.g. cwd), 1285 # identity is irrelevant. Should we add a mechanism to check 1286 # for substitution in the cases where it matters? 1287 return False 1288 1289 1290def post_test_cleanup(): 1291 support.reap_children() 1292 1293def runtest_inner(test, verbose, quiet, huntrleaks=False, pgo=False, testdir=None): 1294 support.unload(test) 1295 if verbose: 1296 capture_stdout = None 1297 else: 1298 capture_stdout = StringIO.StringIO() 1299 1300 test_time = 0.0 1301 refleak = False # True if the test leaked references. 1302 try: 1303 save_stdout = sys.stdout 1304 try: 1305 if capture_stdout: 1306 sys.stdout = capture_stdout 1307 abstest = get_abs_module(testdir, test) 1308 clear_caches() 1309 with saved_test_environment(test, verbose, quiet, pgo) as environment: 1310 start_time = time.time() 1311 the_package = __import__(abstest, globals(), locals(), []) 1312 if abstest.startswith('test.'): 1313 the_module = getattr(the_package, test) 1314 else: 1315 the_module = the_package 1316 # Old tests run to completion simply as a side-effect of 1317 # being imported. For tests based on unittest or doctest, 1318 # explicitly invoke their test_main() function (if it exists). 1319 indirect_test = getattr(the_module, "test_main", None) 1320 if huntrleaks: 1321 refleak = dash_R(the_module, test, indirect_test, 1322 huntrleaks) 1323 else: 1324 if indirect_test is not None: 1325 indirect_test() 1326 test_time = time.time() - start_time 1327 post_test_cleanup() 1328 finally: 1329 sys.stdout = save_stdout 1330 except support.ResourceDenied, msg: 1331 if not quiet and not pgo: 1332 print test, "skipped --", msg 1333 sys.stdout.flush() 1334 return RESOURCE_DENIED, test_time 1335 except unittest.SkipTest, msg: 1336 if not quiet and not pgo: 1337 print test, "skipped --", msg 1338 sys.stdout.flush() 1339 return SKIPPED, test_time 1340 except KeyboardInterrupt: 1341 raise 1342 except support.TestFailed, msg: 1343 if not pgo: 1344 print >>sys.stderr, "test", test, "failed --", msg 1345 sys.stderr.flush() 1346 return FAILED, test_time 1347 except: 1348 type, value = sys.exc_info()[:2] 1349 if not pgo: 1350 print >>sys.stderr, "test", test, "crashed --", str(type) + ":", value 1351 sys.stderr.flush() 1352 if verbose and not pgo: 1353 traceback.print_exc(file=sys.stderr) 1354 sys.stderr.flush() 1355 return FAILED, test_time 1356 else: 1357 if refleak: 1358 return FAILED, test_time 1359 if environment.changed: 1360 return ENV_CHANGED, test_time 1361 # Except in verbose mode, tests should not print anything 1362 if verbose or huntrleaks: 1363 return PASSED, test_time 1364 output = capture_stdout.getvalue() 1365 if not output: 1366 return PASSED, test_time 1367 print "test", test, "produced unexpected output:" 1368 print "*" * 70 1369 print output 1370 print "*" * 70 1371 sys.stdout.flush() 1372 return FAILED, test_time 1373 1374def cleanup_test_droppings(testname, verbose): 1375 import stat 1376 import gc 1377 1378 # First kill any dangling references to open files etc. 1379 gc.collect() 1380 1381 # Try to clean up junk commonly left behind. While tests shouldn't leave 1382 # any files or directories behind, when a test fails that can be tedious 1383 # for it to arrange. The consequences can be especially nasty on Windows, 1384 # since if a test leaves a file open, it cannot be deleted by name (while 1385 # there's nothing we can do about that here either, we can display the 1386 # name of the offending test, which is a real help). 1387 for name in (support.TESTFN, 1388 "db_home", 1389 ): 1390 if not os.path.exists(name): 1391 continue 1392 1393 if os.path.isdir(name): 1394 kind, nuker = "directory", shutil.rmtree 1395 elif os.path.isfile(name): 1396 kind, nuker = "file", os.unlink 1397 else: 1398 raise SystemError("os.path says %r exists but is neither " 1399 "directory nor file" % name) 1400 1401 if verbose: 1402 print "%r left behind %s %r" % (testname, kind, name) 1403 try: 1404 # if we have chmod, fix possible permissions problems 1405 # that might prevent cleanup 1406 if (hasattr(os, 'chmod')): 1407 os.chmod(name, stat.S_IRWXU | stat.S_IRWXG | stat.S_IRWXO) 1408 nuker(name) 1409 except Exception, msg: 1410 print >> sys.stderr, ("%r left behind %s %r and it couldn't be " 1411 "removed: %s" % (testname, kind, name, msg)) 1412 1413def dash_R(the_module, test, indirect_test, huntrleaks): 1414 """Run a test multiple times, looking for reference leaks. 1415 1416 Returns: 1417 False if the test didn't leak references; True if we detected refleaks. 1418 """ 1419 # This code is hackish and inelegant, but it seems to do the job. 1420 import copy_reg, _abcoll, _pyio 1421 1422 if not hasattr(sys, 'gettotalrefcount'): 1423 raise Exception("Tracking reference leaks requires a debug build " 1424 "of Python") 1425 1426 # Save current values for dash_R_cleanup() to restore. 1427 fs = warnings.filters[:] 1428 ps = copy_reg.dispatch_table.copy() 1429 pic = sys.path_importer_cache.copy() 1430 try: 1431 import zipimport 1432 except ImportError: 1433 zdc = None # Run unmodified on platforms without zipimport support 1434 else: 1435 zdc = zipimport._zip_directory_cache.copy() 1436 abcs = {} 1437 modules = _abcoll, _pyio 1438 for abc in [getattr(mod, a) for mod in modules for a in mod.__all__]: 1439 # XXX isinstance(abc, ABCMeta) leads to infinite recursion 1440 if not hasattr(abc, '_abc_registry'): 1441 continue 1442 for obj in abc.__subclasses__() + [abc]: 1443 abcs[obj] = obj._abc_registry.copy() 1444 1445 if indirect_test: 1446 def run_the_test(): 1447 indirect_test() 1448 else: 1449 def run_the_test(): 1450 imp.reload(the_module) 1451 1452 deltas = [] 1453 nwarmup, ntracked, fname = huntrleaks 1454 fname = os.path.join(support.SAVEDCWD, fname) 1455 repcount = nwarmup + ntracked 1456 rc_deltas = [0] * ntracked 1457 fd_deltas = [0] * ntracked 1458 1459 print >> sys.stderr, "beginning", repcount, "repetitions" 1460 print >> sys.stderr, ("1234567890"*(repcount//10 + 1))[:repcount] 1461 dash_R_cleanup(fs, ps, pic, zdc, abcs) 1462 # initialize variables to make pyflakes quiet 1463 rc_before = fd_before = 0 1464 for i in range(repcount): 1465 run_the_test() 1466 sys.stderr.write('.') 1467 dash_R_cleanup(fs, ps, pic, zdc, abcs) 1468 rc_after = sys.gettotalrefcount() 1469 fd_after = support.fd_count() 1470 if i >= nwarmup: 1471 rc_deltas[i - nwarmup] = rc_after - rc_before 1472 fd_deltas[i - nwarmup] = fd_after - fd_before 1473 rc_before = rc_after 1474 fd_before = fd_after 1475 print >> sys.stderr 1476 1477 # These checkers return False on success, True on failure 1478 def check_rc_deltas(deltas): 1479 # Checker for reference counters and memomry blocks. 1480 # 1481 # bpo-30776: Try to ignore false positives: 1482 # 1483 # [3, 0, 0] 1484 # [0, 1, 0] 1485 # [8, -8, 1] 1486 # 1487 # Expected leaks: 1488 # 1489 # [5, 5, 6] 1490 # [10, 1, 1] 1491 return all(delta >= 1 for delta in deltas) 1492 1493 def check_fd_deltas(deltas): 1494 return any(deltas) 1495 1496 failed = False 1497 for deltas, item_name, checker in [ 1498 (rc_deltas, 'references', check_rc_deltas), 1499 (fd_deltas, 'file descriptors', check_fd_deltas) 1500 ]: 1501 if checker(deltas): 1502 msg = '%s leaked %s %s, sum=%s' % (test, deltas, item_name, sum(deltas)) 1503 print >> sys.stderr, msg 1504 with open(fname, "a") as refrep: 1505 print >> refrep, msg 1506 refrep.flush() 1507 failed = True 1508 return failed 1509 1510def dash_R_cleanup(fs, ps, pic, zdc, abcs): 1511 import gc, copy_reg 1512 1513 # Restore some original values. 1514 warnings.filters[:] = fs 1515 copy_reg.dispatch_table.clear() 1516 copy_reg.dispatch_table.update(ps) 1517 sys.path_importer_cache.clear() 1518 sys.path_importer_cache.update(pic) 1519 try: 1520 import zipimport 1521 except ImportError: 1522 pass # Run unmodified on platforms without zipimport support 1523 else: 1524 zipimport._zip_directory_cache.clear() 1525 zipimport._zip_directory_cache.update(zdc) 1526 1527 # clear type cache 1528 sys._clear_type_cache() 1529 1530 # Clear ABC registries, restoring previously saved ABC registries. 1531 for abc, registry in abcs.items(): 1532 abc._abc_registry = registry.copy() 1533 abc._abc_cache.clear() 1534 abc._abc_negative_cache.clear() 1535 1536 clear_caches() 1537 1538def clear_caches(): 1539 import gc 1540 1541 # Clear the warnings registry, so they can be displayed again 1542 for mod in sys.modules.values(): 1543 if hasattr(mod, '__warningregistry__'): 1544 del mod.__warningregistry__ 1545 1546 # Clear assorted module caches. 1547 # Don't worry about resetting the cache if the module is not loaded 1548 try: 1549 distutils_dir_util = sys.modules['distutils.dir_util'] 1550 except KeyError: 1551 pass 1552 else: 1553 distutils_dir_util._path_created.clear() 1554 1555 re.purge() 1556 1557 try: 1558 _strptime = sys.modules['_strptime'] 1559 except KeyError: 1560 pass 1561 else: 1562 _strptime._regex_cache.clear() 1563 1564 try: 1565 urlparse = sys.modules['urlparse'] 1566 except KeyError: 1567 pass 1568 else: 1569 urlparse.clear_cache() 1570 1571 try: 1572 urllib = sys.modules['urllib'] 1573 except KeyError: 1574 pass 1575 else: 1576 urllib.urlcleanup() 1577 1578 try: 1579 urllib2 = sys.modules['urllib2'] 1580 except KeyError: 1581 pass 1582 else: 1583 urllib2.install_opener(None) 1584 1585 try: 1586 dircache = sys.modules['dircache'] 1587 except KeyError: 1588 pass 1589 else: 1590 dircache.reset() 1591 1592 try: 1593 linecache = sys.modules['linecache'] 1594 except KeyError: 1595 pass 1596 else: 1597 linecache.clearcache() 1598 1599 try: 1600 mimetypes = sys.modules['mimetypes'] 1601 except KeyError: 1602 pass 1603 else: 1604 mimetypes._default_mime_types() 1605 1606 try: 1607 filecmp = sys.modules['filecmp'] 1608 except KeyError: 1609 pass 1610 else: 1611 filecmp._cache.clear() 1612 1613 try: 1614 struct = sys.modules['struct'] 1615 except KeyError: 1616 pass 1617 else: 1618 struct._clearcache() 1619 1620 try: 1621 doctest = sys.modules['doctest'] 1622 except KeyError: 1623 pass 1624 else: 1625 doctest.master = None 1626 1627 try: 1628 ctypes = sys.modules['ctypes'] 1629 except KeyError: 1630 pass 1631 else: 1632 ctypes._reset_cache() 1633 1634 # Collect cyclic trash. 1635 gc.collect() 1636 1637def warm_caches(): 1638 """Create explicitly internal singletons which are created on demand 1639 to prevent false positive when hunting reference leaks.""" 1640 # char cache 1641 for i in range(256): 1642 chr(i) 1643 # unicode cache 1644 for i in range(256): 1645 unichr(i) 1646 # int cache 1647 list(range(-5, 257)) 1648 1649def findtestdir(path=None): 1650 return path or os.path.dirname(__file__) or os.curdir 1651 1652def removepy(names): 1653 if not names: 1654 return 1655 for idx, name in enumerate(names): 1656 basename, ext = os.path.splitext(name) 1657 if ext == '.py': 1658 names[idx] = basename 1659 1660def count(n, word): 1661 if n == 1: 1662 return "%d %s" % (n, word) 1663 else: 1664 return "%d %ss" % (n, word) 1665 1666def printlist(x, width=70, indent=4, file=None): 1667 """Print the elements of iterable x to stdout. 1668 1669 Optional arg width (default 70) is the maximum line length. 1670 Optional arg indent (default 4) is the number of blanks with which to 1671 begin each line. 1672 """ 1673 1674 from textwrap import fill 1675 blanks = ' ' * indent 1676 # Print the sorted list: 'x' may be a '--random' list or a set() 1677 print >>file, fill(' '.join(str(elt) for elt in sorted(x)), width, 1678 initial_indent=blanks, subsequent_indent=blanks) 1679 1680def get_abs_module(testdir, test): 1681 if test.startswith('test.') or testdir: 1682 return test 1683 else: 1684 # Always import it from the test package 1685 return 'test.' + test 1686 1687def _list_cases(suite): 1688 for test in suite: 1689 if isinstance(test, unittest.TestSuite): 1690 _list_cases(test) 1691 elif isinstance(test, unittest.TestCase): 1692 if support.match_test(test): 1693 print(test.id()) 1694 1695def list_cases(testdir, selected, match_tests): 1696 support.verbose = False 1697 support.set_match_tests(match_tests) 1698 1699 save_modules = set(sys.modules) 1700 skipped = [] 1701 for test in selected: 1702 abstest = get_abs_module(testdir, test) 1703 try: 1704 suite = unittest.defaultTestLoader.loadTestsFromName(abstest) 1705 _list_cases(suite) 1706 except unittest.SkipTest: 1707 skipped.append(test) 1708 1709 unload_test_modules(save_modules) 1710 1711 if skipped: 1712 print >>sys.stderr 1713 print >>sys.stderr, count(len(skipped), "test"), "skipped:" 1714 printlist(skipped, file=sys.stderr) 1715 1716# Map sys.platform to a string containing the basenames of tests 1717# expected to be skipped on that platform. 1718# 1719# Special cases: 1720# test_pep277 1721# The _ExpectedSkips constructor adds this to the set of expected 1722# skips if not os.path.supports_unicode_filenames. 1723# test_timeout 1724# Controlled by test_timeout.skip_expected. Requires the network 1725# resource and a socket module. 1726# 1727# Tests that are expected to be skipped everywhere except on one platform 1728# are also handled separately. 1729 1730_expectations = { 1731 'win32': 1732 """ 1733 test__locale 1734 test_bsddb185 1735 test_bsddb3 1736 test_commands 1737 test_crypt 1738 test_curses 1739 test_dbm 1740 test_dl 1741 test_fcntl 1742 test_fork1 1743 test_epoll 1744 test_gdbm 1745 test_grp 1746 test_ioctl 1747 test_largefile 1748 test_kqueue 1749 test_mhlib 1750 test_openpty 1751 test_ossaudiodev 1752 test_pipes 1753 test_poll 1754 test_posix 1755 test_pty 1756 test_pwd 1757 test_resource 1758 test_signal 1759 test_spwd 1760 test_threadsignals 1761 test_timing 1762 test_wait3 1763 test_wait4 1764 """, 1765 'linux2': 1766 """ 1767 test_bsddb185 1768 test_curses 1769 test_dl 1770 test_largefile 1771 test_kqueue 1772 test_ossaudiodev 1773 """, 1774 'unixware7': 1775 """ 1776 test_bsddb 1777 test_bsddb185 1778 test_dl 1779 test_epoll 1780 test_largefile 1781 test_kqueue 1782 test_minidom 1783 test_openpty 1784 test_pyexpat 1785 test_sax 1786 test_sundry 1787 """, 1788 'openunix8': 1789 """ 1790 test_bsddb 1791 test_bsddb185 1792 test_dl 1793 test_epoll 1794 test_largefile 1795 test_kqueue 1796 test_minidom 1797 test_openpty 1798 test_pyexpat 1799 test_sax 1800 test_sundry 1801 """, 1802 'sco_sv3': 1803 """ 1804 test_asynchat 1805 test_bsddb 1806 test_bsddb185 1807 test_dl 1808 test_fork1 1809 test_epoll 1810 test_gettext 1811 test_largefile 1812 test_locale 1813 test_kqueue 1814 test_minidom 1815 test_openpty 1816 test_pyexpat 1817 test_queue 1818 test_sax 1819 test_sundry 1820 test_thread 1821 test_threaded_import 1822 test_threadedtempfile 1823 test_threading 1824 """, 1825 'riscos': 1826 """ 1827 test_asynchat 1828 test_atexit 1829 test_bsddb 1830 test_bsddb185 1831 test_bsddb3 1832 test_commands 1833 test_crypt 1834 test_dbm 1835 test_dl 1836 test_fcntl 1837 test_fork1 1838 test_epoll 1839 test_gdbm 1840 test_grp 1841 test_largefile 1842 test_locale 1843 test_kqueue 1844 test_mmap 1845 test_openpty 1846 test_poll 1847 test_popen2 1848 test_pty 1849 test_pwd 1850 test_strop 1851 test_sundry 1852 test_thread 1853 test_threaded_import 1854 test_threadedtempfile 1855 test_threading 1856 test_timing 1857 """, 1858 'darwin': 1859 """ 1860 test__locale 1861 test_bsddb 1862 test_bsddb3 1863 test_curses 1864 test_epoll 1865 test_gdb 1866 test_gdbm 1867 test_largefile 1868 test_locale 1869 test_kqueue 1870 test_minidom 1871 test_ossaudiodev 1872 test_poll 1873 """, 1874 'sunos5': 1875 """ 1876 test_bsddb 1877 test_bsddb185 1878 test_curses 1879 test_dbm 1880 test_epoll 1881 test_kqueue 1882 test_gdbm 1883 test_gzip 1884 test_openpty 1885 test_zipfile 1886 test_zlib 1887 """, 1888 'hp-ux11': 1889 """ 1890 test_bsddb 1891 test_bsddb185 1892 test_curses 1893 test_dl 1894 test_epoll 1895 test_gdbm 1896 test_gzip 1897 test_largefile 1898 test_locale 1899 test_kqueue 1900 test_minidom 1901 test_openpty 1902 test_pyexpat 1903 test_sax 1904 test_zipfile 1905 test_zlib 1906 """, 1907 'atheos': 1908 """ 1909 test_bsddb185 1910 test_curses 1911 test_dl 1912 test_gdbm 1913 test_epoll 1914 test_largefile 1915 test_locale 1916 test_kqueue 1917 test_mhlib 1918 test_mmap 1919 test_poll 1920 test_popen2 1921 test_resource 1922 """, 1923 'cygwin': 1924 """ 1925 test_bsddb185 1926 test_bsddb3 1927 test_curses 1928 test_dbm 1929 test_epoll 1930 test_ioctl 1931 test_kqueue 1932 test_largefile 1933 test_locale 1934 test_ossaudiodev 1935 test_socketserver 1936 """, 1937 'os2emx': 1938 """ 1939 test_audioop 1940 test_bsddb185 1941 test_bsddb3 1942 test_commands 1943 test_curses 1944 test_dl 1945 test_epoll 1946 test_kqueue 1947 test_largefile 1948 test_mhlib 1949 test_mmap 1950 test_openpty 1951 test_ossaudiodev 1952 test_pty 1953 test_resource 1954 test_signal 1955 """, 1956 'freebsd4': 1957 """ 1958 test_bsddb 1959 test_bsddb3 1960 test_epoll 1961 test_gdbm 1962 test_locale 1963 test_ossaudiodev 1964 test_pep277 1965 test_pty 1966 test_socketserver 1967 test_tcl 1968 test_tk 1969 test_ttk_guionly 1970 test_ttk_textonly 1971 test_timeout 1972 test_urllibnet 1973 test_multiprocessing 1974 """, 1975 'aix5': 1976 """ 1977 test_bsddb 1978 test_bsddb185 1979 test_bsddb3 1980 test_bz2 1981 test_dl 1982 test_epoll 1983 test_gdbm 1984 test_gzip 1985 test_kqueue 1986 test_ossaudiodev 1987 test_tcl 1988 test_tk 1989 test_ttk_guionly 1990 test_ttk_textonly 1991 test_zipimport 1992 test_zlib 1993 """, 1994 'openbsd3': 1995 """ 1996 test_ascii_formatd 1997 test_bsddb 1998 test_bsddb3 1999 test_ctypes 2000 test_dl 2001 test_epoll 2002 test_gdbm 2003 test_locale 2004 test_normalization 2005 test_ossaudiodev 2006 test_pep277 2007 test_tcl 2008 test_tk 2009 test_ttk_guionly 2010 test_ttk_textonly 2011 test_multiprocessing 2012 """, 2013 'netbsd3': 2014 """ 2015 test_ascii_formatd 2016 test_bsddb 2017 test_bsddb185 2018 test_bsddb3 2019 test_ctypes 2020 test_curses 2021 test_dl 2022 test_epoll 2023 test_gdbm 2024 test_locale 2025 test_ossaudiodev 2026 test_pep277 2027 test_tcl 2028 test_tk 2029 test_ttk_guionly 2030 test_ttk_textonly 2031 test_multiprocessing 2032 """, 2033} 2034_expectations['freebsd5'] = _expectations['freebsd4'] 2035_expectations['freebsd6'] = _expectations['freebsd4'] 2036_expectations['freebsd7'] = _expectations['freebsd4'] 2037_expectations['freebsd8'] = _expectations['freebsd4'] 2038 2039class _ExpectedSkips: 2040 def __init__(self): 2041 import os.path 2042 from test import test_timeout 2043 2044 self.valid = False 2045 if sys.platform in _expectations: 2046 s = _expectations[sys.platform] 2047 self.expected = set(s.split()) 2048 2049 # expected to be skipped on every platform, even Linux 2050 self.expected.add('test_linuxaudiodev') 2051 2052 if not os.path.supports_unicode_filenames: 2053 self.expected.add('test_pep277') 2054 2055 if test_timeout.skip_expected: 2056 self.expected.add('test_timeout') 2057 2058 if sys.maxint == 9223372036854775807L: 2059 self.expected.add('test_imageop') 2060 2061 if sys.platform != "darwin": 2062 MAC_ONLY = ["test_macos", "test_macostools", "test_aepack", 2063 "test_plistlib", "test_scriptpackages", 2064 "test_applesingle"] 2065 for skip in MAC_ONLY: 2066 self.expected.add(skip) 2067 elif len(u'\0'.encode('unicode-internal')) == 4: 2068 self.expected.add("test_macostools") 2069 2070 2071 if sys.platform != "win32": 2072 # test_sqlite is only reliable on Windows where the library 2073 # is distributed with Python 2074 WIN_ONLY = ["test_unicode_file", "test_winreg", 2075 "test_winsound", "test_startfile", 2076 "test_sqlite", "test_msilib"] 2077 for skip in WIN_ONLY: 2078 self.expected.add(skip) 2079 2080 if sys.platform != 'irix': 2081 IRIX_ONLY = ["test_imageop", "test_al", "test_cd", "test_cl", 2082 "test_gl", "test_imgfile"] 2083 for skip in IRIX_ONLY: 2084 self.expected.add(skip) 2085 2086 if sys.platform != 'sunos5': 2087 self.expected.add('test_sunaudiodev') 2088 self.expected.add('test_nis') 2089 2090 if not sys.py3kwarning: 2091 self.expected.add('test_py3kwarn') 2092 2093 self.valid = True 2094 2095 def isvalid(self): 2096 "Return true iff _ExpectedSkips knows about the current platform." 2097 return self.valid 2098 2099 def getexpected(self): 2100 """Return set of test names we expect to skip on current platform. 2101 2102 self.isvalid() must be true. 2103 """ 2104 2105 assert self.isvalid() 2106 return self.expected 2107 2108def main_in_temp_cwd(): 2109 """Run main() in a temporary working directory.""" 2110 global TEMPDIR 2111 2112 # When tests are run from the Python build directory, it is best practice 2113 # to keep the test files in a subfolder. It eases the cleanup of leftover 2114 # files using command "make distclean". 2115 if sysconfig.is_python_build(): 2116 TEMPDIR = os.path.join(sysconfig.get_config_var('srcdir'), 'build') 2117 TEMPDIR = os.path.abspath(TEMPDIR) 2118 if not os.path.exists(TEMPDIR): 2119 os.mkdir(TEMPDIR) 2120 2121 # Define a writable temp dir that will be used as cwd while running 2122 # the tests. The name of the dir includes the pid to allow parallel 2123 # testing (see the -j option). 2124 TESTCWD = 'test_python_{}'.format(os.getpid()) 2125 2126 TESTCWD = os.path.join(TEMPDIR, TESTCWD) 2127 2128 # Run the tests in a context manager that temporary changes the CWD to a 2129 # temporary and writable directory. If it's not possible to create or 2130 # change the CWD, the original CWD will be used. The original CWD is 2131 # available from support.SAVEDCWD. 2132 with support.temp_cwd(TESTCWD, quiet=True): 2133 main() 2134 2135if __name__ == '__main__': 2136 # findtestdir() gets the dirname out of __file__, so we have to make it 2137 # absolute before changing the working directory. 2138 # For example __file__ may be relative when running trace or profile. 2139 # See issue #9323. 2140 global __file__ 2141 __file__ = os.path.abspath(__file__) 2142 2143 # sanity check 2144 assert __file__ == os.path.abspath(sys.argv[0]) 2145 2146 main_in_temp_cwd() 2147