1**************************** 2 What's New In Python 3.7 3**************************** 4 5:Editor: Elvis Pranskevichus <elvis@magic.io> 6 7.. Rules for maintenance: 8 9 * Anyone can add text to this document. Do not spend very much time 10 on the wording of your changes, because your text will probably 11 get rewritten to some degree. 12 13 * The maintainer will go through Misc/NEWS periodically and add 14 changes; it's therefore more important to add your changes to 15 Misc/NEWS than to this file. 16 17 * This is not a complete list of every single change; completeness 18 is the purpose of Misc/NEWS. Some changes I consider too small 19 or esoteric to include. If such a change is added to the text, 20 I'll just remove it. (This is another reason you shouldn't spend 21 too much time on writing your addition.) 22 23 * If you want to draw your new text to the attention of the 24 maintainer, add 'XXX' to the beginning of the paragraph or 25 section. 26 27 * It's OK to just add a fragmentary note about a change. For 28 example: "XXX Describe the transmogrify() function added to the 29 socket module." The maintainer will research the change and 30 write the necessary text. 31 32 * You can comment out your additions if you like, but it's not 33 necessary (especially when a final release is some months away). 34 35 * Credit the author of a patch or bugfix. Just the name is 36 sufficient; the e-mail address isn't necessary. 37 38 * It's helpful to add the bug/patch number as a comment: 39 40 XXX Describe the transmogrify() function added to the socket 41 module. 42 (Contributed by P.Y. Developer in :issue:`12345`.) 43 44 This saves the maintainer the effort of going through the Git log 45 when researching a change. 46 47This article explains the new features in Python 3.7, compared to 3.6. 48Python 3.7 was released on June 27, 2018. 49For full details, see the :ref:`changelog <changelog>`. 50 51 52Summary -- Release Highlights 53============================= 54 55.. This section singles out the most important changes in Python 3.7. 56 Brevity is key. 57 58New syntax features: 59 60* :ref:`PEP 563 <whatsnew37-pep563>`, postponed evaluation of type annotations. 61 62Backwards incompatible syntax changes: 63 64* :keyword:`async` and :keyword:`await` are now reserved keywords. 65 66New library modules: 67 68* :mod:`contextvars`: :ref:`PEP 567 -- Context Variables <whatsnew37-pep567>` 69* :mod:`dataclasses`: :ref:`PEP 557 -- Data Classes <whatsnew37-pep557>` 70* :ref:`whatsnew37_importlib_resources` 71 72New built-in features: 73 74* :ref:`PEP 553 <whatsnew37-pep553>`, the new :func:`breakpoint` function. 75 76Python data model improvements: 77 78* :ref:`PEP 562 <whatsnew37-pep562>`, customization of access to 79 module attributes. 80 81* :ref:`PEP 560 <whatsnew37-pep560>`, core support for typing module and 82 generic types. 83 84* the insertion-order preservation nature of :ref:`dict <typesmapping>` 85 objects `has been declared`_ to be an official 86 part of the Python language spec. 87 88.. _has been declared: https://mail.python.org/pipermail/python-dev/2017-December/151283.html 89 90Significant improvements in the standard library: 91 92* The :mod:`asyncio` module has received new features, significant 93 :ref:`usability and performance improvements <whatsnew37_asyncio>`. 94 95* The :mod:`time` module gained support for 96 :ref:`functions with nanosecond resolution <whatsnew37-pep564>`. 97 98CPython implementation improvements: 99 100* Avoiding the use of ASCII as a default text encoding: 101 102 * :ref:`PEP 538 <whatsnew37-pep538>`, legacy C locale coercion 103 * :ref:`PEP 540 <whatsnew37-pep540>`, forced UTF-8 runtime mode 104* :ref:`PEP 552 <whatsnew37-pep552>`, deterministic .pycs 105* :ref:`New Python Development Mode <whatsnew37-devmode>` 106* :ref:`PEP 565 <whatsnew37-pep565>`, improved :exc:`DeprecationWarning` 107 handling 108 109C API improvements: 110 111* :ref:`PEP 539 <whatsnew37-pep539>`, new C API for thread-local storage 112 113Documentation improvements: 114 115* :ref:`PEP 545 <whatsnew37-pep545>`, Python documentation translations 116* New documentation translations: `Japanese <https://docs.python.org/ja/>`_, 117 `French <https://docs.python.org/fr/>`_, and 118 `Korean <https://docs.python.org/ko/>`_. 119 120This release features notable performance improvements in many areas. 121The :ref:`whatsnew37-perf` section lists them in detail. 122 123For a list of changes that may affect compatibility with previous Python 124releases please refer to the :ref:`porting-to-python-37` section. 125 126 127New Features 128============ 129 130.. _whatsnew37-pep563: 131 132PEP 563: Postponed Evaluation of Annotations 133-------------------------------------------- 134 135The advent of type hints in Python uncovered two glaring usability issues 136with the functionality of annotations added in :pep:`3107` and refined 137further in :pep:`526`: 138 139* annotations could only use names which were already available in the 140 current scope, in other words they didn't support forward references 141 of any kind; and 142 143* annotating source code had adverse effects on startup time of Python 144 programs. 145 146Both of these issues are fixed by postponing the evaluation of 147annotations. Instead of compiling code which executes expressions in 148annotations at their definition time, the compiler stores the annotation 149in a string form equivalent to the AST of the expression in question. 150If needed, annotations can be resolved at runtime using 151:func:`typing.get_type_hints`. In the common case where this is not 152required, the annotations are cheaper to store (since short strings 153are interned by the interpreter) and make startup time faster. 154 155Usability-wise, annotations now support forward references, making the 156following syntax valid:: 157 158 class C: 159 @classmethod 160 def from_string(cls, source: str) -> C: 161 ... 162 163 def validate_b(self, obj: B) -> bool: 164 ... 165 166 class B: 167 ... 168 169Since this change breaks compatibility, the new behavior needs to be enabled 170on a per-module basis in Python 3.7 using a :mod:`__future__` import:: 171 172 from __future__ import annotations 173 174It will become the default in Python 3.10. 175 176.. seealso:: 177 178 :pep:`563` -- Postponed evaluation of annotations 179 PEP written and implemented by Łukasz Langa. 180 181 182.. _whatsnew37-pep538: 183 184PEP 538: Legacy C Locale Coercion 185--------------------------------- 186 187An ongoing challenge within the Python 3 series has been determining a sensible 188default strategy for handling the "7-bit ASCII" text encoding assumption 189currently implied by the use of the default C or POSIX locale on non-Windows 190platforms. 191 192:pep:`538` updates the default interpreter command line interface to 193automatically coerce that locale to an available UTF-8 based locale as 194described in the documentation of the new :envvar:`PYTHONCOERCECLOCALE` 195environment variable. Automatically setting ``LC_CTYPE`` this way means that 196both the core interpreter and locale-aware C extensions (such as 197:mod:`readline`) will assume the use of UTF-8 as the default text encoding, 198rather than ASCII. 199 200The platform support definition in :pep:`11` has also been updated to limit 201full text handling support to suitably configured non-ASCII based locales. 202 203As part of this change, the default error handler for :data:`~sys.stdin` and 204:data:`~sys.stdout` is now ``surrogateescape`` (rather than ``strict``) when 205using any of the defined coercion target locales (currently ``C.UTF-8``, 206``C.utf8``, and ``UTF-8``). The default error handler for :data:`~sys.stderr` 207continues to be ``backslashreplace``, regardless of locale. 208 209Locale coercion is silent by default, but to assist in debugging potentially 210locale related integration problems, explicit warnings (emitted directly on 211:data:`~sys.stderr`) can be requested by setting ``PYTHONCOERCECLOCALE=warn``. 212This setting will also cause the Python runtime to emit a warning if the 213legacy C locale remains active when the core interpreter is initialized. 214 215While :pep:`538`'s locale coercion has the benefit of also affecting extension 216modules (such as GNU ``readline``), as well as child processes (including those 217running non-Python applications and older versions of Python), it has the 218downside of requiring that a suitable target locale be present on the running 219system. To better handle the case where no suitable target locale is available 220(as occurs on RHEL/CentOS 7, for example), Python 3.7 also implements 221:ref:`whatsnew37-pep540`. 222 223.. seealso:: 224 225 :pep:`538` -- Coercing the legacy C locale to a UTF-8 based locale 226 PEP written and implemented by Nick Coghlan. 227 228 229.. _whatsnew37-pep540: 230 231PEP 540: Forced UTF-8 Runtime Mode 232----------------------------------- 233 234The new :option:`-X` ``utf8`` command line option and :envvar:`PYTHONUTF8` 235environment variable can be used to enable the :ref:`Python UTF-8 Mode 236<utf8-mode>`. 237 238When in UTF-8 mode, CPython ignores the locale settings, and uses the 239UTF-8 encoding by default. The error handlers for :data:`sys.stdin` and 240:data:`sys.stdout` streams are set to ``surrogateescape``. 241 242The forced UTF-8 mode can be used to change the text handling behavior in 243an embedded Python interpreter without changing the locale settings of 244an embedding application. 245 246While :pep:`540`'s UTF-8 mode has the benefit of working regardless of which 247locales are available on the running system, it has the downside of having no 248effect on extension modules (such as GNU ``readline``), child processes running 249non-Python applications, and child processes running older versions of Python. 250To reduce the risk of corrupting text data when communicating with such 251components, Python 3.7 also implements :ref:`whatsnew37-pep540`). 252 253The UTF-8 mode is enabled by default when the locale is ``C`` or ``POSIX``, and 254the :pep:`538` locale coercion feature fails to change it to a UTF-8 based 255alternative (whether that failure is due to ``PYTHONCOERCECLOCALE=0`` being set, 256``LC_ALL`` being set, or the lack of a suitable target locale). 257 258.. seealso:: 259 260 :pep:`540` -- Add a new UTF-8 mode 261 PEP written and implemented by Victor Stinner 262 263 264.. _whatsnew37-pep553: 265 266PEP 553: Built-in ``breakpoint()`` 267---------------------------------- 268 269Python 3.7 includes the new built-in :func:`breakpoint` function as 270an easy and consistent way to enter the Python debugger. 271 272Built-in ``breakpoint()`` calls :func:`sys.breakpointhook`. By default, the 273latter imports :mod:`pdb` and then calls ``pdb.set_trace()``, but by binding 274``sys.breakpointhook()`` to the function of your choosing, ``breakpoint()`` can 275enter any debugger. Additionally, the environment variable 276:envvar:`PYTHONBREAKPOINT` can be set to the callable of your debugger of 277choice. Set ``PYTHONBREAKPOINT=0`` to completely disable built-in 278``breakpoint()``. 279 280.. seealso:: 281 282 :pep:`553` -- Built-in breakpoint() 283 PEP written and implemented by Barry Warsaw 284 285 286.. _whatsnew37-pep539: 287 288PEP 539: New C API for Thread-Local Storage 289------------------------------------------- 290 291While Python provides a C API for thread-local storage support; the existing 292:ref:`Thread Local Storage (TLS) API <thread-local-storage-api>` has used 293:c:expr:`int` to represent TLS keys across all platforms. This has not 294generally been a problem for officially support platforms, but that is neither 295POSIX-compliant, nor portable in any practical sense. 296 297:pep:`539` changes this by providing a new :ref:`Thread Specific Storage (TSS) 298API <thread-specific-storage-api>` to CPython which supersedes use of the 299existing TLS API within the CPython interpreter, while deprecating the existing 300API. The TSS API uses a new type :c:type:`Py_tss_t` instead of :c:expr:`int` 301to represent TSS keys--an opaque type the definition of which may depend on 302the underlying TLS implementation. Therefore, this will allow to build CPython 303on platforms where the native TLS key is defined in a way that cannot be safely 304cast to :c:expr:`int`. 305 306Note that on platforms where the native TLS key is defined in a way that cannot 307be safely cast to :c:expr:`int`, all functions of the existing TLS API will be 308no-op and immediately return failure. This indicates clearly that the old API 309is not supported on platforms where it cannot be used reliably, and that no 310effort will be made to add such support. 311 312.. seealso:: 313 314 :pep:`539` -- A New C-API for Thread-Local Storage in CPython 315 PEP written by Erik M. Bray; implementation by Masayuki Yamamoto. 316 317 318.. _whatsnew37-pep562: 319 320PEP 562: Customization of Access to Module Attributes 321----------------------------------------------------- 322 323Python 3.7 allows defining :meth:`__getattr__` on modules and will call 324it whenever a module attribute is otherwise not found. Defining 325:meth:`__dir__` on modules is now also allowed. 326 327A typical example of where this may be useful is module attribute deprecation 328and lazy loading. 329 330.. seealso:: 331 332 :pep:`562` -- Module ``__getattr__`` and ``__dir__`` 333 PEP written and implemented by Ivan Levkivskyi 334 335 336.. _whatsnew37-pep564: 337 338PEP 564: New Time Functions With Nanosecond Resolution 339------------------------------------------------------ 340 341The resolution of clocks in modern systems can exceed the limited precision 342of a floating-point number returned by the :func:`time.time` function 343and its variants. To avoid loss of precision, :pep:`564` adds six new 344"nanosecond" variants of the existing timer functions to the :mod:`time` 345module: 346 347* :func:`time.clock_gettime_ns` 348* :func:`time.clock_settime_ns` 349* :func:`time.monotonic_ns` 350* :func:`time.perf_counter_ns` 351* :func:`time.process_time_ns` 352* :func:`time.time_ns` 353 354The new functions return the number of nanoseconds as an integer value. 355 356:pep:`Measurements <0564#annex-clocks-resolution-in-python>` 357show that on Linux and Windows the resolution of :func:`time.time_ns` is 358approximately 3 times better than that of :func:`time.time`. 359 360.. seealso:: 361 362 :pep:`564` -- Add new time functions with nanosecond resolution 363 PEP written and implemented by Victor Stinner 364 365 366.. _whatsnew37-pep565: 367 368PEP 565: Show DeprecationWarning in ``__main__`` 369------------------------------------------------ 370 371The default handling of :exc:`DeprecationWarning` has been changed such that 372these warnings are once more shown by default, but only when the code 373triggering them is running directly in the :mod:`__main__` module. As a result, 374developers of single file scripts and those using Python interactively should 375once again start seeing deprecation warnings for the APIs they use, but 376deprecation warnings triggered by imported application, library and framework 377modules will continue to be hidden by default. 378 379As a result of this change, the standard library now allows developers to choose 380between three different deprecation warning behaviours: 381 382* :exc:`FutureWarning`: always displayed by default, recommended for warnings 383 intended to be seen by application end users (e.g. for deprecated application 384 configuration settings). 385* :exc:`DeprecationWarning`: displayed by default only in :mod:`__main__` and when 386 running tests, recommended for warnings intended to be seen by other Python 387 developers where a version upgrade may result in changed behaviour or an 388 error. 389* :exc:`PendingDeprecationWarning`: displayed by default only when running 390 tests, intended for cases where a future version upgrade will change the 391 warning category to :exc:`DeprecationWarning` or :exc:`FutureWarning`. 392 393Previously both :exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning` 394were only visible when running tests, which meant that developers primarily 395writing single file scripts or using Python interactively could be surprised 396by breaking changes in the APIs they used. 397 398.. seealso:: 399 400 :pep:`565` -- Show DeprecationWarning in ``__main__`` 401 PEP written and implemented by Nick Coghlan 402 403 404.. _whatsnew37-pep560: 405 406PEP 560: Core Support for ``typing`` module and Generic Types 407------------------------------------------------------------- 408 409Initially :pep:`484` was designed in such way that it would not introduce *any* 410changes to the core CPython interpreter. Now type hints and the :mod:`typing` 411module are extensively used by the community, so this restriction is removed. 412The PEP introduces two special methods :meth:`__class_getitem__` and 413``__mro_entries__``, these methods are now used by most classes and special 414constructs in :mod:`typing`. As a result, the speed of various operations 415with types increased up to 7 times, the generic types can be used without 416metaclass conflicts, and several long standing bugs in :mod:`typing` module are 417fixed. 418 419.. seealso:: 420 421 :pep:`560` -- Core support for typing module and generic types 422 PEP written and implemented by Ivan Levkivskyi 423 424 425.. _whatsnew37-pep552: 426 427PEP 552: Hash-based .pyc Files 428------------------------------ 429 430Python has traditionally checked the up-to-dateness of bytecode cache files 431(i.e., ``.pyc`` files) by comparing the source metadata (last-modified timestamp 432and size) with source metadata saved in the cache file header when it was 433generated. While effective, this invalidation method has its drawbacks. When 434filesystem timestamps are too coarse, Python can miss source updates, leading to 435user confusion. Additionally, having a timestamp in the cache file is 436problematic for `build reproducibility <https://reproducible-builds.org/>`_ and 437content-based build systems. 438 439:pep:`552` extends the pyc format to allow the hash of the source file to be 440used for invalidation instead of the source timestamp. Such ``.pyc`` files are 441called "hash-based". By default, Python still uses timestamp-based invalidation 442and does not generate hash-based ``.pyc`` files at runtime. Hash-based ``.pyc`` 443files may be generated with :mod:`py_compile` or :mod:`compileall`. 444 445Hash-based ``.pyc`` files come in two variants: checked and unchecked. Python 446validates checked hash-based ``.pyc`` files against the corresponding source 447files at runtime but doesn't do so for unchecked hash-based pycs. Unchecked 448hash-based ``.pyc`` files are a useful performance optimization for environments 449where a system external to Python (e.g., the build system) is responsible for 450keeping ``.pyc`` files up-to-date. 451 452See :ref:`pyc-invalidation` for more information. 453 454.. seealso:: 455 456 :pep:`552` -- Deterministic pycs 457 PEP written and implemented by Benjamin Peterson 458 459 460.. _whatsnew37-pep545: 461 462PEP 545: Python Documentation Translations 463------------------------------------------ 464 465:pep:`545` describes the process of creating and maintaining Python 466documentation translations. 467 468Three new translations have been added: 469 470- Japanese: https://docs.python.org/ja/ 471- French: https://docs.python.org/fr/ 472- Korean: https://docs.python.org/ko/ 473 474.. seealso:: 475 476 :pep:`545` -- Python Documentation Translations 477 PEP written and implemented by Julien Palard, Inada Naoki, and 478 Victor Stinner. 479 480 481.. _whatsnew37-devmode: 482 483Python Development Mode (-X dev) 484-------------------------------- 485 486The new :option:`-X` ``dev`` command line option or the new 487:envvar:`PYTHONDEVMODE` environment variable can be used to enable 488:ref:`Python Development Mode <devmode>`. When in development mode, Python performs 489additional runtime checks that are too expensive to be enabled by default. 490See :ref:`Python Development Mode <devmode>` documentation for the full 491description. 492 493 494Other Language Changes 495====================== 496 497* An :keyword:`await` expression and comprehensions containing an 498 :keyword:`async for` clause were illegal in the expressions in 499 :ref:`formatted string literals <f-strings>` due to a problem with the 500 implementation. In Python 3.7 this restriction was lifted. 501 502* More than 255 arguments can now be passed to a function, and a function can 503 now have more than 255 parameters. (Contributed by Serhiy Storchaka in 504 :issue:`12844` and :issue:`18896`.) 505 506* :meth:`bytes.fromhex` and :meth:`bytearray.fromhex` now ignore all ASCII 507 whitespace, not only spaces. (Contributed by Robert Xiao in :issue:`28927`.) 508 509* :class:`str`, :class:`bytes`, and :class:`bytearray` gained support for 510 the new :meth:`isascii() <str.isascii>` method, which can be used to 511 test if a string or bytes contain only the ASCII characters. 512 (Contributed by INADA Naoki in :issue:`32677`.) 513 514* :exc:`ImportError` now displays module name and module ``__file__`` path when 515 ``from ... import ...`` fails. (Contributed by Matthias Bussonnier in 516 :issue:`29546`.) 517 518* Circular imports involving absolute imports with binding a submodule to 519 a name are now supported. 520 (Contributed by Serhiy Storchaka in :issue:`30024`.) 521 522* ``object.__format__(x, '')`` is now equivalent to ``str(x)`` rather than 523 ``format(str(self), '')``. 524 (Contributed by Serhiy Storchaka in :issue:`28974`.) 525 526* In order to better support dynamic creation of stack traces, 527 :class:`types.TracebackType` can now be instantiated from Python code, and 528 the :attr:`~traceback.tb_next` attribute on 529 :ref:`tracebacks <traceback-objects>` is now writable. 530 (Contributed by Nathaniel J. Smith in :issue:`30579`.) 531 532* When using the :option:`-m` switch, ``sys.path[0]`` is now eagerly expanded 533 to the full starting directory path, rather than being left as the empty 534 directory (which allows imports from the *current* working directory at the 535 time when an import occurs) 536 (Contributed by Nick Coghlan in :issue:`33053`.) 537 538* The new :option:`-X` ``importtime`` option or the 539 :envvar:`PYTHONPROFILEIMPORTTIME` environment variable can be used to show 540 the timing of each module import. 541 (Contributed by Inada Naoki in :issue:`31415`.) 542 543 544New Modules 545=========== 546 547.. _whatsnew37-pep567: 548 549contextvars 550----------- 551 552The new :mod:`contextvars` module and a set of 553:ref:`new C APIs <contextvarsobjects>` introduce 554support for *context variables*. Context variables are conceptually 555similar to thread-local variables. Unlike TLS, context variables 556support asynchronous code correctly. 557 558The :mod:`asyncio` and :mod:`decimal` modules have been updated to use 559and support context variables out of the box. Particularly the active 560decimal context is now stored in a context variable, which allows 561decimal operations to work with the correct context in asynchronous code. 562 563.. seealso:: 564 565 :pep:`567` -- Context Variables 566 PEP written and implemented by Yury Selivanov 567 568 569.. _whatsnew37-pep557: 570 571dataclasses 572----------- 573 574The new :func:`~dataclasses.dataclass` decorator provides a way to declare 575*data classes*. A data class describes its attributes using class variable 576annotations. Its constructor and other magic methods, such as 577:meth:`~object.__repr__`, :meth:`~object.__eq__`, and 578:meth:`~object.__hash__` are generated automatically. 579 580Example:: 581 582 @dataclass 583 class Point: 584 x: float 585 y: float 586 z: float = 0.0 587 588 p = Point(1.5, 2.5) 589 print(p) # produces "Point(x=1.5, y=2.5, z=0.0)" 590 591.. seealso:: 592 593 :pep:`557` -- Data Classes 594 PEP written and implemented by Eric V. Smith 595 596 597.. _whatsnew37_importlib_resources: 598 599importlib.resources 600------------------- 601 602The new :mod:`importlib.resources` module provides several new APIs and one 603new ABC for access to, opening, and reading *resources* inside packages. 604Resources are roughly similar to files inside packages, but they needn't 605be actual files on the physical file system. Module loaders can provide a 606:meth:`get_resource_reader` function which returns 607a :class:`importlib.abc.ResourceReader` instance to support this 608new API. Built-in file path loaders and zip file loaders both support this. 609 610Contributed by Barry Warsaw and Brett Cannon in :issue:`32248`. 611 612.. seealso:: 613 614 `importlib_resources <https://importlib-resources.readthedocs.io/en/latest/>`_ 615 -- a PyPI backport for earlier Python versions. 616 617 618Improved Modules 619================ 620 621 622argparse 623-------- 624 625The new :meth:`ArgumentParser.parse_intermixed_args() 626<argparse.ArgumentParser.parse_intermixed_args>` 627method allows intermixing options and positional arguments. 628(Contributed by paul.j3 in :issue:`14191`.) 629 630 631.. _whatsnew37_asyncio: 632 633asyncio 634------- 635 636The :mod:`asyncio` module has received many new features, usability and 637:ref:`performance improvements <whatsnew37-asyncio-perf>`. Notable changes 638include: 639 640* The new :term:`provisional <provisional API>` :func:`asyncio.run` function can 641 be used to run a coroutine from synchronous code by automatically creating and 642 destroying the event loop. 643 (Contributed by Yury Selivanov in :issue:`32314`.) 644 645* asyncio gained support for :mod:`contextvars`. 646 :meth:`loop.call_soon() <asyncio.loop.call_soon>`, 647 :meth:`loop.call_soon_threadsafe() <asyncio.loop.call_soon_threadsafe>`, 648 :meth:`loop.call_later() <asyncio.loop.call_later>`, 649 :meth:`loop.call_at() <asyncio.loop.call_at>`, and 650 :meth:`Future.add_done_callback() <asyncio.Future.add_done_callback>` 651 have a new optional keyword-only *context* parameter. 652 :class:`Tasks <asyncio.Task>` now track their context automatically. 653 See :pep:`567` for more details. 654 (Contributed by Yury Selivanov in :issue:`32436`.) 655 656* The new :func:`asyncio.create_task` function has been added as a shortcut 657 to ``asyncio.get_event_loop().create_task()``. 658 (Contributed by Andrew Svetlov in :issue:`32311`.) 659 660* The new :meth:`loop.start_tls() <asyncio.loop.start_tls>` 661 method can be used to upgrade an existing connection to TLS. 662 (Contributed by Yury Selivanov in :issue:`23749`.) 663 664* The new :meth:`loop.sock_recv_into() <asyncio.loop.sock_recv_into>` 665 method allows reading data from a socket directly into a provided buffer making 666 it possible to reduce data copies. 667 (Contributed by Antoine Pitrou in :issue:`31819`.) 668 669* The new :func:`asyncio.current_task` function returns the currently running 670 :class:`~asyncio.Task` instance, and the new :func:`asyncio.all_tasks` 671 function returns a set of all existing ``Task`` instances in a given loop. 672 The :meth:`!Task.current_task` and 673 :meth:`!Task.all_tasks` methods have been deprecated. 674 (Contributed by Andrew Svetlov in :issue:`32250`.) 675 676* The new *provisional* :class:`~asyncio.BufferedProtocol` class allows 677 implementing streaming protocols with manual control over the receive buffer. 678 (Contributed by Yury Selivanov in :issue:`32251`.) 679 680* The new :func:`asyncio.get_running_loop` function returns the currently 681 running loop, and raises a :exc:`RuntimeError` if no loop is running. 682 This is in contrast with :func:`asyncio.get_event_loop`, which will *create* 683 a new event loop if none is running. 684 (Contributed by Yury Selivanov in :issue:`32269`.) 685 686* The new :meth:`StreamWriter.wait_closed() <asyncio.StreamWriter.wait_closed>` 687 coroutine method allows waiting until the stream writer is closed. The new 688 :meth:`StreamWriter.is_closing() <asyncio.StreamWriter.is_closing>` method 689 can be used to determine if the writer is closing. 690 (Contributed by Andrew Svetlov in :issue:`32391`.) 691 692* The new :meth:`loop.sock_sendfile() <asyncio.loop.sock_sendfile>` 693 coroutine method allows sending files using :mod:`os.sendfile` when possible. 694 (Contributed by Andrew Svetlov in :issue:`32410`.) 695 696* The new :meth:`Future.get_loop() <asyncio.Future.get_loop>` and 697 ``Task.get_loop()`` methods return the instance of the loop on which a task or 698 a future were created. 699 :meth:`Server.get_loop() <asyncio.Server.get_loop>` allows doing the same for 700 :class:`asyncio.Server` objects. 701 (Contributed by Yury Selivanov in :issue:`32415` and 702 Srinivas Reddy Thatiparthy in :issue:`32418`.) 703 704* It is now possible to control how instances of :class:`asyncio.Server` begin 705 serving. Previously, the server would start serving immediately when created. 706 The new *start_serving* keyword argument to 707 :meth:`loop.create_server() <asyncio.loop.create_server>` and 708 :meth:`loop.create_unix_server() <asyncio.loop.create_unix_server>`, 709 as well as :meth:`Server.start_serving() <asyncio.Server.start_serving>`, and 710 :meth:`Server.serve_forever() <asyncio.Server.serve_forever>` 711 can be used to decouple server instantiation and serving. The new 712 :meth:`Server.is_serving() <asyncio.Server.is_serving>` method returns ``True`` 713 if the server is serving. :class:`~asyncio.Server` objects are now 714 asynchronous context managers:: 715 716 srv = await loop.create_server(...) 717 718 async with srv: 719 # some code 720 721 # At this point, srv is closed and no longer accepts new connections. 722 723 (Contributed by Yury Selivanov in :issue:`32662`.) 724 725* Callback objects returned by 726 :func:`loop.call_later() <asyncio.loop.call_later>` 727 gained the new :meth:`when() <asyncio.TimerHandle.when>` method which 728 returns an absolute scheduled callback timestamp. 729 (Contributed by Andrew Svetlov in :issue:`32741`.) 730 731* The :meth:`loop.create_datagram_endpoint() \ 732 <asyncio.loop.create_datagram_endpoint>` method 733 gained support for Unix sockets. 734 (Contributed by Quentin Dawans in :issue:`31245`.) 735 736* The :func:`asyncio.open_connection`, :func:`asyncio.start_server` functions, 737 :meth:`loop.create_connection() <asyncio.loop.create_connection>`, 738 :meth:`loop.create_server() <asyncio.loop.create_server>`, 739 :meth:`loop.create_accepted_socket() <asyncio.loop.connect_accepted_socket>` 740 methods and their corresponding UNIX socket variants now accept the 741 *ssl_handshake_timeout* keyword argument. 742 (Contributed by Neil Aspinall in :issue:`29970`.) 743 744* The new :meth:`Handle.cancelled() <asyncio.Handle.cancelled>` method returns 745 ``True`` if the callback was cancelled. 746 (Contributed by Marat Sharafutdinov in :issue:`31943`.) 747 748* The asyncio source has been converted to use the 749 :keyword:`async`/:keyword:`await` syntax. 750 (Contributed by Andrew Svetlov in :issue:`32193`.) 751 752* The new :meth:`ReadTransport.is_reading() <asyncio.ReadTransport.is_reading>` 753 method can be used to determine the reading state of the transport. 754 Additionally, calls to 755 :meth:`ReadTransport.resume_reading() <asyncio.ReadTransport.resume_reading>` 756 and :meth:`ReadTransport.pause_reading() <asyncio.ReadTransport.pause_reading>` 757 are now idempotent. 758 (Contributed by Yury Selivanov in :issue:`32356`.) 759 760* Loop methods which accept socket paths now support passing 761 :term:`path-like objects <path-like object>`. 762 (Contributed by Yury Selivanov in :issue:`32066`.) 763 764* In :mod:`asyncio` TCP sockets on Linux are now created with ``TCP_NODELAY`` 765 flag set by default. 766 (Contributed by Yury Selivanov and Victor Stinner in :issue:`27456`.) 767 768* Exceptions occurring in cancelled tasks are no longer logged. 769 (Contributed by Yury Selivanov in :issue:`30508`.) 770 771* New ``WindowsSelectorEventLoopPolicy`` and 772 ``WindowsProactorEventLoopPolicy`` classes. 773 (Contributed by Yury Selivanov in :issue:`33792`.) 774 775Several ``asyncio`` APIs have been 776:ref:`deprecated <whatsnew37-asyncio-deprecated>`. 777 778 779binascii 780-------- 781 782The :func:`~binascii.b2a_uu` function now accepts an optional *backtick* 783keyword argument. When it's true, zeros are represented by ``'`'`` 784instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.) 785 786 787calendar 788-------- 789 790The :class:`~calendar.HTMLCalendar` class has new class attributes which ease 791the customization of CSS classes in the produced HTML calendar. 792(Contributed by Oz Tiram in :issue:`30095`.) 793 794 795collections 796----------- 797 798``collections.namedtuple()`` now supports default values. 799(Contributed by Raymond Hettinger in :issue:`32320`.) 800 801 802compileall 803---------- 804 805:func:`compileall.compile_dir` learned the new *invalidation_mode* parameter, 806which can be used to enable 807:ref:`hash-based .pyc invalidation <whatsnew37-pep552>`. The invalidation 808mode can also be specified on the command line using the new 809``--invalidation-mode`` argument. 810(Contributed by Benjamin Peterson in :issue:`31650`.) 811 812 813concurrent.futures 814------------------ 815 816:class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` and 817:class:`ThreadPoolExecutor <concurrent.futures.ThreadPoolExecutor>` now 818support the new *initializer* and *initargs* constructor arguments. 819(Contributed by Antoine Pitrou in :issue:`21423`.) 820 821The :class:`ProcessPoolExecutor <concurrent.futures.ProcessPoolExecutor>` 822can now take the multiprocessing context via the new *mp_context* argument. 823(Contributed by Thomas Moreau in :issue:`31540`.) 824 825 826contextlib 827---------- 828 829The new :func:`~contextlib.nullcontext` is a simpler and faster no-op 830context manager than :class:`~contextlib.ExitStack`. 831(Contributed by Jesse-Bakker in :issue:`10049`.) 832 833The new :func:`~contextlib.asynccontextmanager`, 834:class:`~contextlib.AbstractAsyncContextManager`, and 835:class:`~contextlib.AsyncExitStack` have been added to 836complement their synchronous counterparts. (Contributed 837by Jelle Zijlstra in :issue:`29679` and :issue:`30241`, 838and by Alexander Mohr and Ilya Kulakov in :issue:`29302`.) 839 840 841cProfile 842-------- 843 844The :mod:`cProfile` command line now accepts ``-m module_name`` as an 845alternative to script path. (Contributed by Sanyam Khurana in :issue:`21862`.) 846 847 848crypt 849----- 850 851The :mod:`!crypt` module now supports the Blowfish hashing method. 852(Contributed by Serhiy Storchaka in :issue:`31664`.) 853 854The :func:`!mksalt` function now allows specifying the number of rounds 855for hashing. (Contributed by Serhiy Storchaka in :issue:`31702`.) 856 857 858datetime 859-------- 860 861The new :meth:`datetime.fromisoformat() <datetime.datetime.fromisoformat>` 862method constructs a :class:`~datetime.datetime` object from a string 863in one of the formats output by 864:meth:`datetime.isoformat() <datetime.datetime.isoformat>`. 865(Contributed by Paul Ganssle in :issue:`15873`.) 866 867The :class:`tzinfo <datetime.tzinfo>` class now supports sub-minute offsets. 868(Contributed by Alexander Belopolsky in :issue:`5288`.) 869 870 871dbm 872--- 873 874:mod:`dbm.dumb` now supports reading read-only files and no longer writes the 875index file when it is not changed. 876 877 878decimal 879------- 880 881The :mod:`decimal` module now uses :ref:`context variables <whatsnew37-pep567>` 882to store the decimal context. 883(Contributed by Yury Selivanov in :issue:`32630`.) 884 885 886dis 887--- 888 889The :func:`~dis.dis` function is now able to 890disassemble nested code objects (the code of comprehensions, generator 891expressions and nested functions, and the code used for building nested 892classes). The maximum depth of disassembly recursion is controlled by 893the new *depth* parameter. 894(Contributed by Serhiy Storchaka in :issue:`11822`.) 895 896 897distutils 898--------- 899 900``README.rst`` is now included in the list of distutils standard READMEs and 901therefore included in source distributions. 902(Contributed by Ryan Gonzalez in :issue:`11913`.) 903 904 905enum 906---- 907 908The :class:`Enum <enum.Enum>` learned the new ``_ignore_`` class property, 909which allows listing the names of properties which should not become 910enum members. 911(Contributed by Ethan Furman in :issue:`31801`.) 912 913In Python 3.8, attempting to check for non-Enum objects in :class:`Enum` 914classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly, 915attempting to check for non-Flag objects in a :class:`Flag` member will 916raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations 917return :const:`False` instead and are deprecated. 918(Contributed by Ethan Furman in :issue:`33217`.) 919 920 921functools 922--------- 923 924:func:`functools.singledispatch` now supports registering implementations 925using type annotations. 926(Contributed by Łukasz Langa in :issue:`32227`.) 927 928 929gc 930-- 931 932The new :func:`gc.freeze` function allows freezing all objects tracked 933by the garbage collector and excluding them from future collections. 934This can be used before a POSIX ``fork()`` call to make the GC copy-on-write 935friendly or to speed up collection. The new :func:`gc.unfreeze` functions 936reverses this operation. Additionally, :func:`gc.get_freeze_count` can 937be used to obtain the number of frozen objects. 938(Contributed by Li Zekun in :issue:`31558`.) 939 940 941hmac 942---- 943 944The :mod:`hmac` module now has an optimized one-shot :func:`~hmac.digest` 945function, which is up to three times faster than :func:`~hmac.HMAC`. 946(Contributed by Christian Heimes in :issue:`32433`.) 947 948 949http.client 950----------- 951 952:class:`~http.client.HTTPConnection` and :class:`~http.client.HTTPSConnection` 953now support the new *blocksize* argument for improved upload throughput. 954(Contributed by Nir Soffer in :issue:`31945`.) 955 956 957http.server 958----------- 959 960:class:`~http.server.SimpleHTTPRequestHandler` now supports the HTTP 961``If-Modified-Since`` header. The server returns the 304 response status if 962the target file was not modified after the time specified in the header. 963(Contributed by Pierre Quentel in :issue:`29654`.) 964 965:class:`~http.server.SimpleHTTPRequestHandler` accepts the new *directory* 966argument, in addition to the new ``--directory`` command line argument. 967With this parameter, the server serves the specified directory, by default it 968uses the current working directory. 969(Contributed by Stéphane Wirtel and Julien Palard in :issue:`28707`.) 970 971The new :class:`ThreadingHTTPServer <http.server.ThreadingHTTPServer>` class 972uses threads to handle requests using :class:`~socketserver.ThreadingMixin`. 973It is used when ``http.server`` is run with ``-m``. 974(Contributed by Julien Palard in :issue:`31639`.) 975 976 977idlelib and IDLE 978---------------- 979 980Multiple fixes for autocompletion. (Contributed by Louie Lu in :issue:`15786`.) 981 982Module Browser (on the File menu, formerly called Class Browser), 983now displays nested functions and classes in addition to top-level 984functions and classes. 985(Contributed by Guilherme Polo, Cheryl Sabella, and Terry Jan Reedy 986in :issue:`1612262`.) 987 988The Settings dialog (Options, Configure IDLE) has been partly rewritten 989to improve both appearance and function. 990(Contributed by Cheryl Sabella and Terry Jan Reedy in multiple issues.) 991 992The font sample now includes a selection of non-Latin characters so that 993users can better see the effect of selecting a particular font. 994(Contributed by Terry Jan Reedy in :issue:`13802`.) 995The sample can be edited to include other characters. 996(Contributed by Serhiy Storchaka in :issue:`31860`.) 997 998The IDLE features formerly implemented as extensions have been reimplemented 999as normal features. Their settings have been moved from the Extensions tab 1000to other dialog tabs. 1001(Contributed by Charles Wohlganger and Terry Jan Reedy in :issue:`27099`.) 1002 1003Editor code context option revised. Box displays all context lines up to 1004maxlines. Clicking on a context line jumps the editor to that line. Context 1005colors for custom themes is added to Highlights tab of Settings dialog. 1006(Contributed by Cheryl Sabella and Terry Jan Reedy in :issue:`33642`, 1007:issue:`33768`, and :issue:`33679`.) 1008 1009On Windows, a new API call tells Windows that tk scales for DPI. On Windows 10108.1+ or 10, with DPI compatibility properties of the Python binary 1011unchanged, and a monitor resolution greater than 96 DPI, this should 1012make text and lines sharper. It should otherwise have no effect. 1013(Contributed by Terry Jan Reedy in :issue:`33656`.) 1014 1015New in 3.7.1: 1016 1017Output over N lines (50 by default) is squeezed down to a button. 1018N can be changed in the PyShell section of the General page of the 1019Settings dialog. Fewer, but possibly extra long, lines can be squeezed by 1020right clicking on the output. Squeezed output can be expanded in place 1021by double-clicking the button or into the clipboard or a separate window 1022by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) 1023 1024The changes above have been backported to 3.6 maintenance releases. 1025 1026NEW in 3.7.4: 1027 1028Add "Run Customized" to the Run menu to run a module with customized 1029settings. Any command line arguments entered are added to sys.argv. 1030They re-appear in the box for the next customized run. One can also 1031suppress the normal Shell main module restart. (Contributed by Cheryl 1032Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.) 1033 1034New in 3.7.5: 1035 1036Add optional line numbers for IDLE editor windows. Windows 1037open without line numbers unless set otherwise in the General 1038tab of the configuration dialog. Line numbers for an existing 1039window are shown and hidden in the Options menu. 1040(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.) 1041 1042 1043importlib 1044--------- 1045 1046The :class:`importlib.abc.ResourceReader` ABC was introduced to 1047support the loading of resources from packages. See also 1048:ref:`whatsnew37_importlib_resources`. 1049(Contributed by Barry Warsaw, Brett Cannon in :issue:`32248`.) 1050 1051:func:`importlib.reload` now raises :exc:`ModuleNotFoundError` if the module 1052lacks a spec. 1053(Contributed by Garvit Khatri in :issue:`29851`.) 1054 1055:func:`importlib.find_spec` now raises :exc:`ModuleNotFoundError` instead of 1056:exc:`AttributeError` if the specified parent module is not a package (i.e. 1057lacks a ``__path__`` attribute). 1058(Contributed by Milan Oberkirch in :issue:`30436`.) 1059 1060The new :func:`importlib.source_hash` can be used to compute the hash of 1061the passed source. A :ref:`hash-based .pyc file <whatsnew37-pep552>` 1062embeds the value returned by this function. 1063 1064 1065io 1066-- 1067 1068The new :meth:`TextIOWrapper.reconfigure() <io.TextIOWrapper.reconfigure>` 1069method can be used to reconfigure the text stream with the new settings. 1070(Contributed by Antoine Pitrou in :issue:`30526` and 1071INADA Naoki in :issue:`15216`.) 1072 1073 1074ipaddress 1075--------- 1076 1077The new ``subnet_of()`` and ``supernet_of()`` methods of 1078:class:`ipaddress.IPv6Network` and :class:`ipaddress.IPv4Network` can 1079be used for network containment tests. 1080(Contributed by Michel Albert and Cheryl Sabella in :issue:`20825`.) 1081 1082 1083itertools 1084--------- 1085 1086:func:`itertools.islice` now accepts 1087:meth:`integer-like objects <object.__index__>` as start, stop, 1088and slice arguments. 1089(Contributed by Will Roberts in :issue:`30537`.) 1090 1091 1092locale 1093------ 1094 1095The new *monetary* argument to :func:`locale.format_string` can be used 1096to make the conversion use monetary thousands separators and 1097grouping strings. (Contributed by Garvit in :issue:`10379`.) 1098 1099The :func:`locale.getpreferredencoding` function now always returns ``'UTF-8'`` 1100on Android or when in the :ref:`forced UTF-8 mode <whatsnew37-pep540>`. 1101 1102 1103logging 1104------- 1105 1106:class:`~logging.Logger` instances can now be pickled. 1107(Contributed by Vinay Sajip in :issue:`30520`.) 1108 1109The new :meth:`StreamHandler.setStream() <logging.StreamHandler.setStream>` 1110method can be used to replace the logger stream after handler creation. 1111(Contributed by Vinay Sajip in :issue:`30522`.) 1112 1113It is now possible to specify keyword arguments to handler constructors in 1114configuration passed to :func:`logging.config.fileConfig`. 1115(Contributed by Preston Landers in :issue:`31080`.) 1116 1117 1118math 1119---- 1120 1121The new :func:`math.remainder` function implements the IEEE 754-style remainder 1122operation. (Contributed by Mark Dickinson in :issue:`29962`.) 1123 1124 1125mimetypes 1126--------- 1127 1128The MIME type of .bmp has been changed from ``'image/x-ms-bmp'`` to 1129``'image/bmp'``. 1130(Contributed by Nitish Chandra in :issue:`22589`.) 1131 1132 1133msilib 1134------ 1135 1136The new :meth:`!Database.Close` method can be used 1137to close the :abbr:`MSI` database. 1138(Contributed by Berker Peksag in :issue:`20486`.) 1139 1140 1141multiprocessing 1142--------------- 1143 1144The new :meth:`Process.close() <multiprocessing.Process.close>` method 1145explicitly closes the process object and releases all resources associated 1146with it. :exc:`ValueError` is raised if the underlying process is still 1147running. 1148(Contributed by Antoine Pitrou in :issue:`30596`.) 1149 1150The new :meth:`Process.kill() <multiprocessing.Process.kill>` method can 1151be used to terminate the process using the :data:`SIGKILL` signal on Unix. 1152(Contributed by Vitor Pereira in :issue:`30794`.) 1153 1154Non-daemonic threads created by :class:`~multiprocessing.Process` are now 1155joined on process exit. 1156(Contributed by Antoine Pitrou in :issue:`18966`.) 1157 1158 1159os 1160-- 1161 1162:func:`os.fwalk` now accepts the *path* argument as :class:`bytes`. 1163(Contributed by Serhiy Storchaka in :issue:`28682`.) 1164 1165:func:`os.scandir` gained support for :ref:`file descriptors <path_fd>`. 1166(Contributed by Serhiy Storchaka in :issue:`25996`.) 1167 1168The new :func:`~os.register_at_fork` function allows registering Python 1169callbacks to be executed at process fork. 1170(Contributed by Antoine Pitrou in :issue:`16500`.) 1171 1172Added :func:`os.preadv` (combine the functionality of :func:`os.readv` and 1173:func:`os.pread`) and :func:`os.pwritev` functions (combine the functionality 1174of :func:`os.writev` and :func:`os.pwrite`). (Contributed by Pablo Galindo in 1175:issue:`31368`.) 1176 1177The mode argument of :func:`os.makedirs` no longer affects the file 1178permission bits of newly created intermediate-level directories. 1179(Contributed by Serhiy Storchaka in :issue:`19930`.) 1180 1181:func:`os.dup2` now returns the new file descriptor. Previously, ``None`` 1182was always returned. 1183(Contributed by Benjamin Peterson in :issue:`32441`.) 1184 1185The structure returned by :func:`os.stat` now contains the 1186:attr:`~os.stat_result.st_fstype` attribute on Solaris and its derivatives. 1187(Contributed by Jesús Cea Avión in :issue:`32659`.) 1188 1189 1190pathlib 1191------- 1192 1193The new :meth:`Path.is_mount() <pathlib.Path.is_mount>` method is now available 1194on POSIX systems and can be used to determine whether a path is a mount point. 1195(Contributed by Cooper Ry Lees in :issue:`30897`.) 1196 1197 1198pdb 1199--- 1200 1201:func:`pdb.set_trace` now takes an optional *header* keyword-only 1202argument. If given, it is printed to the console just before debugging 1203begins. (Contributed by Barry Warsaw in :issue:`31389`.) 1204 1205:mod:`pdb` command line now accepts ``-m module_name`` as an alternative to 1206script file. (Contributed by Mario Corchero in :issue:`32206`.) 1207 1208 1209py_compile 1210---------- 1211 1212:func:`py_compile.compile` -- and by extension, :mod:`compileall` -- now 1213respects the :envvar:`SOURCE_DATE_EPOCH` environment variable by 1214unconditionally creating ``.pyc`` files for hash-based validation. 1215This allows for guaranteeing 1216`reproducible builds <https://reproducible-builds.org/>`_ of ``.pyc`` 1217files when they are created eagerly. (Contributed by Bernhard M. Wiedemann 1218in :issue:`29708`.) 1219 1220 1221pydoc 1222----- 1223 1224The pydoc server can now bind to an arbitrary hostname specified by the 1225new ``-n`` command-line argument. 1226(Contributed by Feanil Patel in :issue:`31128`.) 1227 1228 1229queue 1230----- 1231 1232The new :class:`~queue.SimpleQueue` class is an unbounded :abbr:`FIFO` queue. 1233(Contributed by Antoine Pitrou in :issue:`14976`.) 1234 1235 1236re 1237-- 1238 1239The flags :const:`re.ASCII`, :const:`re.LOCALE` and :const:`re.UNICODE` 1240can be set within the scope of a group. 1241(Contributed by Serhiy Storchaka in :issue:`31690`.) 1242 1243:func:`re.split` now supports splitting on a pattern like ``r'\b'``, 1244``'^$'`` or ``(?=-)`` that matches an empty string. 1245(Contributed by Serhiy Storchaka in :issue:`25054`.) 1246 1247Regular expressions compiled with the :const:`re.LOCALE` flag no longer 1248depend on the locale at compile time. Locale settings are applied only 1249when the compiled regular expression is used. 1250(Contributed by Serhiy Storchaka in :issue:`30215`.) 1251 1252:exc:`FutureWarning` is now emitted if a regular expression contains 1253character set constructs that will change semantically in the future, 1254such as nested sets and set operations. 1255(Contributed by Serhiy Storchaka in :issue:`30349`.) 1256 1257Compiled regular expression and match objects can now be copied 1258using :func:`copy.copy` and :func:`copy.deepcopy`. 1259(Contributed by Serhiy Storchaka in :issue:`10076`.) 1260 1261 1262signal 1263------ 1264 1265The new *warn_on_full_buffer* argument to the :func:`signal.set_wakeup_fd` 1266function makes it possible to specify whether Python prints a warning on 1267stderr when the wakeup buffer overflows. 1268(Contributed by Nathaniel J. Smith in :issue:`30050`.) 1269 1270 1271socket 1272------ 1273 1274The new :func:`socket.getblocking() <socket.socket.getblocking>` method 1275returns ``True`` if the socket is in blocking mode and ``False`` otherwise. 1276(Contributed by Yury Selivanov in :issue:`32373`.) 1277 1278The new :func:`socket.close` function closes the passed socket file descriptor. 1279This function should be used instead of :func:`os.close` for better 1280compatibility across platforms. 1281(Contributed by Christian Heimes in :issue:`32454`.) 1282 1283The :mod:`socket` module now exposes the :const:`socket.TCP_CONGESTION` 1284(Linux 2.6.13), :const:`socket.TCP_USER_TIMEOUT` (Linux 2.6.37), and 1285:const:`socket.TCP_NOTSENT_LOWAT` (Linux 3.12) constants. 1286(Contributed by Omar Sandoval in :issue:`26273` and 1287Nathaniel J. Smith in :issue:`29728`.) 1288 1289Support for :const:`socket.AF_VSOCK` sockets has been added to allow 1290communication between virtual machines and their hosts. 1291(Contributed by Cathy Avery in :issue:`27584`.) 1292 1293Sockets now auto-detect family, type and protocol from file descriptor 1294by default. 1295(Contributed by Christian Heimes in :issue:`28134`.) 1296 1297 1298socketserver 1299------------ 1300 1301:meth:`socketserver.ThreadingMixIn.server_close` now waits until all non-daemon 1302threads complete. :meth:`socketserver.ForkingMixIn.server_close` now waits 1303until all child processes complete. 1304 1305Add a new :attr:`socketserver.ForkingMixIn.block_on_close` class attribute to 1306:class:`socketserver.ForkingMixIn` and :class:`socketserver.ThreadingMixIn` 1307classes. Set the class attribute to ``False`` to get the pre-3.7 behaviour. 1308 1309 1310sqlite3 1311------- 1312 1313:class:`sqlite3.Connection` now exposes the :meth:`~sqlite3.Connection.backup` 1314method when the underlying SQLite library is at version 3.6.11 or higher. 1315(Contributed by Lele Gaifax in :issue:`27645`.) 1316 1317The *database* argument of :func:`sqlite3.connect` now accepts any 1318:term:`path-like object`, instead of just a string. 1319(Contributed by Anders Lorentsen in :issue:`31843`.) 1320 1321 1322ssl 1323--- 1324 1325The :mod:`ssl` module now uses OpenSSL's builtin API instead of 1326:func:`~ssl.match_hostname` to check a host name or an IP address. Values 1327are validated during TLS handshake. Any certificate validation error 1328including failing the host name check now raises 1329:exc:`~ssl.SSLCertVerificationError` and aborts the handshake with a proper 1330TLS Alert message. The new exception contains additional information. 1331Host name validation can be customized with 1332:attr:`SSLContext.hostname_checks_common_name <ssl.SSLContext.hostname_checks_common_name>`. 1333(Contributed by Christian Heimes in :issue:`31399`.) 1334 1335.. note:: 1336 The improved host name check requires a *libssl* implementation compatible 1337 with OpenSSL 1.0.2 or 1.1. Consequently, OpenSSL 0.9.8 and 1.0.1 are no 1338 longer supported (see :ref:`37-platform-support-removals` for more details). 1339 The ssl module is mostly compatible with LibreSSL 2.7.2 and newer. 1340 1341The ``ssl`` module no longer sends IP addresses in SNI TLS extension. 1342(Contributed by Christian Heimes in :issue:`32185`.) 1343 1344:func:`~ssl.match_hostname` no longer supports partial wildcards like 1345``www*.example.org``. 1346(Contributed by Mandeep Singh in :issue:`23033` and Christian Heimes in 1347:issue:`31399`.) 1348 1349The default cipher suite selection of the ``ssl`` module now uses a blacklist 1350approach rather than a hard-coded whitelist. Python no longer re-enables 1351ciphers that have been blocked by OpenSSL security updates. Default cipher 1352suite selection can be configured at compile time. 1353(Contributed by Christian Heimes in :issue:`31429`.) 1354 1355Validation of server certificates containing internationalized domain names 1356(IDNs) is now supported. As part of this change, the 1357:attr:`SSLSocket.server_hostname <ssl.SSLSocket.server_hostname>` attribute 1358now stores the expected hostname in A-label form (``"xn--pythn-mua.org"``), 1359rather than the U-label form (``"pythön.org"``). (Contributed by 1360Nathaniel J. Smith and Christian Heimes in :issue:`28414`.) 1361 1362The ``ssl`` module has preliminary and experimental support for TLS 1.3 and 1363OpenSSL 1.1.1. At the time of Python 3.7.0 release, OpenSSL 1.1.1 is still 1364under development and TLS 1.3 hasn't been finalized yet. The TLS 1.3 1365handshake and protocol behaves slightly differently than TLS 1.2 and earlier, 1366see :ref:`ssl-tlsv1_3`. 1367(Contributed by Christian Heimes in :issue:`32947`, :issue:`20995`, 1368:issue:`29136`, :issue:`30622` and :issue:`33618`) 1369 1370:class:`~ssl.SSLSocket` and :class:`~ssl.SSLObject` no longer have a public 1371constructor. Direct instantiation was never a documented and supported 1372feature. Instances must be created with :class:`~ssl.SSLContext` methods 1373:meth:`~ssl.SSLContext.wrap_socket` and :meth:`~ssl.SSLContext.wrap_bio`. 1374(Contributed by Christian Heimes in :issue:`32951`) 1375 1376OpenSSL 1.1 APIs for setting the minimum and maximum TLS protocol version are 1377available as :attr:`SSLContext.minimum_version <ssl.SSLContext.minimum_version>` 1378and :attr:`SSLContext.maximum_version <ssl.SSLContext.maximum_version>`. 1379Supported protocols are indicated by several new flags, such as 1380:data:`~ssl.HAS_TLSv1_1`. 1381(Contributed by Christian Heimes in :issue:`32609`.) 1382 1383Added :attr:`ssl.SSLContext.post_handshake_auth` to enable and 1384:meth:`ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3 1385post-handshake authentication. 1386(Contributed by Christian Heimes in :gh:`78851`.) 1387 1388string 1389------ 1390 1391:class:`string.Template` now lets you to optionally modify the regular 1392expression pattern for braced placeholders and non-braced placeholders 1393separately. (Contributed by Barry Warsaw in :issue:`1198569`.) 1394 1395 1396subprocess 1397---------- 1398 1399The :func:`subprocess.run` function accepts the new *capture_output* 1400keyword argument. When true, stdout and stderr will be captured. 1401This is equivalent to passing :const:`subprocess.PIPE` as *stdout* and 1402*stderr* arguments. 1403(Contributed by Bo Bayles in :issue:`32102`.) 1404 1405The ``subprocess.run`` function and the :class:`subprocess.Popen` constructor 1406now accept the *text* keyword argument as an alias 1407to *universal_newlines*. 1408(Contributed by Andrew Clegg in :issue:`31756`.) 1409 1410On Windows the default for *close_fds* was changed from ``False`` to 1411``True`` when redirecting the standard handles. It's now possible to set 1412*close_fds* to true when redirecting the standard handles. See 1413:class:`subprocess.Popen`. This means that *close_fds* now defaults to 1414``True`` on all supported platforms. 1415(Contributed by Segev Finer in :issue:`19764`.) 1416 1417The subprocess module is now more graceful when handling 1418:exc:`KeyboardInterrupt` during :func:`subprocess.call`, 1419:func:`subprocess.run`, or in a :class:`~subprocess.Popen` 1420context manager. It now waits a short amount of time for the child 1421to exit, before continuing the handling of the ``KeyboardInterrupt`` 1422exception. 1423(Contributed by Gregory P. Smith in :issue:`25942`.) 1424 1425 1426sys 1427--- 1428 1429The new :func:`sys.breakpointhook` hook function is called by the 1430built-in :func:`breakpoint`. 1431(Contributed by Barry Warsaw in :issue:`31353`.) 1432 1433On Android, the new :func:`sys.getandroidapilevel` returns the build-time 1434Android API version. 1435(Contributed by Victor Stinner in :issue:`28740`.) 1436 1437The new :func:`sys.get_coroutine_origin_tracking_depth` function returns 1438the current coroutine origin tracking depth, as set by 1439the new :func:`sys.set_coroutine_origin_tracking_depth`. :mod:`asyncio` 1440has been converted to use this new API instead of 1441the deprecated :func:`sys.set_coroutine_wrapper`. 1442(Contributed by Nathaniel J. Smith in :issue:`32591`.) 1443 1444 1445time 1446---- 1447 1448:pep:`564` adds six new functions with nanosecond resolution to the 1449:mod:`time` module: 1450 1451* :func:`time.clock_gettime_ns` 1452* :func:`time.clock_settime_ns` 1453* :func:`time.monotonic_ns` 1454* :func:`time.perf_counter_ns` 1455* :func:`time.process_time_ns` 1456* :func:`time.time_ns` 1457 1458New clock identifiers have been added: 1459 1460* :const:`time.CLOCK_BOOTTIME` (Linux): Identical to 1461 :const:`time.CLOCK_MONOTONIC`, except it also includes any time that the 1462 system is suspended. 1463* :const:`time.CLOCK_PROF` (FreeBSD, NetBSD and OpenBSD): High-resolution 1464 per-process CPU timer. 1465* :const:`time.CLOCK_UPTIME` (FreeBSD, OpenBSD): Time whose absolute value is 1466 the time the system has been running and not suspended, providing accurate 1467 uptime measurement. 1468 1469The new :func:`time.thread_time` and :func:`time.thread_time_ns` functions 1470can be used to get per-thread CPU time measurements. 1471(Contributed by Antoine Pitrou in :issue:`32025`.) 1472 1473The new :func:`time.pthread_getcpuclockid` function returns the clock ID 1474of the thread-specific CPU-time clock. 1475 1476 1477tkinter 1478------- 1479 1480The new :class:`tkinter.ttk.Spinbox` class is now available. 1481(Contributed by Alan Moore in :issue:`32585`.) 1482 1483 1484tracemalloc 1485----------- 1486 1487:class:`tracemalloc.Traceback` behaves more like regular tracebacks, 1488sorting the frames from oldest to most recent. 1489:meth:`Traceback.format() <tracemalloc.Traceback.format>` 1490now accepts negative *limit*, truncating the result to the 1491``abs(limit)`` oldest frames. To get the old behaviour, use 1492the new *most_recent_first* argument to ``Traceback.format()``. 1493(Contributed by Jesse Bakker in :issue:`32121`.) 1494 1495 1496types 1497----- 1498 1499The new :class:`~types.WrapperDescriptorType`, 1500:class:`~types.MethodWrapperType`, :class:`~types.MethodDescriptorType`, 1501and :class:`~types.ClassMethodDescriptorType` classes are now available. 1502(Contributed by Manuel Krebber and Guido van Rossum in :issue:`29377`, 1503and Serhiy Storchaka in :issue:`32265`.) 1504 1505The new :func:`types.resolve_bases` function resolves MRO entries 1506dynamically as specified by :pep:`560`. 1507(Contributed by Ivan Levkivskyi in :issue:`32717`.) 1508 1509 1510unicodedata 1511----------- 1512 1513The internal :mod:`unicodedata` database has been upgraded to use `Unicode 11 1514<https://www.unicode.org/versions/Unicode11.0.0/>`_. (Contributed by Benjamin 1515Peterson.) 1516 1517 1518unittest 1519-------- 1520 1521The new ``-k`` command-line option allows filtering tests by a name 1522substring or a Unix shell-like pattern. 1523For example, ``python -m unittest -k foo`` runs 1524``foo_tests.SomeTest.test_something``, ``bar_tests.SomeTest.test_foo``, 1525but not ``bar_tests.FooTest.test_something``. 1526(Contributed by Jonas Haag in :issue:`32071`.) 1527 1528 1529unittest.mock 1530------------- 1531 1532The :const:`~unittest.mock.sentinel` attributes now preserve their identity 1533when they are :mod:`copied <copy>` or :mod:`pickled <pickle>`. (Contributed by 1534Serhiy Storchaka in :issue:`20804`.) 1535 1536The new :func:`~unittest.mock.seal` function allows sealing 1537:class:`~unittest.mock.Mock` instances, which will disallow further creation 1538of attribute mocks. The seal is applied recursively to all attributes that 1539are themselves mocks. 1540(Contributed by Mario Corchero in :issue:`30541`.) 1541 1542 1543urllib.parse 1544------------ 1545 1546:func:`urllib.parse.quote` has been updated from :rfc:`2396` to :rfc:`3986`, 1547adding ``~`` to the set of characters that are never quoted by default. 1548(Contributed by Christian Theune and Ratnadeep Debnath in :issue:`16285`.) 1549 1550 1551uu 1552-- 1553 1554The :func:`!uu.encode` function now accepts an optional *backtick* 1555keyword argument. When it's true, zeros are represented by ``'`'`` 1556instead of spaces. (Contributed by Xiang Zhang in :issue:`30103`.) 1557 1558 1559uuid 1560---- 1561 1562The new :attr:`UUID.is_safe <uuid.UUID.is_safe>` attribute relays information 1563from the platform about whether generated UUIDs are generated with a 1564multiprocessing-safe method. 1565(Contributed by Barry Warsaw in :issue:`22807`.) 1566 1567:func:`uuid.getnode` now prefers universally administered 1568MAC addresses over locally administered MAC addresses. 1569This makes a better guarantee for global uniqueness of UUIDs returned 1570from :func:`uuid.uuid1`. If only locally administered MAC addresses are 1571available, the first such one found is returned. 1572(Contributed by Barry Warsaw in :issue:`32107`.) 1573 1574 1575warnings 1576-------- 1577 1578The initialization of the default warnings filters has changed as follows: 1579 1580* warnings enabled via command line options (including those for :option:`-b` 1581 and the new CPython-specific :option:`-X` ``dev`` option) are always passed 1582 to the warnings machinery via the :data:`sys.warnoptions` attribute. 1583 1584* warnings filters enabled via the command line or the environment now have the 1585 following order of precedence: 1586 1587 * the ``BytesWarning`` filter for :option:`-b` (or ``-bb``) 1588 * any filters specified with the :option:`-W` option 1589 * any filters specified with the :envvar:`PYTHONWARNINGS` environment 1590 variable 1591 * any other CPython specific filters (e.g. the ``default`` filter added 1592 for the new ``-X dev`` mode) 1593 * any implicit filters defined directly by the warnings machinery 1594 1595* in :ref:`CPython debug builds <debug-build>`, all warnings are now displayed 1596 by default (the implicit filter list is empty) 1597 1598(Contributed by Nick Coghlan and Victor Stinner in :issue:`20361`, 1599:issue:`32043`, and :issue:`32230`.) 1600 1601Deprecation warnings are once again shown by default in single-file scripts and 1602at the interactive prompt. See :ref:`whatsnew37-pep565` for details. 1603(Contributed by Nick Coghlan in :issue:`31975`.) 1604 1605 1606xml 1607--- 1608 1609As mitigation against DTD and external entity retrieval, the 1610:mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 1611external entities by default. 1612(Contributed by Christian Heimes in :gh:`61441`.) 1613 1614 1615xml.etree 1616--------- 1617 1618:ref:`ElementPath <elementtree-xpath>` predicates in the :meth:`find` 1619methods can now compare text of the current node with ``[. = "text"]``, 1620not only text in children. Predicates also allow adding spaces for 1621better readability. (Contributed by Stefan Behnel in :issue:`31648`.) 1622 1623 1624xmlrpc.server 1625------------- 1626 1627:meth:`SimpleXMLRPCDispatcher.register_function <xmlrpc.server.SimpleXMLRPCDispatcher>` 1628can now be used as a decorator. (Contributed by Xiang Zhang in 1629:issue:`7769`.) 1630 1631 1632zipapp 1633------ 1634 1635Function :func:`~zipapp.create_archive` now accepts an optional *filter* 1636argument to allow the user to select which files should be included in the 1637archive. (Contributed by Irmen de Jong in :issue:`31072`.) 1638 1639Function :func:`~zipapp.create_archive` now accepts an optional *compressed* 1640argument to generate a compressed archive. A command line option 1641``--compress`` has also been added to support compression. 1642(Contributed by Zhiming Wang in :issue:`31638`.) 1643 1644 1645zipfile 1646------- 1647 1648:class:`~zipfile.ZipFile` now accepts the new *compresslevel* parameter to 1649control the compression level. 1650(Contributed by Bo Bayles in :issue:`21417`.) 1651 1652Subdirectories in archives created by ``ZipFile`` are now stored in 1653alphabetical order. 1654(Contributed by Bernhard M. Wiedemann in :issue:`30693`.) 1655 1656 1657C API Changes 1658============= 1659 1660A new API for thread-local storage has been implemented. See 1661:ref:`whatsnew37-pep539` for an overview and 1662:ref:`thread-specific-storage-api` for a complete reference. 1663(Contributed by Masayuki Yamamoto in :issue:`25658`.) 1664 1665The new :ref:`context variables <whatsnew37-pep567>` functionality 1666exposes a number of :ref:`new C APIs <contextvarsobjects>`. 1667 1668The new :c:func:`PyImport_GetModule` function returns the previously 1669imported module with the given name. 1670(Contributed by Eric Snow in :issue:`28411`.) 1671 1672The new :c:macro:`Py_RETURN_RICHCOMPARE` macro eases writing rich 1673comparison functions. 1674(Contributed by Petr Victorin in :issue:`23699`.) 1675 1676The new :c:macro:`Py_UNREACHABLE` macro can be used to mark unreachable 1677code paths. 1678(Contributed by Barry Warsaw in :issue:`31338`.) 1679 1680The :mod:`tracemalloc` now exposes a C API through the new 1681:c:func:`PyTraceMalloc_Track` and :c:func:`PyTraceMalloc_Untrack` 1682functions. 1683(Contributed by Victor Stinner in :issue:`30054`.) 1684 1685The new :c:func:`import__find__load__start` and 1686:c:func:`import__find__load__done` static markers can be used to trace 1687module imports. 1688(Contributed by Christian Heimes in :issue:`31574`.) 1689 1690The fields :c:member:`!name` and :c:member:`!doc` of structures 1691:c:type:`PyMemberDef`, :c:type:`PyGetSetDef`, 1692:c:type:`PyStructSequence_Field`, :c:type:`PyStructSequence_Desc`, 1693and :c:struct:`wrapperbase` are now of type ``const char *`` rather of 1694``char *``. (Contributed by Serhiy Storchaka in :issue:`28761`.) 1695 1696The result of :c:func:`PyUnicode_AsUTF8AndSize` and :c:func:`PyUnicode_AsUTF8` 1697is now of type ``const char *`` rather of ``char *``. (Contributed by Serhiy 1698Storchaka in :issue:`28769`.) 1699 1700The result of :c:func:`PyMapping_Keys`, :c:func:`PyMapping_Values` and 1701:c:func:`PyMapping_Items` is now always a list, rather than a list or a 1702tuple. (Contributed by Oren Milman in :issue:`28280`.) 1703 1704Added functions :c:func:`PySlice_Unpack` and :c:func:`PySlice_AdjustIndices`. 1705(Contributed by Serhiy Storchaka in :issue:`27867`.) 1706 1707:c:func:`PyOS_AfterFork` is deprecated in favour of the new functions 1708:c:func:`PyOS_BeforeFork`, :c:func:`PyOS_AfterFork_Parent` and 1709:c:func:`PyOS_AfterFork_Child`. (Contributed by Antoine Pitrou in 1710:issue:`16500`.) 1711 1712The ``PyExc_RecursionErrorInst`` singleton that was part of the public API 1713has been removed as its members being never cleared may cause a segfault 1714during finalization of the interpreter. Contributed by Xavier de Gaye in 1715:issue:`22898` and :issue:`30697`. 1716 1717Added C API support for timezones with timezone constructors 1718:c:func:`PyTimeZone_FromOffset` and :c:func:`PyTimeZone_FromOffsetAndName`, 1719and access to the UTC singleton with :c:data:`PyDateTime_TimeZone_UTC`. 1720Contributed by Paul Ganssle in :issue:`10381`. 1721 1722The type of results of :c:func:`PyThread_start_new_thread` and 1723:c:func:`PyThread_get_thread_ident`, and the *id* parameter of 1724:c:func:`PyThreadState_SetAsyncExc` changed from :c:expr:`long` to 1725:c:expr:`unsigned long`. 1726(Contributed by Serhiy Storchaka in :issue:`6532`.) 1727 1728:c:func:`PyUnicode_AsWideCharString` now raises a :exc:`ValueError` if the 1729second argument is ``NULL`` and the :c:expr:`wchar_t*` string contains null 1730characters. (Contributed by Serhiy Storchaka in :issue:`30708`.) 1731 1732Changes to the startup sequence and the management of dynamic memory 1733allocators mean that the long documented requirement to call 1734:c:func:`Py_Initialize` before calling most C API functions is now 1735relied on more heavily, and failing to abide by it may lead to segfaults in 1736embedding applications. See the :ref:`porting-to-python-37` section in this 1737document and the :ref:`pre-init-safe` section in the C API documentation 1738for more details. 1739 1740The new :c:func:`PyInterpreterState_GetID` returns the unique ID for a 1741given interpreter. 1742(Contributed by Eric Snow in :issue:`29102`.) 1743 1744:c:func:`Py_DecodeLocale`, :c:func:`Py_EncodeLocale` now use the UTF-8 1745encoding when the :ref:`UTF-8 mode <whatsnew37-pep540>` is enabled. 1746(Contributed by Victor Stinner in :issue:`29240`.) 1747 1748:c:func:`PyUnicode_DecodeLocaleAndSize` and :c:func:`PyUnicode_EncodeLocale` 1749now use the current locale encoding for ``surrogateescape`` error handler. 1750(Contributed by Victor Stinner in :issue:`29240`.) 1751 1752The *start* and *end* parameters of :c:func:`PyUnicode_FindChar` are 1753now adjusted to behave like string slices. 1754(Contributed by Xiang Zhang in :issue:`28822`.) 1755 1756 1757Build Changes 1758============= 1759 1760Support for building ``--without-threads`` has been removed. The 1761:mod:`threading` module is now always available. 1762(Contributed by Antoine Pitrou in :issue:`31370`.). 1763 1764A full copy of libffi is no longer bundled for use when building the 1765:mod:`_ctypes <ctypes>` module on non-OSX UNIX platforms. An installed copy 1766of libffi is now required when building ``_ctypes`` on such platforms. 1767(Contributed by Zachary Ware in :issue:`27979`.) 1768 1769The Windows build process no longer depends on Subversion to pull in external 1770sources, a Python script is used to download zipfiles from GitHub instead. 1771If Python 3.6 is not found on the system (via ``py -3.6``), NuGet is used to 1772download a copy of 32-bit Python for this purpose. (Contributed by Zachary 1773Ware in :issue:`30450`.) 1774 1775The :mod:`ssl` module requires OpenSSL 1.0.2 or 1.1 compatible libssl. 1776OpenSSL 1.0.1 has reached end of lifetime on 2016-12-31 and is no longer 1777supported. LibreSSL is temporarily not supported as well. LibreSSL releases 1778up to version 2.6.4 are missing required OpenSSL 1.0.2 APIs. 1779 1780 1781.. _whatsnew37-perf: 1782 1783Optimizations 1784============= 1785 1786The overhead of calling many methods of various standard library classes 1787implemented in C has been significantly reduced by porting more code 1788to use the ``METH_FASTCALL`` convention. 1789(Contributed by Victor Stinner in :issue:`29300`, :issue:`29507`, 1790:issue:`29452`, and :issue:`29286`.) 1791 1792Various optimizations have reduced Python startup time by 10% on Linux and 1793up to 30% on macOS. 1794(Contributed by Victor Stinner, INADA Naoki in :issue:`29585`, and 1795Ivan Levkivskyi in :issue:`31333`.) 1796 1797Method calls are now up to 20% faster due to the bytecode changes which 1798avoid creating bound method instances. 1799(Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) 1800 1801.. _whatsnew37-asyncio-perf: 1802 1803The :mod:`asyncio` module received a number of notable optimizations for 1804commonly used functions: 1805 1806* The :func:`asyncio.get_event_loop` function has been reimplemented in C to 1807 make it up to 15 times faster. 1808 (Contributed by Yury Selivanov in :issue:`32296`.) 1809 1810* :class:`asyncio.Future` callback management has been optimized. 1811 (Contributed by Yury Selivanov in :issue:`32348`.) 1812 1813* :func:`asyncio.gather` is now up to 15% faster. 1814 (Contributed by Yury Selivanov in :issue:`32355`.) 1815 1816* :func:`asyncio.sleep` is now up to 2 times faster when the *delay* 1817 argument is zero or negative. 1818 (Contributed by Andrew Svetlov in :issue:`32351`.) 1819 1820* The performance overhead of asyncio debug mode has been reduced. 1821 (Contributed by Antoine Pitrou in :issue:`31970`.) 1822 1823As a result of :ref:`PEP 560 work <whatsnew37-pep560>`, the import time 1824of :mod:`typing` has been reduced by a factor of 7, and many typing operations 1825are now faster. 1826(Contributed by Ivan Levkivskyi in :issue:`32226`.) 1827 1828:func:`sorted` and :meth:`list.sort` have been optimized for common cases 1829to be up to 40-75% faster. 1830(Contributed by Elliot Gorokhovsky in :issue:`28685`.) 1831 1832:meth:`dict.copy` is now up to 5.5 times faster. 1833(Contributed by Yury Selivanov in :issue:`31179`.) 1834 1835:func:`hasattr` and :func:`getattr` are now about 4 times faster when 1836*name* is not found and *obj* does not override :meth:`object.__getattr__` 1837or :meth:`object.__getattribute__`. 1838(Contributed by INADA Naoki in :issue:`32544`.) 1839 1840Searching for certain Unicode characters (like Ukrainian capital "Є") 1841in a string was up to 25 times slower than searching for other characters. 1842It is now only 3 times slower in the worst case. 1843(Contributed by Serhiy Storchaka in :issue:`24821`.) 1844 1845The :func:`collections.namedtuple` factory has been reimplemented to 1846make the creation of named tuples 4 to 6 times faster. 1847(Contributed by Jelle Zijlstra with further improvements by INADA Naoki, 1848Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.) 1849 1850:meth:`date.fromordinal` and :meth:`date.fromtimestamp` are now up to 185130% faster in the common case. 1852(Contributed by Paul Ganssle in :issue:`32403`.) 1853 1854The :func:`os.fwalk` function is now up to 2 times faster thanks to 1855the use of :func:`os.scandir`. 1856(Contributed by Serhiy Storchaka in :issue:`25996`.) 1857 1858The speed of the :func:`shutil.rmtree` function has been improved by 185920--40% thanks to the use of the :func:`os.scandir` function. 1860(Contributed by Serhiy Storchaka in :issue:`28564`.) 1861 1862Optimized case-insensitive matching and searching of :mod:`regular 1863expressions <re>`. Searching some patterns can now be up to 20 times faster. 1864(Contributed by Serhiy Storchaka in :issue:`30285`.) 1865 1866:func:`re.compile` now converts ``flags`` parameter to int object if 1867it is ``RegexFlag``. It is now as fast as Python 3.5, and faster than 1868Python 3.6 by about 10% depending on the pattern. 1869(Contributed by INADA Naoki in :issue:`31671`.) 1870 1871The :meth:`~selectors.BaseSelector.modify` methods of classes 1872:class:`selectors.EpollSelector`, :class:`selectors.PollSelector` 1873and :class:`selectors.DevpollSelector` may be around 10% faster under 1874heavy loads. (Contributed by Giampaolo Rodola' in :issue:`30014`) 1875 1876Constant folding has been moved from the peephole optimizer to the new AST 1877optimizer, which is able perform optimizations more consistently. 1878(Contributed by Eugene Toder and INADA Naoki in :issue:`29469` and 1879:issue:`11549`.) 1880 1881Most functions and methods in :mod:`abc` have been rewritten in C. 1882This makes creation of abstract base classes, and calling :func:`isinstance` 1883and :func:`issubclass` on them 1.5x faster. This also reduces Python 1884start-up time by up to 10%. (Contributed by Ivan Levkivskyi and INADA Naoki 1885in :issue:`31333`) 1886 1887Significant speed improvements to alternate constructors for 1888:class:`datetime.date` and :class:`datetime.datetime` by using fast-path 1889constructors when not constructing subclasses. (Contributed by Paul Ganssle 1890in :issue:`32403`) 1891 1892The speed of comparison of :class:`array.array` instances has been 1893improved considerably in certain cases. It is now from 10x to 70x faster 1894when comparing arrays holding values of the same integer type. 1895(Contributed by Adrian Wielgosik in :issue:`24700`.) 1896 1897The :func:`math.erf` and :func:`math.erfc` functions now use the (faster) 1898C library implementation on most platforms. 1899(Contributed by Serhiy Storchaka in :issue:`26121`.) 1900 1901 1902Other CPython Implementation Changes 1903==================================== 1904 1905* Trace hooks may now opt out of receiving the ``line`` and opt into receiving 1906 the ``opcode`` events from the interpreter by setting the corresponding new 1907 :attr:`~frame.f_trace_lines` and :attr:`~frame.f_trace_opcodes` attributes on the 1908 frame being traced. (Contributed by Nick Coghlan in :issue:`31344`.) 1909 1910* Fixed some consistency problems with namespace package module attributes. 1911 Namespace module objects now have an ``__file__`` that is set to ``None`` 1912 (previously unset), and their ``__spec__.origin`` is also set to ``None`` 1913 (previously the string ``"namespace"``). See :issue:`32305`. Also, the 1914 namespace module object's ``__spec__.loader`` is set to the same value as 1915 ``__loader__`` (previously, the former was set to ``None``). See 1916 :issue:`32303`. 1917 1918* The :func:`locals` dictionary now displays in the lexical order that 1919 variables were defined. Previously, the order was undefined. 1920 (Contributed by Raymond Hettinger in :issue:`32690`.) 1921 1922* The ``distutils`` ``upload`` command no longer tries to change CR 1923 end-of-line characters to CRLF. This fixes a corruption issue with sdists 1924 that ended with a byte equivalent to CR. 1925 (Contributed by Bo Bayles in :issue:`32304`.) 1926 1927 1928Deprecated Python Behavior 1929========================== 1930 1931Yield expressions (both ``yield`` and ``yield from`` clauses) are now deprecated 1932in comprehensions and generator expressions (aside from the iterable expression 1933in the leftmost :keyword:`!for` clause). This ensures that comprehensions 1934always immediately return a container of the appropriate type (rather than 1935potentially returning a :term:`generator iterator` object), while generator 1936expressions won't attempt to interleave their implicit output with the output 1937from any explicit yield expressions. In Python 3.7, such expressions emit 1938:exc:`DeprecationWarning` when compiled, in Python 3.8 this will be a 1939:exc:`SyntaxError`. 1940(Contributed by Serhiy Storchaka in :issue:`10544`.) 1941 1942Returning a subclass of :class:`complex` from :meth:`object.__complex__` is 1943deprecated and will be an error in future Python versions. This makes 1944``__complex__()`` consistent with :meth:`object.__int__` and 1945:meth:`object.__float__`. 1946(Contributed by Serhiy Storchaka in :issue:`28894`.) 1947 1948 1949 1950Deprecated Python modules, functions and methods 1951================================================ 1952 1953aifc 1954---- 1955 1956:func:`!aifc.openfp` has been deprecated and will be removed in Python 3.9. 1957Use :func:`!aifc.open` instead. 1958(Contributed by Brian Curtin in :issue:`31985`.) 1959 1960 1961.. _whatsnew37-asyncio-deprecated: 1962 1963asyncio 1964------- 1965 1966Support for directly ``await``-ing instances of :class:`asyncio.Lock` and 1967other asyncio synchronization primitives has been deprecated. An 1968asynchronous context manager must be used in order to acquire and release 1969the synchronization resource. 1970(Contributed by Andrew Svetlov in :issue:`32253`.) 1971 1972The :meth:`!asyncio.Task.current_task` and :meth:`!asyncio.Task.all_tasks` 1973methods have been deprecated. 1974(Contributed by Andrew Svetlov in :issue:`32250`.) 1975 1976 1977collections 1978----------- 1979 1980In Python 3.8, the abstract base classes in :mod:`collections.abc` will no 1981longer be exposed in the regular :mod:`collections` module. This will help 1982create a clearer distinction between the concrete classes and the abstract 1983base classes. 1984(Contributed by Serhiy Storchaka in :issue:`25988`.) 1985 1986 1987dbm 1988--- 1989 1990:mod:`dbm.dumb` now supports reading read-only files and no longer writes the 1991index file when it is not changed. A deprecation warning is now emitted 1992if the index file is missing and recreated in the ``'r'`` and ``'w'`` 1993modes (this will be an error in future Python releases). 1994(Contributed by Serhiy Storchaka in :issue:`28847`.) 1995 1996 1997enum 1998---- 1999 2000In Python 3.8, attempting to check for non-Enum objects in :class:`Enum` 2001classes will raise a :exc:`TypeError` (e.g. ``1 in Color``); similarly, 2002attempting to check for non-Flag objects in a :class:`Flag` member will 2003raise :exc:`TypeError` (e.g. ``1 in Perm.RW``); currently, both operations 2004return :const:`False` instead. 2005(Contributed by Ethan Furman in :issue:`33217`.) 2006 2007 2008gettext 2009------- 2010 2011Using non-integer value for selecting a plural form in :mod:`gettext` is 2012now deprecated. It never correctly worked. (Contributed by Serhiy Storchaka 2013in :issue:`28692`.) 2014 2015 2016importlib 2017--------- 2018 2019Methods 2020:meth:`!MetaPathFinder.find_module` 2021(replaced by 2022:meth:`MetaPathFinder.find_spec() <importlib.abc.MetaPathFinder.find_spec>`) 2023and 2024:meth:`!PathEntryFinder.find_loader` 2025(replaced by 2026:meth:`PathEntryFinder.find_spec() <importlib.abc.PathEntryFinder.find_spec>`) 2027both deprecated in Python 3.4 now emit :exc:`DeprecationWarning`. 2028(Contributed by Matthias Bussonnier in :issue:`29576`.) 2029 2030The :class:`importlib.abc.ResourceLoader` ABC has been deprecated in 2031favour of :class:`importlib.abc.ResourceReader`. 2032 2033 2034locale 2035------ 2036 2037:func:`locale.format` has been deprecated, use :meth:`locale.format_string` 2038instead. (Contributed by Garvit in :issue:`10379`.) 2039 2040 2041macpath 2042------- 2043 2044The :mod:`macpath` is now deprecated and will be removed in Python 3.8. 2045(Contributed by Chi Hsuan Yen in :issue:`9850`.) 2046 2047 2048threading 2049--------- 2050 2051:mod:`!dummy_threading` and :mod:`!_dummy_thread` have been deprecated. It is 2052no longer possible to build Python with threading disabled. 2053Use :mod:`threading` instead. 2054(Contributed by Antoine Pitrou in :issue:`31370`.) 2055 2056 2057socket 2058------ 2059 2060The silent argument value truncation in :func:`socket.htons` and 2061:func:`socket.ntohs` has been deprecated. In future versions of Python, 2062if the passed argument is larger than 16 bits, an exception will be raised. 2063(Contributed by Oren Milman in :issue:`28332`.) 2064 2065 2066ssl 2067--- 2068 2069:func:`ssl.wrap_socket` is deprecated. Use 2070:meth:`ssl.SSLContext.wrap_socket` instead. 2071(Contributed by Christian Heimes in :issue:`28124`.) 2072 2073 2074sunau 2075----- 2076 2077:func:`!sunau.openfp` has been deprecated and will be removed in Python 3.9. 2078Use :func:`!sunau.open` instead. 2079(Contributed by Brian Curtin in :issue:`31985`.) 2080 2081 2082sys 2083--- 2084 2085Deprecated :func:`sys.set_coroutine_wrapper` and 2086:func:`sys.get_coroutine_wrapper`. 2087 2088The undocumented ``sys.callstats()`` function has been deprecated and 2089will be removed in a future Python version. 2090(Contributed by Victor Stinner in :issue:`28799`.) 2091 2092 2093wave 2094---- 2095 2096:func:`wave.openfp` has been deprecated and will be removed in Python 3.9. 2097Use :func:`wave.open` instead. 2098(Contributed by Brian Curtin in :issue:`31985`.) 2099 2100 2101Deprecated functions and types of the C API 2102=========================================== 2103 2104Function :c:func:`PySlice_GetIndicesEx` is deprecated and replaced with 2105a macro if ``Py_LIMITED_API`` is not set or set to a value in the range 2106between ``0x03050400`` and ``0x03060000`` (not inclusive), or is ``0x03060100`` 2107or higher. (Contributed by Serhiy Storchaka in :issue:`27867`.) 2108 2109:c:func:`PyOS_AfterFork` has been deprecated. Use :c:func:`PyOS_BeforeFork`, 2110:c:func:`PyOS_AfterFork_Parent` or :c:func:`PyOS_AfterFork_Child()` instead. 2111(Contributed by Antoine Pitrou in :issue:`16500`.) 2112 2113 2114.. _37-platform-support-removals: 2115 2116Platform Support Removals 2117========================= 2118 2119* FreeBSD 9 and older are no longer officially supported. 2120* For full Unicode support, including within extension modules, \*nix platforms 2121 are now expected to provide at least one of ``C.UTF-8`` (full locale), 2122 ``C.utf8`` (full locale) or ``UTF-8`` (``LC_CTYPE``-only locale) as an 2123 alternative to the legacy ``ASCII``-based ``C`` locale. 2124* OpenSSL 0.9.8 and 1.0.1 are no longer supported, which means building CPython 2125 3.7 with SSL/TLS support on older platforms still using these versions 2126 requires custom build options that link to a more recent version of OpenSSL. 2127 2128 Notably, this issue affects the Debian 8 (aka "jessie") and Ubuntu 14.04 2129 (aka "Trusty") LTS Linux distributions, as they still use OpenSSL 1.0.1 by 2130 default. 2131 2132 Debian 9 ("stretch") and Ubuntu 16.04 ("xenial"), as well as recent releases 2133 of other LTS Linux releases (e.g. RHEL/CentOS 7.5, SLES 12-SP3), use OpenSSL 2134 1.0.2 or later, and remain supported in the default build configuration. 2135 2136 CPython's own `CI configuration file 2137 <https://github.com/python/cpython/blob/v3.7.13/.travis.yml>`_ provides an 2138 example of using the SSL 2139 :source:`compatibility testing infrastructure <Tools/ssl/multissltests.py>` in 2140 CPython's test suite to build and link against OpenSSL 1.1.0 rather than an 2141 outdated system provided OpenSSL. 2142 2143 2144API and Feature Removals 2145======================== 2146 2147The following features and APIs have been removed from Python 3.7: 2148 2149* The ``os.stat_float_times()`` function has been removed. It was introduced in 2150 Python 2.3 for backward compatibility with Python 2.2, and was deprecated 2151 since Python 3.1. 2152 2153* Unknown escapes consisting of ``'\'`` and an ASCII letter in replacement 2154 templates for :func:`re.sub` were deprecated in Python 3.5, and will now 2155 cause an error. 2156 2157* Removed support of the *exclude* argument in :meth:`tarfile.TarFile.add`. 2158 It was deprecated in Python 2.7 and 3.2. Use the *filter* argument instead. 2159 2160* The :func:`!ntpath.splitunc` function was deprecated in 2161 Python 3.1, and has now been removed. Use :func:`~os.path.splitdrive` 2162 instead. 2163 2164* :func:`collections.namedtuple` no longer supports the *verbose* parameter 2165 or ``_source`` attribute which showed the generated source code for the 2166 named tuple class. This was part of an optimization designed to speed-up 2167 class creation. (Contributed by Jelle Zijlstra with further improvements 2168 by INADA Naoki, Serhiy Storchaka, and Raymond Hettinger in :issue:`28638`.) 2169 2170* Functions :func:`bool`, :func:`float`, :func:`list` and :func:`tuple` no 2171 longer take keyword arguments. The first argument of :func:`int` can now 2172 be passed only as positional argument. 2173 2174* Removed previously deprecated in Python 2.4 classes ``Plist``, ``Dict`` and 2175 ``_InternalDict`` in the :mod:`plistlib` module. Dict values in the result 2176 of functions :func:`~plistlib.readPlist` and 2177 :func:`~plistlib.readPlistFromBytes` are now normal dicts. You no longer 2178 can use attribute access to access items of these dictionaries. 2179 2180* The ``asyncio.windows_utils.socketpair()`` function has been 2181 removed. Use the :func:`socket.socketpair` function instead, 2182 it is available on all platforms since Python 3.5. 2183 ``asyncio.windows_utils.socketpair`` was just an alias to 2184 ``socket.socketpair`` on Python 3.5 and newer. 2185 2186* :mod:`asyncio` no longer exports the :mod:`selectors` and 2187 :mod:`!_overlapped` modules as ``asyncio.selectors`` and 2188 ``asyncio._overlapped``. Replace ``from asyncio import selectors`` with 2189 ``import selectors``. 2190 2191* Direct instantiation of :class:`ssl.SSLSocket` and :class:`ssl.SSLObject` 2192 objects is now prohibited. The constructors were never documented, tested, 2193 or designed as public constructors. Users were supposed to use 2194 :func:`ssl.wrap_socket` or :class:`ssl.SSLContext`. 2195 (Contributed by Christian Heimes in :issue:`32951`.) 2196 2197* The unused ``distutils`` ``install_misc`` command has been removed. 2198 (Contributed by Eric N. Vander Weele in :issue:`29218`.) 2199 2200 2201Module Removals 2202=============== 2203 2204The ``fpectl`` module has been removed. It was never enabled by 2205default, never worked correctly on x86-64, and it changed the Python 2206ABI in ways that caused unexpected breakage of C extensions. 2207(Contributed by Nathaniel J. Smith in :issue:`29137`.) 2208 2209 2210Windows-only Changes 2211==================== 2212 2213The python launcher, (py.exe), can accept 32 & 64 bit specifiers **without** 2214having to specify a minor version as well. So ``py -3-32`` and ``py -3-64`` 2215become valid as well as ``py -3.7-32``, also the -*m*-64 and -*m.n*-64 forms 2216are now accepted to force 64 bit python even if 32 bit would have otherwise 2217been used. If the specified version is not available py.exe will error exit. 2218(Contributed by Steve Barnes in :issue:`30291`.) 2219 2220The launcher can be run as ``py -0`` to produce a list of the installed pythons, 2221*with default marked with an asterisk*. Running ``py -0p`` will include the paths. 2222If py is run with a version specifier that cannot be matched it will also print 2223the *short form* list of available specifiers. 2224(Contributed by Steve Barnes in :issue:`30362`.) 2225 2226 2227.. _porting-to-python-37: 2228 2229Porting to Python 3.7 2230===================== 2231 2232This section lists previously described changes and other bugfixes 2233that may require changes to your code. 2234 2235 2236Changes in Python Behavior 2237-------------------------- 2238 2239* :keyword:`async` and :keyword:`await` names are now reserved keywords. 2240 Code using these names as identifiers will now raise a :exc:`SyntaxError`. 2241 (Contributed by Jelle Zijlstra in :issue:`30406`.) 2242 2243* :pep:`479` is enabled for all code in Python 3.7, meaning that 2244 :exc:`StopIteration` exceptions raised directly or indirectly in 2245 coroutines and generators are transformed into :exc:`RuntimeError` 2246 exceptions. 2247 (Contributed by Yury Selivanov in :issue:`32670`.) 2248 2249* :meth:`object.__aiter__` methods can no longer be declared as 2250 asynchronous. (Contributed by Yury Selivanov in :issue:`31709`.) 2251 2252* Due to an oversight, earlier Python versions erroneously accepted the 2253 following syntax:: 2254 2255 f(1 for x in [1],) 2256 2257 class C(1 for x in [1]): 2258 pass 2259 2260 Python 3.7 now correctly raises a :exc:`SyntaxError`, as a generator 2261 expression always needs to be directly inside a set of parentheses 2262 and cannot have a comma on either side, and the duplication of the 2263 parentheses can be omitted only on calls. 2264 (Contributed by Serhiy Storchaka in :issue:`32012` and :issue:`32023`.) 2265 2266* When using the :option:`-m` switch, the initial working directory is now added 2267 to :data:`sys.path`, rather than an empty string (which dynamically denoted 2268 the current working directory at the time of each import). Any programs that 2269 are checking for the empty string, or otherwise relying on the previous 2270 behaviour, will need to be updated accordingly (e.g. by also checking for 2271 ``os.getcwd()`` or ``os.path.dirname(__main__.__file__)``, depending on why 2272 the code was checking for the empty string in the first place). 2273 2274 2275Changes in the Python API 2276------------------------- 2277 2278* :meth:`socketserver.ThreadingMixIn.server_close` now waits until all 2279 non-daemon threads complete. Set the new 2280 :attr:`socketserver.ThreadingMixIn.block_on_close` class attribute to 2281 ``False`` to get the pre-3.7 behaviour. 2282 (Contributed by Victor Stinner in :issue:`31233` and :issue:`33540`.) 2283 2284* :meth:`socketserver.ForkingMixIn.server_close` now waits until all 2285 child processes complete. Set the new 2286 :attr:`socketserver.ForkingMixIn.block_on_close` class attribute to ``False`` 2287 to get the pre-3.7 behaviour. 2288 (Contributed by Victor Stinner in :issue:`31151` and :issue:`33540`.) 2289 2290* The :func:`locale.localeconv` function now temporarily sets the ``LC_CTYPE`` 2291 locale to the value of ``LC_NUMERIC`` in some cases. 2292 (Contributed by Victor Stinner in :issue:`31900`.) 2293 2294* :meth:`pkgutil.walk_packages` now raises a :exc:`ValueError` if *path* is 2295 a string. Previously an empty list was returned. 2296 (Contributed by Sanyam Khurana in :issue:`24744`.) 2297 2298* A format string argument for :meth:`string.Formatter.format` 2299 is now :ref:`positional-only <positional-only_parameter>`. 2300 Passing it as a keyword argument was deprecated in Python 3.5. (Contributed 2301 by Serhiy Storchaka in :issue:`29193`.) 2302 2303* Attributes :attr:`~http.cookies.Morsel.key`, 2304 :attr:`~http.cookies.Morsel.value` and 2305 :attr:`~http.cookies.Morsel.coded_value` of class 2306 :class:`http.cookies.Morsel` are now read-only. 2307 Assigning to them was deprecated in Python 3.5. 2308 Use the :meth:`~http.cookies.Morsel.set` method for setting them. 2309 (Contributed by Serhiy Storchaka in :issue:`29192`.) 2310 2311* The *mode* argument of :func:`os.makedirs` no longer affects the file 2312 permission bits of newly created intermediate-level directories. 2313 To set their file permission bits you can set the umask before invoking 2314 ``makedirs()``. 2315 (Contributed by Serhiy Storchaka in :issue:`19930`.) 2316 2317* The :attr:`struct.Struct.format` type is now :class:`str` instead of 2318 :class:`bytes`. (Contributed by Victor Stinner in :issue:`21071`.) 2319 2320* :func:`!cgi.parse_multipart` now accepts the *encoding* and *errors* 2321 arguments and returns the same results as 2322 :class:`!FieldStorage`: for non-file fields, the value associated to a key 2323 is a list of strings, not bytes. 2324 (Contributed by Pierre Quentel in :issue:`29979`.) 2325 2326* Due to internal changes in :mod:`socket`, calling :func:`socket.fromshare` 2327 on a socket created by :func:`socket.share <socket.socket.share>` in older 2328 Python versions is not supported. 2329 2330* ``repr`` for :exc:`BaseException` has changed to not include the trailing 2331 comma. Most exceptions are affected by this change. 2332 (Contributed by Serhiy Storchaka in :issue:`30399`.) 2333 2334* ``repr`` for :class:`datetime.timedelta` has changed to include the keyword 2335 arguments in the output. (Contributed by Utkarsh Upadhyay in :issue:`30302`.) 2336 2337* Because :func:`shutil.rmtree` is now implemented using the :func:`os.scandir` 2338 function, the user specified handler *onerror* is now called with the first 2339 argument ``os.scandir`` instead of ``os.listdir`` when listing the directory 2340 is failed. 2341 2342* Support for nested sets and set operations in regular expressions as in 2343 `Unicode Technical Standard #18`_ might be added in the future. This would 2344 change the syntax. To facilitate this future change a :exc:`FutureWarning` 2345 will be raised in ambiguous cases for the time being. 2346 That include sets starting with a literal ``'['`` or containing literal 2347 character sequences ``'--'``, ``'&&'``, ``'~~'``, and ``'||'``. To 2348 avoid a warning, escape them with a backslash. 2349 (Contributed by Serhiy Storchaka in :issue:`30349`.) 2350 2351 .. _Unicode Technical Standard #18: https://unicode.org/reports/tr18/ 2352 2353* The result of splitting a string on a :mod:`regular expression <re>` 2354 that could match an empty string has been changed. For example 2355 splitting on ``r'\s*'`` will now split not only on whitespaces as it 2356 did previously, but also on empty strings before all non-whitespace 2357 characters and just before the end of the string. 2358 The previous behavior can be restored by changing the pattern 2359 to ``r'\s+'``. A :exc:`FutureWarning` was emitted for such patterns since 2360 Python 3.5. 2361 2362 For patterns that match both empty and non-empty strings, the result of 2363 searching for all matches may also be changed in other cases. For example 2364 in the string ``'a\n\n'``, the pattern ``r'(?m)^\s*?$'`` will not only 2365 match empty strings at positions 2 and 3, but also the string ``'\n'`` at 2366 positions 2--3. To match only blank lines, the pattern should be rewritten 2367 as ``r'(?m)^[^\S\n]*$'``. 2368 2369 :func:`re.sub` now replaces empty matches adjacent to a previous 2370 non-empty match. For example ``re.sub('x*', '-', 'abxd')`` returns now 2371 ``'-a-b--d-'`` instead of ``'-a-b-d-'`` (the first minus between 'b' and 2372 'd' replaces 'x', and the second minus replaces an empty string between 2373 'x' and 'd'). 2374 2375 (Contributed by Serhiy Storchaka in :issue:`25054` and :issue:`32308`.) 2376 2377* Change :func:`re.escape` to only escape regex special characters instead 2378 of escaping all characters other than ASCII letters, numbers, and ``'_'``. 2379 (Contributed by Serhiy Storchaka in :issue:`29995`.) 2380 2381* :class:`tracemalloc.Traceback` frames are now sorted from oldest to most 2382 recent to be more consistent with :mod:`traceback`. 2383 (Contributed by Jesse Bakker in :issue:`32121`.) 2384 2385* On OSes that support :const:`socket.SOCK_NONBLOCK` or 2386 :const:`socket.SOCK_CLOEXEC` bit flags, the 2387 :attr:`socket.type <socket.socket.type>` no longer has them applied. 2388 Therefore, checks like ``if sock.type == socket.SOCK_STREAM`` 2389 work as expected on all platforms. 2390 (Contributed by Yury Selivanov in :issue:`32331`.) 2391 2392* On Windows the default for the *close_fds* argument of 2393 :class:`subprocess.Popen` was changed from :const:`False` to :const:`True` 2394 when redirecting the standard handles. If you previously depended on handles 2395 being inherited when using :class:`subprocess.Popen` with standard io 2396 redirection, you will have to pass ``close_fds=False`` to preserve the 2397 previous behaviour, or use 2398 :attr:`STARTUPINFO.lpAttributeList <subprocess.STARTUPINFO.lpAttributeList>`. 2399 2400* :meth:`importlib.machinery.PathFinder.invalidate_caches` -- which implicitly 2401 affects :func:`importlib.invalidate_caches` -- now deletes entries 2402 in :data:`sys.path_importer_cache` which are set to ``None``. 2403 (Contributed by Brett Cannon in :issue:`33169`.) 2404 2405* In :mod:`asyncio`, 2406 :meth:`loop.sock_recv() <asyncio.loop.sock_recv>`, 2407 :meth:`loop.sock_sendall() <asyncio.loop.sock_sendall>`, 2408 :meth:`loop.sock_accept() <asyncio.loop.sock_accept>`, 2409 :meth:`loop.getaddrinfo() <asyncio.loop.getaddrinfo>`, 2410 :meth:`loop.getnameinfo() <asyncio.loop.getnameinfo>` 2411 have been changed to be proper coroutine methods to match their 2412 documentation. Previously, these methods returned :class:`asyncio.Future` 2413 instances. 2414 (Contributed by Yury Selivanov in :issue:`32327`.) 2415 2416* :attr:`asyncio.Server.sockets` now returns a copy of the internal list 2417 of server sockets, instead of returning it directly. 2418 (Contributed by Yury Selivanov in :issue:`32662`.) 2419 2420* :attr:`Struct.format <struct.Struct.format>` is now a :class:`str` instance 2421 instead of a :class:`bytes` instance. 2422 (Contributed by Victor Stinner in :issue:`21071`.) 2423 2424* :mod:`argparse` subparsers can now be made mandatory by passing ``required=True`` 2425 to :meth:`ArgumentParser.add_subparsers() <argparse.ArgumentParser.add_subparsers>`. 2426 (Contributed by Anthony Sottile in :issue:`26510`.) 2427 2428* :meth:`ast.literal_eval` is now stricter. Addition and subtraction of 2429 arbitrary numbers are no longer allowed. 2430 (Contributed by Serhiy Storchaka in :issue:`31778`.) 2431 2432* :meth:`Calendar.itermonthdates <calendar.Calendar.itermonthdates>` 2433 will now consistently raise an exception when a date falls outside of the 2434 ``0001-01-01`` through ``9999-12-31`` range. To support applications that 2435 cannot tolerate such exceptions, the new 2436 :meth:`Calendar.itermonthdays3 <calendar.Calendar.itermonthdays3>` and 2437 :meth:`Calendar.itermonthdays4 <calendar.Calendar.itermonthdays4>` can be used. 2438 The new methods return tuples and are not restricted by the range supported by 2439 :class:`datetime.date`. 2440 (Contributed by Alexander Belopolsky in :issue:`28292`.) 2441 2442* :class:`collections.ChainMap` now preserves the order of the underlying 2443 mappings. (Contributed by Raymond Hettinger in :issue:`32792`.) 2444 2445* The ``submit()`` method of :class:`concurrent.futures.ThreadPoolExecutor` 2446 and :class:`concurrent.futures.ProcessPoolExecutor` now raises 2447 a :exc:`RuntimeError` if called during interpreter shutdown. 2448 (Contributed by Mark Nemec in :issue:`33097`.) 2449 2450* The :class:`configparser.ConfigParser` constructor now uses ``read_dict()`` 2451 to process the default values, making its behavior consistent with the 2452 rest of the parser. Non-string keys and values in the defaults 2453 dictionary are now being implicitly converted to strings. 2454 (Contributed by James Tocknell in :issue:`23835`.) 2455 2456* Several undocumented internal imports were removed. 2457 One example is that ``os.errno`` is no longer available; use ``import errno`` 2458 directly instead. 2459 Note that such undocumented internal imports may be removed any time without 2460 notice, even in micro version releases. 2461 2462 2463Changes in the C API 2464-------------------- 2465 2466The function :c:func:`PySlice_GetIndicesEx` is considered unsafe for 2467resizable sequences. If the slice indices are not instances of :class:`int`, 2468but objects that implement the :meth:`!__index__` method, the sequence can be 2469resized after passing its length to :c:func:`!PySlice_GetIndicesEx`. This 2470can lead to returning indices out of the length of the sequence. For 2471avoiding possible problems use new functions :c:func:`PySlice_Unpack` and 2472:c:func:`PySlice_AdjustIndices`. 2473(Contributed by Serhiy Storchaka in :issue:`27867`.) 2474 2475 2476CPython bytecode changes 2477------------------------ 2478 2479There are two new opcodes: :opcode:`LOAD_METHOD` and :opcode:`!CALL_METHOD`. 2480(Contributed by Yury Selivanov and INADA Naoki in :issue:`26110`.) 2481 2482The :opcode:`!STORE_ANNOTATION` opcode has been removed. 2483(Contributed by Mark Shannon in :issue:`32550`.) 2484 2485 2486Windows-only Changes 2487-------------------- 2488 2489The file used to override :data:`sys.path` is now called 2490``<python-executable>._pth`` instead of ``'sys.path'``. 2491See :ref:`windows_finding_modules` for more information. 2492(Contributed by Steve Dower in :issue:`28137`.) 2493 2494 2495Other CPython implementation changes 2496------------------------------------ 2497 2498In preparation for potential future changes to the public CPython runtime 2499initialization API (see :pep:`432` for an initial, but somewhat outdated, 2500draft), CPython's internal startup 2501and configuration management logic has been significantly refactored. While 2502these updates are intended to be entirely transparent to both embedding 2503applications and users of the regular CPython CLI, they're being mentioned 2504here as the refactoring changes the internal order of various operations 2505during interpreter startup, and hence may uncover previously latent defects, 2506either in embedding applications, or in CPython itself. 2507(Initially contributed by Nick Coghlan and Eric Snow as part of 2508:issue:`22257`, and further updated by Nick, Eric, and Victor Stinner in a 2509number of other issues). Some known details affected: 2510 2511* :c:func:`!PySys_AddWarnOptionUnicode` is not currently usable by embedding 2512 applications due to the requirement to create a Unicode object prior to 2513 calling ``Py_Initialize``. Use :c:func:`!PySys_AddWarnOption` instead. 2514 2515* warnings filters added by an embedding application with 2516 :c:func:`!PySys_AddWarnOption` should now more consistently take precedence 2517 over the default filters set by the interpreter 2518 2519Due to changes in the way the default warnings filters are configured, 2520setting :c:data:`Py_BytesWarningFlag` to a value greater than one is no longer 2521sufficient to both emit :exc:`BytesWarning` messages and have them converted 2522to exceptions. Instead, the flag must be set (to cause the warnings to be 2523emitted in the first place), and an explicit ``error::BytesWarning`` 2524warnings filter added to convert them to exceptions. 2525 2526Due to a change in the way docstrings are handled by the compiler, the 2527implicit ``return None`` in a function body consisting solely of a docstring 2528is now marked as occurring on the same line as the docstring, not on the 2529function's header line. 2530 2531The current exception state has been moved from the frame object to the co-routine. 2532This simplified the interpreter and fixed a couple of obscure bugs caused by 2533having swap exception state when entering or exiting a generator. 2534(Contributed by Mark Shannon in :issue:`25612`.) 2535 2536Notable changes in Python 3.7.1 2537=============================== 2538 2539Starting in 3.7.1, :c:func:`Py_Initialize` now consistently reads and respects 2540all of the same environment settings as :c:func:`Py_Main` (in earlier Python 2541versions, it respected an ill-defined subset of those environment variables, 2542while in Python 3.7.0 it didn't read any of them due to :issue:`34247`). If 2543this behavior is unwanted, set :c:data:`Py_IgnoreEnvironmentFlag` to 1 before 2544calling :c:func:`Py_Initialize`. 2545 2546In 3.7.1 the C API for Context Variables 2547:ref:`was updated <contextvarsobjects_pointertype_change>` to use 2548:c:type:`PyObject` pointers. See also :issue:`34762`. 2549 2550In 3.7.1 the :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token 2551when provided with input that does not have a trailing new line. This behavior 2552now matches what the C tokenizer does internally. 2553(Contributed by Ammar Askar in :issue:`33899`.) 2554 2555Notable changes in Python 3.7.2 2556=============================== 2557 2558In 3.7.2, :mod:`venv` on Windows no longer copies the original binaries, but 2559creates redirector scripts named ``python.exe`` and ``pythonw.exe`` instead. 2560This resolves a long standing issue where all virtual environments would have 2561to be upgraded or recreated with each Python update. However, note that this 2562release will still require recreation of virtual environments in order to get 2563the new scripts. 2564 2565Notable changes in Python 3.7.6 2566=============================== 2567 2568Due to significant security concerns, the *reuse_address* parameter of 2569:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is 2570because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more 2571details, see the documentation for ``loop.create_datagram_endpoint()``. 2572(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 2573:issue:`37228`.) 2574 2575Notable changes in Python 3.7.10 2576================================ 2577 2578Earlier Python versions allowed using both ``;`` and ``&`` as 2579query parameter separators in :func:`urllib.parse.parse_qs` and 2580:func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with 2581newer W3C recommendations, this has been changed to allow only a single 2582separator key, with ``&`` as the default. This change also affects 2583:func:`!cgi.parse` and :func:`!cgi.parse_multipart` as they use the affected 2584functions internally. For more details, please see their respective 2585documentation. 2586(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.) 2587 2588Notable changes in Python 3.7.11 2589================================ 2590 2591A security fix alters the :class:`ftplib.FTP` behavior to not trust the 2592IPv4 address sent from the remote server when setting up a passive data 2593channel. We reuse the ftp server IP address instead. For unusual code 2594requiring the old behavior, set a ``trust_server_pasv_ipv4_address`` 2595attribute on your FTP instance to ``True``. (See :gh:`87451`) 2596 2597 2598The presence of newline or tab characters in parts of a URL allows for some 2599forms of attacks. Following the WHATWG specification that updates RFC 3986, 2600ASCII newline ``\n``, ``\r`` and tab ``\t`` characters are stripped from the 2601URL by the parser :func:`urllib.parse` preventing such attacks. The removal 2602characters are controlled by a new module level variable 2603``urllib.parse._UNSAFE_URL_BYTES_TO_REMOVE``. (See :gh:`88048`) 2604 2605Notable security feature in 3.7.14 2606================================== 2607 2608Converting between :class:`int` and :class:`str` in bases other than 2 2609(binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) 2610now raises a :exc:`ValueError` if the number of digits in string form is 2611above a limit to avoid potential denial of service attacks due to the 2612algorithmic complexity. This is a mitigation for :cve:`2020-10735`. 2613This limit can be configured or disabled by environment variable, command 2614line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion 2615length limitation <int_max_str_digits>` documentation. The default limit 2616is 4300 digits in string form. 2617