1**************************** 2 What's New In Python 3.8 3**************************** 4 5.. Rules for maintenance: 6 7 * Anyone can add text to this document. Do not spend very much time 8 on the wording of your changes, because your text will probably 9 get rewritten to some degree. 10 11 * The maintainer will go through Misc/NEWS periodically and add 12 changes; it's therefore more important to add your changes to 13 Misc/NEWS than to this file. 14 15 * This is not a complete list of every single change; completeness 16 is the purpose of Misc/NEWS. Some changes I consider too small 17 or esoteric to include. If such a change is added to the text, 18 I'll just remove it. (This is another reason you shouldn't spend 19 too much time on writing your addition.) 20 21 * If you want to draw your new text to the attention of the 22 maintainer, add 'XXX' to the beginning of the paragraph or 23 section. 24 25 * It's OK to just add a fragmentary note about a change. For 26 example: "XXX Describe the transmogrify() function added to the 27 socket module." The maintainer will research the change and 28 write the necessary text. 29 30 * You can comment out your additions if you like, but it's not 31 necessary (especially when a final release is some months away). 32 33 * Credit the author of a patch or bugfix. Just the name is 34 sufficient; the e-mail address isn't necessary. 35 36 * It's helpful to add the bug/patch number as a comment: 37 38 XXX Describe the transmogrify() function added to the socket 39 module. 40 (Contributed by P.Y. Developer in :issue:`12345`.) 41 42 This saves the maintainer the effort of going through the Git log 43 when researching a change. 44 45:Editor: Raymond Hettinger 46 47This article explains the new features in Python 3.8, compared to 3.7. 48For full details, see the :ref:`changelog <changelog>`. 49 50Python 3.8 was released on October 14th, 2019. 51 52.. testsetup:: 53 54 from datetime import date 55 from math import cos, radians 56 from unicodedata import normalize 57 import re 58 import math 59 60 61Summary -- Release highlights 62============================= 63 64.. This section singles out the most important changes in Python 3.8. 65 Brevity is key. 66 67 68.. PEP-sized items next. 69 70 71 72New Features 73============ 74 75Assignment expressions 76---------------------- 77 78There is new syntax ``:=`` that assigns values to variables as part of a larger 79expression. It is affectionately known as "the walrus operator" due to 80its resemblance to `the eyes and tusks of a walrus 81<https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(8247646168).jpg>`_. 82 83In this example, the assignment expression helps avoid calling 84:func:`len` twice:: 85 86 if (n := len(a)) > 10: 87 print(f"List is too long ({n} elements, expected <= 10)") 88 89A similar benefit arises during regular expression matching where 90match objects are needed twice, once to test whether a match 91occurred and another to extract a subgroup:: 92 93 discount = 0.0 94 if (mo := re.search(r'(\d+)% discount', advertisement)): 95 discount = float(mo.group(1)) / 100.0 96 97The operator is also useful with while-loops that compute 98a value to test loop termination and then need that same 99value again in the body of the loop:: 100 101 # Loop over fixed length blocks 102 while (block := f.read(256)) != '': 103 process(block) 104 105Another motivating use case arises in list comprehensions where 106a value computed in a filtering condition is also needed in 107the expression body:: 108 109 [clean_name.title() for name in names 110 if (clean_name := normalize('NFC', name)) in allowed_names] 111 112Try to limit use of the walrus operator to clean cases that reduce 113complexity and improve readability. 114 115See :pep:`572` for a full description. 116 117(Contributed by Emily Morehouse in :issue:`35224`.) 118 119 120Positional-only parameters 121-------------------------- 122 123There is a new function parameter syntax ``/`` to indicate that some 124function parameters must be specified positionally and cannot be used as 125keyword arguments. This is the same notation shown by ``help()`` for C 126functions annotated with Larry Hastings' `Argument Clinic 127<https://docs.python.org/3/howto/clinic.html>`_ tool. 128 129In the following example, parameters *a* and *b* are positional-only, 130while *c* or *d* can be positional or keyword, and *e* or *f* are 131required to be keywords:: 132 133 def f(a, b, /, c, d, *, e, f): 134 print(a, b, c, d, e, f) 135 136The following is a valid call:: 137 138 f(10, 20, 30, d=40, e=50, f=60) 139 140However, these are invalid calls:: 141 142 f(10, b=20, c=30, d=40, e=50, f=60) # b cannot be a keyword argument 143 f(10, 20, 30, 40, 50, f=60) # e must be a keyword argument 144 145One use case for this notation is that it allows pure Python functions 146to fully emulate behaviors of existing C coded functions. For example, 147the built-in :func:`pow` function does not accept keyword arguments:: 148 149 def pow(x, y, z=None, /): 150 "Emulate the built in pow() function" 151 r = x ** y 152 return r if z is None else r%z 153 154Another use case is to preclude keyword arguments when the parameter 155name is not helpful. For example, the builtin :func:`len` function has 156the signature ``len(obj, /)``. This precludes awkward calls such as:: 157 158 len(obj='hello') # The "obj" keyword argument impairs readability 159 160A further benefit of marking a parameter as positional-only is that it 161allows the parameter name to be changed in the future without risk of 162breaking client code. For example, in the :mod:`statistics` module, the 163parameter name *dist* may be changed in the future. This was made 164possible with the following function specification:: 165 166 def quantiles(dist, /, *, n=4, method='exclusive') 167 ... 168 169Since the parameters to the left of ``/`` are not exposed as possible 170keywords, the parameters names remain available for use in ``**kwargs``:: 171 172 >>> def f(a, b, /, **kwargs): 173 ... print(a, b, kwargs) 174 ... 175 >>> f(10, 20, a=1, b=2, c=3) # a and b are used in two ways 176 10 20 {'a': 1, 'b': 2, 'c': 3} 177 178This greatly simplifies the implementation of functions and methods 179that need to accept arbitrary keyword arguments. For example, here 180is an excerpt from code in the :mod:`collections` module:: 181 182 class Counter(dict): 183 184 def __init__(self, iterable=None, /, **kwds): 185 # Note "iterable" is a possible keyword argument 186 187See :pep:`570` for a full description. 188 189(Contributed by Pablo Galindo in :issue:`36540`.) 190 191.. TODO: Pablo will sprint on docs at PyCon US 2019. 192 193 194Parallel filesystem cache for compiled bytecode files 195----------------------------------------------------- 196 197The new :envvar:`PYTHONPYCACHEPREFIX` setting (also available as 198:option:`-X` ``pycache_prefix``) configures the implicit bytecode 199cache to use a separate parallel filesystem tree, rather than 200the default ``__pycache__`` subdirectories within each source 201directory. 202 203The location of the cache is reported in :data:`sys.pycache_prefix` 204(:const:`None` indicates the default location in ``__pycache__`` 205subdirectories). 206 207(Contributed by Carl Meyer in :issue:`33499`.) 208 209 210Debug build uses the same ABI as release build 211----------------------------------------------- 212 213Python now uses the same ABI whether it's built in release or debug mode. On 214Unix, when Python is built in debug mode, it is now possible to load C 215extensions built in release mode and C extensions built using the stable ABI. 216 217Release builds and debug builds are now ABI compatible: defining the 218``Py_DEBUG`` macro no longer implies the ``Py_TRACE_REFS`` macro, which 219introduces the only ABI incompatibility. The ``Py_TRACE_REFS`` macro, which 220adds the :func:`sys.getobjects` function and the :envvar:`PYTHONDUMPREFS` 221environment variable, can be set using the new ``./configure --with-trace-refs`` 222build option. 223(Contributed by Victor Stinner in :issue:`36465`.) 224 225On Unix, C extensions are no longer linked to libpython except on Android 226and Cygwin. 227It is now possible 228for a statically linked Python to load a C extension built using a shared 229library Python. 230(Contributed by Victor Stinner in :issue:`21536`.) 231 232On Unix, when Python is built in debug mode, import now also looks for C 233extensions compiled in release mode and for C extensions compiled with the 234stable ABI. 235(Contributed by Victor Stinner in :issue:`36722`.) 236 237To embed Python into an application, a new ``--embed`` option must be passed to 238``python3-config --libs --embed`` to get ``-lpython3.8`` (link the application 239to libpython). To support both 3.8 and older, try ``python3-config --libs 240--embed`` first and fallback to ``python3-config --libs`` (without ``--embed``) 241if the previous command fails. 242 243Add a pkg-config ``python-3.8-embed`` module to embed Python into an 244application: ``pkg-config python-3.8-embed --libs`` includes ``-lpython3.8``. 245To support both 3.8 and older, try ``pkg-config python-X.Y-embed --libs`` first 246and fallback to ``pkg-config python-X.Y --libs`` (without ``--embed``) if the 247previous command fails (replace ``X.Y`` with the Python version). 248 249On the other hand, ``pkg-config python3.8 --libs`` no longer contains 250``-lpython3.8``. C extensions must not be linked to libpython (except on 251Android and Cygwin, whose cases are handled by the script); 252this change is backward incompatible on purpose. 253(Contributed by Victor Stinner in :issue:`36721`.) 254 255 256f-strings support ``=`` for self-documenting expressions and debugging 257---------------------------------------------------------------------- 258 259Added an ``=`` specifier to :term:`f-string`\s. An f-string such as 260``f'{expr=}'`` will expand to the text of the expression, an equal sign, 261then the representation of the evaluated expression. For example: 262 263 >>> user = 'eric_idle' 264 >>> member_since = date(1975, 7, 31) 265 >>> f'{user=} {member_since=}' 266 "user='eric_idle' member_since=datetime.date(1975, 7, 31)" 267 268The usual :ref:`f-string format specifiers <f-strings>` allow more 269control over how the result of the expression is displayed:: 270 271 >>> delta = date.today() - member_since 272 >>> f'{user=!s} {delta.days=:,d}' 273 'user=eric_idle delta.days=16,075' 274 275The ``=`` specifier will display the whole expression so that 276calculations can be shown:: 277 278 >>> print(f'{theta=} {cos(radians(theta))=:.3f}') 279 theta=30 cos(radians(theta))=0.866 280 281(Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.) 282 283 284PEP 578: Python Runtime Audit Hooks 285----------------------------------- 286 287The PEP adds an Audit Hook and Verified Open Hook. Both are available from 288Python and native code, allowing applications and frameworks written in pure 289Python code to take advantage of extra notifications, while also allowing 290embedders or system administrators to deploy builds of Python where auditing is 291always enabled. 292 293See :pep:`578` for full details. 294 295 296PEP 587: Python Initialization Configuration 297-------------------------------------------- 298 299The :pep:`587` adds a new C API to configure the Python Initialization 300providing finer control on the whole configuration and better error reporting. 301 302New structures: 303 304* :c:type:`PyConfig` 305* :c:type:`PyPreConfig` 306* :c:type:`PyStatus` 307* :c:type:`PyWideStringList` 308 309New functions: 310 311* :c:func:`PyConfig_Clear` 312* :c:func:`PyConfig_InitIsolatedConfig` 313* :c:func:`PyConfig_InitPythonConfig` 314* :c:func:`PyConfig_Read` 315* :c:func:`PyConfig_SetArgv` 316* :c:func:`PyConfig_SetBytesArgv` 317* :c:func:`PyConfig_SetBytesString` 318* :c:func:`PyConfig_SetString` 319* :c:func:`PyPreConfig_InitIsolatedConfig` 320* :c:func:`PyPreConfig_InitPythonConfig` 321* :c:func:`PyStatus_Error` 322* :c:func:`PyStatus_Exception` 323* :c:func:`PyStatus_Exit` 324* :c:func:`PyStatus_IsError` 325* :c:func:`PyStatus_IsExit` 326* :c:func:`PyStatus_NoMemory` 327* :c:func:`PyStatus_Ok` 328* :c:func:`PyWideStringList_Append` 329* :c:func:`PyWideStringList_Insert` 330* :c:func:`Py_BytesMain` 331* :c:func:`Py_ExitStatusException` 332* :c:func:`Py_InitializeFromConfig` 333* :c:func:`Py_PreInitialize` 334* :c:func:`Py_PreInitializeFromArgs` 335* :c:func:`Py_PreInitializeFromBytesArgs` 336* :c:func:`Py_RunMain` 337 338This PEP also adds ``_PyRuntimeState.preconfig`` (:c:type:`PyPreConfig` type) 339and ``PyInterpreterState.config`` (:c:type:`PyConfig` type) fields to these 340internal structures. ``PyInterpreterState.config`` becomes the new 341reference configuration, replacing global configuration variables and 342other private variables. 343 344See :ref:`Python Initialization Configuration <init-config>` for the 345documentation. 346 347See :pep:`587` for a full description. 348 349(Contributed by Victor Stinner in :issue:`36763`.) 350 351 352Vectorcall: a fast calling protocol for CPython 353----------------------------------------------- 354 355The "vectorcall" protocol is added to the Python/C API. 356It is meant to formalize existing optimizations which were already done 357for various classes. 358Any extension type implementing a callable can use this protocol. 359 360This is currently provisional. 361The aim is to make it fully public in Python 3.9. 362 363See :pep:`590` for a full description. 364 365(Contributed by Jeroen Demeyer and Mark Shannon in :issue:`36974`.) 366 367 368Pickle protocol 5 with out-of-band data buffers 369----------------------------------------------- 370 371When :mod:`pickle` is used to transfer large data between Python processes 372in order to take advantage of multi-core or multi-machine processing, 373it is important to optimize the transfer by reducing memory copies, and 374possibly by applying custom techniques such as data-dependent compression. 375 376The :mod:`pickle` protocol 5 introduces support for out-of-band buffers 377where :pep:`3118`-compatible data can be transmitted separately from the 378main pickle stream, at the discretion of the communication layer. 379 380See :pep:`574` for a full description. 381 382(Contributed by Antoine Pitrou in :issue:`36785`.) 383 384 385Other Language Changes 386====================== 387 388* A :keyword:`continue` statement was illegal in the :keyword:`finally` clause 389 due to a problem with the implementation. In Python 3.8 this restriction 390 was lifted. 391 (Contributed by Serhiy Storchaka in :issue:`32489`.) 392 393* The :class:`bool`, :class:`int`, and :class:`fractions.Fraction` types 394 now have an :meth:`~int.as_integer_ratio` method like that found in 395 :class:`float` and :class:`decimal.Decimal`. This minor API extension 396 makes it possible to write ``numerator, denominator = 397 x.as_integer_ratio()`` and have it work across multiple numeric types. 398 (Contributed by Lisa Roach in :issue:`33073` and Raymond Hettinger in 399 :issue:`37819`.) 400 401* Constructors of :class:`int`, :class:`float` and :class:`complex` will now 402 use the :meth:`~object.__index__` special method, if available and the 403 corresponding method :meth:`~object.__int__`, :meth:`~object.__float__` 404 or :meth:`~object.__complex__` is not available. 405 (Contributed by Serhiy Storchaka in :issue:`20092`.) 406 407* Added support of ``\N{name}`` escapes in :mod:`regular expressions <re>`:: 408 409 >>> notice = 'Copyright © 2019' 410 >>> copyright_year_pattern = re.compile(r'\N{copyright sign}\s*(\d{4})') 411 >>> int(copyright_year_pattern.search(notice).group(1)) 412 2019 413 414 (Contributed by Jonathan Eunice and Serhiy Storchaka in :issue:`30688`.) 415 416* Dict and dictviews are now iterable in reversed insertion order using 417 :func:`reversed`. (Contributed by Rémi Lapeyre in :issue:`33462`.) 418 419* The syntax allowed for keyword names in function calls was further 420 restricted. In particular, ``f((keyword)=arg)`` is no longer allowed. It was 421 never intended to permit more than a bare name on the left-hand side of a 422 keyword argument assignment term. 423 (Contributed by Benjamin Peterson in :issue:`34641`.) 424 425* Generalized iterable unpacking in :keyword:`yield` and 426 :keyword:`return` statements no longer requires enclosing parentheses. 427 This brings the *yield* and *return* syntax into better agreement with 428 normal assignment syntax:: 429 430 >>> def parse(family): 431 lastname, *members = family.split() 432 return lastname.upper(), *members 433 434 >>> parse('simpsons homer marge bart lisa sally') 435 ('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'sally') 436 437 (Contributed by David Cuthbert and Jordan Chapman in :issue:`32117`.) 438 439* When a comma is missed in code such as ``[(10, 20) (30, 40)]``, the 440 compiler displays a :exc:`SyntaxWarning` with a helpful suggestion. 441 This improves on just having a :exc:`TypeError` indicating that the 442 first tuple was not callable. (Contributed by Serhiy Storchaka in 443 :issue:`15248`.) 444 445* Arithmetic operations between subclasses of :class:`datetime.date` or 446 :class:`datetime.datetime` and :class:`datetime.timedelta` objects now return 447 an instance of the subclass, rather than the base class. This also affects 448 the return type of operations whose implementation (directly or indirectly) 449 uses :class:`datetime.timedelta` arithmetic, such as 450 :meth:`~datetime.datetime.astimezone`. 451 (Contributed by Paul Ganssle in :issue:`32417`.) 452 453* When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the 454 resulting :exc:`KeyboardInterrupt` exception is not caught, the Python process 455 now exits via a SIGINT signal or with the correct exit code such that the 456 calling process can detect that it died due to a Ctrl-C. Shells on POSIX 457 and Windows use this to properly terminate scripts in interactive sessions. 458 (Contributed by Google via Gregory P. Smith in :issue:`1054041`.) 459 460* Some advanced styles of programming require updating the 461 :class:`types.CodeType` object for an existing function. Since code 462 objects are immutable, a new code object needs to be created, one 463 that is modeled on the existing code object. With 19 parameters, 464 this was somewhat tedious. Now, the new ``replace()`` method makes 465 it possible to create a clone with a few altered parameters. 466 467 Here's an example that alters the :func:`statistics.mean` function to 468 prevent the *data* parameter from being used as a keyword argument:: 469 470 >>> from statistics import mean 471 >>> mean(data=[10, 20, 90]) 472 40 473 >>> mean.__code__ = mean.__code__.replace(co_posonlyargcount=1) 474 >>> mean(data=[10, 20, 90]) 475 Traceback (most recent call last): 476 ... 477 TypeError: mean() got some positional-only arguments passed as keyword arguments: 'data' 478 479 (Contributed by Victor Stinner in :issue:`37032`.) 480 481* For integers, the three-argument form of the :func:`pow` function now 482 permits the exponent to be negative in the case where the base is 483 relatively prime to the modulus. It then computes a modular inverse to 484 the base when the exponent is ``-1``, and a suitable power of that 485 inverse for other negative exponents. For example, to compute the 486 `modular multiplicative inverse 487 <https://en.wikipedia.org/wiki/Modular_multiplicative_inverse>`_ of 38 488 modulo 137, write:: 489 490 >>> pow(38, -1, 137) 491 119 492 >>> 119 * 38 % 137 493 1 494 495 Modular inverses arise in the solution of `linear Diophantine 496 equations <https://en.wikipedia.org/wiki/Diophantine_equation>`_. 497 For example, to find integer solutions for ``4258�� + 147�� = 369``, 498 first rewrite as ``4258�� ≡ 369 (mod 147)`` then solve: 499 500 >>> x = 369 * pow(4258, -1, 147) % 147 501 >>> y = (4258 * x - 369) // -147 502 >>> 4258 * x + 147 * y 503 369 504 505 (Contributed by Mark Dickinson in :issue:`36027`.) 506 507* Dict comprehensions have been synced-up with dict literals so that the 508 key is computed first and the value second:: 509 510 >>> # Dict comprehension 511 >>> cast = {input('role? '): input('actor? ') for i in range(2)} 512 role? King Arthur 513 actor? Chapman 514 role? Black Knight 515 actor? Cleese 516 517 >>> # Dict literal 518 >>> cast = {input('role? '): input('actor? ')} 519 role? Sir Robin 520 actor? Eric Idle 521 522 The guaranteed execution order is helpful with assignment expressions 523 because variables assigned in the key expression will be available in 524 the value expression:: 525 526 >>> names = ['Martin von Löwis', 'Łukasz Langa', 'Walter Dörwald'] 527 >>> {(n := normalize('NFC', name)).casefold() : n for name in names} 528 {'martin von löwis': 'Martin von Löwis', 529 'łukasz langa': 'Łukasz Langa', 530 'walter dörwald': 'Walter Dörwald'} 531 532 (Contributed by Jörn Heissler in :issue:`35224`.) 533 534* The :meth:`object.__reduce__` method can now return a tuple from two to 535 six elements long. Formerly, five was the limit. The new, optional sixth 536 element is a callable with a ``(obj, state)`` signature. This allows the 537 direct control over the state-updating behavior of a specific object. If 538 not *None*, this callable will have priority over the object's 539 :meth:`~__setstate__` method. 540 (Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.) 541 542New Modules 543=========== 544 545* The new :mod:`importlib.metadata` module provides (provisional) support for 546 reading metadata from third-party packages. For example, it can extract an 547 installed package's version number, list of entry points, and more:: 548 549 >>> # Note following example requires that the popular "requests" 550 >>> # package has been installed. 551 >>> 552 >>> from importlib.metadata import version, requires, files 553 >>> version('requests') 554 '2.22.0' 555 >>> list(requires('requests')) 556 ['chardet (<3.1.0,>=3.0.2)'] 557 >>> list(files('requests'))[:5] 558 [PackagePath('requests-2.22.0.dist-info/INSTALLER'), 559 PackagePath('requests-2.22.0.dist-info/LICENSE'), 560 PackagePath('requests-2.22.0.dist-info/METADATA'), 561 PackagePath('requests-2.22.0.dist-info/RECORD'), 562 PackagePath('requests-2.22.0.dist-info/WHEEL')] 563 564 (Contributed by Barry Warsaw and Jason R. Coombs in :issue:`34632`.) 565 566 567Improved Modules 568================ 569 570ast 571--- 572 573AST nodes now have ``end_lineno`` and ``end_col_offset`` attributes, 574which give the precise location of the end of the node. (This only 575applies to nodes that have ``lineno`` and ``col_offset`` attributes.) 576 577New function :func:`ast.get_source_segment` returns the source code 578for a specific AST node. 579 580(Contributed by Ivan Levkivskyi in :issue:`33416`.) 581 582The :func:`ast.parse` function has some new flags: 583 584* ``type_comments=True`` causes it to return the text of :pep:`484` and 585 :pep:`526` type comments associated with certain AST nodes; 586 587* ``mode='func_type'`` can be used to parse :pep:`484` "signature type 588 comments" (returned for function definition AST nodes); 589 590* ``feature_version=(3, N)`` allows specifying an earlier Python 3 591 version. For example, ``feature_version=(3, 4)`` will treat 592 :keyword:`async` and :keyword:`await` as non-reserved words. 593 594(Contributed by Guido van Rossum in :issue:`35766`.) 595 596 597asyncio 598------- 599 600:func:`asyncio.run` has graduated from the provisional to stable API. This 601function can be used to execute a :term:`coroutine` and return the result while 602automatically managing the event loop. For example:: 603 604 import asyncio 605 606 async def main(): 607 await asyncio.sleep(0) 608 return 42 609 610 asyncio.run(main()) 611 612This is *roughly* equivalent to:: 613 614 import asyncio 615 616 async def main(): 617 await asyncio.sleep(0) 618 return 42 619 620 loop = asyncio.new_event_loop() 621 asyncio.set_event_loop(loop) 622 try: 623 loop.run_until_complete(main()) 624 finally: 625 asyncio.set_event_loop(None) 626 loop.close() 627 628 629The actual implementation is significantly more complex. Thus, 630:func:`asyncio.run` should be the preferred way of running asyncio programs. 631 632(Contributed by Yury Selivanov in :issue:`32314`.) 633 634Running ``python -m asyncio`` launches a natively async REPL. This allows rapid 635experimentation with code that has a top-level :keyword:`await`. There is no 636longer a need to directly call ``asyncio.run()`` which would spawn a new event 637loop on every invocation: 638 639.. code-block:: none 640 641 $ python -m asyncio 642 asyncio REPL 3.8.0 643 Use "await" directly instead of "asyncio.run()". 644 Type "help", "copyright", "credits" or "license" for more information. 645 >>> import asyncio 646 >>> await asyncio.sleep(10, result='hello') 647 hello 648 649(Contributed by Yury Selivanov in :issue:`37028`.) 650 651The exception :class:`asyncio.CancelledError` now inherits from 652:class:`BaseException` rather than :class:`Exception`. 653(Contributed by Yury Selivanov in :issue:`32528`.) 654 655On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`. 656(Contributed by Victor Stinner in :issue:`34687`.) 657 658:class:`~asyncio.ProactorEventLoop` now also supports UDP. 659(Contributed by Adam Meily and Andrew Svetlov in :issue:`29883`.) 660 661:class:`~asyncio.ProactorEventLoop` can now be interrupted by 662:exc:`KeyboardInterrupt` ("CTRL+C"). 663(Contributed by Vladimir Matveev in :issue:`23057`.) 664 665Added :meth:`asyncio.Task.get_coro` for getting the wrapped coroutine 666within an :class:`asyncio.Task`. 667(Contributed by Alex Grönholm in :issue:`36999`.) 668 669Asyncio tasks can now be named, either by passing the ``name`` keyword 670argument to :func:`asyncio.create_task` or 671the :meth:`~asyncio.loop.create_task` event loop method, or by 672calling the :meth:`~asyncio.Task.set_name` method on the task object. The 673task name is visible in the ``repr()`` output of :class:`asyncio.Task` and 674can also be retrieved using the :meth:`~asyncio.Task.get_name` method. 675(Contributed by Alex Grönholm in :issue:`34270`.) 676 677Added support for 678`Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>`_ to 679:func:`asyncio.loop.create_connection`. To specify the behavior, two new 680parameters have been added: *happy_eyeballs_delay* and *interleave*. The Happy 681Eyeballs algorithm improves responsiveness in applications that support IPv4 682and IPv6 by attempting to simultaneously connect using both. 683(Contributed by twisteroid ambassador in :issue:`33530`.) 684 685 686builtins 687-------- 688 689The :func:`compile` built-in has been improved to accept the 690``ast.PyCF_ALLOW_TOP_LEVEL_AWAIT`` flag. With this new flag passed, 691:func:`compile` will allow top-level ``await``, ``async for`` and ``async with`` 692constructs that are usually considered invalid syntax. Asynchronous code object 693marked with the ``CO_COROUTINE`` flag may then be returned. 694(Contributed by Matthias Bussonnier in :issue:`34616`) 695 696 697collections 698----------- 699 700The :meth:`~collections.somenamedtuple._asdict` method for 701:func:`collections.namedtuple` now returns a :class:`dict` instead of a 702:class:`collections.OrderedDict`. This works because regular dicts have 703guaranteed ordering since Python 3.7. If the extra features of 704:class:`OrderedDict` are required, the suggested remediation is to cast the 705result to the desired type: ``OrderedDict(nt._asdict())``. 706(Contributed by Raymond Hettinger in :issue:`35864`.) 707 708 709cProfile 710-------- 711 712The :class:`cProfile.Profile <profile.Profile>` class can now be used as a context manager. 713Profile a block of code by running:: 714 715 import cProfile 716 717 with cProfile.Profile() as profiler: 718 # code to be profiled 719 ... 720 721(Contributed by Scott Sanderson in :issue:`29235`.) 722 723 724csv 725--- 726 727The :class:`csv.DictReader` now returns instances of :class:`dict` instead of 728a :class:`collections.OrderedDict`. The tool is now faster and uses less 729memory while still preserving the field order. 730(Contributed by Michael Seek in :issue:`34003`.) 731 732 733curses 734------- 735 736Added a new variable holding structured version information for the 737underlying ncurses library: :data:`~curses.ncurses_version`. 738(Contributed by Serhiy Storchaka in :issue:`31680`.) 739 740 741ctypes 742------ 743 744On Windows, :class:`~ctypes.CDLL` and subclasses now accept a *winmode* parameter 745to specify flags for the underlying ``LoadLibraryEx`` call. The default flags are 746set to only load DLL dependencies from trusted locations, including the path 747where the DLL is stored (if a full or partial path is used to load the initial 748DLL) and paths added by :func:`~os.add_dll_directory`. 749(Contributed by Steve Dower in :issue:`36085`.) 750 751 752datetime 753-------- 754 755Added new alternate constructors :meth:`datetime.date.fromisocalendar` and 756:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and 757:class:`datetime` objects respectively from ISO year, week number, and weekday; 758these are the inverse of each class's ``isocalendar`` method. 759(Contributed by Paul Ganssle in :issue:`36004`.) 760 761 762functools 763--------- 764 765:func:`functools.lru_cache` can now be used as a straight decorator rather 766than as a function returning a decorator. So both of these are now supported:: 767 768 @lru_cache 769 def f(x): 770 ... 771 772 @lru_cache(maxsize=256) 773 def f(x): 774 ... 775 776(Contributed by Raymond Hettinger in :issue:`36772`.) 777 778Added a new :func:`functools.cached_property` decorator, for computed properties 779cached for the life of the instance. :: 780 781 import functools 782 import statistics 783 784 class Dataset: 785 def __init__(self, sequence_of_numbers): 786 self.data = sequence_of_numbers 787 788 @functools.cached_property 789 def variance(self): 790 return statistics.variance(self.data) 791 792(Contributed by Carl Meyer in :issue:`21145`) 793 794 795Added a new :func:`functools.singledispatchmethod` decorator that converts 796methods into :term:`generic functions <generic function>` using 797:term:`single dispatch`:: 798 799 from functools import singledispatchmethod 800 from contextlib import suppress 801 802 class TaskManager: 803 804 def __init__(self, tasks): 805 self.tasks = list(tasks) 806 807 @singledispatchmethod 808 def discard(self, value): 809 with suppress(ValueError): 810 self.tasks.remove(value) 811 812 @discard.register(list) 813 def _(self, tasks): 814 targets = set(tasks) 815 self.tasks = [x for x in self.tasks if x not in targets] 816 817(Contributed by Ethan Smith in :issue:`32380`) 818 819gc 820-- 821 822:func:`~gc.get_objects` can now receive an optional *generation* parameter 823indicating a generation to get objects from. 824(Contributed by Pablo Galindo in :issue:`36016`.) 825 826 827gettext 828------- 829 830Added :func:`~gettext.pgettext` and its variants. 831(Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in :issue:`2504`.) 832 833 834gzip 835---- 836 837Added the *mtime* parameter to :func:`gzip.compress` for reproducible output. 838(Contributed by Guo Ci Teo in :issue:`34898`.) 839 840A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError` 841for certain types of invalid or corrupt gzip files. 842(Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in 843:issue:`6584`.) 844 845 846IDLE and idlelib 847---------------- 848 849Output over N lines (50 by default) is squeezed down to a button. 850N can be changed in the PyShell section of the General page of the 851Settings dialog. Fewer, but possibly extra long, lines can be squeezed by 852right clicking on the output. Squeezed output can be expanded in place 853by double-clicking the button or into the clipboard or a separate window 854by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) 855 856Add "Run Customized" to the Run menu to run a module with customized 857settings. Any command line arguments entered are added to sys.argv. 858They also re-appear in the box for the next customized run. One can also 859suppress the normal Shell main module restart. (Contributed by Cheryl 860Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.) 861 862Added optional line numbers for IDLE editor windows. Windows 863open without line numbers unless set otherwise in the General 864tab of the configuration dialog. Line numbers for an existing 865window are shown and hidden in the Options menu. 866(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.) 867 868OS native encoding is now used for converting between Python strings and Tcl 869objects. This allows IDLE to work with emoji and other non-BMP characters. 870These characters can be displayed or copied and pasted to or from the 871clipboard. Converting strings from Tcl to Python and back now never fails. 872(Many people worked on this for eight years but the problem was finally 873solved by Serhiy Storchaka in :issue:`13153`.) 874 875The changes above have been backported to 3.7 maintenance releases. 876 877 878inspect 879------- 880 881The :func:`inspect.getdoc` function can now find docstrings for ``__slots__`` 882if that attribute is a :class:`dict` where the values are docstrings. 883This provides documentation options similar to what we already have 884for :func:`property`, :func:`classmethod`, and :func:`staticmethod`:: 885 886 class AudioClip: 887 __slots__ = {'bit_rate': 'expressed in kilohertz to one decimal place', 888 'duration': 'in seconds, rounded up to an integer'} 889 def __init__(self, bit_rate, duration): 890 self.bit_rate = round(bit_rate / 1000.0, 1) 891 self.duration = ceil(duration) 892 893(Contributed by Raymond Hettinger in :issue:`36326`.) 894 895 896io 897-- 898 899In development mode (:option:`-X` ``env``) and in debug build, the 900:class:`io.IOBase` finalizer now logs the exception if the ``close()`` method 901fails. The exception is ignored silently by default in release build. 902(Contributed by Victor Stinner in :issue:`18748`.) 903 904 905itertools 906--------- 907 908The :func:`itertools.accumulate` function added an option *initial* keyword 909argument to specify an initial value:: 910 911 >>> from itertools import accumulate 912 >>> list(accumulate([10, 5, 30, 15], initial=1000)) 913 [1000, 1010, 1015, 1045, 1060] 914 915(Contributed by Lisa Roach in :issue:`34659`.) 916 917 918json.tool 919--------- 920 921Add option ``--json-lines`` to parse every input line as a separate JSON object. 922(Contributed by Weipeng Hong in :issue:`31553`.) 923 924 925logging 926------- 927 928Added a *force* keyword argument to :func:`logging.basicConfig()` 929When set to true, any existing handlers attached 930to the root logger are removed and closed before carrying out the 931configuration specified by the other arguments. 932 933This solves a long-standing problem. Once a logger or *basicConfig()* had 934been called, subsequent calls to *basicConfig()* were silently ignored. 935This made it difficult to update, experiment with, or teach the various 936logging configuration options using the interactive prompt or a Jupyter 937notebook. 938 939(Suggested by Raymond Hettinger, implemented by Dong-hee Na, and 940reviewed by Vinay Sajip in :issue:`33897`.) 941 942 943math 944---- 945 946Added new function :func:`math.dist` for computing Euclidean distance 947between two points. (Contributed by Raymond Hettinger in :issue:`33089`.) 948 949Expanded the :func:`math.hypot` function to handle multiple dimensions. 950Formerly, it only supported the 2-D case. 951(Contributed by Raymond Hettinger in :issue:`33089`.) 952 953Added new function, :func:`math.prod`, as analogous function to :func:`sum` 954that returns the product of a 'start' value (default: 1) times an iterable of 955numbers:: 956 957 >>> prior = 0.8 958 >>> likelihoods = [0.625, 0.84, 0.30] 959 >>> math.prod(likelihoods, start=prior) 960 0.126 961 962(Contributed by Pablo Galindo in :issue:`35606`.) 963 964Added two new combinatoric functions :func:`math.perm` and :func:`math.comb`:: 965 966 >>> math.perm(10, 3) # Permutations of 10 things taken 3 at a time 967 720 968 >>> math.comb(10, 3) # Combinations of 10 things taken 3 at a time 969 120 970 971(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond 972Hettinger in :issue:`37128`, :issue:`37178`, and :issue:`35431`.) 973 974Added a new function :func:`math.isqrt` for computing accurate integer square 975roots without conversion to floating point. The new function supports 976arbitrarily large integers. It is faster than ``floor(sqrt(n))`` but slower 977than :func:`math.sqrt`:: 978 979 >>> r = 650320427 980 >>> s = r ** 2 981 >>> isqrt(s - 1) # correct 982 650320426 983 >>> floor(sqrt(s - 1)) # incorrect 984 650320427 985 986(Contributed by Mark Dickinson in :issue:`36887`.) 987 988The function :func:`math.factorial` no longer accepts arguments that are not 989int-like. (Contributed by Pablo Galindo in :issue:`33083`.) 990 991 992mmap 993---- 994 995The :class:`mmap.mmap` class now has an :meth:`~mmap.mmap.madvise` method to 996access the ``madvise()`` system call. 997(Contributed by Zackery Spytz in :issue:`32941`.) 998 999 1000multiprocessing 1001--------------- 1002 1003Added new :mod:`multiprocessing.shared_memory` module. 1004(Contributed by Davin Potts in :issue:`35813`.) 1005 1006On macOS, the *spawn* start method is now used by default. 1007(Contributed by Victor Stinner in :issue:`33725`.) 1008 1009 1010os 1011-- 1012 1013Added new function :func:`~os.add_dll_directory` on Windows for providing 1014additional search paths for native dependencies when importing extension 1015modules or loading DLLs using :mod:`ctypes`. 1016(Contributed by Steve Dower in :issue:`36085`.) 1017 1018A new :func:`os.memfd_create` function was added to wrap the 1019``memfd_create()`` syscall. 1020(Contributed by Zackery Spytz and Christian Heimes in :issue:`26836`.) 1021 1022On Windows, much of the manual logic for handling reparse points (including 1023symlinks and directory junctions) has been delegated to the operating system. 1024Specifically, :func:`os.stat` will now traverse anything supported by the 1025operating system, while :func:`os.lstat` will only open reparse points that 1026identify as "name surrogates" while others are opened as for :func:`os.stat`. 1027In all cases, :attr:`stat_result.st_mode` will only have ``S_IFLNK`` set for 1028symbolic links and not other kinds of reparse points. To identify other kinds 1029of reparse point, check the new :attr:`stat_result.st_reparse_tag` attribute. 1030 1031On Windows, :func:`os.readlink` is now able to read directory junctions. Note 1032that :func:`~os.path.islink` will return ``False`` for directory junctions, 1033and so code that checks ``islink`` first will continue to treat junctions as 1034directories, while code that handles errors from :func:`os.readlink` may now 1035treat junctions as links. 1036 1037(Contributed by Steve Dower in :issue:`37834`.) 1038 1039 1040os.path 1041------- 1042 1043:mod:`os.path` functions that return a boolean result like 1044:func:`~os.path.exists`, :func:`~os.path.lexists`, :func:`~os.path.isdir`, 1045:func:`~os.path.isfile`, :func:`~os.path.islink`, and :func:`~os.path.ismount` 1046now return ``False`` instead of raising :exc:`ValueError` or its subclasses 1047:exc:`UnicodeEncodeError` and :exc:`UnicodeDecodeError` for paths that contain 1048characters or bytes unrepresentable at the OS level. 1049(Contributed by Serhiy Storchaka in :issue:`33721`.) 1050 1051:func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` 1052environment variable and does not use :envvar:`HOME`, which is not normally set 1053for regular user accounts. 1054(Contributed by Anthony Sottile in :issue:`36264`.) 1055 1056:func:`~os.path.isdir` on Windows no longer returns ``True`` for a link to a 1057non-existent directory. 1058 1059:func:`~os.path.realpath` on Windows now resolves reparse points, including 1060symlinks and directory junctions. 1061 1062(Contributed by Steve Dower in :issue:`37834`.) 1063 1064 1065pathlib 1066------- 1067 1068:mod:`pathlib.Path` methods that return a boolean result like 1069:meth:`~pathlib.Path.exists()`, :meth:`~pathlib.Path.is_dir()`, 1070:meth:`~pathlib.Path.is_file()`, :meth:`~pathlib.Path.is_mount()`, 1071:meth:`~pathlib.Path.is_symlink()`, :meth:`~pathlib.Path.is_block_device()`, 1072:meth:`~pathlib.Path.is_char_device()`, :meth:`~pathlib.Path.is_fifo()`, 1073:meth:`~pathlib.Path.is_socket()` now return ``False`` instead of raising 1074:exc:`ValueError` or its subclass :exc:`UnicodeEncodeError` for paths that 1075contain characters unrepresentable at the OS level. 1076(Contributed by Serhiy Storchaka in :issue:`33721`.) 1077 1078Added :meth:`pathlib.Path.link_to()` which creates a hard link pointing 1079to a path. 1080(Contributed by Joannah Nanjekye in :issue:`26978`) 1081 1082 1083pickle 1084------ 1085 1086:mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler` 1087can now override the pickling logic of functions and classes by defining the 1088special :meth:`~pickle.Pickler.reducer_override` method. 1089(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.) 1090 1091 1092plistlib 1093-------- 1094 1095Added new :class:`plistlib.UID` and enabled support for reading and writing 1096NSKeyedArchiver-encoded binary plists. 1097(Contributed by Jon Janzen in :issue:`26707`.) 1098 1099 1100pprint 1101------ 1102 1103The :mod:`pprint` module added a *sort_dicts* parameter to several functions. 1104By default, those functions continue to sort dictionaries before rendering or 1105printing. However, if *sort_dicts* is set to false, the dictionaries retain 1106the order that keys were inserted. This can be useful for comparison to JSON 1107inputs during debugging. 1108 1109In addition, there is a convenience new function, :func:`pprint.pp` that is 1110like :func:`pprint.pprint` but with *sort_dicts* defaulting to ``False``:: 1111 1112 >>> from pprint import pprint, pp 1113 >>> d = dict(source='input.txt', operation='filter', destination='output.txt') 1114 >>> pp(d, width=40) # Original order 1115 {'source': 'input.txt', 1116 'operation': 'filter', 1117 'destination': 'output.txt'} 1118 >>> pprint(d, width=40) # Keys sorted alphabetically 1119 {'destination': 'output.txt', 1120 'operation': 'filter', 1121 'source': 'input.txt'} 1122 1123(Contributed by Rémi Lapeyre in :issue:`30670`.) 1124 1125 1126py_compile 1127---------- 1128 1129:func:`py_compile.compile` now supports silent mode. 1130(Contributed by Joannah Nanjekye in :issue:`22640`.) 1131 1132 1133shlex 1134----- 1135 1136The new :func:`shlex.join` function acts as the inverse of :func:`shlex.split`. 1137(Contributed by Bo Bayles in :issue:`32102`.) 1138 1139 1140shutil 1141------ 1142 1143:func:`shutil.copytree` now accepts a new ``dirs_exist_ok`` keyword argument. 1144(Contributed by Josh Bronson in :issue:`20849`.) 1145 1146:func:`shutil.make_archive` now defaults to the modern pax (POSIX.1-2001) 1147format for new archives to improve portability and standards conformance, 1148inherited from the corresponding change to the :mod:`tarfile` module. 1149(Contributed by C.A.M. Gerlach in :issue:`30661`.) 1150 1151:func:`shutil.rmtree` on Windows now removes directory junctions without 1152recursively removing their contents first. 1153(Contributed by Steve Dower in :issue:`37834`.) 1154 1155 1156socket 1157------ 1158 1159Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6()` 1160convenience functions to automate the necessary tasks usually involved when 1161creating a server socket, including accepting both IPv4 and IPv6 connections 1162on the same socket. (Contributed by Giampaolo Rodolà in :issue:`17561`.) 1163 1164The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and 1165:func:`socket.if_indextoname()` functions have been implemented on Windows. 1166(Contributed by Zackery Spytz in :issue:`37007`.) 1167 1168 1169ssl 1170--- 1171 1172Added :attr:`~ssl.SSLContext.post_handshake_auth` to enable and 1173:meth:`~ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3 1174post-handshake authentication. 1175(Contributed by Christian Heimes in :issue:`34670`.) 1176 1177 1178statistics 1179---------- 1180 1181Added :func:`statistics.fmean` as a faster, floating point variant of 1182:func:`statistics.mean()`. (Contributed by Raymond Hettinger and 1183Steven D'Aprano in :issue:`35904`.) 1184 1185Added :func:`statistics.geometric_mean()` 1186(Contributed by Raymond Hettinger in :issue:`27181`.) 1187 1188Added :func:`statistics.multimode` that returns a list of the most 1189common values. (Contributed by Raymond Hettinger in :issue:`35892`.) 1190 1191Added :func:`statistics.quantiles` that divides data or a distribution 1192in to equiprobable intervals (e.g. quartiles, deciles, or percentiles). 1193(Contributed by Raymond Hettinger in :issue:`36546`.) 1194 1195Added :class:`statistics.NormalDist`, a tool for creating 1196and manipulating normal distributions of a random variable. 1197(Contributed by Raymond Hettinger in :issue:`36018`.) 1198 1199:: 1200 1201 >>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14]) 1202 >>> temperature_feb.mean 1203 6.0 1204 >>> temperature_feb.stdev 1205 6.356099432828281 1206 1207 >>> temperature_feb.cdf(3) # Chance of being under 3 degrees 1208 0.3184678262814532 1209 >>> # Relative chance of being 7 degrees versus 10 degrees 1210 >>> temperature_feb.pdf(7) / temperature_feb.pdf(10) 1211 1.2039930378537762 1212 1213 >>> el_niño = NormalDist(4, 2.5) 1214 >>> temperature_feb += el_niño # Add in a climate effect 1215 >>> temperature_feb 1216 NormalDist(mu=10.0, sigma=6.830080526611674) 1217 1218 >>> temperature_feb * (9/5) + 32 # Convert to Fahrenheit 1219 NormalDist(mu=50.0, sigma=12.294144947901014) 1220 >>> temperature_feb.samples(3) # Generate random samples 1221 [7.672102882379219, 12.000027119750287, 4.647488369766392] 1222 1223 1224sys 1225--- 1226 1227Add new :func:`sys.unraisablehook` function which can be overridden to control 1228how "unraisable exceptions" are handled. It is called when an exception has 1229occurred but there is no way for Python to handle it. For example, when a 1230destructor raises an exception or during garbage collection 1231(:func:`gc.collect`). 1232(Contributed by Victor Stinner in :issue:`36829`.) 1233 1234 1235tarfile 1236------- 1237 1238The :mod:`tarfile` module now defaults to the modern pax (POSIX.1-2001) 1239format for new archives, instead of the previous GNU-specific one. 1240This improves cross-platform portability with a consistent encoding (UTF-8) 1241in a standardized and extensible format, and offers several other benefits. 1242(Contributed by C.A.M. Gerlach in :issue:`36268`.) 1243 1244 1245threading 1246--------- 1247 1248Add a new :func:`threading.excepthook` function which handles uncaught 1249:meth:`threading.Thread.run` exception. It can be overridden to control how 1250uncaught :meth:`threading.Thread.run` exceptions are handled. 1251(Contributed by Victor Stinner in :issue:`1230540`.) 1252 1253Add a new :func:`threading.get_native_id` function and 1254a :data:`~threading.Thread.native_id` 1255attribute to the :class:`threading.Thread` class. These return the native 1256integral Thread ID of the current thread assigned by the kernel. 1257This feature is only available on certain platforms, see 1258:func:`get_native_id <threading.get_native_id>` for more information. 1259(Contributed by Jake Tesler in :issue:`36084`.) 1260 1261 1262tokenize 1263-------- 1264 1265The :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token when 1266provided with input that does not have a trailing new line. This behavior 1267now matches what the C tokenizer does internally. 1268(Contributed by Ammar Askar in :issue:`33899`.) 1269 1270 1271tkinter 1272------- 1273 1274Added methods :meth:`~tkinter.Spinbox.selection_from`, 1275:meth:`~tkinter.Spinbox.selection_present`, 1276:meth:`~tkinter.Spinbox.selection_range` and 1277:meth:`~tkinter.Spinbox.selection_to` 1278in the :class:`tkinter.Spinbox` class. 1279(Contributed by Juliette Monsel in :issue:`34829`.) 1280 1281Added method :meth:`~tkinter.Canvas.moveto` 1282in the :class:`tkinter.Canvas` class. 1283(Contributed by Juliette Monsel in :issue:`23831`.) 1284 1285The :class:`tkinter.PhotoImage` class now has 1286:meth:`~tkinter.PhotoImage.transparency_get` and 1287:meth:`~tkinter.PhotoImage.transparency_set` methods. (Contributed by 1288Zackery Spytz in :issue:`25451`.) 1289 1290 1291time 1292---- 1293 1294Added new clock :data:`~time.CLOCK_UPTIME_RAW` for macOS 10.12. 1295(Contributed by Joannah Nanjekye in :issue:`35702`.) 1296 1297 1298typing 1299------ 1300 1301The :mod:`typing` module incorporates several new features: 1302 1303* A dictionary type with per-key types. See :pep:`589` and 1304 :class:`typing.TypedDict`. 1305 TypedDict uses only string keys. By default, every key is required 1306 to be present. Specify "total=False" to allow keys to be optional:: 1307 1308 class Location(TypedDict, total=False): 1309 lat_long: tuple 1310 grid_square: str 1311 xy_coordinate: tuple 1312 1313* Literal types. See :pep:`586` and :class:`typing.Literal`. 1314 Literal types indicate that a parameter or return value 1315 is constrained to one or more specific literal values:: 1316 1317 def get_status(port: int) -> Literal['connected', 'disconnected']: 1318 ... 1319 1320* "Final" variables, functions, methods and classes. See :pep:`591`, 1321 :class:`typing.Final` and :func:`typing.final`. 1322 The final qualifier instructs a static type checker to restrict 1323 subclassing, overriding, or reassignment:: 1324 1325 pi: Final[float] = 3.1415926536 1326 1327* Protocol definitions. See :pep:`544`, :class:`typing.Protocol` and 1328 :func:`typing.runtime_checkable`. Simple ABCs like 1329 :class:`typing.SupportsInt` are now ``Protocol`` subclasses. 1330 1331* New protocol class :class:`typing.SupportsIndex`. 1332 1333* New functions :func:`typing.get_origin` and :func:`typing.get_args`. 1334 1335 1336unicodedata 1337----------- 1338 1339The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0 1340<http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release. 1341 1342New function :func:`~unicodedata.is_normalized` can be used to verify a string 1343is in a specific normal form, often much faster than by actually normalizing 1344the string. (Contributed by Max Belanger, David Euresti, and Greg Price in 1345:issue:`32285` and :issue:`37966`). 1346 1347 1348unittest 1349-------- 1350 1351Added :class:`~unittest.mock.AsyncMock` to support an asynchronous version of 1352:class:`~unittest.mock.Mock`. Appropriate new assert functions for testing 1353have been added as well. 1354(Contributed by Lisa Roach in :issue:`26467`). 1355 1356Added :func:`~unittest.addModuleCleanup()` and 1357:meth:`~unittest.TestCase.addClassCleanup()` to unittest to support 1358cleanups for :func:`~unittest.setUpModule()` and 1359:meth:`~unittest.TestCase.setUpClass()`. 1360(Contributed by Lisa Roach in :issue:`24412`.) 1361 1362Several mock assert functions now also print a list of actual calls upon 1363failure. (Contributed by Petter Strandmark in :issue:`35047`.) 1364 1365:mod:`unittest` module gained support for coroutines to be used as test cases 1366with :class:`unittest.IsolatedAsyncioTestCase`. 1367(Contributed by Andrew Svetlov in :issue:`32972`.) 1368 1369Example:: 1370 1371 import unittest 1372 1373 1374 class TestRequest(unittest.IsolatedAsyncioTestCase): 1375 1376 async def asyncSetUp(self): 1377 self.connection = await AsyncConnection() 1378 1379 async def test_get(self): 1380 response = await self.connection.get("https://example.com") 1381 self.assertEqual(response.status_code, 200) 1382 1383 async def asyncTearDown(self): 1384 await self.connection.close() 1385 1386 1387 if __name__ == "__main__": 1388 unittest.main() 1389 1390 1391venv 1392---- 1393 1394:mod:`venv` now includes an ``Activate.ps1`` script on all platforms for 1395activating virtual environments under PowerShell Core 6.1. 1396(Contributed by Brett Cannon in :issue:`32718`.) 1397 1398 1399weakref 1400------- 1401 1402The proxy objects returned by :func:`weakref.proxy` now support the matrix 1403multiplication operators ``@`` and ``@=`` in addition to the other 1404numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.) 1405 1406 1407xml 1408--- 1409 1410As mitigation against DTD and external entity retrieval, the 1411:mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 1412external entities by default. 1413(Contributed by Christian Heimes in :issue:`17239`.) 1414 1415The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module 1416support wildcard searches like ``{*}tag`` which ignores the namespace 1417and ``{namespace}*`` which returns all tags in the given namespace. 1418(Contributed by Stefan Behnel in :issue:`28238`.) 1419 1420The :mod:`xml.etree.ElementTree` module provides a new function 1421:func:`–xml.etree.ElementTree.canonicalize()` that implements C14N 2.0. 1422(Contributed by Stefan Behnel in :issue:`13611`.) 1423 1424The target object of :class:`xml.etree.ElementTree.XMLParser` can 1425receive namespace declaration events through the new callback methods 1426``start_ns()`` and ``end_ns()``. Additionally, the 1427:class:`xml.etree.ElementTree.TreeBuilder` target can be configured 1428to process events about comments and processing instructions to include 1429them in the generated tree. 1430(Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.) 1431 1432 1433xmlrpc 1434------ 1435 1436:class:`xmlrpc.client.ServerProxy` now supports an optional *headers* keyword 1437argument for a sequence of HTTP headers to be sent with each request. Among 1438other things, this makes it possible to upgrade from default basic 1439authentication to faster session authentication. 1440(Contributed by Cédric Krier in :issue:`35153`.) 1441 1442 1443Optimizations 1444============= 1445 1446* The :mod:`subprocess` module can now use the :func:`os.posix_spawn` function 1447 in some cases for better performance. Currently, it is only used on macOS 1448 and Linux (using glibc 2.24 or newer) if all these conditions are met: 1449 1450 * *close_fds* is false; 1451 * *preexec_fn*, *pass_fds*, *cwd* and *start_new_session* parameters 1452 are not set; 1453 * the *executable* path contains a directory. 1454 1455 (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.) 1456 1457* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`, 1458 :func:`shutil.copytree` and :func:`shutil.move` use platform-specific 1459 "fast-copy" syscalls on Linux and macOS in order to copy the file 1460 more efficiently. 1461 "fast-copy" means that the copying operation occurs within the kernel, 1462 avoiding the use of userspace buffers in Python as in 1463 "``outfd.write(infd.read())``". 1464 On Windows :func:`shutil.copyfile` uses a bigger default buffer size (1 MiB 1465 instead of 16 KiB) and a :func:`memoryview`-based variant of 1466 :func:`shutil.copyfileobj` is used. 1467 The speedup for copying a 512 MiB file within the same partition is about 1468 +26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycles 1469 are consumed. 1470 See :ref:`shutil-platform-dependent-efficient-copy-operations` section. 1471 (Contributed by Giampaolo Rodolà in :issue:`33671`.) 1472 1473* :func:`shutil.copytree` uses :func:`os.scandir` function and all copy 1474 functions depending from it use cached :func:`os.stat` values. The speedup 1475 for copying a directory with 8000 files is around +9% on Linux, +20% on 1476 Windows and +30% on a Windows SMB share. Also the number of :func:`os.stat` 1477 syscalls is reduced by 38% making :func:`shutil.copytree` especially faster 1478 on network filesystems. (Contributed by Giampaolo Rodolà in :issue:`33695`.) 1479 1480* The default protocol in the :mod:`pickle` module is now Protocol 4, 1481 first introduced in Python 3.4. It offers better performance and smaller 1482 size compared to Protocol 3 available since Python 3.0. 1483 1484* Removed one ``Py_ssize_t`` member from ``PyGC_Head``. All GC tracked 1485 objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes. 1486 (Contributed by Inada Naoki in :issue:`33597`.) 1487 1488* :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint. 1489 (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`) 1490 1491* Improved performance of :func:`operator.itemgetter` by 33%. Optimized 1492 argument handling and added a fast path for the common case of a single 1493 non-negative integer index into a tuple (which is the typical use case in 1494 the standard library). (Contributed by Raymond Hettinger in 1495 :issue:`35664`.) 1496 1497* Sped-up field lookups in :func:`collections.namedtuple`. They are now more 1498 than two times faster, making them the fastest form of instance variable 1499 lookup in Python. (Contributed by Raymond Hettinger, Pablo Galindo, and 1500 Joe Jevnik, Serhiy Storchaka in :issue:`32492`.) 1501 1502* The :class:`list` constructor does not overallocate the internal item buffer 1503 if the input iterable has a known length (the input implements ``__len__``). 1504 This makes the created list 12% smaller on average. (Contributed by 1505 Raymond Hettinger and Pablo Galindo in :issue:`33234`.) 1506 1507* Doubled the speed of class variable writes. When a non-dunder attribute 1508 was updated, there was an unnecessary call to update slots. 1509 (Contributed by Stefan Behnel, Pablo Galindo Salgado, Raymond Hettinger, 1510 Neil Schemenauer, and Serhiy Storchaka in :issue:`36012`.) 1511 1512* Reduced an overhead of converting arguments passed to many builtin functions 1513 and methods. This sped up calling some simple builtin functions and 1514 methods up to 20--50%. (Contributed by Serhiy Storchaka in :issue:`23867`, 1515 :issue:`35582` and :issue:`36127`.) 1516 1517* ``LOAD_GLOBAL`` instruction now uses new "per opcode cache" mechanism. 1518 It is about 40% faster now. (Contributed by Yury Selivanov and Inada Naoki in 1519 :issue:`26219`.) 1520 1521 1522Build and C API Changes 1523======================= 1524 1525* Default :data:`sys.abiflags` became an empty string: the ``m`` flag for 1526 pymalloc became useless (builds with and without pymalloc are ABI compatible) 1527 and so has been removed. (Contributed by Victor Stinner in :issue:`36707`.) 1528 1529 Example of changes: 1530 1531 * Only ``python3.8`` program is installed, ``python3.8m`` program is gone. 1532 * Only ``python3.8-config`` script is installed, ``python3.8m-config`` script 1533 is gone. 1534 * The ``m`` flag has been removed from the suffix of dynamic library 1535 filenames: extension modules in the standard library as well as those 1536 produced and installed by third-party packages, like those downloaded from 1537 PyPI. On Linux, for example, the Python 3.7 suffix 1538 ``.cpython-37m-x86_64-linux-gnu.so`` became 1539 ``.cpython-38-x86_64-linux-gnu.so`` in Python 3.8. 1540 1541* The header files have been reorganized to better separate the different kinds 1542 of APIs: 1543 1544 * ``Include/*.h`` should be the portable public stable C API. 1545 * ``Include/cpython/*.h`` should be the unstable C API specific to CPython; 1546 public API, with some private API prefixed by ``_Py`` or ``_PY``. 1547 * ``Include/internal/*.h`` is the private internal C API very specific to 1548 CPython. This API comes with no backward compatibility warranty and should 1549 not be used outside CPython. It is only exposed for very specific needs 1550 like debuggers and profiles which has to access to CPython internals 1551 without calling functions. This API is now installed by ``make install``. 1552 1553 (Contributed by Victor Stinner in :issue:`35134` and :issue:`35081`, 1554 work initiated by Eric Snow in Python 3.7.) 1555 1556* Some macros have been converted to static inline functions: parameter types 1557 and return type are well defined, they don't have issues specific to macros, 1558 variables have a local scopes. Examples: 1559 1560 * :c:func:`Py_INCREF`, :c:func:`Py_DECREF` 1561 * :c:func:`Py_XINCREF`, :c:func:`Py_XDECREF` 1562 * :c:func:`PyObject_INIT`, :c:func:`PyObject_INIT_VAR` 1563 * Private functions: :c:func:`_PyObject_GC_TRACK`, 1564 :c:func:`_PyObject_GC_UNTRACK`, :c:func:`_Py_Dealloc` 1565 1566 (Contributed by Victor Stinner in :issue:`35059`.) 1567 1568* The :c:func:`PyByteArray_Init` and :c:func:`PyByteArray_Fini` functions have 1569 been removed. They did nothing since Python 2.7.4 and Python 3.2.0, were 1570 excluded from the limited API (stable ABI), and were not documented. 1571 (Contributed by Victor Stinner in :issue:`35713`.) 1572 1573* The result of :c:func:`PyExceptionClass_Name` is now of type 1574 ``const char *`` rather of ``char *``. 1575 (Contributed by Serhiy Storchaka in :issue:`33818`.) 1576 1577* The duality of ``Modules/Setup.dist`` and ``Modules/Setup`` has been 1578 removed. Previously, when updating the CPython source tree, one had 1579 to manually copy ``Modules/Setup.dist`` (inside the source tree) to 1580 ``Modules/Setup`` (inside the build tree) in order to reflect any changes 1581 upstream. This was of a small benefit to packagers at the expense of 1582 a frequent annoyance to developers following CPython development, as 1583 forgetting to copy the file could produce build failures. 1584 1585 Now the build system always reads from ``Modules/Setup`` inside the source 1586 tree. People who want to customize that file are encouraged to maintain 1587 their changes in a git fork of CPython or as patch files, as they would do 1588 for any other change to the source tree. 1589 1590 (Contributed by Antoine Pitrou in :issue:`32430`.) 1591 1592* Functions that convert Python number to C integer like 1593 :c:func:`PyLong_AsLong` and argument parsing functions like 1594 :c:func:`PyArg_ParseTuple` with integer converting format units like ``'i'`` 1595 will now use the :meth:`~object.__index__` special method instead of 1596 :meth:`~object.__int__`, if available. The deprecation warning will be 1597 emitted for objects with the ``__int__()`` method but without the 1598 ``__index__()`` method (like :class:`~decimal.Decimal` and 1599 :class:`~fractions.Fraction`). :c:func:`PyNumber_Check` will now return 1600 ``1`` for objects implementing ``__index__()``. 1601 :c:func:`PyNumber_Long`, :c:func:`PyNumber_Float` and 1602 :c:func:`PyFloat_AsDouble` also now use the ``__index__()`` method if 1603 available. 1604 (Contributed by Serhiy Storchaka in :issue:`36048` and :issue:`20092`.) 1605 1606* Heap-allocated type objects will now increase their reference count 1607 in :c:func:`PyObject_Init` (and its parallel macro ``PyObject_INIT``) 1608 instead of in :c:func:`PyType_GenericAlloc`. Types that modify instance 1609 allocation or deallocation may need to be adjusted. 1610 (Contributed by Eddie Elizondo in :issue:`35810`.) 1611 1612* The new function :c:func:`PyCode_NewWithPosOnlyArgs` allows to create 1613 code objects like :c:func:`PyCode_New`, but with an extra *posonlyargcount* 1614 parameter for indicating the number of positional-only arguments. 1615 (Contributed by Pablo Galindo in :issue:`37221`.) 1616 1617* :c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full 1618 path (:c:func:`Py_GetProgramFullPath`) rather than to the program name 1619 (:c:func:`Py_GetProgramName`). 1620 (Contributed by Victor Stinner in :issue:`38234`.) 1621 1622 1623Deprecated 1624========== 1625 1626* The distutils ``bdist_wininst`` command is now deprecated, use 1627 ``bdist_wheel`` (wheel packages) instead. 1628 (Contributed by Victor Stinner in :issue:`37481`.) 1629 1630* Deprecated methods ``getchildren()`` and ``getiterator()`` in 1631 the :mod:`~xml.etree.ElementTree` module now emit a 1632 :exc:`DeprecationWarning` instead of :exc:`PendingDeprecationWarning`. 1633 They will be removed in Python 3.9. 1634 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1635 1636* Passing an object that is not an instance of 1637 :class:`concurrent.futures.ThreadPoolExecutor` to 1638 :meth:`loop.set_default_executor() <asyncio.loop.set_default_executor>` is 1639 deprecated and will be prohibited in Python 3.9. 1640 (Contributed by Elvis Pranskevichus in :issue:`34075`.) 1641 1642* The :meth:`__getitem__` methods of :class:`xml.dom.pulldom.DOMEventStream`, 1643 :class:`wsgiref.util.FileWrapper` and :class:`fileinput.FileInput` have been 1644 deprecated. 1645 1646 Implementations of these methods have been ignoring their *index* parameter, 1647 and returning the next item instead. 1648 (Contributed by Berker Peksag in :issue:`9372`.) 1649 1650* The :class:`typing.NamedTuple` class has deprecated the ``_field_types`` 1651 attribute in favor of the ``__annotations__`` attribute which has the same 1652 information. (Contributed by Raymond Hettinger in :issue:`36320`.) 1653 1654* :mod:`ast` classes ``Num``, ``Str``, ``Bytes``, ``NameConstant`` and 1655 ``Ellipsis`` are considered deprecated and will be removed in future Python 1656 versions. :class:`~ast.Constant` should be used instead. 1657 (Contributed by Serhiy Storchaka in :issue:`32892`.) 1658 1659* :class:`ast.NodeVisitor` methods ``visit_Num()``, ``visit_Str()``, 1660 ``visit_Bytes()``, ``visit_NameConstant()`` and ``visit_Ellipsis()`` are 1661 deprecated now and will not be called in future Python versions. 1662 Add the :meth:`~ast.NodeVisitor.visit_Constant` method to handle all 1663 constant nodes. 1664 (Contributed by Serhiy Storchaka in :issue:`36917`.) 1665 1666* The :func:`asyncio.coroutine` :term:`decorator` is deprecated and will be 1667 removed in version 3.10. Instead of ``@asyncio.coroutine``, use 1668 :keyword:`async def` instead. 1669 (Contributed by Andrew Svetlov in :issue:`36921`.) 1670 1671* In :mod:`asyncio`, the explicit passing of a *loop* argument has been 1672 deprecated and will be removed in version 3.10 for the following: 1673 :func:`asyncio.sleep`, :func:`asyncio.gather`, :func:`asyncio.shield`, 1674 :func:`asyncio.wait_for`, :func:`asyncio.wait`, :func:`asyncio.as_completed`, 1675 :class:`asyncio.Task`, :class:`asyncio.Lock`, :class:`asyncio.Event`, 1676 :class:`asyncio.Condition`, :class:`asyncio.Semaphore`, 1677 :class:`asyncio.BoundedSemaphore`, :class:`asyncio.Queue`, 1678 :func:`asyncio.create_subprocess_exec`, and 1679 :func:`asyncio.create_subprocess_shell`. 1680 1681* The explicit passing of coroutine objects to :func:`asyncio.wait` has been 1682 deprecated and will be removed in version 3.11. 1683 (Contributed by Yury Selivanov in :issue:`34790`.) 1684 1685* The following functions and methods are deprecated in the :mod:`gettext` 1686 module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`, 1687 :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`. 1688 They return encoded bytes, and it's possible that you will get unexpected 1689 Unicode-related exceptions if there are encoding problems with the 1690 translated strings. It's much better to use alternatives which return 1691 Unicode strings in Python 3. These functions have been broken for a long time. 1692 1693 Function :func:`~gettext.bind_textdomain_codeset`, methods 1694 :meth:`~gettext.NullTranslations.output_charset` and 1695 :meth:`~gettext.NullTranslations.set_output_charset`, and the *codeset* 1696 parameter of functions :func:`~gettext.translation` and 1697 :func:`~gettext.install` are also deprecated, since they are only used for 1698 for the ``l*gettext()`` functions. 1699 (Contributed by Serhiy Storchaka in :issue:`33710`.) 1700 1701* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` 1702 has been deprecated. 1703 (Contributed by Dong-hee Na in :issue:`35283`.) 1704 1705* Many builtin and extension functions that take integer arguments will 1706 now emit a deprecation warning for :class:`~decimal.Decimal`\ s, 1707 :class:`~fractions.Fraction`\ s and any other objects that can be converted 1708 to integers only with a loss (e.g. that have the :meth:`~object.__int__` 1709 method but do not have the :meth:`~object.__index__` method). In future 1710 version they will be errors. 1711 (Contributed by Serhiy Storchaka in :issue:`36048`.) 1712 1713* Deprecated passing the following arguments as keyword arguments: 1714 1715 - *func* in :func:`functools.partialmethod`, :func:`weakref.finalize`, 1716 :meth:`profile.Profile.runcall`, :meth:`cProfile.Profile.runcall`, 1717 :meth:`bdb.Bdb.runcall`, :meth:`trace.Trace.runfunc` and 1718 :func:`curses.wrapper`. 1719 - *function* in :meth:`unittest.TestCase.addCleanup`. 1720 - *fn* in the :meth:`~concurrent.futures.Executor.submit` method of 1721 :class:`concurrent.futures.ThreadPoolExecutor` and 1722 :class:`concurrent.futures.ProcessPoolExecutor`. 1723 - *callback* in :meth:`contextlib.ExitStack.callback`, 1724 :meth:`contextlib.AsyncExitStack.callback` and 1725 :meth:`contextlib.AsyncExitStack.push_async_callback`. 1726 - *c* and *typeid* in the :meth:`~multiprocessing.managers.Server.create` 1727 method of :class:`multiprocessing.managers.Server` and 1728 :class:`multiprocessing.managers.SharedMemoryServer`. 1729 - *obj* in :func:`weakref.finalize`. 1730 1731 In future releases of Python, they will be :ref:`positional-only 1732 <positional-only_parameter>`. 1733 (Contributed by Serhiy Storchaka in :issue:`36492`.) 1734 1735 1736API and Feature Removals 1737======================== 1738 1739The following features and APIs have been removed from Python 3.8: 1740 1741* Starting with Python 3.3, importing ABCs from :mod:`collections` was 1742 deprecated, and importing should be done from :mod:`collections.abc`. Being 1743 able to import from collections was marked for removal in 3.8, but has been 1744 delayed to 3.9. (See :issue:`36952`.) 1745 1746* The :mod:`macpath` module, deprecated in Python 3.7, has been removed. 1747 (Contributed by Victor Stinner in :issue:`35471`.) 1748 1749* The function :func:`platform.popen` has been removed, after having been 1750 deprecated since Python 3.3: use :func:`os.popen` instead. 1751 (Contributed by Victor Stinner in :issue:`35345`.) 1752 1753* The function :func:`time.clock` has been removed, after having been 1754 deprecated since Python 3.3: use :func:`time.perf_counter` or 1755 :func:`time.process_time` instead, depending 1756 on your requirements, to have well-defined behavior. 1757 (Contributed by Matthias Bussonnier in :issue:`36895`.) 1758 1759* The ``pyvenv`` script has been removed in favor of ``python3.8 -m venv`` 1760 to help eliminate confusion as to what Python interpreter the ``pyvenv`` 1761 script is tied to. (Contributed by Brett Cannon in :issue:`25427`.) 1762 1763* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from the :mod:`cgi` 1764 module. They are deprecated in Python 3.2 or older. They should be imported 1765 from the ``urllib.parse`` and ``html`` modules instead. 1766 1767* ``filemode`` function is removed from the :mod:`tarfile` module. 1768 It is not documented and deprecated since Python 3.3. 1769 1770* The :class:`~xml.etree.ElementTree.XMLParser` constructor no longer accepts 1771 the *html* argument. It never had an effect and was deprecated in Python 3.4. 1772 All other parameters are now :ref:`keyword-only <keyword-only_parameter>`. 1773 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1774 1775* Removed the ``doctype()`` method of :class:`~xml.etree.ElementTree.XMLParser`. 1776 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1777 1778* "unicode_internal" codec is removed. 1779 (Contributed by Inada Naoki in :issue:`36297`.) 1780 1781* The ``Cache`` and ``Statement`` objects of the :mod:`sqlite3` module are not 1782 exposed to the user. 1783 (Contributed by Aviv Palivoda in :issue:`30262`.) 1784 1785* The ``bufsize`` keyword argument of :func:`fileinput.input` and 1786 :func:`fileinput.FileInput` which was ignored and deprecated since Python 3.6 1787 has been removed. :issue:`36952` (Contributed by Matthias Bussonnier.) 1788 1789* The functions :func:`sys.set_coroutine_wrapper` and 1790 :func:`sys.get_coroutine_wrapper` deprecated in Python 3.7 have been removed; 1791 :issue:`36933` (Contributed by Matthias Bussonnier.) 1792 1793 1794Porting to Python 3.8 1795===================== 1796 1797This section lists previously described changes and other bugfixes 1798that may require changes to your code. 1799 1800 1801Changes in Python behavior 1802-------------------------- 1803 1804* Yield expressions (both ``yield`` and ``yield from`` clauses) are now disallowed 1805 in comprehensions and generator expressions (aside from the iterable expression 1806 in the leftmost :keyword:`!for` clause). 1807 (Contributed by Serhiy Storchaka in :issue:`10544`.) 1808 1809* The compiler now produces a :exc:`SyntaxWarning` when identity checks 1810 (``is`` and ``is not``) are used with certain types of literals 1811 (e.g. strings, numbers). These can often work by accident in CPython, 1812 but are not guaranteed by the language spec. The warning advises users 1813 to use equality tests (``==`` and ``!=``) instead. 1814 (Contributed by Serhiy Storchaka in :issue:`34850`.) 1815 1816* The CPython interpreter can swallow exceptions in some circumstances. 1817 In Python 3.8 this happens in fewer cases. In particular, exceptions 1818 raised when getting the attribute from the type dictionary are no longer 1819 ignored. (Contributed by Serhiy Storchaka in :issue:`35459`.) 1820 1821* Removed ``__str__`` implementations from builtin types :class:`bool`, 1822 :class:`int`, :class:`float`, :class:`complex` and few classes from 1823 the standard library. They now inherit ``__str__()`` from :class:`object`. 1824 As result, defining the ``__repr__()`` method in the subclass of these 1825 classes will affect their string representation. 1826 (Contributed by Serhiy Storchaka in :issue:`36793`.) 1827 1828* On AIX, :attr:`sys.platform` doesn't contain the major version anymore. 1829 It is always ``'aix'``, instead of ``'aix3'`` .. ``'aix7'``. Since 1830 older Python versions include the version number, so it is recommended to 1831 always use ``sys.platform.startswith('aix')``. 1832 (Contributed by M. Felt in :issue:`36588`.) 1833 1834* :c:func:`PyEval_AcquireLock` and :c:func:`PyEval_AcquireThread` now 1835 terminate the current thread if called while the interpreter is 1836 finalizing, making them consistent with :c:func:`PyEval_RestoreThread`, 1837 :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this 1838 behavior is not desired, guard the call by checking :c:func:`_Py_IsFinalizing` 1839 or :c:func:`sys.is_finalizing`. 1840 (Contributed by Joannah Nanjekye in :issue:`36475`.) 1841 1842 1843Changes in the Python API 1844------------------------- 1845 1846* The :func:`os.getcwdb` function now uses the UTF-8 encoding on Windows, 1847 rather than the ANSI code page: see :pep:`529` for the rationale. The 1848 function is no longer deprecated on Windows. 1849 (Contributed by Victor Stinner in :issue:`37412`.) 1850 1851* :class:`subprocess.Popen` can now use :func:`os.posix_spawn` in some cases 1852 for better performance. On Windows Subsystem for Linux and QEMU User 1853 Emulation, the :class:`Popen` constructor using :func:`os.posix_spawn` no longer raises an 1854 exception on errors like "missing program". Instead the child process fails with a 1855 non-zero :attr:`~Popen.returncode`. 1856 (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.) 1857 1858* The *preexec_fn* argument of * :class:`subprocess.Popen` is no longer 1859 compatible with subinterpreters. The use of the parameter in a 1860 subinterpreter now raises :exc:`RuntimeError`. 1861 (Contributed by Eric Snow in :issue:`34651`, modified by Christian Heimes 1862 in :issue:`37951`.) 1863 1864* The :meth:`imap.IMAP4.logout` method no longer silently ignores arbitrary 1865 exceptions. 1866 (Contributed by Victor Stinner in :issue:`36348`.) 1867 1868* The function :func:`platform.popen` has been removed, after having been deprecated since 1869 Python 3.3: use :func:`os.popen` instead. 1870 (Contributed by Victor Stinner in :issue:`35345`.) 1871 1872* The :func:`statistics.mode` function no longer raises an exception 1873 when given multimodal data. Instead, it returns the first mode 1874 encountered in the input data. (Contributed by Raymond Hettinger 1875 in :issue:`35892`.) 1876 1877* The :meth:`~tkinter.ttk.Treeview.selection` method of the 1878 :class:`tkinter.ttk.Treeview` class no longer takes arguments. Using it with 1879 arguments for changing the selection was deprecated in Python 3.6. Use 1880 specialized methods like :meth:`~tkinter.ttk.Treeview.selection_set` for 1881 changing the selection. (Contributed by Serhiy Storchaka in :issue:`31508`.) 1882 1883* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of 1884 :mod:`xml.dom.minidom`, and the :meth:`write` method of :mod:`xml.etree`, 1885 now preserve the attribute order specified by the user. 1886 (Contributed by Diego Rojas and Raymond Hettinger in :issue:`34160`.) 1887 1888* A :mod:`dbm.dumb` database opened with flags ``'r'`` is now read-only. 1889 :func:`dbm.dumb.open` with flags ``'r'`` and ``'w'`` no longer creates 1890 a database if it does not exist. 1891 (Contributed by Serhiy Storchaka in :issue:`32749`.) 1892 1893* The ``doctype()`` method defined in a subclass of 1894 :class:`~xml.etree.ElementTree.XMLParser` will no longer be called and will 1895 emit a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarning`. 1896 Define the :meth:`doctype() <xml.etree.ElementTree.TreeBuilder.doctype>` 1897 method on a target for handling an XML doctype declaration. 1898 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1899 1900* A :exc:`RuntimeError` is now raised when the custom metaclass doesn't 1901 provide the ``__classcell__`` entry in the namespace passed to 1902 ``type.__new__``. A :exc:`DeprecationWarning` was emitted in Python 1903 3.6--3.7. (Contributed by Serhiy Storchaka in :issue:`23722`.) 1904 1905* The :class:`cProfile.Profile` class can now be used as a context 1906 manager. (Contributed by Scott Sanderson in :issue:`29235`.) 1907 1908* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`, 1909 :func:`shutil.copytree` and :func:`shutil.move` use platform-specific 1910 "fast-copy" syscalls (see 1911 :ref:`shutil-platform-dependent-efficient-copy-operations` section). 1912 1913* :func:`shutil.copyfile` default buffer size on Windows was changed from 1914 16 KiB to 1 MiB. 1915 1916* The ``PyGC_Head`` struct has changed completely. All code that touched the 1917 struct member should be rewritten. (See :issue:`33597`.) 1918 1919* The :c:type:`PyInterpreterState` struct has been moved into the "internal" 1920 header files (specifically Include/internal/pycore_pystate.h). An 1921 opaque ``PyInterpreterState`` is still available as part of the public 1922 API (and stable ABI). The docs indicate that none of the struct's 1923 fields are public, so we hope no one has been using them. However, 1924 if you do rely on one or more of those private fields and have no 1925 alternative then please open a BPO issue. We'll work on helping 1926 you adjust (possibly including adding accessor functions to the 1927 public API). (See :issue:`35886`.) 1928 1929* The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on 1930 success and raises an exception on error under all platforms. Previously, 1931 its behavior was platform-dependent: a nonzero value was returned on success; 1932 zero was returned on error under Windows. A zero value was returned on 1933 success; an exception was raised on error under Unix. 1934 (Contributed by Berker Peksag in :issue:`2122`.) 1935 1936* :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 1937 external entities by default. 1938 (Contributed by Christian Heimes in :issue:`17239`.) 1939 1940* Deleting a key from a read-only :mod:`dbm` database (:mod:`dbm.dumb`, 1941 :mod:`dbm.gnu` or :mod:`dbm.ndbm`) raises :attr:`error` (:exc:`dbm.dumb.error`, 1942 :exc:`dbm.gnu.error` or :exc:`dbm.ndbm.error`) instead of :exc:`KeyError`. 1943 (Contributed by Xiang Zhang in :issue:`33106`.) 1944 1945* :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` 1946 environment variable and does not use :envvar:`HOME`, which is not normally 1947 set for regular user accounts. 1948 (Contributed by Anthony Sottile in :issue:`36264`.) 1949 1950* The exception :class:`asyncio.CancelledError` now inherits from 1951 :class:`BaseException` rather than :class:`Exception`. 1952 (Contributed by Yury Selivanov in :issue:`32528`.) 1953 1954* The function :func:`asyncio.wait_for` now correctly waits for cancellation 1955 when using an instance of :class:`asyncio.Task`. Previously, upon reaching 1956 *timeout*, it was cancelled and immediately returned. 1957 (Contributed by Elvis Pranskevichus in :issue:`32751`.) 1958 1959* The function :func:`asyncio.BaseTransport.get_extra_info` now returns a safe 1960 to use socket object when 'socket' is passed to the *name* parameter. 1961 (Contributed by Yury Selivanov in :issue:`37027`.) 1962 1963* :class:`asyncio.BufferedProtocol` has graduated to the stable API. 1964 1965.. _bpo-36085-whatsnew: 1966 1967* DLL dependencies for extension modules and DLLs loaded with :mod:`ctypes` on 1968 Windows are now resolved more securely. Only the system paths, the directory 1969 containing the DLL or PYD file, and directories added with 1970 :func:`~os.add_dll_directory` are searched for load-time dependencies. 1971 Specifically, :envvar:`PATH` and the current working directory are no longer 1972 used, and modifications to these will no longer have any effect on normal DLL 1973 resolution. If your application relies on these mechanisms, you should check 1974 for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs 1975 directory while loading your library. Note that Windows 7 users will need to 1976 ensure that Windows Update KB2533623 has been installed (this is also verified 1977 by the installer). 1978 (Contributed by Steve Dower in :issue:`36085`.) 1979 1980* The header files and functions related to pgen have been removed after its 1981 replacement by a pure Python implementation. (Contributed by Pablo Galindo 1982 in :issue:`36623`.) 1983 1984* :class:`types.CodeType` has a new parameter in the second position of the 1985 constructor (*posonlyargcount*) to support positional-only arguments defined 1986 in :pep:`570`. The first argument (*argcount*) now represents the total 1987 number of positional arguments (including positional-only arguments). The new 1988 ``replace()`` method of :class:`types.CodeType` can be used to make the code 1989 future-proof. 1990 1991 1992Changes in the C API 1993-------------------- 1994 1995* The :c:type:`PyCompilerFlags` structure got a new *cf_feature_version* 1996 field. It should be initialized to ``PY_MINOR_VERSION``. The field is ignored 1997 by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in 1998 *cf_flags*. 1999 (Contributed by Guido van Rossum in :issue:`35766`.) 2000 2001* The :c:func:`PyEval_ReInitThreads` function has been removed from the C API. 2002 It should not be called explicitly: use :c:func:`PyOS_AfterFork_Child` 2003 instead. 2004 (Contributed by Victor Stinner in :issue:`36728`.) 2005 2006* On Unix, C extensions are no longer linked to libpython except on Android 2007 and Cygwin. When Python is embedded, ``libpython`` must not be loaded with 2008 ``RTLD_LOCAL``, but ``RTLD_GLOBAL`` instead. Previously, using 2009 ``RTLD_LOCAL``, it was already not possible to load C extensions which 2010 were not linked to ``libpython``, like C extensions of the standard 2011 library built by the ``*shared*`` section of ``Modules/Setup``. 2012 (Contributed by Victor Stinner in :issue:`21536`.) 2013 2014* Use of ``#`` variants of formats in parsing or building value (e.g. 2015 :c:func:`PyArg_ParseTuple`, :c:func:`Py_BuildValue`, :c:func:`PyObject_CallFunction`, 2016 etc.) without ``PY_SSIZE_T_CLEAN`` defined raises ``DeprecationWarning`` now. 2017 It will be removed in 3.10 or 4.0. Read :ref:`arg-parsing` for detail. 2018 (Contributed by Inada Naoki in :issue:`36381`.) 2019 2020* Instances of heap-allocated types (such as those created with 2021 :c:func:`PyType_FromSpec`) hold a reference to their type object. 2022 Increasing the reference count of these type objects has been moved from 2023 :c:func:`PyType_GenericAlloc` to the more low-level functions, 2024 :c:func:`PyObject_Init` and :c:func:`PyObject_INIT`. 2025 This makes types created through :c:func:`PyType_FromSpec` behave like 2026 other classes in managed code. 2027 2028 Statically allocated types are not affected. 2029 2030 For the vast majority of cases, there should be no side effect. 2031 However, types that manually increase the reference count after allocating 2032 an instance (perhaps to work around the bug) may now become immortal. 2033 To avoid this, these classes need to call Py_DECREF on the type object 2034 during instance deallocation. 2035 2036 To correctly port these types into 3.8, please apply the following 2037 changes: 2038 2039 * Remove :c:macro:`Py_INCREF` on the type object after allocating an 2040 instance - if any. 2041 This may happen after calling :c:func:`PyObject_New`, 2042 :c:func:`PyObject_NewVar`, :c:func:`PyObject_GC_New`, 2043 :c:func:`PyObject_GC_NewVar`, or any other custom allocator that uses 2044 :c:func:`PyObject_Init` or :c:func:`PyObject_INIT`. 2045 2046 Example: 2047 2048 .. code-block:: c 2049 2050 static foo_struct * 2051 foo_new(PyObject *type) { 2052 foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type); 2053 if (foo == NULL) 2054 return NULL; 2055 #if PY_VERSION_HEX < 0x03080000 2056 // Workaround for Python issue 35810; no longer necessary in Python 3.8 2057 PY_INCREF(type) 2058 #endif 2059 return foo; 2060 } 2061 2062 * Ensure that all custom ``tp_dealloc`` functions of heap-allocated types 2063 decrease the type's reference count. 2064 2065 Example: 2066 2067 .. code-block:: c 2068 2069 static void 2070 foo_dealloc(foo_struct *instance) { 2071 PyObject *type = Py_TYPE(instance); 2072 PyObject_GC_Del(instance); 2073 #if PY_VERSION_HEX >= 0x03080000 2074 // This was not needed before Python 3.8 (Python issue 35810) 2075 Py_DECREF(type); 2076 #endif 2077 } 2078 2079 (Contributed by Eddie Elizondo in :issue:`35810`.) 2080 2081* The :c:macro:`Py_DEPRECATED()` macro has been implemented for MSVC. 2082 The macro now must be placed before the symbol name. 2083 2084 Example: 2085 2086 .. code-block:: c 2087 2088 Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void); 2089 2090 (Contributed by Zackery Spytz in :issue:`33407`.) 2091 2092* The interpreter does not pretend to support binary compatibility of 2093 extension types across feature releases, anymore. A :c:type:`PyTypeObject` 2094 exported by a third-party extension module is supposed to have all the 2095 slots expected in the current Python version, including 2096 :c:member:`~PyTypeObject.tp_finalize` (:const:`Py_TPFLAGS_HAVE_FINALIZE` 2097 is not checked anymore before reading :c:member:`~PyTypeObject.tp_finalize`). 2098 2099 (Contributed by Antoine Pitrou in :issue:`32388`.) 2100 2101* The :c:func:`PyCode_New` has a new parameter in the second position (*posonlyargcount*) 2102 to support :pep:`570`, indicating the number of positional-only arguments. 2103 2104* The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` now accept 2105 two additional ``int`` arguments *end_lineno* and *end_col_offset*. 2106 2107* The :file:`libpython38.a` file to allow MinGW tools to link directly against 2108 :file:`python38.dll` is no longer included in the regular Windows distribution. 2109 If you require this file, it may be generated with the ``gendef`` and 2110 ``dlltool`` tools, which are part of the MinGW binutils package: 2111 2112 .. code-block:: shell 2113 2114 gendef python38.dll > tmp.def 2115 dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a 2116 2117 The location of an installed :file:`pythonXY.dll` will depend on the 2118 installation options and the version and language of Windows. See 2119 :ref:`using-on-windows` for more information. The resulting library should be 2120 placed in the same directory as :file:`pythonXY.lib`, which is generally the 2121 :file:`libs` directory under your Python installation. 2122 2123 (Contributed by Steve Dower in :issue:`37351`.) 2124 2125 2126CPython bytecode changes 2127------------------------ 2128 2129* The interpreter loop has been simplified by moving the logic of unrolling 2130 the stack of blocks into the compiler. The compiler emits now explicit 2131 instructions for adjusting the stack of values and calling the 2132 cleaning-up code for :keyword:`break`, :keyword:`continue` and 2133 :keyword:`return`. 2134 2135 Removed opcodes :opcode:`BREAK_LOOP`, :opcode:`CONTINUE_LOOP`, 2136 :opcode:`SETUP_LOOP` and :opcode:`SETUP_EXCEPT`. Added new opcodes 2137 :opcode:`ROT_FOUR`, :opcode:`BEGIN_FINALLY`, :opcode:`CALL_FINALLY` and 2138 :opcode:`POP_FINALLY`. Changed the behavior of :opcode:`END_FINALLY` 2139 and :opcode:`WITH_CLEANUP_START`. 2140 2141 (Contributed by Mark Shannon, Antoine Pitrou and Serhiy Storchaka in 2142 :issue:`17611`.) 2143 2144* Added new opcode :opcode:`END_ASYNC_FOR` for handling exceptions raised 2145 when awaiting a next item in an :keyword:`async for` loop. 2146 (Contributed by Serhiy Storchaka in :issue:`33041`.) 2147 2148* The :opcode:`MAP_ADD` now expects the value as the first element in the 2149 stack and the key as the second element. This change was made so the key 2150 is always evaluated before the value in dictionary comprehensions, as 2151 proposed by :pep:`572`. (Contributed by Jörn Heissler in :issue:`35224`.) 2152 2153 2154Demos and Tools 2155--------------- 2156 2157Added a benchmark script for timing various ways to access variables: 2158``Tools/scripts/var_access_benchmark.py``. 2159(Contributed by Raymond Hettinger in :issue:`35884`.) 2160 2161Here's a summary of performance improvements since Python 3.3: 2162 2163.. code-block:: none 2164 2165 Python version 3.3 3.4 3.5 3.6 3.7 3.8 2166 -------------- --- --- --- --- --- --- 2167 2168 Variable and attribute read access: 2169 read_local 4.0 7.1 7.1 5.4 5.1 3.9 2170 read_nonlocal 5.3 7.1 8.1 5.8 5.4 4.4 2171 read_global 13.3 15.5 19.0 14.3 13.6 7.6 2172 read_builtin 20.0 21.1 21.6 18.5 19.0 7.5 2173 read_classvar_from_class 20.5 25.6 26.5 20.7 19.5 18.4 2174 read_classvar_from_instance 18.5 22.8 23.5 18.8 17.1 16.4 2175 read_instancevar 26.8 32.4 33.1 28.0 26.3 25.4 2176 read_instancevar_slots 23.7 27.8 31.3 20.8 20.8 20.2 2177 read_namedtuple 68.5 73.8 57.5 45.0 46.8 18.4 2178 read_boundmethod 29.8 37.6 37.9 29.6 26.9 27.7 2179 2180 Variable and attribute write access: 2181 write_local 4.6 8.7 9.3 5.5 5.3 4.3 2182 write_nonlocal 7.3 10.5 11.1 5.6 5.5 4.7 2183 write_global 15.9 19.7 21.2 18.0 18.0 15.8 2184 write_classvar 81.9 92.9 96.0 104.6 102.1 39.2 2185 write_instancevar 36.4 44.6 45.8 40.0 38.9 35.5 2186 write_instancevar_slots 28.7 35.6 36.1 27.3 26.6 25.7 2187 2188 Data structure read access: 2189 read_list 19.2 24.2 24.5 20.8 20.8 19.0 2190 read_deque 19.9 24.7 25.5 20.2 20.6 19.8 2191 read_dict 19.7 24.3 25.7 22.3 23.0 21.0 2192 read_strdict 17.9 22.6 24.3 19.5 21.2 18.9 2193 2194 Data structure write access: 2195 write_list 21.2 27.1 28.5 22.5 21.6 20.0 2196 write_deque 23.8 28.7 30.1 22.7 21.8 23.5 2197 write_dict 25.9 31.4 33.3 29.3 29.2 24.7 2198 write_strdict 22.9 28.4 29.9 27.5 25.2 23.1 2199 2200 Stack (or queue) operations: 2201 list_append_pop 144.2 93.4 112.7 75.4 74.2 50.8 2202 deque_append_pop 30.4 43.5 57.0 49.4 49.2 42.5 2203 deque_append_popleft 30.8 43.7 57.3 49.7 49.7 42.8 2204 2205 Timing loop: 2206 loop_overhead 0.3 0.5 0.6 0.4 0.3 0.3 2207 2208 (Measured from the macOS 64-bit builds found at python.org) 2209 2210Notable changes in Python 3.8.1 2211=============================== 2212 2213Due to significant security concerns, the *reuse_address* parameter of 2214:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is 2215because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more 2216details, see the documentation for ``loop.create_datagram_endpoint()``. 2217(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 2218:issue:`37228`.) 2219
.jpg){kind=link}