1**************************** 2 What's New In Python 3.8 3**************************** 4 5.. Rules for maintenance: 6 7 * Anyone can add text to this document. Do not spend very much time 8 on the wording of your changes, because your text will probably 9 get rewritten to some degree. 10 11 * The maintainer will go through Misc/NEWS periodically and add 12 changes; it's therefore more important to add your changes to 13 Misc/NEWS than to this file. 14 15 * This is not a complete list of every single change; completeness 16 is the purpose of Misc/NEWS. Some changes I consider too small 17 or esoteric to include. If such a change is added to the text, 18 I'll just remove it. (This is another reason you shouldn't spend 19 too much time on writing your addition.) 20 21 * If you want to draw your new text to the attention of the 22 maintainer, add 'XXX' to the beginning of the paragraph or 23 section. 24 25 * It's OK to just add a fragmentary note about a change. For 26 example: "XXX Describe the transmogrify() function added to the 27 socket module." The maintainer will research the change and 28 write the necessary text. 29 30 * You can comment out your additions if you like, but it's not 31 necessary (especially when a final release is some months away). 32 33 * Credit the author of a patch or bugfix. Just the name is 34 sufficient; the e-mail address isn't necessary. 35 36 * It's helpful to add the bug/patch number as a comment: 37 38 XXX Describe the transmogrify() function added to the socket 39 module. 40 (Contributed by P.Y. Developer in :issue:`12345`.) 41 42 This saves the maintainer the effort of going through the Git log 43 when researching a change. 44 45:Editor: Raymond Hettinger 46 47This article explains the new features in Python 3.8, compared to 3.7. 48For full details, see the :ref:`changelog <changelog>`. 49 50Python 3.8 was released on October 14th, 2019. 51 52.. testsetup:: 53 54 from datetime import date 55 from math import cos, radians 56 from unicodedata import normalize 57 import re 58 import math 59 60 61Summary -- Release highlights 62============================= 63 64.. This section singles out the most important changes in Python 3.8. 65 Brevity is key. 66 67 68.. PEP-sized items next. 69 70 71 72New Features 73============ 74 75Assignment expressions 76---------------------- 77 78There is new syntax ``:=`` that assigns values to variables as part of a larger 79expression. It is affectionately known as "the walrus operator" due to 80its resemblance to `the eyes and tusks of a walrus 81<https://en.wikipedia.org/wiki/Walrus#/media/File:Pacific_Walrus_-_Bull_(8247646168).jpg>`_. 82 83In this example, the assignment expression helps avoid calling 84:func:`len` twice:: 85 86 if (n := len(a)) > 10: 87 print(f"List is too long ({n} elements, expected <= 10)") 88 89A similar benefit arises during regular expression matching where 90match objects are needed twice, once to test whether a match 91occurred and another to extract a subgroup:: 92 93 discount = 0.0 94 if (mo := re.search(r'(\d+)% discount', advertisement)): 95 discount = float(mo.group(1)) / 100.0 96 97The operator is also useful with while-loops that compute 98a value to test loop termination and then need that same 99value again in the body of the loop:: 100 101 # Loop over fixed length blocks 102 while (block := f.read(256)) != '': 103 process(block) 104 105Another motivating use case arises in list comprehensions where 106a value computed in a filtering condition is also needed in 107the expression body:: 108 109 [clean_name.title() for name in names 110 if (clean_name := normalize('NFC', name)) in allowed_names] 111 112Try to limit use of the walrus operator to clean cases that reduce 113complexity and improve readability. 114 115See :pep:`572` for a full description. 116 117(Contributed by Emily Morehouse in :issue:`35224`.) 118 119 120Positional-only parameters 121-------------------------- 122 123There is a new function parameter syntax ``/`` to indicate that some 124function parameters must be specified positionally and cannot be used as 125keyword arguments. This is the same notation shown by ``help()`` for C 126functions annotated with Larry Hastings' `Argument Clinic 127<https://docs.python.org/3/howto/clinic.html>`_ tool. 128 129In the following example, parameters *a* and *b* are positional-only, 130while *c* or *d* can be positional or keyword, and *e* or *f* are 131required to be keywords:: 132 133 def f(a, b, /, c, d, *, e, f): 134 print(a, b, c, d, e, f) 135 136The following is a valid call:: 137 138 f(10, 20, 30, d=40, e=50, f=60) 139 140However, these are invalid calls:: 141 142 f(10, b=20, c=30, d=40, e=50, f=60) # b cannot be a keyword argument 143 f(10, 20, 30, 40, 50, f=60) # e must be a keyword argument 144 145One use case for this notation is that it allows pure Python functions 146to fully emulate behaviors of existing C coded functions. For example, 147the built-in :func:`divmod` function does not accept keyword arguments:: 148 149 def divmod(a, b, /): 150 "Emulate the built in divmod() function" 151 return (a // b, a % b) 152 153Another use case is to preclude keyword arguments when the parameter 154name is not helpful. For example, the builtin :func:`len` function has 155the signature ``len(obj, /)``. This precludes awkward calls such as:: 156 157 len(obj='hello') # The "obj" keyword argument impairs readability 158 159A further benefit of marking a parameter as positional-only is that it 160allows the parameter name to be changed in the future without risk of 161breaking client code. For example, in the :mod:`statistics` module, the 162parameter name *dist* may be changed in the future. This was made 163possible with the following function specification:: 164 165 def quantiles(dist, /, *, n=4, method='exclusive') 166 ... 167 168Since the parameters to the left of ``/`` are not exposed as possible 169keywords, the parameters names remain available for use in ``**kwargs``:: 170 171 >>> def f(a, b, /, **kwargs): 172 ... print(a, b, kwargs) 173 ... 174 >>> f(10, 20, a=1, b=2, c=3) # a and b are used in two ways 175 10 20 {'a': 1, 'b': 2, 'c': 3} 176 177This greatly simplifies the implementation of functions and methods 178that need to accept arbitrary keyword arguments. For example, here 179is an excerpt from code in the :mod:`collections` module:: 180 181 class Counter(dict): 182 183 def __init__(self, iterable=None, /, **kwds): 184 # Note "iterable" is a possible keyword argument 185 186See :pep:`570` for a full description. 187 188(Contributed by Pablo Galindo in :issue:`36540`.) 189 190.. TODO: Pablo will sprint on docs at PyCon US 2019. 191 192 193Parallel filesystem cache for compiled bytecode files 194----------------------------------------------------- 195 196The new :envvar:`PYTHONPYCACHEPREFIX` setting (also available as 197:option:`-X` ``pycache_prefix``) configures the implicit bytecode 198cache to use a separate parallel filesystem tree, rather than 199the default ``__pycache__`` subdirectories within each source 200directory. 201 202The location of the cache is reported in :data:`sys.pycache_prefix` 203(:const:`None` indicates the default location in ``__pycache__`` 204subdirectories). 205 206(Contributed by Carl Meyer in :issue:`33499`.) 207 208 209Debug build uses the same ABI as release build 210----------------------------------------------- 211 212Python now uses the same ABI whether it's built in release or debug mode. On 213Unix, when Python is built in debug mode, it is now possible to load C 214extensions built in release mode and C extensions built using the stable ABI. 215 216Release builds and debug builds are now ABI compatible: defining the 217``Py_DEBUG`` macro no longer implies the ``Py_TRACE_REFS`` macro, which 218introduces the only ABI incompatibility. The ``Py_TRACE_REFS`` macro, which 219adds the :func:`sys.getobjects` function and the :envvar:`PYTHONDUMPREFS` 220environment variable, can be set using the new ``./configure --with-trace-refs`` 221build option. 222(Contributed by Victor Stinner in :issue:`36465`.) 223 224On Unix, C extensions are no longer linked to libpython except on Android 225and Cygwin. 226It is now possible 227for a statically linked Python to load a C extension built using a shared 228library Python. 229(Contributed by Victor Stinner in :issue:`21536`.) 230 231On Unix, when Python is built in debug mode, import now also looks for C 232extensions compiled in release mode and for C extensions compiled with the 233stable ABI. 234(Contributed by Victor Stinner in :issue:`36722`.) 235 236To embed Python into an application, a new ``--embed`` option must be passed to 237``python3-config --libs --embed`` to get ``-lpython3.8`` (link the application 238to libpython). To support both 3.8 and older, try ``python3-config --libs 239--embed`` first and fallback to ``python3-config --libs`` (without ``--embed``) 240if the previous command fails. 241 242Add a pkg-config ``python-3.8-embed`` module to embed Python into an 243application: ``pkg-config python-3.8-embed --libs`` includes ``-lpython3.8``. 244To support both 3.8 and older, try ``pkg-config python-X.Y-embed --libs`` first 245and fallback to ``pkg-config python-X.Y --libs`` (without ``--embed``) if the 246previous command fails (replace ``X.Y`` with the Python version). 247 248On the other hand, ``pkg-config python3.8 --libs`` no longer contains 249``-lpython3.8``. C extensions must not be linked to libpython (except on 250Android and Cygwin, whose cases are handled by the script); 251this change is backward incompatible on purpose. 252(Contributed by Victor Stinner in :issue:`36721`.) 253 254 255f-strings support ``=`` for self-documenting expressions and debugging 256---------------------------------------------------------------------- 257 258Added an ``=`` specifier to :term:`f-string`\s. An f-string such as 259``f'{expr=}'`` will expand to the text of the expression, an equal sign, 260then the representation of the evaluated expression. For example: 261 262 >>> user = 'eric_idle' 263 >>> member_since = date(1975, 7, 31) 264 >>> f'{user=} {member_since=}' 265 "user='eric_idle' member_since=datetime.date(1975, 7, 31)" 266 267The usual :ref:`f-string format specifiers <f-strings>` allow more 268control over how the result of the expression is displayed:: 269 270 >>> delta = date.today() - member_since 271 >>> f'{user=!s} {delta.days=:,d}' 272 'user=eric_idle delta.days=16,075' 273 274The ``=`` specifier will display the whole expression so that 275calculations can be shown:: 276 277 >>> print(f'{theta=} {cos(radians(theta))=:.3f}') 278 theta=30 cos(radians(theta))=0.866 279 280(Contributed by Eric V. Smith and Larry Hastings in :issue:`36817`.) 281 282 283PEP 578: Python Runtime Audit Hooks 284----------------------------------- 285 286The PEP adds an Audit Hook and Verified Open Hook. Both are available from 287Python and native code, allowing applications and frameworks written in pure 288Python code to take advantage of extra notifications, while also allowing 289embedders or system administrators to deploy builds of Python where auditing is 290always enabled. 291 292See :pep:`578` for full details. 293 294 295PEP 587: Python Initialization Configuration 296-------------------------------------------- 297 298The :pep:`587` adds a new C API to configure the Python Initialization 299providing finer control on the whole configuration and better error reporting. 300 301New structures: 302 303* :c:type:`PyConfig` 304* :c:type:`PyPreConfig` 305* :c:type:`PyStatus` 306* :c:type:`PyWideStringList` 307 308New functions: 309 310* :c:func:`PyConfig_Clear` 311* :c:func:`PyConfig_InitIsolatedConfig` 312* :c:func:`PyConfig_InitPythonConfig` 313* :c:func:`PyConfig_Read` 314* :c:func:`PyConfig_SetArgv` 315* :c:func:`PyConfig_SetBytesArgv` 316* :c:func:`PyConfig_SetBytesString` 317* :c:func:`PyConfig_SetString` 318* :c:func:`PyPreConfig_InitIsolatedConfig` 319* :c:func:`PyPreConfig_InitPythonConfig` 320* :c:func:`PyStatus_Error` 321* :c:func:`PyStatus_Exception` 322* :c:func:`PyStatus_Exit` 323* :c:func:`PyStatus_IsError` 324* :c:func:`PyStatus_IsExit` 325* :c:func:`PyStatus_NoMemory` 326* :c:func:`PyStatus_Ok` 327* :c:func:`PyWideStringList_Append` 328* :c:func:`PyWideStringList_Insert` 329* :c:func:`Py_BytesMain` 330* :c:func:`Py_ExitStatusException` 331* :c:func:`Py_InitializeFromConfig` 332* :c:func:`Py_PreInitialize` 333* :c:func:`Py_PreInitializeFromArgs` 334* :c:func:`Py_PreInitializeFromBytesArgs` 335* :c:func:`Py_RunMain` 336 337This PEP also adds ``_PyRuntimeState.preconfig`` (:c:type:`PyPreConfig` type) 338and ``PyInterpreterState.config`` (:c:type:`PyConfig` type) fields to these 339internal structures. ``PyInterpreterState.config`` becomes the new 340reference configuration, replacing global configuration variables and 341other private variables. 342 343See :ref:`Python Initialization Configuration <init-config>` for the 344documentation. 345 346See :pep:`587` for a full description. 347 348(Contributed by Victor Stinner in :issue:`36763`.) 349 350 351Vectorcall: a fast calling protocol for CPython 352----------------------------------------------- 353 354The "vectorcall" protocol is added to the Python/C API. 355It is meant to formalize existing optimizations which were already done 356for various classes. 357Any extension type implementing a callable can use this protocol. 358 359This is currently provisional. 360The aim is to make it fully public in Python 3.9. 361 362See :pep:`590` for a full description. 363 364(Contributed by Jeroen Demeyer and Mark Shannon in :issue:`36974`.) 365 366 367Pickle protocol 5 with out-of-band data buffers 368----------------------------------------------- 369 370When :mod:`pickle` is used to transfer large data between Python processes 371in order to take advantage of multi-core or multi-machine processing, 372it is important to optimize the transfer by reducing memory copies, and 373possibly by applying custom techniques such as data-dependent compression. 374 375The :mod:`pickle` protocol 5 introduces support for out-of-band buffers 376where :pep:`3118`-compatible data can be transmitted separately from the 377main pickle stream, at the discretion of the communication layer. 378 379See :pep:`574` for a full description. 380 381(Contributed by Antoine Pitrou in :issue:`36785`.) 382 383 384Other Language Changes 385====================== 386 387* A :keyword:`continue` statement was illegal in the :keyword:`finally` clause 388 due to a problem with the implementation. In Python 3.8 this restriction 389 was lifted. 390 (Contributed by Serhiy Storchaka in :issue:`32489`.) 391 392* The :class:`bool`, :class:`int`, and :class:`fractions.Fraction` types 393 now have an :meth:`~int.as_integer_ratio` method like that found in 394 :class:`float` and :class:`decimal.Decimal`. This minor API extension 395 makes it possible to write ``numerator, denominator = 396 x.as_integer_ratio()`` and have it work across multiple numeric types. 397 (Contributed by Lisa Roach in :issue:`33073` and Raymond Hettinger in 398 :issue:`37819`.) 399 400* Constructors of :class:`int`, :class:`float` and :class:`complex` will now 401 use the :meth:`~object.__index__` special method, if available and the 402 corresponding method :meth:`~object.__int__`, :meth:`~object.__float__` 403 or :meth:`~object.__complex__` is not available. 404 (Contributed by Serhiy Storchaka in :issue:`20092`.) 405 406* Added support of ``\N{name}`` escapes in :mod:`regular expressions <re>`:: 407 408 >>> notice = 'Copyright © 2019' 409 >>> copyright_year_pattern = re.compile(r'\N{copyright sign}\s*(\d{4})') 410 >>> int(copyright_year_pattern.search(notice).group(1)) 411 2019 412 413 (Contributed by Jonathan Eunice and Serhiy Storchaka in :issue:`30688`.) 414 415* Dict and dictviews are now iterable in reversed insertion order using 416 :func:`reversed`. (Contributed by Rémi Lapeyre in :issue:`33462`.) 417 418* The syntax allowed for keyword names in function calls was further 419 restricted. In particular, ``f((keyword)=arg)`` is no longer allowed. It was 420 never intended to permit more than a bare name on the left-hand side of a 421 keyword argument assignment term. 422 (Contributed by Benjamin Peterson in :issue:`34641`.) 423 424* Generalized iterable unpacking in :keyword:`yield` and 425 :keyword:`return` statements no longer requires enclosing parentheses. 426 This brings the *yield* and *return* syntax into better agreement with 427 normal assignment syntax:: 428 429 >>> def parse(family): 430 lastname, *members = family.split() 431 return lastname.upper(), *members 432 433 >>> parse('simpsons homer marge bart lisa sally') 434 ('SIMPSONS', 'homer', 'marge', 'bart', 'lisa', 'sally') 435 436 (Contributed by David Cuthbert and Jordan Chapman in :issue:`32117`.) 437 438* When a comma is missed in code such as ``[(10, 20) (30, 40)]``, the 439 compiler displays a :exc:`SyntaxWarning` with a helpful suggestion. 440 This improves on just having a :exc:`TypeError` indicating that the 441 first tuple was not callable. (Contributed by Serhiy Storchaka in 442 :issue:`15248`.) 443 444* Arithmetic operations between subclasses of :class:`datetime.date` or 445 :class:`datetime.datetime` and :class:`datetime.timedelta` objects now return 446 an instance of the subclass, rather than the base class. This also affects 447 the return type of operations whose implementation (directly or indirectly) 448 uses :class:`datetime.timedelta` arithmetic, such as 449 :meth:`~datetime.datetime.astimezone`. 450 (Contributed by Paul Ganssle in :issue:`32417`.) 451 452* When the Python interpreter is interrupted by Ctrl-C (SIGINT) and the 453 resulting :exc:`KeyboardInterrupt` exception is not caught, the Python process 454 now exits via a SIGINT signal or with the correct exit code such that the 455 calling process can detect that it died due to a Ctrl-C. Shells on POSIX 456 and Windows use this to properly terminate scripts in interactive sessions. 457 (Contributed by Google via Gregory P. Smith in :issue:`1054041`.) 458 459* Some advanced styles of programming require updating the 460 :class:`types.CodeType` object for an existing function. Since code 461 objects are immutable, a new code object needs to be created, one 462 that is modeled on the existing code object. With 19 parameters, 463 this was somewhat tedious. Now, the new ``replace()`` method makes 464 it possible to create a clone with a few altered parameters. 465 466 Here's an example that alters the :func:`statistics.mean` function to 467 prevent the *data* parameter from being used as a keyword argument:: 468 469 >>> from statistics import mean 470 >>> mean(data=[10, 20, 90]) 471 40 472 >>> mean.__code__ = mean.__code__.replace(co_posonlyargcount=1) 473 >>> mean(data=[10, 20, 90]) 474 Traceback (most recent call last): 475 ... 476 TypeError: mean() got some positional-only arguments passed as keyword arguments: 'data' 477 478 (Contributed by Victor Stinner in :issue:`37032`.) 479 480* For integers, the three-argument form of the :func:`pow` function now 481 permits the exponent to be negative in the case where the base is 482 relatively prime to the modulus. It then computes a modular inverse to 483 the base when the exponent is ``-1``, and a suitable power of that 484 inverse for other negative exponents. For example, to compute the 485 `modular multiplicative inverse 486 <https://en.wikipedia.org/wiki/Modular_multiplicative_inverse>`_ of 38 487 modulo 137, write:: 488 489 >>> pow(38, -1, 137) 490 119 491 >>> 119 * 38 % 137 492 1 493 494 Modular inverses arise in the solution of `linear Diophantine 495 equations <https://en.wikipedia.org/wiki/Diophantine_equation>`_. 496 For example, to find integer solutions for ``4258�� + 147�� = 369``, 497 first rewrite as ``4258�� ≡ 369 (mod 147)`` then solve: 498 499 >>> x = 369 * pow(4258, -1, 147) % 147 500 >>> y = (4258 * x - 369) // -147 501 >>> 4258 * x + 147 * y 502 369 503 504 (Contributed by Mark Dickinson in :issue:`36027`.) 505 506* Dict comprehensions have been synced-up with dict literals so that the 507 key is computed first and the value second:: 508 509 >>> # Dict comprehension 510 >>> cast = {input('role? '): input('actor? ') for i in range(2)} 511 role? King Arthur 512 actor? Chapman 513 role? Black Knight 514 actor? Cleese 515 516 >>> # Dict literal 517 >>> cast = {input('role? '): input('actor? ')} 518 role? Sir Robin 519 actor? Eric Idle 520 521 The guaranteed execution order is helpful with assignment expressions 522 because variables assigned in the key expression will be available in 523 the value expression:: 524 525 >>> names = ['Martin von Löwis', 'Łukasz Langa', 'Walter Dörwald'] 526 >>> {(n := normalize('NFC', name)).casefold() : n for name in names} 527 {'martin von löwis': 'Martin von Löwis', 528 'łukasz langa': 'Łukasz Langa', 529 'walter dörwald': 'Walter Dörwald'} 530 531 (Contributed by Jörn Heissler in :issue:`35224`.) 532 533* The :meth:`object.__reduce__` method can now return a tuple from two to 534 six elements long. Formerly, five was the limit. The new, optional sixth 535 element is a callable with a ``(obj, state)`` signature. This allows the 536 direct control over the state-updating behavior of a specific object. If 537 not *None*, this callable will have priority over the object's 538 :meth:`~__setstate__` method. 539 (Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.) 540 541New Modules 542=========== 543 544* The new :mod:`importlib.metadata` module provides (provisional) support for 545 reading metadata from third-party packages. For example, it can extract an 546 installed package's version number, list of entry points, and more:: 547 548 >>> # Note following example requires that the popular "requests" 549 >>> # package has been installed. 550 >>> 551 >>> from importlib.metadata import version, requires, files 552 >>> version('requests') 553 '2.22.0' 554 >>> list(requires('requests')) 555 ['chardet (<3.1.0,>=3.0.2)'] 556 >>> list(files('requests'))[:5] 557 [PackagePath('requests-2.22.0.dist-info/INSTALLER'), 558 PackagePath('requests-2.22.0.dist-info/LICENSE'), 559 PackagePath('requests-2.22.0.dist-info/METADATA'), 560 PackagePath('requests-2.22.0.dist-info/RECORD'), 561 PackagePath('requests-2.22.0.dist-info/WHEEL')] 562 563 (Contributed by Barry Warsaw and Jason R. Coombs in :issue:`34632`.) 564 565 566Improved Modules 567================ 568 569ast 570--- 571 572AST nodes now have ``end_lineno`` and ``end_col_offset`` attributes, 573which give the precise location of the end of the node. (This only 574applies to nodes that have ``lineno`` and ``col_offset`` attributes.) 575 576New function :func:`ast.get_source_segment` returns the source code 577for a specific AST node. 578 579(Contributed by Ivan Levkivskyi in :issue:`33416`.) 580 581The :func:`ast.parse` function has some new flags: 582 583* ``type_comments=True`` causes it to return the text of :pep:`484` and 584 :pep:`526` type comments associated with certain AST nodes; 585 586* ``mode='func_type'`` can be used to parse :pep:`484` "signature type 587 comments" (returned for function definition AST nodes); 588 589* ``feature_version=(3, N)`` allows specifying an earlier Python 3 590 version. For example, ``feature_version=(3, 4)`` will treat 591 :keyword:`async` and :keyword:`await` as non-reserved words. 592 593(Contributed by Guido van Rossum in :issue:`35766`.) 594 595 596asyncio 597------- 598 599:func:`asyncio.run` has graduated from the provisional to stable API. This 600function can be used to execute a :term:`coroutine` and return the result while 601automatically managing the event loop. For example:: 602 603 import asyncio 604 605 async def main(): 606 await asyncio.sleep(0) 607 return 42 608 609 asyncio.run(main()) 610 611This is *roughly* equivalent to:: 612 613 import asyncio 614 615 async def main(): 616 await asyncio.sleep(0) 617 return 42 618 619 loop = asyncio.new_event_loop() 620 asyncio.set_event_loop(loop) 621 try: 622 loop.run_until_complete(main()) 623 finally: 624 asyncio.set_event_loop(None) 625 loop.close() 626 627 628The actual implementation is significantly more complex. Thus, 629:func:`asyncio.run` should be the preferred way of running asyncio programs. 630 631(Contributed by Yury Selivanov in :issue:`32314`.) 632 633Running ``python -m asyncio`` launches a natively async REPL. This allows rapid 634experimentation with code that has a top-level :keyword:`await`. There is no 635longer a need to directly call ``asyncio.run()`` which would spawn a new event 636loop on every invocation: 637 638.. code-block:: none 639 640 $ python -m asyncio 641 asyncio REPL 3.8.0 642 Use "await" directly instead of "asyncio.run()". 643 Type "help", "copyright", "credits" or "license" for more information. 644 >>> import asyncio 645 >>> await asyncio.sleep(10, result='hello') 646 hello 647 648(Contributed by Yury Selivanov in :issue:`37028`.) 649 650The exception :class:`asyncio.CancelledError` now inherits from 651:class:`BaseException` rather than :class:`Exception` and no longer inherits 652from :class:`concurrent.futures.CancelledError`. 653(Contributed by Yury Selivanov in :issue:`32528`.) 654 655On Windows, the default event loop is now :class:`~asyncio.ProactorEventLoop`. 656(Contributed by Victor Stinner in :issue:`34687`.) 657 658:class:`~asyncio.ProactorEventLoop` now also supports UDP. 659(Contributed by Adam Meily and Andrew Svetlov in :issue:`29883`.) 660 661:class:`~asyncio.ProactorEventLoop` can now be interrupted by 662:exc:`KeyboardInterrupt` ("CTRL+C"). 663(Contributed by Vladimir Matveev in :issue:`23057`.) 664 665Added :meth:`asyncio.Task.get_coro` for getting the wrapped coroutine 666within an :class:`asyncio.Task`. 667(Contributed by Alex Grönholm in :issue:`36999`.) 668 669Asyncio tasks can now be named, either by passing the ``name`` keyword 670argument to :func:`asyncio.create_task` or 671the :meth:`~asyncio.loop.create_task` event loop method, or by 672calling the :meth:`~asyncio.Task.set_name` method on the task object. The 673task name is visible in the ``repr()`` output of :class:`asyncio.Task` and 674can also be retrieved using the :meth:`~asyncio.Task.get_name` method. 675(Contributed by Alex Grönholm in :issue:`34270`.) 676 677Added support for 678`Happy Eyeballs <https://en.wikipedia.org/wiki/Happy_Eyeballs>`_ to 679:func:`asyncio.loop.create_connection`. To specify the behavior, two new 680parameters have been added: *happy_eyeballs_delay* and *interleave*. The Happy 681Eyeballs algorithm improves responsiveness in applications that support IPv4 682and IPv6 by attempting to simultaneously connect using both. 683(Contributed by twisteroid ambassador in :issue:`33530`.) 684 685 686builtins 687-------- 688 689The :func:`compile` built-in has been improved to accept the 690``ast.PyCF_ALLOW_TOP_LEVEL_AWAIT`` flag. With this new flag passed, 691:func:`compile` will allow top-level ``await``, ``async for`` and ``async with`` 692constructs that are usually considered invalid syntax. Asynchronous code object 693marked with the ``CO_COROUTINE`` flag may then be returned. 694(Contributed by Matthias Bussonnier in :issue:`34616`) 695 696 697collections 698----------- 699 700The :meth:`~collections.somenamedtuple._asdict` method for 701:func:`collections.namedtuple` now returns a :class:`dict` instead of a 702:class:`collections.OrderedDict`. This works because regular dicts have 703guaranteed ordering since Python 3.7. If the extra features of 704:class:`OrderedDict` are required, the suggested remediation is to cast the 705result to the desired type: ``OrderedDict(nt._asdict())``. 706(Contributed by Raymond Hettinger in :issue:`35864`.) 707 708 709cProfile 710-------- 711 712The :class:`cProfile.Profile <profile.Profile>` class can now be used as a context manager. 713Profile a block of code by running:: 714 715 import cProfile 716 717 with cProfile.Profile() as profiler: 718 # code to be profiled 719 ... 720 721(Contributed by Scott Sanderson in :issue:`29235`.) 722 723 724csv 725--- 726 727The :class:`csv.DictReader` now returns instances of :class:`dict` instead of 728a :class:`collections.OrderedDict`. The tool is now faster and uses less 729memory while still preserving the field order. 730(Contributed by Michael Selik in :issue:`34003`.) 731 732 733curses 734------- 735 736Added a new variable holding structured version information for the 737underlying ncurses library: :data:`~curses.ncurses_version`. 738(Contributed by Serhiy Storchaka in :issue:`31680`.) 739 740 741ctypes 742------ 743 744On Windows, :class:`~ctypes.CDLL` and subclasses now accept a *winmode* parameter 745to specify flags for the underlying ``LoadLibraryEx`` call. The default flags are 746set to only load DLL dependencies from trusted locations, including the path 747where the DLL is stored (if a full or partial path is used to load the initial 748DLL) and paths added by :func:`~os.add_dll_directory`. 749(Contributed by Steve Dower in :issue:`36085`.) 750 751 752datetime 753-------- 754 755Added new alternate constructors :meth:`datetime.date.fromisocalendar` and 756:meth:`datetime.datetime.fromisocalendar`, which construct :class:`date` and 757:class:`datetime` objects respectively from ISO year, week number, and weekday; 758these are the inverse of each class's ``isocalendar`` method. 759(Contributed by Paul Ganssle in :issue:`36004`.) 760 761 762functools 763--------- 764 765:func:`functools.lru_cache` can now be used as a straight decorator rather 766than as a function returning a decorator. So both of these are now supported:: 767 768 @lru_cache 769 def f(x): 770 ... 771 772 @lru_cache(maxsize=256) 773 def f(x): 774 ... 775 776(Contributed by Raymond Hettinger in :issue:`36772`.) 777 778Added a new :func:`functools.cached_property` decorator, for computed properties 779cached for the life of the instance. :: 780 781 import functools 782 import statistics 783 784 class Dataset: 785 def __init__(self, sequence_of_numbers): 786 self.data = sequence_of_numbers 787 788 @functools.cached_property 789 def variance(self): 790 return statistics.variance(self.data) 791 792(Contributed by Carl Meyer in :issue:`21145`) 793 794 795Added a new :func:`functools.singledispatchmethod` decorator that converts 796methods into :term:`generic functions <generic function>` using 797:term:`single dispatch`:: 798 799 from functools import singledispatchmethod 800 from contextlib import suppress 801 802 class TaskManager: 803 804 def __init__(self, tasks): 805 self.tasks = list(tasks) 806 807 @singledispatchmethod 808 def discard(self, value): 809 with suppress(ValueError): 810 self.tasks.remove(value) 811 812 @discard.register(list) 813 def _(self, tasks): 814 targets = set(tasks) 815 self.tasks = [x for x in self.tasks if x not in targets] 816 817(Contributed by Ethan Smith in :issue:`32380`) 818 819gc 820-- 821 822:func:`~gc.get_objects` can now receive an optional *generation* parameter 823indicating a generation to get objects from. 824(Contributed by Pablo Galindo in :issue:`36016`.) 825 826 827gettext 828------- 829 830Added :func:`~gettext.pgettext` and its variants. 831(Contributed by Franz Glasner, Éric Araujo, and Cheryl Sabella in :issue:`2504`.) 832 833 834gzip 835---- 836 837Added the *mtime* parameter to :func:`gzip.compress` for reproducible output. 838(Contributed by Guo Ci Teo in :issue:`34898`.) 839 840A :exc:`~gzip.BadGzipFile` exception is now raised instead of :exc:`OSError` 841for certain types of invalid or corrupt gzip files. 842(Contributed by Filip Gruszczyński, Michele Orrù, and Zackery Spytz in 843:issue:`6584`.) 844 845 846IDLE and idlelib 847---------------- 848 849Output over N lines (50 by default) is squeezed down to a button. 850N can be changed in the PyShell section of the General page of the 851Settings dialog. Fewer, but possibly extra long, lines can be squeezed by 852right clicking on the output. Squeezed output can be expanded in place 853by double-clicking the button or into the clipboard or a separate window 854by right-clicking the button. (Contributed by Tal Einat in :issue:`1529353`.) 855 856Add "Run Customized" to the Run menu to run a module with customized 857settings. Any command line arguments entered are added to sys.argv. 858They also re-appear in the box for the next customized run. One can also 859suppress the normal Shell main module restart. (Contributed by Cheryl 860Sabella, Terry Jan Reedy, and others in :issue:`5680` and :issue:`37627`.) 861 862Added optional line numbers for IDLE editor windows. Windows 863open without line numbers unless set otherwise in the General 864tab of the configuration dialog. Line numbers for an existing 865window are shown and hidden in the Options menu. 866(Contributed by Tal Einat and Saimadhav Heblikar in :issue:`17535`.) 867 868OS native encoding is now used for converting between Python strings and Tcl 869objects. This allows IDLE to work with emoji and other non-BMP characters. 870These characters can be displayed or copied and pasted to or from the 871clipboard. Converting strings from Tcl to Python and back now never fails. 872(Many people worked on this for eight years but the problem was finally 873solved by Serhiy Storchaka in :issue:`13153`.) 874 875The changes above have been backported to 3.7 maintenance releases. 876 877 878inspect 879------- 880 881The :func:`inspect.getdoc` function can now find docstrings for ``__slots__`` 882if that attribute is a :class:`dict` where the values are docstrings. 883This provides documentation options similar to what we already have 884for :func:`property`, :func:`classmethod`, and :func:`staticmethod`:: 885 886 class AudioClip: 887 __slots__ = {'bit_rate': 'expressed in kilohertz to one decimal place', 888 'duration': 'in seconds, rounded up to an integer'} 889 def __init__(self, bit_rate, duration): 890 self.bit_rate = round(bit_rate / 1000.0, 1) 891 self.duration = ceil(duration) 892 893(Contributed by Raymond Hettinger in :issue:`36326`.) 894 895 896io 897-- 898 899In development mode (:option:`-X` ``env``) and in debug build, the 900:class:`io.IOBase` finalizer now logs the exception if the ``close()`` method 901fails. The exception is ignored silently by default in release build. 902(Contributed by Victor Stinner in :issue:`18748`.) 903 904 905itertools 906--------- 907 908The :func:`itertools.accumulate` function added an option *initial* keyword 909argument to specify an initial value:: 910 911 >>> from itertools import accumulate 912 >>> list(accumulate([10, 5, 30, 15], initial=1000)) 913 [1000, 1010, 1015, 1045, 1060] 914 915(Contributed by Lisa Roach in :issue:`34659`.) 916 917 918json.tool 919--------- 920 921Add option ``--json-lines`` to parse every input line as a separate JSON object. 922(Contributed by Weipeng Hong in :issue:`31553`.) 923 924 925logging 926------- 927 928Added a *force* keyword argument to :func:`logging.basicConfig()` 929When set to true, any existing handlers attached 930to the root logger are removed and closed before carrying out the 931configuration specified by the other arguments. 932 933This solves a long-standing problem. Once a logger or *basicConfig()* had 934been called, subsequent calls to *basicConfig()* were silently ignored. 935This made it difficult to update, experiment with, or teach the various 936logging configuration options using the interactive prompt or a Jupyter 937notebook. 938 939(Suggested by Raymond Hettinger, implemented by Dong-hee Na, and 940reviewed by Vinay Sajip in :issue:`33897`.) 941 942 943math 944---- 945 946Added new function :func:`math.dist` for computing Euclidean distance 947between two points. (Contributed by Raymond Hettinger in :issue:`33089`.) 948 949Expanded the :func:`math.hypot` function to handle multiple dimensions. 950Formerly, it only supported the 2-D case. 951(Contributed by Raymond Hettinger in :issue:`33089`.) 952 953Added new function, :func:`math.prod`, as analogous function to :func:`sum` 954that returns the product of a 'start' value (default: 1) times an iterable of 955numbers:: 956 957 >>> prior = 0.8 958 >>> likelihoods = [0.625, 0.84, 0.30] 959 >>> math.prod(likelihoods, start=prior) 960 0.126 961 962(Contributed by Pablo Galindo in :issue:`35606`.) 963 964Added two new combinatoric functions :func:`math.perm` and :func:`math.comb`:: 965 966 >>> math.perm(10, 3) # Permutations of 10 things taken 3 at a time 967 720 968 >>> math.comb(10, 3) # Combinations of 10 things taken 3 at a time 969 120 970 971(Contributed by Yash Aggarwal, Keller Fuchs, Serhiy Storchaka, and Raymond 972Hettinger in :issue:`37128`, :issue:`37178`, and :issue:`35431`.) 973 974Added a new function :func:`math.isqrt` for computing accurate integer square 975roots without conversion to floating point. The new function supports 976arbitrarily large integers. It is faster than ``floor(sqrt(n))`` but slower 977than :func:`math.sqrt`:: 978 979 >>> r = 650320427 980 >>> s = r ** 2 981 >>> isqrt(s - 1) # correct 982 650320426 983 >>> floor(sqrt(s - 1)) # incorrect 984 650320427 985 986(Contributed by Mark Dickinson in :issue:`36887`.) 987 988The function :func:`math.factorial` no longer accepts arguments that are not 989int-like. (Contributed by Pablo Galindo in :issue:`33083`.) 990 991 992mmap 993---- 994 995The :class:`mmap.mmap` class now has an :meth:`~mmap.mmap.madvise` method to 996access the ``madvise()`` system call. 997(Contributed by Zackery Spytz in :issue:`32941`.) 998 999 1000multiprocessing 1001--------------- 1002 1003Added new :mod:`multiprocessing.shared_memory` module. 1004(Contributed by Davin Potts in :issue:`35813`.) 1005 1006On macOS, the *spawn* start method is now used by default. 1007(Contributed by Victor Stinner in :issue:`33725`.) 1008 1009 1010os 1011-- 1012 1013Added new function :func:`~os.add_dll_directory` on Windows for providing 1014additional search paths for native dependencies when importing extension 1015modules or loading DLLs using :mod:`ctypes`. 1016(Contributed by Steve Dower in :issue:`36085`.) 1017 1018A new :func:`os.memfd_create` function was added to wrap the 1019``memfd_create()`` syscall. 1020(Contributed by Zackery Spytz and Christian Heimes in :issue:`26836`.) 1021 1022On Windows, much of the manual logic for handling reparse points (including 1023symlinks and directory junctions) has been delegated to the operating system. 1024Specifically, :func:`os.stat` will now traverse anything supported by the 1025operating system, while :func:`os.lstat` will only open reparse points that 1026identify as "name surrogates" while others are opened as for :func:`os.stat`. 1027In all cases, :attr:`stat_result.st_mode` will only have ``S_IFLNK`` set for 1028symbolic links and not other kinds of reparse points. To identify other kinds 1029of reparse point, check the new :attr:`stat_result.st_reparse_tag` attribute. 1030 1031On Windows, :func:`os.readlink` is now able to read directory junctions. Note 1032that :func:`~os.path.islink` will return ``False`` for directory junctions, 1033and so code that checks ``islink`` first will continue to treat junctions as 1034directories, while code that handles errors from :func:`os.readlink` may now 1035treat junctions as links. 1036 1037(Contributed by Steve Dower in :issue:`37834`.) 1038 1039 1040os.path 1041------- 1042 1043:mod:`os.path` functions that return a boolean result like 1044:func:`~os.path.exists`, :func:`~os.path.lexists`, :func:`~os.path.isdir`, 1045:func:`~os.path.isfile`, :func:`~os.path.islink`, and :func:`~os.path.ismount` 1046now return ``False`` instead of raising :exc:`ValueError` or its subclasses 1047:exc:`UnicodeEncodeError` and :exc:`UnicodeDecodeError` for paths that contain 1048characters or bytes unrepresentable at the OS level. 1049(Contributed by Serhiy Storchaka in :issue:`33721`.) 1050 1051:func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` 1052environment variable and does not use :envvar:`HOME`, which is not normally set 1053for regular user accounts. 1054(Contributed by Anthony Sottile in :issue:`36264`.) 1055 1056:func:`~os.path.isdir` on Windows no longer returns ``True`` for a link to a 1057non-existent directory. 1058 1059:func:`~os.path.realpath` on Windows now resolves reparse points, including 1060symlinks and directory junctions. 1061 1062(Contributed by Steve Dower in :issue:`37834`.) 1063 1064 1065pathlib 1066------- 1067 1068:mod:`pathlib.Path` methods that return a boolean result like 1069:meth:`~pathlib.Path.exists()`, :meth:`~pathlib.Path.is_dir()`, 1070:meth:`~pathlib.Path.is_file()`, :meth:`~pathlib.Path.is_mount()`, 1071:meth:`~pathlib.Path.is_symlink()`, :meth:`~pathlib.Path.is_block_device()`, 1072:meth:`~pathlib.Path.is_char_device()`, :meth:`~pathlib.Path.is_fifo()`, 1073:meth:`~pathlib.Path.is_socket()` now return ``False`` instead of raising 1074:exc:`ValueError` or its subclass :exc:`UnicodeEncodeError` for paths that 1075contain characters unrepresentable at the OS level. 1076(Contributed by Serhiy Storchaka in :issue:`33721`.) 1077 1078Added :meth:`pathlib.Path.link_to()` which creates a hard link pointing 1079to a path. 1080(Contributed by Joannah Nanjekye in :issue:`26978`) 1081 1082 1083pickle 1084------ 1085 1086:mod:`pickle` extensions subclassing the C-optimized :class:`~pickle.Pickler` 1087can now override the pickling logic of functions and classes by defining the 1088special :meth:`~pickle.Pickler.reducer_override` method. 1089(Contributed by Pierre Glaser and Olivier Grisel in :issue:`35900`.) 1090 1091 1092plistlib 1093-------- 1094 1095Added new :class:`plistlib.UID` and enabled support for reading and writing 1096NSKeyedArchiver-encoded binary plists. 1097(Contributed by Jon Janzen in :issue:`26707`.) 1098 1099 1100pprint 1101------ 1102 1103The :mod:`pprint` module added a *sort_dicts* parameter to several functions. 1104By default, those functions continue to sort dictionaries before rendering or 1105printing. However, if *sort_dicts* is set to false, the dictionaries retain 1106the order that keys were inserted. This can be useful for comparison to JSON 1107inputs during debugging. 1108 1109In addition, there is a convenience new function, :func:`pprint.pp` that is 1110like :func:`pprint.pprint` but with *sort_dicts* defaulting to ``False``:: 1111 1112 >>> from pprint import pprint, pp 1113 >>> d = dict(source='input.txt', operation='filter', destination='output.txt') 1114 >>> pp(d, width=40) # Original order 1115 {'source': 'input.txt', 1116 'operation': 'filter', 1117 'destination': 'output.txt'} 1118 >>> pprint(d, width=40) # Keys sorted alphabetically 1119 {'destination': 'output.txt', 1120 'operation': 'filter', 1121 'source': 'input.txt'} 1122 1123(Contributed by Rémi Lapeyre in :issue:`30670`.) 1124 1125 1126py_compile 1127---------- 1128 1129:func:`py_compile.compile` now supports silent mode. 1130(Contributed by Joannah Nanjekye in :issue:`22640`.) 1131 1132 1133shlex 1134----- 1135 1136The new :func:`shlex.join` function acts as the inverse of :func:`shlex.split`. 1137(Contributed by Bo Bayles in :issue:`32102`.) 1138 1139 1140shutil 1141------ 1142 1143:func:`shutil.copytree` now accepts a new ``dirs_exist_ok`` keyword argument. 1144(Contributed by Josh Bronson in :issue:`20849`.) 1145 1146:func:`shutil.make_archive` now defaults to the modern pax (POSIX.1-2001) 1147format for new archives to improve portability and standards conformance, 1148inherited from the corresponding change to the :mod:`tarfile` module. 1149(Contributed by C.A.M. Gerlach in :issue:`30661`.) 1150 1151:func:`shutil.rmtree` on Windows now removes directory junctions without 1152recursively removing their contents first. 1153(Contributed by Steve Dower in :issue:`37834`.) 1154 1155 1156socket 1157------ 1158 1159Added :meth:`~socket.create_server()` and :meth:`~socket.has_dualstack_ipv6()` 1160convenience functions to automate the necessary tasks usually involved when 1161creating a server socket, including accepting both IPv4 and IPv6 connections 1162on the same socket. (Contributed by Giampaolo Rodolà in :issue:`17561`.) 1163 1164The :func:`socket.if_nameindex()`, :func:`socket.if_nametoindex()`, and 1165:func:`socket.if_indextoname()` functions have been implemented on Windows. 1166(Contributed by Zackery Spytz in :issue:`37007`.) 1167 1168 1169ssl 1170--- 1171 1172Added :attr:`~ssl.SSLContext.post_handshake_auth` to enable and 1173:meth:`~ssl.SSLSocket.verify_client_post_handshake` to initiate TLS 1.3 1174post-handshake authentication. 1175(Contributed by Christian Heimes in :issue:`34670`.) 1176 1177 1178statistics 1179---------- 1180 1181Added :func:`statistics.fmean` as a faster, floating point variant of 1182:func:`statistics.mean()`. (Contributed by Raymond Hettinger and 1183Steven D'Aprano in :issue:`35904`.) 1184 1185Added :func:`statistics.geometric_mean()` 1186(Contributed by Raymond Hettinger in :issue:`27181`.) 1187 1188Added :func:`statistics.multimode` that returns a list of the most 1189common values. (Contributed by Raymond Hettinger in :issue:`35892`.) 1190 1191Added :func:`statistics.quantiles` that divides data or a distribution 1192in to equiprobable intervals (e.g. quartiles, deciles, or percentiles). 1193(Contributed by Raymond Hettinger in :issue:`36546`.) 1194 1195Added :class:`statistics.NormalDist`, a tool for creating 1196and manipulating normal distributions of a random variable. 1197(Contributed by Raymond Hettinger in :issue:`36018`.) 1198 1199:: 1200 1201 >>> temperature_feb = NormalDist.from_samples([4, 12, -3, 2, 7, 14]) 1202 >>> temperature_feb.mean 1203 6.0 1204 >>> temperature_feb.stdev 1205 6.356099432828281 1206 1207 >>> temperature_feb.cdf(3) # Chance of being under 3 degrees 1208 0.3184678262814532 1209 >>> # Relative chance of being 7 degrees versus 10 degrees 1210 >>> temperature_feb.pdf(7) / temperature_feb.pdf(10) 1211 1.2039930378537762 1212 1213 >>> el_niño = NormalDist(4, 2.5) 1214 >>> temperature_feb += el_niño # Add in a climate effect 1215 >>> temperature_feb 1216 NormalDist(mu=10.0, sigma=6.830080526611674) 1217 1218 >>> temperature_feb * (9/5) + 32 # Convert to Fahrenheit 1219 NormalDist(mu=50.0, sigma=12.294144947901014) 1220 >>> temperature_feb.samples(3) # Generate random samples 1221 [7.672102882379219, 12.000027119750287, 4.647488369766392] 1222 1223 1224sys 1225--- 1226 1227Add new :func:`sys.unraisablehook` function which can be overridden to control 1228how "unraisable exceptions" are handled. It is called when an exception has 1229occurred but there is no way for Python to handle it. For example, when a 1230destructor raises an exception or during garbage collection 1231(:func:`gc.collect`). 1232(Contributed by Victor Stinner in :issue:`36829`.) 1233 1234 1235tarfile 1236------- 1237 1238The :mod:`tarfile` module now defaults to the modern pax (POSIX.1-2001) 1239format for new archives, instead of the previous GNU-specific one. 1240This improves cross-platform portability with a consistent encoding (UTF-8) 1241in a standardized and extensible format, and offers several other benefits. 1242(Contributed by C.A.M. Gerlach in :issue:`36268`.) 1243 1244 1245threading 1246--------- 1247 1248Add a new :func:`threading.excepthook` function which handles uncaught 1249:meth:`threading.Thread.run` exception. It can be overridden to control how 1250uncaught :meth:`threading.Thread.run` exceptions are handled. 1251(Contributed by Victor Stinner in :issue:`1230540`.) 1252 1253Add a new :func:`threading.get_native_id` function and 1254a :data:`~threading.Thread.native_id` 1255attribute to the :class:`threading.Thread` class. These return the native 1256integral Thread ID of the current thread assigned by the kernel. 1257This feature is only available on certain platforms, see 1258:func:`get_native_id <threading.get_native_id>` for more information. 1259(Contributed by Jake Tesler in :issue:`36084`.) 1260 1261 1262tokenize 1263-------- 1264 1265The :mod:`tokenize` module now implicitly emits a ``NEWLINE`` token when 1266provided with input that does not have a trailing new line. This behavior 1267now matches what the C tokenizer does internally. 1268(Contributed by Ammar Askar in :issue:`33899`.) 1269 1270 1271tkinter 1272------- 1273 1274Added methods :meth:`~tkinter.Spinbox.selection_from`, 1275:meth:`~tkinter.Spinbox.selection_present`, 1276:meth:`~tkinter.Spinbox.selection_range` and 1277:meth:`~tkinter.Spinbox.selection_to` 1278in the :class:`tkinter.Spinbox` class. 1279(Contributed by Juliette Monsel in :issue:`34829`.) 1280 1281Added method :meth:`~tkinter.Canvas.moveto` 1282in the :class:`tkinter.Canvas` class. 1283(Contributed by Juliette Monsel in :issue:`23831`.) 1284 1285The :class:`tkinter.PhotoImage` class now has 1286:meth:`~tkinter.PhotoImage.transparency_get` and 1287:meth:`~tkinter.PhotoImage.transparency_set` methods. (Contributed by 1288Zackery Spytz in :issue:`25451`.) 1289 1290 1291time 1292---- 1293 1294Added new clock :data:`~time.CLOCK_UPTIME_RAW` for macOS 10.12. 1295(Contributed by Joannah Nanjekye in :issue:`35702`.) 1296 1297 1298typing 1299------ 1300 1301The :mod:`typing` module incorporates several new features: 1302 1303* A dictionary type with per-key types. See :pep:`589` and 1304 :class:`typing.TypedDict`. 1305 TypedDict uses only string keys. By default, every key is required 1306 to be present. Specify "total=False" to allow keys to be optional:: 1307 1308 class Location(TypedDict, total=False): 1309 lat_long: tuple 1310 grid_square: str 1311 xy_coordinate: tuple 1312 1313* Literal types. See :pep:`586` and :class:`typing.Literal`. 1314 Literal types indicate that a parameter or return value 1315 is constrained to one or more specific literal values:: 1316 1317 def get_status(port: int) -> Literal['connected', 'disconnected']: 1318 ... 1319 1320* "Final" variables, functions, methods and classes. See :pep:`591`, 1321 :class:`typing.Final` and :func:`typing.final`. 1322 The final qualifier instructs a static type checker to restrict 1323 subclassing, overriding, or reassignment:: 1324 1325 pi: Final[float] = 3.1415926536 1326 1327* Protocol definitions. See :pep:`544`, :class:`typing.Protocol` and 1328 :func:`typing.runtime_checkable`. Simple ABCs like 1329 :class:`typing.SupportsInt` are now ``Protocol`` subclasses. 1330 1331* New protocol class :class:`typing.SupportsIndex`. 1332 1333* New functions :func:`typing.get_origin` and :func:`typing.get_args`. 1334 1335 1336unicodedata 1337----------- 1338 1339The :mod:`unicodedata` module has been upgraded to use the `Unicode 12.1.0 1340<http://blog.unicode.org/2019/05/unicode-12-1-en.html>`_ release. 1341 1342New function :func:`~unicodedata.is_normalized` can be used to verify a string 1343is in a specific normal form, often much faster than by actually normalizing 1344the string. (Contributed by Max Belanger, David Euresti, and Greg Price in 1345:issue:`32285` and :issue:`37966`). 1346 1347 1348unittest 1349-------- 1350 1351Added :class:`~unittest.mock.AsyncMock` to support an asynchronous version of 1352:class:`~unittest.mock.Mock`. Appropriate new assert functions for testing 1353have been added as well. 1354(Contributed by Lisa Roach in :issue:`26467`). 1355 1356Added :func:`~unittest.addModuleCleanup()` and 1357:meth:`~unittest.TestCase.addClassCleanup()` to unittest to support 1358cleanups for :func:`~unittest.setUpModule()` and 1359:meth:`~unittest.TestCase.setUpClass()`. 1360(Contributed by Lisa Roach in :issue:`24412`.) 1361 1362Several mock assert functions now also print a list of actual calls upon 1363failure. (Contributed by Petter Strandmark in :issue:`35047`.) 1364 1365:mod:`unittest` module gained support for coroutines to be used as test cases 1366with :class:`unittest.IsolatedAsyncioTestCase`. 1367(Contributed by Andrew Svetlov in :issue:`32972`.) 1368 1369Example:: 1370 1371 import unittest 1372 1373 1374 class TestRequest(unittest.IsolatedAsyncioTestCase): 1375 1376 async def asyncSetUp(self): 1377 self.connection = await AsyncConnection() 1378 1379 async def test_get(self): 1380 response = await self.connection.get("https://example.com") 1381 self.assertEqual(response.status_code, 200) 1382 1383 async def asyncTearDown(self): 1384 await self.connection.close() 1385 1386 1387 if __name__ == "__main__": 1388 unittest.main() 1389 1390 1391venv 1392---- 1393 1394:mod:`venv` now includes an ``Activate.ps1`` script on all platforms for 1395activating virtual environments under PowerShell Core 6.1. 1396(Contributed by Brett Cannon in :issue:`32718`.) 1397 1398 1399weakref 1400------- 1401 1402The proxy objects returned by :func:`weakref.proxy` now support the matrix 1403multiplication operators ``@`` and ``@=`` in addition to the other 1404numeric operators. (Contributed by Mark Dickinson in :issue:`36669`.) 1405 1406 1407xml 1408--- 1409 1410As mitigation against DTD and external entity retrieval, the 1411:mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 1412external entities by default. 1413(Contributed by Christian Heimes in :issue:`17239`.) 1414 1415The ``.find*()`` methods in the :mod:`xml.etree.ElementTree` module 1416support wildcard searches like ``{*}tag`` which ignores the namespace 1417and ``{namespace}*`` which returns all tags in the given namespace. 1418(Contributed by Stefan Behnel in :issue:`28238`.) 1419 1420The :mod:`xml.etree.ElementTree` module provides a new function 1421:func:`–xml.etree.ElementTree.canonicalize()` that implements C14N 2.0. 1422(Contributed by Stefan Behnel in :issue:`13611`.) 1423 1424The target object of :class:`xml.etree.ElementTree.XMLParser` can 1425receive namespace declaration events through the new callback methods 1426``start_ns()`` and ``end_ns()``. Additionally, the 1427:class:`xml.etree.ElementTree.TreeBuilder` target can be configured 1428to process events about comments and processing instructions to include 1429them in the generated tree. 1430(Contributed by Stefan Behnel in :issue:`36676` and :issue:`36673`.) 1431 1432 1433xmlrpc 1434------ 1435 1436:class:`xmlrpc.client.ServerProxy` now supports an optional *headers* keyword 1437argument for a sequence of HTTP headers to be sent with each request. Among 1438other things, this makes it possible to upgrade from default basic 1439authentication to faster session authentication. 1440(Contributed by Cédric Krier in :issue:`35153`.) 1441 1442 1443Optimizations 1444============= 1445 1446* The :mod:`subprocess` module can now use the :func:`os.posix_spawn` function 1447 in some cases for better performance. Currently, it is only used on macOS 1448 and Linux (using glibc 2.24 or newer) if all these conditions are met: 1449 1450 * *close_fds* is false; 1451 * *preexec_fn*, *pass_fds*, *cwd* and *start_new_session* parameters 1452 are not set; 1453 * the *executable* path contains a directory. 1454 1455 (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.) 1456 1457* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`, 1458 :func:`shutil.copytree` and :func:`shutil.move` use platform-specific 1459 "fast-copy" syscalls on Linux and macOS in order to copy the file 1460 more efficiently. 1461 "fast-copy" means that the copying operation occurs within the kernel, 1462 avoiding the use of userspace buffers in Python as in 1463 "``outfd.write(infd.read())``". 1464 On Windows :func:`shutil.copyfile` uses a bigger default buffer size (1 MiB 1465 instead of 16 KiB) and a :func:`memoryview`-based variant of 1466 :func:`shutil.copyfileobj` is used. 1467 The speedup for copying a 512 MiB file within the same partition is about 1468 +26% on Linux, +50% on macOS and +40% on Windows. Also, much less CPU cycles 1469 are consumed. 1470 See :ref:`shutil-platform-dependent-efficient-copy-operations` section. 1471 (Contributed by Giampaolo Rodolà in :issue:`33671`.) 1472 1473* :func:`shutil.copytree` uses :func:`os.scandir` function and all copy 1474 functions depending from it use cached :func:`os.stat` values. The speedup 1475 for copying a directory with 8000 files is around +9% on Linux, +20% on 1476 Windows and +30% on a Windows SMB share. Also the number of :func:`os.stat` 1477 syscalls is reduced by 38% making :func:`shutil.copytree` especially faster 1478 on network filesystems. (Contributed by Giampaolo Rodolà in :issue:`33695`.) 1479 1480* The default protocol in the :mod:`pickle` module is now Protocol 4, 1481 first introduced in Python 3.4. It offers better performance and smaller 1482 size compared to Protocol 3 available since Python 3.0. 1483 1484* Removed one ``Py_ssize_t`` member from ``PyGC_Head``. All GC tracked 1485 objects (e.g. tuple, list, dict) size is reduced 4 or 8 bytes. 1486 (Contributed by Inada Naoki in :issue:`33597`.) 1487 1488* :class:`uuid.UUID` now uses ``__slots__`` to reduce its memory footprint. 1489 (Contributed by Wouter Bolsterlee and Tal Einat in :issue:`30977`) 1490 1491* Improved performance of :func:`operator.itemgetter` by 33%. Optimized 1492 argument handling and added a fast path for the common case of a single 1493 non-negative integer index into a tuple (which is the typical use case in 1494 the standard library). (Contributed by Raymond Hettinger in 1495 :issue:`35664`.) 1496 1497* Sped-up field lookups in :func:`collections.namedtuple`. They are now more 1498 than two times faster, making them the fastest form of instance variable 1499 lookup in Python. (Contributed by Raymond Hettinger, Pablo Galindo, and 1500 Joe Jevnik, Serhiy Storchaka in :issue:`32492`.) 1501 1502* The :class:`list` constructor does not overallocate the internal item buffer 1503 if the input iterable has a known length (the input implements ``__len__``). 1504 This makes the created list 12% smaller on average. (Contributed by 1505 Raymond Hettinger and Pablo Galindo in :issue:`33234`.) 1506 1507* Doubled the speed of class variable writes. When a non-dunder attribute 1508 was updated, there was an unnecessary call to update slots. 1509 (Contributed by Stefan Behnel, Pablo Galindo Salgado, Raymond Hettinger, 1510 Neil Schemenauer, and Serhiy Storchaka in :issue:`36012`.) 1511 1512* Reduced an overhead of converting arguments passed to many builtin functions 1513 and methods. This sped up calling some simple builtin functions and 1514 methods up to 20--50%. (Contributed by Serhiy Storchaka in :issue:`23867`, 1515 :issue:`35582` and :issue:`36127`.) 1516 1517* ``LOAD_GLOBAL`` instruction now uses new "per opcode cache" mechanism. 1518 It is about 40% faster now. (Contributed by Yury Selivanov and Inada Naoki in 1519 :issue:`26219`.) 1520 1521 1522Build and C API Changes 1523======================= 1524 1525* Default :data:`sys.abiflags` became an empty string: the ``m`` flag for 1526 pymalloc became useless (builds with and without pymalloc are ABI compatible) 1527 and so has been removed. (Contributed by Victor Stinner in :issue:`36707`.) 1528 1529 Example of changes: 1530 1531 * Only ``python3.8`` program is installed, ``python3.8m`` program is gone. 1532 * Only ``python3.8-config`` script is installed, ``python3.8m-config`` script 1533 is gone. 1534 * The ``m`` flag has been removed from the suffix of dynamic library 1535 filenames: extension modules in the standard library as well as those 1536 produced and installed by third-party packages, like those downloaded from 1537 PyPI. On Linux, for example, the Python 3.7 suffix 1538 ``.cpython-37m-x86_64-linux-gnu.so`` became 1539 ``.cpython-38-x86_64-linux-gnu.so`` in Python 3.8. 1540 1541* The header files have been reorganized to better separate the different kinds 1542 of APIs: 1543 1544 * ``Include/*.h`` should be the portable public stable C API. 1545 * ``Include/cpython/*.h`` should be the unstable C API specific to CPython; 1546 public API, with some private API prefixed by ``_Py`` or ``_PY``. 1547 * ``Include/internal/*.h`` is the private internal C API very specific to 1548 CPython. This API comes with no backward compatibility warranty and should 1549 not be used outside CPython. It is only exposed for very specific needs 1550 like debuggers and profiles which has to access to CPython internals 1551 without calling functions. This API is now installed by ``make install``. 1552 1553 (Contributed by Victor Stinner in :issue:`35134` and :issue:`35081`, 1554 work initiated by Eric Snow in Python 3.7.) 1555 1556* Some macros have been converted to static inline functions: parameter types 1557 and return type are well defined, they don't have issues specific to macros, 1558 variables have a local scopes. Examples: 1559 1560 * :c:func:`Py_INCREF`, :c:func:`Py_DECREF` 1561 * :c:func:`Py_XINCREF`, :c:func:`Py_XDECREF` 1562 * :c:func:`PyObject_INIT`, :c:func:`PyObject_INIT_VAR` 1563 * Private functions: :c:func:`_PyObject_GC_TRACK`, 1564 :c:func:`_PyObject_GC_UNTRACK`, :c:func:`_Py_Dealloc` 1565 1566 (Contributed by Victor Stinner in :issue:`35059`.) 1567 1568* The :c:func:`PyByteArray_Init` and :c:func:`PyByteArray_Fini` functions have 1569 been removed. They did nothing since Python 2.7.4 and Python 3.2.0, were 1570 excluded from the limited API (stable ABI), and were not documented. 1571 (Contributed by Victor Stinner in :issue:`35713`.) 1572 1573* The result of :c:func:`PyExceptionClass_Name` is now of type 1574 ``const char *`` rather of ``char *``. 1575 (Contributed by Serhiy Storchaka in :issue:`33818`.) 1576 1577* The duality of ``Modules/Setup.dist`` and ``Modules/Setup`` has been 1578 removed. Previously, when updating the CPython source tree, one had 1579 to manually copy ``Modules/Setup.dist`` (inside the source tree) to 1580 ``Modules/Setup`` (inside the build tree) in order to reflect any changes 1581 upstream. This was of a small benefit to packagers at the expense of 1582 a frequent annoyance to developers following CPython development, as 1583 forgetting to copy the file could produce build failures. 1584 1585 Now the build system always reads from ``Modules/Setup`` inside the source 1586 tree. People who want to customize that file are encouraged to maintain 1587 their changes in a git fork of CPython or as patch files, as they would do 1588 for any other change to the source tree. 1589 1590 (Contributed by Antoine Pitrou in :issue:`32430`.) 1591 1592* Functions that convert Python number to C integer like 1593 :c:func:`PyLong_AsLong` and argument parsing functions like 1594 :c:func:`PyArg_ParseTuple` with integer converting format units like ``'i'`` 1595 will now use the :meth:`~object.__index__` special method instead of 1596 :meth:`~object.__int__`, if available. The deprecation warning will be 1597 emitted for objects with the ``__int__()`` method but without the 1598 ``__index__()`` method (like :class:`~decimal.Decimal` and 1599 :class:`~fractions.Fraction`). :c:func:`PyNumber_Check` will now return 1600 ``1`` for objects implementing ``__index__()``. 1601 :c:func:`PyNumber_Long`, :c:func:`PyNumber_Float` and 1602 :c:func:`PyFloat_AsDouble` also now use the ``__index__()`` method if 1603 available. 1604 (Contributed by Serhiy Storchaka in :issue:`36048` and :issue:`20092`.) 1605 1606* Heap-allocated type objects will now increase their reference count 1607 in :c:func:`PyObject_Init` (and its parallel macro ``PyObject_INIT``) 1608 instead of in :c:func:`PyType_GenericAlloc`. Types that modify instance 1609 allocation or deallocation may need to be adjusted. 1610 (Contributed by Eddie Elizondo in :issue:`35810`.) 1611 1612* The new function :c:func:`PyCode_NewWithPosOnlyArgs` allows to create 1613 code objects like :c:func:`PyCode_New`, but with an extra *posonlyargcount* 1614 parameter for indicating the number of positional-only arguments. 1615 (Contributed by Pablo Galindo in :issue:`37221`.) 1616 1617* :c:func:`Py_SetPath` now sets :data:`sys.executable` to the program full 1618 path (:c:func:`Py_GetProgramFullPath`) rather than to the program name 1619 (:c:func:`Py_GetProgramName`). 1620 (Contributed by Victor Stinner in :issue:`38234`.) 1621 1622 1623Deprecated 1624========== 1625 1626* The distutils ``bdist_wininst`` command is now deprecated, use 1627 ``bdist_wheel`` (wheel packages) instead. 1628 (Contributed by Victor Stinner in :issue:`37481`.) 1629 1630* Deprecated methods ``getchildren()`` and ``getiterator()`` in 1631 the :mod:`~xml.etree.ElementTree` module now emit a 1632 :exc:`DeprecationWarning` instead of :exc:`PendingDeprecationWarning`. 1633 They will be removed in Python 3.9. 1634 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1635 1636* Passing an object that is not an instance of 1637 :class:`concurrent.futures.ThreadPoolExecutor` to 1638 :meth:`loop.set_default_executor() <asyncio.loop.set_default_executor>` is 1639 deprecated and will be prohibited in Python 3.9. 1640 (Contributed by Elvis Pranskevichus in :issue:`34075`.) 1641 1642* The :meth:`__getitem__` methods of :class:`xml.dom.pulldom.DOMEventStream`, 1643 :class:`wsgiref.util.FileWrapper` and :class:`fileinput.FileInput` have been 1644 deprecated. 1645 1646 Implementations of these methods have been ignoring their *index* parameter, 1647 and returning the next item instead. 1648 (Contributed by Berker Peksag in :issue:`9372`.) 1649 1650* The :class:`typing.NamedTuple` class has deprecated the ``_field_types`` 1651 attribute in favor of the ``__annotations__`` attribute which has the same 1652 information. (Contributed by Raymond Hettinger in :issue:`36320`.) 1653 1654* :mod:`ast` classes ``Num``, ``Str``, ``Bytes``, ``NameConstant`` and 1655 ``Ellipsis`` are considered deprecated and will be removed in future Python 1656 versions. :class:`~ast.Constant` should be used instead. 1657 (Contributed by Serhiy Storchaka in :issue:`32892`.) 1658 1659* :class:`ast.NodeVisitor` methods ``visit_Num()``, ``visit_Str()``, 1660 ``visit_Bytes()``, ``visit_NameConstant()`` and ``visit_Ellipsis()`` are 1661 deprecated now and will not be called in future Python versions. 1662 Add the :meth:`~ast.NodeVisitor.visit_Constant` method to handle all 1663 constant nodes. 1664 (Contributed by Serhiy Storchaka in :issue:`36917`.) 1665 1666* The :func:`asyncio.coroutine` :term:`decorator` is deprecated and will be 1667 removed in version 3.10. Instead of ``@asyncio.coroutine``, use 1668 :keyword:`async def` instead. 1669 (Contributed by Andrew Svetlov in :issue:`36921`.) 1670 1671* In :mod:`asyncio`, the explicit passing of a *loop* argument has been 1672 deprecated and will be removed in version 3.10 for the following: 1673 :func:`asyncio.sleep`, :func:`asyncio.gather`, :func:`asyncio.shield`, 1674 :func:`asyncio.wait_for`, :func:`asyncio.wait`, :func:`asyncio.as_completed`, 1675 :class:`asyncio.Task`, :class:`asyncio.Lock`, :class:`asyncio.Event`, 1676 :class:`asyncio.Condition`, :class:`asyncio.Semaphore`, 1677 :class:`asyncio.BoundedSemaphore`, :class:`asyncio.Queue`, 1678 :func:`asyncio.create_subprocess_exec`, and 1679 :func:`asyncio.create_subprocess_shell`. 1680 1681* The explicit passing of coroutine objects to :func:`asyncio.wait` has been 1682 deprecated and will be removed in version 3.11. 1683 (Contributed by Yury Selivanov in :issue:`34790`.) 1684 1685* The following functions and methods are deprecated in the :mod:`gettext` 1686 module: :func:`~gettext.lgettext`, :func:`~gettext.ldgettext`, 1687 :func:`~gettext.lngettext` and :func:`~gettext.ldngettext`. 1688 They return encoded bytes, and it's possible that you will get unexpected 1689 Unicode-related exceptions if there are encoding problems with the 1690 translated strings. It's much better to use alternatives which return 1691 Unicode strings in Python 3. These functions have been broken for a long time. 1692 1693 Function :func:`~gettext.bind_textdomain_codeset`, methods 1694 :meth:`~gettext.NullTranslations.output_charset` and 1695 :meth:`~gettext.NullTranslations.set_output_charset`, and the *codeset* 1696 parameter of functions :func:`~gettext.translation` and 1697 :func:`~gettext.install` are also deprecated, since they are only used for 1698 for the ``l*gettext()`` functions. 1699 (Contributed by Serhiy Storchaka in :issue:`33710`.) 1700 1701* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` 1702 has been deprecated. 1703 (Contributed by Dong-hee Na in :issue:`35283`.) 1704 1705* Many builtin and extension functions that take integer arguments will 1706 now emit a deprecation warning for :class:`~decimal.Decimal`\ s, 1707 :class:`~fractions.Fraction`\ s and any other objects that can be converted 1708 to integers only with a loss (e.g. that have the :meth:`~object.__int__` 1709 method but do not have the :meth:`~object.__index__` method). In future 1710 version they will be errors. 1711 (Contributed by Serhiy Storchaka in :issue:`36048`.) 1712 1713* Deprecated passing the following arguments as keyword arguments: 1714 1715 - *func* in :func:`functools.partialmethod`, :func:`weakref.finalize`, 1716 :meth:`profile.Profile.runcall`, :meth:`cProfile.Profile.runcall`, 1717 :meth:`bdb.Bdb.runcall`, :meth:`trace.Trace.runfunc` and 1718 :func:`curses.wrapper`. 1719 - *function* in :meth:`unittest.TestCase.addCleanup`. 1720 - *fn* in the :meth:`~concurrent.futures.Executor.submit` method of 1721 :class:`concurrent.futures.ThreadPoolExecutor` and 1722 :class:`concurrent.futures.ProcessPoolExecutor`. 1723 - *callback* in :meth:`contextlib.ExitStack.callback`, 1724 :meth:`contextlib.AsyncExitStack.callback` and 1725 :meth:`contextlib.AsyncExitStack.push_async_callback`. 1726 - *c* and *typeid* in the :meth:`~multiprocessing.managers.Server.create` 1727 method of :class:`multiprocessing.managers.Server` and 1728 :class:`multiprocessing.managers.SharedMemoryServer`. 1729 - *obj* in :func:`weakref.finalize`. 1730 1731 In future releases of Python, they will be :ref:`positional-only 1732 <positional-only_parameter>`. 1733 (Contributed by Serhiy Storchaka in :issue:`36492`.) 1734 1735 1736API and Feature Removals 1737======================== 1738 1739The following features and APIs have been removed from Python 3.8: 1740 1741* Starting with Python 3.3, importing ABCs from :mod:`collections` was 1742 deprecated, and importing should be done from :mod:`collections.abc`. Being 1743 able to import from collections was marked for removal in 3.8, but has been 1744 delayed to 3.9. (See :issue:`36952`.) 1745 1746* The :mod:`macpath` module, deprecated in Python 3.7, has been removed. 1747 (Contributed by Victor Stinner in :issue:`35471`.) 1748 1749* The function :func:`platform.popen` has been removed, after having been 1750 deprecated since Python 3.3: use :func:`os.popen` instead. 1751 (Contributed by Victor Stinner in :issue:`35345`.) 1752 1753* The function :func:`time.clock` has been removed, after having been 1754 deprecated since Python 3.3: use :func:`time.perf_counter` or 1755 :func:`time.process_time` instead, depending 1756 on your requirements, to have well-defined behavior. 1757 (Contributed by Matthias Bussonnier in :issue:`36895`.) 1758 1759* The ``pyvenv`` script has been removed in favor of ``python3.8 -m venv`` 1760 to help eliminate confusion as to what Python interpreter the ``pyvenv`` 1761 script is tied to. (Contributed by Brett Cannon in :issue:`25427`.) 1762 1763* ``parse_qs``, ``parse_qsl``, and ``escape`` are removed from the :mod:`cgi` 1764 module. They are deprecated in Python 3.2 or older. They should be imported 1765 from the ``urllib.parse`` and ``html`` modules instead. 1766 1767* ``filemode`` function is removed from the :mod:`tarfile` module. 1768 It is not documented and deprecated since Python 3.3. 1769 1770* The :class:`~xml.etree.ElementTree.XMLParser` constructor no longer accepts 1771 the *html* argument. It never had an effect and was deprecated in Python 3.4. 1772 All other parameters are now :ref:`keyword-only <keyword-only_parameter>`. 1773 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1774 1775* Removed the ``doctype()`` method of :class:`~xml.etree.ElementTree.XMLParser`. 1776 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1777 1778* "unicode_internal" codec is removed. 1779 (Contributed by Inada Naoki in :issue:`36297`.) 1780 1781* The ``Cache`` and ``Statement`` objects of the :mod:`sqlite3` module are not 1782 exposed to the user. 1783 (Contributed by Aviv Palivoda in :issue:`30262`.) 1784 1785* The ``bufsize`` keyword argument of :func:`fileinput.input` and 1786 :func:`fileinput.FileInput` which was ignored and deprecated since Python 3.6 1787 has been removed. :issue:`36952` (Contributed by Matthias Bussonnier.) 1788 1789* The functions :func:`sys.set_coroutine_wrapper` and 1790 :func:`sys.get_coroutine_wrapper` deprecated in Python 3.7 have been removed; 1791 :issue:`36933` (Contributed by Matthias Bussonnier.) 1792 1793 1794Porting to Python 3.8 1795===================== 1796 1797This section lists previously described changes and other bugfixes 1798that may require changes to your code. 1799 1800 1801Changes in Python behavior 1802-------------------------- 1803 1804* Yield expressions (both ``yield`` and ``yield from`` clauses) are now disallowed 1805 in comprehensions and generator expressions (aside from the iterable expression 1806 in the leftmost :keyword:`!for` clause). 1807 (Contributed by Serhiy Storchaka in :issue:`10544`.) 1808 1809* The compiler now produces a :exc:`SyntaxWarning` when identity checks 1810 (``is`` and ``is not``) are used with certain types of literals 1811 (e.g. strings, numbers). These can often work by accident in CPython, 1812 but are not guaranteed by the language spec. The warning advises users 1813 to use equality tests (``==`` and ``!=``) instead. 1814 (Contributed by Serhiy Storchaka in :issue:`34850`.) 1815 1816* The CPython interpreter can swallow exceptions in some circumstances. 1817 In Python 3.8 this happens in fewer cases. In particular, exceptions 1818 raised when getting the attribute from the type dictionary are no longer 1819 ignored. (Contributed by Serhiy Storchaka in :issue:`35459`.) 1820 1821* Removed ``__str__`` implementations from builtin types :class:`bool`, 1822 :class:`int`, :class:`float`, :class:`complex` and few classes from 1823 the standard library. They now inherit ``__str__()`` from :class:`object`. 1824 As result, defining the ``__repr__()`` method in the subclass of these 1825 classes will affect their string representation. 1826 (Contributed by Serhiy Storchaka in :issue:`36793`.) 1827 1828* On AIX, :attr:`sys.platform` doesn't contain the major version anymore. 1829 It is always ``'aix'``, instead of ``'aix3'`` .. ``'aix7'``. Since 1830 older Python versions include the version number, so it is recommended to 1831 always use ``sys.platform.startswith('aix')``. 1832 (Contributed by M. Felt in :issue:`36588`.) 1833 1834* :c:func:`PyEval_AcquireLock` and :c:func:`PyEval_AcquireThread` now 1835 terminate the current thread if called while the interpreter is 1836 finalizing, making them consistent with :c:func:`PyEval_RestoreThread`, 1837 :c:func:`Py_END_ALLOW_THREADS`, and :c:func:`PyGILState_Ensure`. If this 1838 behavior is not desired, guard the call by checking :c:func:`_Py_IsFinalizing` 1839 or :c:func:`sys.is_finalizing`. 1840 (Contributed by Joannah Nanjekye in :issue:`36475`.) 1841 1842 1843Changes in the Python API 1844------------------------- 1845 1846* The :func:`os.getcwdb` function now uses the UTF-8 encoding on Windows, 1847 rather than the ANSI code page: see :pep:`529` for the rationale. The 1848 function is no longer deprecated on Windows. 1849 (Contributed by Victor Stinner in :issue:`37412`.) 1850 1851* :class:`subprocess.Popen` can now use :func:`os.posix_spawn` in some cases 1852 for better performance. On Windows Subsystem for Linux and QEMU User 1853 Emulation, the :class:`Popen` constructor using :func:`os.posix_spawn` no longer raises an 1854 exception on errors like "missing program". Instead the child process fails with a 1855 non-zero :attr:`~Popen.returncode`. 1856 (Contributed by Joannah Nanjekye and Victor Stinner in :issue:`35537`.) 1857 1858* The *preexec_fn* argument of * :class:`subprocess.Popen` is no longer 1859 compatible with subinterpreters. The use of the parameter in a 1860 subinterpreter now raises :exc:`RuntimeError`. 1861 (Contributed by Eric Snow in :issue:`34651`, modified by Christian Heimes 1862 in :issue:`37951`.) 1863 1864* The :meth:`imap.IMAP4.logout` method no longer silently ignores arbitrary 1865 exceptions. 1866 (Contributed by Victor Stinner in :issue:`36348`.) 1867 1868* The function :func:`platform.popen` has been removed, after having been deprecated since 1869 Python 3.3: use :func:`os.popen` instead. 1870 (Contributed by Victor Stinner in :issue:`35345`.) 1871 1872* The :func:`statistics.mode` function no longer raises an exception 1873 when given multimodal data. Instead, it returns the first mode 1874 encountered in the input data. (Contributed by Raymond Hettinger 1875 in :issue:`35892`.) 1876 1877* The :meth:`~tkinter.ttk.Treeview.selection` method of the 1878 :class:`tkinter.ttk.Treeview` class no longer takes arguments. Using it with 1879 arguments for changing the selection was deprecated in Python 3.6. Use 1880 specialized methods like :meth:`~tkinter.ttk.Treeview.selection_set` for 1881 changing the selection. (Contributed by Serhiy Storchaka in :issue:`31508`.) 1882 1883* The :meth:`writexml`, :meth:`toxml` and :meth:`toprettyxml` methods of 1884 :mod:`xml.dom.minidom`, and the :meth:`write` method of :mod:`xml.etree`, 1885 now preserve the attribute order specified by the user. 1886 (Contributed by Diego Rojas and Raymond Hettinger in :issue:`34160`.) 1887 1888* A :mod:`dbm.dumb` database opened with flags ``'r'`` is now read-only. 1889 :func:`dbm.dumb.open` with flags ``'r'`` and ``'w'`` no longer creates 1890 a database if it does not exist. 1891 (Contributed by Serhiy Storchaka in :issue:`32749`.) 1892 1893* The ``doctype()`` method defined in a subclass of 1894 :class:`~xml.etree.ElementTree.XMLParser` will no longer be called and will 1895 emit a :exc:`RuntimeWarning` instead of a :exc:`DeprecationWarning`. 1896 Define the :meth:`doctype() <xml.etree.ElementTree.TreeBuilder.doctype>` 1897 method on a target for handling an XML doctype declaration. 1898 (Contributed by Serhiy Storchaka in :issue:`29209`.) 1899 1900* A :exc:`RuntimeError` is now raised when the custom metaclass doesn't 1901 provide the ``__classcell__`` entry in the namespace passed to 1902 ``type.__new__``. A :exc:`DeprecationWarning` was emitted in Python 1903 3.6--3.7. (Contributed by Serhiy Storchaka in :issue:`23722`.) 1904 1905* The :class:`cProfile.Profile` class can now be used as a context 1906 manager. (Contributed by Scott Sanderson in :issue:`29235`.) 1907 1908* :func:`shutil.copyfile`, :func:`shutil.copy`, :func:`shutil.copy2`, 1909 :func:`shutil.copytree` and :func:`shutil.move` use platform-specific 1910 "fast-copy" syscalls (see 1911 :ref:`shutil-platform-dependent-efficient-copy-operations` section). 1912 1913* :func:`shutil.copyfile` default buffer size on Windows was changed from 1914 16 KiB to 1 MiB. 1915 1916* The ``PyGC_Head`` struct has changed completely. All code that touched the 1917 struct member should be rewritten. (See :issue:`33597`.) 1918 1919* The :c:type:`PyInterpreterState` struct has been moved into the "internal" 1920 header files (specifically Include/internal/pycore_pystate.h). An 1921 opaque ``PyInterpreterState`` is still available as part of the public 1922 API (and stable ABI). The docs indicate that none of the struct's 1923 fields are public, so we hope no one has been using them. However, 1924 if you do rely on one or more of those private fields and have no 1925 alternative then please open a BPO issue. We'll work on helping 1926 you adjust (possibly including adding accessor functions to the 1927 public API). (See :issue:`35886`.) 1928 1929* The :meth:`mmap.flush() <mmap.mmap.flush>` method now returns ``None`` on 1930 success and raises an exception on error under all platforms. Previously, 1931 its behavior was platform-dependent: a nonzero value was returned on success; 1932 zero was returned on error under Windows. A zero value was returned on 1933 success; an exception was raised on error under Unix. 1934 (Contributed by Berker Peksag in :issue:`2122`.) 1935 1936* :mod:`xml.dom.minidom` and :mod:`xml.sax` modules no longer process 1937 external entities by default. 1938 (Contributed by Christian Heimes in :issue:`17239`.) 1939 1940* Deleting a key from a read-only :mod:`dbm` database (:mod:`dbm.dumb`, 1941 :mod:`dbm.gnu` or :mod:`dbm.ndbm`) raises :attr:`error` (:exc:`dbm.dumb.error`, 1942 :exc:`dbm.gnu.error` or :exc:`dbm.ndbm.error`) instead of :exc:`KeyError`. 1943 (Contributed by Xiang Zhang in :issue:`33106`.) 1944 1945* Simplified AST for literals. All constants will be represented as 1946 :class:`ast.Constant` instances. Instantiating old classes ``Num``, 1947 ``Str``, ``Bytes``, ``NameConstant`` and ``Ellipsis`` will return 1948 an instance of ``Constant``. 1949 (Contributed by Serhiy Storchaka in :issue:`32892`.) 1950 1951* :func:`~os.path.expanduser` on Windows now prefers the :envvar:`USERPROFILE` 1952 environment variable and does not use :envvar:`HOME`, which is not normally 1953 set for regular user accounts. 1954 (Contributed by Anthony Sottile in :issue:`36264`.) 1955 1956* The exception :class:`asyncio.CancelledError` now inherits from 1957 :class:`BaseException` rather than :class:`Exception` and no longer inherits 1958 from :class:`concurrent.futures.CancelledError`. 1959 (Contributed by Yury Selivanov in :issue:`32528`.) 1960 1961* The function :func:`asyncio.wait_for` now correctly waits for cancellation 1962 when using an instance of :class:`asyncio.Task`. Previously, upon reaching 1963 *timeout*, it was cancelled and immediately returned. 1964 (Contributed by Elvis Pranskevichus in :issue:`32751`.) 1965 1966* The function :func:`asyncio.BaseTransport.get_extra_info` now returns a safe 1967 to use socket object when 'socket' is passed to the *name* parameter. 1968 (Contributed by Yury Selivanov in :issue:`37027`.) 1969 1970* :class:`asyncio.BufferedProtocol` has graduated to the stable API. 1971 1972.. _bpo-36085-whatsnew: 1973 1974* DLL dependencies for extension modules and DLLs loaded with :mod:`ctypes` on 1975 Windows are now resolved more securely. Only the system paths, the directory 1976 containing the DLL or PYD file, and directories added with 1977 :func:`~os.add_dll_directory` are searched for load-time dependencies. 1978 Specifically, :envvar:`PATH` and the current working directory are no longer 1979 used, and modifications to these will no longer have any effect on normal DLL 1980 resolution. If your application relies on these mechanisms, you should check 1981 for :func:`~os.add_dll_directory` and if it exists, use it to add your DLLs 1982 directory while loading your library. Note that Windows 7 users will need to 1983 ensure that Windows Update KB2533623 has been installed (this is also verified 1984 by the installer). 1985 (Contributed by Steve Dower in :issue:`36085`.) 1986 1987* The header files and functions related to pgen have been removed after its 1988 replacement by a pure Python implementation. (Contributed by Pablo Galindo 1989 in :issue:`36623`.) 1990 1991* :class:`types.CodeType` has a new parameter in the second position of the 1992 constructor (*posonlyargcount*) to support positional-only arguments defined 1993 in :pep:`570`. The first argument (*argcount*) now represents the total 1994 number of positional arguments (including positional-only arguments). The new 1995 ``replace()`` method of :class:`types.CodeType` can be used to make the code 1996 future-proof. 1997 1998 1999Changes in the C API 2000-------------------- 2001 2002* The :c:type:`PyCompilerFlags` structure got a new *cf_feature_version* 2003 field. It should be initialized to ``PY_MINOR_VERSION``. The field is ignored 2004 by default, and is used if and only if ``PyCF_ONLY_AST`` flag is set in 2005 *cf_flags*. 2006 (Contributed by Guido van Rossum in :issue:`35766`.) 2007 2008* The :c:func:`PyEval_ReInitThreads` function has been removed from the C API. 2009 It should not be called explicitly: use :c:func:`PyOS_AfterFork_Child` 2010 instead. 2011 (Contributed by Victor Stinner in :issue:`36728`.) 2012 2013* On Unix, C extensions are no longer linked to libpython except on Android 2014 and Cygwin. When Python is embedded, ``libpython`` must not be loaded with 2015 ``RTLD_LOCAL``, but ``RTLD_GLOBAL`` instead. Previously, using 2016 ``RTLD_LOCAL``, it was already not possible to load C extensions which 2017 were not linked to ``libpython``, like C extensions of the standard 2018 library built by the ``*shared*`` section of ``Modules/Setup``. 2019 (Contributed by Victor Stinner in :issue:`21536`.) 2020 2021* Use of ``#`` variants of formats in parsing or building value (e.g. 2022 :c:func:`PyArg_ParseTuple`, :c:func:`Py_BuildValue`, :c:func:`PyObject_CallFunction`, 2023 etc.) without ``PY_SSIZE_T_CLEAN`` defined raises ``DeprecationWarning`` now. 2024 It will be removed in 3.10 or 4.0. Read :ref:`arg-parsing` for detail. 2025 (Contributed by Inada Naoki in :issue:`36381`.) 2026 2027* Instances of heap-allocated types (such as those created with 2028 :c:func:`PyType_FromSpec`) hold a reference to their type object. 2029 Increasing the reference count of these type objects has been moved from 2030 :c:func:`PyType_GenericAlloc` to the more low-level functions, 2031 :c:func:`PyObject_Init` and :c:func:`PyObject_INIT`. 2032 This makes types created through :c:func:`PyType_FromSpec` behave like 2033 other classes in managed code. 2034 2035 Statically allocated types are not affected. 2036 2037 For the vast majority of cases, there should be no side effect. 2038 However, types that manually increase the reference count after allocating 2039 an instance (perhaps to work around the bug) may now become immortal. 2040 To avoid this, these classes need to call Py_DECREF on the type object 2041 during instance deallocation. 2042 2043 To correctly port these types into 3.8, please apply the following 2044 changes: 2045 2046 * Remove :c:macro:`Py_INCREF` on the type object after allocating an 2047 instance - if any. 2048 This may happen after calling :c:func:`PyObject_New`, 2049 :c:func:`PyObject_NewVar`, :c:func:`PyObject_GC_New`, 2050 :c:func:`PyObject_GC_NewVar`, or any other custom allocator that uses 2051 :c:func:`PyObject_Init` or :c:func:`PyObject_INIT`. 2052 2053 Example: 2054 2055 .. code-block:: c 2056 2057 static foo_struct * 2058 foo_new(PyObject *type) { 2059 foo_struct *foo = PyObject_GC_New(foo_struct, (PyTypeObject *) type); 2060 if (foo == NULL) 2061 return NULL; 2062 #if PY_VERSION_HEX < 0x03080000 2063 // Workaround for Python issue 35810; no longer necessary in Python 3.8 2064 PY_INCREF(type) 2065 #endif 2066 return foo; 2067 } 2068 2069 * Ensure that all custom ``tp_dealloc`` functions of heap-allocated types 2070 decrease the type's reference count. 2071 2072 Example: 2073 2074 .. code-block:: c 2075 2076 static void 2077 foo_dealloc(foo_struct *instance) { 2078 PyObject *type = Py_TYPE(instance); 2079 PyObject_GC_Del(instance); 2080 #if PY_VERSION_HEX >= 0x03080000 2081 // This was not needed before Python 3.8 (Python issue 35810) 2082 Py_DECREF(type); 2083 #endif 2084 } 2085 2086 (Contributed by Eddie Elizondo in :issue:`35810`.) 2087 2088* The :c:macro:`Py_DEPRECATED()` macro has been implemented for MSVC. 2089 The macro now must be placed before the symbol name. 2090 2091 Example: 2092 2093 .. code-block:: c 2094 2095 Py_DEPRECATED(3.8) PyAPI_FUNC(int) Py_OldFunction(void); 2096 2097 (Contributed by Zackery Spytz in :issue:`33407`.) 2098 2099* The interpreter does not pretend to support binary compatibility of 2100 extension types across feature releases, anymore. A :c:type:`PyTypeObject` 2101 exported by a third-party extension module is supposed to have all the 2102 slots expected in the current Python version, including 2103 :c:member:`~PyTypeObject.tp_finalize` (:const:`Py_TPFLAGS_HAVE_FINALIZE` 2104 is not checked anymore before reading :c:member:`~PyTypeObject.tp_finalize`). 2105 2106 (Contributed by Antoine Pitrou in :issue:`32388`.) 2107 2108* The :c:func:`PyCode_New` has a new parameter in the second position (*posonlyargcount*) 2109 to support :pep:`570`, indicating the number of positional-only arguments. 2110 2111* The functions :c:func:`PyNode_AddChild` and :c:func:`PyParser_AddToken` now accept 2112 two additional ``int`` arguments *end_lineno* and *end_col_offset*. 2113 2114* The :file:`libpython38.a` file to allow MinGW tools to link directly against 2115 :file:`python38.dll` is no longer included in the regular Windows distribution. 2116 If you require this file, it may be generated with the ``gendef`` and 2117 ``dlltool`` tools, which are part of the MinGW binutils package: 2118 2119 .. code-block:: shell 2120 2121 gendef - python38.dll > tmp.def 2122 dlltool --dllname python38.dll --def tmp.def --output-lib libpython38.a 2123 2124 The location of an installed :file:`pythonXY.dll` will depend on the 2125 installation options and the version and language of Windows. See 2126 :ref:`using-on-windows` for more information. The resulting library should be 2127 placed in the same directory as :file:`pythonXY.lib`, which is generally the 2128 :file:`libs` directory under your Python installation. 2129 2130 (Contributed by Steve Dower in :issue:`37351`.) 2131 2132 2133CPython bytecode changes 2134------------------------ 2135 2136* The interpreter loop has been simplified by moving the logic of unrolling 2137 the stack of blocks into the compiler. The compiler emits now explicit 2138 instructions for adjusting the stack of values and calling the 2139 cleaning-up code for :keyword:`break`, :keyword:`continue` and 2140 :keyword:`return`. 2141 2142 Removed opcodes :opcode:`BREAK_LOOP`, :opcode:`CONTINUE_LOOP`, 2143 :opcode:`SETUP_LOOP` and :opcode:`SETUP_EXCEPT`. Added new opcodes 2144 :opcode:`ROT_FOUR`, :opcode:`BEGIN_FINALLY`, :opcode:`CALL_FINALLY` and 2145 :opcode:`POP_FINALLY`. Changed the behavior of :opcode:`END_FINALLY` 2146 and :opcode:`WITH_CLEANUP_START`. 2147 2148 (Contributed by Mark Shannon, Antoine Pitrou and Serhiy Storchaka in 2149 :issue:`17611`.) 2150 2151* Added new opcode :opcode:`END_ASYNC_FOR` for handling exceptions raised 2152 when awaiting a next item in an :keyword:`async for` loop. 2153 (Contributed by Serhiy Storchaka in :issue:`33041`.) 2154 2155* The :opcode:`MAP_ADD` now expects the value as the first element in the 2156 stack and the key as the second element. This change was made so the key 2157 is always evaluated before the value in dictionary comprehensions, as 2158 proposed by :pep:`572`. (Contributed by Jörn Heissler in :issue:`35224`.) 2159 2160 2161Demos and Tools 2162--------------- 2163 2164Added a benchmark script for timing various ways to access variables: 2165``Tools/scripts/var_access_benchmark.py``. 2166(Contributed by Raymond Hettinger in :issue:`35884`.) 2167 2168Here's a summary of performance improvements since Python 3.3: 2169 2170.. code-block:: none 2171 2172 Python version 3.3 3.4 3.5 3.6 3.7 3.8 2173 -------------- --- --- --- --- --- --- 2174 2175 Variable and attribute read access: 2176 read_local 4.0 7.1 7.1 5.4 5.1 3.9 2177 read_nonlocal 5.3 7.1 8.1 5.8 5.4 4.4 2178 read_global 13.3 15.5 19.0 14.3 13.6 7.6 2179 read_builtin 20.0 21.1 21.6 18.5 19.0 7.5 2180 read_classvar_from_class 20.5 25.6 26.5 20.7 19.5 18.4 2181 read_classvar_from_instance 18.5 22.8 23.5 18.8 17.1 16.4 2182 read_instancevar 26.8 32.4 33.1 28.0 26.3 25.4 2183 read_instancevar_slots 23.7 27.8 31.3 20.8 20.8 20.2 2184 read_namedtuple 68.5 73.8 57.5 45.0 46.8 18.4 2185 read_boundmethod 29.8 37.6 37.9 29.6 26.9 27.7 2186 2187 Variable and attribute write access: 2188 write_local 4.6 8.7 9.3 5.5 5.3 4.3 2189 write_nonlocal 7.3 10.5 11.1 5.6 5.5 4.7 2190 write_global 15.9 19.7 21.2 18.0 18.0 15.8 2191 write_classvar 81.9 92.9 96.0 104.6 102.1 39.2 2192 write_instancevar 36.4 44.6 45.8 40.0 38.9 35.5 2193 write_instancevar_slots 28.7 35.6 36.1 27.3 26.6 25.7 2194 2195 Data structure read access: 2196 read_list 19.2 24.2 24.5 20.8 20.8 19.0 2197 read_deque 19.9 24.7 25.5 20.2 20.6 19.8 2198 read_dict 19.7 24.3 25.7 22.3 23.0 21.0 2199 read_strdict 17.9 22.6 24.3 19.5 21.2 18.9 2200 2201 Data structure write access: 2202 write_list 21.2 27.1 28.5 22.5 21.6 20.0 2203 write_deque 23.8 28.7 30.1 22.7 21.8 23.5 2204 write_dict 25.9 31.4 33.3 29.3 29.2 24.7 2205 write_strdict 22.9 28.4 29.9 27.5 25.2 23.1 2206 2207 Stack (or queue) operations: 2208 list_append_pop 144.2 93.4 112.7 75.4 74.2 50.8 2209 deque_append_pop 30.4 43.5 57.0 49.4 49.2 42.5 2210 deque_append_popleft 30.8 43.7 57.3 49.7 49.7 42.8 2211 2212 Timing loop: 2213 loop_overhead 0.3 0.5 0.6 0.4 0.3 0.3 2214 2215The benchmarks were measured on an 2216`Intel® Core™ i7-4960HQ processor 2217<https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html>`_ 2218running the macOS 64-bit builds found at 2219`python.org <https://www.python.org/downloads/mac-osx/>`_. 2220The benchmark script displays timings in nanoseconds. 2221 2222 2223Notable changes in Python 3.8.1 2224=============================== 2225 2226Due to significant security concerns, the *reuse_address* parameter of 2227:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is 2228because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more 2229details, see the documentation for ``loop.create_datagram_endpoint()``. 2230(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 2231:issue:`37228`.) 2232 2233Notable changes in Python 3.8.2 2234=============================== 2235 2236Fixed a regression with the ``ignore`` callback of :func:`shutil.copytree`. 2237The argument types are now str and List[str] again. 2238(Contributed by Manuel Barkhau and Giampaolo Rodola in :issue:`39390`.) 2239 2240Notable changes in Python 3.8.3 2241=============================== 2242 2243The constant values of future flags in the :mod:`__future__` module 2244are updated in order to prevent collision with compiler flags. Previously 2245``PyCF_ALLOW_TOP_LEVEL_AWAIT`` was clashing with ``CO_FUTURE_DIVISION``. 2246(Contributed by Batuhan Taskaya in :issue:`39562`) 2247
.jpg){kind=link}