1**************************** 2 What's New In Python 3.9 3**************************** 4 5:Release: |release| 6:Date: |today| 7:Editor: Łukasz Langa 8 9.. Rules for maintenance: 10 11 * Anyone can add text to this document. Your text might get 12 rewritten to some degree. 13 14 * The maintainer will go through Misc/NEWS periodically and add 15 changes; it's therefore more important to add your changes to 16 Misc/NEWS than to this file. 17 18 * This is not a complete list of every single change; completeness 19 is the purpose of Misc/NEWS. Some changes will be too small 20 or esoteric to include. If such a change is added to the text, 21 it might get removed during final editing. 22 23 * If you want to draw your new text to the attention of the 24 maintainer, add 'XXX' to the beginning of the paragraph or 25 section. 26 27 * It's OK to just add a fragmentary note about a change. For 28 example: "XXX Describe the transmogrify() function added to the 29 socket module." The maintainer will research the change and 30 write the necessary text. 31 32 * You can comment out your additions if you like, but it's not 33 necessary (especially when a final release is some months away). 34 35 * Credit the author of a patch or bugfix. Just the name is 36 sufficient; the e-mail address isn't necessary. 37 38 * It's helpful to add the bug/patch number as a comment: 39 40 XXX Describe the transmogrify() function added to the socket 41 module. 42 (Contributed by P.Y. Developer in :issue:`12345`.) 43 44 This saves the maintainer the effort of going through the Mercurial log 45 when researching a change. 46 47This article explains the new features in Python 3.9, compared to 3.8. 48Python 3.9 was released on October 5th, 2020. 49 50For full details, see the :ref:`changelog <changelog>`. 51 52.. seealso:: 53 54 :pep:`596` - Python 3.9 Release Schedule 55 56 57Summary -- Release highlights 58============================= 59 60.. This section singles out the most important changes in Python 3.9. 61 Brevity is key. 62 63New syntax features: 64 65* :pep:`584`, union operators added to ``dict``; 66* :pep:`585`, type hinting generics in standard collections; 67* :pep:`614`, relaxed grammar restrictions on decorators. 68 69New built-in features: 70 71* :pep:`616`, string methods to remove prefixes and suffixes. 72 73New features in the standard library: 74 75* :pep:`593`, flexible function and variable annotations; 76* :func:`os.pidfd_open` added that allows process management without races 77 and signals. 78 79Interpreter improvements: 80 81* :pep:`573`, fast access to module state from methods of C extension 82 types; 83* :pep:`617`, CPython now uses a new parser based on PEG; 84* a number of Python builtins (range, tuple, set, frozenset, list, dict) are 85 now sped up using :pep:`590` vectorcall; 86* garbage collection does not block on resurrected objects; 87* a number of Python modules (:mod:`_abc`, :mod:`audioop`, :mod:`_bz2`, 88 :mod:`_codecs`, :mod:`_contextvars`, :mod:`_crypt`, :mod:`_functools`, 89 :mod:`_json`, :mod:`_locale`, :mod:`math`, :mod:`operator`, :mod:`resource`, 90 :mod:`time`, :mod:`_weakref`) now use multiphase initialization as defined 91 by PEP 489; 92* a number of standard library modules (:mod:`audioop`, :mod:`ast`, :mod:`grp`, 93 :mod:`_hashlib`, :mod:`pwd`, :mod:`_posixsubprocess`, :mod:`random`, 94 :mod:`select`, :mod:`struct`, :mod:`termios`, :mod:`zlib`) are now using 95 the stable ABI defined by PEP 384. 96 97New library modules: 98 99* :pep:`615`, the IANA Time Zone Database is now present in the standard 100 library in the :mod:`zoneinfo` module; 101* an implementation of a topological sort of a graph is now provided in 102 the new :mod:`graphlib` module. 103 104Release process changes: 105 106* :pep:`602`, CPython adopts an annual release cycle. 107 108 109You should check for DeprecationWarning in your code 110==================================================== 111 112When Python 2.7 was still supported, a lot of functionality in Python 3 113was kept for backward compatibility with Python 2.7. With the end of Python 1142 support, these backward compatibility layers have been removed, or will 115be removed soon. Most of them emitted a :exc:`DeprecationWarning` warning for 116several years. For example, using ``collections.Mapping`` instead of 117``collections.abc.Mapping`` emits a :exc:`DeprecationWarning` since Python 1183.3, released in 2012. 119 120Test your application with the :option:`-W` ``default`` command-line option to see 121:exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, or even with 122:option:`-W` ``error`` to treat them as errors. :ref:`Warnings Filter 123<warning-filter>` can be used to ignore warnings from third-party code. 124 125Python 3.9 is the last version providing those Python 2 backward compatibility 126layers, to give more time to Python projects maintainers to organize the 127removal of the Python 2 support and add support for Python 3.9. 128 129Aliases to :ref:`Abstract Base Classes <collections-abstract-base-classes>` in 130the :mod:`collections` module, like ``collections.Mapping`` alias to 131:class:`collections.abc.Mapping`, are kept for one last release for backward 132compatibility. They will be removed from Python 3.10. 133 134More generally, try to run your tests in the :ref:`Python Development Mode 135<devmode>` which helps to prepare your code to make it compatible with the 136next Python version. 137 138Note: a number of pre-existing deprecations were removed in this version of 139Python as well. Consult the :ref:`removed-in-python-39` section. 140 141 142New Features 143============ 144 145Dictionary Merge & Update Operators 146----------------------------------- 147 148Merge (``|``) and update (``|=``) operators have been added to the built-in 149:class:`dict` class. Those complement the existing ``dict.update`` and 150``{**d1, **d2}`` methods of merging dictionaries. 151 152Example:: 153 154 >>> x = {"key1": "value1 from x", "key2": "value2 from x"} 155 >>> y = {"key2": "value2 from y", "key3": "value3 from y"} 156 >>> x | y 157 {'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'} 158 >>> y | x 159 {'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'} 160 161See :pep:`584` for a full description. 162(Contributed by Brandt Bucher in :issue:`36144`.) 163 164New String Methods to Remove Prefixes and Suffixes 165-------------------------------------------------- 166 167:meth:`str.removeprefix(prefix)<str.removeprefix>` and 168:meth:`str.removesuffix(suffix)<str.removesuffix>` have been added 169to easily remove an unneeded prefix or a suffix from a string. Corresponding 170``bytes``, ``bytearray``, and ``collections.UserString`` methods have also been 171added. See :pep:`616` for a full description. (Contributed by Dennis Sweeney in 172:issue:`39939`.) 173 174Type Hinting Generics in Standard Collections 175--------------------------------------------- 176 177In type annotations you can now use built-in collection types such as 178``list`` and ``dict`` as generic types instead of importing the 179corresponding capitalized types (e.g. ``List`` or ``Dict``) from 180``typing``. Some other types in the standard library are also now generic, 181for example ``queue.Queue``. 182 183Example: 184 185.. code-block:: python 186 187 def greet_all(names: list[str]) -> None: 188 for name in names: 189 print("Hello", name) 190 191See :pep:`585` for more details. (Contributed by Guido van Rossum, 192Ethan Smith, and Batuhan Taşkaya in :issue:`39481`.) 193 194New Parser 195---------- 196 197Python 3.9 uses a new parser, based on `PEG 198<https://en.wikipedia.org/wiki/Parsing_expression_grammar>`_ instead 199of `LL(1) <https://en.wikipedia.org/wiki/LL_parser>`_. The new 200parser's performance is roughly comparable to that of the old parser, 201but the PEG formalism is more flexible than LL(1) when it comes to 202designing new language features. We'll start using this flexibility 203in Python 3.10 and later. 204 205The :mod:`ast` module uses the new parser and produces the same AST as 206the old parser. 207 208In Python 3.10, the old parser will be deleted and so will all 209functionality that depends on it (primarily the :mod:`parser` module, 210which has long been deprecated). In Python 3.9 *only*, you can switch 211back to the LL(1) parser using a command line switch (``-X 212oldparser``) or an environment variable (``PYTHONOLDPARSER=1``). 213 214See :pep:`617` for more details. (Contributed by Guido van Rossum, 215Pablo Galindo and Lysandros Nikolaou in :issue:`40334`.) 216 217 218Other Language Changes 219====================== 220 221* :func:`__import__` now raises :exc:`ImportError` instead of 222 :exc:`ValueError`, which used to occur when a relative import went past 223 its top-level package. 224 (Contributed by Ngalim Siregar in :issue:`37444`.) 225 226* Python now gets the absolute path of the script filename specified on 227 the command line (ex: ``python3 script.py``): the ``__file__`` attribute of 228 the :mod:`__main__` module became an absolute path, rather than a relative 229 path. These paths now remain valid after the current directory is changed 230 by :func:`os.chdir`. As a side effect, the traceback also displays the 231 absolute path for :mod:`__main__` module frames in this case. 232 (Contributed by Victor Stinner in :issue:`20443`.) 233 234* In the :ref:`Python Development Mode <devmode>` and in :ref:`debug build <debug-build>`, the 235 *encoding* and *errors* arguments are now checked for string encoding and 236 decoding operations. Examples: :func:`open`, :meth:`str.encode` and 237 :meth:`bytes.decode`. 238 239 By default, for best performance, the *errors* argument is only checked at 240 the first encoding/decoding error and the *encoding* argument is sometimes 241 ignored for empty strings. 242 (Contributed by Victor Stinner in :issue:`37388`.) 243 244* ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for 245 all non-zero ``n``. It is now consistent with ``"".replace("", s)``. 246 There are similar changes for :class:`bytes` and :class:`bytearray` objects. 247 (Contributed by Serhiy Storchaka in :issue:`28029`.) 248 249* Any valid expression can now be used as a :term:`decorator`. Previously, the 250 grammar was much more restrictive. See :pep:`614` for details. 251 (Contributed by Brandt Bucher in :issue:`39702`.) 252 253* Improved help for the :mod:`typing` module. Docstrings are now shown for 254 all special forms and special generic aliases (like ``Union`` and ``List``). 255 Using :func:`help` with generic alias like ``List[int]`` will show the help 256 for the correspondent concrete type (``list`` in this case). 257 (Contributed by Serhiy Storchaka in :issue:`40257`.) 258 259* Parallel running of :meth:`~agen.aclose` / :meth:`~agen.asend` / 260 :meth:`~agen.athrow` is now prohibited, and ``ag_running`` now reflects 261 the actual running status of the async generator. 262 (Contributed by Yury Selivanov in :issue:`30773`.) 263 264* Unexpected errors in calling the ``__iter__`` method are no longer masked by 265 ``TypeError`` in the :keyword:`in` operator and functions 266 :func:`~operator.contains`, :func:`~operator.indexOf` and 267 :func:`~operator.countOf` of the :mod:`operator` module. 268 (Contributed by Serhiy Storchaka in :issue:`40824`.) 269 270* Unparenthesized lambda expressions can no longer be the expression part in an 271 ``if`` clause in comprehensions and generator expressions. See :issue:`41848` 272 and :issue:`43755` for details. 273 274 275New Modules 276=========== 277 278zoneinfo 279-------- 280 281The :mod:`zoneinfo` module brings support for the IANA time zone database to 282the standard library. It adds :class:`zoneinfo.ZoneInfo`, a concrete 283:class:`datetime.tzinfo` implementation backed by the system's time zone data. 284 285Example:: 286 287 >>> from zoneinfo import ZoneInfo 288 >>> from datetime import datetime, timedelta 289 290 >>> # Daylight saving time 291 >>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles")) 292 >>> print(dt) 293 2020-10-31 12:00:00-07:00 294 >>> dt.tzname() 295 'PDT' 296 297 >>> # Standard time 298 >>> dt += timedelta(days=7) 299 >>> print(dt) 300 2020-11-07 12:00:00-08:00 301 >>> print(dt.tzname()) 302 PST 303 304 305As a fall-back source of data for platforms that don't ship the IANA database, 306the |tzdata|_ module was released as a first-party package -- distributed via 307PyPI and maintained by the CPython core team. 308 309.. |tzdata| replace:: ``tzdata`` 310.. _tzdata: https://pypi.org/project/tzdata/ 311 312.. seealso:: 313 314 :pep:`615` -- Support for the IANA Time Zone Database in the Standard Library 315 PEP written and implemented by Paul Ganssle 316 317 318graphlib 319--------- 320 321A new module, :mod:`graphlib`, was added that contains the 322:class:`graphlib.TopologicalSorter` class to offer functionality to perform 323topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and 324Larry Hastings in :issue:`17005`.) 325 326 327Improved Modules 328================ 329 330ast 331--- 332 333Added the *indent* option to :func:`~ast.dump` which allows it to produce a 334multiline indented output. 335(Contributed by Serhiy Storchaka in :issue:`37995`.) 336 337Added :func:`ast.unparse` as a function in the :mod:`ast` module that can 338be used to unparse an :class:`ast.AST` object and produce a string with code 339that would produce an equivalent :class:`ast.AST` object when parsed. 340(Contributed by Pablo Galindo and Batuhan Taskaya in :issue:`38870`.) 341 342Added docstrings to AST nodes that contains the ASDL signature used to 343construct that node. (Contributed by Batuhan Taskaya in :issue:`39638`.) 344 345asyncio 346------- 347 348Due to significant security concerns, the *reuse_address* parameter of 349:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is 350because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more 351details, see the documentation for ``loop.create_datagram_endpoint()``. 352(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in 353:issue:`37228`.) 354 355Added a new :term:`coroutine` :meth:`~asyncio.loop.shutdown_default_executor` 356that schedules a shutdown for the default executor that waits on the 357:class:`~concurrent.futures.ThreadPoolExecutor` to finish closing. Also, 358:func:`asyncio.run` has been updated to use the new :term:`coroutine`. 359(Contributed by Kyle Stanley in :issue:`34037`.) 360 361Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher 362implementation that polls process file descriptors. (:issue:`38692`) 363 364Added a new :term:`coroutine` :func:`asyncio.to_thread`. It is mainly used for 365running IO-bound functions in a separate thread to avoid blocking the event 366loop, and essentially works as a high-level version of 367:meth:`~asyncio.loop.run_in_executor` that can directly take keyword arguments. 368(Contributed by Kyle Stanley and Yury Selivanov in :issue:`32309`.) 369 370When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now 371wait until the cancellation is complete also in the case when *timeout* is 372<= 0, like it does with positive timeouts. 373(Contributed by Elvis Pranskevichus in :issue:`32751`.) 374 375:mod:`asyncio` now raises :exc:`TyperError` when calling incompatible 376methods with an :class:`ssl.SSLSocket` socket. 377(Contributed by Ido Michael in :issue:`37404`.) 378 379compileall 380---------- 381 382Added new possibility to use hardlinks for duplicated ``.pyc`` files: *hardlink_dupes* parameter and --hardlink-dupes command line option. 383(Contributed by Lumír 'Frenzy' Balhar in :issue:`40495`.) 384 385Added new options for path manipulation in resulting ``.pyc`` files: *stripdir*, *prependdir*, *limit_sl_dest* parameters and -s, -p, -e command line options. 386Added the possibility to specify the option for an optimization level multiple times. 387(Contributed by Lumír 'Frenzy' Balhar in :issue:`38112`.) 388 389concurrent.futures 390------------------ 391 392Added a new *cancel_futures* parameter to 393:meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures 394which have not started running, instead of waiting for them to complete before 395shutting down the executor. 396(Contributed by Kyle Stanley in :issue:`39349`.) 397 398Removed daemon threads from :class:`~concurrent.futures.ThreadPoolExecutor` 399and :class:`~concurrent.futures.ProcessPoolExecutor`. This improves 400compatibility with subinterpreters and predictability in their shutdown 401processes. (Contributed by Kyle Stanley in :issue:`39812`.) 402 403Workers in :class:`~concurrent.futures.ProcessPoolExecutor` are now spawned on 404demand, only when there are no available idle workers to reuse. This optimizes 405startup overhead and reduces the amount of lost CPU time to idle workers. 406(Contributed by Kyle Stanley in :issue:`39207`.) 407 408curses 409------ 410 411Added :func:`curses.get_escdelay`, :func:`curses.set_escdelay`, 412:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions. 413(Contributed by Anthony Sottile in :issue:`38312`.) 414 415datetime 416-------- 417The :meth:`~datetime.date.isocalendar()` of :class:`datetime.date` 418and :meth:`~datetime.datetime.isocalendar()` of :class:`datetime.datetime` 419methods now returns a :func:`~collections.namedtuple` instead of a :class:`tuple`. 420(Contributed by Dong-hee Na in :issue:`24416`.) 421 422distutils 423--------- 424 425The :command:`upload` command now creates SHA2-256 and Blake2b-256 hash 426digests. It skips MD5 on platforms that block MD5 digest. 427(Contributed by Christian Heimes in :issue:`40698`.) 428 429fcntl 430----- 431 432Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK` 433and :data:`~fcntl.F_OFD_SETLKW`. 434(Contributed by Dong-hee Na in :issue:`38602`.) 435 436ftplib 437------- 438 439:class:`~ftplib.FTP` and :class:`~ftplib.FTP_TLS` now raise a :class:`ValueError` 440if the given timeout for their constructor is zero to prevent the creation of 441a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 442 443gc 444-- 445 446When the garbage collector makes a collection in which some objects resurrect 447(they are reachable from outside the isolated cycles after the finalizers have 448been executed), do not block the collection of all objects that are still 449unreachable. (Contributed by Pablo Galindo and Tim Peters in :issue:`38379`.) 450 451Added a new function :func:`gc.is_finalized` to check if an object has been 452finalized by the garbage collector. (Contributed by Pablo Galindo in 453:issue:`39322`.) 454 455hashlib 456------- 457 458The :mod:`hashlib` module can now use SHA3 hashes and SHAKE XOF from OpenSSL 459when available. 460(Contributed by Christian Heimes in :issue:`37630`.) 461 462Builtin hash modules can now be disabled with 463``./configure --without-builtin-hashlib-hashes`` or selectively enabled with 464e.g. ``./configure --with-builtin-hashlib-hashes=sha3,blake2`` to force use 465of OpenSSL based implementation. 466(Contributed by Christian Heimes in :issue:`40479`) 467 468 469http 470---- 471 472HTTP status codes ``103 EARLY_HINTS``, ``418 IM_A_TEAPOT`` and ``425 TOO_EARLY`` are added to 473:class:`http.HTTPStatus`. (Contributed by Dong-hee Na in :issue:`39509` and Ross Rhodes in :issue:`39507`.) 474 475IDLE and idlelib 476---------------- 477 478Added option to toggle cursor blink off. (Contributed by Zackery Spytz 479in :issue:`4603`.) 480 481Escape key now closes IDLE completion windows. (Contributed by Johnny 482Najera in :issue:`38944`.) 483 484Added keywords to module name completion list. (Contributed by Terry J. 485Reedy in :issue:`37765`.) 486 487The changes above have been backported to 3.8 maintenance releases. 488 489imaplib 490------- 491 492:class:`~imaplib.IMAP4` and :class:`~imaplib.IMAP4_SSL` now have 493an optional *timeout* parameter for their constructors. 494Also, the :meth:`~imaplib.IMAP4.open` method now has an optional *timeout* parameter 495with this change. The overridden methods of :class:`~imaplib.IMAP4_SSL` and 496:class:`~imaplib.IMAP4_stream` were applied to this change. 497(Contributed by Dong-hee Na in :issue:`38615`.) 498 499:meth:`imaplib.IMAP4.unselect` is added. 500:meth:`imaplib.IMAP4.unselect` frees server's resources associated with the 501selected mailbox and returns the server to the authenticated 502state. This command performs the same actions as :meth:`imaplib.IMAP4.close`, except 503that no messages are permanently removed from the currently 504selected mailbox. (Contributed by Dong-hee Na in :issue:`40375`.) 505 506importlib 507--------- 508 509To improve consistency with import statements, :func:`importlib.util.resolve_name` 510now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative 511import attempts. 512(Contributed by Ngalim Siregar in :issue:`37444`.) 513 514Import loaders which publish immutable module objects can now publish 515immutable packages in addition to individual modules. 516(Contributed by Dino Viehland in :issue:`39336`.) 517 518Added :func:`importlib.resources.files` function with support for 519subdirectories in package data, matching backport in ``importlib_resources`` 520version 1.5. 521(Contributed by Jason R. Coombs in :issue:`39791`.) 522 523Refreshed ``importlib.metadata`` from ``importlib_metadata`` version 1.6.1. 524 525inspect 526------- 527 528:attr:`inspect.BoundArguments.arguments` is changed from ``OrderedDict`` to regular 529dict. (Contributed by Inada Naoki in :issue:`36350` and :issue:`39775`.) 530 531ipaddress 532--------- 533 534:mod:`ipaddress` now supports IPv6 Scoped Addresses (IPv6 address with suffix ``%<scope_id>``). 535 536Scoped IPv6 addresses can be parsed using :class:`ipaddress.IPv6Address`. 537If present, scope zone ID is available through the :attr:`~ipaddress.IPv6Address.scope_id` attribute. 538(Contributed by Oleksandr Pavliuk in :issue:`34788`.) 539 540Starting with Python 3.9.5 the :mod:`ipaddress` module no longer 541accepts any leading zeros in IPv4 address strings. 542(Contributed by Christian Heimes in :issue:`36384`). 543 544math 545---- 546 547Expanded the :func:`math.gcd` function to handle multiple arguments. 548Formerly, it only supported two arguments. 549(Contributed by Serhiy Storchaka in :issue:`39648`.) 550 551Added :func:`math.lcm`: return the least common multiple of specified arguments. 552(Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in 553:issue:`39479` and :issue:`39648`.) 554 555Added :func:`math.nextafter`: return the next floating-point value after *x* 556towards *y*. 557(Contributed by Victor Stinner in :issue:`39288`.) 558 559Added :func:`math.ulp`: return the value of the least significant bit 560of a float. 561(Contributed by Victor Stinner in :issue:`39310`.) 562 563multiprocessing 564--------------- 565 566The :class:`multiprocessing.SimpleQueue` class has a new 567:meth:`~multiprocessing.SimpleQueue.close` method to explicitly close the 568queue. 569(Contributed by Victor Stinner in :issue:`30966`.) 570 571nntplib 572------- 573 574:class:`~nntplib.NNTP` and :class:`~nntplib.NNTP_SSL` now raise a :class:`ValueError` 575if the given timeout for their constructor is zero to prevent the creation of 576a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 577 578os 579-- 580 581Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`. 582(Contributed by Dong-hee Na in :issue:`38493`.) 583 584Exposed the Linux-specific :func:`os.pidfd_open` (:issue:`38692`) and 585:data:`os.P_PIDFD` (:issue:`38713`) for process management with file 586descriptors. 587 588The :func:`os.unsetenv` function is now also available on Windows. 589(Contributed by Victor Stinner in :issue:`39413`.) 590 591The :func:`os.putenv` and :func:`os.unsetenv` functions are now always 592available. 593(Contributed by Victor Stinner in :issue:`39395`.) 594 595Added :func:`os.waitstatus_to_exitcode` function: 596convert a wait status to an exit code. 597(Contributed by Victor Stinner in :issue:`40094`.) 598 599pathlib 600------- 601 602Added :meth:`pathlib.Path.readlink()` which acts similarly to 603:func:`os.readlink`. 604(Contributed by Girts Folkmanis in :issue:`30618`) 605 606pdb 607--- 608 609On Windows now :class:`~pdb.Pdb` supports ``~/.pdbrc``. 610(Contributed by Tim Hopper and Dan Lidral-Porter in :issue:`20523`.) 611 612poplib 613------ 614 615:class:`~poplib.POP3` and :class:`~poplib.POP3_SSL` now raise a :class:`ValueError` 616if the given timeout for their constructor is zero to prevent the creation of 617a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 618 619pprint 620------ 621 622:mod:`pprint` can now pretty-print :class:`types.SimpleNamespace`. 623(Contributed by Carl Bordum Hansen in :issue:`37376`.) 624 625pydoc 626----- 627 628The documentation string is now shown not only for class, function, 629method etc, but for any object that has its own ``__doc__`` attribute. 630(Contributed by Serhiy Storchaka in :issue:`40257`.) 631 632random 633------ 634 635Added a new :attr:`random.Random.randbytes` method: generate random bytes. 636(Contributed by Victor Stinner in :issue:`40286`.) 637 638signal 639------ 640 641Exposed the Linux-specific :func:`signal.pidfd_send_signal` for sending to 642signals to a process using a file descriptor instead of a pid. (:issue:`38712`) 643 644smtplib 645------- 646 647:class:`~smtplib.SMTP` and :class:`~smtplib.SMTP_SSL` now raise a :class:`ValueError` 648if the given timeout for their constructor is zero to prevent the creation of 649a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.) 650 651:class:`~smtplib.LMTP` constructor now has an optional *timeout* parameter. 652(Contributed by Dong-hee Na in :issue:`39329`.) 653 654socket 655------ 656 657The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_JOIN_FILTERS` 658constant on Linux 4.1 and greater. 659(Contributed by Stefan Tatschner and Zackery Spytz in :issue:`25780`.) 660 661The socket module now supports the :data:`~socket.CAN_J1939` protocol on 662platforms that support it. (Contributed by Karl Ding in :issue:`40291`.) 663 664The socket module now has the :func:`socket.send_fds` and 665:func:`socket.recv_fds` functions. (Contributed by Joannah Nanjekye, Shinya 666Okano and Victor Stinner in :issue:`28724`.) 667 668 669time 670---- 671 672On AIX, :func:`~time.thread_time` is now implemented with ``thread_cputime()`` 673which has nanosecond resolution, rather than 674``clock_gettime(CLOCK_THREAD_CPUTIME_ID)`` which has a resolution of 10 ms. 675(Contributed by Batuhan Taskaya in :issue:`40192`) 676 677sys 678--- 679 680Added a new :attr:`sys.platlibdir` attribute: name of the platform-specific 681library directory. It is used to build the path of standard library and the 682paths of installed extension modules. It is equal to ``"lib"`` on most 683platforms. On Fedora and SuSE, it is equal to ``"lib64"`` on 64-bit platforms. 684(Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in :issue:`1294959`.) 685 686Previously, :attr:`sys.stderr` was block-buffered when non-interactive. Now 687``stderr`` defaults to always being line-buffered. 688(Contributed by Jendrik Seipp in :issue:`13601`.) 689 690tracemalloc 691----------- 692 693Added :func:`tracemalloc.reset_peak` to set the peak size of traced memory 694blocks to the current size, to measure the peak of specific pieces of code. 695(Contributed by Huon Wilson in :issue:`40630`.) 696 697typing 698------ 699 700:pep:`593` introduced an :data:`typing.Annotated` type to decorate existing 701types with context-specific metadata and new ``include_extras`` parameter to 702:func:`typing.get_type_hints` to access the metadata at runtime. (Contributed 703by Till Varoquaux and Konstantin Kashin.) 704 705unicodedata 706----------- 707 708The Unicode database has been updated to version 13.0.0. (:issue:`39926`). 709 710venv 711---- 712 713The activation scripts provided by :mod:`venv` now all specify their prompt 714customization consistently by always using the value specified by 715``__VENV_PROMPT__``. Previously some scripts unconditionally used 716``__VENV_PROMPT__``, others only if it happened to be set (which was the default 717case), and one used ``__VENV_NAME__`` instead. 718(Contributed by Brett Cannon in :issue:`37663`.) 719 720xml 721--- 722 723White space characters within attributes are now preserved when serializing 724:mod:`xml.etree.ElementTree` to XML file. EOLNs are no longer normalized 725to "\n". This is the result of discussion about how to interpret 726section 2.11 of XML spec. 727(Contributed by Mefistotelis in :issue:`39011`.) 728 729 730Optimizations 731============= 732 733* Optimized the idiom for assignment a temporary variable in comprehensions. 734 Now ``for y in [expr]`` in comprehensions is as fast as a simple assignment 735 ``y = expr``. For example: 736 737 sums = [s for s in [0] for x in data for s in [s + x]] 738 739 Unlike the ``:=`` operator this idiom does not leak a variable to the 740 outer scope. 741 742 (Contributed by Serhiy Storchaka in :issue:`32856`.) 743 744* Optimized signal handling in multithreaded applications. If a thread different 745 than the main thread gets a signal, the bytecode evaluation loop is no longer 746 interrupted at each bytecode instruction to check for pending signals which 747 cannot be handled. Only the main thread of the main interpreter can handle 748 signals. 749 750 Previously, the bytecode evaluation loop was interrupted at each instruction 751 until the main thread handles signals. 752 (Contributed by Victor Stinner in :issue:`40010`.) 753 754* Optimized the :mod:`subprocess` module on FreeBSD using ``closefrom()``. 755 (Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor 756 Stinner in :issue:`38061`.) 757 758* :c:func:`PyLong_FromDouble` is now up to 1.87x faster for values that 759 fit into :c:type:`long`. 760 (Contributed by Sergey Fedoseev in :issue:`37986`.) 761 762* A number of Python builtins (:class:`range`, :class:`tuple`, :class:`set`, 763 :class:`frozenset`, :class:`list`, :class:`dict`) are now sped up by using 764 :pep:`590` vectorcall protocol. 765 (Contributed by Dong-hee Na, Mark Shannon, Jeroen Demeyer and Petr Viktorin in :issue:`37207`.) 766 767* Optimized :func:`~set.difference_update` for the case when the other set 768 is much larger than the base set. 769 (Suggested by Evgeny Kapun with code contributed by Michele Orrù in :issue:`8425`.) 770 771* Python's small object allocator (``obmalloc.c``) now allows (no more than) 772 one empty arena to remain available for immediate reuse, without returning 773 it to the OS. This prevents thrashing in simple loops where an arena could 774 be created and destroyed anew on each iteration. 775 (Contributed by Tim Peters in :issue:`37257`.) 776 777* :term:`floor division` of float operation now has a better performance. Also 778 the message of :exc:`ZeroDivisionError` for this operation is updated. 779 (Contributed by Dong-hee Na in :issue:`39434`.) 780 781* Decoding short ASCII strings with UTF-8 and ascii codecs is now about 782 15% faster. (Contributed by Inada Naoki in :issue:`37348`.) 783 784Here's a summary of performance improvements from Python 3.4 through Python 3.9: 785 786.. code-block:: none 787 788 Python version 3.4 3.5 3.6 3.7 3.8 3.9 789 -------------- --- --- --- --- --- --- 790 791 Variable and attribute read access: 792 read_local 7.1 7.1 5.4 5.1 3.9 3.9 793 read_nonlocal 7.1 8.1 5.8 5.4 4.4 4.5 794 read_global 15.5 19.0 14.3 13.6 7.6 7.8 795 read_builtin 21.1 21.6 18.5 19.0 7.5 7.8 796 read_classvar_from_class 25.6 26.5 20.7 19.5 18.4 17.9 797 read_classvar_from_instance 22.8 23.5 18.8 17.1 16.4 16.9 798 read_instancevar 32.4 33.1 28.0 26.3 25.4 25.3 799 read_instancevar_slots 27.8 31.3 20.8 20.8 20.2 20.5 800 read_namedtuple 73.8 57.5 45.0 46.8 18.4 18.7 801 read_boundmethod 37.6 37.9 29.6 26.9 27.7 41.1 802 803 Variable and attribute write access: 804 write_local 8.7 9.3 5.5 5.3 4.3 4.3 805 write_nonlocal 10.5 11.1 5.6 5.5 4.7 4.8 806 write_global 19.7 21.2 18.0 18.0 15.8 16.7 807 write_classvar 92.9 96.0 104.6 102.1 39.2 39.8 808 write_instancevar 44.6 45.8 40.0 38.9 35.5 37.4 809 write_instancevar_slots 35.6 36.1 27.3 26.6 25.7 25.8 810 811 Data structure read access: 812 read_list 24.2 24.5 20.8 20.8 19.0 19.5 813 read_deque 24.7 25.5 20.2 20.6 19.8 20.2 814 read_dict 24.3 25.7 22.3 23.0 21.0 22.4 815 read_strdict 22.6 24.3 19.5 21.2 18.9 21.5 816 817 Data structure write access: 818 write_list 27.1 28.5 22.5 21.6 20.0 20.0 819 write_deque 28.7 30.1 22.7 21.8 23.5 21.7 820 write_dict 31.4 33.3 29.3 29.2 24.7 25.4 821 write_strdict 28.4 29.9 27.5 25.2 23.1 24.5 822 823 Stack (or queue) operations: 824 list_append_pop 93.4 112.7 75.4 74.2 50.8 50.6 825 deque_append_pop 43.5 57.0 49.4 49.2 42.5 44.2 826 deque_append_popleft 43.7 57.3 49.7 49.7 42.8 46.4 827 828 Timing loop: 829 loop_overhead 0.5 0.6 0.4 0.3 0.3 0.3 830 831These results were generated from the variable access benchmark script at: 832``Tools/scripts/var_access_benchmark.py``. The benchmark script displays timings 833in nanoseconds. The benchmarks were measured on an 834`Intel® Core™ i7-4960HQ processor 835<https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html>`_ 836running the macOS 64-bit builds found at 837`python.org <https://www.python.org/downloads/mac-osx/>`_. 838 839 840Deprecated 841========== 842 843* The distutils ``bdist_msi`` command is now deprecated, use 844 ``bdist_wheel`` (wheel packages) instead. 845 (Contributed by Hugo van Kemenade in :issue:`39586`.) 846 847* Currently :func:`math.factorial` accepts :class:`float` instances with 848 non-negative integer values (like ``5.0``). It raises a :exc:`ValueError` 849 for non-integral and negative floats. It is now deprecated. In future 850 Python versions it will raise a :exc:`TypeError` for all floats. 851 (Contributed by Serhiy Storchaka in :issue:`37315`.) 852 853* The :mod:`parser` and :mod:`symbol` modules are deprecated and will be 854 removed in future versions of Python. For the majority of use cases, 855 users can leverage the Abstract Syntax Tree (AST) generation and compilation 856 stage, using the :mod:`ast` module. 857 858* The Public C API functions :c:func:`PyParser_SimpleParseStringFlags`, 859 :c:func:`PyParser_SimpleParseStringFlagsFilename`, 860 :c:func:`PyParser_SimpleParseFileFlags` and :c:func:`PyNode_Compile` 861 are deprecated and will be removed in Python 3.10 together with the old parser. 862 863* Using :data:`NotImplemented` in a boolean context has been deprecated, 864 as it is almost exclusively the result of incorrect rich comparator 865 implementations. It will be made a :exc:`TypeError` in a future version 866 of Python. 867 (Contributed by Josh Rosenberg in :issue:`35712`.) 868 869* The :mod:`random` module currently accepts any hashable type as a 870 possible seed value. Unfortunately, some of those types are not 871 guaranteed to have a deterministic hash value. After Python 3.9, 872 the module will restrict its seeds to :const:`None`, :class:`int`, 873 :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`. 874 875* Opening the :class:`~gzip.GzipFile` file for writing without specifying 876 the *mode* argument is deprecated. In future Python versions it will always 877 be opened for reading by default. Specify the *mode* argument for opening 878 it for writing and silencing a warning. 879 (Contributed by Serhiy Storchaka in :issue:`28286`.) 880 881* Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in 882 favour of the ``splitlist()`` method which has more consistent and 883 predicable behavior. 884 (Contributed by Serhiy Storchaka in :issue:`38371`.) 885 886* The explicit passing of coroutine objects to :func:`asyncio.wait` has been 887 deprecated and will be removed in version 3.11. 888 (Contributed by Yury Selivanov and Kyle Stanley in :issue:`34790`.) 889 890* binhex4 and hexbin4 standards are now deprecated. The :mod:`binhex` module 891 and the following :mod:`binascii` functions are now deprecated: 892 893 * :func:`~binascii.b2a_hqx`, :func:`~binascii.a2b_hqx` 894 * :func:`~binascii.rlecode_hqx`, :func:`~binascii.rledecode_hqx` 895 896 (Contributed by Victor Stinner in :issue:`39353`.) 897 898* :mod:`ast` classes ``slice``, ``Index`` and ``ExtSlice`` are considered deprecated 899 and will be removed in future Python versions. ``value`` itself should be 900 used instead of ``Index(value)``. ``Tuple(slices, Load())`` should be 901 used instead of ``ExtSlice(slices)``. 902 (Contributed by Serhiy Storchaka in :issue:`34822`.) 903 904* :mod:`ast` classes ``Suite``, ``Param``, ``AugLoad`` and ``AugStore`` 905 are considered deprecated and will be removed in future Python versions. 906 They were not generated by the parser and not accepted by the code 907 generator in Python 3. 908 (Contributed by Batuhan Taskaya in :issue:`39639` and :issue:`39969` 909 and Serhiy Storchaka in :issue:`39988`.) 910 911* The :c:func:`PyEval_InitThreads` and :c:func:`PyEval_ThreadsInitialized` 912 functions are now deprecated and will be removed in Python 3.11. Calling 913 :c:func:`PyEval_InitThreads` now does nothing. The :term:`GIL` is initialized 914 by :c:func:`Py_Initialize()` since Python 3.7. 915 (Contributed by Victor Stinner in :issue:`39877`.) 916 917* Passing ``None`` as the first argument to the :func:`shlex.split` function 918 has been deprecated. (Contributed by Zackery Spytz in :issue:`33262`.) 919 920* :func:`smtpd.MailmanProxy` is now deprecated as it is unusable without 921 an external module, ``mailman``. (Contributed by Samuel Colvin in :issue:`35800`.) 922 923* The :mod:`lib2to3` module now emits a :exc:`PendingDeprecationWarning`. 924 Python 3.9 switched to a PEG parser (see :pep:`617`), and Python 3.10 may 925 include new language syntax that is not parsable by lib2to3's LL(1) parser. 926 The ``lib2to3`` module may be removed from the standard library in a future 927 Python version. Consider third-party alternatives such as `LibCST`_ or 928 `parso`_. 929 (Contributed by Carl Meyer in :issue:`40360`.) 930 931* The *random* parameter of :func:`random.shuffle` has been deprecated. 932 (Contributed by Raymond Hettinger in :issue:`40465`) 933 934.. _LibCST: https://libcst.readthedocs.io/ 935.. _parso: https://parso.readthedocs.io/ 936 937.. _removed-in-python-39: 938 939Removed 940======= 941 942* The erroneous version at :data:`unittest.mock.__version__` has been removed. 943 944* :class:`nntplib.NNTP`: ``xpath()`` and ``xgtitle()`` methods have been removed. 945 These methods are deprecated since Python 3.3. Generally, these extensions 946 are not supported or not enabled by NNTP server administrators. 947 For ``xgtitle()``, please use :meth:`nntplib.NNTP.descriptions` or 948 :meth:`nntplib.NNTP.description` instead. 949 (Contributed by Dong-hee Na in :issue:`39366`.) 950 951* :class:`array.array`: ``tostring()`` and ``fromstring()`` methods have been 952 removed. They were aliases to ``tobytes()`` and ``frombytes()``, deprecated 953 since Python 3.2. 954 (Contributed by Victor Stinner in :issue:`38916`.) 955 956* The undocumented ``sys.callstats()`` function has been removed. Since Python 957 3.7, it was deprecated and always returned :const:`None`. It required a special 958 build option ``CALL_PROFILE`` which was already removed in Python 3.7. 959 (Contributed by Victor Stinner in :issue:`37414`.) 960 961* The ``sys.getcheckinterval()`` and ``sys.setcheckinterval()`` functions have 962 been removed. They were deprecated since Python 3.2. Use 963 :func:`sys.getswitchinterval` and :func:`sys.setswitchinterval` instead. 964 (Contributed by Victor Stinner in :issue:`37392`.) 965 966* The C function ``PyImport_Cleanup()`` has been removed. It was documented as: 967 "Empty the module table. For internal use only." 968 (Contributed by Victor Stinner in :issue:`36710`.) 969 970* ``_dummy_thread`` and ``dummy_threading`` modules have been removed. These 971 modules were deprecated since Python 3.7 which requires threading support. 972 (Contributed by Victor Stinner in :issue:`37312`.) 973 974* ``aifc.openfp()`` alias to ``aifc.open()``, ``sunau.openfp()`` alias to 975 ``sunau.open()``, and ``wave.openfp()`` alias to :func:`wave.open()` have been 976 removed. They were deprecated since Python 3.7. 977 (Contributed by Victor Stinner in :issue:`37320`.) 978 979* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread` 980 has been removed. It was deprecated since Python 3.8. 981 Use :meth:`~threading.Thread.is_alive()` instead. 982 (Contributed by Dong-hee Na in :issue:`37804`.) 983 984* Methods ``getchildren()`` and ``getiterator()`` of classes 985 :class:`~xml.etree.ElementTree.ElementTree` and 986 :class:`~xml.etree.ElementTree.Element` in the :mod:`~xml.etree.ElementTree` 987 module have been removed. They were deprecated in Python 3.2. 988 Use ``iter(x)`` or ``list(x)`` instead of ``x.getchildren()`` and 989 ``x.iter()`` or ``list(x.iter())`` instead of ``x.getiterator()``. 990 (Contributed by Serhiy Storchaka in :issue:`36543`.) 991 992* The old :mod:`plistlib` API has been removed, it was deprecated since Python 993 3.4. Use the :func:`~plistlib.load`, :func:`~plistlib.loads`, :func:`~plistlib.dump`, and 994 :func:`~plistlib.dumps` functions. Additionally, the *use_builtin_types* parameter was 995 removed, standard :class:`bytes` objects are always used instead. 996 (Contributed by Jon Janzen in :issue:`36409`.) 997 998* The C function ``PyGen_NeedsFinalizing`` has been removed. It was not 999 documented, tested, or used anywhere within CPython after the implementation 1000 of :pep:`442`. Patch by Joannah Nanjekye. 1001 (Contributed by Joannah Nanjekye in :issue:`15088`) 1002 1003* ``base64.encodestring()`` and ``base64.decodestring()``, aliases deprecated 1004 since Python 3.1, have been removed: use :func:`base64.encodebytes` and 1005 :func:`base64.decodebytes` instead. 1006 (Contributed by Victor Stinner in :issue:`39351`.) 1007 1008* ``fractions.gcd()`` function has been removed, it was deprecated since Python 1009 3.5 (:issue:`22486`): use :func:`math.gcd` instead. 1010 (Contributed by Victor Stinner in :issue:`39350`.) 1011 1012* The *buffering* parameter of :class:`bz2.BZ2File` has been removed. Since 1013 Python 3.0, it was ignored and using it emitted a :exc:`DeprecationWarning`. 1014 Pass an open file object to control how the file is opened. 1015 (Contributed by Victor Stinner in :issue:`39357`.) 1016 1017* The *encoding* parameter of :func:`json.loads` has been removed. 1018 As of Python 3.1, it was deprecated and ignored; using it has emitted a 1019 :exc:`DeprecationWarning` since Python 3.8. 1020 (Contributed by Inada Naoki in :issue:`39377`) 1021 1022* ``with (await asyncio.lock):`` and ``with (yield from asyncio.lock):`` statements are 1023 not longer supported, use ``async with lock`` instead. The same is correct for 1024 ``asyncio.Condition`` and ``asyncio.Semaphore``. 1025 (Contributed by Andrew Svetlov in :issue:`34793`.) 1026 1027* The :func:`sys.getcounts` function, the ``-X showalloccount`` command line 1028 option and the ``show_alloc_count`` field of the C structure 1029 :c:type:`PyConfig` have been removed. They required a special Python build by 1030 defining ``COUNT_ALLOCS`` macro. 1031 (Contributed by Victor Stinner in :issue:`39489`.) 1032 1033* The ``_field_types`` attribute of the :class:`typing.NamedTuple` class 1034 has been removed. It was deprecated since Python 3.8. Use 1035 the ``__annotations__`` attribute instead. 1036 (Contributed by Serhiy Storchaka in :issue:`40182`.) 1037 1038* The :meth:`symtable.SymbolTable.has_exec` method has been removed. It was 1039 deprecated since 2006, and only returning ``False`` when it's called. 1040 (Contributed by Batuhan Taskaya in :issue:`40208`) 1041 1042* The :meth:`asyncio.Task.current_task` and :meth:`asyncio.Task.all_tasks` 1043 have been removed. They were deprecated since Python 3.7 and you can use 1044 :func:`asyncio.current_task` and :func:`asyncio.all_tasks` instead. 1045 (Contributed by Rémi Lapeyre in :issue:`40967`) 1046 1047* The ``unescape()`` method in the :class:`html.parser.HTMLParser` class 1048 has been removed (it was deprecated since Python 3.4). :func:`html.unescape` 1049 should be used for converting character references to the corresponding 1050 unicode characters. 1051 1052 1053Porting to Python 3.9 1054===================== 1055 1056This section lists previously described changes and other bugfixes 1057that may require changes to your code. 1058 1059 1060Changes in the Python API 1061------------------------- 1062 1063* :func:`__import__` and :func:`importlib.util.resolve_name` now raise 1064 :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers 1065 catching the specific exception type and supporting both Python 3.9 and 1066 earlier versions will need to catch both using ``except (ImportError, ValueError):``. 1067 1068* The :mod:`venv` activation scripts no longer special-case when 1069 ``__VENV_PROMPT__`` is set to ``""``. 1070 1071* The :meth:`select.epoll.unregister` method no longer ignores the 1072 :data:`~errno.EBADF` error. 1073 (Contributed by Victor Stinner in :issue:`39239`.) 1074 1075* The *compresslevel* parameter of :class:`bz2.BZ2File` became keyword-only, 1076 since the *buffering* parameter has been removed. 1077 (Contributed by Victor Stinner in :issue:`39357`.) 1078 1079* Simplified AST for subscription. Simple indices will be represented by 1080 their value, extended slices will be represented as tuples. 1081 ``Index(value)`` will return a ``value`` itself, ``ExtSlice(slices)`` 1082 will return ``Tuple(slices, Load())``. 1083 (Contributed by Serhiy Storchaka in :issue:`34822`.) 1084 1085* The :mod:`importlib` module now ignores the :envvar:`PYTHONCASEOK` 1086 environment variable when the :option:`-E` or :option:`-I` command line 1087 options are being used. 1088 1089* The *encoding* parameter has been added to the classes :class:`ftplib.FTP` and 1090 :class:`ftplib.FTP_TLS` as a keyword-only parameter, and the default encoding 1091 is changed from Latin-1 to UTF-8 to follow :rfc:`2640`. 1092 1093* :meth:`asyncio.loop.shutdown_default_executor` has been added to 1094 :class:`~asyncio.AbstractEventLoop`, meaning alternative event loops that 1095 inherit from it should have this method defined. 1096 (Contributed by Kyle Stanley in :issue:`34037`.) 1097 1098* The constant values of future flags in the :mod:`__future__` module 1099 is updated in order to prevent collision with compiler flags. Previously 1100 ``PyCF_ALLOW_TOP_LEVEL_AWAIT`` was clashing with ``CO_FUTURE_DIVISION``. 1101 (Contributed by Batuhan Taskaya in :issue:`39562`) 1102 1103* ``array('u')`` now uses ``wchar_t`` as C type instead of ``Py_UNICODE``. 1104 This change doesn't affect to its behavior because ``Py_UNICODE`` is alias 1105 of ``wchar_t`` since Python 3.3. 1106 (Contributed by Inada Naoki in :issue:`34538`.) 1107 1108* The :func:`logging.getLogger` API now returns the root logger when passed 1109 the name ``'root'``, whereas previously it returned a non-root logger named 1110 ``'root'``. This could affect cases where user code explicitly wants a 1111 non-root logger named ``'root'``, or instantiates a logger using 1112 ``logging.getLogger(__name__)`` in some top-level module called ``'root.py'``. 1113 (Contributed by Vinay Sajip in :issue:`37742`.) 1114 1115* Division handling of :class:`~pathlib.PurePath` now returns ``NotImplemented`` 1116 instead of raising a :exc:`TypeError` when passed something other than an 1117 instance of ``str`` or :class:`~pathlib.PurePath`. This allows creating 1118 compatible classes that don't inherit from those mentioned types. 1119 (Contributed by Roger Aiudi in :issue:`34775`). 1120 1121* Starting with Python 3.9.5 the :mod:`ipaddress` module no longer 1122 accepts any leading zeros in IPv4 address strings. Leading zeros are 1123 ambiguous and interpreted as octal notation by some libraries. For example 1124 the legacy function :func:`socket.inet_aton` treats leading zeros as octal 1125 notatation. glibc implementation of modern :func:`~socket.inet_pton` does 1126 not accept any leading zeros. 1127 (Contributed by Christian Heimes in :issue:`36384`). 1128 1129* :func:`codecs.lookup` now normalizes the encoding name the same way as 1130 :func:`encodings.normalize_encoding`, except that :func:`codecs.lookup` also 1131 converts the name to lower case. For example, ``"latex+latin1"`` encoding 1132 name is now normalized to ``"latex_latin1"``. 1133 (Contributed by Jordon Xu in :issue:`37751`.) 1134 1135 1136Changes in the C API 1137-------------------- 1138 1139* Instances of :ref:`heap-allocated types <heap-types>` (such as those created with 1140 :c:func:`PyType_FromSpec` and similar APIs) hold a reference to their type 1141 object since Python 3.8. As indicated in the "Changes in the C API" of Python 1142 3.8, for the vast majority of cases, there should be no side effect but for 1143 types that have a custom :c:member:`~PyTypeObject.tp_traverse` function, 1144 ensure that all custom ``tp_traverse`` functions of heap-allocated types 1145 visit the object's type. 1146 1147 Example: 1148 1149 .. code-block:: c 1150 1151 int 1152 foo_traverse(foo_struct *self, visitproc visit, void *arg) { 1153 // Rest of the traverse function 1154 #if PY_VERSION_HEX >= 0x03090000 1155 // This was not needed before Python 3.9 (Python issue 35810 and 40217) 1156 Py_VISIT(Py_TYPE(self)); 1157 #endif 1158 } 1159 1160 If your traverse function delegates to ``tp_traverse`` of its base class 1161 (or another type), ensure that ``Py_TYPE(self)`` is visited only once. 1162 Note that only :ref:`heap type <heap-types>` are expected to visit the type 1163 in ``tp_traverse``. 1164 1165 For example, if your ``tp_traverse`` function includes: 1166 1167 .. code-block:: c 1168 1169 base->tp_traverse(self, visit, arg) 1170 1171 then add: 1172 1173 .. code-block:: c 1174 1175 #if PY_VERSION_HEX >= 0x03090000 1176 // This was not needed before Python 3.9 (bpo-35810 and bpo-40217) 1177 if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) { 1178 // a heap type's tp_traverse already visited Py_TYPE(self) 1179 } else { 1180 Py_VISIT(Py_TYPE(self)); 1181 } 1182 #else 1183 1184 (See :issue:`35810` and :issue:`40217` for more information.) 1185 1186* The functions ``PyEval_CallObject``, ``PyEval_CallFunction``, 1187 ``PyEval_CallMethod`` and ``PyEval_CallObjectWithKeywords`` are deprecated. 1188 Use :c:func:`PyObject_Call` and its variants instead. 1189 (See more details in :issue:`29548`.) 1190 1191CPython bytecode changes 1192------------------------ 1193 1194* The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the 1195 :keyword:`assert` statement. Previously, the assert statement would not work 1196 correctly if the :exc:`AssertionError` exception was being shadowed. 1197 (Contributed by Zackery Spytz in :issue:`34880`.) 1198 1199* The :opcode:`COMPARE_OP` opcode was split into four distinct instructions: 1200 1201 * ``COMPARE_OP`` for rich comparisons 1202 * ``IS_OP`` for 'is' and 'is not' tests 1203 * ``CONTAINS_OP`` for 'in' and 'not in' tests 1204 * ``JUMP_IF_NOT_EXC_MATCH`` for checking exceptions in 'try-except' 1205 statements. 1206 1207 (Contributed by Mark Shannon in :issue:`39156`.) 1208 1209 1210Build Changes 1211============= 1212 1213* Added ``--with-platlibdir`` option to the ``configure`` script: name of the 1214 platform-specific library directory, stored in the new :attr:`sys.platlibdir` 1215 attribute. See :attr:`sys.platlibdir` attribute for more information. 1216 (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis 1217 and Victor Stinner in :issue:`1294959`.) 1218 1219* The ``COUNT_ALLOCS`` special build macro has been removed. 1220 (Contributed by Victor Stinner in :issue:`39489`.) 1221 1222* On non-Windows platforms, the :c:func:`setenv` and :c:func:`unsetenv` 1223 functions are now required to build Python. 1224 (Contributed by Victor Stinner in :issue:`39395`.) 1225 1226* On non-Windows platforms, creating ``bdist_wininst`` installers is now 1227 officially unsupported. (See :issue:`10945` for more details.) 1228 1229* When building Python on macOS from source, ``_tkinter`` now links with 1230 non-system Tcl and Tk frameworks if they are installed in 1231 ``/Library/Frameworks``, as had been the case on older releases 1232 of macOS. If a macOS SDK is explicitly configured, by using 1233 :option:`--enable-universalsdk` or ``-isysroot``, only the SDK itself is 1234 searched. The default behavior can still be overridden with 1235 :option:`--with-tcltk-includes` and :option:`--with-tcltk-libs`. 1236 (Contributed by Ned Deily in :issue:`34956`.) 1237 1238* Python can now be built for Windows 10 ARM64. 1239 (Contributed by Steve Dower in :issue:`33125`.) 1240 1241* Some individual tests are now skipped when ``--pgo`` is used. The tests 1242 in question increased the PGO task time significantly and likely 1243 didn't help improve optimization of the final executable. This 1244 speeds up the task by a factor of about 15x. Running the full unit test 1245 suite is slow. This change may result in a slightly less optimized build 1246 since not as many code branches will be executed. If you are willing to 1247 wait for the much slower build, the old behavior can be restored using 1248 ``./configure [..] PROFILE_TASK="-m test --pgo-extended"``. We make no 1249 guarantees as to which PGO task set produces a faster build. Users who care 1250 should run their own relevant benchmarks as results can depend on the 1251 environment, workload, and compiler tool chain. 1252 (See :issue:`36044` and :issue:`37707` for more details.) 1253 1254 1255C API Changes 1256============= 1257 1258New Features 1259------------ 1260 1261* :pep:`573`: Added :c:func:`PyType_FromModuleAndSpec` to associate 1262 a module with a class; :c:func:`PyType_GetModule` and 1263 :c:func:`PyType_GetModuleState` to retrieve the module and its state; and 1264 :c:data:`PyCMethod` and :c:data:`METH_METHOD` to allow a method to 1265 access the class it was defined in. 1266 (Contributed by Marcel Plch and Petr Viktorin in :issue:`38787`.) 1267 1268* Added :c:func:`PyFrame_GetCode` function: get a frame code. 1269 Added :c:func:`PyFrame_GetBack` function: get the frame next outer frame. 1270 (Contributed by Victor Stinner in :issue:`40421`.) 1271 1272* Added :c:func:`PyFrame_GetLineNumber` to the limited C API. 1273 (Contributed by Victor Stinner in :issue:`40421`.) 1274 1275* Added :c:func:`PyThreadState_GetInterpreter` and 1276 :c:func:`PyInterpreterState_Get` functions to get the interpreter. 1277 Added :c:func:`PyThreadState_GetFrame` function to get the current frame of a 1278 Python thread state. 1279 Added :c:func:`PyThreadState_GetID` function: get the unique identifier of a 1280 Python thread state. 1281 (Contributed by Victor Stinner in :issue:`39947`.) 1282 1283* Added a new public :c:func:`PyObject_CallNoArgs` function to the C API, which 1284 calls a callable Python object without any arguments. It is the most efficient 1285 way to call a callable Python object without any argument. 1286 (Contributed by Victor Stinner in :issue:`37194`.) 1287 1288* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined): 1289 1290 * Provide :c:func:`Py_EnterRecursiveCall` and :c:func:`Py_LeaveRecursiveCall` 1291 as regular functions for the limited API. Previously, there were defined as 1292 macros, but these macros didn't compile with the limited C API which cannot 1293 access ``PyThreadState.recursion_depth`` field (the structure is opaque in 1294 the limited C API). 1295 1296 * ``PyObject_INIT()`` and ``PyObject_INIT_VAR()`` become regular "opaque" 1297 function to hide implementation details. 1298 1299 (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.) 1300 1301* The :c:func:`PyModule_AddType` function is added to help adding a type 1302 to a module. 1303 (Contributed by Dong-hee Na in :issue:`40024`.) 1304 1305* Added the functions :c:func:`PyObject_GC_IsTracked` and 1306 :c:func:`PyObject_GC_IsFinalized` to the public API to allow to query if 1307 Python objects are being currently tracked or have been already finalized by 1308 the garbage collector respectively. 1309 (Contributed by Pablo Galindo Salgado in :issue:`40241`.) 1310 1311* Added :c:func:`_PyObject_FunctionStr` to get a user-friendly string 1312 representation of a function-like object. 1313 (Patch by Jeroen Demeyer in :issue:`37645`.) 1314 1315* Added :c:func:`PyObject_CallOneArg` for calling an object with one 1316 positional argument 1317 (Patch by Jeroen Demeyer in :issue:`37483`.) 1318 1319 1320Porting to Python 3.9 1321--------------------- 1322 1323* ``PyInterpreterState.eval_frame`` (:pep:`523`) now requires a new mandatory 1324 *tstate* parameter (``PyThreadState*``). 1325 (Contributed by Victor Stinner in :issue:`38500`.) 1326 1327* Extension modules: :c:member:`~PyModuleDef.m_traverse`, 1328 :c:member:`~PyModuleDef.m_clear` and :c:member:`~PyModuleDef.m_free` 1329 functions of :c:type:`PyModuleDef` are no longer called if the module state 1330 was requested but is not allocated yet. This is the case immediately after 1331 the module is created and before the module is executed 1332 (:c:data:`Py_mod_exec` function). More precisely, these functions are not called 1333 if :c:member:`~PyModuleDef.m_size` is greater than 0 and the module state (as 1334 returned by :c:func:`PyModule_GetState`) is ``NULL``. 1335 1336 Extension modules without module state (``m_size <= 0``) are not affected. 1337 1338* If :c:func:`Py_AddPendingCall` is called in a subinterpreter, the function is 1339 now scheduled to be called from the subinterpreter, rather than being called 1340 from the main interpreter. Each subinterpreter now has its own list of 1341 scheduled calls. 1342 (Contributed by Victor Stinner in :issue:`39984`.) 1343 1344* The Windows registry is no longer used to initialize :data:`sys.path` when 1345 the ``-E`` option is used (if :c:member:`PyConfig.use_environment` is set to 1346 ``0``). This is significant when embedding Python on Windows. 1347 (Contributed by Zackery Spytz in :issue:`8901`.) 1348 1349* The global variable :c:data:`PyStructSequence_UnnamedField` is now a constant 1350 and refers to a constant string. 1351 (Contributed by Serhiy Storchaka in :issue:`38650`.) 1352 1353* The :c:type:`PyGC_Head` structure is now opaque. It is only defined in the 1354 internal C API (``pycore_gc.h``). 1355 (Contributed by Victor Stinner in :issue:`40241`.) 1356 1357* The ``Py_UNICODE_COPY``, ``Py_UNICODE_FILL``, ``PyUnicode_WSTR_LENGTH``, 1358 :c:func:`PyUnicode_FromUnicode`, :c:func:`PyUnicode_AsUnicode`, 1359 ``_PyUnicode_AsUnicode``, and :c:func:`PyUnicode_AsUnicodeAndSize` are 1360 marked as deprecated in C. They have been deprecated by :pep:`393` since 1361 Python 3.3. 1362 (Contributed by Inada Naoki in :issue:`36346`.) 1363 1364* The :c:func:`Py_FatalError` function is replaced with a macro which logs 1365 automatically the name of the current function, unless the 1366 ``Py_LIMITED_API`` macro is defined. 1367 (Contributed by Victor Stinner in :issue:`39882`.) 1368 1369* The vectorcall protocol now requires that the caller passes only strings as 1370 keyword names. (See :issue:`37540` for more information.) 1371 1372* Implementation details of a number of macros and functions are now hidden: 1373 1374 * :c:func:`PyObject_IS_GC` macro was converted to a function. 1375 1376 * The :c:func:`PyObject_NEW` macro becomes an alias to the 1377 :c:func:`PyObject_New` macro, and the :c:func:`PyObject_NEW_VAR` macro 1378 becomes an alias to the :c:func:`PyObject_NewVar` macro. They no longer 1379 access directly the :c:member:`PyTypeObject.tp_basicsize` member. 1380 1381 * :c:func:`PyObject_GET_WEAKREFS_LISTPTR` macro was converted to a function: 1382 the macro accessed directly the :c:member:`PyTypeObject.tp_weaklistoffset` 1383 member. 1384 1385 * :c:func:`PyObject_CheckBuffer` macro was converted to a function: the macro 1386 accessed directly the :c:member:`PyTypeObject.tp_as_buffer` member. 1387 1388 * :c:func:`PyIndex_Check` is now always declared as an opaque function to hide 1389 implementation details: removed the ``PyIndex_Check()`` macro. The macro accessed 1390 directly the :c:member:`PyTypeObject.tp_as_number` member. 1391 1392 (See :issue:`40170` for more details.) 1393 1394Removed 1395------- 1396 1397* Excluded ``PyFPE_START_PROTECT()`` and ``PyFPE_END_PROTECT()`` macros of 1398 ``pyfpe.h`` from the limited C API. 1399 (Contributed by Victor Stinner in :issue:`38835`.) 1400 1401* The ``tp_print`` slot of :ref:`PyTypeObject <type-structs>` has been removed. 1402 It was used for printing objects to files in Python 2.7 and before. Since 1403 Python 3.0, it has been ignored and unused. 1404 (Contributed by Jeroen Demeyer in :issue:`36974`.) 1405 1406* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined): 1407 1408 * Excluded the following functions from the limited C API: 1409 1410 * ``PyThreadState_DeleteCurrent()`` 1411 (Contributed by Joannah Nanjekye in :issue:`37878`.) 1412 * ``_Py_CheckRecursionLimit`` 1413 * ``_Py_NewReference()`` 1414 * ``_Py_ForgetReference()`` 1415 * ``_PyTraceMalloc_NewReference()`` 1416 * ``_Py_GetRefTotal()`` 1417 * The trashcan mechanism which never worked in the limited C API. 1418 * ``PyTrash_UNWIND_LEVEL`` 1419 * ``Py_TRASHCAN_BEGIN_CONDITION`` 1420 * ``Py_TRASHCAN_BEGIN`` 1421 * ``Py_TRASHCAN_END`` 1422 * ``Py_TRASHCAN_SAFE_BEGIN`` 1423 * ``Py_TRASHCAN_SAFE_END`` 1424 1425 * Moved following functions and definitions to the internal C API: 1426 1427 * ``_PyDebug_PrintTotalRefs()`` 1428 * ``_Py_PrintReferences()`` 1429 * ``_Py_PrintReferenceAddresses()`` 1430 * ``_Py_tracemalloc_config`` 1431 * ``_Py_AddToAllObjects()`` (specific to ``Py_TRACE_REFS`` build) 1432 1433 (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.) 1434 1435* Removed ``_PyRuntime.getframe`` hook and removed ``_PyThreadState_GetFrame`` 1436 macro which was an alias to ``_PyRuntime.getframe``. They were only exposed 1437 by the internal C API. Removed also ``PyThreadFrameGetter`` type. 1438 (Contributed by Victor Stinner in :issue:`39946`.) 1439 1440* Removed the following functions from the C API. Call :c:func:`PyGC_Collect` 1441 explicitly to clear all free lists. 1442 (Contributed by Inada Naoki and Victor Stinner in :issue:`37340`, 1443 :issue:`38896` and :issue:`40428`.) 1444 1445 * ``PyAsyncGen_ClearFreeLists()`` 1446 * ``PyContext_ClearFreeList()`` 1447 * ``PyDict_ClearFreeList()`` 1448 * ``PyFloat_ClearFreeList()`` 1449 * ``PyFrame_ClearFreeList()`` 1450 * ``PyList_ClearFreeList()`` 1451 * ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``: 1452 the free lists of bound method objects have been removed. 1453 * ``PySet_ClearFreeList()``: the set free list has been removed 1454 in Python 3.4. 1455 * ``PyTuple_ClearFreeList()`` 1456 * ``PyUnicode_ClearFreeList()``: the Unicode free list has been removed in 1457 Python 3.3. 1458 1459* Removed ``_PyUnicode_ClearStaticStrings()`` function. 1460 (Contributed by Victor Stinner in :issue:`39465`.) 1461 1462* Removed ``Py_UNICODE_MATCH``. It has been deprecated by :pep:`393`, and 1463 broken since Python 3.3. The :c:func:`PyUnicode_Tailmatch` function can be 1464 used instead. 1465 (Contributed by Inada Naoki in :issue:`36346`.) 1466 1467* Cleaned header files of interfaces defined but with no implementation. 1468 The public API symbols being removed are: 1469 ``_PyBytes_InsertThousandsGroupingLocale``, 1470 ``_PyBytes_InsertThousandsGrouping``, ``_Py_InitializeFromArgs``, 1471 ``_Py_InitializeFromWideArgs``, ``_PyFloat_Repr``, ``_PyFloat_Digits``, 1472 ``_PyFloat_DigitsInit``, ``PyFrame_ExtendStack``, ``_PyAIterWrapper_Type``, 1473 ``PyNullImporter_Type``, ``PyCmpWrapper_Type``, ``PySortWrapper_Type``, 1474 ``PyNoArgsFunction``. 1475 (Contributed by Pablo Galindo Salgado in :issue:`39372`.) 1476 1477Notable changes in Python 3.9.1 1478=============================== 1479 1480typing 1481------ 1482 1483The behavior of :class:`typing.Literal` was changed to conform with :pep:`586` 1484and to match the behavior of static type checkers specified in the PEP. 1485 14861. ``Literal`` now de-duplicates parameters. 14872. Equality comparisons between ``Literal`` objects are now order independent. 14883. ``Literal`` comparisons now respect types. For example, 1489 ``Literal[0] == Literal[False]`` previously evaluated to ``True``. It is 1490 now ``False``. To support this change, the internally used type cache now 1491 supports differentiating types. 14924. ``Literal`` objects will now raise a :exc:`TypeError` exception during 1493 equality comparisons if any of their parameters are not :term:`hashable`. 1494 Note that declaring ``Literal`` with mutable parameters will not throw 1495 an error:: 1496 1497 >>> from typing import Literal 1498 >>> Literal[{0}] 1499 >>> Literal[{0}] == Literal[{False}] 1500 Traceback (most recent call last): 1501 File "<stdin>", line 1, in <module> 1502 TypeError: unhashable type: 'set' 1503 1504(Contributed by Yurii Karabas in :issue:`42345`.) 1505 1506macOS 11.0 (Big Sur) and Apple Silicon Mac support 1507-------------------------------------------------- 1508 1509As of 3.9.1, Python now fully supports building and running on macOS 11.0 1510(Big Sur) and on Apple Silicon Macs (based on the ``ARM64`` architecture). 1511A new universal build variant, ``universal2``, is now available to natively 1512support both ``ARM64`` and ``Intel 64`` in one set of executables. Binaries 1513can also now be built on current versions of macOS to be deployed on a range 1514of older macOS versions (tested to 10.9) while making some newer OS 1515functions and options conditionally available based on the operating system 1516version in use at runtime ("weaklinking"). 1517 1518(Contributed by Ronald Oussoren and Lawrence D'Anna in :issue:`41100`.) 1519 1520Notable changes in Python 3.9.2 1521=============================== 1522 1523collections.abc 1524--------------- 1525 1526:class:`collections.abc.Callable` generic now flattens type parameters, similar 1527to what :data:`typing.Callable` currently does. This means that 1528``collections.abc.Callable[[int, str], str]`` will have ``__args__`` of 1529``(int, str, str)``; previously this was ``([int, str], str)``. To allow this 1530change, :class:`types.GenericAlias` can now be subclassed, and a subclass will 1531be returned when subscripting the :class:`collections.abc.Callable` type. 1532Code which accesses the arguments via :func:`typing.get_args` or ``__args__`` 1533need to account for this change. A :exc:`DeprecationWarning` may be emitted for 1534invalid forms of parameterizing :class:`collections.abc.Callable` which may have 1535passed silently in Python 3.9.1. This :exc:`DeprecationWarning` will 1536become a :exc:`TypeError` in Python 3.10. 1537(Contributed by Ken Jin in :issue:`42195`.) 1538 1539urllib.parse 1540------------ 1541 1542Earlier Python versions allowed using both ``;`` and ``&`` as 1543query parameter separators in :func:`urllib.parse.parse_qs` and 1544:func:`urllib.parse.parse_qsl`. Due to security concerns, and to conform with 1545newer W3C recommendations, this has been changed to allow only a single 1546separator key, with ``&`` as the default. This change also affects 1547:func:`cgi.parse` and :func:`cgi.parse_multipart` as they use the affected 1548functions internally. For more details, please see their respective 1549documentation. 1550(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.) 1551