• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1****************************
2  What's New In Python 3.9
3****************************
4
5:Release: |release|
6:Date: |today|
7:Editor: Łukasz Langa
8
9.. Rules for maintenance:
10
11   * Anyone can add text to this document.  Your text might get
12   rewritten to some degree.
13
14   * The maintainer will go through Misc/NEWS periodically and add
15   changes; it's therefore more important to add your changes to
16   Misc/NEWS than to this file.
17
18   * This is not a complete list of every single change; completeness
19   is the purpose of Misc/NEWS.  Some changes will be too small
20   or esoteric to include.  If such a change is added to the text,
21   it might get removed during final editing.
22
23   * If you want to draw your new text to the attention of the
24   maintainer, add 'XXX' to the beginning of the paragraph or
25   section.
26
27   * It's OK to just add a fragmentary note about a change.  For
28   example: "XXX Describe the transmogrify() function added to the
29   socket module."  The maintainer will research the change and
30   write the necessary text.
31
32   * You can comment out your additions if you like, but it's not
33   necessary (especially when a final release is some months away).
34
35   * Credit the author of a patch or bugfix.   Just the name is
36   sufficient; the e-mail address isn't necessary.
37
38   * It's helpful to add the bug/patch number as a comment:
39
40   XXX Describe the transmogrify() function added to the socket
41   module.
42   (Contributed by P.Y. Developer in :issue:`12345`.)
43
44   This saves the maintainer the effort of going through the Mercurial log
45   when researching a change.
46
47This article explains the new features in Python 3.9, compared to 3.8.
48Python 3.9 was released on October 5th, 2020.
49
50For full details, see the :ref:`changelog <changelog>`.
51
52.. seealso::
53
54    :pep:`596` - Python 3.9 Release Schedule
55
56
57Summary -- Release highlights
58=============================
59
60.. This section singles out the most important changes in Python 3.9.
61   Brevity is key.
62
63New syntax features:
64
65* :pep:`584`, union operators added to ``dict``;
66* :pep:`585`, type hinting generics in standard collections;
67* :pep:`614`, relaxed grammar restrictions on decorators.
68
69New built-in features:
70
71* :pep:`616`, string methods to remove prefixes and suffixes.
72
73New features in the standard library:
74
75* :pep:`593`, flexible function and variable annotations;
76* :func:`os.pidfd_open` added that allows process management without races
77  and signals.
78
79Interpreter improvements:
80
81* :pep:`573`, fast access to module state from methods of C extension
82  types;
83* :pep:`617`, CPython now uses a new parser based on PEG;
84* a number of Python builtins (range, tuple, set, frozenset, list, dict) are
85  now sped up using :pep:`590` vectorcall;
86* garbage collection does not block on resurrected objects;
87* a number of Python modules (:mod:`_abc`, :mod:`audioop`, :mod:`_bz2`,
88  :mod:`_codecs`, :mod:`_contextvars`, :mod:`_crypt`, :mod:`_functools`,
89  :mod:`_json`, :mod:`_locale`, :mod:`math`, :mod:`operator`, :mod:`resource`,
90  :mod:`time`, :mod:`_weakref`) now use multiphase initialization as defined
91  by PEP 489;
92* a number of standard library modules (:mod:`audioop`, :mod:`ast`, :mod:`grp`,
93  :mod:`_hashlib`, :mod:`pwd`, :mod:`_posixsubprocess`, :mod:`random`,
94  :mod:`select`, :mod:`struct`, :mod:`termios`, :mod:`zlib`) are now using
95  the stable ABI defined by PEP 384.
96
97New library modules:
98
99* :pep:`615`, the IANA Time Zone Database is now present in the standard
100  library in the :mod:`zoneinfo` module;
101* an implementation of a topological sort of a graph is now provided in
102  the new :mod:`graphlib` module.
103
104Release process changes:
105
106* :pep:`602`, CPython adopts an annual release cycle.
107
108
109You should check for DeprecationWarning in your code
110====================================================
111
112When Python 2.7 was still supported, a lot of functionality in Python 3
113was kept for backward compatibility with Python 2.7. With the end of Python
1142 support, these backward compatibility layers have been removed, or will
115be removed soon. Most of them emitted a :exc:`DeprecationWarning` warning for
116several years. For example, using ``collections.Mapping`` instead of
117``collections.abc.Mapping`` emits a :exc:`DeprecationWarning` since Python
1183.3, released in 2012.
119
120Test your application with the :option:`-W` ``default`` command-line option to see
121:exc:`DeprecationWarning` and :exc:`PendingDeprecationWarning`, or even with
122:option:`-W` ``error`` to treat them as errors. :ref:`Warnings Filter
123<warning-filter>` can be used to ignore warnings from third-party code.
124
125Python 3.9 is the last version providing those Python 2 backward compatibility
126layers, to give more time to Python projects maintainers to organize the
127removal of the Python 2 support and add support for Python 3.9.
128
129Aliases to :ref:`Abstract Base Classes <collections-abstract-base-classes>` in
130the :mod:`collections` module, like ``collections.Mapping`` alias to
131:class:`collections.abc.Mapping`, are kept for one last release for backward
132compatibility. They will be removed from Python 3.10.
133
134More generally, try to run your tests in the :ref:`Python Development Mode
135<devmode>` which helps to prepare your code to make it compatible with the
136next Python version.
137
138Note: a number of pre-existing deprecations were removed in this version of
139Python as well. Consult the :ref:`removed-in-python-39` section.
140
141
142New Features
143============
144
145Dictionary Merge & Update Operators
146-----------------------------------
147
148Merge (``|``) and update (``|=``) operators have been added to the built-in
149:class:`dict` class. Those complement the existing ``dict.update`` and
150``{**d1, **d2}`` methods of merging dictionaries.
151
152Example::
153
154  >>> x = {"key1": "value1 from x", "key2": "value2 from x"}
155  >>> y = {"key2": "value2 from y", "key3": "value3 from y"}
156  >>> x | y
157  {'key1': 'value1 from x', 'key2': 'value2 from y', 'key3': 'value3 from y'}
158  >>> y | x
159  {'key2': 'value2 from x', 'key3': 'value3 from y', 'key1': 'value1 from x'}
160
161See :pep:`584` for a full description.
162(Contributed by Brandt Bucher in :issue:`36144`.)
163
164New String Methods to Remove Prefixes and Suffixes
165--------------------------------------------------
166
167:meth:`str.removeprefix(prefix)<str.removeprefix>` and
168:meth:`str.removesuffix(suffix)<str.removesuffix>` have been added
169to easily remove an unneeded prefix or a suffix from a string. Corresponding
170``bytes``, ``bytearray``, and ``collections.UserString`` methods have also been
171added. See :pep:`616` for a full description. (Contributed by Dennis Sweeney in
172:issue:`39939`.)
173
174Type Hinting Generics in Standard Collections
175---------------------------------------------
176
177In type annotations you can now use built-in collection types such as
178``list`` and ``dict`` as generic types instead of importing the
179corresponding capitalized types (e.g. ``List`` or ``Dict``) from
180``typing``.  Some other types in the standard library are also now generic,
181for example ``queue.Queue``.
182
183Example:
184
185.. code-block:: python
186
187   def greet_all(names: list[str]) -> None:
188       for name in names:
189           print("Hello", name)
190
191See :pep:`585` for more details.  (Contributed by Guido van Rossum,
192Ethan Smith, and Batuhan Taşkaya in :issue:`39481`.)
193
194New Parser
195----------
196
197Python 3.9 uses a new parser, based on `PEG
198<https://en.wikipedia.org/wiki/Parsing_expression_grammar>`_ instead
199of `LL(1) <https://en.wikipedia.org/wiki/LL_parser>`_.  The new
200parser's performance is roughly comparable to that of the old parser,
201but the PEG formalism is more flexible than LL(1) when it comes to
202designing new language features.  We'll start using this flexibility
203in Python 3.10 and later.
204
205The :mod:`ast` module uses the new parser and produces the same AST as
206the old parser.
207
208In Python 3.10, the old parser will be deleted and so will all
209functionality that depends on it (primarily the :mod:`parser` module,
210which has long been deprecated).  In Python 3.9 *only*, you can switch
211back to the LL(1) parser using a command line switch (``-X
212oldparser``) or an environment variable (``PYTHONOLDPARSER=1``).
213
214See :pep:`617` for more details.  (Contributed by Guido van Rossum,
215Pablo Galindo and Lysandros Nikolaou in :issue:`40334`.)
216
217
218Other Language Changes
219======================
220
221* :func:`__import__` now raises :exc:`ImportError` instead of
222  :exc:`ValueError`, which used to occur when a relative import went past
223  its top-level package.
224  (Contributed by Ngalim Siregar in :issue:`37444`.)
225
226* Python now gets the absolute path of the script filename specified on
227  the command line (ex: ``python3 script.py``): the ``__file__`` attribute of
228  the :mod:`__main__` module became an absolute path, rather than a relative
229  path. These paths now remain valid after the current directory is changed
230  by :func:`os.chdir`. As a side effect, the traceback also displays the
231  absolute path for :mod:`__main__` module frames in this case.
232  (Contributed by Victor Stinner in :issue:`20443`.)
233
234* In the :ref:`Python Development Mode <devmode>` and in :ref:`debug build <debug-build>`, the
235  *encoding* and *errors* arguments are now checked for string encoding and
236  decoding operations. Examples: :func:`open`, :meth:`str.encode` and
237  :meth:`bytes.decode`.
238
239  By default, for best performance, the *errors* argument is only checked at
240  the first encoding/decoding error and the *encoding* argument is sometimes
241  ignored for empty strings.
242  (Contributed by Victor Stinner in :issue:`37388`.)
243
244* ``"".replace("", s, n)`` now returns ``s`` instead of an empty string for
245  all non-zero ``n``.  It is now consistent with ``"".replace("", s)``.
246  There are similar changes for :class:`bytes` and :class:`bytearray` objects.
247  (Contributed by Serhiy Storchaka in :issue:`28029`.)
248
249* Any valid expression can now be used as a :term:`decorator`.  Previously, the
250  grammar was much more restrictive.  See :pep:`614` for details.
251  (Contributed by Brandt Bucher in :issue:`39702`.)
252
253* Improved help for the :mod:`typing` module. Docstrings are now shown for
254  all special forms and special generic aliases (like ``Union`` and ``List``).
255  Using :func:`help` with generic alias like ``List[int]`` will show the help
256  for the correspondent concrete type (``list`` in this case).
257  (Contributed by Serhiy Storchaka in :issue:`40257`.)
258
259* Parallel running of :meth:`~agen.aclose` / :meth:`~agen.asend` /
260  :meth:`~agen.athrow` is now prohibited, and ``ag_running`` now reflects
261  the actual running status of the async generator.
262  (Contributed by Yury Selivanov in :issue:`30773`.)
263
264* Unexpected errors in calling the ``__iter__`` method are no longer masked by
265  ``TypeError`` in the :keyword:`in` operator and functions
266  :func:`~operator.contains`, :func:`~operator.indexOf` and
267  :func:`~operator.countOf` of the :mod:`operator` module.
268  (Contributed by Serhiy Storchaka in :issue:`40824`.)
269
270* Unparenthesized lambda expressions can no longer be the expression part in an
271  ``if`` clause in comprehensions and generator expressions. See :issue:`41848`
272  and :issue:`43755` for details.
273
274
275New Modules
276===========
277
278zoneinfo
279--------
280
281The :mod:`zoneinfo` module brings support for the IANA time zone database to
282the standard library. It adds :class:`zoneinfo.ZoneInfo`, a concrete
283:class:`datetime.tzinfo` implementation backed by the system's time zone data.
284
285Example::
286
287    >>> from zoneinfo import ZoneInfo
288    >>> from datetime import datetime, timedelta
289
290    >>> # Daylight saving time
291    >>> dt = datetime(2020, 10, 31, 12, tzinfo=ZoneInfo("America/Los_Angeles"))
292    >>> print(dt)
293    2020-10-31 12:00:00-07:00
294    >>> dt.tzname()
295    'PDT'
296
297    >>> # Standard time
298    >>> dt += timedelta(days=7)
299    >>> print(dt)
300    2020-11-07 12:00:00-08:00
301    >>> print(dt.tzname())
302    PST
303
304
305As a fall-back source of data for platforms that don't ship the IANA database,
306the |tzdata|_ module was released as a first-party package -- distributed via
307PyPI and maintained by the CPython core team.
308
309.. |tzdata| replace:: ``tzdata``
310.. _tzdata: https://pypi.org/project/tzdata/
311
312.. seealso::
313
314    :pep:`615` -- Support for the IANA Time Zone Database in the Standard Library
315        PEP written and implemented by Paul Ganssle
316
317
318graphlib
319---------
320
321A new module, :mod:`graphlib`, was added that contains the
322:class:`graphlib.TopologicalSorter` class to offer functionality to perform
323topological sorting of graphs. (Contributed by Pablo Galindo, Tim Peters and
324Larry Hastings in :issue:`17005`.)
325
326
327Improved Modules
328================
329
330ast
331---
332
333Added the *indent* option to :func:`~ast.dump` which allows it to produce a
334multiline indented output.
335(Contributed by Serhiy Storchaka in :issue:`37995`.)
336
337Added :func:`ast.unparse` as a function in the :mod:`ast` module that can
338be used to unparse an :class:`ast.AST` object and produce a string with code
339that would produce an equivalent :class:`ast.AST` object when parsed.
340(Contributed by Pablo Galindo and Batuhan Taskaya in :issue:`38870`.)
341
342Added docstrings to AST nodes that contains the ASDL signature used to
343construct that node. (Contributed by Batuhan Taskaya in :issue:`39638`.)
344
345asyncio
346-------
347
348Due to significant security concerns, the *reuse_address* parameter of
349:meth:`asyncio.loop.create_datagram_endpoint` is no longer supported. This is
350because of the behavior of the socket option ``SO_REUSEADDR`` in UDP. For more
351details, see the documentation for ``loop.create_datagram_endpoint()``.
352(Contributed by Kyle Stanley, Antoine Pitrou, and Yury Selivanov in
353:issue:`37228`.)
354
355Added a new :term:`coroutine` :meth:`~asyncio.loop.shutdown_default_executor`
356that schedules a shutdown for the default executor that waits on the
357:class:`~concurrent.futures.ThreadPoolExecutor` to finish closing. Also,
358:func:`asyncio.run` has been updated to use the new :term:`coroutine`.
359(Contributed by Kyle Stanley in :issue:`34037`.)
360
361Added :class:`asyncio.PidfdChildWatcher`, a Linux-specific child watcher
362implementation that polls process file descriptors. (:issue:`38692`)
363
364Added a new :term:`coroutine` :func:`asyncio.to_thread`. It is mainly used for
365running IO-bound functions in a separate thread to avoid blocking the event
366loop, and essentially works as a high-level version of
367:meth:`~asyncio.loop.run_in_executor` that can directly take keyword arguments.
368(Contributed by Kyle Stanley and Yury Selivanov in :issue:`32309`.)
369
370When cancelling the task due to a timeout, :meth:`asyncio.wait_for` will now
371wait until the cancellation is complete also in the case when *timeout* is
372<= 0, like it does with positive timeouts.
373(Contributed by Elvis Pranskevichus in :issue:`32751`.)
374
375:mod:`asyncio` now raises :exc:`TyperError` when calling incompatible
376methods with an :class:`ssl.SSLSocket` socket.
377(Contributed by Ido Michael in :issue:`37404`.)
378
379compileall
380----------
381
382Added new possibility to use hardlinks for duplicated ``.pyc`` files: *hardlink_dupes* parameter and --hardlink-dupes command line option.
383(Contributed by  Lumír 'Frenzy' Balhar in :issue:`40495`.)
384
385Added new options for path manipulation in resulting ``.pyc`` files: *stripdir*, *prependdir*, *limit_sl_dest* parameters and -s, -p, -e command line options.
386Added the possibility to specify the option for an optimization level multiple times.
387(Contributed by Lumír 'Frenzy' Balhar in :issue:`38112`.)
388
389concurrent.futures
390------------------
391
392Added a new *cancel_futures* parameter to
393:meth:`concurrent.futures.Executor.shutdown` that cancels all pending futures
394which have not started running, instead of waiting for them to complete before
395shutting down the executor.
396(Contributed by Kyle Stanley in :issue:`39349`.)
397
398Removed daemon threads from :class:`~concurrent.futures.ThreadPoolExecutor`
399and :class:`~concurrent.futures.ProcessPoolExecutor`. This improves
400compatibility with subinterpreters and predictability in their shutdown
401processes. (Contributed by Kyle Stanley in :issue:`39812`.)
402
403Workers in :class:`~concurrent.futures.ProcessPoolExecutor` are now spawned on
404demand, only when there are no available idle workers to reuse. This optimizes
405startup overhead and reduces the amount of lost CPU time to idle workers.
406(Contributed by Kyle Stanley in :issue:`39207`.)
407
408curses
409------
410
411Added :func:`curses.get_escdelay`, :func:`curses.set_escdelay`,
412:func:`curses.get_tabsize`, and :func:`curses.set_tabsize` functions.
413(Contributed by Anthony Sottile in :issue:`38312`.)
414
415datetime
416--------
417The :meth:`~datetime.date.isocalendar()` of :class:`datetime.date`
418and :meth:`~datetime.datetime.isocalendar()` of :class:`datetime.datetime`
419methods now returns a :func:`~collections.namedtuple` instead of a :class:`tuple`.
420(Contributed by Dong-hee Na in :issue:`24416`.)
421
422distutils
423---------
424
425The :command:`upload` command now creates SHA2-256 and Blake2b-256 hash
426digests. It skips MD5 on platforms that block MD5 digest.
427(Contributed by Christian Heimes in :issue:`40698`.)
428
429fcntl
430-----
431
432Added constants :data:`~fcntl.F_OFD_GETLK`, :data:`~fcntl.F_OFD_SETLK`
433and :data:`~fcntl.F_OFD_SETLKW`.
434(Contributed by Dong-hee Na in :issue:`38602`.)
435
436ftplib
437-------
438
439:class:`~ftplib.FTP` and :class:`~ftplib.FTP_TLS` now raise a :class:`ValueError`
440if the given timeout for their constructor is zero to prevent the creation of
441a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)
442
443gc
444--
445
446When the garbage collector makes a collection in which some objects resurrect
447(they are reachable from outside the isolated cycles after the finalizers have
448been executed), do not block the collection of all objects that are still
449unreachable. (Contributed by Pablo Galindo and Tim Peters in :issue:`38379`.)
450
451Added a new function :func:`gc.is_finalized` to check if an object has been
452finalized by the garbage collector. (Contributed by Pablo Galindo in
453:issue:`39322`.)
454
455hashlib
456-------
457
458The :mod:`hashlib` module can now use SHA3 hashes and SHAKE XOF from OpenSSL
459when available.
460(Contributed by Christian Heimes in :issue:`37630`.)
461
462Builtin hash modules can now be disabled with
463``./configure --without-builtin-hashlib-hashes`` or selectively enabled with
464e.g. ``./configure --with-builtin-hashlib-hashes=sha3,blake2`` to force use
465of OpenSSL based implementation.
466(Contributed by Christian Heimes in :issue:`40479`)
467
468
469http
470----
471
472HTTP status codes ``103 EARLY_HINTS``, ``418 IM_A_TEAPOT`` and ``425 TOO_EARLY`` are added to
473:class:`http.HTTPStatus`. (Contributed by Dong-hee Na in :issue:`39509` and Ross Rhodes in :issue:`39507`.)
474
475IDLE and idlelib
476----------------
477
478Added option to toggle cursor blink off.  (Contributed by Zackery Spytz
479in :issue:`4603`.)
480
481Escape key now closes IDLE completion windows.  (Contributed by Johnny
482Najera in :issue:`38944`.)
483
484Added keywords to module name completion list.  (Contributed by Terry J.
485Reedy in :issue:`37765`.)
486
487The changes above have been backported to 3.8 maintenance releases.
488
489imaplib
490-------
491
492:class:`~imaplib.IMAP4` and :class:`~imaplib.IMAP4_SSL` now have
493an optional *timeout* parameter for their constructors.
494Also, the :meth:`~imaplib.IMAP4.open` method now has an optional *timeout* parameter
495with this change. The overridden methods of :class:`~imaplib.IMAP4_SSL` and
496:class:`~imaplib.IMAP4_stream` were applied to this change.
497(Contributed by Dong-hee Na in :issue:`38615`.)
498
499:meth:`imaplib.IMAP4.unselect` is added.
500:meth:`imaplib.IMAP4.unselect` frees server's resources associated with the
501selected mailbox and returns the server to the authenticated
502state. This command performs the same actions as :meth:`imaplib.IMAP4.close`, except
503that no messages are permanently removed from the currently
504selected mailbox. (Contributed by Dong-hee Na in :issue:`40375`.)
505
506importlib
507---------
508
509To improve consistency with import statements, :func:`importlib.util.resolve_name`
510now raises :exc:`ImportError` instead of :exc:`ValueError` for invalid relative
511import attempts.
512(Contributed by Ngalim Siregar in :issue:`37444`.)
513
514Import loaders which publish immutable module objects can now publish
515immutable packages in addition to individual modules.
516(Contributed by Dino Viehland in :issue:`39336`.)
517
518Added :func:`importlib.resources.files` function with support for
519subdirectories in package data, matching backport in ``importlib_resources``
520version 1.5.
521(Contributed by Jason R. Coombs in :issue:`39791`.)
522
523Refreshed ``importlib.metadata`` from ``importlib_metadata`` version 1.6.1.
524
525inspect
526-------
527
528:attr:`inspect.BoundArguments.arguments` is changed from ``OrderedDict`` to regular
529dict.  (Contributed by Inada Naoki in :issue:`36350` and :issue:`39775`.)
530
531ipaddress
532---------
533
534:mod:`ipaddress` now supports IPv6 Scoped Addresses (IPv6 address with suffix ``%<scope_id>``).
535
536Scoped IPv6 addresses can be parsed using :class:`ipaddress.IPv6Address`.
537If present, scope zone ID is available through the :attr:`~ipaddress.IPv6Address.scope_id` attribute.
538(Contributed by Oleksandr Pavliuk in :issue:`34788`.)
539
540Starting with Python 3.9.5 the :mod:`ipaddress` module no longer
541accepts any leading zeros in IPv4 address strings.
542(Contributed by Christian Heimes in :issue:`36384`).
543
544math
545----
546
547Expanded the :func:`math.gcd` function to handle multiple arguments.
548Formerly, it only supported two arguments.
549(Contributed by Serhiy Storchaka in :issue:`39648`.)
550
551Added :func:`math.lcm`: return the least common multiple of specified arguments.
552(Contributed by Mark Dickinson, Ananthakrishnan and Serhiy Storchaka in
553:issue:`39479` and :issue:`39648`.)
554
555Added :func:`math.nextafter`: return the next floating-point value after *x*
556towards *y*.
557(Contributed by Victor Stinner in :issue:`39288`.)
558
559Added :func:`math.ulp`: return the value of the least significant bit
560of a float.
561(Contributed by Victor Stinner in :issue:`39310`.)
562
563multiprocessing
564---------------
565
566The :class:`multiprocessing.SimpleQueue` class has a new
567:meth:`~multiprocessing.SimpleQueue.close` method to explicitly close the
568queue.
569(Contributed by Victor Stinner in :issue:`30966`.)
570
571nntplib
572-------
573
574:class:`~nntplib.NNTP` and :class:`~nntplib.NNTP_SSL` now raise a :class:`ValueError`
575if the given timeout for their constructor is zero to prevent the creation of
576a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)
577
578os
579--
580
581Added :data:`~os.CLD_KILLED` and :data:`~os.CLD_STOPPED` for :attr:`si_code`.
582(Contributed by Dong-hee Na in :issue:`38493`.)
583
584Exposed the Linux-specific :func:`os.pidfd_open` (:issue:`38692`) and
585:data:`os.P_PIDFD` (:issue:`38713`) for process management with file
586descriptors.
587
588The :func:`os.unsetenv` function is now also available on Windows.
589(Contributed by Victor Stinner in :issue:`39413`.)
590
591The :func:`os.putenv` and :func:`os.unsetenv` functions are now always
592available.
593(Contributed by Victor Stinner in :issue:`39395`.)
594
595Added :func:`os.waitstatus_to_exitcode` function:
596convert a wait status to an exit code.
597(Contributed by Victor Stinner in :issue:`40094`.)
598
599pathlib
600-------
601
602Added :meth:`pathlib.Path.readlink()` which acts similarly to
603:func:`os.readlink`.
604(Contributed by Girts Folkmanis in :issue:`30618`)
605
606pdb
607---
608
609On Windows now :class:`~pdb.Pdb` supports ``~/.pdbrc``.
610(Contributed by Tim Hopper and Dan Lidral-Porter in :issue:`20523`.)
611
612poplib
613------
614
615:class:`~poplib.POP3` and :class:`~poplib.POP3_SSL` now raise a :class:`ValueError`
616if the given timeout for their constructor is zero to prevent the creation of
617a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)
618
619pprint
620------
621
622:mod:`pprint` can now pretty-print :class:`types.SimpleNamespace`.
623(Contributed by Carl Bordum Hansen in :issue:`37376`.)
624
625pydoc
626-----
627
628The documentation string is now shown not only for class, function,
629method etc, but for any object that has its own ``__doc__`` attribute.
630(Contributed by Serhiy Storchaka in :issue:`40257`.)
631
632random
633------
634
635Added a new :attr:`random.Random.randbytes` method: generate random bytes.
636(Contributed by Victor Stinner in :issue:`40286`.)
637
638signal
639------
640
641Exposed the Linux-specific :func:`signal.pidfd_send_signal` for sending to
642signals to a process using a file descriptor instead of a pid. (:issue:`38712`)
643
644smtplib
645-------
646
647:class:`~smtplib.SMTP` and :class:`~smtplib.SMTP_SSL` now raise a :class:`ValueError`
648if the given timeout for their constructor is zero to prevent the creation of
649a non-blocking socket. (Contributed by Dong-hee Na in :issue:`39259`.)
650
651:class:`~smtplib.LMTP` constructor  now has an optional *timeout* parameter.
652(Contributed by Dong-hee Na in :issue:`39329`.)
653
654socket
655------
656
657The :mod:`socket` module now exports the :data:`~socket.CAN_RAW_JOIN_FILTERS`
658constant on Linux 4.1 and greater.
659(Contributed by Stefan Tatschner and Zackery Spytz in :issue:`25780`.)
660
661The socket module now supports the :data:`~socket.CAN_J1939` protocol on
662platforms that support it.  (Contributed by Karl Ding in :issue:`40291`.)
663
664The socket module now has the :func:`socket.send_fds` and
665:func:`socket.recv_fds` functions. (Contributed by Joannah Nanjekye, Shinya
666Okano and Victor Stinner in :issue:`28724`.)
667
668
669time
670----
671
672On AIX, :func:`~time.thread_time` is now implemented with ``thread_cputime()``
673which has nanosecond resolution, rather than
674``clock_gettime(CLOCK_THREAD_CPUTIME_ID)`` which has a resolution of 10 ms.
675(Contributed by Batuhan Taskaya in :issue:`40192`)
676
677sys
678---
679
680Added a new :attr:`sys.platlibdir` attribute: name of the platform-specific
681library directory. It is used to build the path of standard library and the
682paths of installed extension modules. It is equal to ``"lib"`` on most
683platforms.  On Fedora and SuSE, it is equal to ``"lib64"`` on 64-bit platforms.
684(Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis and Victor Stinner in :issue:`1294959`.)
685
686Previously, :attr:`sys.stderr` was block-buffered when non-interactive. Now
687``stderr`` defaults to always being line-buffered.
688(Contributed by Jendrik Seipp in :issue:`13601`.)
689
690tracemalloc
691-----------
692
693Added :func:`tracemalloc.reset_peak` to set the peak size of traced memory
694blocks to the current size, to measure the peak of specific pieces of code.
695(Contributed by Huon Wilson in :issue:`40630`.)
696
697typing
698------
699
700:pep:`593` introduced an :data:`typing.Annotated` type to decorate existing
701types with context-specific metadata and new ``include_extras`` parameter to
702:func:`typing.get_type_hints` to access the metadata at runtime. (Contributed
703by Till Varoquaux and Konstantin Kashin.)
704
705unicodedata
706-----------
707
708The Unicode database has been updated to version 13.0.0. (:issue:`39926`).
709
710venv
711----
712
713The activation scripts provided by :mod:`venv` now all specify their prompt
714customization consistently by always using the value specified by
715``__VENV_PROMPT__``. Previously some scripts unconditionally used
716``__VENV_PROMPT__``, others only if it happened to be set (which was the default
717case), and one used ``__VENV_NAME__`` instead.
718(Contributed by Brett Cannon in :issue:`37663`.)
719
720xml
721---
722
723White space characters within attributes are now preserved when serializing
724:mod:`xml.etree.ElementTree` to XML file. EOLNs are no longer normalized
725to "\n". This is the result of discussion about how to interpret
726section 2.11 of XML spec.
727(Contributed by Mefistotelis in :issue:`39011`.)
728
729
730Optimizations
731=============
732
733* Optimized the idiom for assignment a temporary variable in comprehensions.
734  Now ``for y in [expr]`` in comprehensions is as fast as a simple assignment
735  ``y = expr``.  For example:
736
737     sums = [s for s in [0] for x in data for s in [s + x]]
738
739  Unlike the ``:=`` operator this idiom does not leak a variable to the
740  outer scope.
741
742  (Contributed by Serhiy Storchaka in :issue:`32856`.)
743
744* Optimized signal handling in multithreaded applications. If a thread different
745  than the main thread gets a signal, the bytecode evaluation loop is no longer
746  interrupted at each bytecode instruction to check for pending signals which
747  cannot be handled. Only the main thread of the main interpreter can handle
748  signals.
749
750  Previously, the bytecode evaluation loop was interrupted at each instruction
751  until the main thread handles signals.
752  (Contributed by Victor Stinner in :issue:`40010`.)
753
754* Optimized the :mod:`subprocess` module on FreeBSD using ``closefrom()``.
755  (Contributed by Ed Maste, Conrad Meyer, Kyle Evans, Kubilay Kocak and Victor
756  Stinner in :issue:`38061`.)
757
758* :c:func:`PyLong_FromDouble` is now up to 1.87x faster for values that
759  fit into :c:type:`long`.
760  (Contributed by Sergey Fedoseev in :issue:`37986`.)
761
762* A number of Python builtins (:class:`range`, :class:`tuple`, :class:`set`,
763  :class:`frozenset`, :class:`list`, :class:`dict`) are now sped up by using
764  :pep:`590` vectorcall protocol.
765  (Contributed by Dong-hee Na, Mark Shannon, Jeroen Demeyer and Petr Viktorin in :issue:`37207`.)
766
767* Optimized :func:`~set.difference_update` for the case when the other set
768  is much larger than the base set.
769  (Suggested by Evgeny Kapun with code contributed by Michele Orrù in :issue:`8425`.)
770
771* Python's small object allocator (``obmalloc.c``) now allows (no more than)
772  one empty arena to remain available for immediate reuse, without returning
773  it to the OS.  This prevents thrashing in simple loops where an arena could
774  be created and destroyed anew on each iteration.
775  (Contributed by Tim Peters in :issue:`37257`.)
776
777* :term:`floor division` of float operation now has a better performance. Also
778  the message of :exc:`ZeroDivisionError` for this operation is updated.
779  (Contributed by Dong-hee Na in :issue:`39434`.)
780
781* Decoding short ASCII strings with UTF-8 and ascii codecs is now about
782  15% faster.  (Contributed by Inada Naoki in :issue:`37348`.)
783
784Here's a summary of performance improvements from Python 3.4 through Python 3.9:
785
786.. code-block:: none
787
788    Python version                       3.4     3.5     3.6     3.7     3.8    3.9
789    --------------                       ---     ---     ---     ---     ---    ---
790
791    Variable and attribute read access:
792        read_local                       7.1     7.1     5.4     5.1     3.9    3.9
793        read_nonlocal                    7.1     8.1     5.8     5.4     4.4    4.5
794        read_global                     15.5    19.0    14.3    13.6     7.6    7.8
795        read_builtin                    21.1    21.6    18.5    19.0     7.5    7.8
796        read_classvar_from_class        25.6    26.5    20.7    19.5    18.4   17.9
797        read_classvar_from_instance     22.8    23.5    18.8    17.1    16.4   16.9
798        read_instancevar                32.4    33.1    28.0    26.3    25.4   25.3
799        read_instancevar_slots          27.8    31.3    20.8    20.8    20.2   20.5
800        read_namedtuple                 73.8    57.5    45.0    46.8    18.4   18.7
801        read_boundmethod                37.6    37.9    29.6    26.9    27.7   41.1
802
803    Variable and attribute write access:
804        write_local                      8.7     9.3     5.5     5.3     4.3    4.3
805        write_nonlocal                  10.5    11.1     5.6     5.5     4.7    4.8
806        write_global                    19.7    21.2    18.0    18.0    15.8   16.7
807        write_classvar                  92.9    96.0   104.6   102.1    39.2   39.8
808        write_instancevar               44.6    45.8    40.0    38.9    35.5   37.4
809        write_instancevar_slots         35.6    36.1    27.3    26.6    25.7   25.8
810
811    Data structure read access:
812        read_list                       24.2    24.5    20.8    20.8    19.0   19.5
813        read_deque                      24.7    25.5    20.2    20.6    19.8   20.2
814        read_dict                       24.3    25.7    22.3    23.0    21.0   22.4
815        read_strdict                    22.6    24.3    19.5    21.2    18.9   21.5
816
817    Data structure write access:
818        write_list                      27.1    28.5    22.5    21.6    20.0   20.0
819        write_deque                     28.7    30.1    22.7    21.8    23.5   21.7
820        write_dict                      31.4    33.3    29.3    29.2    24.7   25.4
821        write_strdict                   28.4    29.9    27.5    25.2    23.1   24.5
822
823    Stack (or queue) operations:
824        list_append_pop                 93.4   112.7    75.4    74.2    50.8   50.6
825        deque_append_pop                43.5    57.0    49.4    49.2    42.5   44.2
826        deque_append_popleft            43.7    57.3    49.7    49.7    42.8   46.4
827
828    Timing loop:
829        loop_overhead                    0.5     0.6     0.4     0.3     0.3    0.3
830
831These results were generated from the variable access benchmark script at:
832``Tools/scripts/var_access_benchmark.py``. The benchmark script displays timings
833in nanoseconds.  The benchmarks were measured on an
834`Intel® Core™ i7-4960HQ processor
835<https://ark.intel.com/content/www/us/en/ark/products/76088/intel-core-i7-4960hq-processor-6m-cache-up-to-3-80-ghz.html>`_
836running the macOS 64-bit builds found at
837`python.org <https://www.python.org/downloads/mac-osx/>`_.
838
839
840Deprecated
841==========
842
843* The distutils ``bdist_msi`` command is now deprecated, use
844  ``bdist_wheel`` (wheel packages) instead.
845  (Contributed by Hugo van Kemenade in :issue:`39586`.)
846
847* Currently :func:`math.factorial` accepts :class:`float` instances with
848  non-negative integer values (like ``5.0``).  It raises a :exc:`ValueError`
849  for non-integral and negative floats.  It is now deprecated.  In future
850  Python versions it will raise a :exc:`TypeError` for all floats.
851  (Contributed by Serhiy Storchaka in :issue:`37315`.)
852
853* The :mod:`parser` and :mod:`symbol` modules are deprecated and will be
854  removed in future versions of Python. For the majority of use cases,
855  users can leverage the Abstract Syntax Tree (AST) generation and compilation
856  stage, using the :mod:`ast` module.
857
858* The Public C API functions :c:func:`PyParser_SimpleParseStringFlags`,
859  :c:func:`PyParser_SimpleParseStringFlagsFilename`,
860  :c:func:`PyParser_SimpleParseFileFlags` and :c:func:`PyNode_Compile`
861  are deprecated and will be removed in Python 3.10 together with the old parser.
862
863* Using :data:`NotImplemented` in a boolean context has been deprecated,
864  as it is almost exclusively the result of incorrect rich comparator
865  implementations. It will be made a :exc:`TypeError` in a future version
866  of Python.
867  (Contributed by Josh Rosenberg in :issue:`35712`.)
868
869* The :mod:`random` module currently accepts any hashable type as a
870  possible seed value.  Unfortunately, some of those types are not
871  guaranteed to have a deterministic hash value.  After Python 3.9,
872  the module will restrict its seeds to :const:`None`, :class:`int`,
873  :class:`float`, :class:`str`, :class:`bytes`, and :class:`bytearray`.
874
875* Opening the :class:`~gzip.GzipFile` file for writing without specifying
876  the *mode* argument is deprecated.  In future Python versions it will always
877  be opened for reading by default.  Specify the *mode* argument for opening
878  it for writing and silencing a warning.
879  (Contributed by Serhiy Storchaka in :issue:`28286`.)
880
881* Deprecated the ``split()`` method of :class:`_tkinter.TkappType` in
882  favour of the ``splitlist()`` method which has more consistent and
883  predicable behavior.
884  (Contributed by Serhiy Storchaka in :issue:`38371`.)
885
886* The explicit passing of coroutine objects to :func:`asyncio.wait` has been
887  deprecated and will be removed in version 3.11.
888  (Contributed by Yury Selivanov and Kyle Stanley in :issue:`34790`.)
889
890* binhex4 and hexbin4 standards are now deprecated. The :mod:`binhex` module
891  and the following :mod:`binascii` functions are now deprecated:
892
893  * :func:`~binascii.b2a_hqx`, :func:`~binascii.a2b_hqx`
894  * :func:`~binascii.rlecode_hqx`, :func:`~binascii.rledecode_hqx`
895
896  (Contributed by Victor Stinner in :issue:`39353`.)
897
898* :mod:`ast` classes ``slice``, ``Index`` and ``ExtSlice`` are considered deprecated
899  and will be removed in future Python versions.  ``value`` itself should be
900  used instead of ``Index(value)``.  ``Tuple(slices, Load())`` should be
901  used instead of ``ExtSlice(slices)``.
902  (Contributed by Serhiy Storchaka in :issue:`34822`.)
903
904* :mod:`ast` classes ``Suite``, ``Param``, ``AugLoad`` and ``AugStore``
905  are considered deprecated and will be removed in future Python versions.
906  They were not generated by the parser and not accepted by the code
907  generator in Python 3.
908  (Contributed by Batuhan Taskaya in :issue:`39639` and :issue:`39969`
909  and Serhiy Storchaka in :issue:`39988`.)
910
911* The :c:func:`PyEval_InitThreads` and :c:func:`PyEval_ThreadsInitialized`
912  functions are now deprecated and will be removed in Python 3.11. Calling
913  :c:func:`PyEval_InitThreads` now does nothing. The :term:`GIL` is initialized
914  by :c:func:`Py_Initialize()` since Python 3.7.
915  (Contributed by Victor Stinner in :issue:`39877`.)
916
917* Passing ``None`` as the first argument to the :func:`shlex.split` function
918  has been deprecated.  (Contributed by Zackery Spytz in :issue:`33262`.)
919
920* :func:`smtpd.MailmanProxy` is now deprecated as it is unusable without
921  an external module, ``mailman``.  (Contributed by Samuel Colvin in :issue:`35800`.)
922
923* The :mod:`lib2to3` module now emits a :exc:`PendingDeprecationWarning`.
924  Python 3.9 switched to a PEG parser (see :pep:`617`), and Python 3.10 may
925  include new language syntax that is not parsable by lib2to3's LL(1) parser.
926  The ``lib2to3`` module may be removed from the standard library in a future
927  Python version. Consider third-party alternatives such as `LibCST`_ or
928  `parso`_.
929  (Contributed by Carl Meyer in :issue:`40360`.)
930
931* The *random* parameter of :func:`random.shuffle` has been deprecated.
932  (Contributed by Raymond Hettinger in :issue:`40465`)
933
934.. _LibCST: https://libcst.readthedocs.io/
935.. _parso: https://parso.readthedocs.io/
936
937.. _removed-in-python-39:
938
939Removed
940=======
941
942* The erroneous version at :data:`unittest.mock.__version__` has been removed.
943
944* :class:`nntplib.NNTP`: ``xpath()`` and ``xgtitle()`` methods have been removed.
945  These methods are deprecated since Python 3.3. Generally, these extensions
946  are not supported or not enabled by NNTP server administrators.
947  For ``xgtitle()``, please use :meth:`nntplib.NNTP.descriptions` or
948  :meth:`nntplib.NNTP.description` instead.
949  (Contributed by Dong-hee Na in :issue:`39366`.)
950
951* :class:`array.array`: ``tostring()`` and ``fromstring()`` methods have been
952  removed. They were aliases to ``tobytes()`` and ``frombytes()``, deprecated
953  since Python 3.2.
954  (Contributed by Victor Stinner in :issue:`38916`.)
955
956* The undocumented ``sys.callstats()`` function has been removed. Since Python
957  3.7, it was deprecated and always returned :const:`None`. It required a special
958  build option ``CALL_PROFILE`` which was already removed in Python 3.7.
959  (Contributed by Victor Stinner in :issue:`37414`.)
960
961* The ``sys.getcheckinterval()`` and ``sys.setcheckinterval()`` functions have
962  been removed. They were deprecated since Python 3.2. Use
963  :func:`sys.getswitchinterval` and :func:`sys.setswitchinterval` instead.
964  (Contributed by Victor Stinner in :issue:`37392`.)
965
966* The C function ``PyImport_Cleanup()`` has been removed. It was documented as:
967  "Empty the module table.  For internal use only."
968  (Contributed by Victor Stinner in :issue:`36710`.)
969
970* ``_dummy_thread`` and ``dummy_threading`` modules have been removed. These
971  modules were deprecated since Python 3.7 which requires threading support.
972  (Contributed by Victor Stinner in :issue:`37312`.)
973
974* ``aifc.openfp()`` alias to ``aifc.open()``, ``sunau.openfp()`` alias to
975  ``sunau.open()``, and ``wave.openfp()`` alias to :func:`wave.open()` have been
976  removed. They were deprecated since Python 3.7.
977  (Contributed by Victor Stinner in :issue:`37320`.)
978
979* The :meth:`~threading.Thread.isAlive()` method of :class:`threading.Thread`
980  has been removed. It was deprecated since Python 3.8.
981  Use :meth:`~threading.Thread.is_alive()` instead.
982  (Contributed by Dong-hee Na in :issue:`37804`.)
983
984* Methods ``getchildren()`` and ``getiterator()`` of classes
985  :class:`~xml.etree.ElementTree.ElementTree` and
986  :class:`~xml.etree.ElementTree.Element` in the :mod:`~xml.etree.ElementTree`
987  module have been removed.  They were deprecated in Python 3.2.
988  Use ``iter(x)`` or ``list(x)`` instead of ``x.getchildren()`` and
989  ``x.iter()`` or ``list(x.iter())`` instead of ``x.getiterator()``.
990  (Contributed by Serhiy Storchaka in :issue:`36543`.)
991
992* The old :mod:`plistlib` API has been removed, it was deprecated since Python
993  3.4. Use the :func:`~plistlib.load`, :func:`~plistlib.loads`, :func:`~plistlib.dump`, and
994  :func:`~plistlib.dumps` functions. Additionally, the *use_builtin_types* parameter was
995  removed, standard :class:`bytes` objects are always used instead.
996  (Contributed by Jon Janzen in :issue:`36409`.)
997
998* The C function ``PyGen_NeedsFinalizing`` has been removed. It was not
999  documented, tested, or used anywhere within CPython after the implementation
1000  of :pep:`442`. Patch by Joannah Nanjekye.
1001  (Contributed by Joannah Nanjekye in :issue:`15088`)
1002
1003* ``base64.encodestring()`` and ``base64.decodestring()``, aliases deprecated
1004  since Python 3.1, have been removed: use :func:`base64.encodebytes` and
1005  :func:`base64.decodebytes` instead.
1006  (Contributed by Victor Stinner in :issue:`39351`.)
1007
1008* ``fractions.gcd()`` function has been removed, it was deprecated since Python
1009  3.5 (:issue:`22486`): use :func:`math.gcd` instead.
1010  (Contributed by Victor Stinner in :issue:`39350`.)
1011
1012* The *buffering* parameter of :class:`bz2.BZ2File` has been removed. Since
1013  Python 3.0, it was ignored and using it emitted a :exc:`DeprecationWarning`.
1014  Pass an open file object to control how the file is opened.
1015  (Contributed by Victor Stinner in :issue:`39357`.)
1016
1017* The *encoding* parameter of :func:`json.loads` has been removed.
1018  As of Python 3.1, it was deprecated and ignored; using it has emitted a
1019  :exc:`DeprecationWarning` since Python 3.8.
1020  (Contributed by Inada Naoki in :issue:`39377`)
1021
1022* ``with (await asyncio.lock):`` and ``with (yield from asyncio.lock):`` statements are
1023  not longer supported, use ``async with lock`` instead.  The same is correct for
1024  ``asyncio.Condition`` and ``asyncio.Semaphore``.
1025  (Contributed by Andrew Svetlov in :issue:`34793`.)
1026
1027* The :func:`sys.getcounts` function, the ``-X showalloccount`` command line
1028  option and the ``show_alloc_count`` field of the C structure
1029  :c:type:`PyConfig` have been removed. They required a special Python build by
1030  defining ``COUNT_ALLOCS`` macro.
1031  (Contributed by Victor Stinner in :issue:`39489`.)
1032
1033* The ``_field_types`` attribute of the :class:`typing.NamedTuple` class
1034  has been removed.  It was deprecated since Python 3.8.  Use
1035  the ``__annotations__`` attribute instead.
1036  (Contributed by Serhiy Storchaka in :issue:`40182`.)
1037
1038* The :meth:`symtable.SymbolTable.has_exec` method has been removed. It was
1039  deprecated since 2006, and only returning ``False`` when it's called.
1040  (Contributed by Batuhan Taskaya in :issue:`40208`)
1041
1042* The :meth:`asyncio.Task.current_task` and :meth:`asyncio.Task.all_tasks`
1043  have been removed. They were deprecated since Python 3.7 and you can use
1044  :func:`asyncio.current_task` and :func:`asyncio.all_tasks` instead.
1045  (Contributed by Rémi Lapeyre in :issue:`40967`)
1046
1047* The ``unescape()`` method in the :class:`html.parser.HTMLParser` class
1048  has been removed (it was deprecated since Python 3.4).  :func:`html.unescape`
1049  should be used for converting character references to the corresponding
1050  unicode characters.
1051
1052
1053Porting to Python 3.9
1054=====================
1055
1056This section lists previously described changes and other bugfixes
1057that may require changes to your code.
1058
1059
1060Changes in the Python API
1061-------------------------
1062
1063* :func:`__import__` and :func:`importlib.util.resolve_name` now raise
1064  :exc:`ImportError` where it previously raised :exc:`ValueError`. Callers
1065  catching the specific exception type and supporting both Python 3.9 and
1066  earlier versions will need to catch both using ``except (ImportError, ValueError):``.
1067
1068* The :mod:`venv` activation scripts no longer special-case when
1069  ``__VENV_PROMPT__`` is set to ``""``.
1070
1071* The :meth:`select.epoll.unregister` method no longer ignores the
1072  :data:`~errno.EBADF` error.
1073  (Contributed by Victor Stinner in :issue:`39239`.)
1074
1075* The *compresslevel* parameter of :class:`bz2.BZ2File` became keyword-only,
1076  since the *buffering* parameter has been removed.
1077  (Contributed by Victor Stinner in :issue:`39357`.)
1078
1079* Simplified AST for subscription. Simple indices will be represented by
1080  their value, extended slices will be represented as tuples.
1081  ``Index(value)`` will return a ``value`` itself, ``ExtSlice(slices)``
1082  will return ``Tuple(slices, Load())``.
1083  (Contributed by Serhiy Storchaka in :issue:`34822`.)
1084
1085* The :mod:`importlib` module now ignores the :envvar:`PYTHONCASEOK`
1086  environment variable when the :option:`-E` or :option:`-I` command line
1087  options are being used.
1088
1089* The *encoding* parameter has been added to the classes :class:`ftplib.FTP` and
1090  :class:`ftplib.FTP_TLS` as a keyword-only parameter, and the default encoding
1091  is changed from Latin-1 to UTF-8 to follow :rfc:`2640`.
1092
1093* :meth:`asyncio.loop.shutdown_default_executor` has been added to
1094  :class:`~asyncio.AbstractEventLoop`, meaning alternative event loops that
1095  inherit from it should have this method defined.
1096  (Contributed by Kyle Stanley in :issue:`34037`.)
1097
1098* The constant values of future flags in the :mod:`__future__` module
1099  is updated in order to prevent collision with compiler flags. Previously
1100  ``PyCF_ALLOW_TOP_LEVEL_AWAIT`` was clashing with ``CO_FUTURE_DIVISION``.
1101  (Contributed by Batuhan Taskaya in :issue:`39562`)
1102
1103* ``array('u')`` now uses ``wchar_t`` as C type instead of ``Py_UNICODE``.
1104  This change doesn't affect to its behavior because ``Py_UNICODE`` is alias
1105  of ``wchar_t`` since Python 3.3.
1106  (Contributed by Inada Naoki in :issue:`34538`.)
1107
1108* The :func:`logging.getLogger` API now returns the root logger when passed
1109  the name ``'root'``, whereas previously it returned a non-root logger named
1110  ``'root'``. This could affect cases where user code explicitly wants a
1111  non-root logger named ``'root'``, or instantiates a logger using
1112  ``logging.getLogger(__name__)`` in some top-level module called ``'root.py'``.
1113  (Contributed by Vinay Sajip in :issue:`37742`.)
1114
1115* Division handling of :class:`~pathlib.PurePath` now returns ``NotImplemented``
1116  instead of raising a :exc:`TypeError` when passed something other than an
1117  instance of ``str`` or :class:`~pathlib.PurePath`.  This allows creating
1118  compatible classes that don't inherit from those mentioned types.
1119  (Contributed by Roger Aiudi in :issue:`34775`).
1120
1121* Starting with Python 3.9.5 the :mod:`ipaddress` module no longer
1122  accepts any leading zeros in IPv4 address strings. Leading zeros are
1123  ambiguous and interpreted as octal notation by some libraries. For example
1124  the legacy function :func:`socket.inet_aton` treats leading zeros as octal
1125  notatation. glibc implementation of modern :func:`~socket.inet_pton` does
1126  not accept any leading zeros.
1127  (Contributed by Christian Heimes in :issue:`36384`).
1128
1129* :func:`codecs.lookup` now normalizes the encoding name the same way as
1130  :func:`encodings.normalize_encoding`, except that :func:`codecs.lookup` also
1131  converts the name to lower case. For example, ``"latex+latin1"`` encoding
1132  name is now normalized to ``"latex_latin1"``.
1133  (Contributed by Jordon Xu in :issue:`37751`.)
1134
1135
1136Changes in the C API
1137--------------------
1138
1139* Instances of :ref:`heap-allocated types <heap-types>` (such as those created with
1140  :c:func:`PyType_FromSpec` and similar APIs) hold a reference to their type
1141  object since Python 3.8. As indicated in the "Changes in the C API" of Python
1142  3.8, for the vast majority of cases, there should be no side effect but for
1143  types that have a custom :c:member:`~PyTypeObject.tp_traverse` function,
1144  ensure that all custom ``tp_traverse`` functions of heap-allocated types
1145  visit the object's type.
1146
1147    Example:
1148
1149    .. code-block:: c
1150
1151        int
1152        foo_traverse(foo_struct *self, visitproc visit, void *arg) {
1153        // Rest of the traverse function
1154        #if PY_VERSION_HEX >= 0x03090000
1155            // This was not needed before Python 3.9 (Python issue 35810 and 40217)
1156            Py_VISIT(Py_TYPE(self));
1157        #endif
1158        }
1159
1160  If your traverse function delegates to ``tp_traverse`` of its base class
1161  (or another type), ensure that ``Py_TYPE(self)`` is visited only once.
1162  Note that only :ref:`heap type <heap-types>` are expected to visit the type
1163  in ``tp_traverse``.
1164
1165    For example, if your ``tp_traverse`` function includes:
1166
1167    .. code-block:: c
1168
1169        base->tp_traverse(self, visit, arg)
1170
1171    then add:
1172
1173    .. code-block:: c
1174
1175        #if PY_VERSION_HEX >= 0x03090000
1176            // This was not needed before Python 3.9 (bpo-35810 and bpo-40217)
1177            if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
1178                // a heap type's tp_traverse already visited Py_TYPE(self)
1179            } else {
1180                Py_VISIT(Py_TYPE(self));
1181            }
1182        #else
1183
1184  (See :issue:`35810` and :issue:`40217` for more information.)
1185
1186* The functions ``PyEval_CallObject``, ``PyEval_CallFunction``,
1187  ``PyEval_CallMethod`` and ``PyEval_CallObjectWithKeywords`` are deprecated.
1188  Use :c:func:`PyObject_Call` and its variants instead.
1189  (See more details in :issue:`29548`.)
1190
1191CPython bytecode changes
1192------------------------
1193
1194* The :opcode:`LOAD_ASSERTION_ERROR` opcode was added for handling the
1195  :keyword:`assert` statement. Previously, the assert statement would not work
1196  correctly if the :exc:`AssertionError` exception was being shadowed.
1197  (Contributed by Zackery Spytz in :issue:`34880`.)
1198
1199* The :opcode:`COMPARE_OP` opcode was split into four distinct instructions:
1200
1201  * ``COMPARE_OP`` for rich comparisons
1202  * ``IS_OP`` for 'is' and 'is not' tests
1203  * ``CONTAINS_OP`` for 'in' and 'not in' tests
1204  * ``JUMP_IF_NOT_EXC_MATCH`` for checking exceptions in 'try-except'
1205    statements.
1206
1207  (Contributed by Mark Shannon in :issue:`39156`.)
1208
1209
1210Build Changes
1211=============
1212
1213* Added ``--with-platlibdir`` option to the ``configure`` script: name of the
1214  platform-specific library directory, stored in the new :attr:`sys.platlibdir`
1215  attribute. See :attr:`sys.platlibdir` attribute for more information.
1216  (Contributed by Jan Matějek, Matěj Cepl, Charalampos Stratakis
1217  and Victor Stinner in :issue:`1294959`.)
1218
1219* The ``COUNT_ALLOCS`` special build macro has been removed.
1220  (Contributed by Victor Stinner in :issue:`39489`.)
1221
1222* On non-Windows platforms, the :c:func:`setenv` and :c:func:`unsetenv`
1223  functions are now required to build Python.
1224  (Contributed by Victor Stinner in :issue:`39395`.)
1225
1226* On non-Windows platforms, creating ``bdist_wininst`` installers is now
1227  officially unsupported.  (See :issue:`10945` for more details.)
1228
1229* When building Python on macOS from source, ``_tkinter`` now links with
1230  non-system Tcl and Tk frameworks if they are installed in
1231  ``/Library/Frameworks``, as had been the case on older releases
1232  of macOS. If a macOS SDK is explicitly configured, by using
1233  :option:`--enable-universalsdk` or ``-isysroot``, only the SDK itself is
1234  searched. The default behavior can still be overridden with
1235  :option:`--with-tcltk-includes` and :option:`--with-tcltk-libs`.
1236  (Contributed by Ned Deily in :issue:`34956`.)
1237
1238* Python can now be built for Windows 10 ARM64.
1239  (Contributed by Steve Dower in :issue:`33125`.)
1240
1241* Some individual tests are now skipped when ``--pgo`` is used.  The tests
1242  in question increased the PGO task time significantly and likely
1243  didn't help improve optimization of the final executable. This
1244  speeds up the task by a factor of about 15x.  Running the full unit test
1245  suite is slow.  This change may result in a slightly less optimized build
1246  since not as many code branches will be executed.  If you are willing to
1247  wait for the much slower build, the old behavior can be restored using
1248  ``./configure [..] PROFILE_TASK="-m test --pgo-extended"``.  We make no
1249  guarantees as to which PGO task set produces a faster build.  Users who care
1250  should run their own relevant benchmarks as results can depend on the
1251  environment, workload, and compiler tool chain.
1252  (See :issue:`36044` and :issue:`37707` for more details.)
1253
1254
1255C API Changes
1256=============
1257
1258New Features
1259------------
1260
1261* :pep:`573`: Added :c:func:`PyType_FromModuleAndSpec` to associate
1262  a module with a class; :c:func:`PyType_GetModule` and
1263  :c:func:`PyType_GetModuleState` to retrieve the module and its state; and
1264  :c:data:`PyCMethod` and :c:data:`METH_METHOD` to allow a method to
1265  access the class it was defined in.
1266  (Contributed by Marcel Plch and Petr Viktorin in :issue:`38787`.)
1267
1268* Added :c:func:`PyFrame_GetCode` function: get a frame code.
1269  Added :c:func:`PyFrame_GetBack` function: get the frame next outer frame.
1270  (Contributed by Victor Stinner in :issue:`40421`.)
1271
1272* Added :c:func:`PyFrame_GetLineNumber` to the limited C API.
1273  (Contributed by Victor Stinner in :issue:`40421`.)
1274
1275* Added :c:func:`PyThreadState_GetInterpreter` and
1276  :c:func:`PyInterpreterState_Get` functions to get the interpreter.
1277  Added :c:func:`PyThreadState_GetFrame` function to get the current frame of a
1278  Python thread state.
1279  Added :c:func:`PyThreadState_GetID` function: get the unique identifier of a
1280  Python thread state.
1281  (Contributed by Victor Stinner in :issue:`39947`.)
1282
1283* Added a new public :c:func:`PyObject_CallNoArgs` function to the C API, which
1284  calls a callable Python object without any arguments. It is the most efficient
1285  way to call a callable Python object without any argument.
1286  (Contributed by Victor Stinner in :issue:`37194`.)
1287
1288* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined):
1289
1290  * Provide :c:func:`Py_EnterRecursiveCall` and :c:func:`Py_LeaveRecursiveCall`
1291    as regular functions for the limited API. Previously, there were defined as
1292    macros, but these macros didn't compile with the limited C API which cannot
1293    access ``PyThreadState.recursion_depth`` field (the structure is opaque in
1294    the limited C API).
1295
1296  * ``PyObject_INIT()`` and ``PyObject_INIT_VAR()`` become regular "opaque"
1297    function to hide implementation details.
1298
1299  (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.)
1300
1301* The :c:func:`PyModule_AddType` function is added to help adding a type
1302  to a module.
1303  (Contributed by Dong-hee Na in :issue:`40024`.)
1304
1305* Added the functions :c:func:`PyObject_GC_IsTracked` and
1306  :c:func:`PyObject_GC_IsFinalized` to the public API to allow to query if
1307  Python objects are being currently tracked or have been already finalized by
1308  the garbage collector respectively.
1309  (Contributed by Pablo Galindo Salgado in :issue:`40241`.)
1310
1311* Added :c:func:`_PyObject_FunctionStr` to get a user-friendly string
1312  representation of a function-like object.
1313  (Patch by Jeroen Demeyer in :issue:`37645`.)
1314
1315* Added :c:func:`PyObject_CallOneArg` for calling an object with one
1316  positional argument
1317  (Patch by Jeroen Demeyer in :issue:`37483`.)
1318
1319
1320Porting to Python 3.9
1321---------------------
1322
1323* ``PyInterpreterState.eval_frame`` (:pep:`523`) now requires a new mandatory
1324  *tstate* parameter (``PyThreadState*``).
1325  (Contributed by Victor Stinner in :issue:`38500`.)
1326
1327* Extension modules: :c:member:`~PyModuleDef.m_traverse`,
1328  :c:member:`~PyModuleDef.m_clear` and :c:member:`~PyModuleDef.m_free`
1329  functions of :c:type:`PyModuleDef` are no longer called if the module state
1330  was requested but is not allocated yet. This is the case immediately after
1331  the module is created and before the module is executed
1332  (:c:data:`Py_mod_exec` function). More precisely, these functions are not called
1333  if :c:member:`~PyModuleDef.m_size` is greater than 0 and the module state (as
1334  returned by :c:func:`PyModule_GetState`) is ``NULL``.
1335
1336  Extension modules without module state (``m_size <= 0``) are not affected.
1337
1338* If :c:func:`Py_AddPendingCall` is called in a subinterpreter, the function is
1339  now scheduled to be called from the subinterpreter, rather than being called
1340  from the main interpreter. Each subinterpreter now has its own list of
1341  scheduled calls.
1342  (Contributed by Victor Stinner in :issue:`39984`.)
1343
1344* The Windows registry is no longer used to initialize :data:`sys.path` when
1345  the ``-E`` option is used (if :c:member:`PyConfig.use_environment` is set to
1346  ``0``). This is significant when embedding Python on Windows.
1347  (Contributed by Zackery Spytz in :issue:`8901`.)
1348
1349* The global variable :c:data:`PyStructSequence_UnnamedField` is now a constant
1350  and refers to a constant string.
1351  (Contributed by Serhiy Storchaka in :issue:`38650`.)
1352
1353* The :c:type:`PyGC_Head` structure is now opaque. It is only defined in the
1354  internal C API (``pycore_gc.h``).
1355  (Contributed by Victor Stinner in :issue:`40241`.)
1356
1357* The ``Py_UNICODE_COPY``, ``Py_UNICODE_FILL``, ``PyUnicode_WSTR_LENGTH``,
1358  :c:func:`PyUnicode_FromUnicode`, :c:func:`PyUnicode_AsUnicode`,
1359  ``_PyUnicode_AsUnicode``, and :c:func:`PyUnicode_AsUnicodeAndSize` are
1360  marked as deprecated in C.  They have been deprecated by :pep:`393` since
1361  Python 3.3.
1362  (Contributed by Inada Naoki in :issue:`36346`.)
1363
1364* The :c:func:`Py_FatalError` function is replaced with a macro which logs
1365  automatically the name of the current function, unless the
1366  ``Py_LIMITED_API`` macro is defined.
1367  (Contributed by Victor Stinner in :issue:`39882`.)
1368
1369* The vectorcall protocol now requires that the caller passes only strings as
1370  keyword names. (See :issue:`37540` for more information.)
1371
1372* Implementation details of a number of macros and functions are now hidden:
1373
1374  * :c:func:`PyObject_IS_GC` macro was converted to a function.
1375
1376  * The :c:func:`PyObject_NEW` macro becomes an alias to the
1377    :c:func:`PyObject_New` macro, and the :c:func:`PyObject_NEW_VAR` macro
1378    becomes an alias to the :c:func:`PyObject_NewVar` macro. They no longer
1379    access directly the :c:member:`PyTypeObject.tp_basicsize` member.
1380
1381  * :c:func:`PyObject_GET_WEAKREFS_LISTPTR` macro was converted to a function:
1382    the macro accessed directly the :c:member:`PyTypeObject.tp_weaklistoffset`
1383    member.
1384
1385  * :c:func:`PyObject_CheckBuffer` macro was converted to a function: the macro
1386    accessed directly the :c:member:`PyTypeObject.tp_as_buffer` member.
1387
1388  * :c:func:`PyIndex_Check` is now always declared as an opaque function to hide
1389    implementation details: removed the ``PyIndex_Check()`` macro. The macro accessed
1390    directly the :c:member:`PyTypeObject.tp_as_number` member.
1391
1392  (See :issue:`40170` for more details.)
1393
1394Removed
1395-------
1396
1397* Excluded ``PyFPE_START_PROTECT()`` and ``PyFPE_END_PROTECT()`` macros of
1398  ``pyfpe.h`` from the limited C API.
1399  (Contributed by Victor Stinner in :issue:`38835`.)
1400
1401* The ``tp_print`` slot of :ref:`PyTypeObject <type-structs>` has been removed.
1402  It was used for printing objects to files in Python 2.7 and before. Since
1403  Python 3.0, it has been ignored and unused.
1404  (Contributed by Jeroen Demeyer in :issue:`36974`.)
1405
1406* Changes in the limited C API (if ``Py_LIMITED_API`` macro is defined):
1407
1408  * Excluded the following functions from the limited C API:
1409
1410    * ``PyThreadState_DeleteCurrent()``
1411      (Contributed by Joannah Nanjekye in :issue:`37878`.)
1412    * ``_Py_CheckRecursionLimit``
1413    * ``_Py_NewReference()``
1414    * ``_Py_ForgetReference()``
1415    * ``_PyTraceMalloc_NewReference()``
1416    * ``_Py_GetRefTotal()``
1417    * The trashcan mechanism which never worked in the limited C API.
1418    * ``PyTrash_UNWIND_LEVEL``
1419    * ``Py_TRASHCAN_BEGIN_CONDITION``
1420    * ``Py_TRASHCAN_BEGIN``
1421    * ``Py_TRASHCAN_END``
1422    * ``Py_TRASHCAN_SAFE_BEGIN``
1423    * ``Py_TRASHCAN_SAFE_END``
1424
1425  * Moved following functions and definitions to the internal C API:
1426
1427    * ``_PyDebug_PrintTotalRefs()``
1428    * ``_Py_PrintReferences()``
1429    * ``_Py_PrintReferenceAddresses()``
1430    * ``_Py_tracemalloc_config``
1431    * ``_Py_AddToAllObjects()`` (specific to ``Py_TRACE_REFS`` build)
1432
1433  (Contributed by Victor Stinner in :issue:`38644` and :issue:`39542`.)
1434
1435* Removed ``_PyRuntime.getframe`` hook and removed ``_PyThreadState_GetFrame``
1436  macro which was an alias to ``_PyRuntime.getframe``. They were only exposed
1437  by the internal C API. Removed also ``PyThreadFrameGetter`` type.
1438  (Contributed by Victor Stinner in :issue:`39946`.)
1439
1440* Removed the following functions from the C API. Call :c:func:`PyGC_Collect`
1441  explicitly to clear all free lists.
1442  (Contributed by Inada Naoki and Victor Stinner in :issue:`37340`,
1443  :issue:`38896` and :issue:`40428`.)
1444
1445  * ``PyAsyncGen_ClearFreeLists()``
1446  * ``PyContext_ClearFreeList()``
1447  * ``PyDict_ClearFreeList()``
1448  * ``PyFloat_ClearFreeList()``
1449  * ``PyFrame_ClearFreeList()``
1450  * ``PyList_ClearFreeList()``
1451  * ``PyMethod_ClearFreeList()`` and ``PyCFunction_ClearFreeList()``:
1452    the free lists of bound method objects have been removed.
1453  * ``PySet_ClearFreeList()``: the set free list has been removed
1454    in Python 3.4.
1455  * ``PyTuple_ClearFreeList()``
1456  * ``PyUnicode_ClearFreeList()``: the Unicode free list has been removed in
1457    Python 3.3.
1458
1459* Removed ``_PyUnicode_ClearStaticStrings()`` function.
1460  (Contributed by Victor Stinner in :issue:`39465`.)
1461
1462* Removed ``Py_UNICODE_MATCH``. It has been deprecated by :pep:`393`, and
1463  broken since Python 3.3. The :c:func:`PyUnicode_Tailmatch` function can be
1464  used instead.
1465  (Contributed by Inada Naoki in :issue:`36346`.)
1466
1467* Cleaned header files of interfaces defined but with no implementation.
1468  The public API symbols being removed are:
1469  ``_PyBytes_InsertThousandsGroupingLocale``,
1470  ``_PyBytes_InsertThousandsGrouping``, ``_Py_InitializeFromArgs``,
1471  ``_Py_InitializeFromWideArgs``, ``_PyFloat_Repr``, ``_PyFloat_Digits``,
1472  ``_PyFloat_DigitsInit``, ``PyFrame_ExtendStack``, ``_PyAIterWrapper_Type``,
1473  ``PyNullImporter_Type``, ``PyCmpWrapper_Type``, ``PySortWrapper_Type``,
1474  ``PyNoArgsFunction``.
1475  (Contributed by Pablo Galindo Salgado in :issue:`39372`.)
1476
1477Notable changes in Python 3.9.1
1478===============================
1479
1480typing
1481------
1482
1483The behavior of :class:`typing.Literal` was changed to conform with :pep:`586`
1484and to match the behavior of static type checkers specified in the PEP.
1485
14861. ``Literal`` now de-duplicates parameters.
14872. Equality comparisons between ``Literal`` objects are now order independent.
14883. ``Literal`` comparisons now respect types.  For example,
1489   ``Literal[0] == Literal[False]`` previously evaluated to ``True``.  It is
1490   now ``False``.  To support this change, the internally used type cache now
1491   supports differentiating types.
14924. ``Literal`` objects will now raise a :exc:`TypeError` exception during
1493   equality comparisons if any of their parameters are not :term:`hashable`.
1494   Note that declaring ``Literal`` with mutable parameters will not throw
1495   an error::
1496
1497      >>> from typing import Literal
1498      >>> Literal[{0}]
1499      >>> Literal[{0}] == Literal[{False}]
1500      Traceback (most recent call last):
1501        File "<stdin>", line 1, in <module>
1502      TypeError: unhashable type: 'set'
1503
1504(Contributed by Yurii Karabas in :issue:`42345`.)
1505
1506macOS 11.0 (Big Sur) and Apple Silicon Mac support
1507--------------------------------------------------
1508
1509As of 3.9.1, Python now fully supports building and running on macOS 11.0
1510(Big Sur) and on Apple Silicon Macs (based on the ``ARM64`` architecture).
1511A new universal build variant, ``universal2``, is now available to natively
1512support both ``ARM64`` and ``Intel 64`` in one set of executables. Binaries
1513can also now be built on current versions of macOS to be deployed on a range
1514of older macOS versions (tested to 10.9) while making some newer OS
1515functions and options conditionally available based on the operating system
1516version in use at runtime ("weaklinking").
1517
1518(Contributed by Ronald Oussoren and Lawrence D'Anna in :issue:`41100`.)
1519
1520Notable changes in Python 3.9.2
1521===============================
1522
1523collections.abc
1524---------------
1525
1526:class:`collections.abc.Callable` generic now flattens type parameters, similar
1527to what :data:`typing.Callable` currently does.  This means that
1528``collections.abc.Callable[[int, str], str]`` will have ``__args__`` of
1529``(int, str, str)``; previously this was ``([int, str], str)``.  To allow this
1530change, :class:`types.GenericAlias` can now be subclassed, and a subclass will
1531be returned when subscripting the :class:`collections.abc.Callable` type.
1532Code which accesses the arguments via :func:`typing.get_args` or ``__args__``
1533need to account for this change.  A :exc:`DeprecationWarning` may be emitted for
1534invalid forms of parameterizing :class:`collections.abc.Callable` which may have
1535passed silently in Python 3.9.1.  This :exc:`DeprecationWarning` will
1536become a :exc:`TypeError` in Python 3.10.
1537(Contributed by Ken Jin in :issue:`42195`.)
1538
1539urllib.parse
1540------------
1541
1542Earlier Python versions allowed using both ``;`` and ``&`` as
1543query parameter separators in :func:`urllib.parse.parse_qs` and
1544:func:`urllib.parse.parse_qsl`.  Due to security concerns, and to conform with
1545newer W3C recommendations, this has been changed to allow only a single
1546separator key, with ``&`` as the default.  This change also affects
1547:func:`cgi.parse` and :func:`cgi.parse_multipart` as they use the affected
1548functions internally. For more details, please see their respective
1549documentation.
1550(Contributed by Adam Goldschmidt, Senthil Kumaran and Ken Jin in :issue:`42967`.)
1551