1**************************** 2 What's New In Python 3.11 3**************************** 4 5:Editor: Pablo Galindo Salgado 6 7.. Rules for maintenance: 8 9 * Anyone can add text to this document. Do not spend very much time 10 on the wording of your changes, because your text will probably 11 get rewritten to some degree. 12 13 * The maintainer will go through Misc/NEWS periodically and add 14 changes; it's therefore more important to add your changes to 15 Misc/NEWS than to this file. 16 17 * This is not a complete list of every single change; completeness 18 is the purpose of Misc/NEWS. Some changes I consider too small 19 or esoteric to include. If such a change is added to the text, 20 I'll just remove it. (This is another reason you shouldn't spend 21 too much time on writing your addition.) 22 23 * If you want to draw your new text to the attention of the 24 maintainer, add 'XXX' to the beginning of the paragraph or 25 section. 26 27 * It's OK to just add a fragmentary note about a change. For 28 example: "XXX Describe the transmogrify() function added to the 29 socket module." The maintainer will research the change and 30 write the necessary text. 31 32 * You can comment out your additions if you like, but it's not 33 necessary (especially when a final release is some months away). 34 35 * Credit the author of a patch or bugfix. Just the name is 36 sufficient; the e-mail address isn't necessary. 37 38 * It's helpful to add the bug/patch number as a comment: 39 40 XXX Describe the transmogrify() function added to the socket 41 module. 42 (Contributed by P.Y. Developer in :issue:`12345`.) 43 44 This saves the maintainer the effort of going through the Mercurial log 45 when researching a change. 46 47This article explains the new features in Python 3.11, compared to 3.10. 48Python 3.11 was released on October 24, 2022. 49For full details, see the :ref:`changelog <changelog>`. 50 51 52.. _whatsnew311-summary: 53 54Summary -- Release highlights 55============================= 56 57.. This section singles out the most important changes in Python 3.11. 58 Brevity is key. 59 60* Python 3.11 is between 10-60% faster than Python 3.10. 61 On average, we measured a 1.25x speedup on the standard benchmark suite. 62 See :ref:`whatsnew311-faster-cpython` for details. 63 64.. PEP-sized items next. 65 66New syntax features: 67 68* :ref:`whatsnew311-pep654` 69 70New built-in features: 71 72* :ref:`whatsnew311-pep678` 73 74New standard library modules: 75 76* :pep:`680`: :mod:`tomllib` — 77 Support for parsing `TOML <https://toml.io/>`_ in the Standard Library 78 79Interpreter improvements: 80 81* :ref:`whatsnew311-pep657` 82* New :option:`-P` command line option and :envvar:`PYTHONSAFEPATH` environment 83 variable to :ref:`disable automatically prepending potentially unsafe paths 84 <whatsnew311-pythonsafepath>` to :data:`sys.path` 85 86New typing features: 87 88* :ref:`whatsnew311-pep646` 89* :ref:`whatsnew311-pep655` 90* :ref:`whatsnew311-pep673` 91* :ref:`whatsnew311-pep675` 92* :ref:`whatsnew311-pep681` 93 94Important deprecations, removals and restrictions: 95 96* :pep:`594`: 97 :ref:`Many legacy standard library modules have been deprecated 98 <whatsnew311-pep594>` and will be removed in Python 3.13 99* :pep:`624`: 100 :ref:`Py_UNICODE encoder APIs have been removed <whatsnew311-pep624>` 101* :pep:`670`: 102 :ref:`Macros converted to static inline functions <whatsnew311-pep670>` 103 104 105.. _whatsnew311-features: 106 107New Features 108============ 109 110.. _whatsnew311-pep657: 111 112PEP 657: Fine-grained error locations in tracebacks 113--------------------------------------------------- 114 115When printing tracebacks, the interpreter will now point to the exact expression 116that caused the error, instead of just the line. For example: 117 118.. code-block:: python 119 120 Traceback (most recent call last): 121 File "distance.py", line 11, in <module> 122 print(manhattan_distance(p1, p2)) 123 ^^^^^^^^^^^^^^^^^^^^^^^^^^ 124 File "distance.py", line 6, in manhattan_distance 125 return abs(point_1.x - point_2.x) + abs(point_1.y - point_2.y) 126 ^^^^^^^^^ 127 AttributeError: 'NoneType' object has no attribute 'x' 128 129Previous versions of the interpreter would point to just the line, making it 130ambiguous which object was ``None``. These enhanced errors can also be helpful 131when dealing with deeply nested :class:`dict` objects and multiple function calls: 132 133.. code-block:: python 134 135 Traceback (most recent call last): 136 File "query.py", line 37, in <module> 137 magic_arithmetic('foo') 138 File "query.py", line 18, in magic_arithmetic 139 return add_counts(x) / 25 140 ^^^^^^^^^^^^^ 141 File "query.py", line 24, in add_counts 142 return 25 + query_user(user1) + query_user(user2) 143 ^^^^^^^^^^^^^^^^^ 144 File "query.py", line 32, in query_user 145 return 1 + query_count(db, response['a']['b']['c']['user'], retry=True) 146 ~~~~~~~~~~~~~~~~~~^^^^^ 147 TypeError: 'NoneType' object is not subscriptable 148 149As well as complex arithmetic expressions: 150 151.. code-block:: python 152 153 Traceback (most recent call last): 154 File "calculation.py", line 54, in <module> 155 result = (x / y / z) * (a / b / c) 156 ~~~~~~^~~ 157 ZeroDivisionError: division by zero 158 159Additionally, the information used by the enhanced traceback feature 160is made available via a general API, that can be used to correlate 161:term:`bytecode` :ref:`instructions <bytecodes>` with source code location. 162This information can be retrieved using: 163 164- The :meth:`codeobject.co_positions` method in Python. 165- The :c:func:`PyCode_Addr2Location` function in the C API. 166 167See :pep:`657` for more details. (Contributed by Pablo Galindo, Batuhan Taskaya 168and Ammar Askar in :issue:`43950`.) 169 170.. note:: 171 This feature requires storing column positions in :ref:`codeobjects`, 172 which may result in a small increase in interpreter memory usage 173 and disk usage for compiled Python files. 174 To avoid storing the extra information 175 and deactivate printing the extra traceback information, 176 use the :option:`-X no_debug_ranges <-X>` command line option 177 or the :envvar:`PYTHONNODEBUGRANGES` environment variable. 178 179 180.. _whatsnew311-pep654: 181 182PEP 654: Exception Groups and ``except*`` 183----------------------------------------- 184 185:pep:`654` introduces language features that enable a program 186to raise and handle multiple unrelated exceptions simultaneously. 187The builtin types :exc:`ExceptionGroup` and :exc:`BaseExceptionGroup` 188make it possible to group exceptions and raise them together, 189and the new :keyword:`except* <except_star>` syntax generalizes 190:keyword:`except` to match subgroups of exception groups. 191 192See :pep:`654` for more details. 193 194(Contributed by Irit Katriel in :issue:`45292`. PEP written by 195Irit Katriel, Yury Selivanov and Guido van Rossum.) 196 197 198.. _whatsnew311-pep678: 199 200PEP 678: Exceptions can be enriched with notes 201---------------------------------------------- 202 203The :meth:`~BaseException.add_note` method is added to :exc:`BaseException`. 204It can be used to enrich exceptions with context information 205that is not available at the time when the exception is raised. 206The added notes appear in the default traceback. 207 208See :pep:`678` for more details. 209 210(Contributed by Irit Katriel in :issue:`45607`. 211PEP written by Zac Hatfield-Dodds.) 212 213 214.. _whatsnew311-windows-launcher: 215 216Windows ``py.exe`` launcher improvements 217---------------------------------------- 218 219The copy of the :ref:`launcher` included with Python 3.11 has been significantly 220updated. It now supports company/tag syntax as defined in :pep:`514` using the 221:samp:`-V:{<company>}/{<tag>}` argument instead of the limited :samp:`-{<major>}.{<minor>}`. 222This allows launching distributions other than ``PythonCore``, 223the one hosted on `python.org <https://www.python.org>`_. 224 225When using ``-V:`` selectors, either company or tag can be omitted, but all 226installs will be searched. For example, ``-V:OtherPython/`` will select the 227"best" tag registered for ``OtherPython``, while ``-V:3.11`` or ``-V:/3.11`` 228will select the "best" distribution with tag ``3.11``. 229 230When using the legacy :samp:`-{<major>}`, :samp:`-{<major>}.{<minor>}`, 231:samp:`-{<major>}-{<bitness>}` or :samp:`-{<major>}.{<minor>}-{<bitness>}` arguments, 232all existing behaviour should be preserved from past versions, 233and only releases from ``PythonCore`` will be selected. 234However, the ``-64`` suffix now implies "not 32-bit" (not necessarily x86-64), 235as there are multiple supported 64-bit platforms. 23632-bit runtimes are detected by checking the runtime's tag for a ``-32`` suffix. 237All releases of Python since 3.5 have included this in their 32-bit builds. 238 239 240.. _new-feat-related-type-hints-311: 241.. _whatsnew311-typing-features: 242 243New Features Related to Type Hints 244================================== 245 246This section covers major changes affecting :pep:`484` type hints and 247the :mod:`typing` module. 248 249 250.. _whatsnew311-pep646: 251 252PEP 646: Variadic generics 253-------------------------- 254 255:pep:`484` previously introduced :data:`~typing.TypeVar`, enabling creation 256of generics parameterised with a single type. :pep:`646` adds 257:data:`~typing.TypeVarTuple`, enabling parameterisation 258with an *arbitrary* number of types. In other words, 259a :data:`~typing.TypeVarTuple` is a *variadic* type variable, 260enabling *variadic* generics. 261 262This enables a wide variety of use cases. 263In particular, it allows the type of array-like structures 264in numerical computing libraries such as NumPy and TensorFlow to be 265parameterised with the array *shape*. Static type checkers will now 266be able to catch shape-related bugs in code that uses these libraries. 267 268See :pep:`646` for more details. 269 270(Contributed by Matthew Rahtz in :issue:`43224`, with contributions by 271Serhiy Storchaka and Jelle Zijlstra. PEP written by Mark Mendoza, Matthew 272Rahtz, Pradeep Kumar Srinivasan, and Vincent Siles.) 273 274 275.. _whatsnew311-pep655: 276 277PEP 655: Marking individual ``TypedDict`` items as required or not-required 278--------------------------------------------------------------------------- 279 280:data:`~typing.Required` and :data:`~typing.NotRequired` provide a 281straightforward way to mark whether individual items in a 282:class:`~typing.TypedDict` must be present. Previously, this was only possible 283using inheritance. 284 285All fields are still required by default, 286unless the *total* parameter is set to ``False``, 287in which case all fields are still not-required by default. 288For example, the following specifies a :class:`!TypedDict` 289with one required and one not-required key:: 290 291 class Movie(TypedDict): 292 title: str 293 year: NotRequired[int] 294 295 m1: Movie = {"title": "Black Panther", "year": 2018} # OK 296 m2: Movie = {"title": "Star Wars"} # OK (year is not required) 297 m3: Movie = {"year": 2022} # ERROR (missing required field title) 298 299The following definition is equivalent:: 300 301 class Movie(TypedDict, total=False): 302 title: Required[str] 303 year: int 304 305See :pep:`655` for more details. 306 307(Contributed by David Foster and Jelle Zijlstra in :issue:`47087`. PEP 308written by David Foster.) 309 310 311.. _whatsnew311-pep673: 312 313PEP 673: ``Self`` type 314---------------------- 315 316The new :data:`~typing.Self` annotation provides a simple and intuitive 317way to annotate methods that return an instance of their class. This 318behaves the same as the :class:`~typing.TypeVar`-based approach 319:pep:`specified in PEP 484 <484#annotating-instance-and-class-methods>`, 320but is more concise and easier to follow. 321 322Common use cases include alternative constructors provided as 323:func:`classmethod <classmethod>`\s, 324and :meth:`~object.__enter__` methods that return ``self``:: 325 326 class MyLock: 327 def __enter__(self) -> Self: 328 self.lock() 329 return self 330 331 ... 332 333 class MyInt: 334 @classmethod 335 def fromhex(cls, s: str) -> Self: 336 return cls(int(s, 16)) 337 338 ... 339 340:data:`~typing.Self` can also be used to annotate method parameters 341or attributes of the same type as their enclosing class. 342 343See :pep:`673` for more details. 344 345(Contributed by James Hilton-Balfe in :issue:`46534`. PEP written by 346Pradeep Kumar Srinivasan and James Hilton-Balfe.) 347 348 349.. _whatsnew311-pep675: 350 351PEP 675: Arbitrary literal string type 352-------------------------------------- 353 354The new :data:`~typing.LiteralString` annotation may be used to indicate 355that a function parameter can be of any literal string type. This allows 356a function to accept arbitrary literal string types, as well as strings 357created from other literal strings. Type checkers can then 358enforce that sensitive functions, such as those that execute SQL 359statements or shell commands, are called only with static arguments, 360providing protection against injection attacks. 361 362For example, a SQL query function could be annotated as follows:: 363 364 def run_query(sql: LiteralString) -> ... 365 ... 366 367 def caller( 368 arbitrary_string: str, 369 query_string: LiteralString, 370 table_name: LiteralString, 371 ) -> None: 372 run_query("SELECT * FROM students") # ok 373 run_query(query_string) # ok 374 run_query("SELECT * FROM " + table_name) # ok 375 run_query(arbitrary_string) # type checker error 376 run_query( # type checker error 377 f"SELECT * FROM students WHERE name = {arbitrary_string}" 378 ) 379 380See :pep:`675` for more details. 381 382(Contributed by Jelle Zijlstra in :issue:`47088`. PEP written by Pradeep 383Kumar Srinivasan and Graham Bleaney.) 384 385 386.. _whatsnew311-pep681: 387 388PEP 681: Data class transforms 389------------------------------ 390 391:data:`~typing.dataclass_transform` may be used to 392decorate a class, metaclass, or a function that is itself a decorator. 393The presence of ``@dataclass_transform()`` tells a static type checker that the 394decorated object performs runtime "magic" that transforms a class, 395giving it :func:`dataclass <dataclasses.dataclass>`-like behaviors. 396 397For example:: 398 399 # The create_model decorator is defined by a library. 400 @typing.dataclass_transform() 401 def create_model(cls: Type[T]) -> Type[T]: 402 cls.__init__ = ... 403 cls.__eq__ = ... 404 cls.__ne__ = ... 405 return cls 406 407 # The create_model decorator can now be used to create new model classes: 408 @create_model 409 class CustomerModel: 410 id: int 411 name: str 412 413 c = CustomerModel(id=327, name="Eric Idle") 414 415See :pep:`681` for more details. 416 417(Contributed by Jelle Zijlstra in :gh:`91860`. PEP written by 418Erik De Bonte and Eric Traut.) 419 420 421.. _whatsnew311-pep563-deferred: 422 423PEP 563 may not be the future 424----------------------------- 425 426:pep:`563` Postponed Evaluation of Annotations 427(the ``from __future__ import annotations`` :ref:`future statement <future>`) 428that was originally planned for release in Python 3.10 429has been put on hold indefinitely. 430See `this message from the Steering Council <https://mail.python.org/archives/list/python-dev@python.org/message/VIZEBX5EYMSYIJNDBF6DMUMZOCWHARSO/>`__ 431for more information. 432 433 434.. _whatsnew311-other-lang-changes: 435 436Other Language Changes 437====================== 438 439* Starred unpacking expressions can now be used in :keyword:`for` statements. 440 (See :issue:`46725` for more details.) 441 442* Asynchronous :ref:`comprehensions <comprehensions>` are now allowed 443 inside comprehensions in :ref:`asynchronous functions <async def>`. 444 Outer comprehensions implicitly become asynchronous in this case. 445 (Contributed by Serhiy Storchaka in :issue:`33346`.) 446 447* A :exc:`TypeError` is now raised instead of an :exc:`AttributeError` in 448 :keyword:`with` statements and :meth:`contextlib.ExitStack.enter_context` 449 for objects that do not support the :term:`context manager` protocol, 450 and in :keyword:`async with` statements and 451 :meth:`contextlib.AsyncExitStack.enter_async_context` 452 for objects not supporting the :term:`asynchronous context manager` protocol. 453 (Contributed by Serhiy Storchaka in :issue:`12022` and :issue:`44471`.) 454 455* Added :meth:`object.__getstate__`, which provides the default 456 implementation of the :meth:`!__getstate__` method. :mod:`copy`\ing 457 and :mod:`pickle`\ing instances of subclasses of builtin types 458 :class:`bytearray`, :class:`set`, :class:`frozenset`, 459 :class:`collections.OrderedDict`, :class:`collections.deque`, 460 :class:`weakref.WeakSet`, and :class:`datetime.tzinfo` now copies and 461 pickles instance attributes implemented as :term:`slots <__slots__>`. 462 This change has an unintended side effect: It trips up a small minority 463 of existing Python projects not expecting :meth:`object.__getstate__` to 464 exist. See the later comments on :gh:`70766` for discussions of what 465 workarounds such code may need. 466 (Contributed by Serhiy Storchaka in :issue:`26579`.) 467 468.. _whatsnew311-pythonsafepath: 469 470* Added a :option:`-P` command line option 471 and a :envvar:`PYTHONSAFEPATH` environment variable, 472 which disable the automatic prepending to :data:`sys.path` 473 of the script's directory when running a script, 474 or the current directory when using :option:`-c` and :option:`-m`. 475 This ensures only stdlib and installed modules 476 are picked up by :keyword:`import`, 477 and avoids unintentionally or maliciously shadowing modules 478 with those in a local (and typically user-writable) directory. 479 (Contributed by Victor Stinner in :gh:`57684`.) 480 481* A ``"z"`` option was added to the :ref:`formatspec` that 482 coerces negative to positive zero after rounding to the format precision. 483 See :pep:`682` for more details. 484 (Contributed by John Belmonte in :gh:`90153`.) 485 486* Bytes are no longer accepted on :data:`sys.path`. Support broke sometime 487 between Python 3.2 and 3.6, with no one noticing until after Python 3.10.0 488 was released. In addition, bringing back support would be problematic due to 489 interactions between :option:`-b` and :data:`sys.path_importer_cache` when 490 there is a mixture of :class:`str` and :class:`bytes` keys. 491 (Contributed by Thomas Grainger in :gh:`91181`.) 492 493 494.. _whatsnew311-other-implementation-changes: 495 496Other CPython Implementation Changes 497==================================== 498 499* The special methods :meth:`~object.__complex__` for :class:`complex` 500 and :meth:`~object.__bytes__` for :class:`bytes` are implemented to support 501 the :class:`typing.SupportsComplex` and :class:`typing.SupportsBytes` protocols. 502 (Contributed by Mark Dickinson and Donghee Na in :issue:`24234`.) 503 504* ``siphash13`` is added as a new internal hashing algorithm. 505 It has similar security properties as ``siphash24``, 506 but it is slightly faster for long inputs. 507 :class:`str`, :class:`bytes`, and some other types 508 now use it as the default algorithm for :func:`hash`. 509 :pep:`552` :ref:`hash-based .pyc files <pyc-invalidation>` 510 now use ``siphash13`` too. 511 (Contributed by Inada Naoki in :issue:`29410`.) 512 513* When an active exception is re-raised by a :keyword:`raise` statement with no parameters, 514 the traceback attached to this exception is now always ``sys.exc_info()[1].__traceback__``. 515 This means that changes made to the traceback in the current :keyword:`except` clause are 516 reflected in the re-raised exception. 517 (Contributed by Irit Katriel in :issue:`45711`.) 518 519* The interpreter state's representation of handled exceptions 520 (aka ``exc_info`` or ``_PyErr_StackItem``) 521 now only has the ``exc_value`` field; ``exc_type`` and ``exc_traceback`` 522 have been removed, as they can be derived from ``exc_value``. 523 (Contributed by Irit Katriel in :issue:`45711`.) 524 525* A new :ref:`command line option <install-quiet-option>`, ``AppendPath``, 526 has been added for the Windows installer. 527 It behaves similarly to ``PrependPath``, 528 but appends the install and scripts directories instead of prepending them. 529 (Contributed by Bastian Neuburger in :issue:`44934`.) 530 531* The :c:member:`PyConfig.module_search_paths_set` field must now be set to ``1`` for 532 initialization to use :c:member:`PyConfig.module_search_paths` to initialize 533 :data:`sys.path`. Otherwise, initialization will recalculate the path and replace 534 any values added to ``module_search_paths``. 535 536* The output of the :option:`--help` option now fits in 50 lines/80 columns. 537 Information about :ref:`Python environment variables <using-on-envvars>` 538 and :option:`-X` options is now available using the respective 539 :option:`--help-env` and :option:`--help-xoptions` flags, 540 and with the new :option:`--help-all`. 541 (Contributed by Éric Araujo in :issue:`46142`.) 542 543* Converting between :class:`int` and :class:`str` in bases other than 2 544 (binary), 4, 8 (octal), 16 (hexadecimal), or 32 such as base 10 (decimal) 545 now raises a :exc:`ValueError` if the number of digits in string form is 546 above a limit to avoid potential denial of service attacks due to the 547 algorithmic complexity. This is a mitigation for :cve:`2020-10735`. 548 This limit can be configured or disabled by environment variable, command 549 line flag, or :mod:`sys` APIs. See the :ref:`integer string conversion 550 length limitation <int_max_str_digits>` documentation. The default limit 551 is 4300 digits in string form. 552 553 554.. _whatsnew311-new-modules: 555 556New Modules 557=========== 558 559* :mod:`tomllib`: For parsing `TOML <https://toml.io/>`_. 560 See :pep:`680` for more details. 561 (Contributed by Taneli Hukkinen in :issue:`40059`.) 562 563* :mod:`wsgiref.types`: 564 :pep:`WSGI <3333>`-specific types for static type checking. 565 (Contributed by Sebastian Rittau in :issue:`42012`.) 566 567 568.. _whatsnew311-improved-modules: 569 570Improved Modules 571================ 572 573.. _whatsnew311-asyncio: 574 575asyncio 576------- 577 578* Added the :class:`~asyncio.TaskGroup` class, 579 an :ref:`asynchronous context manager <async-context-managers>` 580 holding a group of tasks that will wait for all of them upon exit. 581 For new code this is recommended over using 582 :func:`~asyncio.create_task` and :func:`~asyncio.gather` directly. 583 (Contributed by Yury Selivanov and others in :gh:`90908`.) 584 585* Added :func:`~asyncio.timeout`, an asynchronous context manager for 586 setting a timeout on asynchronous operations. For new code this is 587 recommended over using :func:`~asyncio.wait_for` directly. 588 (Contributed by Andrew Svetlov in :gh:`90927`.) 589 590* Added the :class:`~asyncio.Runner` class, which exposes the machinery 591 used by :func:`~asyncio.run`. 592 (Contributed by Andrew Svetlov in :gh:`91218`.) 593 594* Added the :class:`~asyncio.Barrier` class to the synchronization 595 primitives in the asyncio library, and the related 596 :exc:`~asyncio.BrokenBarrierError` exception. 597 (Contributed by Yves Duprat and Andrew Svetlov in :gh:`87518`.) 598 599* Added keyword argument *all_errors* to :meth:`asyncio.loop.create_connection` 600 so that multiple connection errors can be raised as an :exc:`ExceptionGroup`. 601 602* Added the :meth:`asyncio.StreamWriter.start_tls` method for 603 upgrading existing stream-based connections to TLS. 604 (Contributed by Ian Good in :issue:`34975`.) 605 606* Added raw datagram socket functions to the event loop: 607 :meth:`~asyncio.loop.sock_sendto`, 608 :meth:`~asyncio.loop.sock_recvfrom` and 609 :meth:`~asyncio.loop.sock_recvfrom_into`. 610 These have implementations in :class:`~asyncio.SelectorEventLoop` and 611 :class:`~asyncio.ProactorEventLoop`. 612 (Contributed by Alex Grönholm in :issue:`46805`.) 613 614* Added :meth:`~asyncio.Task.cancelling` and 615 :meth:`~asyncio.Task.uncancel` methods to :class:`~asyncio.Task`. 616 These are primarily intended for internal use, 617 notably by :class:`~asyncio.TaskGroup`. 618 619 620.. _whatsnew311-contextlib: 621 622contextlib 623---------- 624 625* Added non parallel-safe :func:`~contextlib.chdir` context manager to change 626 the current working directory and then restore it on exit. Simple wrapper 627 around :func:`~os.chdir`. (Contributed by Filipe Laíns in :issue:`25625`) 628 629 630.. _whatsnew311-dataclasses: 631 632dataclasses 633----------- 634 635* Change field default mutability check, allowing only defaults which are 636 :term:`hashable` instead of any object which is not an instance of 637 :class:`dict`, :class:`list` or :class:`set`. (Contributed by Eric V. Smith in 638 :issue:`44674`.) 639 640 641.. _whatsnew311-datetime: 642 643datetime 644-------- 645 646* Add :const:`datetime.UTC`, a convenience alias for 647 :attr:`datetime.timezone.utc`. (Contributed by Kabir Kwatra in :gh:`91973`.) 648 649* :meth:`datetime.date.fromisoformat`, :meth:`datetime.time.fromisoformat` and 650 :meth:`datetime.datetime.fromisoformat` can now be used to parse most ISO 8601 651 formats (barring only those that support fractional hours and minutes). 652 (Contributed by Paul Ganssle in :gh:`80010`.) 653 654 655.. _whatsnew311-enum: 656 657enum 658---- 659 660* Renamed :class:`!EnumMeta` to :class:`~enum.EnumType` 661 (:class:`!EnumMeta` kept as an alias). 662 663* Added :class:`~enum.StrEnum`, 664 with members that can be used as (and must be) strings. 665 666* Added :class:`~enum.ReprEnum`, 667 which only modifies the :meth:`~object.__repr__` of members 668 while returning their literal values (rather than names) 669 for :meth:`~object.__str__` and :meth:`~object.__format__` 670 (used by :func:`str`, :func:`format` and :term:`f-string`\s). 671 672* Changed :meth:`Enum.__format__() <enum.Enum.__format__>` (the default for 673 :func:`format`, :meth:`str.format` and :term:`f-string`\s) to always produce 674 the same result as :meth:`Enum.__str__() <enum.Enum.__str__>`: for enums inheriting from 675 :class:`~enum.ReprEnum` it will be the member's value; for all other enums 676 it will be the enum and member name (e.g. ``Color.RED``). 677 678* Added a new *boundary* class parameter to :class:`~enum.Flag` enums 679 and the :class:`~enum.FlagBoundary` enum with its options, 680 to control how to handle out-of-range flag values. 681 682* Added the :func:`~enum.verify` enum decorator 683 and the :class:`~enum.EnumCheck` enum with its options, 684 to check enum classes against several specific constraints. 685 686* Added the :func:`~enum.member` and :func:`~enum.nonmember` decorators, 687 to ensure the decorated object is/is not converted to an enum member. 688 689* Added the :func:`~enum.property` decorator, 690 which works like :func:`property` except for enums. 691 Use this instead of :func:`types.DynamicClassAttribute`. 692 693* Added the :func:`~enum.global_enum` enum decorator, 694 which adjusts :meth:`~object.__repr__` and :meth:`~object.__str__` 695 to show values as members of their module rather than the enum class. 696 For example, ``'re.ASCII'`` for the :const:`~re.ASCII` member 697 of :class:`re.RegexFlag` rather than ``'RegexFlag.ASCII'``. 698 699* Enhanced :class:`~enum.Flag` to support 700 :func:`len`, iteration and :keyword:`in`/:keyword:`not in` on its members. 701 For example, the following now works: 702 ``len(AFlag(3)) == 2 and list(AFlag(3)) == (AFlag.ONE, AFlag.TWO)`` 703 704* Changed :class:`~enum.Enum` and :class:`~enum.Flag` 705 so that members are now defined 706 before :meth:`~object.__init_subclass__` is called; 707 :func:`dir` now includes methods, etc., from mixed-in data types. 708 709* Changed :class:`~enum.Flag` 710 to only consider primary values (power of two) canonical 711 while composite values (``3``, ``6``, ``10``, etc.) are considered aliases; 712 inverted flags are coerced to their positive equivalent. 713 714 715.. _whatsnew311-fcntl: 716 717fcntl 718----- 719 720* On FreeBSD, the :data:`!F_DUP2FD` and :data:`!F_DUP2FD_CLOEXEC` flags respectively 721 are supported, the former equals to ``dup2`` usage while the latter set 722 the ``FD_CLOEXEC`` flag in addition. 723 724 725.. _whatsnew311-fractions: 726 727fractions 728--------- 729 730* Support :PEP:`515`-style initialization of :class:`~fractions.Fraction` from 731 string. (Contributed by Sergey B Kirpichev in :issue:`44258`.) 732 733* :class:`~fractions.Fraction` now implements an ``__int__`` method, so 734 that an ``isinstance(some_fraction, typing.SupportsInt)`` check passes. 735 (Contributed by Mark Dickinson in :issue:`44547`.) 736 737 738.. _whatsnew311-functools: 739 740functools 741--------- 742 743* :func:`functools.singledispatch` now supports :data:`types.UnionType` 744 and :data:`typing.Union` as annotations to the dispatch argument.:: 745 746 >>> from functools import singledispatch 747 >>> @singledispatch 748 ... def fun(arg, verbose=False): 749 ... if verbose: 750 ... print("Let me just say,", end=" ") 751 ... print(arg) 752 ... 753 >>> @fun.register 754 ... def _(arg: int | float, verbose=False): 755 ... if verbose: 756 ... print("Strength in numbers, eh?", end=" ") 757 ... print(arg) 758 ... 759 >>> from typing import Union 760 >>> @fun.register 761 ... def _(arg: Union[list, set], verbose=False): 762 ... if verbose: 763 ... print("Enumerate this:") 764 ... for i, elem in enumerate(arg): 765 ... print(i, elem) 766 ... 767 768 (Contributed by Yurii Karabas in :issue:`46014`.) 769 770 771.. _whatsnew311-gzip: 772 773gzip 774---- 775 776* The :func:`gzip.compress` function is now faster when used with the 777 **mtime=0** argument as it delegates the compression entirely to a single 778 :func:`zlib.compress` operation. There is one side effect of this change: The 779 gzip file header contains an "OS" byte in its header. That was traditionally 780 always set to a value of 255 representing "unknown" by the :mod:`gzip` 781 module. Now, when using :func:`~gzip.compress` with **mtime=0**, it may be 782 set to a different value by the underlying zlib C library Python was linked 783 against. 784 (See :gh:`112346` for details on the side effect.) 785 786.. _whatsnew311-hashlib: 787 788hashlib 789------- 790 791* :func:`hashlib.blake2b` and :func:`hashlib.blake2s` now prefer `libb2`_ 792 over Python's vendored copy. 793 (Contributed by Christian Heimes in :issue:`47095`.) 794 795* The internal ``_sha3`` module with SHA3 and SHAKE algorithms now uses 796 *tiny_sha3* instead of the *Keccak Code Package* to reduce code and binary 797 size. The :mod:`hashlib` module prefers optimized SHA3 and SHAKE 798 implementations from OpenSSL. The change affects only installations without 799 OpenSSL support. 800 (Contributed by Christian Heimes in :issue:`47098`.) 801 802* Add :func:`hashlib.file_digest`, a helper function for efficient hashing 803 of files or file-like objects. 804 (Contributed by Christian Heimes in :gh:`89313`.) 805 806 807.. _whatsnew311-idle: 808 809IDLE and idlelib 810---------------- 811 812* Apply syntax highlighting to ``.pyi`` files. (Contributed by Alex 813 Waygood and Terry Jan Reedy in :issue:`45447`.) 814 815* Include prompts when saving Shell with inputs and outputs. 816 (Contributed by Terry Jan Reedy in :gh:`95191`.) 817 818 819.. _whatsnew311-inspect: 820 821inspect 822------- 823 824* Add :func:`~inspect.getmembers_static` to return all members without 825 triggering dynamic lookup via the descriptor protocol. (Contributed by 826 Weipeng Hong in :issue:`30533`.) 827 828* Add :func:`~inspect.ismethodwrapper` 829 for checking if the type of an object is a :class:`~types.MethodWrapperType`. 830 (Contributed by Hakan Çelik in :issue:`29418`.) 831 832* Change the frame-related functions in the :mod:`inspect` module to return new 833 :class:`~inspect.FrameInfo` and :class:`~inspect.Traceback` class instances 834 (backwards compatible with the previous :term:`named tuple`-like interfaces) 835 that includes the extended :pep:`657` position information (end 836 line number, column and end column). The affected functions are: 837 838 * :func:`inspect.getframeinfo` 839 * :func:`inspect.getouterframes` 840 * :func:`inspect.getinnerframes`, 841 * :func:`inspect.stack` 842 * :func:`inspect.trace` 843 844 (Contributed by Pablo Galindo in :gh:`88116`.) 845 846 847.. _whatsnew311-locale: 848 849locale 850------ 851 852* Add :func:`locale.getencoding` to get the current locale encoding. It is similar to 853 ``locale.getpreferredencoding(False)`` but ignores the 854 :ref:`Python UTF-8 Mode <utf8-mode>`. 855 856 857.. _whatsnew311-logging: 858 859logging 860------- 861 862* Added :func:`~logging.getLevelNamesMapping` 863 to return a mapping from logging level names (e.g. ``'CRITICAL'``) 864 to the values of their corresponding :ref:`levels` (e.g. ``50``, by default). 865 (Contributed by Andrei Kulakovin in :gh:`88024`.) 866 867* Added a :meth:`~logging.handlers.SysLogHandler.createSocket` method 868 to :class:`~logging.handlers.SysLogHandler`, to match 869 :meth:`SocketHandler.createSocket() 870 <logging.handlers.SocketHandler.createSocket>`. 871 It is called automatically during handler initialization 872 and when emitting an event, if there is no active socket. 873 (Contributed by Kirill Pinchuk in :gh:`88457`.) 874 875 876.. _whatsnew311-math: 877 878math 879---- 880 881* Add :func:`math.exp2`: return 2 raised to the power of x. 882 (Contributed by Gideon Mitchell in :issue:`45917`.) 883 884* Add :func:`math.cbrt`: return the cube root of x. 885 (Contributed by Ajith Ramachandran in :issue:`44357`.) 886 887* The behaviour of two :func:`math.pow` corner cases was changed, for 888 consistency with the IEEE 754 specification. The operations 889 ``math.pow(0.0, -math.inf)`` and ``math.pow(-0.0, -math.inf)`` now return 890 ``inf``. Previously they raised :exc:`ValueError`. (Contributed by Mark 891 Dickinson in :issue:`44339`.) 892 893* The :data:`math.nan` value is now always available. 894 (Contributed by Victor Stinner in :issue:`46917`.) 895 896 897.. _whatsnew311-operator: 898 899operator 900-------- 901 902* A new function ``operator.call`` has been added, such that 903 ``operator.call(obj, *args, **kwargs) == obj(*args, **kwargs)``. 904 (Contributed by Antony Lee in :issue:`44019`.) 905 906 907.. _whatsnew311-os: 908 909os 910-- 911 912* On Windows, :func:`os.urandom` now uses ``BCryptGenRandom()``, 913 instead of ``CryptGenRandom()`` which is deprecated. 914 (Contributed by Donghee Na in :issue:`44611`.) 915 916 917.. _whatsnew311-pathlib: 918 919pathlib 920------- 921 922* :meth:`~pathlib.Path.glob` and :meth:`~pathlib.Path.rglob` return only 923 directories if *pattern* ends with a pathname components separator: 924 :data:`~os.sep` or :data:`~os.altsep`. 925 (Contributed by Eisuke Kawasima in :issue:`22276` and :issue:`33392`.) 926 927 928.. _whatsnew311-re: 929 930re 931-- 932 933* Atomic grouping (``(?>...)``) and possessive quantifiers (``*+``, ``++``, 934 ``?+``, ``{m,n}+``) are now supported in regular expressions. 935 (Contributed by Jeffrey C. Jacobs and Serhiy Storchaka in :issue:`433030`.) 936 937 938.. _whatsnew311-shutil: 939 940shutil 941------ 942 943* Add optional parameter *dir_fd* in :func:`shutil.rmtree`. 944 (Contributed by Serhiy Storchaka in :issue:`46245`.) 945 946 947.. _whatsnew311-socket: 948 949socket 950------ 951 952* Add CAN Socket support for NetBSD. 953 (Contributed by Thomas Klausner in :issue:`30512`.) 954 955* :meth:`~socket.create_connection` has an option to raise, in case of 956 failure to connect, an :exc:`ExceptionGroup` containing all errors 957 instead of only raising the last error. 958 (Contributed by Irit Katriel in :issue:`29980`.) 959 960 961.. _whatsnew311-sqlite3: 962 963sqlite3 964------- 965 966* You can now disable the authorizer by passing :const:`None` to 967 :meth:`~sqlite3.Connection.set_authorizer`. 968 (Contributed by Erlend E. Aasland in :issue:`44491`.) 969 970* Collation name :meth:`~sqlite3.Connection.create_collation` can now 971 contain any Unicode character. Collation names with invalid characters 972 now raise :exc:`UnicodeEncodeError` instead of :exc:`sqlite3.ProgrammingError`. 973 (Contributed by Erlend E. Aasland in :issue:`44688`.) 974 975* :mod:`sqlite3` exceptions now include the SQLite extended error code as 976 :attr:`~sqlite3.Error.sqlite_errorcode` and the SQLite error name as 977 :attr:`~sqlite3.Error.sqlite_errorname`. 978 (Contributed by Aviv Palivoda, Daniel Shahaf, and Erlend E. Aasland in 979 :issue:`16379` and :issue:`24139`.) 980 981* Add :meth:`~sqlite3.Connection.setlimit` and 982 :meth:`~sqlite3.Connection.getlimit` to :class:`sqlite3.Connection` for 983 setting and getting SQLite limits by connection basis. 984 (Contributed by Erlend E. Aasland in :issue:`45243`.) 985 986* :mod:`sqlite3` now sets :attr:`sqlite3.threadsafety` based on the default 987 threading mode the underlying SQLite library has been compiled with. 988 (Contributed by Erlend E. Aasland in :issue:`45613`.) 989 990* :mod:`sqlite3` C callbacks now use unraisable exceptions if callback 991 tracebacks are enabled. Users can now register an 992 :func:`unraisable hook handler <sys.unraisablehook>` to improve their debug 993 experience. 994 (Contributed by Erlend E. Aasland in :issue:`45828`.) 995 996* Fetch across rollback no longer raises :exc:`~sqlite3.InterfaceError`. 997 Instead we leave it to the SQLite library to handle these cases. 998 (Contributed by Erlend E. Aasland in :issue:`44092`.) 999 1000* Add :meth:`~sqlite3.Connection.serialize` and 1001 :meth:`~sqlite3.Connection.deserialize` to :class:`sqlite3.Connection` for 1002 serializing and deserializing databases. 1003 (Contributed by Erlend E. Aasland in :issue:`41930`.) 1004 1005* Add :meth:`~sqlite3.Connection.create_window_function` to 1006 :class:`sqlite3.Connection` for creating aggregate window functions. 1007 (Contributed by Erlend E. Aasland in :issue:`34916`.) 1008 1009* Add :meth:`~sqlite3.Connection.blobopen` to :class:`sqlite3.Connection`. 1010 :class:`sqlite3.Blob` allows incremental I/O operations on blobs. 1011 (Contributed by Aviv Palivoda and Erlend E. Aasland in :issue:`24905`.) 1012 1013 1014.. _whatsnew311-string: 1015 1016string 1017------ 1018 1019* Add :meth:`~string.Template.get_identifiers` 1020 and :meth:`~string.Template.is_valid` to :class:`string.Template`, 1021 which respectively return all valid placeholders, 1022 and whether any invalid placeholders are present. 1023 (Contributed by Ben Kehoe in :gh:`90465`.) 1024 1025 1026.. _whatsnew311-sys: 1027 1028sys 1029--- 1030 1031* :func:`sys.exc_info` now derives the ``type`` and ``traceback`` fields 1032 from the ``value`` (the exception instance), so when an exception is 1033 modified while it is being handled, the changes are reflected in 1034 the results of subsequent calls to :func:`!exc_info`. 1035 (Contributed by Irit Katriel in :issue:`45711`.) 1036 1037* Add :func:`sys.exception` which returns the active exception instance 1038 (equivalent to ``sys.exc_info()[1]``). 1039 (Contributed by Irit Katriel in :issue:`46328`.) 1040 1041* Add the :data:`sys.flags.safe_path <sys.flags>` flag. 1042 (Contributed by Victor Stinner in :gh:`57684`.) 1043 1044 1045.. _whatsnew311-sysconfig: 1046 1047sysconfig 1048--------- 1049 1050* Three new :ref:`installation schemes <installation_paths>` 1051 (*posix_venv*, *nt_venv* and *venv*) were added and are used when Python 1052 creates new virtual environments or when it is running from a virtual 1053 environment. 1054 The first two schemes (*posix_venv* and *nt_venv*) are OS-specific 1055 for non-Windows and Windows, the *venv* is essentially an alias to one of 1056 them according to the OS Python runs on. 1057 This is useful for downstream distributors who modify 1058 :func:`sysconfig.get_preferred_scheme`. 1059 Third party code that creates new virtual environments should use the new 1060 *venv* installation scheme to determine the paths, as does :mod:`venv`. 1061 (Contributed by Miro Hrončok in :issue:`45413`.) 1062 1063 1064.. _whatsnew311-tempfile: 1065 1066tempfile 1067-------- 1068 1069* :class:`~tempfile.SpooledTemporaryFile` objects now fully implement the methods 1070 of :class:`io.BufferedIOBase` or :class:`io.TextIOBase` 1071 (depending on file mode). 1072 This lets them work correctly with APIs that expect file-like objects, 1073 such as compression modules. 1074 (Contributed by Carey Metcalfe in :gh:`70363`.) 1075 1076 1077.. _whatsnew311-threading: 1078 1079threading 1080--------- 1081 1082* On Unix, if the ``sem_clockwait()`` function is available in the C library 1083 (glibc 2.30 and newer), the :meth:`threading.Lock.acquire` method now uses 1084 the monotonic clock (:const:`time.CLOCK_MONOTONIC`) for the timeout, rather 1085 than using the system clock (:const:`time.CLOCK_REALTIME`), to not be affected 1086 by system clock changes. 1087 (Contributed by Victor Stinner in :issue:`41710`.) 1088 1089 1090.. _whatsnew311-time: 1091 1092time 1093---- 1094 1095* On Unix, :func:`time.sleep` now uses the ``clock_nanosleep()`` or 1096 ``nanosleep()`` function, if available, which has a resolution of 1 nanosecond 1097 (10\ :sup:`-9` seconds), rather than using ``select()`` which has a resolution 1098 of 1 microsecond (10\ :sup:`-6` seconds). 1099 (Contributed by Benjamin Szőke and Victor Stinner in :issue:`21302`.) 1100 1101* On Windows 8.1 and newer, :func:`time.sleep` now uses a waitable timer based 1102 on `high-resolution timers 1103 <https://docs.microsoft.com/en-us/windows-hardware/drivers/kernel/high-resolution-timers>`_ 1104 which has a resolution of 100 nanoseconds (10\ :sup:`-7` seconds). Previously, 1105 it had a resolution of 1 millisecond (10\ :sup:`-3` seconds). 1106 (Contributed by Benjamin Szőke, Donghee Na, Eryk Sun and Victor Stinner in :issue:`21302` and :issue:`45429`.) 1107 1108 1109.. _whatsnew311-tkinter: 1110 1111tkinter 1112------- 1113 1114* Added method ``info_patchlevel()`` which returns the exact version of 1115 the Tcl library as a named tuple similar to :data:`sys.version_info`. 1116 (Contributed by Serhiy Storchaka in :gh:`91827`.) 1117 1118 1119.. _whatsnew311-traceback: 1120 1121traceback 1122--------- 1123 1124* Add :func:`traceback.StackSummary.format_frame_summary` to allow users 1125 to override which frames appear in the traceback, and how they are 1126 formatted. 1127 (Contributed by Ammar Askar in :issue:`44569`.) 1128 1129* Add :func:`traceback.TracebackException.print`, which prints the 1130 formatted :exc:`~traceback.TracebackException` instance to a file. 1131 (Contributed by Irit Katriel in :issue:`33809`.) 1132 1133 1134.. _whatsnew311-typing: 1135 1136typing 1137------ 1138 1139For major changes, see :ref:`new-feat-related-type-hints-311`. 1140 1141* Add :func:`typing.assert_never` and :class:`typing.Never`. 1142 :func:`typing.assert_never` is useful for asking a type checker to confirm 1143 that a line of code is not reachable. At runtime, it raises an 1144 :exc:`AssertionError`. 1145 (Contributed by Jelle Zijlstra in :gh:`90633`.) 1146 1147* Add :func:`typing.reveal_type`. This is useful for asking a type checker 1148 what type it has inferred for a given expression. At runtime it prints 1149 the type of the received value. 1150 (Contributed by Jelle Zijlstra in :gh:`90572`.) 1151 1152* Add :func:`typing.assert_type`. This is useful for asking a type checker 1153 to confirm that the type it has inferred for a given expression matches 1154 the given type. At runtime it simply returns the received value. 1155 (Contributed by Jelle Zijlstra in :gh:`90638`.) 1156 1157* :data:`typing.TypedDict` types can now be generic. (Contributed by 1158 Samodya Abeysiriwardane in :gh:`89026`.) 1159 1160* :class:`~typing.NamedTuple` types can now be generic. 1161 (Contributed by Serhiy Storchaka in :issue:`43923`.) 1162 1163* Allow subclassing of :class:`typing.Any`. This is useful for avoiding 1164 type checker errors related to highly dynamic class, such as mocks. 1165 (Contributed by Shantanu Jain in :gh:`91154`.) 1166 1167* The :func:`typing.final` decorator now sets the ``__final__`` attributed on 1168 the decorated object. 1169 (Contributed by Jelle Zijlstra in :gh:`90500`.) 1170 1171* The :func:`typing.get_overloads` function can be used for introspecting 1172 the overloads of a function. :func:`typing.clear_overloads` can be used 1173 to clear all registered overloads of a function. 1174 (Contributed by Jelle Zijlstra in :gh:`89263`.) 1175 1176* The :meth:`~object.__init__` method of :class:`~typing.Protocol` subclasses 1177 is now preserved. (Contributed by Adrian Garcia Badarasco in :gh:`88970`.) 1178 1179* The representation of empty tuple types (``Tuple[()]``) is simplified. 1180 This affects introspection, e.g. ``get_args(Tuple[()])`` now evaluates 1181 to ``()`` instead of ``((),)``. 1182 (Contributed by Serhiy Storchaka in :gh:`91137`.) 1183 1184* Loosen runtime requirements for type annotations by removing the callable 1185 check in the private ``typing._type_check`` function. (Contributed by 1186 Gregory Beauregard in :gh:`90802`.) 1187 1188* :func:`typing.get_type_hints` now supports evaluating strings as forward 1189 references in :ref:`PEP 585 generic aliases <types-genericalias>`. 1190 (Contributed by Niklas Rosenstein in :gh:`85542`.) 1191 1192* :func:`typing.get_type_hints` no longer adds :data:`~typing.Optional` 1193 to parameters with ``None`` as a default. (Contributed by Nikita Sobolev 1194 in :gh:`90353`.) 1195 1196* :func:`typing.get_type_hints` now supports evaluating bare stringified 1197 :data:`~typing.ClassVar` annotations. (Contributed by Gregory Beauregard 1198 in :gh:`90711`.) 1199 1200* :func:`typing.no_type_check` no longer modifies external classes and functions. 1201 It also now correctly marks classmethods as not to be type checked. (Contributed 1202 by Nikita Sobolev in :gh:`90729`.) 1203 1204 1205.. _whatsnew311-unicodedata: 1206 1207unicodedata 1208----------- 1209 1210* The Unicode database has been updated to version 14.0.0. 1211 (Contributed by Benjamin Peterson in :issue:`45190`). 1212 1213 1214.. _whatsnew311-unittest: 1215 1216unittest 1217-------- 1218 1219* Added methods :meth:`~unittest.TestCase.enterContext` and 1220 :meth:`~unittest.TestCase.enterClassContext` of class 1221 :class:`~unittest.TestCase`, method 1222 :meth:`~unittest.IsolatedAsyncioTestCase.enterAsyncContext` of 1223 class :class:`~unittest.IsolatedAsyncioTestCase` and function 1224 :func:`unittest.enterModuleContext`. 1225 (Contributed by Serhiy Storchaka in :issue:`45046`.) 1226 1227 1228.. _whatsnew311-venv: 1229 1230venv 1231---- 1232 1233* When new Python virtual environments are created, the *venv* 1234 :ref:`sysconfig installation scheme <installation_paths>` is used 1235 to determine the paths inside the environment. 1236 When Python runs in a virtual environment, the same installation scheme 1237 is the default. 1238 That means that downstream distributors can change the default sysconfig install 1239 scheme without changing behavior of virtual environments. 1240 Third party code that also creates new virtual environments should do the same. 1241 (Contributed by Miro Hrončok in :issue:`45413`.) 1242 1243 1244.. _whatsnew311-warnings: 1245 1246warnings 1247-------- 1248 1249* :func:`warnings.catch_warnings` now accepts arguments for :func:`warnings.simplefilter`, 1250 providing a more concise way to locally ignore warnings or convert them to errors. 1251 (Contributed by Zac Hatfield-Dodds in :issue:`47074`.) 1252 1253 1254.. _whatsnew311-zipfile: 1255 1256zipfile 1257------- 1258 1259* Added support for specifying member name encoding for reading metadata 1260 in a :class:`~zipfile.ZipFile`'s directory and file headers. 1261 (Contributed by Stephen J. Turnbull and Serhiy Storchaka in :issue:`28080`.) 1262 1263* Added :meth:`ZipFile.mkdir() <zipfile.ZipFile.mkdir>` 1264 for creating new directories inside ZIP archives. 1265 (Contributed by Sam Ezeh in :gh:`49083`.) 1266 1267* Added :attr:`~zipfile.Path.stem`, :attr:`~zipfile.Path.suffix` 1268 and :attr:`~zipfile.Path.suffixes` to :class:`zipfile.Path`. 1269 (Contributed by Miguel Brito in :gh:`88261`.) 1270 1271 1272.. _whatsnew311-optimizations: 1273 1274Optimizations 1275============= 1276 1277This section covers specific optimizations independent of the 1278:ref:`whatsnew311-faster-cpython` project, which is covered in its own section. 1279 1280* The compiler now optimizes simple 1281 :ref:`printf-style % formatting <old-string-formatting>` on string literals 1282 containing only the format codes ``%s``, ``%r`` and ``%a`` and makes it as 1283 fast as a corresponding :term:`f-string` expression. 1284 (Contributed by Serhiy Storchaka in :issue:`28307`.) 1285 1286* Integer division (``//``) is better tuned for optimization by compilers. 1287 It is now around 20% faster on x86-64 when dividing an :class:`int` 1288 by a value smaller than ``2**30``. 1289 (Contributed by Gregory P. Smith and Tim Peters in :gh:`90564`.) 1290 1291* :func:`sum` is now nearly 30% faster for integers smaller than ``2**30``. 1292 (Contributed by Stefan Behnel in :gh:`68264`.) 1293 1294* Resizing lists is streamlined for the common case, 1295 speeding up :meth:`list.append` by ≈15% 1296 and simple :term:`list comprehension`\s by up to 20-30% 1297 (Contributed by Dennis Sweeney in :gh:`91165`.) 1298 1299* Dictionaries don't store hash values when all keys are Unicode objects, 1300 decreasing :class:`dict` size. 1301 For example, ``sys.getsizeof(dict.fromkeys("abcdefg"))`` 1302 is reduced from 352 bytes to 272 bytes (23% smaller) on 64-bit platforms. 1303 (Contributed by Inada Naoki in :issue:`46845`.) 1304 1305* Using :class:`asyncio.DatagramProtocol` is now orders of magnitude faster 1306 when transferring large files over UDP, 1307 with speeds over 100 times higher for a ≈60 MiB file. 1308 (Contributed by msoxzw in :gh:`91487`.) 1309 1310* :mod:`math` functions :func:`~math.comb` and :func:`~math.perm` are now 1311 ≈10 times faster for large arguments (with a larger speedup for larger *k*). 1312 (Contributed by Serhiy Storchaka in :issue:`37295`.) 1313 1314* The :mod:`statistics` functions :func:`~statistics.mean`, 1315 :func:`~statistics.variance` and :func:`~statistics.stdev` now consume 1316 iterators in one pass rather than converting them to a :class:`list` first. 1317 This is twice as fast and can save substantial memory. 1318 (Contributed by Raymond Hettinger in :gh:`90415`.) 1319 1320* :func:`unicodedata.normalize` 1321 now normalizes pure-ASCII strings in constant time. 1322 (Contributed by Donghee Na in :issue:`44987`.) 1323 1324 1325.. _whatsnew311-faster-cpython: 1326 1327Faster CPython 1328============== 1329 1330CPython 3.11 is an average of 1331`25% faster <https://github.com/faster-cpython/ideas#published-results>`_ 1332than CPython 3.10 as measured with the 1333`pyperformance <https://github.com/python/pyperformance>`_ benchmark suite, 1334when compiled with GCC on Ubuntu Linux. 1335Depending on your workload, the overall speedup could be 10-60%. 1336 1337This project focuses on two major areas in Python: 1338:ref:`whatsnew311-faster-startup` and :ref:`whatsnew311-faster-runtime`. 1339Optimizations not covered by this project are listed separately under 1340:ref:`whatsnew311-optimizations`. 1341 1342 1343.. _whatsnew311-faster-startup: 1344 1345Faster Startup 1346-------------- 1347 1348.. _whatsnew311-faster-imports: 1349 1350Frozen imports / Static code objects 1351^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1352 1353Python caches :term:`bytecode` in the :ref:`__pycache__ <tut-pycache>` 1354directory to speed up module loading. 1355 1356Previously in 3.10, Python module execution looked like this: 1357 1358.. code-block:: text 1359 1360 Read __pycache__ -> Unmarshal -> Heap allocated code object -> Evaluate 1361 1362In Python 3.11, the core modules essential for Python startup are "frozen". 1363This means that their :ref:`codeobjects` (and bytecode) 1364are statically allocated by the interpreter. 1365This reduces the steps in module execution process to: 1366 1367.. code-block:: text 1368 1369 Statically allocated code object -> Evaluate 1370 1371Interpreter startup is now 10-15% faster in Python 3.11. This has a big 1372impact for short-running programs using Python. 1373 1374(Contributed by Eric Snow, Guido van Rossum and Kumar Aditya in many issues.) 1375 1376 1377.. _whatsnew311-faster-runtime: 1378 1379Faster Runtime 1380-------------- 1381 1382.. _whatsnew311-lazy-python-frames: 1383 1384Cheaper, lazy Python frames 1385^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1386 1387Python frames, holding execution information, 1388are created whenever Python calls a Python function. 1389The following are new frame optimizations: 1390 1391- Streamlined the frame creation process. 1392- Avoided memory allocation by generously re-using frame space on the C stack. 1393- Streamlined the internal frame struct to contain only essential information. 1394 Frames previously held extra debugging and memory management information. 1395 1396Old-style :ref:`frame objects <frame-objects>` 1397are now created only when requested by debuggers 1398or by Python introspection functions such as :func:`sys._getframe` and 1399:func:`inspect.currentframe`. For most user code, no frame objects are 1400created at all. As a result, nearly all Python functions calls have sped 1401up significantly. We measured a 3-7% speedup in pyperformance. 1402 1403(Contributed by Mark Shannon in :issue:`44590`.) 1404 1405 1406.. _inline-calls: 1407.. _whatsnew311-inline-calls: 1408 1409Inlined Python function calls 1410^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1411 1412During a Python function call, Python will call an evaluating C function to 1413interpret that function's code. This effectively limits pure Python recursion to 1414what's safe for the C stack. 1415 1416In 3.11, when CPython detects Python code calling another Python function, 1417it sets up a new frame, and "jumps" to the new code inside the new frame. This 1418avoids calling the C interpreting function altogether. 1419 1420Most Python function calls now consume no C stack space, speeding them up. 1421In simple recursive functions like fibonacci or 1422factorial, we observed a 1.7x speedup. This also means recursive functions 1423can recurse significantly deeper 1424(if the user increases the recursion limit with :func:`sys.setrecursionlimit`). 1425We measured a 1-3% improvement in pyperformance. 1426 1427(Contributed by Pablo Galindo and Mark Shannon in :issue:`45256`.) 1428 1429 1430.. _whatsnew311-pep659: 1431 1432PEP 659: Specializing Adaptive Interpreter 1433^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1434 1435:pep:`659` is one of the key parts of the Faster CPython project. The general 1436idea is that while Python is a dynamic language, most code has regions where 1437objects and types rarely change. This concept is known as *type stability*. 1438 1439At runtime, Python will try to look for common patterns and type stability 1440in the executing code. Python will then replace the current operation with a 1441more specialized one. This specialized operation uses fast paths available only 1442to those use cases/types, which generally outperform their generic 1443counterparts. This also brings in another concept called *inline caching*, where 1444Python caches the results of expensive operations directly in the 1445:term:`bytecode`. 1446 1447The specializer will also combine certain common instruction pairs into one 1448superinstruction, reducing the overhead during execution. 1449 1450Python will only specialize 1451when it sees code that is "hot" (executed multiple times). This prevents Python 1452from wasting time on run-once code. Python can also de-specialize when code is 1453too dynamic or when the use changes. Specialization is attempted periodically, 1454and specialization attempts are not too expensive, 1455allowing specialization to adapt to new circumstances. 1456 1457(PEP written by Mark Shannon, with ideas inspired by Stefan Brunthaler. 1458See :pep:`659` for more information. Implementation by Mark Shannon and Brandt 1459Bucher, with additional help from Irit Katriel and Dennis Sweeney.) 1460 1461.. 1462 If I missed out anyone, please add them. 1463 1464+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1465| Operation | Form | Specialization | Operation speedup | Contributor(s) | 1466| | | | (up to) | | 1467+===============+====================+=======================================================+===================+===================+ 1468| Binary | ``x + x`` | Binary add, multiply and subtract for common types | 10% | Mark Shannon, | 1469| operations | | such as :class:`int`, :class:`float` and :class:`str` | | Donghee Na, | 1470| | ``x - x`` | take custom fast paths for their underlying types. | | Brandt Bucher, | 1471| | | | | Dennis Sweeney | 1472| | ``x * x`` | | | | 1473+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1474| Subscript | ``a[i]`` | Subscripting container types such as :class:`list`, | 10-25% | Irit Katriel, | 1475| | | :class:`tuple` and :class:`dict` directly index | | Mark Shannon | 1476| | | the underlying data structures. | | | 1477| | | | | | 1478| | | Subscripting custom :meth:`~object.__getitem__` | | | 1479| | | is also inlined similar to :ref:`inline-calls`. | | | 1480+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1481| Store | ``a[i] = z`` | Similar to subscripting specialization above. | 10-25% | Dennis Sweeney | 1482| subscript | | | | | 1483+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1484| Calls | ``f(arg)`` | Calls to common builtin (C) functions and types such | 20% | Mark Shannon, | 1485| | | as :func:`len` and :class:`str` directly call their | | Ken Jin | 1486| | ``C(arg)`` | underlying C version. This avoids going through the | | | 1487| | | internal calling convention. | | | 1488+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1489| Load | ``print`` | The object's index in the globals/builtins namespace | [#load-global]_ | Mark Shannon | 1490| global | | is cached. Loading globals and builtins require | | | 1491| variable | ``len`` | zero namespace lookups. | | | 1492+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1493| Load | ``o.attr`` | Similar to loading global variables. The attribute's | [#load-attr]_ | Mark Shannon | 1494| attribute | | index inside the class/object's namespace is cached. | | | 1495| | | In most cases, attribute loading will require zero | | | 1496| | | namespace lookups. | | | 1497+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1498| Load | ``o.meth()`` | The actual address of the method is cached. Method | 10-20% | Ken Jin, | 1499| methods for | | loading now has no namespace lookups -- even for | | Mark Shannon | 1500| call | | classes with long inheritance chains. | | | 1501+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1502| Store | ``o.attr = z`` | Similar to load attribute optimization. | 2% | Mark Shannon | 1503| attribute | | | in pyperformance | | 1504+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1505| Unpack | ``*seq`` | Specialized for common containers such as | 8% | Brandt Bucher | 1506| Sequence | | :class:`list` and :class:`tuple`. | | | 1507| | | Avoids internal calling convention. | | | 1508+---------------+--------------------+-------------------------------------------------------+-------------------+-------------------+ 1509 1510.. [#load-global] A similar optimization already existed since Python 3.8. 1511 3.11 specializes for more forms and reduces some overhead. 1512 1513.. [#load-attr] A similar optimization already existed since Python 3.10. 1514 3.11 specializes for more forms. Furthermore, all attribute loads should 1515 be sped up by :issue:`45947`. 1516 1517 1518.. _whatsnew311-faster-cpython-misc: 1519 1520Misc 1521---- 1522 1523* Objects now require less memory due to lazily created object namespaces. 1524 Their namespace dictionaries now also share keys more freely. 1525 (Contributed Mark Shannon in :issue:`45340` and :issue:`40116`.) 1526 1527* "Zero-cost" exceptions are implemented, eliminating the cost 1528 of :keyword:`try` statements when no exception is raised. 1529 (Contributed by Mark Shannon in :issue:`40222`.) 1530 1531* A more concise representation of exceptions in the interpreter reduced the 1532 time required for catching an exception by about 10%. 1533 (Contributed by Irit Katriel in :issue:`45711`.) 1534 1535* :mod:`re`'s regular expression matching engine has been partially refactored, 1536 and now uses computed gotos (or "threaded code") on supported platforms. As a 1537 result, Python 3.11 executes the `pyperformance regular expression benchmarks 1538 <https://pyperformance.readthedocs.io/benchmarks.html#regex-dna>`_ up to 10% 1539 faster than Python 3.10. 1540 (Contributed by Brandt Bucher in :gh:`91404`.) 1541 1542 1543.. _whatsnew311-faster-cpython-faq: 1544 1545FAQ 1546--- 1547 1548.. _faster-cpython-faq-my-code: 1549 1550How should I write my code to utilize these speedups? 1551^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1552 1553Write Pythonic code that follows common best practices; 1554you don't have to change your code. 1555The Faster CPython project optimizes for common code patterns we observe. 1556 1557 1558.. _faster-cpython-faq-memory: 1559 1560Will CPython 3.11 use more memory? 1561^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1562 1563Maybe not; we don't expect memory use to exceed 20% higher than 3.10. 1564This is offset by memory optimizations for frame objects and object 1565dictionaries as mentioned above. 1566 1567 1568.. _faster-cpython-ymmv: 1569 1570I don't see any speedups in my workload. Why? 1571^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 1572 1573Certain code won't have noticeable benefits. If your code spends most of 1574its time on I/O operations, or already does most of its 1575computation in a C extension library like NumPy, there won't be significant 1576speedups. This project currently benefits pure-Python workloads the most. 1577 1578Furthermore, the pyperformance figures are a geometric mean. Even within the 1579pyperformance benchmarks, certain benchmarks have slowed down slightly, while 1580others have sped up by nearly 2x! 1581 1582 1583.. _faster-cpython-jit: 1584 1585Is there a JIT compiler? 1586^^^^^^^^^^^^^^^^^^^^^^^^ 1587 1588No. We're still exploring other optimizations. 1589 1590 1591.. _whatsnew311-faster-cpython-about: 1592 1593About 1594----- 1595 1596Faster CPython explores optimizations for :term:`CPython`. The main team is 1597funded by Microsoft to work on this full-time. Pablo Galindo Salgado is also 1598funded by Bloomberg LP to work on the project part-time. Finally, many 1599contributors are volunteers from the community. 1600 1601 1602.. _whatsnew311-bytecode-changes: 1603 1604CPython bytecode changes 1605======================== 1606 1607The bytecode now contains inline cache entries, 1608which take the form of the newly-added :opcode:`CACHE` instructions. 1609Many opcodes expect to be followed by an exact number of caches, 1610and instruct the interpreter to skip over them at runtime. 1611Populated caches can look like arbitrary instructions, 1612so great care should be taken when reading or modifying 1613raw, adaptive bytecode containing quickened data. 1614 1615 1616.. _whatsnew311-added-opcodes: 1617 1618New opcodes 1619----------- 1620 1621* :opcode:`!ASYNC_GEN_WRAP`, :opcode:`RETURN_GENERATOR` and :opcode:`SEND`, 1622 used in generators and co-routines. 1623 1624* :opcode:`COPY_FREE_VARS`, 1625 which avoids needing special caller-side code for closures. 1626 1627* :opcode:`JUMP_BACKWARD_NO_INTERRUPT`, 1628 for use in certain loops where handling interrupts is undesirable. 1629 1630* :opcode:`MAKE_CELL`, to create :ref:`cell-objects`. 1631 1632* :opcode:`CHECK_EG_MATCH` and :opcode:`!PREP_RERAISE_STAR`, 1633 to handle the :ref:`new exception groups and except* <whatsnew311-pep654>` 1634 added in :pep:`654`. 1635 1636* :opcode:`PUSH_EXC_INFO`, for use in exception handlers. 1637 1638* :opcode:`RESUME`, a no-op, 1639 for internal tracing, debugging and optimization checks. 1640 1641 1642.. _whatsnew311-replaced-opcodes: 1643 1644Replaced opcodes 1645---------------- 1646 1647+------------------------------------+------------------------------------+-----------------------------------------+ 1648| Replaced Opcode(s) | New Opcode(s) | Notes | 1649+====================================+====================================+=========================================+ 1650| | :opcode:`!BINARY_*` | :opcode:`BINARY_OP` | Replaced all numeric binary/in-place | 1651| | :opcode:`!INPLACE_*` | | opcodes with a single opcode | 1652+------------------------------------+------------------------------------+-----------------------------------------+ 1653| | :opcode:`!CALL_FUNCTION` | | :opcode:`CALL` | Decouples argument shifting for methods | 1654| | :opcode:`!CALL_FUNCTION_KW` | | :opcode:`!KW_NAMES` | from handling of keyword arguments; | 1655| | :opcode:`!CALL_METHOD` | | :opcode:`!PRECALL` | allows better specialization of calls | 1656| | | :opcode:`PUSH_NULL` | | 1657+------------------------------------+------------------------------------+-----------------------------------------+ 1658| | :opcode:`!DUP_TOP` | | :opcode:`COPY` | Stack manipulation instructions | 1659| | :opcode:`!DUP_TOP_TWO` | | :opcode:`SWAP` | | 1660| | :opcode:`!ROT_TWO` | | | 1661| | :opcode:`!ROT_THREE` | | | 1662| | :opcode:`!ROT_FOUR` | | | 1663| | :opcode:`!ROT_N` | | | 1664+------------------------------------+------------------------------------+-----------------------------------------+ 1665| | :opcode:`!JUMP_IF_NOT_EXC_MATCH` | | :opcode:`CHECK_EXC_MATCH` | Now performs check but doesn't jump | 1666+------------------------------------+------------------------------------+-----------------------------------------+ 1667| | :opcode:`!JUMP_ABSOLUTE` | | :opcode:`JUMP_BACKWARD` | See [#bytecode-jump]_; | 1668| | :opcode:`!POP_JUMP_IF_FALSE` | | :opcode:`!POP_JUMP_BACKWARD_IF_*`| ``TRUE``, ``FALSE``, | 1669| | :opcode:`!POP_JUMP_IF_TRUE` | | :opcode:`!POP_JUMP_FORWARD_IF_*` | ``NONE`` and ``NOT_NONE`` variants | 1670| | | for each direction | 1671| | | | 1672+------------------------------------+------------------------------------+-----------------------------------------+ 1673| | :opcode:`!SETUP_WITH` | :opcode:`BEFORE_WITH` | :keyword:`with` block setup | 1674| | :opcode:`!SETUP_ASYNC_WITH` | | | 1675+------------------------------------+------------------------------------+-----------------------------------------+ 1676 1677.. [#bytecode-jump] All jump opcodes are now relative, including the 1678 existing :opcode:`!JUMP_IF_TRUE_OR_POP` and :opcode:`!JUMP_IF_FALSE_OR_POP`. 1679 The argument is now an offset from the current instruction 1680 rather than an absolute location. 1681 1682 1683.. _whatsnew311-changed-opcodes: 1684.. _whatsnew311-removed-opcodes: 1685.. _whatsnew311-changed-removed-opcodes: 1686 1687Changed/removed opcodes 1688----------------------- 1689 1690* Changed :opcode:`MATCH_CLASS` and :opcode:`MATCH_KEYS` 1691 to no longer push an additional boolean value to indicate success/failure. 1692 Instead, ``None`` is pushed on failure 1693 in place of the tuple of extracted values. 1694 1695* Changed opcodes that work with exceptions to reflect them 1696 now being represented as one item on the stack instead of three 1697 (see :gh:`89874`). 1698 1699* Removed :opcode:`!COPY_DICT_WITHOUT_KEYS`, :opcode:`!GEN_START`, 1700 :opcode:`!POP_BLOCK`, :opcode:`!SETUP_FINALLY` and :opcode:`!YIELD_FROM`. 1701 1702 1703.. _whatsnew311-deprecated: 1704.. _whatsnew311-python-api-deprecated: 1705 1706Deprecated 1707========== 1708 1709This section lists Python APIs that have been deprecated in Python 3.11. 1710 1711Deprecated C APIs are :ref:`listed separately <whatsnew311-c-api-deprecated>`. 1712 1713 1714.. _whatsnew311-deprecated-language: 1715.. _whatsnew311-deprecated-builtins: 1716 1717Language/Builtins 1718----------------- 1719 1720* Chaining :class:`classmethod` descriptors (introduced in :issue:`19072`) 1721 is now deprecated. It can no longer be used to wrap other descriptors 1722 such as :class:`property`. The core design of this feature was flawed 1723 and caused a number of downstream problems. To "pass-through" a 1724 :class:`classmethod`, consider using the :attr:`!__wrapped__` attribute 1725 that was added in Python 3.10. 1726 (Contributed by Raymond Hettinger in :gh:`89519`.) 1727 1728* Octal escapes in string and bytes literals with values larger than ``0o377`` 1729 (255 in decimal) now produce a :exc:`DeprecationWarning`. 1730 In a future Python version, they will raise a :exc:`SyntaxWarning` and 1731 eventually a :exc:`SyntaxError`. 1732 (Contributed by Serhiy Storchaka in :gh:`81548`.) 1733 1734* The delegation of :func:`int` to :meth:`~object.__trunc__` is now deprecated. 1735 Calling ``int(a)`` when ``type(a)`` implements :meth:`!__trunc__` but not 1736 :meth:`~object.__int__` or :meth:`~object.__index__` now raises 1737 a :exc:`DeprecationWarning`. 1738 (Contributed by Zackery Spytz in :issue:`44977`.) 1739 1740 1741.. _whatsnew311-deprecated-modules: 1742 1743Modules 1744------- 1745 1746.. _whatsnew311-pep594: 1747 1748* :pep:`594` led to the deprecations of the following modules 1749 slated for removal in Python 3.13: 1750 1751 +---------------------+---------------------+---------------------+---------------------+---------------------+ 1752 | :mod:`!aifc` | :mod:`!chunk` | :mod:`!msilib` | :mod:`!pipes` | :mod:`!telnetlib` | 1753 +---------------------+---------------------+---------------------+---------------------+---------------------+ 1754 | :mod:`!audioop` | :mod:`!crypt` | :mod:`!nis` | :mod:`!sndhdr` | :mod:`!uu` | 1755 +---------------------+---------------------+---------------------+---------------------+---------------------+ 1756 | :mod:`!cgi` | :mod:`!imghdr` | :mod:`!nntplib` | :mod:`!spwd` | :mod:`!xdrlib` | 1757 +---------------------+---------------------+---------------------+---------------------+---------------------+ 1758 | :mod:`!cgitb` | :mod:`!mailcap` | :mod:`!ossaudiodev` | :mod:`!sunau` | | 1759 +---------------------+---------------------+---------------------+---------------------+---------------------+ 1760 1761 (Contributed by Brett Cannon in :issue:`47061` and Victor Stinner in 1762 :gh:`68966`.) 1763 1764* The :mod:`!asynchat`, :mod:`!asyncore` and :mod:`!smtpd` modules have been 1765 deprecated since at least Python 3.6. Their documentation and deprecation 1766 warnings have now been updated to note they will be removed in Python 3.12. 1767 (Contributed by Hugo van Kemenade in :issue:`47022`.) 1768 1769* The :mod:`!lib2to3` package and ``2to3`` tool 1770 are now deprecated and may not be able to parse Python 3.10 or newer. 1771 See :pep:`617`, introducing the new PEG parser, for details. 1772 (Contributed by Victor Stinner in :issue:`40360`.) 1773 1774* Undocumented modules :mod:`!sre_compile`, :mod:`!sre_constants` 1775 and :mod:`!sre_parse` are now deprecated. 1776 (Contributed by Serhiy Storchaka in :issue:`47152`.) 1777 1778 1779.. _whatsnew311-deprecated-stdlib: 1780 1781Standard Library 1782---------------- 1783 1784* The following have been deprecated in :mod:`configparser` since Python 3.2. 1785 Their deprecation warnings have now been updated to note they will be removed 1786 in Python 3.12: 1787 1788 * the :class:`!configparser.SafeConfigParser` class 1789 * the :attr:`!configparser.ParsingError.filename` property 1790 * the :meth:`!configparser.RawConfigParser.readfp` method 1791 1792 (Contributed by Hugo van Kemenade in :issue:`45173`.) 1793 1794* :class:`!configparser.LegacyInterpolation` has been deprecated in the docstring 1795 since Python 3.2, and is not listed in the :mod:`configparser` documentation. 1796 It now emits a :exc:`DeprecationWarning` and will be removed 1797 in Python 3.13. Use :class:`configparser.BasicInterpolation` or 1798 :class:`configparser.ExtendedInterpolation` instead. 1799 (Contributed by Hugo van Kemenade in :issue:`46607`.) 1800 1801* The older set of :mod:`importlib.resources` functions were deprecated 1802 in favor of the replacements added in Python 3.9 1803 and will be removed in a future Python version, 1804 due to not supporting resources located within package subdirectories: 1805 1806 * :func:`!importlib.resources.contents` 1807 * :func:`!importlib.resources.is_resource` 1808 * :func:`!importlib.resources.open_binary` 1809 * :func:`!importlib.resources.open_text` 1810 * :func:`!importlib.resources.read_binary` 1811 * :func:`!importlib.resources.read_text` 1812 * :func:`!importlib.resources.path` 1813 1814* The :func:`locale.getdefaultlocale` function is deprecated and will be 1815 removed in Python 3.15. Use :func:`locale.setlocale`, 1816 :func:`locale.getpreferredencoding(False) <locale.getpreferredencoding>` and 1817 :func:`locale.getlocale` functions instead. 1818 (Contributed by Victor Stinner in :gh:`90817`.) 1819 1820* The :func:`!locale.resetlocale` function is deprecated and will be 1821 removed in Python 3.13. Use ``locale.setlocale(locale.LC_ALL, "")`` instead. 1822 (Contributed by Victor Stinner in :gh:`90817`.) 1823 1824* Stricter rules will now be applied for numerical group references 1825 and group names in :ref:`regular expressions <re-syntax>`. 1826 Only sequences of ASCII digits will now be accepted as a numerical reference, 1827 and the group name in :class:`bytes` patterns and replacement strings 1828 can only contain ASCII letters, digits and underscores. 1829 For now, a deprecation warning is raised for syntax violating these rules. 1830 (Contributed by Serhiy Storchaka in :gh:`91760`.) 1831 1832* In the :mod:`re` module, the :func:`!re.template` function 1833 and the corresponding :const:`!re.TEMPLATE` and :const:`!re.T` flags 1834 are deprecated, as they were undocumented and lacked an obvious purpose. 1835 They will be removed in Python 3.13. 1836 (Contributed by Serhiy Storchaka and Miro Hrončok in :gh:`92728`.) 1837 1838* :func:`!turtle.settiltangle` has been deprecated since Python 3.1; 1839 it now emits a deprecation warning and will be removed in Python 3.13. Use 1840 :func:`turtle.tiltangle` instead (it was earlier incorrectly marked 1841 as deprecated, and its docstring is now corrected). 1842 (Contributed by Hugo van Kemenade in :issue:`45837`.) 1843 1844* :class:`typing.Text`, which exists solely to provide compatibility support 1845 between Python 2 and Python 3 code, is now deprecated. Its removal is 1846 currently unplanned, but users are encouraged to use :class:`str` instead 1847 wherever possible. 1848 (Contributed by Alex Waygood in :gh:`92332`.) 1849 1850* The keyword argument syntax for constructing :data:`typing.TypedDict` types 1851 is now deprecated. Support will be removed in Python 3.13. (Contributed by 1852 Jingchen Ye in :gh:`90224`.) 1853 1854* :class:`!webbrowser.MacOSX` is deprecated and will be removed in Python 3.13. 1855 It is untested, undocumented, and not used by :mod:`webbrowser` itself. 1856 (Contributed by Donghee Na in :issue:`42255`.) 1857 1858* The behavior of returning a value from a :class:`~unittest.TestCase` and 1859 :class:`~unittest.IsolatedAsyncioTestCase` test methods (other than the 1860 default ``None`` value) is now deprecated. 1861 1862* Deprecated the following not-formally-documented :mod:`unittest` functions, 1863 scheduled for removal in Python 3.13: 1864 1865 * :func:`!unittest.findTestCases` 1866 * :func:`!unittest.makeSuite` 1867 * :func:`!unittest.getTestCaseNames` 1868 1869 Use :class:`~unittest.TestLoader` methods instead: 1870 1871 * :meth:`unittest.TestLoader.loadTestsFromModule` 1872 * :meth:`unittest.TestLoader.loadTestsFromTestCase` 1873 * :meth:`unittest.TestLoader.getTestCaseNames` 1874 1875 (Contributed by Erlend E. Aasland in :issue:`5846`.) 1876 1877* :meth:`!unittest.TestProgram.usageExit` is marked deprecated, to be removed 1878 in 3.13. 1879 (Contributed by Carlos Damázio in :gh:`67048`.) 1880 1881 1882.. _whatsnew311-pending-removal: 1883.. _whatsnew311-python-api-pending-removal: 1884 1885Pending Removal in Python 3.12 1886============================== 1887 1888The following Python APIs have been deprecated in earlier Python releases, 1889and will be removed in Python 3.12. 1890 1891C APIs pending removal are 1892:ref:`listed separately <whatsnew311-c-api-pending-removal>`. 1893 1894* The :mod:`!asynchat` module 1895* The :mod:`!asyncore` module 1896* The :ref:`entire distutils package <distutils-deprecated>` 1897* The :mod:`!imp` module 1898* The :class:`typing.io <typing.IO>` namespace 1899* The :class:`typing.re <typing.Pattern>` namespace 1900* :func:`!cgi.log` 1901* :func:`!importlib.find_loader` 1902* :meth:`!importlib.abc.Loader.module_repr` 1903* :meth:`!importlib.abc.MetaPathFinder.find_module` 1904* :meth:`!importlib.abc.PathEntryFinder.find_loader` 1905* :meth:`!importlib.abc.PathEntryFinder.find_module` 1906* :meth:`!importlib.machinery.BuiltinImporter.find_module` 1907* :meth:`!importlib.machinery.BuiltinLoader.module_repr` 1908* :meth:`!importlib.machinery.FileFinder.find_loader` 1909* :meth:`!importlib.machinery.FileFinder.find_module` 1910* :meth:`!importlib.machinery.FrozenImporter.find_module` 1911* :meth:`!importlib.machinery.FrozenLoader.module_repr` 1912* :meth:`!importlib.machinery.PathFinder.find_module` 1913* :meth:`!importlib.machinery.WindowsRegistryFinder.find_module` 1914* :func:`!importlib.util.module_for_loader` 1915* :func:`!importlib.util.set_loader_wrapper` 1916* :func:`!importlib.util.set_package_wrapper` 1917* :class:`!pkgutil.ImpImporter` 1918* :class:`!pkgutil.ImpLoader` 1919* :meth:`!pathlib.Path.link_to` 1920* :func:`!sqlite3.enable_shared_cache` 1921* :func:`!sqlite3.OptimizedUnicode` 1922* :envvar:`!PYTHONTHREADDEBUG` environment variable 1923* The following deprecated aliases in :mod:`unittest`: 1924 1925 ============================ =============================== =============== 1926 Deprecated alias Method Name Deprecated in 1927 ============================ =============================== =============== 1928 ``failUnless`` :meth:`.assertTrue` 3.1 1929 ``failIf`` :meth:`.assertFalse` 3.1 1930 ``failUnlessEqual`` :meth:`.assertEqual` 3.1 1931 ``failIfEqual`` :meth:`.assertNotEqual` 3.1 1932 ``failUnlessAlmostEqual`` :meth:`.assertAlmostEqual` 3.1 1933 ``failIfAlmostEqual`` :meth:`.assertNotAlmostEqual` 3.1 1934 ``failUnlessRaises`` :meth:`.assertRaises` 3.1 1935 ``assert_`` :meth:`.assertTrue` 3.2 1936 ``assertEquals`` :meth:`.assertEqual` 3.2 1937 ``assertNotEquals`` :meth:`.assertNotEqual` 3.2 1938 ``assertAlmostEquals`` :meth:`.assertAlmostEqual` 3.2 1939 ``assertNotAlmostEquals`` :meth:`.assertNotAlmostEqual` 3.2 1940 ``assertRegexpMatches`` :meth:`.assertRegex` 3.2 1941 ``assertRaisesRegexp`` :meth:`.assertRaisesRegex` 3.2 1942 ``assertNotRegexpMatches`` :meth:`.assertNotRegex` 3.5 1943 ============================ =============================== =============== 1944 1945.. _whatsnew311-removed: 1946.. _whatsnew311-python-api-removed: 1947 1948Removed 1949======= 1950 1951This section lists Python APIs that have been removed in Python 3.11. 1952 1953Removed C APIs are :ref:`listed separately <whatsnew311-c-api-removed>`. 1954 1955* Removed the :func:`!@asyncio.coroutine` :term:`decorator` 1956 enabling legacy generator-based coroutines to be compatible with 1957 :keyword:`async` / :keyword:`await` code. 1958 The function has been deprecated since Python 3.8 and the removal was 1959 initially scheduled for Python 3.10. Use :keyword:`async def` instead. 1960 (Contributed by Illia Volochii in :issue:`43216`.) 1961 1962* Removed :class:`!asyncio.coroutines.CoroWrapper` used for wrapping legacy 1963 generator-based coroutine objects in the debug mode. 1964 (Contributed by Illia Volochii in :issue:`43216`.) 1965 1966* Due to significant security concerns, the *reuse_address* parameter of 1967 :meth:`asyncio.loop.create_datagram_endpoint`, disabled in Python 3.9, is 1968 now entirely removed. This is because of the behavior of the socket option 1969 ``SO_REUSEADDR`` in UDP. 1970 (Contributed by Hugo van Kemenade in :issue:`45129`.) 1971 1972* Removed the :mod:`!binhex` module, deprecated in Python 3.9. 1973 Also removed the related, similarly-deprecated :mod:`binascii` functions: 1974 1975 * :func:`!binascii.a2b_hqx` 1976 * :func:`!binascii.b2a_hqx` 1977 * :func:`!binascii.rlecode_hqx` 1978 * :func:`!binascii.rldecode_hqx` 1979 1980 The :func:`binascii.crc_hqx` function remains available. 1981 1982 (Contributed by Victor Stinner in :issue:`45085`.) 1983 1984* Removed the :mod:`!distutils` ``bdist_msi`` command deprecated in Python 3.9. 1985 Use ``bdist_wheel`` (wheel packages) instead. 1986 (Contributed by Hugo van Kemenade in :issue:`45124`.) 1987 1988* Removed the :meth:`~object.__getitem__` methods of 1989 :class:`xml.dom.pulldom.DOMEventStream`, :class:`wsgiref.util.FileWrapper` 1990 and :class:`fileinput.FileInput`, deprecated since Python 3.9. 1991 (Contributed by Hugo van Kemenade in :issue:`45132`.) 1992 1993* Removed the deprecated :mod:`gettext` functions 1994 :func:`!lgettext`, :func:`!ldgettext`, 1995 :func:`!lngettext` and :func:`!ldngettext`. 1996 Also removed the :func:`!bind_textdomain_codeset` function, 1997 the :meth:`!NullTranslations.output_charset` and 1998 :meth:`!NullTranslations.set_output_charset` methods, 1999 and the *codeset* parameter of :func:`!translation` and :func:`!install`, 2000 since they are only used for the :func:`!l*gettext` functions. 2001 (Contributed by Donghee Na and Serhiy Storchaka in :issue:`44235`.) 2002 2003* Removed from the :mod:`inspect` module: 2004 2005 * The :func:`!getargspec` function, deprecated since Python 3.0; 2006 use :func:`inspect.signature` or :func:`inspect.getfullargspec` instead. 2007 2008 * The :func:`!formatargspec` function, deprecated since Python 3.5; 2009 use the :func:`inspect.signature` function 2010 or the :class:`inspect.Signature` object directly. 2011 2012 * The undocumented :meth:`!Signature.from_builtin` 2013 and :meth:`!Signature.from_function` methods, deprecated since Python 3.5; 2014 use the :meth:`Signature.from_callable() <inspect.Signature.from_callable>` 2015 method instead. 2016 2017 (Contributed by Hugo van Kemenade in :issue:`45320`.) 2018 2019* Removed the :meth:`~object.__class_getitem__` method 2020 from :class:`pathlib.PurePath`, 2021 because it was not used and added by mistake in previous versions. 2022 (Contributed by Nikita Sobolev in :issue:`46483`.) 2023 2024* Removed the :class:`!MailmanProxy` class in the :mod:`!smtpd` module, 2025 as it is unusable without the external :mod:`!mailman` package. 2026 (Contributed by Donghee Na in :issue:`35800`.) 2027 2028* Removed the deprecated :meth:`!split` method of :class:`!_tkinter.TkappType`. 2029 (Contributed by Erlend E. Aasland in :issue:`38371`.) 2030 2031* Removed namespace package support from :mod:`unittest` discovery. 2032 It was introduced in Python 3.4 but has been broken since Python 3.7. 2033 (Contributed by Inada Naoki in :issue:`23882`.) 2034 2035* Removed the undocumented private :meth:`!float.__set_format__` method, 2036 previously known as :meth:`!float.__setformat__` in Python 3.7. 2037 Its docstring said: "You probably don't want to use this function. 2038 It exists mainly to be used in Python's test suite." 2039 (Contributed by Victor Stinner in :issue:`46852`.) 2040 2041* The :option:`!--experimental-isolated-subinterpreters` configure flag 2042 (and corresponding :c:macro:`!EXPERIMENTAL_ISOLATED_SUBINTERPRETERS` macro) 2043 have been removed. 2044 2045* :pypi:`Pynche` 2046 --- The Pythonically Natural Color and Hue Editor --- has been moved out 2047 of ``Tools/scripts`` and is `being developed independently 2048 <https://gitlab.com/warsaw/pynche/-/tree/main>`_ from the Python source tree. 2049 2050 2051.. _whatsnew311-porting: 2052.. _whatsnew311-python-api-porting: 2053 2054Porting to Python 3.11 2055====================== 2056 2057This section lists previously described changes and other bugfixes 2058in the Python API that may require changes to your Python code. 2059 2060Porting notes for the C API are 2061:ref:`listed separately <whatsnew311-c-api-porting>`. 2062 2063* :func:`open`, :func:`io.open`, :func:`codecs.open` and 2064 :class:`fileinput.FileInput` no longer accept ``'U'`` ("universal newline") 2065 in the file mode. In Python 3, "universal newline" mode is used by default 2066 whenever a file is opened in text mode, 2067 and the ``'U'`` flag has been deprecated since Python 3.3. 2068 The :ref:`newline parameter <open-newline-parameter>` 2069 to these functions controls how universal newlines work. 2070 (Contributed by Victor Stinner in :issue:`37330`.) 2071 2072* :class:`ast.AST` node positions are now validated when provided to 2073 :func:`compile` and other related functions. If invalid positions are detected, 2074 a :exc:`ValueError` will be raised. (Contributed by Pablo Galindo in :gh:`93351`) 2075 2076* Prohibited passing non-:class:`concurrent.futures.ThreadPoolExecutor` 2077 executors to :meth:`asyncio.loop.set_default_executor` 2078 following a deprecation in Python 3.8. 2079 (Contributed by Illia Volochii in :issue:`43234`.) 2080 2081* :mod:`calendar`: The :class:`calendar.LocaleTextCalendar` and 2082 :class:`calendar.LocaleHTMLCalendar` classes now use 2083 :func:`locale.getlocale`, instead of using :func:`locale.getdefaultlocale`, 2084 if no locale is specified. 2085 (Contributed by Victor Stinner in :issue:`46659`.) 2086 2087* The :mod:`pdb` module now reads the :file:`.pdbrc` configuration file with 2088 the ``'UTF-8'`` encoding. 2089 (Contributed by Srinivas Reddy Thatiparthy (శ్రీనివాస్ రెడ్డి తాటిపర్తి) in :issue:`41137`.) 2090 2091* The *population* parameter of :func:`random.sample` must be a sequence, 2092 and automatic conversion of :class:`set`\s to :class:`list`\s 2093 is no longer supported. Also, if the sample size 2094 is larger than the population size, a :exc:`ValueError` is raised. 2095 (Contributed by Raymond Hettinger in :issue:`40465`.) 2096 2097* The *random* optional parameter of :func:`random.shuffle` was removed. 2098 It was previously an arbitrary random function to use for the shuffle; 2099 now, :func:`random.random` (its previous default) will always be used. 2100 2101* In :mod:`re` :ref:`re-syntax`, global inline flags (e.g. ``(?i)``) 2102 can now only be used at the start of regular expressions. 2103 Using them elsewhere has been deprecated since Python 3.6. 2104 (Contributed by Serhiy Storchaka in :issue:`47066`.) 2105 2106* In the :mod:`re` module, several long-standing bugs where fixed that, 2107 in rare cases, could cause capture groups to get the wrong result. 2108 Therefore, this could change the captured output in these cases. 2109 (Contributed by Ma Lin in :issue:`35859`.) 2110 2111 2112.. _whatsnew311-build-changes: 2113 2114Build Changes 2115============= 2116 2117* CPython now has :pep:`11` :pep:`Tier 3 support <11#tier-3>` for 2118 cross compiling to the `WebAssembly <https://webassembly.org/>`_ platforms 2119 `Emscripten <https://emscripten.org/>`_ 2120 (``wasm32-unknown-emscripten``, i.e. Python in the browser) 2121 and `WebAssembly System Interface (WASI) <https://wasi.dev/>`_ 2122 (``wasm32-unknown-wasi``). 2123 The effort is inspired by previous work like `Pyodide <https://pyodide.org/>`_. 2124 These platforms provide a limited subset of POSIX APIs; Python standard 2125 libraries features and modules related to networking, processes, threading, 2126 signals, mmap, and users/groups are not available or don't work. 2127 (Emscripten contributed by Christian Heimes and Ethan Smith in :gh:`84461` 2128 and WASI contributed by Christian Heimes in :gh:`90473`; 2129 platforms promoted in :gh:`95085`) 2130 2131* Building CPython now requires: 2132 2133 * A `C11 <https://en.cppreference.com/w/c/11>`_ compiler and standard library. 2134 `Optional C11 features 2135 <https://en.wikipedia.org/wiki/C11_(C_standard_revision)#Optional_features>`_ 2136 are not required. 2137 (Contributed by Victor Stinner in :issue:`46656`, 2138 :issue:`45440` and :issue:`46640`.) 2139 2140 * Support for `IEEE 754 <https://en.wikipedia.org/wiki/IEEE_754>`_ 2141 floating-point numbers. 2142 (Contributed by Victor Stinner in :issue:`46917`.) 2143 2144* The :c:macro:`!Py_NO_NAN` macro has been removed. 2145 Since CPython now requires IEEE 754 floats, NaN values are always available. 2146 (Contributed by Victor Stinner in :issue:`46656`.) 2147 2148* The :mod:`tkinter` package now requires `Tcl/Tk <https://www.tcl.tk>`_ 2149 version 8.5.12 or newer. 2150 (Contributed by Serhiy Storchaka in :issue:`46996`.) 2151 2152* Build dependencies, compiler flags, and linker flags for most stdlib 2153 extension modules are now detected by :program:`configure`. libffi, libnsl, 2154 libsqlite3, zlib, bzip2, liblzma, libcrypt, Tcl/Tk, and uuid flags 2155 are detected by `pkg-config 2156 <https://www.freedesktop.org/wiki/Software/pkg-config/>`_ (when available). 2157 :mod:`tkinter` now requires a pkg-config command 2158 to detect development settings for `Tcl/Tk`_ headers and libraries. 2159 (Contributed by Christian Heimes and Erlend Egeberg Aasland in 2160 :issue:`45847`, :issue:`45747`, and :issue:`45763`.) 2161 2162* libpython is no longer linked against libcrypt. 2163 (Contributed by Mike Gilbert in :issue:`45433`.) 2164 2165* CPython can now be built with the 2166 `ThinLTO <https://clang.llvm.org/docs/ThinLTO.html>`_ option 2167 via passing ``thin`` to :option:`--with-lto`, i.e. ``--with-lto=thin``. 2168 (Contributed by Donghee Na and Brett Holman in :issue:`44340`.) 2169 2170* Freelists for object structs can now be disabled. A new :program:`configure` 2171 option :option:`--without-freelists` can be used to disable all freelists 2172 except empty tuple singleton. 2173 (Contributed by Christian Heimes in :issue:`45522`.) 2174 2175* ``Modules/Setup`` and ``Modules/makesetup`` have been improved and tied up. 2176 Extension modules can now be built through ``makesetup``. All except some 2177 test modules can be linked statically into a main binary or library. 2178 (Contributed by Brett Cannon and Christian Heimes in :issue:`45548`, 2179 :issue:`45570`, :issue:`45571`, and :issue:`43974`.) 2180 2181 .. note:: 2182 Use the environment variables :envvar:`!TCLTK_CFLAGS` and 2183 :envvar:`!TCLTK_LIBS` to manually specify the location of Tcl/Tk headers 2184 and libraries. The :program:`configure` options 2185 :option:`!--with-tcltk-includes` and :option:`!--with-tcltk-libs` 2186 have been removed. 2187 2188 On RHEL 7 and CentOS 7 the development packages do not provide ``tcl.pc`` 2189 and ``tk.pc``; use ``TCLTK_LIBS="-ltk8.5 -ltkstub8.5 -ltcl8.5"``. 2190 The directory ``Misc/rhel7`` contains ``.pc`` files and instructions 2191 on how to build Python with RHEL 7's and CentOS 7's Tcl/Tk and OpenSSL. 2192 2193* CPython will now use 30-bit digits by default for the Python :class:`int` 2194 implementation. Previously, the default was to use 30-bit digits on platforms 2195 with ``SIZEOF_VOID_P >= 8``, and 15-bit digits otherwise. It's still possible 2196 to explicitly request use of 15-bit digits via either the 2197 :option:`--enable-big-digits` option to the configure script 2198 or (for Windows) the ``PYLONG_BITS_IN_DIGIT`` variable in ``PC/pyconfig.h``, 2199 but this option may be removed at some point in the future. 2200 (Contributed by Mark Dickinson in :issue:`45569`.) 2201 2202 2203.. _whatsnew311-c-api: 2204 2205C API Changes 2206============= 2207 2208.. _whatsnew311-c-api-new-features: 2209 2210New Features 2211------------ 2212 2213* Add a new :c:func:`PyType_GetName` function to get type's short name. 2214 (Contributed by Hai Shi in :issue:`42035`.) 2215 2216* Add a new :c:func:`PyType_GetQualName` function to get type's qualified name. 2217 (Contributed by Hai Shi in :issue:`42035`.) 2218 2219* Add new :c:func:`PyThreadState_EnterTracing` and 2220 :c:func:`PyThreadState_LeaveTracing` functions to the limited C API to 2221 suspend and resume tracing and profiling. 2222 (Contributed by Victor Stinner in :issue:`43760`.) 2223 2224* Added the :c:data:`Py_Version` constant which bears the same value as 2225 :c:macro:`PY_VERSION_HEX`. 2226 (Contributed by Gabriele N. Tornetta in :issue:`43931`.) 2227 2228* :c:type:`Py_buffer` and APIs are now part of the limited API and the stable 2229 ABI: 2230 2231 * :c:func:`PyObject_CheckBuffer` 2232 * :c:func:`PyObject_GetBuffer` 2233 * :c:func:`PyBuffer_GetPointer` 2234 * :c:func:`PyBuffer_SizeFromFormat` 2235 * :c:func:`PyBuffer_ToContiguous` 2236 * :c:func:`PyBuffer_FromContiguous` 2237 * :c:func:`PyObject_CopyData` 2238 * :c:func:`PyBuffer_IsContiguous` 2239 * :c:func:`PyBuffer_FillContiguousStrides` 2240 * :c:func:`PyBuffer_FillInfo` 2241 * :c:func:`PyBuffer_Release` 2242 * :c:func:`PyMemoryView_FromBuffer` 2243 * :c:member:`~PyBufferProcs.bf_getbuffer` and 2244 :c:member:`~PyBufferProcs.bf_releasebuffer` type slots 2245 2246 (Contributed by Christian Heimes in :issue:`45459`.) 2247 2248* Added the :c:func:`PyType_GetModuleByDef` function, used to get the module 2249 in which a method was defined, in cases where this information is not 2250 available directly (via :c:type:`PyCMethod`). 2251 (Contributed by Petr Viktorin in :issue:`46613`.) 2252 2253* Add new functions to pack and unpack C double (serialize and deserialize): 2254 :c:func:`PyFloat_Pack2`, :c:func:`PyFloat_Pack4`, :c:func:`PyFloat_Pack8`, 2255 :c:func:`PyFloat_Unpack2`, :c:func:`PyFloat_Unpack4` and 2256 :c:func:`PyFloat_Unpack8`. 2257 (Contributed by Victor Stinner in :issue:`46906`.) 2258 2259* Add new functions to get frame object attributes: 2260 :c:func:`PyFrame_GetBuiltins`, :c:func:`PyFrame_GetGenerator`, 2261 :c:func:`PyFrame_GetGlobals`, :c:func:`PyFrame_GetLasti`. 2262 2263* Added two new functions to get and set the active exception instance: 2264 :c:func:`PyErr_GetHandledException` and :c:func:`PyErr_SetHandledException`. 2265 These are alternatives to :c:func:`PyErr_SetExcInfo()` and 2266 :c:func:`PyErr_GetExcInfo()` which work with the legacy 3-tuple 2267 representation of exceptions. 2268 (Contributed by Irit Katriel in :issue:`46343`.) 2269 2270* Added the :c:member:`PyConfig.safe_path` member. 2271 (Contributed by Victor Stinner in :gh:`57684`.) 2272 2273 2274.. _whatsnew311-c-api-porting: 2275 2276Porting to Python 3.11 2277---------------------- 2278 2279.. _whatsnew311-pep670: 2280 2281* Some macros have been converted to static inline functions to avoid 2282 `macro pitfalls <https://gcc.gnu.org/onlinedocs/cpp/Macro-Pitfalls.html>`_. 2283 The change should be mostly transparent to users, 2284 as the replacement functions will cast their arguments to the expected types 2285 to avoid compiler warnings due to static type checks. 2286 However, when the limited C API is set to >=3.11, 2287 these casts are not done, 2288 and callers will need to cast arguments to their expected types. 2289 See :pep:`670` for more details. 2290 (Contributed by Victor Stinner and Erlend E. Aasland in :gh:`89653`.) 2291 2292* :c:func:`PyErr_SetExcInfo()` no longer uses the ``type`` and ``traceback`` 2293 arguments, the interpreter now derives those values from the exception 2294 instance (the ``value`` argument). The function still steals references 2295 of all three arguments. 2296 (Contributed by Irit Katriel in :issue:`45711`.) 2297 2298* :c:func:`PyErr_GetExcInfo()` now derives the ``type`` and ``traceback`` 2299 fields of the result from the exception instance (the ``value`` field). 2300 (Contributed by Irit Katriel in :issue:`45711`.) 2301 2302* :c:struct:`_frozen` has a new ``is_package`` field to indicate whether 2303 or not the frozen module is a package. Previously, a negative value 2304 in the ``size`` field was the indicator. Now only non-negative values 2305 be used for ``size``. 2306 (Contributed by Kumar Aditya in :issue:`46608`.) 2307 2308* :c:func:`_PyFrameEvalFunction` now takes ``_PyInterpreterFrame*`` 2309 as its second parameter, instead of ``PyFrameObject*``. 2310 See :pep:`523` for more details of how to use this function pointer type. 2311 2312* :c:func:`!PyCode_New` and :c:func:`!PyCode_NewWithPosOnlyArgs` now take 2313 an additional ``exception_table`` argument. 2314 Using these functions should be avoided, if at all possible. 2315 To get a custom code object: create a code object using the compiler, 2316 then get a modified version with the ``replace`` method. 2317 2318* :c:type:`PyCodeObject` no longer has the ``co_code``, ``co_varnames``, 2319 ``co_cellvars`` and ``co_freevars`` fields. Instead, use 2320 :c:func:`PyCode_GetCode`, :c:func:`PyCode_GetVarnames`, 2321 :c:func:`PyCode_GetCellvars` and :c:func:`PyCode_GetFreevars` respectively 2322 to access them via the C API. 2323 (Contributed by Brandt Bucher in :issue:`46841` and Ken Jin in :gh:`92154` 2324 and :gh:`94936`.) 2325 2326* The old trashcan macros (``Py_TRASHCAN_SAFE_BEGIN``/``Py_TRASHCAN_SAFE_END``) 2327 are now deprecated. They should be replaced by the new macros 2328 ``Py_TRASHCAN_BEGIN`` and ``Py_TRASHCAN_END``. 2329 2330 A tp_dealloc function that has the old macros, such as:: 2331 2332 static void 2333 mytype_dealloc(mytype *p) 2334 { 2335 PyObject_GC_UnTrack(p); 2336 Py_TRASHCAN_SAFE_BEGIN(p); 2337 ... 2338 Py_TRASHCAN_SAFE_END 2339 } 2340 2341 should migrate to the new macros as follows:: 2342 2343 static void 2344 mytype_dealloc(mytype *p) 2345 { 2346 PyObject_GC_UnTrack(p); 2347 Py_TRASHCAN_BEGIN(p, mytype_dealloc) 2348 ... 2349 Py_TRASHCAN_END 2350 } 2351 2352 Note that ``Py_TRASHCAN_BEGIN`` has a second argument which 2353 should be the deallocation function it is in. 2354 2355 To support older Python versions in the same codebase, you 2356 can define the following macros and use them throughout 2357 the code (credit: these were copied from the ``mypy`` codebase):: 2358 2359 #if PY_VERSION_HEX >= 0x03080000 2360 # define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_BEGIN(op, dealloc) 2361 # define CPy_TRASHCAN_END(op) Py_TRASHCAN_END 2362 #else 2363 # define CPy_TRASHCAN_BEGIN(op, dealloc) Py_TRASHCAN_SAFE_BEGIN(op) 2364 # define CPy_TRASHCAN_END(op) Py_TRASHCAN_SAFE_END(op) 2365 #endif 2366 2367* The :c:func:`PyType_Ready` function now raises an error if a type is defined 2368 with the :c:macro:`Py_TPFLAGS_HAVE_GC` flag set but has no traverse function 2369 (:c:member:`PyTypeObject.tp_traverse`). 2370 (Contributed by Victor Stinner in :issue:`44263`.) 2371 2372* Heap types with the :c:macro:`Py_TPFLAGS_IMMUTABLETYPE` flag can now inherit 2373 the :pep:`590` vectorcall protocol. Previously, this was only possible for 2374 :ref:`static types <static-types>`. 2375 (Contributed by Erlend E. Aasland in :issue:`43908`) 2376 2377* Since :c:func:`Py_TYPE()` is changed to a inline static function, 2378 ``Py_TYPE(obj) = new_type`` must be replaced with 2379 ``Py_SET_TYPE(obj, new_type)``: see the :c:func:`Py_SET_TYPE()` function 2380 (available since Python 3.9). For backward compatibility, this macro can be 2381 used:: 2382 2383 #if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_TYPE) 2384 static inline void _Py_SET_TYPE(PyObject *ob, PyTypeObject *type) 2385 { ob->ob_type = type; } 2386 #define Py_SET_TYPE(ob, type) _Py_SET_TYPE((PyObject*)(ob), type) 2387 #endif 2388 2389 (Contributed by Victor Stinner in :issue:`39573`.) 2390 2391* Since :c:func:`Py_SIZE()` is changed to a inline static function, 2392 ``Py_SIZE(obj) = new_size`` must be replaced with 2393 ``Py_SET_SIZE(obj, new_size)``: see the :c:func:`Py_SET_SIZE()` function 2394 (available since Python 3.9). For backward compatibility, this macro can be 2395 used:: 2396 2397 #if PY_VERSION_HEX < 0x030900A4 && !defined(Py_SET_SIZE) 2398 static inline void _Py_SET_SIZE(PyVarObject *ob, Py_ssize_t size) 2399 { ob->ob_size = size; } 2400 #define Py_SET_SIZE(ob, size) _Py_SET_SIZE((PyVarObject*)(ob), size) 2401 #endif 2402 2403 (Contributed by Victor Stinner in :issue:`39573`.) 2404 2405* ``<Python.h>`` no longer includes the header files ``<stdlib.h>``, 2406 ``<stdio.h>``, ``<errno.h>`` and ``<string.h>`` when the ``Py_LIMITED_API`` 2407 macro is set to ``0x030b0000`` (Python 3.11) or higher. C extensions should 2408 explicitly include the header files after ``#include <Python.h>``. 2409 (Contributed by Victor Stinner in :issue:`45434`.) 2410 2411* The non-limited API files ``cellobject.h``, ``classobject.h``, ``code.h``, ``context.h``, 2412 ``funcobject.h``, ``genobject.h`` and ``longintrepr.h`` have been moved to 2413 the ``Include/cpython`` directory. Moreover, the ``eval.h`` header file was 2414 removed. These files must not be included directly, as they are already 2415 included in ``Python.h``: :ref:`Include Files <api-includes>`. If they have 2416 been included directly, consider including ``Python.h`` instead. 2417 (Contributed by Victor Stinner in :issue:`35134`.) 2418 2419* The :c:func:`!PyUnicode_CHECK_INTERNED` macro has been excluded from the 2420 limited C API. It was never usable there, because it used internal structures 2421 which are not available in the limited C API. 2422 (Contributed by Victor Stinner in :issue:`46007`.) 2423 2424* The following frame functions and type are now directly available with 2425 ``#include <Python.h>``, it's no longer needed to add 2426 ``#include <frameobject.h>``: 2427 2428 * :c:func:`PyFrame_Check` 2429 * :c:func:`PyFrame_GetBack` 2430 * :c:func:`PyFrame_GetBuiltins` 2431 * :c:func:`PyFrame_GetGenerator` 2432 * :c:func:`PyFrame_GetGlobals` 2433 * :c:func:`PyFrame_GetLasti` 2434 * :c:func:`PyFrame_GetLocals` 2435 * :c:type:`PyFrame_Type` 2436 2437 (Contributed by Victor Stinner in :gh:`93937`.) 2438 2439.. _pyframeobject-3.11-hiding: 2440 2441* The :c:type:`PyFrameObject` structure members have been removed from the 2442 public C API. 2443 2444 While the documentation notes that the :c:type:`PyFrameObject` fields are 2445 subject to change at any time, they have been stable for a long time and were 2446 used in several popular extensions. 2447 2448 In Python 3.11, the frame struct was reorganized to allow performance 2449 optimizations. Some fields were removed entirely, as they were details of the 2450 old implementation. 2451 2452 :c:type:`PyFrameObject` fields: 2453 2454 * ``f_back``: use :c:func:`PyFrame_GetBack`. 2455 * ``f_blockstack``: removed. 2456 * ``f_builtins``: use :c:func:`PyFrame_GetBuiltins`. 2457 * ``f_code``: use :c:func:`PyFrame_GetCode`. 2458 * ``f_gen``: use :c:func:`PyFrame_GetGenerator`. 2459 * ``f_globals``: use :c:func:`PyFrame_GetGlobals`. 2460 * ``f_iblock``: removed. 2461 * ``f_lasti``: use :c:func:`PyFrame_GetLasti`. 2462 Code using ``f_lasti`` with ``PyCode_Addr2Line()`` should use 2463 :c:func:`PyFrame_GetLineNumber` instead; it may be faster. 2464 * ``f_lineno``: use :c:func:`PyFrame_GetLineNumber` 2465 * ``f_locals``: use :c:func:`PyFrame_GetLocals`. 2466 * ``f_stackdepth``: removed. 2467 * ``f_state``: no public API (renamed to ``f_frame.f_state``). 2468 * ``f_trace``: no public API. 2469 * ``f_trace_lines``: use ``PyObject_GetAttrString((PyObject*)frame, "f_trace_lines")``. 2470 * ``f_trace_opcodes``: use ``PyObject_GetAttrString((PyObject*)frame, "f_trace_opcodes")``. 2471 * ``f_localsplus``: no public API (renamed to ``f_frame.localsplus``). 2472 * ``f_valuestack``: removed. 2473 2474 The Python frame object is now created lazily. A side effect is that the 2475 :attr:`~frame.f_back` member must not be accessed directly, 2476 since its value is now also 2477 computed lazily. The :c:func:`PyFrame_GetBack` function must be called 2478 instead. 2479 2480 Debuggers that accessed the :attr:`~frame.f_locals` directly *must* call 2481 :c:func:`PyFrame_GetLocals` instead. They no longer need to call 2482 :c:func:`!PyFrame_FastToLocalsWithError` or :c:func:`!PyFrame_LocalsToFast`, 2483 in fact they should not call those functions. The necessary updating of the 2484 frame is now managed by the virtual machine. 2485 2486 Code defining ``PyFrame_GetCode()`` on Python 3.8 and older:: 2487 2488 #if PY_VERSION_HEX < 0x030900B1 2489 static inline PyCodeObject* PyFrame_GetCode(PyFrameObject *frame) 2490 { 2491 Py_INCREF(frame->f_code); 2492 return frame->f_code; 2493 } 2494 #endif 2495 2496 Code defining ``PyFrame_GetBack()`` on Python 3.8 and older:: 2497 2498 #if PY_VERSION_HEX < 0x030900B1 2499 static inline PyFrameObject* PyFrame_GetBack(PyFrameObject *frame) 2500 { 2501 Py_XINCREF(frame->f_back); 2502 return frame->f_back; 2503 } 2504 #endif 2505 2506 Or use the `pythoncapi_compat project 2507 <https://github.com/python/pythoncapi-compat>`__ to get these two 2508 functions on older Python versions. 2509 2510* Changes of the :c:type:`PyThreadState` structure members: 2511 2512 * ``frame``: removed, use :c:func:`PyThreadState_GetFrame` (function added 2513 to Python 3.9 by :issue:`40429`). 2514 Warning: the function returns a :term:`strong reference`, need to call 2515 :c:func:`Py_XDECREF`. 2516 * ``tracing``: changed, use :c:func:`PyThreadState_EnterTracing` 2517 and :c:func:`PyThreadState_LeaveTracing` 2518 (functions added to Python 3.11 by :issue:`43760`). 2519 * ``recursion_depth``: removed, 2520 use ``(tstate->recursion_limit - tstate->recursion_remaining)`` instead. 2521 * ``stackcheck_counter``: removed. 2522 2523 Code defining ``PyThreadState_GetFrame()`` on Python 3.8 and older:: 2524 2525 #if PY_VERSION_HEX < 0x030900B1 2526 static inline PyFrameObject* PyThreadState_GetFrame(PyThreadState *tstate) 2527 { 2528 Py_XINCREF(tstate->frame); 2529 return tstate->frame; 2530 } 2531 #endif 2532 2533 Code defining ``PyThreadState_EnterTracing()`` and 2534 ``PyThreadState_LeaveTracing()`` on Python 3.10 and older:: 2535 2536 #if PY_VERSION_HEX < 0x030B00A2 2537 static inline void PyThreadState_EnterTracing(PyThreadState *tstate) 2538 { 2539 tstate->tracing++; 2540 #if PY_VERSION_HEX >= 0x030A00A1 2541 tstate->cframe->use_tracing = 0; 2542 #else 2543 tstate->use_tracing = 0; 2544 #endif 2545 } 2546 2547 static inline void PyThreadState_LeaveTracing(PyThreadState *tstate) 2548 { 2549 int use_tracing = (tstate->c_tracefunc != NULL || tstate->c_profilefunc != NULL); 2550 tstate->tracing--; 2551 #if PY_VERSION_HEX >= 0x030A00A1 2552 tstate->cframe->use_tracing = use_tracing; 2553 #else 2554 tstate->use_tracing = use_tracing; 2555 #endif 2556 } 2557 #endif 2558 2559 Or use `the pythoncapi-compat project 2560 <https://github.com/python/pythoncapi-compat>`__ to get these functions 2561 on old Python functions. 2562 2563* Distributors are encouraged to build Python with the optimized Blake2 2564 library `libb2`_. 2565 2566* The :c:member:`PyConfig.module_search_paths_set` field must now be set to 1 for 2567 initialization to use :c:member:`PyConfig.module_search_paths` to initialize 2568 :data:`sys.path`. Otherwise, initialization will recalculate the path and replace 2569 any values added to ``module_search_paths``. 2570 2571* :c:func:`PyConfig_Read` no longer calculates the initial search path, and will not 2572 fill any values into :c:member:`PyConfig.module_search_paths`. To calculate default 2573 paths and then modify them, finish initialization and use :c:func:`PySys_GetObject` 2574 to retrieve :data:`sys.path` as a Python list object and modify it directly. 2575 2576 2577.. _whatsnew311-c-api-deprecated: 2578 2579Deprecated 2580---------- 2581 2582* Deprecate the following functions to configure the Python initialization: 2583 2584 * :c:func:`!PySys_AddWarnOptionUnicode` 2585 * :c:func:`!PySys_AddWarnOption` 2586 * :c:func:`!PySys_AddXOption` 2587 * :c:func:`!PySys_HasWarnOptions` 2588 * :c:func:`!PySys_SetArgvEx` 2589 * :c:func:`!PySys_SetArgv` 2590 * :c:func:`!PySys_SetPath` 2591 * :c:func:`!Py_SetPath` 2592 * :c:func:`!Py_SetProgramName` 2593 * :c:func:`!Py_SetPythonHome` 2594 * :c:func:`!Py_SetStandardStreamEncoding` 2595 * :c:func:`!_Py_SetProgramFullPath` 2596 2597 Use the new :c:type:`PyConfig` API of the :ref:`Python Initialization Configuration 2598 <init-config>` instead (:pep:`587`). 2599 (Contributed by Victor Stinner in :gh:`88279`.) 2600 2601* Deprecate the ``ob_shash`` member of the :c:type:`PyBytesObject`. Use :c:func:`PyObject_Hash` instead. 2602 (Contributed by Inada Naoki in :issue:`46864`.) 2603 2604 2605.. _whatsnew311-c-api-pending-removal: 2606 2607Pending Removal in Python 3.12 2608------------------------------ 2609 2610The following C APIs have been deprecated in earlier Python releases, 2611and will be removed in Python 3.12. 2612 2613* :c:func:`!PyUnicode_AS_DATA` 2614* :c:func:`!PyUnicode_AS_UNICODE` 2615* :c:func:`!PyUnicode_AsUnicodeAndSize` 2616* :c:func:`!PyUnicode_AsUnicode` 2617* :c:func:`!PyUnicode_FromUnicode` 2618* :c:func:`!PyUnicode_GET_DATA_SIZE` 2619* :c:func:`!PyUnicode_GET_SIZE` 2620* :c:func:`!PyUnicode_GetSize` 2621* :c:func:`!PyUnicode_IS_COMPACT` 2622* :c:func:`!PyUnicode_IS_READY` 2623* :c:func:`PyUnicode_READY` 2624* :c:func:`!PyUnicode_WSTR_LENGTH` 2625* :c:func:`!_PyUnicode_AsUnicode` 2626* :c:macro:`!PyUnicode_WCHAR_KIND` 2627* :c:type:`PyUnicodeObject` 2628* :c:func:`!PyUnicode_InternImmortal` 2629 2630 2631.. _whatsnew311-c-api-removed: 2632 2633Removed 2634------- 2635 2636* :c:func:`!PyFrame_BlockSetup` and :c:func:`!PyFrame_BlockPop` have been 2637 removed. 2638 (Contributed by Mark Shannon in :issue:`40222`.) 2639 2640* Remove the following math macros using the ``errno`` variable: 2641 2642 * ``Py_ADJUST_ERANGE1()`` 2643 * ``Py_ADJUST_ERANGE2()`` 2644 * ``Py_OVERFLOWED()`` 2645 * ``Py_SET_ERANGE_IF_OVERFLOW()`` 2646 * ``Py_SET_ERRNO_ON_MATH_ERROR()`` 2647 2648 (Contributed by Victor Stinner in :issue:`45412`.) 2649 2650* Remove ``Py_UNICODE_COPY()`` and ``Py_UNICODE_FILL()`` macros, deprecated 2651 since Python 3.3. Use ``PyUnicode_CopyCharacters()`` or ``memcpy()`` 2652 (``wchar_t*`` string), and ``PyUnicode_Fill()`` functions instead. 2653 (Contributed by Victor Stinner in :issue:`41123`.) 2654 2655* Remove the ``pystrhex.h`` header file. It only contains private functions. 2656 C extensions should only include the main ``<Python.h>`` header file. 2657 (Contributed by Victor Stinner in :issue:`45434`.) 2658 2659* Remove the ``Py_FORCE_DOUBLE()`` macro. It was used by the 2660 ``Py_IS_INFINITY()`` macro. 2661 (Contributed by Victor Stinner in :issue:`45440`.) 2662 2663* The following items are no longer available when :c:macro:`Py_LIMITED_API` 2664 is defined: 2665 2666 * :c:func:`PyMarshal_WriteLongToFile` 2667 * :c:func:`PyMarshal_WriteObjectToFile` 2668 * :c:func:`PyMarshal_ReadObjectFromString` 2669 * :c:func:`PyMarshal_WriteObjectToString` 2670 * the ``Py_MARSHAL_VERSION`` macro 2671 2672 These are not part of the :ref:`limited API <limited-api-list>`. 2673 2674 (Contributed by Victor Stinner in :issue:`45474`.) 2675 2676* Exclude :c:func:`PyWeakref_GET_OBJECT` from the limited C API. It never 2677 worked since the :c:type:`!PyWeakReference` structure is opaque in the 2678 limited C API. 2679 (Contributed by Victor Stinner in :issue:`35134`.) 2680 2681* Remove the ``PyHeapType_GET_MEMBERS()`` macro. It was exposed in the 2682 public C API by mistake, it must only be used by Python internally. 2683 Use the ``PyTypeObject.tp_members`` member instead. 2684 (Contributed by Victor Stinner in :issue:`40170`.) 2685 2686* Remove the ``HAVE_PY_SET_53BIT_PRECISION`` macro (moved to the internal C 2687 API). 2688 (Contributed by Victor Stinner in :issue:`45412`.) 2689 2690.. _whatsnew311-pep624: 2691 2692* Remove the :c:type:`Py_UNICODE` encoder APIs, 2693 as they have been deprecated since Python 3.3, 2694 are little used 2695 and are inefficient relative to the recommended alternatives. 2696 2697 The removed functions are: 2698 2699 * :func:`!PyUnicode_Encode` 2700 * :func:`!PyUnicode_EncodeASCII` 2701 * :func:`!PyUnicode_EncodeLatin1` 2702 * :func:`!PyUnicode_EncodeUTF7` 2703 * :func:`!PyUnicode_EncodeUTF8` 2704 * :func:`!PyUnicode_EncodeUTF16` 2705 * :func:`!PyUnicode_EncodeUTF32` 2706 * :func:`!PyUnicode_EncodeUnicodeEscape` 2707 * :func:`!PyUnicode_EncodeRawUnicodeEscape` 2708 * :func:`!PyUnicode_EncodeCharmap` 2709 * :func:`!PyUnicode_TranslateCharmap` 2710 * :func:`!PyUnicode_EncodeDecimal` 2711 * :func:`!PyUnicode_TransformDecimalToASCII` 2712 2713 See :pep:`624` for details and 2714 :pep:`migration guidance <624#alternative-apis>`. 2715 (Contributed by Inada Naoki in :issue:`44029`.) 2716 2717 2718Notable changes in 3.11.4 2719========================= 2720 2721tarfile 2722------- 2723 2724* The extraction methods in :mod:`tarfile`, and :func:`shutil.unpack_archive`, 2725 have a new a *filter* argument that allows limiting tar features than may be 2726 surprising or dangerous, such as creating files outside the destination 2727 directory. 2728 See :ref:`tarfile-extraction-filter` for details. 2729 In Python 3.12, use without the *filter* argument will show a 2730 :exc:`DeprecationWarning`. 2731 In Python 3.14, the default will switch to ``'data'``. 2732 (Contributed by Petr Viktorin in :pep:`706`.) 2733 2734 2735Notable changes in 3.11.5 2736========================= 2737 2738OpenSSL 2739------- 2740 2741* Windows builds and macOS installers from python.org now use OpenSSL 3.0. 2742 2743 2744.. _libb2: https://www.blake2.net/ 2745