1**************************** 2 What's New In Python 3.4 3**************************** 4 5:Author: R. David Murray <rdmurray@bitdance.com> (Editor) 6 7.. Rules for maintenance: 8 9 * Anyone can add text to this document, but the maintainer reserves the 10 right to rewrite any additions. In particular, for obscure or esoteric 11 features, the maintainer may reduce any addition to a simple reference to 12 the new documentation rather than explaining the feature inline. 13 14 * While the maintainer will periodically go through Misc/NEWS 15 and add changes, it's best not to rely on this. We know from experience 16 that any changes that aren't in the What's New documentation around the 17 time of the original release will remain largely unknown to the community 18 for years, even if they're added later. We also know from experience that 19 other priorities can arise, and the maintainer will run out of time to do 20 updates -- in such cases, end users will be much better served by partial 21 notifications that at least give a hint about new features to 22 investigate. 23 24 * This is not a complete list of every single change; completeness 25 is the purpose of Misc/NEWS. The What's New should focus on changes that 26 are visible to Python *users* and that *require* a feature release (i.e. 27 most bug fixes should only be recorded in Misc/NEWS) 28 29 * PEPs should not be marked Final until they have an entry in What's New. 30 A placeholder entry that is just a section header and a link to the PEP 31 (e.g ":pep:`397` has been implemented") is acceptable. If a PEP has been 32 implemented and noted in What's New, don't forget to mark it as Final! 33 34 * If you want to draw your new text to the attention of the 35 maintainer, add 'XXX' to the beginning of the paragraph or 36 section. 37 38 * It's OK to add just a very brief note about a change. For 39 example: "The :ref:`~socket.transmogrify()` function was added to the 40 :mod:`socket` module." The maintainer will research the change and 41 write the necessary text (if appropriate). The advantage of doing this 42 is that even if no more descriptive text is ever added, readers will at 43 least have a notification that the new feature exists and a link to the 44 relevant documentation. 45 46 * You can comment out your additions if you like, but it's not 47 necessary (especially when a final release is some months away). 48 49 * Credit the author of a patch or bugfix. Just the name is 50 sufficient; the e-mail address isn't necessary. 51 52 * It's helpful to add the bug/patch number as a comment: 53 54 The :ref:`~socket.transmogrify()` function was added to the 55 :mod:`socket` module. (Contributed by P.Y. Developer in :issue:`12345`.) 56 57 This saves the maintainer the effort of going through the Mercurial log 58 when researching a change. 59 60 * Cross referencing tip: :ref:`mod.attr` will display as ``mod.attr``, 61 while :ref:`~mod.attr` will display as ``attr``. 62 63This article explains the new features in Python 3.4, compared to 3.3. 64Python 3.4 was released on March 16, 2014. For full details, see the 65`changelog <https://docs.python.org/3.4/whatsnew/changelog.html>`_. 66 67 68.. seealso:: 69 70 :pep:`429` -- Python 3.4 Release Schedule 71 72 73 74Summary -- Release Highlights 75============================= 76 77.. This section singles out the most important changes in Python 3.4. 78 Brevity is key. 79 80New syntax features: 81 82* No new syntax features were added in Python 3.4. 83 84Other new features: 85 86* :ref:`pip should always be available <whatsnew-pep-453>` (:pep:`453`). 87* :ref:`Newly created file descriptors are non-inheritable <whatsnew-pep-446>` 88 (:pep:`446`). 89* command line option for :ref:`isolated mode <whatsnew-isolated-mode>` 90 (:issue:`16499`). 91* :ref:`improvements in the handling of codecs <codec-handling-improvements>` 92 that are not text encodings (multiple issues). 93* :ref:`A ModuleSpec Type <whatsnew-pep-451>` for the Import System 94 (:pep:`451`). (Affects importer authors.) 95* The :mod:`marshal` format has been made :ref:`more compact and efficient 96 <whatsnew-marshal-3>` (:issue:`16475`). 97 98New library modules: 99 100* :mod:`asyncio`: :ref:`New provisional API for asynchronous IO 101 <whatsnew-asyncio>` (:pep:`3156`). 102* :mod:`ensurepip`: :ref:`Bootstrapping the pip installer <whatsnew-ensurepip>` 103 (:pep:`453`). 104* :mod:`enum`: :ref:`Support for enumeration types <whatsnew-enum>` 105 (:pep:`435`). 106* :mod:`pathlib`: :ref:`Object-oriented filesystem paths <whatsnew-pathlib>` 107 (:pep:`428`). 108* :mod:`selectors`: :ref:`High-level and efficient I/O multiplexing 109 <whatsnew-selectors>`, built upon the :mod:`select` module primitives (part 110 of :pep:`3156`). 111* :mod:`statistics`: A basic :ref:`numerically stable statistics library 112 <whatsnew-statistics>` (:pep:`450`). 113* :mod:`tracemalloc`: :ref:`Trace Python memory allocations 114 <whatsnew-tracemalloc>` (:pep:`454`). 115 116Significantly improved library modules: 117 118* :ref:`Single-dispatch generic functions <whatsnew-singledispatch>` in 119 :mod:`functools` (:pep:`443`). 120* New :mod:`pickle` :ref:`protocol 4 <whatsnew-protocol-4>` (:pep:`3154`). 121* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork 122 on Unix <whatsnew-multiprocessing-no-fork>` (:issue:`8713`). 123* :mod:`email` has a new submodule, :mod:`~email.contentmanager`, and 124 a new :mod:`~email.message.Message` subclass 125 (:class:`~email.contentmanager.EmailMessage`) that :ref:`simplify MIME 126 handling <whatsnew_email_contentmanager>` (:issue:`18891`). 127* The :mod:`inspect` and :mod:`pydoc` modules are now capable of 128 correct introspection of a much wider variety of callable objects, 129 which improves the output of the Python :func:`help` system. 130* The :mod:`ipaddress` module API has been declared stable 131 132Security improvements: 133 134* :ref:`Secure and interchangeable hash algorithm <whatsnew-pep-456>` 135 (:pep:`456`). 136* :ref:`Make newly created file descriptors non-inheritable <whatsnew-pep-446>` 137 (:pep:`446`) to avoid leaking file descriptors to child processes. 138* New command line option for :ref:`isolated mode <whatsnew-isolated-mode>`, 139 (:issue:`16499`). 140* :mod:`multiprocessing` now has :ref:`an option to avoid using os.fork 141 on Unix <whatsnew-multiprocessing-no-fork>`. *spawn* and *forkserver* are 142 more secure because they avoid sharing data with child processes. 143* :mod:`multiprocessing` child processes on Windows no longer inherit 144 all of the parent's inheritable handles, only the necessary ones. 145* A new :func:`hashlib.pbkdf2_hmac` function provides 146 the `PKCS#5 password-based key derivation function 2 147 <https://en.wikipedia.org/wiki/PBKDF2>`_. 148* :ref:`TLSv1.1 and TLSv1.2 support <whatsnew-tls-11-12>` for :mod:`ssl`. 149* :ref:`Retrieving certificates from the Windows system cert store support 150 <whatsnew34-win-cert-store>` for :mod:`ssl`. 151* :ref:`Server-side SNI (Server Name Indication) support 152 <whatsnew34-sni>` for :mod:`ssl`. 153* The :class:`ssl.SSLContext` class has a :ref:`lot of improvements 154 <whatsnew34-sslcontext>`. 155* All modules in the standard library that support SSL now support server 156 certificate verification, including hostname matching 157 (:func:`ssl.match_hostname`) and CRLs (Certificate Revocation lists, see 158 :func:`ssl.SSLContext.load_verify_locations`). 159 160CPython implementation improvements: 161 162* :ref:`Safe object finalization <whatsnew-pep-442>` (:pep:`442`). 163* Leveraging :pep:`442`, in most cases :ref:`module globals are no longer set 164 to None during finalization <whatsnew-pep-442>` (:issue:`18214`). 165* :ref:`Configurable memory allocators <whatsnew-pep-445>` (:pep:`445`). 166* :ref:`Argument Clinic <whatsnew-pep-436>` (:pep:`436`). 167 168Please read on for a comprehensive list of user-facing changes, including many 169other smaller improvements, CPython optimizations, deprecations, and potential 170porting issues. 171 172 173 174New Features 175============ 176 177.. _whatsnew-pep-453: 178 179PEP 453: Explicit Bootstrapping of PIP in Python Installations 180-------------------------------------------------------------- 181 182Bootstrapping pip By Default 183~~~~~~~~~~~~~~~~~~~~~~~~~~~~ 184 185The new :mod:`ensurepip` module (defined in :pep:`453`) provides a standard 186cross-platform mechanism to bootstrap the pip installer into Python 187installations and virtual environments. The version of ``pip`` included 188with Python 3.4.0 is ``pip`` 1.5.4, and future 3.4.x maintenance releases 189will update the bundled version to the latest version of ``pip`` that is 190available at the time of creating the release candidate. 191 192By default, the commands ``pipX`` and ``pipX.Y`` will be installed on all 193platforms (where X.Y stands for the version of the Python installation), 194along with the ``pip`` Python package and its dependencies. On Windows and 195in virtual environments on all platforms, the unversioned ``pip`` command 196will also be installed. On other platforms, the system wide unversioned 197``pip`` command typically refers to the separately installed Python 2 198version. 199 200The ``pyvenv`` command line utility and the :mod:`venv` 201module make use of the :mod:`ensurepip` module to make ``pip`` readily 202available in virtual environments. When using the command line utility, 203``pip`` is installed by default, while when using the :mod:`venv` module 204:ref:`venv-api` installation of ``pip`` must be requested explicitly. 205 206For CPython :ref:`source builds on POSIX systems <building-python-on-unix>`, 207the ``make install`` and ``make altinstall`` commands bootstrap ``pip`` by 208default. This behaviour can be controlled through configure options, and 209overridden through Makefile options. 210 211On Windows and Mac OS X, the CPython installers now default to installing 212``pip`` along with CPython itself (users may opt out of installing it 213during the installation process). Window users will need to opt in to the 214automatic ``PATH`` modifications to have ``pip`` available from the command 215line by default, otherwise it can still be accessed through the Python 216launcher for Windows as ``py -m pip``. 217 218As `discussed in the PEP`__, platform packagers may choose not to install 219these commands by default, as long as, when invoked, they provide clear and 220simple directions on how to install them on that platform (usually using 221the system package manager). 222 223__ https://www.python.org/dev/peps/pep-0453/#recommendations-for-downstream-distributors 224 225.. note:: 226 227 To avoid conflicts between parallel Python 2 and Python 3 installations, 228 only the versioned ``pip3`` and ``pip3.4`` commands are bootstrapped by 229 default when ``ensurepip`` is invoked directly - the ``--default-pip`` 230 option is needed to also request the unversioned ``pip`` command. 231 ``pyvenv`` and the Windows installer ensure that the unqualified ``pip`` 232 command is made available in those environments, and ``pip`` can always be 233 invoked via the ``-m`` switch rather than directly to avoid ambiguity on 234 systems with multiple Python installations. 235 236 237Documentation Changes 238~~~~~~~~~~~~~~~~~~~~~ 239 240As part of this change, the :ref:`installing-index` and 241:ref:`distributing-index` sections of the documentation have been 242completely redesigned as short getting started and FAQ documents. Most 243packaging documentation has now been moved out to the Python Packaging 244Authority maintained `Python Packaging User Guide 245<https://packaging.python.org>`__ and the documentation of the individual 246projects. 247 248However, as this migration is currently still incomplete, the legacy 249versions of those guides remaining available as :ref:`install-index` 250and :ref:`distutils-index`. 251 252.. seealso:: 253 254 :pep:`453` -- Explicit bootstrapping of pip in Python installations 255 PEP written by Donald Stufft and Nick Coghlan, implemented by 256 Donald Stufft, Nick Coghlan, Martin von Löwis and Ned Deily. 257 258 259.. _whatsnew-pep-446: 260 261PEP 446: Newly Created File Descriptors Are Non-Inheritable 262----------------------------------------------------------- 263 264:pep:`446` makes newly created file descriptors :ref:`non-inheritable 265<fd_inheritance>`. In general, this is the behavior an application will 266want: when launching a new process, having currently open files also 267open in the new process can lead to all sorts of hard to find bugs, 268and potentially to security issues. 269 270However, there are occasions when inheritance is desired. To support 271these cases, the following new functions and methods are available: 272 273* :func:`os.get_inheritable`, :func:`os.set_inheritable` 274* :func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable` 275* :meth:`socket.socket.get_inheritable`, :meth:`socket.socket.set_inheritable` 276 277.. seealso:: 278 279 :pep:`446` -- Make newly created file descriptors non-inheritable 280 PEP written and implemented by Victor Stinner. 281 282 283.. _codec-handling-improvements: 284 285Improvements to Codec Handling 286------------------------------ 287 288Since it was first introduced, the :mod:`codecs` module has always been 289intended to operate as a type-neutral dynamic encoding and decoding 290system. However, its close coupling with the Python text model, especially 291the type restricted convenience methods on the builtin :class:`str`, 292:class:`bytes` and :class:`bytearray` types, has historically obscured that 293fact. 294 295As a key step in clarifying the situation, the :meth:`codecs.encode` and 296:meth:`codecs.decode` convenience functions are now properly documented in 297Python 2.7, 3.3 and 3.4. These functions have existed in the :mod:`codecs` 298module (and have been covered by the regression test suite) since Python 2.4, 299but were previously only discoverable through runtime introspection. 300 301Unlike the convenience methods on :class:`str`, :class:`bytes` and 302:class:`bytearray`, the :mod:`codecs` convenience functions support arbitrary 303codecs in both Python 2 and Python 3, rather than being limited to Unicode text 304encodings (in Python 3) or ``basestring`` <-> ``basestring`` conversions (in 305Python 2). 306 307In Python 3.4, the interpreter is able to identify the known non-text 308encodings provided in the standard library and direct users towards these 309general purpose convenience functions when appropriate:: 310 311 >>> b"abcdef".decode("hex") 312 Traceback (most recent call last): 313 File "<stdin>", line 1, in <module> 314 LookupError: 'hex' is not a text encoding; use codecs.decode() to handle arbitrary codecs 315 316 >>> "hello".encode("rot13") 317 Traceback (most recent call last): 318 File "<stdin>", line 1, in <module> 319 LookupError: 'rot13' is not a text encoding; use codecs.encode() to handle arbitrary codecs 320 321 >>> open("foo.txt", encoding="hex") 322 Traceback (most recent call last): 323 File "<stdin>", line 1, in <module> 324 LookupError: 'hex' is not a text encoding; use codecs.open() to handle arbitrary codecs 325 326In a related change, whenever it is feasible without breaking backwards 327compatibility, exceptions raised during encoding and decoding operations 328are wrapped in a chained exception of the same type that mentions the 329name of the codec responsible for producing the error:: 330 331 >>> import codecs 332 333 >>> codecs.decode(b"abcdefgh", "hex") 334 Traceback (most recent call last): 335 File "/usr/lib/python3.4/encodings/hex_codec.py", line 20, in hex_decode 336 return (binascii.a2b_hex(input), len(input)) 337 binascii.Error: Non-hexadecimal digit found 338 339 The above exception was the direct cause of the following exception: 340 341 Traceback (most recent call last): 342 File "<stdin>", line 1, in <module> 343 binascii.Error: decoding with 'hex' codec failed (Error: Non-hexadecimal digit found) 344 345 >>> codecs.encode("hello", "bz2") 346 Traceback (most recent call last): 347 File "/usr/lib/python3.4/encodings/bz2_codec.py", line 17, in bz2_encode 348 return (bz2.compress(input), len(input)) 349 File "/usr/lib/python3.4/bz2.py", line 498, in compress 350 return comp.compress(data) + comp.flush() 351 TypeError: 'str' does not support the buffer interface 352 353 The above exception was the direct cause of the following exception: 354 355 Traceback (most recent call last): 356 File "<stdin>", line 1, in <module> 357 TypeError: encoding with 'bz2' codec failed (TypeError: 'str' does not support the buffer interface) 358 359Finally, as the examples above show, these improvements have permitted 360the restoration of the convenience aliases for the non-Unicode codecs that 361were themselves restored in Python 3.2. This means that encoding binary data 362to and from its hexadecimal representation (for example) can now be written 363as:: 364 365 >>> from codecs import encode, decode 366 >>> encode(b"hello", "hex") 367 b'68656c6c6f' 368 >>> decode(b"68656c6c6f", "hex") 369 b'hello' 370 371The binary and text transforms provided in the standard library are detailed 372in :ref:`binary-transforms` and :ref:`text-transforms`. 373 374(Contributed by Nick Coghlan in :issue:`7475`, :issue:`17827`, 375:issue:`17828` and :issue:`19619`.) 376 377 378.. _whatsnew-pep-451: 379 380PEP 451: A ModuleSpec Type for the Import System 381------------------------------------------------ 382 383:pep:`451` provides an encapsulation of the information about a module that the 384import machinery will use to load it (that is, a module specification). This 385helps simplify both the import implementation and several import-related APIs. 386The change is also a stepping stone for `several future import-related 387improvements`__. 388 389__ https://mail.python.org/pipermail/python-dev/2013-November/130111.html 390 391The public-facing changes from the PEP are entirely backward-compatible. 392Furthermore, they should be transparent to everyone but importer authors. Key 393finder and loader methods have been deprecated, but they will continue working. 394New importers should use the new methods described in the PEP. Existing 395importers should be updated to implement the new methods. See the 396:ref:`deprecated-3.4` section for a list of methods that should be replaced and 397their replacements. 398 399 400Other Language Changes 401---------------------- 402 403Some smaller changes made to the core Python language are: 404 405* Unicode database updated to UCD version 6.3. 406 407* :func:`min` and :func:`max` now accept a *default* keyword-only argument that 408 can be used to specify the value they return if the iterable they are 409 evaluating has no elements. (Contributed by Julian Berman in 410 :issue:`18111`.) 411 412* Module objects are now :mod:`weakref`'able. 413 414* Module ``__file__`` attributes (and related values) should now always 415 contain absolute paths by default, with the sole exception of 416 ``__main__.__file__`` when a script has been executed directly using 417 a relative path. (Contributed by Brett Cannon in :issue:`18416`.) 418 419* All the UTF-\* codecs (except UTF-7) now reject surrogates during both 420 encoding and decoding unless the ``surrogatepass`` error handler is used, 421 with the exception of the UTF-16 decoder (which accepts valid surrogate pairs) 422 and the UTF-16 encoder (which produces them while encoding non-BMP characters). 423 (Contributed by Victor Stinner, Kang-Hao (Kenny) Lu and Serhiy Storchaka in 424 :issue:`12892`.) 425 426* New German EBCDIC :ref:`codec <standard-encodings>` ``cp273``. (Contributed 427 by Michael Bierenfeld and Andrew Kuchling in :issue:`1097797`.) 428 429* New Ukrainian :ref:`codec <standard-encodings>` ``cp1125``. (Contributed by 430 Serhiy Storchaka in :issue:`19668`.) 431 432* :class:`bytes`.join() and :class:`bytearray`.join() now accept arbitrary 433 buffer objects as arguments. (Contributed by Antoine Pitrou in 434 :issue:`15958`.) 435 436* The :class:`int` constructor now accepts any object that has an ``__index__`` 437 method for its *base* argument. (Contributed by Mark Dickinson in 438 :issue:`16772`.) 439 440* Frame objects now have a :func:`~frame.clear` method that clears all 441 references to local variables from the frame. (Contributed by Antoine Pitrou 442 in :issue:`17934`.) 443 444* :class:`memoryview` is now registered as a :class:`Sequence <collections.abc>`, 445 and supports the :func:`reversed` builtin. (Contributed by Nick Coghlan 446 and Claudiu Popa in :issue:`18690` and :issue:`19078`.) 447 448* Signatures reported by :func:`help` have been modified and improved in 449 several cases as a result of the introduction of Argument Clinic and other 450 changes to the :mod:`inspect` and :mod:`pydoc` modules. 451 452* :meth:`~object.__length_hint__` is now part of the formal language 453 specification (see :pep:`424`). (Contributed by Armin Ronacher in 454 :issue:`16148`.) 455 456 457New Modules 458=========== 459 460 461.. _whatsnew-asyncio: 462 463asyncio 464------- 465 466The new :mod:`asyncio` module (defined in :pep:`3156`) provides a standard 467pluggable event loop model for Python, providing solid asynchronous IO 468support in the standard library, and making it easier for other event loop 469implementations to interoperate with the standard library and each other. 470 471For Python 3.4, this module is considered a :term:`provisional API`. 472 473.. seealso:: 474 475 :pep:`3156` -- Asynchronous IO Support Rebooted: the "asyncio" Module 476 PEP written and implementation led by Guido van Rossum. 477 478 479.. _whatsnew-ensurepip: 480 481ensurepip 482--------- 483 484The new :mod:`ensurepip` module is the primary infrastructure for the 485:pep:`453` implementation. In the normal course of events end users will not 486need to interact with this module, but it can be used to manually bootstrap 487``pip`` if the automated bootstrapping into an installation or virtual 488environment was declined. 489 490:mod:`ensurepip` includes a bundled copy of ``pip``, up-to-date as of the first 491release candidate of the release of CPython with which it ships (this applies 492to both maintenance releases and feature releases). ``ensurepip`` does not 493access the internet. If the installation has internet access, after 494``ensurepip`` is run the bundled ``pip`` can be used to upgrade ``pip`` to a 495more recent release than the bundled one. (Note that such an upgraded version 496of ``pip`` is considered to be a separately installed package and will not be 497removed if Python is uninstalled.) 498 499The module is named *ensure*\ pip because if called when ``pip`` is already 500installed, it does nothing. It also has an ``--upgrade`` option that will 501cause it to install the bundled copy of ``pip`` if the existing installed 502version of ``pip`` is older than the bundled copy. 503 504 505.. _whatsnew-enum: 506 507enum 508---- 509 510The new :mod:`enum` module (defined in :pep:`435`) provides a standard 511implementation of enumeration types, allowing other modules (such as 512:mod:`socket`) to provide more informative error messages and better 513debugging support by replacing opaque integer constants with backwards 514compatible enumeration values. 515 516.. seealso:: 517 518 :pep:`435` -- Adding an Enum type to the Python standard library 519 PEP written by Barry Warsaw, Eli Bendersky and Ethan Furman, 520 implemented by Ethan Furman. 521 522 523.. _whatsnew-pathlib: 524 525pathlib 526------- 527 528The new :mod:`pathlib` module offers classes representing filesystem paths 529with semantics appropriate for different operating systems. Path classes are 530divided between *pure paths*, which provide purely computational operations 531without I/O, and *concrete paths*, which inherit from pure paths but also 532provide I/O operations. 533 534For Python 3.4, this module is considered a :term:`provisional API`. 535 536.. seealso:: 537 538 :pep:`428` -- The pathlib module -- object-oriented filesystem paths 539 PEP written and implemented by Antoine Pitrou. 540 541 542.. _whatsnew-selectors: 543 544selectors 545--------- 546 547The new :mod:`selectors` module (created as part of implementing :pep:`3156`) 548allows high-level and efficient I/O multiplexing, built upon the 549:mod:`select` module primitives. 550 551 552.. _whatsnew-statistics: 553 554statistics 555---------- 556 557The new :mod:`statistics` module (defined in :pep:`450`) offers some core 558statistics functionality directly in the standard library. This module 559supports calculation of the mean, median, mode, variance and standard 560deviation of a data series. 561 562.. seealso:: 563 564 :pep:`450` -- Adding A Statistics Module To The Standard Library 565 PEP written and implemented by Steven D'Aprano 566 567.. _whatsnew-tracemalloc: 568 569 570tracemalloc 571----------- 572 573The new :mod:`tracemalloc` module (defined in :pep:`454`) is a debug tool to 574trace memory blocks allocated by Python. It provides the following information: 575 576* Trace where an object was allocated 577* Statistics on allocated memory blocks per filename and per line number: 578 total size, number and average size of allocated memory blocks 579* Compute the differences between two snapshots to detect memory leaks 580 581.. seealso:: 582 583 :pep:`454` -- Add a new tracemalloc module to trace Python memory allocations 584 PEP written and implemented by Victor Stinner 585 586 587 588Improved Modules 589================ 590 591 592abc 593--- 594 595New function :func:`abc.get_cache_token` can be used to know when to invalidate 596caches that are affected by changes in the object graph. (Contributed 597by Łukasz Langa in :issue:`16832`.) 598 599New class :class:`~abc.ABC` has :class:`~abc.ABCMeta` as its meta class. 600Using ``ABC`` as a base class has essentially the same effect as specifying 601``metaclass=abc.ABCMeta``, but is simpler to type and easier to read. 602(Contributed by Bruno Dupuis in :issue:`16049`.) 603 604 605aifc 606---- 607 608The :meth:`~aifc.aifc.getparams` method now returns a namedtuple rather than a 609plain tuple. (Contributed by Claudiu Popa in :issue:`17818`.) 610 611:func:`aifc.open` now supports the context management protocol: when used in a 612:keyword:`with` block, the :meth:`~aifc.aifc.close` method of the returned 613object will be called automatically at the end of the block. (Contributed by 614Serhiy Storchacha in :issue:`16486`.) 615 616The :meth:`~aifc.aifc.writeframesraw` and :meth:`~aifc.aifc.writeframes` 617methods now accept any :term:`bytes-like object`. (Contributed by Serhiy 618Storchaka in :issue:`8311`.) 619 620 621argparse 622-------- 623 624The :class:`~argparse.FileType` class now accepts *encoding* and 625*errors* arguments, which are passed through to :func:`open`. (Contributed 626by Lucas Maystre in :issue:`11175`.) 627 628 629audioop 630------- 631 632:mod:`audioop` now supports 24-bit samples. (Contributed by Serhiy Storchaka 633in :issue:`12866`.) 634 635New :func:`~audioop.byteswap` function converts big-endian samples to 636little-endian and vice versa. (Contributed by Serhiy Storchaka in 637:issue:`19641`.) 638 639All :mod:`audioop` functions now accept any :term:`bytes-like object`. Strings 640are not accepted: they didn't work before, now they raise an error right away. 641(Contributed by Serhiy Storchaka in :issue:`16685`.) 642 643 644base64 645------ 646 647The encoding and decoding functions in :mod:`base64` now accept any 648:term:`bytes-like object` in cases where it previously required a 649:class:`bytes` or :class:`bytearray` instance. (Contributed by Nick Coghlan in 650:issue:`17839`.) 651 652New functions :func:`~base64.a85encode`, :func:`~base64.a85decode`, 653:func:`~base64.b85encode`, and :func:`~base64.b85decode` provide the ability to 654encode and decode binary data from and to ``Ascii85`` and the git/mercurial 655``Base85`` formats, respectively. The ``a85`` functions have options that can 656be used to make them compatible with the variants of the ``Ascii85`` encoding, 657including the Adobe variant. (Contributed by Martin Morrison, the Mercurial 658project, Serhiy Storchaka, and Antoine Pitrou in :issue:`17618`.) 659 660 661collections 662----------- 663 664The :meth:`.ChainMap.new_child` method now accepts an *m* argument specifying 665the child map to add to the chain. This allows an existing mapping and/or a 666custom mapping type to be used for the child. (Contributed by Vinay Sajip in 667:issue:`16613`.) 668 669 670colorsys 671-------- 672 673The number of digits in the coefficients for the RGB --- YIQ conversions have 674been expanded so that they match the FCC NTSC versions. The change in 675results should be less than 1% and may better match results found elsewhere. 676(Contributed by Brian Landers and Serhiy Storchaka in :issue:`14323`.) 677 678 679contextlib 680---------- 681 682The new :class:`contextlib.suppress` context manager helps to clarify the 683intent of code that deliberately suppresses exceptions from a single 684statement. (Contributed by Raymond Hettinger in :issue:`15806` and 685Zero Piraeus in :issue:`19266`.) 686 687The new :func:`contextlib.redirect_stdout` context manager makes it easier 688for utility scripts to handle inflexible APIs that write their output to 689:data:`sys.stdout` and don't provide any options to redirect it. Using the 690context manager, the :data:`sys.stdout` output can be redirected to any 691other stream or, in conjunction with :class:`io.StringIO`, to a string. 692The latter can be especially useful, for example, to capture output 693from a function that was written to implement a command line interface. 694It is recommended only for utility scripts because it affects the 695global state of :data:`sys.stdout`. (Contributed by Raymond Hettinger 696in :issue:`15805`.) 697 698The :mod:`contextlib` documentation has also been updated to include a 699:ref:`discussion <single-use-reusable-and-reentrant-cms>` of the 700differences between single use, reusable and reentrant context managers. 701 702 703dbm 704--- 705 706:func:`dbm.open` objects now support the context management protocol. When 707used in a :keyword:`with` statement, the ``close`` method of the database 708object will be called automatically at the end of the block. (Contributed by 709Claudiu Popa and Nick Coghlan in :issue:`19282`.) 710 711 712dis 713--- 714 715Functions :func:`~dis.show_code`, :func:`~dis.dis`, :func:`~dis.distb`, and 716:func:`~dis.disassemble` now accept a keyword-only *file* argument that 717controls where they write their output. 718 719The :mod:`dis` module is now built around an :class:`~dis.Instruction` class 720that provides object oriented access to the details of each individual bytecode 721operation. 722 723A new method, :func:`~dis.get_instructions`, provides an iterator that emits 724the Instruction stream for a given piece of Python code. Thus it is now 725possible to write a program that inspects and manipulates a bytecode 726object in ways different from those provided by the :mod:`~dis` module 727itself. For example:: 728 729 >>> import dis 730 >>> for instr in dis.get_instructions(lambda x: x + 1): 731 ... print(instr.opname) 732 LOAD_FAST 733 LOAD_CONST 734 BINARY_ADD 735 RETURN_VALUE 736 737The various display tools in the :mod:`dis` module have been rewritten to use 738these new components. 739 740In addition, a new application-friendly class :class:`~dis.Bytecode` provides 741an object-oriented API for inspecting bytecode in both in human-readable form 742and for iterating over instructions. The :class:`~dis.Bytecode` constructor 743takes the same arguments that :func:`~dis.get_instruction` does (plus an 744optional *current_offset*), and the resulting object can be iterated to produce 745:class:`~dis.Instruction` objects. But it also has a :mod:`~dis.Bytecode.dis` 746method, equivalent to calling :mod:`~dis.dis` on the constructor argument, but 747returned as a multi-line string:: 748 749 >>> bytecode = dis.Bytecode(lambda x: x + 1, current_offset=3) 750 >>> for instr in bytecode: 751 ... print('{} ({})'.format(instr.opname, instr.opcode)) 752 LOAD_FAST (124) 753 LOAD_CONST (100) 754 BINARY_ADD (23) 755 RETURN_VALUE (83) 756 >>> bytecode.dis().splitlines() # doctest: +NORMALIZE_WHITESPACE 757 [' 1 0 LOAD_FAST 0 (x)', 758 ' --> 3 LOAD_CONST 1 (1)', 759 ' 6 BINARY_ADD', 760 ' 7 RETURN_VALUE'] 761 762:class:`~dis.Bytecode` also has a class method, 763:meth:`~dis.Bytecode.from_traceback`, that provides the ability to manipulate a 764traceback (that is, ``print(Bytecode.from_traceback(tb).dis())`` is equivalent 765to ``distb(tb)``). 766 767(Contributed by Nick Coghlan, Ryan Kelly and Thomas Kluyver in :issue:`11816` 768and Claudiu Popa in :issue:`17916`.) 769 770New function :func:`~dis.stack_effect` computes the effect on the Python stack 771of a given opcode and argument, information that is not otherwise available. 772(Contributed by Larry Hastings in :issue:`19722`.) 773 774 775doctest 776------- 777 778A new :ref:`option flag <doctest-options>`, :data:`~doctest.FAIL_FAST`, halts 779test running as soon as the first failure is detected. (Contributed by R. 780David Murray and Daniel Urban in :issue:`16522`.) 781 782The :mod:`doctest` command line interface now uses :mod:`argparse`, and has two 783new options, ``-o`` and ``-f``. ``-o`` allows :ref:`doctest options 784<doctest-options>` to be specified on the command line, and ``-f`` is a 785shorthand for ``-o FAIL_FAST`` (to parallel the similar option supported by the 786:mod:`unittest` CLI). (Contributed by R. David Murray in :issue:`11390`.) 787 788:mod:`doctest` will now find doctests in extension module ``__doc__`` strings. 789(Contributed by Zachary Ware in :issue:`3158`.) 790 791 792email 793----- 794 795:meth:`~email.message.Message.as_string` now accepts a *policy* argument to 796override the default policy of the message when generating a string 797representation of it. This means that ``as_string`` can now be used in more 798circumstances, instead of having to create and use a :mod:`~email.generator` in 799order to pass formatting parameters to its ``flatten`` method. (Contributed by 800R. David Murray in :issue:`18600`.) 801 802New method :meth:`~email.message.Message.as_bytes` added to produce a bytes 803representation of the message in a fashion similar to how ``as_string`` 804produces a string representation. It does not accept the *maxheaderlen* 805argument, but does accept the *unixfrom* and *policy* arguments. The 806:class:`~email.message.Message` :meth:`~email.message.Message.__bytes__` method 807calls it, meaning that ``bytes(mymsg)`` will now produce the intuitive 808result: a bytes object containing the fully formatted message. (Contributed 809by R. David Murray in :issue:`18600`.) 810 811The :meth:`.Message.set_param` message now accepts a *replace* keyword argument. 812When specified, the associated header will be updated without changing 813its location in the list of headers. For backward compatibility, the default 814is ``False``. (Contributed by R. David Murray in :issue:`18891`.) 815 816 817.. _whatsnew_email_contentmanager: 818 819A pair of new subclasses of :class:`~email.message.Message` have been added 820(:class:`.EmailMessage` and :class:`.MIMEPart`), along with a new sub-module, 821:mod:`~email.contentmanager` and a new :mod:`~email.policy` attribute 822:attr:`~email.policy.EmailPolicy.content_manager`. All documentation is 823currently in the new module, which is being added as part of email's new 824:term:`provisional API`. These classes provide a number of new methods that 825make extracting content from and inserting content into email messages much 826easier. For details, see the :mod:`~email.contentmanager` documentation and 827the :ref:`email-examples`. These API additions complete the 828bulk of the work that was planned as part of the email6 project. The currently 829provisional API is scheduled to become final in Python 3.5 (possibly with a few 830minor additions in the area of error handling). (Contributed by R. David 831Murray in :issue:`18891`.) 832 833 834filecmp 835------- 836 837A new :func:`~filecmp.clear_cache` function provides the ability to clear the 838:mod:`filecmp` comparison cache, which uses :func:`os.stat` information to 839determine if the file has changed since the last compare. This can be used, 840for example, if the file might have been changed and re-checked in less time 841than the resolution of a particular filesystem's file modification time field. 842(Contributed by Mark Levitt in :issue:`18149`.) 843 844New module attribute :data:`~filecmp.DEFAULT_IGNORES` provides the list of 845directories that are used as the default value for the *ignore* parameter of 846the :func:`~filecmp.dircmp` function. (Contributed by Eli Bendersky in 847:issue:`15442`.) 848 849 850functools 851--------- 852 853The new :func:`~functools.partialmethod` descriptor brings partial argument 854application to descriptors, just as :func:`~functools.partial` provides 855for normal callables. The new descriptor also makes it easier to get 856arbitrary callables (including :func:`~functools.partial` instances) 857to behave like normal instance methods when included in a class definition. 858(Contributed by Alon Horev and Nick Coghlan in :issue:`4331`.) 859 860.. _whatsnew-singledispatch: 861 862The new :func:`~functools.singledispatch` decorator brings support for 863single-dispatch generic functions to the Python standard library. Where 864object oriented programming focuses on grouping multiple operations on a 865common set of data into a class, a generic function focuses on grouping 866multiple implementations of an operation that allows it to work with 867*different* kinds of data. 868 869.. seealso:: 870 871 :pep:`443` -- Single-dispatch generic functions 872 PEP written and implemented by Łukasz Langa. 873 874:func:`~functools.total_ordering` now supports a return value of 875:const:`NotImplemented` from the underlying comparison function. (Contributed 876by Katie Miller in :issue:`10042`.) 877 878A pure-python version of the :func:`~functools.partial` function is now in the 879stdlib; in CPython it is overridden by the C accelerated version, but it is 880available for other implementations to use. (Contributed by Brian Thorne in 881:issue:`12428`.) 882 883 884gc 885-- 886 887New function :func:`~gc.get_stats` returns a list of three per-generation 888dictionaries containing the collections statistics since interpreter startup. 889(Contributed by Antoine Pitrou in :issue:`16351`.) 890 891 892glob 893---- 894 895A new function :func:`~glob.escape` provides a way to escape special characters 896in a filename so that they do not become part of the globbing expansion but are 897instead matched literally. (Contributed by Serhiy Storchaka in :issue:`8402`.) 898 899 900hashlib 901------- 902 903A new :func:`hashlib.pbkdf2_hmac` function provides 904the `PKCS#5 password-based key derivation function 2 905<https://en.wikipedia.org/wiki/PBKDF2>`_. (Contributed by Christian 906Heimes in :issue:`18582`.) 907 908The :attr:`~hashlib.hash.name` attribute of :mod:`hashlib` hash objects is now 909a formally supported interface. It has always existed in CPython's 910:mod:`hashlib` (although it did not return lower case names for all supported 911hashes), but it was not a public interface and so some other Python 912implementations have not previously supported it. (Contributed by Jason R. 913Coombs in :issue:`18532`.) 914 915 916hmac 917---- 918 919:mod:`hmac` now accepts ``bytearray`` as well as ``bytes`` for the *key* 920argument to the :func:`~hmac.new` function, and the *msg* parameter to both the 921:func:`~hmac.new` function and the :meth:`~hmac.HMAC.update` method now 922accepts any type supported by the :mod:`hashlib` module. (Contributed 923by Jonas Borgström in :issue:`18240`.) 924 925The *digestmod* argument to the :func:`hmac.new` function may now be any hash 926digest name recognized by :mod:`hashlib`. In addition, the current behavior in 927which the value of *digestmod* defaults to ``MD5`` is deprecated: in a 928future version of Python there will be no default value. (Contributed by 929Christian Heimes in :issue:`17276`.) 930 931With the addition of :attr:`~hmac.HMAC.block_size` and :attr:`~hmac.HMAC.name` 932attributes (and the formal documentation of the :attr:`~hmac.HMAC.digest_size` 933attribute), the :mod:`hmac` module now conforms fully to the :pep:`247` API. 934(Contributed by Christian Heimes in :issue:`18775`.) 935 936 937html 938---- 939 940New function :func:`~html.unescape` function converts HTML5 character references to 941the corresponding Unicode characters. (Contributed by Ezio Melotti in 942:issue:`2927`.) 943 944:class:`~html.parser.HTMLParser` accepts a new keyword argument 945*convert_charrefs* that, when ``True``, automatically converts all character 946references. For backward-compatibility, its value defaults to ``False``, but 947it will change to ``True`` in a future version of Python, so you are invited to 948set it explicitly and update your code to use this new feature. (Contributed 949by Ezio Melotti in :issue:`13633`.) 950 951The *strict* argument of :class:`~html.parser.HTMLParser` is now deprecated. 952(Contributed by Ezio Melotti in :issue:`15114`.) 953 954 955http 956---- 957 958:meth:`~http.server.BaseHTTPRequestHandler.send_error` now accepts an 959optional additional *explain* parameter which can be used to provide an 960extended error description, overriding the hardcoded default if there is one. 961This extended error description will be formatted using the 962:attr:`~http.server.HTTP.error_message_format` attribute and sent as the body 963of the error response. (Contributed by Karl Cow in :issue:`12921`.) 964 965The :mod:`http.server` :ref:`command line interface <http-server-cli>` now has 966a ``-b/--bind`` option that causes the server to listen on a specific address. 967(Contributed by Malte Swart in :issue:`17764`.) 968 969 970idlelib and IDLE 971---------------- 972 973Since idlelib implements the IDLE shell and editor and is not intended for 974import by other programs, it gets improvements with every release. See 975:file:`Lib/idlelib/NEWS.txt` for a cumulative list of changes since 3.3.0, 976as well as changes made in future 3.4.x releases. This file is also available 977from the IDLE :menuselection:`Help --> About IDLE` dialog. 978 979 980importlib 981--------- 982 983The :class:`~importlib.abc.InspectLoader` ABC defines a new method, 984:meth:`~importlib.abc.InspectLoader.source_to_code` that accepts source 985data and a path and returns a code object. The default implementation 986is equivalent to ``compile(data, path, 'exec', dont_inherit=True)``. 987(Contributed by Eric Snow and Brett Cannon in :issue:`15627`.) 988 989:class:`~importlib.abc.InspectLoader` also now has a default implementation 990for the :meth:`~importlib.abc.InspectLoader.get_code` method. However, 991it will normally be desirable to override the default implementation 992for performance reasons. (Contributed by Brett Cannon in :issue:`18072`.) 993 994The :func:`~importlib.reload` function has been moved from :mod:`imp` to 995:mod:`importlib` as part of the :mod:`imp` module deprecation. (Contributed by 996Berker Peksag in :issue:`18193`.) 997 998:mod:`importlib.util` now has a :data:`~importlib.util.MAGIC_NUMBER` attribute 999providing access to the bytecode version number. This replaces the 1000:func:`~imp.get_magic` function in the deprecated :mod:`imp` module. 1001(Contributed by Brett Cannon in :issue:`18192`.) 1002 1003New :mod:`importlib.util` functions :func:`~importlib.util.cache_from_source` 1004and :func:`~importlib.util.source_from_cache` replace the same-named functions 1005in the deprecated :mod:`imp` module. (Contributed by Brett Cannon in 1006:issue:`18194`.) 1007 1008The :mod:`importlib` bootstrap :class:`.NamespaceLoader` now conforms to 1009the :class:`.InspectLoader` ABC, which means that ``runpy`` and 1010``python -m`` can now be used with namespace packages. (Contributed 1011by Brett Cannon in :issue:`18058`.) 1012 1013:mod:`importlib.util` has a new function :func:`~importlib.util.decode_source` 1014that decodes source from bytes using universal newline processing. This is 1015useful for implementing :meth:`.InspectLoader.get_source` methods. 1016 1017:class:`importlib.machinery.ExtensionFileLoader` now has a 1018:meth:`~importlib.machinery.ExtensionFileLoader.get_filename` method. This was 1019inadvertently omitted in the original implementation. (Contributed by Eric 1020Snow in :issue:`19152`.) 1021 1022 1023inspect 1024------- 1025 1026The :mod:`inspect` module now offers a basic :ref:`command line interface 1027<inspect-module-cli>` to quickly display source code and other 1028information for modules, classes and functions. (Contributed by Claudiu Popa 1029and Nick Coghlan in :issue:`18626`.) 1030 1031:func:`~inspect.unwrap` makes it easy to unravel wrapper function chains 1032created by :func:`functools.wraps` (and any other API that sets the 1033``__wrapped__`` attribute on a wrapper function). (Contributed by 1034Daniel Urban, Aaron Iles and Nick Coghlan in :issue:`13266`.) 1035 1036As part of the implementation of the new :mod:`enum` module, the 1037:mod:`inspect` module now has substantially better support for custom 1038``__dir__`` methods and dynamic class attributes provided through 1039metaclasses. (Contributed by Ethan Furman in :issue:`18929` and 1040:issue:`19030`.) 1041 1042:func:`~inspect.getfullargspec` and :func:`~inspect.getargspec` 1043now use the :func:`~inspect.signature` API. This allows them to 1044support a much broader range of callables, including those with 1045``__signature__`` attributes, those with metadata provided by argument 1046clinic, :func:`functools.partial` objects and more. Note that, unlike 1047:func:`~inspect.signature`, these functions still ignore ``__wrapped__`` 1048attributes, and report the already bound first argument for bound methods, 1049so it is still necessary to update your code to use 1050:func:`~inspect.signature` directly if those features are desired. 1051(Contributed by Yury Selivanov in :issue:`17481`.) 1052 1053:func:`~inspect.signature` now supports duck types of CPython functions, 1054which adds support for functions compiled with Cython. (Contributed 1055by Stefan Behnel and Yury Selivanov in :issue:`17159`.) 1056 1057 1058ipaddress 1059--------- 1060 1061:mod:`ipaddress` was added to the standard library in Python 3.3 as a 1062:term:`provisional API`. With the release of Python 3.4, this qualification 1063has been removed: :mod:`ipaddress` is now considered a stable API, covered 1064by the normal standard library requirements to maintain backwards 1065compatibility. 1066 1067A new :attr:`~ipaddress.IPv4Address.is_global` property is ``True`` if 1068an address is globally routeable. (Contributed by Peter Moody in 1069:issue:`17400`.) 1070 1071 1072logging 1073------- 1074 1075The :class:`~logging.handlers.TimedRotatingFileHandler` has a new *atTime* 1076parameter that can be used to specify the time of day when rollover should 1077happen. (Contributed by Ronald Oussoren in :issue:`9556`.) 1078 1079:class:`~logging.handlers.SocketHandler` and 1080:class:`~logging.handlers.DatagramHandler` now support Unix domain sockets (by 1081setting *port* to ``None``). (Contributed by Vinay Sajip in commit 1082ce46195b56a9.) 1083 1084:func:`~logging.config.fileConfig` now accepts a 1085:class:`configparser.RawConfigParser` subclass instance for the *fname* 1086parameter. This facilitates using a configuration file when logging 1087configuration is just a part of the overall application configuration, or where 1088the application modifies the configuration before passing it to 1089:func:`~logging.config.fileConfig`. (Contributed by Vinay Sajip in 1090:issue:`16110`.) 1091 1092Logging configuration data received from a socket via the 1093:func:`logging.config.listen` function can now be validated before being 1094processed by supplying a verification function as the argument to the new 1095*verify* keyword argument. (Contributed by Vinay Sajip in :issue:`15452`.) 1096 1097 1098.. _whatsnew-marshal-3: 1099 1100marshal 1101------- 1102 1103The default :mod:`marshal` version has been bumped to 3. The code implementing 1104the new version restores the Python2 behavior of recording only one copy of 1105interned strings and preserving the interning on deserialization, and extends 1106this "one copy" ability to any object type (including handling recursive 1107references). This reduces both the size of ``.pyc`` files and the amount of 1108memory a module occupies in memory when it is loaded from a ``.pyc`` (or 1109``.pyo``) file. (Contributed by Kristján Valur Jónsson in :issue:`16475`, 1110with additional speedups by Antoine Pitrou in :issue:`19219`.) 1111 1112 1113mmap 1114---- 1115 1116mmap objects can now be :mod:`weakref`\ ed. (Contributed by Valerie Lambert in 1117:issue:`4885`.) 1118 1119 1120multiprocessing 1121--------------- 1122 1123.. _whatsnew-multiprocessing-no-fork: 1124 1125On Unix two new :ref:`start methods <multiprocessing-start-methods>`, 1126``spawn`` and ``forkserver``, have been added for starting processes using 1127:mod:`multiprocessing`. These make the mixing of processes with threads more 1128robust, and the ``spawn`` method matches the semantics that multiprocessing has 1129always used on Windows. New function 1130:func:`~multiprocessing.get_all_start_methods` reports all start methods 1131available on the platform, :func:`~multiprocessing.get_start_method` reports 1132the current start method, and :func:`~multiprocessing.set_start_method` sets 1133the start method. (Contributed by Richard Oudkerk in :issue:`8713`.) 1134 1135:mod:`multiprocessing` also now has the concept of a ``context``, which 1136determines how child processes are created. New function 1137:func:`~multiprocessing.get_context` returns a context that uses a specified 1138start method. It has the same API as the :mod:`multiprocessing` module itself, 1139so you can use it to create :class:`~multiprocessing.pool.Pool`\ s and other 1140objects that will operate within that context. This allows a framework and an 1141application or different parts of the same application to use multiprocessing 1142without interfering with each other. (Contributed by Richard Oudkerk in 1143:issue:`18999`.) 1144 1145Except when using the old *fork* start method, child processes no longer 1146inherit unneeded handles/file descriptors from their parents (part of 1147:issue:`8713`). 1148 1149:mod:`multiprocessing` now relies on :mod:`runpy` (which implements the 1150``-m`` switch) to initialise ``__main__`` appropriately in child processes 1151when using the ``spawn`` or ``forkserver`` start methods. This resolves some 1152edge cases where combining multiprocessing, the ``-m`` command line switch, 1153and explicit relative imports could cause obscure failures in child 1154processes. (Contributed by Nick Coghlan in :issue:`19946`.) 1155 1156 1157operator 1158-------- 1159 1160New function :func:`~operator.length_hint` provides an implementation of the 1161specification for how the :meth:`~object.__length_hint__` special method should 1162be used, as part of the :pep:`424` formal specification of this language 1163feature. (Contributed by Armin Ronacher in :issue:`16148`.) 1164 1165There is now a pure-python version of the :mod:`operator` module available for 1166reference and for use by alternate implementations of Python. (Contributed by 1167Zachary Ware in :issue:`16694`.) 1168 1169 1170os 1171-- 1172 1173There are new functions to get and set the :ref:`inheritable flag 1174<fd_inheritance>` of a file descriptor (:func:`os.get_inheritable`, 1175:func:`os.set_inheritable`) or a Windows handle 1176(:func:`os.get_handle_inheritable`, :func:`os.set_handle_inheritable`). 1177 1178New function :func:`~os.cpu_count` reports the number of CPUs available on the 1179platform on which Python is running (or ``None`` if the count can't be 1180determined). The :func:`multiprocessing.cpu_count` function is now implemented 1181in terms of this function). (Contributed by Trent Nelson, Yogesh Chaudhari, 1182Victor Stinner, and Charles-François Natali in :issue:`17914`.) 1183 1184:func:`os.path.samestat` is now available on the Windows platform (and the 1185:func:`os.path.samefile` implementation is now shared between Unix and 1186Windows). (Contributed by Brian Curtin in :issue:`11939`.) 1187 1188:func:`os.path.ismount` now recognizes volumes mounted below a drive 1189root on Windows. (Contributed by Tim Golden in :issue:`9035`.) 1190 1191:func:`os.open` supports two new flags on platforms that provide them, 1192:data:`~os.O_PATH` (un-opened file descriptor), and :data:`~os.O_TMPFILE` 1193(unnamed temporary file; as of 3.4.0 release available only on Linux systems 1194with a kernel version of 3.11 or newer that have uapi headers). (Contributed 1195by Christian Heimes in :issue:`18673` and Benjamin Peterson, respectively.) 1196 1197 1198pdb 1199--- 1200 1201:mod:`pdb` has been enhanced to handle generators, :keyword:`yield`, and 1202``yield from`` in a more useful fashion. This is especially helpful when 1203debugging :mod:`asyncio` based programs. (Contributed by Andrew Svetlov and 1204Xavier de Gaye in :issue:`16596`.) 1205 1206The ``print`` command has been removed from :mod:`pdb`, restoring access to the 1207Python :func:`print` function from the pdb command line. Python2's ``pdb`` did 1208not have a ``print`` command; instead, entering ``print`` executed the 1209``print`` statement. In Python3 ``print`` was mistakenly made an alias for the 1210pdb :pdbcmd:`p` command. ``p``, however, prints the ``repr`` of its argument, 1211not the ``str`` like the Python2 ``print`` command did. Worse, the Python3 1212``pdb print`` command shadowed the Python3 ``print`` function, making it 1213inaccessible at the ``pdb`` prompt. (Contributed by Connor Osborn in 1214:issue:`18764`.) 1215 1216 1217.. _whatsnew-protocol-4: 1218 1219pickle 1220------ 1221 1222:mod:`pickle` now supports (but does not use by default) a new pickle protocol, 1223protocol 4. This new protocol addresses a number of issues that were present 1224in previous protocols, such as the serialization of nested classes, very large 1225strings and containers, and classes whose :meth:`__new__` method takes 1226keyword-only arguments. It also provides some efficiency improvements. 1227 1228.. seealso:: 1229 1230 :pep:`3154` -- Pickle protocol 4 1231 PEP written by Antoine Pitrou and implemented by Alexandre Vassalotti. 1232 1233 1234plistlib 1235-------- 1236 1237:mod:`plistlib` now has an API that is similar to the standard pattern for 1238stdlib serialization protocols, with new :func:`~plistlib.load`, 1239:func:`~plistlib.dump`, :func:`~plistlib.loads`, and :func:`~plistlib.dumps` 1240functions. (The older API is now deprecated.) In addition to the already 1241supported XML plist format (:data:`~plistlib.FMT_XML`), it also now supports 1242the binary plist format (:data:`~plistlib.FMT_BINARY`). (Contributed by Ronald 1243Oussoren and others in :issue:`14455`.) 1244 1245 1246poplib 1247------ 1248 1249Two new methods have been added to :mod:`poplib`: :meth:`~poplib.POP3.capa`, 1250which returns the list of capabilities advertised by the POP server, and 1251:meth:`~poplib.POP3.stls`, which switches a clear-text POP3 session into an 1252encrypted POP3 session if the POP server supports it. (Contributed by Lorenzo 1253Catucci in :issue:`4473`.) 1254 1255 1256pprint 1257------ 1258 1259The :mod:`pprint` module's :class:`~pprint.PrettyPrinter` class and its 1260:func:`~pprint.pformat`, and :func:`~pprint.pprint` functions have a new 1261option, *compact*, that controls how the output is formatted. Currently 1262setting *compact* to ``True`` means that sequences will be printed with as many 1263sequence elements as will fit within *width* on each (indented) line. 1264(Contributed by Serhiy Storchaka in :issue:`19132`.) 1265 1266Long strings are now wrapped using Python's normal line continuation 1267syntax. (Contributed by Antoine Pitrou in :issue:`17150`.) 1268 1269 1270pty 1271--- 1272 1273:func:`pty.spawn` now returns the status value from :func:`os.waitpid` on 1274the child process, instead of ``None``. (Contributed by Gregory P. Smith.) 1275 1276 1277pydoc 1278----- 1279 1280The :mod:`pydoc` module is now based directly on the :func:`inspect.signature` 1281introspection API, allowing it to provide signature information for a wider 1282variety of callable objects. This change also means that ``__wrapped__`` 1283attributes are now taken into account when displaying help information. 1284(Contributed by Larry Hastings in :issue:`19674`.) 1285 1286The :mod:`pydoc` module no longer displays the ``self`` parameter for 1287already bound methods. Instead, it aims to always display the exact current 1288signature of the supplied callable. (Contributed by Larry Hastings in 1289:issue:`20710`.) 1290 1291In addition to the changes that have been made to :mod:`pydoc` directly, 1292its handling of custom ``__dir__`` methods and various descriptor 1293behaviours has also been improved substantially by the underlying changes in 1294the :mod:`inspect` module. 1295 1296As the :func:`help` builtin is based on :mod:`pydoc`, the above changes also 1297affect the behaviour of :func:`help`. 1298 1299 1300re 1301-- 1302 1303New :func:`~re.fullmatch` function and :meth:`.regex.fullmatch` method anchor 1304the pattern at both ends of the string to match. This provides a way to be 1305explicit about the goal of the match, which avoids a class of subtle bugs where 1306``$`` characters get lost during code changes or the addition of alternatives 1307to an existing regular expression. (Contributed by Matthew Barnett in 1308:issue:`16203`.) 1309 1310The repr of :ref:`regex objects <re-objects>` now includes the pattern 1311and the flags; the repr of :ref:`match objects <match-objects>` now 1312includes the start, end, and the part of the string that matched. (Contributed 1313by Hugo Lopes Tavares and Serhiy Storchaka in :issue:`13592` and 1314:issue:`17087`.) 1315 1316 1317resource 1318-------- 1319 1320New :func:`~resource.prlimit` function, available on Linux platforms with a 1321kernel version of 2.6.36 or later and glibc of 2.13 or later, provides the 1322ability to query or set the resource limits for processes other than the one 1323making the call. (Contributed by Christian Heimes in :issue:`16595`.) 1324 1325On Linux kernel version 2.6.36 or later, there are also some new 1326Linux specific constants: :attr:`~resource.RLIMIT_MSGQUEUE`, 1327:attr:`~resource.RLIMIT_NICE`, :attr:`~resource.RLIMIT_RTPRIO`, 1328:attr:`~resource.RLIMIT_RTTIME`, and :attr:`~resource.RLIMIT_SIGPENDING`. 1329(Contributed by Christian Heimes in :issue:`19324`.) 1330 1331On FreeBSD version 9 and later, there some new FreeBSD specific constants: 1332:attr:`~resource.RLIMIT_SBSIZE`, :attr:`~resource.RLIMIT_SWAP`, and 1333:attr:`~resource.RLIMIT_NPTS`. (Contributed by Claudiu Popa in 1334:issue:`19343`.) 1335 1336 1337select 1338------ 1339 1340:class:`~select.epoll` objects now support the context management protocol. 1341When used in a :keyword:`with` statement, the :meth:`~select.epoll.close` 1342method will be called automatically at the end of the block. (Contributed 1343by Serhiy Storchaka in :issue:`16488`.) 1344 1345:class:`~select.devpoll` objects now have :meth:`~select.devpoll.fileno` and 1346:meth:`~select.devpoll.close` methods, as well as a new attribute 1347:attr:`~select.devpoll.closed`. (Contributed by Victor Stinner in 1348:issue:`18794`.) 1349 1350 1351shelve 1352------ 1353 1354:class:`~shelve.Shelf` instances may now be used in :keyword:`with` statements, 1355and will be automatically closed at the end of the :keyword:`!with` block. 1356(Contributed by Filip Gruszczyński in :issue:`13896`.) 1357 1358 1359shutil 1360------ 1361 1362:func:`~shutil.copyfile` now raises a specific :exc:`~shutil.Error` subclass, 1363:exc:`~shutil.SameFileError`, when the source and destination are the same 1364file, which allows an application to take appropriate action on this specific 1365error. (Contributed by Atsuo Ishimoto and Hynek Schlawack in 1366:issue:`1492704`.) 1367 1368 1369smtpd 1370----- 1371 1372The :class:`~smtpd.SMTPServer` and :class:`~smtpd.SMTPChannel` classes now 1373accept a *map* keyword argument which, if specified, is passed in to 1374:class:`asynchat.async_chat` as its *map* argument. This allows an application 1375to avoid affecting the global socket map. (Contributed by Vinay Sajip in 1376:issue:`11959`.) 1377 1378 1379smtplib 1380------- 1381 1382:exc:`~smtplib.SMTPException` is now a subclass of :exc:`OSError`, which allows 1383both socket level errors and SMTP protocol level errors to be caught in one 1384try/except statement by code that only cares whether or not an error occurred. 1385(Contributed by Ned Jackson Lovely in :issue:`2118`.) 1386 1387 1388socket 1389------ 1390 1391The socket module now supports the :data:`~socket.CAN_BCM` protocol on 1392platforms that support it. (Contributed by Brian Thorne in :issue:`15359`.) 1393 1394Socket objects have new methods to get or set their :ref:`inheritable flag 1395<fd_inheritance>`, :meth:`~socket.socket.get_inheritable` and 1396:meth:`~socket.socket.set_inheritable`. 1397 1398The ``socket.AF_*`` and ``socket.SOCK_*`` constants are now enumeration values 1399using the new :mod:`enum` module. This allows meaningful names to be printed 1400during debugging, instead of integer "magic numbers". 1401 1402The :data:`~socket.AF_LINK` constant is now available on BSD and OSX. 1403 1404:func:`~socket.inet_pton` and :func:`~socket.inet_ntop` are now supported 1405on Windows. (Contributed by Atsuo Ishimoto in :issue:`7171`.) 1406 1407 1408sqlite3 1409------- 1410 1411A new boolean parameter to the :func:`~sqlite3.connect` function, *uri*, can be 1412used to indicate that the *database* parameter is a ``uri`` (see the `SQLite 1413URI documentation <https://www.sqlite.org/uri.html>`_). (Contributed by poq in 1414:issue:`13773`.) 1415 1416 1417ssl 1418--- 1419 1420.. _whatsnew-tls-11-12: 1421 1422:data:`~ssl.PROTOCOL_TLSv1_1` and :data:`~ssl.PROTOCOL_TLSv1_2` (TLSv1.1 and 1423TLSv1.2 support) have been added; support for these protocols is only available if 1424Python is linked with OpenSSL 1.0.1 or later. (Contributed by Michele Orrù and 1425Antoine Pitrou in :issue:`16692`.) 1426 1427.. _whatsnew34-sslcontext: 1428 1429New function :func:`~ssl.create_default_context` provides a standard way to 1430obtain an :class:`~ssl.SSLContext` whose settings are intended to be a 1431reasonable balance between compatibility and security. These settings are 1432more stringent than the defaults provided by the :class:`~ssl.SSLContext` 1433constructor, and may be adjusted in the future, without prior deprecation, if 1434best-practice security requirements change. The new recommended best 1435practice for using stdlib libraries that support SSL is to use 1436:func:`~ssl.create_default_context` to obtain an :class:`~ssl.SSLContext` 1437object, modify it if needed, and then pass it as the *context* argument 1438of the appropriate stdlib API. (Contributed by Christian Heimes 1439in :issue:`19689`.) 1440 1441:class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_verify_locations` 1442accepts a new optional argument *cadata*, which can be used to provide PEM or 1443DER encoded certificates directly via strings or bytes, respectively. 1444(Contributed by Christian Heimes in :issue:`18138`.) 1445 1446New function :func:`~ssl.get_default_verify_paths` returns 1447a named tuple of the paths and environment variables that the 1448:meth:`~ssl.SSLContext.set_default_verify_paths` method uses to set 1449OpenSSL's default ``cafile`` and ``capath``. This can be an aid in 1450debugging default verification issues. (Contributed by Christian Heimes 1451in :issue:`18143`.) 1452 1453:class:`~ssl.SSLContext` has a new method, 1454:meth:`~ssl.SSLContext.cert_store_stats`, that reports the number of loaded 1455``X.509`` certs, ``X.509 CA`` certs, and certificate revocation lists 1456(``crl``\ s), as well as a :meth:`~ssl.SSLContext.get_ca_certs` method that 1457returns a list of the loaded ``CA`` certificates. (Contributed by Christian 1458Heimes in :issue:`18147`.) 1459 1460If OpenSSL 0.9.8 or later is available, :class:`~ssl.SSLContext` has a new 1461attribute :attr:`~ssl.SSLContext.verify_flags` that can be used to control the 1462certificate verification process by setting it to some combination of the new 1463constants :data:`~ssl.VERIFY_DEFAULT`, :data:`~ssl.VERIFY_CRL_CHECK_LEAF`, 1464:data:`~ssl.VERIFY_CRL_CHECK_CHAIN`, or :data:`~ssl.VERIFY_X509_STRICT`. 1465OpenSSL does not do any CRL verification by default. (Contributed by 1466Christien Heimes in :issue:`8813`.) 1467 1468New :class:`~ssl.SSLContext` method :meth:`~ssl.SSLContext.load_default_certs` 1469loads a set of default "certificate authority" (CA) certificates from default 1470locations, which vary according to the platform. It can be used to load both 1471TLS web server authentication certificates 1472(``purpose=``:data:`~ssl.Purpose.SERVER_AUTH`) for a client to use to verify a 1473server, and certificates for a server to use in verifying client certificates 1474(``purpose=``:data:`~ssl.Purpose.CLIENT_AUTH`). (Contributed by Christian 1475Heimes in :issue:`19292`.) 1476 1477.. _whatsnew34-win-cert-store: 1478 1479Two new windows-only functions, :func:`~ssl.enum_certificates` and 1480:func:`~ssl.enum_crls` provide the ability to retrieve certificates, 1481certificate information, and CRLs from the Windows cert store. (Contributed 1482by Christian Heimes in :issue:`17134`.) 1483 1484.. _whatsnew34-sni: 1485 1486Support for server-side SNI (Server Name Indication) using the new 1487:meth:`ssl.SSLContext.set_servername_callback` method. 1488(Contributed by Daniel Black in :issue:`8109`.) 1489 1490The dictionary returned by :meth:`.SSLSocket.getpeercert` contains additional 1491``X509v3`` extension items: ``crlDistributionPoints``, ``calIssuers``, and 1492``OCSP`` URIs. (Contributed by Christian Heimes in :issue:`18379`.) 1493 1494 1495stat 1496---- 1497 1498The :mod:`stat` module is now backed by a C implementation in :mod:`_stat`. A C 1499implementation is required as most of the values aren't standardized and 1500are platform-dependent. (Contributed by Christian Heimes in :issue:`11016`.) 1501 1502The module supports new :mod:`~stat.ST_MODE` flags, :mod:`~stat.S_IFDOOR`, 1503:attr:`~stat.S_IFPORT`, and :attr:`~stat.S_IFWHT`. (Contributed by 1504Christian Hiemes in :issue:`11016`.) 1505 1506 1507struct 1508------ 1509 1510New function :mod:`~struct.iter_unpack` and a new 1511:meth:`struct.Struct.iter_unpack` method on compiled formats provide streamed 1512unpacking of a buffer containing repeated instances of a given format of data. 1513(Contributed by Antoine Pitrou in :issue:`17804`.) 1514 1515 1516subprocess 1517---------- 1518 1519:func:`~subprocess.check_output` now accepts an *input* argument that can 1520be used to provide the contents of ``stdin`` for the command that is run. 1521(Contributed by Zack Weinberg in :issue:`16624`.) 1522 1523:func:`~subprocess.getstatus` and :func:`~subprocess.getstatusoutput` now 1524work on Windows. This change was actually inadvertently made in 3.3.4. 1525(Contributed by Tim Golden in :issue:`10197`.) 1526 1527 1528sunau 1529----- 1530 1531The :meth:`~sunau.getparams` method now returns a namedtuple rather than a 1532plain tuple. (Contributed by Claudiu Popa in :issue:`18901`.) 1533 1534:meth:`sunau.open` now supports the context management protocol: when used in a 1535:keyword:`with` block, the ``close`` method of the returned object will be 1536called automatically at the end of the block. (Contributed by Serhiy Storchaka 1537in :issue:`18878`.) 1538 1539:meth:`.AU_write.setsampwidth` now supports 24 bit samples, thus adding 1540support for writing 24 sample using the module. (Contributed by 1541Serhiy Storchaka in :issue:`19261`.) 1542 1543The :meth:`~sunau.AU_write.writeframesraw` and 1544:meth:`~sunau.AU_write.writeframes` methods now accept any :term:`bytes-like 1545object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) 1546 1547 1548sys 1549--- 1550 1551New function :func:`sys.getallocatedblocks` returns the current number of 1552blocks allocated by the interpreter. (In CPython with the default 1553``--with-pymalloc`` setting, this is allocations made through the 1554:c:func:`PyObject_Malloc` API.) This can be useful for tracking memory leaks, 1555especially if automated via a test suite. (Contributed by Antoine Pitrou 1556in :issue:`13390`.) 1557 1558When the Python interpreter starts in :ref:`interactive mode 1559<tut-interactive>`, it checks for an :data:`~sys.__interactivehook__` attribute 1560on the :mod:`sys` module. If the attribute exists, its value is called with no 1561arguments just before interactive mode is started. The check is made after the 1562:envvar:`PYTHONSTARTUP` file is read, so it can be set there. The :mod:`site` 1563module :ref:`sets it <rlcompleter-config>` to a function that enables tab 1564completion and history saving (in :file:`~/.python-history`) if the platform 1565supports :mod:`readline`. If you do not want this (new) behavior, you can 1566override it in :envvar:`PYTHONSTARTUP`, :mod:`sitecustomize`, or 1567:mod:`usercustomize` by deleting this attribute from :mod:`sys` (or setting it 1568to some other callable). (Contributed by Éric Araujo and Antoine Pitrou in 1569:issue:`5845`.) 1570 1571 1572tarfile 1573------- 1574 1575The :mod:`tarfile` module now supports a simple :ref:`tarfile-commandline` when 1576called as a script directly or via ``-m``. This can be used to create and 1577extract tarfile archives. (Contributed by Berker Peksag in :issue:`13477`.) 1578 1579 1580textwrap 1581-------- 1582 1583The :class:`~textwrap.TextWrapper` class has two new attributes/constructor 1584arguments: :attr:`~textwrap.TextWrapper.max_lines`, which limits the number of 1585lines in the output, and :attr:`~textwrap.TextWrapper.placeholder`, which is a 1586string that will appear at the end of the output if it has been truncated 1587because of *max_lines*. Building on these capabilities, a new convenience 1588function :func:`~textwrap.shorten` collapses all of the whitespace in the input 1589to single spaces and produces a single line of a given *width* that ends with 1590the *placeholder* (by default, ``[...]``). (Contributed by Antoine Pitrou and 1591Serhiy Storchaka in :issue:`18585` and :issue:`18725`.) 1592 1593 1594threading 1595--------- 1596 1597The :class:`~threading.Thread` object representing the main thread can be 1598obtained from the new :func:`~threading.main_thread` function. In normal 1599conditions this will be the thread from which the Python interpreter was 1600started. (Contributed by Andrew Svetlov in :issue:`18882`.) 1601 1602 1603traceback 1604--------- 1605 1606A new :func:`traceback.clear_frames` function takes a traceback object 1607and clears the local variables in all of the frames it references, 1608reducing the amount of memory consumed. (Contributed by Andrew Kuchling in 1609:issue:`1565525`.) 1610 1611 1612types 1613----- 1614 1615A new :func:`~types.DynamicClassAttribute` descriptor provides a way to define 1616an attribute that acts normally when looked up through an instance object, but 1617which is routed to the *class* ``__getattr__`` when looked up through the 1618class. This allows one to have properties active on a class, and have virtual 1619attributes on the class with the same name (see :mod:`Enum` for an example). 1620(Contributed by Ethan Furman in :issue:`19030`.) 1621 1622 1623urllib 1624------ 1625 1626:mod:`urllib.request` now supports ``data:`` URLs via the 1627:class:`~urllib.request.DataHandler` class. (Contributed by Mathias Panzenböck 1628in :issue:`16423`.) 1629 1630The http method that will be used by a :class:`~urllib.request.Request` class 1631can now be specified by setting a :class:`~urllib.request.Request.method` 1632class attribute on the subclass. (Contributed by Jason R Coombs in 1633:issue:`18978`.) 1634 1635:class:`~urllib.request.Request` objects are now reusable: if the 1636:attr:`~urllib.request.Request.full_url` or :attr:`~urllib.request.Request.data` 1637attributes are modified, all relevant internal properties are updated. This 1638means, for example, that it is now possible to use the same 1639:class:`~urllib.request.Request` object in more than one 1640:meth:`.OpenerDirector.open` call with different *data* arguments, or to 1641modify a :class:`~urllib.request.Request`\ 's ``url`` rather than recomputing it 1642from scratch. There is also a new 1643:meth:`~urllib.request.Request.remove_header` method that can be used to remove 1644headers from a :class:`~urllib.request.Request`. (Contributed by Alexey 1645Kachayev in :issue:`16464`, Daniel Wozniak in :issue:`17485`, and Damien Brecht 1646and Senthil Kumaran in :issue:`17272`.) 1647 1648:class:`~urllib.error.HTTPError` objects now have a 1649:attr:`~urllib.error.HTTPError.headers` attribute that provides access to the 1650HTTP response headers associated with the error. (Contributed by 1651Berker Peksag in :issue:`15701`.) 1652 1653 1654unittest 1655-------- 1656 1657The :class:`~unittest.TestCase` class has a new method, 1658:meth:`~unittest.TestCase.subTest`, that produces a context manager whose 1659:keyword:`with` block becomes a "sub-test". This context manager allows a test 1660method to dynamically generate subtests by, say, calling the ``subTest`` 1661context manager inside a loop. A single test method can thereby produce an 1662indefinite number of separately-identified and separately-counted tests, all of 1663which will run even if one or more of them fail. For example:: 1664 1665 class NumbersTest(unittest.TestCase): 1666 def test_even(self): 1667 for i in range(6): 1668 with self.subTest(i=i): 1669 self.assertEqual(i % 2, 0) 1670 1671will result in six subtests, each identified in the unittest verbose output 1672with a label consisting of the variable name ``i`` and a particular value for 1673that variable (``i=0``, ``i=1``, etc). See :ref:`subtests` for the full 1674version of this example. (Contributed by Antoine Pitrou in :issue:`16997`.) 1675 1676:func:`unittest.main` now accepts an iterable of test names for 1677*defaultTest*, where previously it only accepted a single test name as a 1678string. (Contributed by Jyrki Pulliainen in :issue:`15132`.) 1679 1680If :class:`~unittest.SkipTest` is raised during test discovery (that is, at the 1681module level in the test file), it is now reported as a skip instead of an 1682error. (Contributed by Zach Ware in :issue:`16935`.) 1683 1684:meth:`~unittest.TestLoader.discover` now sorts the discovered files to provide 1685consistent test ordering. (Contributed by Martin Melin and Jeff Ramnani in 1686:issue:`16709`.) 1687 1688:class:`~unittest.TestSuite` now drops references to tests as soon as the test 1689has been run, if the test is successful. On Python interpreters that do 1690garbage collection, this allows the tests to be garbage collected if nothing 1691else is holding a reference to the test. It is possible to override this 1692behavior by creating a :class:`~unittest.TestSuite` subclass that defines a 1693custom ``_removeTestAtIndex`` method. (Contributed by Tom Wardill, Matt 1694McClure, and Andrew Svetlov in :issue:`11798`.) 1695 1696A new test assertion context-manager, :meth:`~unittest.TestCase.assertLogs`, 1697will ensure that a given block of code emits a log message using the 1698:mod:`logging` module. By default the message can come from any logger and 1699have a priority of ``INFO`` or higher, but both the logger name and an 1700alternative minimum logging level may be specified. The object returned by the 1701context manager can be queried for the :class:`~logging.LogRecord`\ s and/or 1702formatted messages that were logged. (Contributed by Antoine Pitrou in 1703:issue:`18937`.) 1704 1705Test discovery now works with namespace packages (Contributed by Claudiu Popa 1706in :issue:`17457`.) 1707 1708:mod:`unittest.mock` objects now inspect their specification signatures when 1709matching calls, which means an argument can now be matched by either position 1710or name, instead of only by position. (Contributed by Antoine Pitrou in 1711:issue:`17015`.) 1712 1713:func:`~mock.mock_open` objects now have ``readline`` and ``readlines`` 1714methods. (Contributed by Toshio Kuratomi in :issue:`17467`.) 1715 1716 1717venv 1718---- 1719 1720:mod:`venv` now includes activation scripts for the ``csh`` and ``fish`` 1721shells. (Contributed by Andrew Svetlov in :issue:`15417`.) 1722 1723:class:`~venv.EnvBuilder` and the :func:`~venv.create` convenience function 1724take a new keyword argument *with_pip*, which defaults to ``False``, that 1725controls whether or not :class:`~venv.EnvBuilder` ensures that ``pip`` is 1726installed in the virtual environment. (Contributed by Nick Coghlan in 1727:issue:`19552` as part of the :pep:`453` implementation.) 1728 1729 1730wave 1731---- 1732 1733The :meth:`~wave.getparams` method now returns a namedtuple rather than a 1734plain tuple. (Contributed by Claudiu Popa in :issue:`17487`.) 1735 1736:meth:`wave.open` now supports the context management protocol. (Contributed 1737by Claudiu Popa in :issue:`17616`.) 1738 1739:mod:`wave` can now :ref:`write output to unseekable files 1740<wave-write-objects>`. (Contributed by David Jones, Guilherme Polo, and Serhiy 1741Storchaka in :issue:`5202`.) 1742 1743The :meth:`~wave.Wave_write.writeframesraw` and 1744:meth:`~wave.Wave_write.writeframes` methods now accept any :term:`bytes-like 1745object`. (Contributed by Serhiy Storchaka in :issue:`8311`.) 1746 1747 1748weakref 1749------- 1750 1751New :class:`~weakref.WeakMethod` class simulates weak references to bound 1752methods. (Contributed by Antoine Pitrou in :issue:`14631`.) 1753 1754New :class:`~weakref.finalize` class makes it possible to register a callback 1755to be invoked when an object is garbage collected, without needing to 1756carefully manage the lifecycle of the weak reference itself. (Contributed by 1757Richard Oudkerk in :issue:`15528`.) 1758 1759The callback, if any, associated with a :class:`~weakref.ref` is now 1760exposed via the :attr:`~weakref.ref.__callback__` attribute. (Contributed 1761by Mark Dickinson in :issue:`17643`.) 1762 1763 1764xml.etree 1765--------- 1766 1767A new parser, :class:`~xml.etree.ElementTree.XMLPullParser`, allows a 1768non-blocking applications to parse XML documents. An example can be 1769seen at :ref:`elementtree-pull-parsing`. (Contributed by Antoine 1770Pitrou in :issue:`17741`.) 1771 1772The :mod:`xml.etree.ElementTree` :func:`~xml.etree.ElementTree.tostring` and 1773:func:`~xml.etree.ElementTree.tostringlist` functions, and the 1774:class:`~xml.etree.ElementTree.ElementTree` 1775:meth:`~xml.etree.ElementTree.ElementTree.write` method, now have a 1776*short_empty_elements* :ref:`keyword-only parameter <keyword-only_parameter>` 1777providing control over whether elements with no content are written in 1778abbreviated (``<tag />``) or expanded (``<tag></tag>``) form. (Contributed by 1779Ariel Poliak and Serhiy Storchaka in :issue:`14377`.) 1780 1781 1782zipfile 1783------- 1784 1785The :meth:`~zipfile.PyZipFile.writepy` method of the 1786:class:`~zipfile.PyZipFile` class has a new *filterfunc* option that can be 1787used to control which directories and files are added to the archive. For 1788example, this could be used to exclude test files from the archive. 1789(Contributed by Christian Tismer in :issue:`19274`.) 1790 1791The *allowZip64* parameter to :class:`~zipfile.ZipFile` and 1792:class:`~zipfile.PyZipfile` is now ``True`` by default. (Contributed by 1793William Mallard in :issue:`17201`.) 1794 1795 1796 1797CPython Implementation Changes 1798============================== 1799 1800 1801.. _whatsnew-pep-445: 1802 1803PEP 445: Customization of CPython Memory Allocators 1804--------------------------------------------------- 1805 1806:pep:`445` adds new C level interfaces to customize memory allocation in 1807the CPython interpreter. 1808 1809.. seealso:: 1810 1811 :pep:`445` -- Add new APIs to customize Python memory allocators 1812 PEP written and implemented by Victor Stinner. 1813 1814 1815.. _whatsnew-pep-442: 1816 1817PEP 442: Safe Object Finalization 1818--------------------------------- 1819 1820:pep:`442` removes the current limitations and quirks of object finalization 1821in CPython. With it, objects with :meth:`__del__` methods, as well as 1822generators with :keyword:`finally` clauses, can be finalized when they are 1823part of a reference cycle. 1824 1825As part of this change, module globals are no longer forcibly set to 1826:const:`None` during interpreter shutdown in most cases, instead relying 1827on the normal operation of the cyclic garbage collector. This avoids a 1828whole class of interpreter-shutdown-time errors, usually involving 1829``__del__`` methods, that have plagued Python since the cyclic GC 1830was first introduced. 1831 1832.. seealso:: 1833 1834 :pep:`442` -- Safe object finalization 1835 PEP written and implemented by Antoine Pitrou. 1836 1837 1838.. _whatsnew-pep-456: 1839 1840PEP 456: Secure and Interchangeable Hash Algorithm 1841-------------------------------------------------- 1842 1843:pep:`456` follows up on earlier security fix work done on Python's hash 1844algorithm to address certain DOS attacks to which public facing APIs backed by 1845dictionary lookups may be subject. (See :issue:`14621` for the start of the 1846current round of improvements.) The PEP unifies CPython's hash code to make it 1847easier for a packager to substitute a different hash algorithm, and switches 1848Python's default implementation to a SipHash implementation on platforms that 1849have a 64 bit data type. Any performance differences in comparison with the 1850older FNV algorithm are trivial. 1851 1852The PEP adds additional fields to the :attr:`sys.hash_info` named tuple to 1853describe the hash algorithm in use by the currently executing binary. Otherwise, 1854the PEP does not alter any existing CPython APIs. 1855 1856 1857.. _whatsnew-pep-436: 1858 1859PEP 436: Argument Clinic 1860------------------------ 1861 1862"Argument Clinic" (:pep:`436`) is now part of the CPython build process 1863and can be used to simplify the process of defining and maintaining 1864accurate signatures for builtins and standard library extension modules 1865implemented in C. 1866 1867Some standard library extension modules have been converted to use Argument 1868Clinic in Python 3.4, and :mod:`pydoc` and :mod:`inspect` have been updated 1869accordingly. 1870 1871It is expected that signature metadata for programmatic introspection will 1872be added to additional callables implemented in C as part of Python 3.4 1873maintenance releases. 1874 1875.. note:: 1876 The Argument Clinic PEP is not fully up to date with the state of the 1877 implementation. This has been deemed acceptable by the release manager 1878 and core development team in this case, as Argument Clinic will not 1879 be made available as a public API for third party use in Python 3.4. 1880 1881.. seealso:: 1882 1883 :pep:`436` -- The Argument Clinic DSL 1884 PEP written and implemented by Larry Hastings. 1885 1886 1887Other Build and C API Changes 1888----------------------------- 1889 1890* The new :c:func:`PyType_GetSlot` function has been added to the stable ABI, 1891 allowing retrieval of function pointers from named type slots when using 1892 the limited API. (Contributed by Martin von Löwis in :issue:`17162`.) 1893 1894* The new :c:func:`Py_SetStandardStreamEncoding` pre-initialization API 1895 allows applications embedding the CPython interpreter to reliably force 1896 a particular encoding and error handler for the standard streams. 1897 (Contributed by Bastien Montagne and Nick Coghlan in :issue:`16129`.) 1898 1899* Most Python C APIs that don't mutate string arguments are now correctly 1900 marked as accepting ``const char *`` rather than ``char *``. (Contributed 1901 by Serhiy Storchaka in :issue:`1772673`.) 1902 1903* A new shell version of ``python-config`` can be used even when a python 1904 interpreter is not available (for example, in cross compilation scenarios). 1905 1906* :c:func:`PyUnicode_FromFormat` now supports width and precision 1907 specifications for ``%s``, ``%A``, ``%U``, ``%V``, ``%S``, and ``%R``. 1908 (Contributed by Ysj Ray and Victor Stinner in :issue:`7330`.) 1909 1910* New function :c:func:`PyStructSequence_InitType2` supplements the 1911 existing :c:func:`PyStructSequence_InitType` function. The difference 1912 is that it returns ``0`` on success and ``-1`` on failure. 1913 1914* The CPython source can now be compiled using the address sanity checking 1915 features of recent versions of GCC and clang: the false alarms in the small 1916 object allocator have been silenced. (Contributed by Dhiru Kholia in 1917 :issue:`18596`.) 1918 1919* The Windows build now uses `Address Space Layout Randomization 1920 <https://en.wikipedia.org/wiki/Address_space_layout_randomization>`_ and `Data Execution Prevention 1921 <https://en.wikipedia.org/wiki/Data_Execution_Prevention>`_. (Contributed by 1922 Christian Heimes in :issue:`16632`.) 1923 1924* New function :c:func:`PyObject_LengthHint` is the C API equivalent 1925 of :func:`operator.length_hint`. (Contributed by Armin Ronacher in 1926 :issue:`16148`.) 1927 1928 1929.. _other-improvements-3.4: 1930 1931Other Improvements 1932------------------ 1933 1934.. _whatsnew-isolated-mode: 1935 1936* The :ref:`python <using-on-cmdline>` command has a new :ref:`option 1937 <using-on-misc-options>`, ``-I``, which causes it to run in "isolated mode", 1938 which means that :data:`sys.path` contains neither the script's directory nor 1939 the user's ``site-packages`` directory, and all :envvar:`PYTHON*` environment 1940 variables are ignored (it implies both ``-s`` and ``-E``). Other 1941 restrictions may also be applied in the future, with the goal being to 1942 isolate the execution of a script from the user's environment. This is 1943 appropriate, for example, when Python is used to run a system script. On 1944 most POSIX systems it can and should be used in the ``#!`` line of system 1945 scripts. (Contributed by Christian Heimes in :issue:`16499`.) 1946 1947* Tab-completion is now enabled by default in the interactive interpreter 1948 on systems that support :mod:`readline`. History is also enabled by default, 1949 and is written to (and read from) the file :file:`~/.python-history`. 1950 (Contributed by Antoine Pitrou and Éric Araujo in :issue:`5845`.) 1951 1952* Invoking the Python interpreter with ``--version`` now outputs the version to 1953 standard output instead of standard error (:issue:`18338`). Similar changes 1954 were made to :mod:`argparse` (:issue:`18920`) and other modules that have 1955 script-like invocation capabilities (:issue:`18922`). 1956 1957* The CPython Windows installer now adds ``.py`` to the :envvar:`PATHEXT` 1958 variable when extensions are registered, allowing users to run a python 1959 script at the windows command prompt by just typing its name without the 1960 ``.py`` extension. (Contributed by Paul Moore in :issue:`18569`.) 1961 1962* A new ``make`` target `coverage-report 1963 <https://devguide.python.org/coverage/#measuring-coverage-of-c-code-with-gcov-and-lcov>`_ 1964 will build python, run the test suite, and generate an HTML coverage report 1965 for the C codebase using ``gcov`` and `lcov 1966 <http://ltp.sourceforge.net/coverage/lcov.php>`_. 1967 1968* The ``-R`` option to the :ref:`python regression test suite <regrtest>` now 1969 also checks for memory allocation leaks, using 1970 :func:`sys.getallocatedblocks()`. (Contributed by Antoine Pitrou in 1971 :issue:`13390`.) 1972 1973* ``python -m`` now works with namespace packages. 1974 1975* The :mod:`stat` module is now implemented in C, which means it gets the 1976 values for its constants from the C header files, instead of having the 1977 values hard-coded in the python module as was previously the case. 1978 1979* Loading multiple python modules from a single OS module (``.so``, ``.dll``) 1980 now works correctly (previously it silently returned the first python 1981 module in the file). (Contributed by Václav Šmilauer in :issue:`16421`.) 1982 1983* A new opcode, :opcode:`LOAD_CLASSDEREF`, has been added to fix a bug in the 1984 loading of free variables in class bodies that could be triggered by certain 1985 uses of :ref:`__prepare__ <prepare>`. (Contributed by Benjamin Peterson in 1986 :issue:`17853`.) 1987 1988* A number of MemoryError-related crashes were identified and fixed by Victor 1989 Stinner using his :pep:`445`-based ``pyfailmalloc`` tool (:issue:`18408`, 1990 :issue:`18520`). 1991 1992* The ``pyvenv`` command now accepts a ``--copies`` option 1993 to use copies rather than symlinks even on systems where symlinks are the 1994 default. (Contributed by Vinay Sajip in :issue:`18807`.) 1995 1996* The ``pyvenv`` command also accepts a ``--without-pip`` 1997 option to suppress the otherwise-automatic bootstrapping of pip into 1998 the virtual environment. (Contributed by Nick Coghlan in :issue:`19552` 1999 as part of the :pep:`453` implementation.) 2000 2001* The encoding name is now optional in the value set for the 2002 :envvar:`PYTHONIOENCODING` environment variable. This makes it possible to 2003 set just the error handler, without changing the default encoding. 2004 (Contributed by Serhiy Storchaka in :issue:`18818`.) 2005 2006* The :mod:`bz2`, :mod:`lzma`, and :mod:`gzip` module ``open`` functions now 2007 support ``x`` (exclusive creation) mode. (Contributed by Tim Heaney and 2008 Vajrasky Kok in :issue:`19201`, :issue:`19222`, and :issue:`19223`.) 2009 2010 2011Significant Optimizations 2012------------------------- 2013 2014* The UTF-32 decoder is now 3x to 4x faster. (Contributed by Serhiy Storchaka 2015 in :issue:`14625`.) 2016 2017* The cost of hash collisions for sets is now reduced. Each hash table 2018 probe now checks a series of consecutive, adjacent key/hash pairs before 2019 continuing to make random probes through the hash table. This exploits 2020 cache locality to make collision resolution less expensive. 2021 The collision resolution scheme can be described as a hybrid of linear 2022 probing and open addressing. The number of additional linear probes 2023 defaults to nine. This can be changed at compile-time by defining 2024 LINEAR_PROBES to be any value. Set LINEAR_PROBES=0 to turn-off 2025 linear probing entirely. (Contributed by Raymond Hettinger in 2026 :issue:`18771`.) 2027 2028* The interpreter starts about 30% faster. A couple of measures lead to the 2029 speedup. The interpreter loads fewer modules on startup, e.g. the :mod:`re`, 2030 :mod:`collections` and :mod:`locale` modules and their dependencies are no 2031 longer imported by default. The marshal module has been improved to load 2032 compiled Python code faster. (Contributed by Antoine Pitrou, Christian 2033 Heimes and Victor Stinner in :issue:`19219`, :issue:`19218`, :issue:`19209`, 2034 :issue:`19205` and :issue:`9548`.) 2035 2036* :class:`bz2.BZ2File` is now as fast or faster than the Python2 version for 2037 most cases. :class:`lzma.LZMAFile` has also been optimized. (Contributed by 2038 Serhiy Storchaka and Nadeem Vawda in :issue:`16034`.) 2039 2040* :func:`random.getrandbits` is 20%-40% faster for small integers (the most 2041 common use case). (Contributed by Serhiy Storchaka in :issue:`16674`.) 2042 2043* By taking advantage of the new storage format for strings, pickling of 2044 strings is now significantly faster. (Contributed by Victor Stinner and 2045 Antoine Pitrou in :issue:`15596`.) 2046 2047* A performance issue in :meth:`io.FileIO.readall` has been solved. This 2048 particularly affects Windows, and significantly speeds up the case of piping 2049 significant amounts of data through :mod:`subprocess`. (Contributed 2050 by Richard Oudkerk in :issue:`15758`.) 2051 2052* :func:`html.escape` is now 10x faster. (Contributed by Matt Bryant in 2053 :issue:`18020`.) 2054 2055* On Windows, the native ``VirtualAlloc`` is now used instead of the CRT 2056 ``malloc`` in ``obmalloc``. Artificial benchmarks show about a 3% memory 2057 savings. 2058 2059* :func:`os.urandom` now uses a lazily-opened persistent file descriptor 2060 so as to avoid using many file descriptors when run in parallel from 2061 multiple threads. (Contributed by Antoine Pitrou in :issue:`18756`.) 2062 2063 2064.. _deprecated-3.4: 2065 2066Deprecated 2067========== 2068 2069This section covers various APIs and other features that have been deprecated 2070in Python 3.4, and will be removed in Python 3.5 or later. In most (but not 2071all) cases, using the deprecated APIs will produce a :exc:`DeprecationWarning` 2072when the interpreter is run with deprecation warnings enabled (for example, by 2073using ``-Wd``). 2074 2075 2076Deprecations in the Python API 2077------------------------------ 2078 2079* As mentioned in :ref:`whatsnew-pep-451`, a number of :mod:`importlib` 2080 methods and functions are deprecated: :meth:`importlib.find_loader` is 2081 replaced by :func:`importlib.util.find_spec`; 2082 :meth:`importlib.machinery.PathFinder.find_module` is replaced by 2083 :meth:`importlib.machinery.PathFinder.find_spec`; 2084 :meth:`importlib.abc.MetaPathFinder.find_module` is replaced by 2085 :meth:`importlib.abc.MetaPathFinder.find_spec`; 2086 :meth:`importlib.abc.PathEntryFinder.find_loader` and 2087 :meth:`~importlib.abc.PathEntryFinder.find_module` are replaced by 2088 :meth:`importlib.abc.PathEntryFinder.find_spec`; all of the ``xxxLoader`` ABC 2089 ``load_module`` methods (:meth:`importlib.abc.Loader.load_module`, 2090 :meth:`importlib.abc.InspectLoader.load_module`, 2091 :meth:`importlib.abc.FileLoader.load_module`, 2092 :meth:`importlib.abc.SourceLoader.load_module`) should no longer be 2093 implemented, instead loaders should implement an 2094 ``exec_module`` method 2095 (:meth:`importlib.abc.Loader.exec_module`, 2096 :meth:`importlib.abc.InspectLoader.exec_module` 2097 :meth:`importlib.abc.SourceLoader.exec_module`) and let the import system 2098 take care of the rest; and 2099 :meth:`importlib.abc.Loader.module_repr`, 2100 :meth:`importlib.util.module_for_loader`, :meth:`importlib.util.set_loader`, 2101 and :meth:`importlib.util.set_package` are no longer needed because their 2102 functions are now handled automatically by the import system. 2103 2104* The :mod:`imp` module is pending deprecation. To keep compatibility with 2105 Python 2/3 code bases, the module's removal is currently not scheduled. 2106 2107* The :mod:`formatter` module is pending deprecation and is slated for removal 2108 in Python 3.6. 2109 2110* ``MD5`` as the default *digestmod* for the :func:`hmac.new` function is 2111 deprecated. Python 3.6 will require an explicit digest name or constructor as 2112 *digestmod* argument. 2113 2114* The internal ``Netrc`` class in the :mod:`ftplib` module has been documented 2115 as deprecated in its docstring for quite some time. It now emits a 2116 :exc:`DeprecationWarning` and will be removed completely in Python 3.5. 2117 2118* The undocumented *endtime* argument to :meth:`subprocess.Popen.wait` should 2119 not have been exposed and is hopefully not in use; it is deprecated and 2120 will mostly likely be removed in Python 3.5. 2121 2122* The *strict* argument of :class:`~html.parser.HTMLParser` is deprecated. 2123 2124* The :mod:`plistlib` :func:`~plistlib.readPlist`, 2125 :func:`~plistlib.writePlist`, :func:`~plistlib.readPlistFromBytes`, and 2126 :func:`~plistlib.writePlistToBytes` functions are deprecated in favor of the 2127 corresponding new functions :func:`~plistlib.load`, :func:`~plistlib.dump`, 2128 :func:`~plistlib.loads`, and :func:`~plistlib.dumps`. :func:`~plistlib.Data` 2129 is deprecated in favor of just using the :class:`bytes` constructor. 2130 2131* The :mod:`sysconfig` key ``SO`` is deprecated, it has been replaced by 2132 ``EXT_SUFFIX``. 2133 2134* The ``U`` mode accepted by various ``open`` functions is deprecated. 2135 In Python3 it does not do anything useful, and should be replaced by 2136 appropriate uses of :class:`io.TextIOWrapper` (if needed) and its *newline* 2137 argument. 2138 2139* The *parser* argument of :func:`xml.etree.ElementTree.iterparse` has 2140 been deprecated, as has the *html* argument of 2141 :func:`~xml.etree.ElementTree.XMLParser`. To prepare for the removal of the 2142 latter, all arguments to ``XMLParser`` should be passed by keyword. 2143 2144 2145Deprecated Features 2146------------------- 2147 2148* Running :ref:`idle` with the ``-n`` flag (no subprocess) is deprecated. 2149 However, the feature will not be removed until :issue:`18823` is resolved. 2150 2151* The site module adding a "site-python" directory to sys.path, if it 2152 exists, is deprecated (:issue:`19375`). 2153 2154 2155 2156Removed 2157======= 2158 2159 2160Operating Systems No Longer Supported 2161------------------------------------- 2162 2163Support for the following operating systems has been removed from the source 2164and build tools: 2165 2166* OS/2 (:issue:`16135`). 2167* Windows 2000 (changeset e52df05b496a). 2168* Windows systems where ``COMSPEC`` points to ``command.com`` (:issue:`14470`). 2169* VMS (:issue:`16136`). 2170 2171 2172API and Feature Removals 2173------------------------ 2174 2175The following obsolete and previously deprecated APIs and features have been 2176removed: 2177 2178* The unmaintained ``Misc/TextMate`` and ``Misc/vim`` directories have been 2179 removed (see the `devguide <https://devguide.python.org>`_ 2180 for suggestions on what to use instead). 2181 2182* The ``SO`` makefile macro is removed (it was replaced by the 2183 ``SHLIB_SUFFIX`` and ``EXT_SUFFIX`` macros) (:issue:`16754`). 2184 2185* The ``PyThreadState.tick_counter`` field has been removed; its value has 2186 been meaningless since Python 3.2, when the "new GIL" was introduced 2187 (:issue:`19199`). 2188 2189* ``PyLoader`` and ``PyPycLoader`` have been removed from :mod:`importlib`. 2190 (Contributed by Taras Lyapun in :issue:`15641`.) 2191 2192* The *strict* argument to :class:`~http.client.HTTPConnection` and 2193 :class:`~http.client.HTTPSConnection` has been removed. HTTP 0.9-style 2194 "Simple Responses" are no longer supported. 2195 2196* The deprecated :mod:`urllib.request.Request` getter and setter methods 2197 ``add_data``, ``has_data``, ``get_data``, ``get_type``, ``get_host``, 2198 ``get_selector``, ``set_proxy``, ``get_origin_req_host``, and 2199 ``is_unverifiable`` have been removed (use direct attribute access instead). 2200 2201* Support for loading the deprecated ``TYPE_INT64`` has been removed from 2202 :mod:`marshal`. (Contributed by Dan Riti in :issue:`15480`.) 2203 2204* :class:`inspect.Signature`: positional-only parameters are now required 2205 to have a valid name. 2206 2207* :meth:`object.__format__` no longer accepts non-empty format strings, it now 2208 raises a :exc:`TypeError` instead. Using a non-empty string has been 2209 deprecated since Python 3.2. This change has been made to prevent a 2210 situation where previously working (but incorrect) code would start failing 2211 if an object gained a __format__ method, which means that your code may now 2212 raise a :exc:`TypeError` if you are using an ``'s'`` format code with objects 2213 that do not have a __format__ method that handles it. See :issue:`7994` for 2214 background. 2215 2216* :meth:`difflib.SequenceMatcher.isbjunk` and 2217 :meth:`difflib.SequenceMatcher.isbpopular` were deprecated in 3.2, and have 2218 now been removed: use ``x in sm.bjunk`` and 2219 ``x in sm.bpopular``, where *sm* is a :class:`~difflib.SequenceMatcher` object 2220 (:issue:`13248`). 2221 2222 2223Code Cleanups 2224------------- 2225 2226* The unused and undocumented internal ``Scanner`` class has been removed from 2227 the :mod:`pydoc` module. 2228 2229* The private and effectively unused ``_gestalt`` module has been removed, 2230 along with the private :mod:`platform` functions ``_mac_ver_lookup``, 2231 ``_mac_ver_gstalt``, and ``_bcd2str``, which would only have ever been called 2232 on badly broken OSX systems (see :issue:`18393`). 2233 2234* The hardcoded copies of certain :mod:`stat` constants that were included in 2235 the :mod:`tarfile` module namespace have been removed. 2236 2237 2238 2239Porting to Python 3.4 2240===================== 2241 2242This section lists previously described changes and other bugfixes 2243that may require changes to your code. 2244 2245 2246Changes in 'python' Command Behavior 2247------------------------------------ 2248 2249* In a posix shell, setting the :envvar:`PATH` environment variable to 2250 an empty value is equivalent to not setting it at all. However, setting 2251 :envvar:`PYTHONPATH` to an empty value was *not* equivalent to not setting it 2252 at all: setting :envvar:`PYTHONPATH` to an empty value was equivalent to 2253 setting it to ``.``, which leads to confusion when reasoning by analogy to 2254 how :envvar:`PATH` works. The behavior now conforms to the posix convention 2255 for :envvar:`PATH`. 2256 2257* The [X refs, Y blocks] output of a debug (``--with-pydebug``) build of the 2258 CPython interpreter is now off by default. It can be re-enabled using the 2259 ``-X showrefcount`` option. (Contributed by Ezio Melotti in :issue:`17323`.) 2260 2261* The python command and most stdlib scripts (as well as :mod:`argparse`) now 2262 output ``--version`` information to ``stdout`` instead of ``stderr`` (for 2263 issue list see :ref:`other-improvements-3.4` above). 2264 2265 2266Changes in the Python API 2267------------------------- 2268 2269* The ABCs defined in :mod:`importlib.abc` now either raise the appropriate 2270 exception or return a default value instead of raising 2271 :exc:`NotImplementedError` blindly. This will only affect code calling 2272 :func:`super` and falling through all the way to the ABCs. For compatibility, 2273 catch both :exc:`NotImplementedError` or the appropriate exception as needed. 2274 2275* The module type now initializes the :attr:`__package__` and :attr:`__loader__` 2276 attributes to ``None`` by default. To determine if these attributes were set 2277 in a backwards-compatible fashion, use e.g. 2278 ``getattr(module, '__loader__', None) is not None``. (:issue:`17115`.) 2279 2280* :meth:`importlib.util.module_for_loader` now sets ``__loader__`` and 2281 ``__package__`` unconditionally to properly support reloading. If this is not 2282 desired then you will need to set these attributes manually. You can use 2283 :func:`importlib.util.module_to_load` for module management. 2284 2285* Import now resets relevant attributes (e.g. ``__name__``, ``__loader__``, 2286 ``__package__``, ``__file__``, ``__cached__``) unconditionally when reloading. 2287 Note that this restores a pre-3.3 behavior in that it means a module is 2288 re-found when re-loaded (:issue:`19413`). 2289 2290* Frozen packages no longer set ``__path__`` to a list containing the package 2291 name, they now set it to an empty list. The previous behavior could cause 2292 the import system to do the wrong thing on submodule imports if there was 2293 also a directory with the same name as the frozen package. The correct way 2294 to determine if a module is a package or not is to use ``hasattr(module, 2295 '__path__')`` (:issue:`18065`). 2296 2297* Frozen modules no longer define a ``__file__`` attribute. It's semantically 2298 incorrect for frozen modules to set the attribute as they are not loaded from 2299 any explicit location. If you must know that a module comes from frozen code 2300 then you can see if the module's ``__spec__.location`` is set to ``'frozen'``, 2301 check if the loader is a subclass of 2302 :class:`importlib.machinery.FrozenImporter`, 2303 or if Python 2 compatibility is necessary you can use :func:`imp.is_frozen`. 2304 2305* :func:`py_compile.compile` now raises :exc:`FileExistsError` if the file path 2306 it would write to is a symlink or a non-regular file. This is to act as a 2307 warning that import will overwrite those files with a regular file regardless 2308 of what type of file path they were originally. 2309 2310* :meth:`importlib.abc.SourceLoader.get_source` no longer raises 2311 :exc:`ImportError` when the source code being loaded triggers a 2312 :exc:`SyntaxError` or :exc:`UnicodeDecodeError`. As :exc:`ImportError` is 2313 meant to be raised only when source code cannot be found but it should, it was 2314 felt to be over-reaching/overloading of that meaning when the source code is 2315 found but improperly structured. If you were catching ImportError before and 2316 wish to continue to ignore syntax or decoding issues, catch all three 2317 exceptions now. 2318 2319* :func:`functools.update_wrapper` and :func:`functools.wraps` now correctly 2320 set the ``__wrapped__`` attribute to the function being wrapped, even if 2321 that function also had its ``__wrapped__`` attribute set. This means 2322 ``__wrapped__`` attributes now correctly link a stack of decorated 2323 functions rather than every ``__wrapped__`` attribute in the chain 2324 referring to the innermost function. Introspection libraries that 2325 assumed the previous behaviour was intentional can use 2326 :func:`inspect.unwrap` to access the first function in the chain that has 2327 no ``__wrapped__`` attribute. 2328 2329* :func:`inspect.getfullargspec` has been reimplemented on top of 2330 :func:`inspect.signature` and hence handles a much wider variety of callable 2331 objects than it did in the past. It is expected that additional builtin and 2332 extension module callables will gain signature metadata over the course of 2333 the Python 3.4 series. Code that assumes that 2334 :func:`inspect.getfullargspec` will fail on non-Python callables may need 2335 to be adjusted accordingly. 2336 2337* :class:`importlib.machinery.PathFinder` now passes on the current working 2338 directory to objects in :data:`sys.path_hooks` for the empty string. This 2339 results in :data:`sys.path_importer_cache` never containing ``''``, thus 2340 iterating through :data:`sys.path_importer_cache` based on :data:`sys.path` 2341 will not find all keys. A module's ``__file__`` when imported in the current 2342 working directory will also now have an absolute path, including when using 2343 ``-m`` with the interpreter (except for ``__main__.__file__`` when a script 2344 has been executed directly using a relative path) (Contributed by Brett 2345 Cannon in :issue:`18416`). is specified on the command-line) 2346 (:issue:`18416`). 2347 2348* The removal of the *strict* argument to :class:`~http.client.HTTPConnection` 2349 and :class:`~http.client.HTTPSConnection` changes the meaning of the 2350 remaining arguments if you are specifying them positionally rather than by 2351 keyword. If you've been paying attention to deprecation warnings your code 2352 should already be specifying any additional arguments via keywords. 2353 2354* Strings between ``from __future__ import ...`` statements now *always* raise 2355 a :exc:`SyntaxError`. Previously if there was no leading docstring, an 2356 interstitial string would sometimes be ignored. This brings CPython into 2357 compliance with the language spec; Jython and PyPy already were. 2358 (:issue:`17434`). 2359 2360* :meth:`ssl.SSLSocket.getpeercert` and :meth:`ssl.SSLSocket.do_handshake` 2361 now raise an :exc:`OSError` with ``ENOTCONN`` when the ``SSLSocket`` is not 2362 connected, instead of the previous behavior of raising an 2363 :exc:`AttributeError`. In addition, :meth:`~ssl.SSLSocket.getpeercert` 2364 will raise a :exc:`ValueError` if the handshake has not yet been done. 2365 2366* :func:`base64.b32decode` now raises a :exc:`binascii.Error` when the 2367 input string contains non-b32-alphabet characters, instead of a 2368 :exc:`TypeError`. This particular :exc:`TypeError` was missed when the other 2369 :exc:`TypeError`\ s were converted. (Contributed by Serhiy Storchaka in 2370 :issue:`18011`.) Note: this change was also inadvertently applied in Python 2371 3.3.3. 2372 2373* The :attr:`~cgi.FieldStorage.file` attribute is now automatically closed when 2374 the creating :class:`cgi.FieldStorage` instance is garbage collected. If you 2375 were pulling the file object out separately from the :class:`cgi.FieldStorage` 2376 instance and not keeping the instance alive, then you should either store the 2377 entire :class:`cgi.FieldStorage` instance or read the contents of the file 2378 before the :class:`cgi.FieldStorage` instance is garbage collected. 2379 2380* Calling ``read`` or ``write`` on a closed SSL socket now raises an 2381 informative :exc:`ValueError` rather than the previous more mysterious 2382 :exc:`AttributeError` (:issue:`9177`). 2383 2384* :meth:`slice.indices` no longer produces an :exc:`OverflowError` for huge 2385 values. As a consequence of this fix, :meth:`slice.indices` now raises a 2386 :exc:`ValueError` if given a negative length; previously it returned nonsense 2387 values (:issue:`14794`). 2388 2389* The :class:`complex` constructor, unlike the :mod:`cmath` functions, was 2390 incorrectly accepting :class:`float` values if an object's ``__complex__`` 2391 special method returned one. This now raises a :exc:`TypeError`. 2392 (:issue:`16290`.) 2393 2394* The :class:`int` constructor in 3.2 and 3.3 erroneously accepts :class:`float` 2395 values for the *base* parameter. It is unlikely anyone was doing this, but 2396 if so, it will now raise a :exc:`TypeError` (:issue:`16772`). 2397 2398* Defaults for keyword-only arguments are now evaluated *after* defaults for 2399 regular keyword arguments, instead of before. Hopefully no one wrote any 2400 code that depends on the previous buggy behavior (:issue:`16967`). 2401 2402* Stale thread states are now cleared after :func:`~os.fork`. This may cause 2403 some system resources to be released that previously were incorrectly kept 2404 perpetually alive (for example, database connections kept in thread-local 2405 storage). (:issue:`17094`.) 2406 2407* Parameter names in ``__annotations__`` dicts are now mangled properly, 2408 similarly to ``__kwdefaults__``. (Contributed by Yury Selivanov in 2409 :issue:`20625`.) 2410 2411* :attr:`hashlib.hash.name` now always returns the identifier in lower case. 2412 Previously some builtin hashes had uppercase names, but now that it is a 2413 formal public interface the naming has been made consistent (:issue:`18532`). 2414 2415* Because :mod:`unittest.TestSuite` now drops references to tests after they 2416 are run, test harnesses that re-use a :class:`~unittest.TestSuite` to re-run 2417 a set of tests may fail. Test suites should not be re-used in this fashion 2418 since it means state is retained between test runs, breaking the test 2419 isolation that :mod:`unittest` is designed to provide. However, if the lack 2420 of isolation is considered acceptable, the old behavior can be restored by 2421 creating a :mod:`~unittest.TestSuite` subclass that defines a 2422 ``_removeTestAtIndex`` method that does nothing (see 2423 :meth:`.TestSuite.__iter__`) (:issue:`11798`). 2424 2425* :mod:`unittest` now uses :mod:`argparse` for command line parsing. There are 2426 certain invalid command forms that used to work that are no longer allowed; 2427 in theory this should not cause backward compatibility issues since the 2428 disallowed command forms didn't make any sense and are unlikely to be in use. 2429 2430* The :func:`re.split`, :func:`re.findall`, and :func:`re.sub` functions, and 2431 the :meth:`~re.match.group` and :meth:`~re.match.groups` methods of 2432 ``match`` objects now always return a *bytes* object when the string 2433 to be matched is a :term:`bytes-like object`. Previously the return type 2434 matched the input type, so if your code was depending on the return value 2435 being, say, a ``bytearray``, you will need to change your code. 2436 2437* :mod:`audioop` functions now raise an error immediately if passed string 2438 input, instead of failing randomly later on (:issue:`16685`). 2439 2440* The new *convert_charrefs* argument to :class:`~html.parser.HTMLParser` 2441 currently defaults to ``False`` for backward compatibility, but will 2442 eventually be changed to default to ``True``. It is recommended that you add 2443 this keyword, with the appropriate value, to any 2444 :class:`~html.parser.HTMLParser` calls in your code (:issue:`13633`). 2445 2446* Since the *digestmod* argument to the :func:`hmac.new` function will in the 2447 future have no default, all calls to :func:`hmac.new` should be changed to 2448 explicitly specify a *digestmod* (:issue:`17276`). 2449 2450* Calling :func:`sysconfig.get_config_var` with the ``SO`` key, or looking 2451 ``SO`` up in the results of a call to :func:`sysconfig.get_config_vars` 2452 is deprecated. This key should be replaced by ``EXT_SUFFIX`` or 2453 ``SHLIB_SUFFIX``, depending on the context (:issue:`19555`). 2454 2455* Any calls to ``open`` functions that specify ``U`` should be modified. 2456 ``U`` is ineffective in Python3 and will eventually raise an error if used. 2457 Depending on the function, the equivalent of its old Python2 behavior can be 2458 achieved using either a *newline* argument, or if necessary by wrapping the 2459 stream in :mod:`~io.TextIOWrapper` to use its *newline* argument 2460 (:issue:`15204`). 2461 2462* If you use ``pyvenv`` in a script and desire that pip 2463 *not* be installed, you must add ``--without-pip`` to your command 2464 invocation. 2465 2466* The default behavior of :func:`json.dump` and :func:`json.dumps` when 2467 an indent is specified has changed: it no longer produces trailing 2468 spaces after the item separating commas at the ends of lines. This 2469 will matter only if you have tests that are doing white-space-sensitive 2470 comparisons of such output (:issue:`16333`). 2471 2472* :mod:`doctest` now looks for doctests in extension module ``__doc__`` 2473 strings, so if your doctest test discovery includes extension modules that 2474 have things that look like doctests in them you may see test failures you've 2475 never seen before when running your tests (:issue:`3158`). 2476 2477* The :mod:`collections.abc` module has been slightly refactored as 2478 part of the Python startup improvements. As a consequence of this, it is no 2479 longer the case that importing :mod:`collections` automatically imports 2480 :mod:`collections.abc`. If your program depended on the (undocumented) 2481 implicit import, you will need to add an explicit ``import collections.abc`` 2482 (:issue:`20784`). 2483 2484 2485Changes in the C API 2486-------------------- 2487 2488* :c:func:`PyEval_EvalFrameEx`, :c:func:`PyObject_Repr`, and 2489 :c:func:`PyObject_Str`, along with some other internal C APIs, now include 2490 a debugging assertion that ensures they are not used in situations where 2491 they may silently discard a currently active exception. In cases where 2492 discarding the active exception is expected and desired (for example, 2493 because it has already been saved locally with :c:func:`PyErr_Fetch` or 2494 is being deliberately replaced with a different exception), an explicit 2495 :c:func:`PyErr_Clear` call will be needed to avoid triggering the 2496 assertion when invoking these operations (directly or indirectly) and 2497 running against a version of Python that is compiled with assertions 2498 enabled. 2499 2500* :c:func:`PyErr_SetImportError` now sets :exc:`TypeError` when its **msg** 2501 argument is not set. Previously only ``NULL`` was returned with no exception 2502 set. 2503 2504* The result of the :c:data:`PyOS_ReadlineFunctionPointer` callback must 2505 now be a string allocated by :c:func:`PyMem_RawMalloc` or 2506 :c:func:`PyMem_RawRealloc`, or ``NULL`` if an error occurred, instead of a 2507 string allocated by :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc` 2508 (:issue:`16742`) 2509 2510* :c:func:`PyThread_set_key_value` now always set the value. In Python 2511 3.3, the function did nothing if the key already exists (if the current 2512 value is a non-``NULL`` pointer). 2513 2514* The ``f_tstate`` (thread state) field of the :c:type:`PyFrameObject` 2515 structure has been removed to fix a bug: see :issue:`14432` for the 2516 rationale. 2517 2518Changed in 3.4.3 2519================ 2520 2521.. _pep-476: 2522 2523PEP 476: Enabling certificate verification by default for stdlib http clients 2524----------------------------------------------------------------------------- 2525 2526:mod:`http.client` and modules which use it, such as :mod:`urllib.request` and 2527:mod:`xmlrpc.client`, will now verify that the server presents a certificate 2528which is signed by a CA in the platform trust store and whose hostname matches 2529the hostname being requested by default, significantly improving security for 2530many applications. 2531 2532For applications which require the old previous behavior, they can pass an 2533alternate context:: 2534 2535 import urllib.request 2536 import ssl 2537 2538 # This disables all verification 2539 context = ssl._create_unverified_context() 2540 2541 # This allows using a specific certificate for the host, which doesn't need 2542 # to be in the trust store 2543 context = ssl.create_default_context(cafile="/path/to/file.crt") 2544 2545 urllib.request.urlopen("https://invalid-cert", context=context) 2546