• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1****************************
2  What's New In Python 3.2
3****************************
4
5:Author: Raymond Hettinger
6
7.. $Id$
8   Rules for maintenance:
9
10   * Anyone can add text to this document.  Do not spend very much time
11   on the wording of your changes, because your text will probably
12   get rewritten.  (Note, during release candidate phase or just before
13   a beta release, please use the tracker instead -- this helps avoid
14   merge conflicts.   If you must add a suggested entry directly,
15   please put it in an XXX comment and the maintainer will take notice).
16
17   * The maintainer will go through Misc/NEWS periodically and add
18   changes; it's therefore more important to add your changes to
19   Misc/NEWS than to this file.
20
21   * This is not a complete list of every single change; completeness
22   is the purpose of Misc/NEWS.  Some changes I consider too small
23   or esoteric to include.  If such a change is added to the text,
24   I'll just remove it.  (This is another reason you shouldn't spend
25   too much time on writing your addition.)
26
27   * If you want to draw your new text to the attention of the
28   maintainer, add 'XXX' to the beginning of the paragraph or
29   section.
30
31   * It's OK to just add a fragmentary note about a change.  For
32   example: "XXX Describe the transmogrify() function added to the
33   socket module."  The maintainer will research the change and
34   write the necessary text.
35
36   * You can comment out your additions if you like, but it's not
37   necessary (especially when a final release is some months away).
38
39   * Credit the author of a patch or bugfix.   Just the name is
40   sufficient; the e-mail address isn't necessary.  It's helpful to
41   add the issue number:
42
43     XXX Describe the transmogrify() function added to the socket
44     module.
45
46     (Contributed by P.Y. Developer; :issue:`12345`.)
47
48   This saves the maintainer the effort of going through the SVN log
49   when researching a change.
50
51This article explains the new features in Python 3.2 as compared to 3.1.  It
52focuses on a few highlights and gives a few examples.  For full details, see the
53`Misc/NEWS
54<https://github.com/python/cpython/blob/076ca6c3c8df3030307e548d9be792ce3c1c6eea/Misc/NEWS>`_
55file.
56
57.. seealso::
58
59   :pep:`392` - Python 3.2 Release Schedule
60
61
62PEP 384: Defining a Stable ABI
63==============================
64
65In the past, extension modules built for one Python version were often
66not usable with other Python versions. Particularly on Windows, every
67feature release of Python required rebuilding all extension modules that
68one wanted to use. This requirement was the result of the free access to
69Python interpreter internals that extension modules could use.
70
71With Python 3.2, an alternative approach becomes available: extension
72modules which restrict themselves to a limited API (by defining
73Py_LIMITED_API) cannot use many of the internals, but are constrained
74to a set of API functions that are promised to be stable for several
75releases. As a consequence, extension modules built for 3.2 in that
76mode will also work with 3.3, 3.4, and so on. Extension modules that
77make use of details of memory structures can still be built, but will
78need to be recompiled for every feature release.
79
80.. seealso::
81
82   :pep:`384` - Defining a Stable ABI
83      PEP written by Martin von Löwis.
84
85
86PEP 389: Argparse Command Line Parsing Module
87=============================================
88
89A new module for command line parsing, :mod:`argparse`, was introduced to
90overcome the limitations of :mod:`optparse` which did not provide support for
91positional arguments (not just options), subcommands, required options and other
92common patterns of specifying and validating options.
93
94This module has already had widespread success in the community as a
95third-party module.  Being more fully featured than its predecessor, the
96:mod:`argparse` module is now the preferred module for command-line processing.
97The older module is still being kept available because of the substantial amount
98of legacy code that depends on it.
99
100Here's an annotated example parser showing features like limiting results to a
101set of choices, specifying a *metavar* in the help screen, validating that one
102or more positional arguments is present, and making a required option::
103
104    import argparse
105    parser = argparse.ArgumentParser(
106                description = 'Manage servers',         # main description for help
107                epilog = 'Tested on Solaris and Linux') # displayed after help
108    parser.add_argument('action',                       # argument name
109                choices = ['deploy', 'start', 'stop'],  # three allowed values
110                help = 'action on each target')         # help msg
111    parser.add_argument('targets',
112                metavar = 'HOSTNAME',                   # var name used in help msg
113                nargs = '+',                            # require one or more targets
114                help = 'url for target machines')       # help msg explanation
115    parser.add_argument('-u', '--user',                 # -u or --user option
116                required = True,                        # make it a required argument
117                help = 'login as user')
118
119Example of calling the parser on a command string::
120
121    >>> cmd = 'deploy sneezy.example.com sleepy.example.com -u skycaptain'
122    >>> result = parser.parse_args(cmd.split())
123    >>> result.action
124    'deploy'
125    >>> result.targets
126    ['sneezy.example.com', 'sleepy.example.com']
127    >>> result.user
128    'skycaptain'
129
130Example of the parser's automatically generated help::
131
132    >>> parser.parse_args('-h'.split())
133
134    usage: manage_cloud.py [-h] -u USER
135                           {deploy,start,stop} HOSTNAME [HOSTNAME ...]
136
137    Manage servers
138
139    positional arguments:
140      {deploy,start,stop}   action on each target
141      HOSTNAME              url for target machines
142
143    optional arguments:
144      -h, --help            show this help message and exit
145      -u USER, --user USER  login as user
146
147    Tested on Solaris and Linux
148
149An especially nice :mod:`argparse` feature is the ability to define subparsers,
150each with their own argument patterns and help displays::
151
152    import argparse
153    parser = argparse.ArgumentParser(prog='HELM')
154    subparsers = parser.add_subparsers()
155
156    parser_l = subparsers.add_parser('launch', help='Launch Control')   # first subgroup
157    parser_l.add_argument('-m', '--missiles', action='store_true')
158    parser_l.add_argument('-t', '--torpedos', action='store_true')
159
160    parser_m = subparsers.add_parser('move', help='Move Vessel',        # second subgroup
161                                     aliases=('steer', 'turn'))         # equivalent names
162    parser_m.add_argument('-c', '--course', type=int, required=True)
163    parser_m.add_argument('-s', '--speed', type=int, default=0)
164
165.. code-block:: shell-session
166
167    $ ./helm.py --help                         # top level help (launch and move)
168    $ ./helm.py launch --help                  # help for launch options
169    $ ./helm.py launch --missiles              # set missiles=True and torpedos=False
170    $ ./helm.py steer --course 180 --speed 5   # set movement parameters
171
172.. seealso::
173
174   :pep:`389` - New Command Line Parsing Module
175      PEP written by Steven Bethard.
176
177   :ref:`upgrading-optparse-code` for details on the differences from :mod:`optparse`.
178
179
180PEP 391:  Dictionary Based Configuration for Logging
181====================================================
182
183The :mod:`logging` module provided two kinds of configuration, one style with
184function calls for each option or another style driven by an external file saved
185in a :mod:`ConfigParser` format.  Those options did not provide the flexibility
186to create configurations from JSON or YAML files, nor did they support
187incremental configuration, which is needed for specifying logger options from a
188command line.
189
190To support a more flexible style, the module now offers
191:func:`logging.config.dictConfig` for specifying logging configuration with
192plain Python dictionaries.  The configuration options include formatters,
193handlers, filters, and loggers.  Here's a working example of a configuration
194dictionary::
195
196   {"version": 1,
197    "formatters": {"brief": {"format": "%(levelname)-8s: %(name)-15s: %(message)s"},
198                   "full": {"format": "%(asctime)s %(name)-15s %(levelname)-8s %(message)s"}
199                   },
200    "handlers": {"console": {
201                      "class": "logging.StreamHandler",
202                      "formatter": "brief",
203                      "level": "INFO",
204                      "stream": "ext://sys.stdout"},
205                 "console_priority": {
206                      "class": "logging.StreamHandler",
207                      "formatter": "full",
208                      "level": "ERROR",
209                      "stream": "ext://sys.stderr"}
210                 },
211    "root": {"level": "DEBUG", "handlers": ["console", "console_priority"]}}
212
213
214If that dictionary is stored in a file called :file:`conf.json`, it can be
215loaded and called with code like this::
216
217   >>> import json, logging.config
218   >>> with open('conf.json') as f:
219   ...     conf = json.load(f)
220   ...
221   >>> logging.config.dictConfig(conf)
222   >>> logging.info("Transaction completed normally")
223   INFO    : root           : Transaction completed normally
224   >>> logging.critical("Abnormal termination")
225   2011-02-17 11:14:36,694 root            CRITICAL Abnormal termination
226
227.. seealso::
228
229   :pep:`391` - Dictionary Based Configuration for Logging
230      PEP written by Vinay Sajip.
231
232
233PEP 3148:  The ``concurrent.futures`` module
234============================================
235
236Code for creating and managing concurrency is being collected in a new top-level
237namespace, *concurrent*.  Its first member is a *futures* package which provides
238a uniform high-level interface for managing threads and processes.
239
240The design for :mod:`concurrent.futures` was inspired by the
241*java.util.concurrent* package.  In that model, a running call and its result
242are represented by a :class:`~concurrent.futures.Future` object that abstracts
243features common to threads, processes, and remote procedure calls.  That object
244supports status checks (running or done), timeouts, cancellations, adding
245callbacks, and access to results or exceptions.
246
247The primary offering of the new module is a pair of executor classes for
248launching and managing calls.  The goal of the executors is to make it easier to
249use existing tools for making parallel calls. They save the effort needed to
250setup a pool of resources, launch the calls, create a results queue, add
251time-out handling, and limit the total number of threads, processes, or remote
252procedure calls.
253
254Ideally, each application should share a single executor across multiple
255components so that process and thread limits can be centrally managed.  This
256solves the design challenge that arises when each component has its own
257competing strategy for resource management.
258
259Both classes share a common interface with three methods:
260:meth:`~concurrent.futures.Executor.submit` for scheduling a callable and
261returning a :class:`~concurrent.futures.Future` object;
262:meth:`~concurrent.futures.Executor.map` for scheduling many asynchronous calls
263at a time, and :meth:`~concurrent.futures.Executor.shutdown` for freeing
264resources.  The class is a :term:`context manager` and can be used in a
265:keyword:`with` statement to assure that resources are automatically released
266when currently pending futures are done executing.
267
268A simple of example of :class:`~concurrent.futures.ThreadPoolExecutor` is a
269launch of four parallel threads for copying files::
270
271  import concurrent.futures, shutil
272  with concurrent.futures.ThreadPoolExecutor(max_workers=4) as e:
273      e.submit(shutil.copy, 'src1.txt', 'dest1.txt')
274      e.submit(shutil.copy, 'src2.txt', 'dest2.txt')
275      e.submit(shutil.copy, 'src3.txt', 'dest3.txt')
276      e.submit(shutil.copy, 'src3.txt', 'dest4.txt')
277
278.. seealso::
279
280   :pep:`3148` - Futures -- Execute Computations Asynchronously
281      PEP written by Brian Quinlan.
282
283   :ref:`Code for Threaded Parallel URL reads<threadpoolexecutor-example>`, an
284   example using threads to fetch multiple web pages in parallel.
285
286   :ref:`Code for computing prime numbers in
287   parallel<processpoolexecutor-example>`, an example demonstrating
288   :class:`~concurrent.futures.ProcessPoolExecutor`.
289
290
291PEP 3147:  PYC Repository Directories
292=====================================
293
294Python's scheme for caching bytecode in *.pyc* files did not work well in
295environments with multiple Python interpreters.  If one interpreter encountered
296a cached file created by another interpreter, it would recompile the source and
297overwrite the cached file, thus losing the benefits of caching.
298
299The issue of "pyc fights" has become more pronounced as it has become
300commonplace for Linux distributions to ship with multiple versions of Python.
301These conflicts also arise with CPython alternatives such as Unladen Swallow.
302
303To solve this problem, Python's import machinery has been extended to use
304distinct filenames for each interpreter.  Instead of Python 3.2 and Python 3.3 and
305Unladen Swallow each competing for a file called "mymodule.pyc", they will now
306look for "mymodule.cpython-32.pyc", "mymodule.cpython-33.pyc", and
307"mymodule.unladen10.pyc".  And to prevent all of these new files from
308cluttering source directories, the *pyc* files are now collected in a
309"__pycache__" directory stored under the package directory.
310
311Aside from the filenames and target directories, the new scheme has a few
312aspects that are visible to the programmer:
313
314* Imported modules now have a :attr:`__cached__` attribute which stores the name
315  of the actual file that was imported:
316
317   >>> import collections
318   >>> collections.__cached__ # doctest: +SKIP
319   'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
320
321* The tag that is unique to each interpreter is accessible from the :mod:`imp`
322  module:
323
324   >>> import imp
325   >>> imp.get_tag() # doctest: +SKIP
326   'cpython-32'
327
328* Scripts that try to deduce source filename from the imported file now need to
329  be smarter.  It is no longer sufficient to simply strip the "c" from a ".pyc"
330  filename.  Instead, use the new functions in the :mod:`imp` module:
331
332  >>> imp.source_from_cache('c:/py32/lib/__pycache__/collections.cpython-32.pyc')
333  'c:/py32/lib/collections.py'
334  >>> imp.cache_from_source('c:/py32/lib/collections.py') # doctest: +SKIP
335  'c:/py32/lib/__pycache__/collections.cpython-32.pyc'
336
337* The :mod:`py_compile` and :mod:`compileall` modules have been updated to
338  reflect the new naming convention and target directory.  The command-line
339  invocation of *compileall* has new options: ``-i`` for
340  specifying a list of files and directories to compile and ``-b`` which causes
341  bytecode files to be written to their legacy location rather than
342  *__pycache__*.
343
344* The :mod:`importlib.abc` module has been updated with new :term:`abstract base
345  classes <abstract base class>` for loading bytecode files.  The obsolete
346  ABCs, :class:`~importlib.abc.PyLoader` and
347  :class:`~importlib.abc.PyPycLoader`, have been deprecated (instructions on how
348  to stay Python 3.1 compatible are included with the documentation).
349
350.. seealso::
351
352   :pep:`3147` - PYC Repository Directories
353      PEP written by Barry Warsaw.
354
355
356PEP 3149: ABI Version Tagged .so Files
357======================================
358
359The PYC repository directory allows multiple bytecode cache files to be
360co-located.  This PEP implements a similar mechanism for shared object files by
361giving them a common directory and distinct names for each version.
362
363The common directory is "pyshared" and the file names are made distinct by
364identifying the Python implementation (such as CPython, PyPy, Jython, etc.), the
365major and minor version numbers, and optional build flags (such as "d" for
366debug, "m" for pymalloc, "u" for wide-unicode).  For an arbitrary package "foo",
367you may see these files when the distribution package is installed::
368
369   /usr/share/pyshared/foo.cpython-32m.so
370   /usr/share/pyshared/foo.cpython-33md.so
371
372In Python itself, the tags are accessible from functions in the :mod:`sysconfig`
373module::
374
375   >>> import sysconfig
376   >>> sysconfig.get_config_var('SOABI')       # find the version tag
377   'cpython-32mu'
378   >>> sysconfig.get_config_var('EXT_SUFFIX')  # find the full filename extension
379   '.cpython-32mu.so'
380
381.. seealso::
382
383   :pep:`3149` - ABI Version Tagged .so Files
384      PEP written by Barry Warsaw.
385
386
387PEP 3333: Python Web Server Gateway Interface v1.0.1
388=====================================================
389
390This informational PEP clarifies how bytes/text issues are to be handled by the
391WSGI protocol.  The challenge is that string handling in Python 3 is most
392conveniently handled with the :class:`str` type even though the HTTP protocol
393is itself bytes oriented.
394
395The PEP differentiates so-called *native strings* that are used for
396request/response headers and metadata versus *byte strings* which are used for
397the bodies of requests and responses.
398
399The *native strings* are always of type :class:`str` but are restricted to code
400points between *U+0000* through *U+00FF* which are translatable to bytes using
401*Latin-1* encoding.  These strings are used for the keys and values in the
402environment dictionary and for response headers and statuses in the
403:func:`start_response` function.  They must follow :rfc:`2616` with respect to
404encoding. That is, they must either be *ISO-8859-1* characters or use
405:rfc:`2047` MIME encoding.
406
407For developers porting WSGI applications from Python 2, here are the salient
408points:
409
410* If the app already used strings for headers in Python 2, no change is needed.
411
412* If instead, the app encoded output headers or decoded input headers, then the
413  headers will need to be re-encoded to Latin-1.  For example, an output header
414  encoded in utf-8 was using ``h.encode('utf-8')`` now needs to convert from
415  bytes to native strings using ``h.encode('utf-8').decode('latin-1')``.
416
417* Values yielded by an application or sent using the :meth:`write` method
418  must be byte strings.  The :func:`start_response` function and environ
419  must use native strings.  The two cannot be mixed.
420
421For server implementers writing CGI-to-WSGI pathways or other CGI-style
422protocols, the users must to be able access the environment using native strings
423even though the underlying platform may have a different convention.  To bridge
424this gap, the :mod:`wsgiref` module has a new function,
425:func:`wsgiref.handlers.read_environ` for transcoding CGI variables from
426:attr:`os.environ` into native strings and returning a new dictionary.
427
428.. seealso::
429
430   :pep:`3333` - Python Web Server Gateway Interface v1.0.1
431      PEP written by Phillip Eby.
432
433
434Other Language Changes
435======================
436
437Some smaller changes made to the core Python language are:
438
439* String formatting for :func:`format` and :meth:`str.format` gained new
440  capabilities for the format character **#**.  Previously, for integers in
441  binary, octal, or hexadecimal, it caused the output to be prefixed with '0b',
442  '0o', or '0x' respectively.  Now it can also handle floats, complex, and
443  Decimal, causing the output to always have a decimal point even when no digits
444  follow it.
445
446  >>> format(20, '#o')
447  '0o24'
448  >>> format(12.34, '#5.0f')
449  '  12.'
450
451  (Suggested by Mark Dickinson and implemented by Eric Smith in :issue:`7094`.)
452
453* There is also a new :meth:`str.format_map` method that extends the
454  capabilities of the existing :meth:`str.format` method by accepting arbitrary
455  :term:`mapping` objects.  This new method makes it possible to use string
456  formatting with any of Python's many dictionary-like objects such as
457  :class:`~collections.defaultdict`, :class:`~shelve.Shelf`,
458  :class:`~configparser.ConfigParser`, or :mod:`dbm`.  It is also useful with
459  custom :class:`dict` subclasses that normalize keys before look-up or that
460  supply a :meth:`__missing__` method for unknown keys::
461
462    >>> import shelve
463    >>> d = shelve.open('tmp.shl')
464    >>> 'The {project_name} status is {status} as of {date}'.format_map(d)
465    'The testing project status is green as of February 15, 2011'
466
467    >>> class LowerCasedDict(dict):
468    ...     def __getitem__(self, key):
469    ...         return dict.__getitem__(self, key.lower())
470    >>> lcd = LowerCasedDict(part='widgets', quantity=10)
471    >>> 'There are {QUANTITY} {Part} in stock'.format_map(lcd)
472    'There are 10 widgets in stock'
473
474    >>> class PlaceholderDict(dict):
475    ...     def __missing__(self, key):
476    ...         return '<{}>'.format(key)
477    >>> 'Hello {name}, welcome to {location}'.format_map(PlaceholderDict())
478    'Hello <name>, welcome to <location>'
479
480 (Suggested by Raymond Hettinger and implemented by Eric Smith in
481 :issue:`6081`.)
482
483* The interpreter can now be started with a quiet option, ``-q``, to prevent
484  the copyright and version information from being displayed in the interactive
485  mode.  The option can be introspected using the :attr:`sys.flags` attribute:
486
487  .. code-block:: shell-session
488
489    $ python -q
490    >>> sys.flags
491    sys.flags(debug=0, division_warning=0, inspect=0, interactive=0,
492    optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0,
493    ignore_environment=0, verbose=0, bytes_warning=0, quiet=1)
494
495  (Contributed by Marcin Wojdyr in :issue:`1772833`).
496
497* The :func:`hasattr` function works by calling :func:`getattr` and detecting
498  whether an exception is raised.  This technique allows it to detect methods
499  created dynamically by :meth:`__getattr__` or :meth:`__getattribute__` which
500  would otherwise be absent from the class dictionary.  Formerly, *hasattr*
501  would catch any exception, possibly masking genuine errors.  Now, *hasattr*
502  has been tightened to only catch :exc:`AttributeError` and let other
503  exceptions pass through::
504
505    >>> class A:
506    ...     @property
507    ...     def f(self):
508    ...         return 1 // 0
509    ...
510    >>> a = A()
511    >>> hasattr(a, 'f')
512    Traceback (most recent call last):
513      ...
514    ZeroDivisionError: integer division or modulo by zero
515
516  (Discovered by Yury Selivanov and fixed by Benjamin Peterson; :issue:`9666`.)
517
518* The :func:`str` of a float or complex number is now the same as its
519  :func:`repr`. Previously, the :func:`str` form was shorter but that just
520  caused confusion and is no longer needed now that the shortest possible
521  :func:`repr` is displayed by default:
522
523   >>> import math
524   >>> repr(math.pi)
525   '3.141592653589793'
526   >>> str(math.pi)
527   '3.141592653589793'
528
529  (Proposed and implemented by Mark Dickinson; :issue:`9337`.)
530
531* :class:`memoryview` objects now have a :meth:`~memoryview.release()` method
532  and they also now support the context management protocol.  This allows timely
533  release of any resources that were acquired when requesting a buffer from the
534  original object.
535
536  >>> with memoryview(b'abcdefgh') as v:
537  ...     print(v.tolist())
538  [97, 98, 99, 100, 101, 102, 103, 104]
539
540  (Added by Antoine Pitrou; :issue:`9757`.)
541
542* Previously it was illegal to delete a name from the local namespace if it
543  occurs as a free variable in a nested block::
544
545       def outer(x):
546           def inner():
547               return x
548           inner()
549           del x
550
551  This is now allowed.  Remember that the target of an :keyword:`except` clause
552  is cleared, so this code which used to work with Python 2.6, raised a
553  :exc:`SyntaxError` with Python 3.1 and now works again::
554
555       def f():
556           def print_error():
557               print(e)
558           try:
559               something
560           except Exception as e:
561               print_error()
562               # implicit "del e" here
563
564  (See :issue:`4617`.)
565
566* The internal :c:type:`structsequence` tool now creates subclasses of tuple.
567  This means that C structures like those returned by :func:`os.stat`,
568  :func:`time.gmtime`, and :attr:`sys.version_info` now work like a
569  :term:`named tuple` and now work with functions and methods that
570  expect a tuple as an argument.  This is a big step forward in making the C
571  structures as flexible as their pure Python counterparts:
572
573  >>> import sys
574  >>> isinstance(sys.version_info, tuple)
575  True
576  >>> 'Version %d.%d.%d %s(%d)' % sys.version_info # doctest: +SKIP
577  'Version 3.2.0 final(0)'
578
579  (Suggested by Arfrever Frehtes Taifersar Arahesis and implemented
580  by Benjamin Peterson in :issue:`8413`.)
581
582* Warnings are now easier to control using the :envvar:`PYTHONWARNINGS`
583  environment variable as an alternative to using ``-W`` at the command line:
584
585  .. code-block:: shell-session
586
587    $ export PYTHONWARNINGS='ignore::RuntimeWarning::,once::UnicodeWarning::'
588
589  (Suggested by Barry Warsaw and implemented by Philip Jenvey in :issue:`7301`.)
590
591* A new warning category, :exc:`ResourceWarning`, has been added.  It is
592  emitted when potential issues with resource consumption or cleanup
593  are detected.  It is silenced by default in normal release builds but
594  can be enabled through the means provided by the :mod:`warnings`
595  module, or on the command line.
596
597  A :exc:`ResourceWarning` is issued at interpreter shutdown if the
598  :data:`gc.garbage` list isn't empty, and if :attr:`gc.DEBUG_UNCOLLECTABLE` is
599  set, all uncollectable objects are printed.  This is meant to make the
600  programmer aware that their code contains object finalization issues.
601
602  A :exc:`ResourceWarning` is also issued when a :term:`file object` is destroyed
603  without having been explicitly closed.  While the deallocator for such
604  object ensures it closes the underlying operating system resource
605  (usually, a file descriptor), the delay in deallocating the object could
606  produce various issues, especially under Windows.  Here is an example
607  of enabling the warning from the command line:
608
609  .. code-block:: shell-session
610
611      $ python -q -Wdefault
612      >>> f = open("foo", "wb")
613      >>> del f
614      __main__:1: ResourceWarning: unclosed file <_io.BufferedWriter name='foo'>
615
616  (Added by Antoine Pitrou and Georg Brandl in :issue:`10093` and :issue:`477863`.)
617
618* :class:`range` objects now support *index* and *count* methods. This is part
619  of an effort to make more objects fully implement the
620  :class:`collections.Sequence` :term:`abstract base class`.  As a result, the
621  language will have a more uniform API.  In addition, :class:`range` objects
622  now support slicing and negative indices, even with values larger than
623  :attr:`sys.maxsize`.  This makes *range* more interoperable with lists::
624
625      >>> range(0, 100, 2).count(10)
626      1
627      >>> range(0, 100, 2).index(10)
628      5
629      >>> range(0, 100, 2)[5]
630      10
631      >>> range(0, 100, 2)[0:5]
632      range(0, 10, 2)
633
634  (Contributed by Daniel Stutzbach in :issue:`9213`, by Alexander Belopolsky
635  in :issue:`2690`, and by Nick Coghlan in :issue:`10889`.)
636
637* The :func:`callable` builtin function from Py2.x was resurrected.  It provides
638  a concise, readable alternative to using an :term:`abstract base class` in an
639  expression like ``isinstance(x, collections.Callable)``:
640
641  >>> callable(max)
642  True
643  >>> callable(20)
644  False
645
646  (See :issue:`10518`.)
647
648* Python's import mechanism can now load modules installed in directories with
649  non-ASCII characters in the path name.  This solved an aggravating problem
650  with home directories for users with non-ASCII characters in their usernames.
651
652 (Required extensive work by Victor Stinner in :issue:`9425`.)
653
654
655New, Improved, and Deprecated Modules
656=====================================
657
658Python's standard library has undergone significant maintenance efforts and
659quality improvements.
660
661The biggest news for Python 3.2 is that the :mod:`email` package, :mod:`mailbox`
662module, and :mod:`nntplib` modules now work correctly with the bytes/text model
663in Python 3.  For the first time, there is correct handling of messages with
664mixed encodings.
665
666Throughout the standard library, there has been more careful attention to
667encodings and text versus bytes issues.  In particular, interactions with the
668operating system are now better able to exchange non-ASCII data using the
669Windows MBCS encoding, locale-aware encodings, or UTF-8.
670
671Another significant win is the addition of substantially better support for
672*SSL* connections and security certificates.
673
674In addition, more classes now implement a :term:`context manager` to support
675convenient and reliable resource clean-up using a :keyword:`with` statement.
676
677email
678-----
679
680The usability of the :mod:`email` package in Python 3 has been mostly fixed by
681the extensive efforts of R. David Murray.  The problem was that emails are
682typically read and stored in the form of :class:`bytes` rather than :class:`str`
683text, and they may contain multiple encodings within a single email.  So, the
684email package had to be extended to parse and generate email messages in bytes
685format.
686
687* New functions :func:`~email.message_from_bytes` and
688  :func:`~email.message_from_binary_file`, and new classes
689  :class:`~email.parser.BytesFeedParser` and :class:`~email.parser.BytesParser`
690  allow binary message data to be parsed into model objects.
691
692* Given bytes input to the model, :meth:`~email.message.Message.get_payload`
693  will by default decode a message body that has a
694  :mailheader:`Content-Transfer-Encoding` of *8bit* using the charset
695  specified in the MIME headers and return the resulting string.
696
697* Given bytes input to the model, :class:`~email.generator.Generator` will
698  convert message bodies that have a :mailheader:`Content-Transfer-Encoding` of
699  *8bit* to instead have a *7bit* :mailheader:`Content-Transfer-Encoding`.
700
701  Headers with unencoded non-ASCII bytes are deemed to be :rfc:`2047`\ -encoded
702  using the *unknown-8bit* character set.
703
704* A new class :class:`~email.generator.BytesGenerator` produces bytes as output,
705  preserving any unchanged non-ASCII data that was present in the input used to
706  build the model, including message bodies with a
707  :mailheader:`Content-Transfer-Encoding` of *8bit*.
708
709* The :mod:`smtplib` :class:`~smtplib.SMTP` class now accepts a byte string
710  for the *msg* argument to the :meth:`~smtplib.SMTP.sendmail` method,
711  and a new method, :meth:`~smtplib.SMTP.send_message` accepts a
712  :class:`~email.message.Message` object and can optionally obtain the
713  *from_addr* and *to_addrs* addresses directly from the object.
714
715(Proposed and implemented by R. David Murray, :issue:`4661` and :issue:`10321`.)
716
717elementtree
718-----------
719
720The :mod:`xml.etree.ElementTree` package and its :mod:`xml.etree.cElementTree`
721counterpart have been updated to version 1.3.
722
723Several new and useful functions and methods have been added:
724
725* :func:`xml.etree.ElementTree.fromstringlist` which builds an XML document
726  from a sequence of fragments
727* :func:`xml.etree.ElementTree.register_namespace` for registering a global
728  namespace prefix
729* :func:`xml.etree.ElementTree.tostringlist` for string representation
730  including all sublists
731* :meth:`xml.etree.ElementTree.Element.extend` for appending a sequence of zero
732  or more elements
733* :meth:`xml.etree.ElementTree.Element.iterfind` searches an element and
734  subelements
735* :meth:`xml.etree.ElementTree.Element.itertext` creates a text iterator over
736  an element and its subelements
737* :meth:`xml.etree.ElementTree.TreeBuilder.end` closes the current element
738* :meth:`xml.etree.ElementTree.TreeBuilder.doctype` handles a doctype
739  declaration
740
741Two methods have been deprecated:
742
743* :meth:`xml.etree.ElementTree.getchildren` use ``list(elem)`` instead.
744* :meth:`xml.etree.ElementTree.getiterator` use ``Element.iter`` instead.
745
746For details of the update, see `Introducing ElementTree
747<http://effbot.org/zone/elementtree-13-intro.htm>`_ on Fredrik Lundh's website.
748
749(Contributed by Florent Xicluna and Fredrik Lundh, :issue:`6472`.)
750
751functools
752---------
753
754* The :mod:`functools` module includes a new decorator for caching function
755  calls.  :func:`functools.lru_cache` can save repeated queries to an external
756  resource whenever the results are expected to be the same.
757
758  For example, adding a caching decorator to a database query function can save
759  database accesses for popular searches:
760
761  >>> import functools
762  >>> @functools.lru_cache(maxsize=300)
763  ... def get_phone_number(name):
764  ...     c = conn.cursor()
765  ...     c.execute('SELECT phonenumber FROM phonelist WHERE name=?', (name,))
766  ...     return c.fetchone()[0]
767
768  >>> for name in user_requests:        # doctest: +SKIP
769  ...     get_phone_number(name)        # cached lookup
770
771  To help with choosing an effective cache size, the wrapped function is
772  instrumented for tracking cache statistics:
773
774  >>> get_phone_number.cache_info()     # doctest: +SKIP
775  CacheInfo(hits=4805, misses=980, maxsize=300, currsize=300)
776
777  If the phonelist table gets updated, the outdated contents of the cache can be
778  cleared with:
779
780  >>> get_phone_number.cache_clear()
781
782  (Contributed by Raymond Hettinger and incorporating design ideas from Jim
783  Baker, Miki Tebeka, and Nick Coghlan; see `recipe 498245
784  <https://code.activestate.com/recipes/498245>`_\, `recipe 577479
785  <https://code.activestate.com/recipes/577479>`_\, :issue:`10586`, and
786  :issue:`10593`.)
787
788* The :func:`functools.wraps` decorator now adds a :attr:`__wrapped__` attribute
789  pointing to the original callable function.  This allows wrapped functions to
790  be introspected.  It also copies :attr:`__annotations__` if defined.  And now
791  it also gracefully skips over missing attributes such as :attr:`__doc__` which
792  might not be defined for the wrapped callable.
793
794  In the above example, the cache can be removed by recovering the original
795  function:
796
797  >>> get_phone_number = get_phone_number.__wrapped__    # uncached function
798
799  (By Nick Coghlan and Terrence Cole; :issue:`9567`, :issue:`3445`, and
800  :issue:`8814`.)
801
802* To help write classes with rich comparison methods, a new decorator
803  :func:`functools.total_ordering` will use existing equality and inequality
804  methods to fill in the remaining methods.
805
806  For example, supplying *__eq__* and *__lt__* will enable
807  :func:`~functools.total_ordering` to fill-in *__le__*, *__gt__* and *__ge__*::
808
809    @total_ordering
810    class Student:
811        def __eq__(self, other):
812            return ((self.lastname.lower(), self.firstname.lower()) ==
813                    (other.lastname.lower(), other.firstname.lower()))
814
815        def __lt__(self, other):
816            return ((self.lastname.lower(), self.firstname.lower()) <
817                    (other.lastname.lower(), other.firstname.lower()))
818
819  With the *total_ordering* decorator, the remaining comparison methods
820  are filled in automatically.
821
822  (Contributed by Raymond Hettinger.)
823
824* To aid in porting programs from Python 2, the :func:`functools.cmp_to_key`
825  function converts an old-style comparison function to
826  modern :term:`key function`:
827
828  >>> # locale-aware sort order
829  >>> sorted(iterable, key=cmp_to_key(locale.strcoll)) # doctest: +SKIP
830
831  For sorting examples and a brief sorting tutorial, see the `Sorting HowTo
832  <https://wiki.python.org/moin/HowTo/Sorting/>`_ tutorial.
833
834  (Contributed by Raymond Hettinger.)
835
836itertools
837---------
838
839* The :mod:`itertools` module has a new :func:`~itertools.accumulate` function
840  modeled on APL's *scan* operator and Numpy's *accumulate* function:
841
842  >>> from itertools import accumulate
843  >>> list(accumulate([8, 2, 50]))
844  [8, 10, 60]
845
846  >>> prob_dist = [0.1, 0.4, 0.2, 0.3]
847  >>> list(accumulate(prob_dist))      # cumulative probability distribution
848  [0.1, 0.5, 0.7, 1.0]
849
850  For an example using :func:`~itertools.accumulate`, see the :ref:`examples for
851  the random module <random-examples>`.
852
853  (Contributed by Raymond Hettinger and incorporating design suggestions
854  from Mark Dickinson.)
855
856collections
857-----------
858
859* The :class:`collections.Counter` class now has two forms of in-place
860  subtraction, the existing *-=* operator for `saturating subtraction
861  <https://en.wikipedia.org/wiki/Saturation_arithmetic>`_ and the new
862  :meth:`~collections.Counter.subtract` method for regular subtraction.  The
863  former is suitable for `multisets <https://en.wikipedia.org/wiki/Multiset>`_
864  which only have positive counts, and the latter is more suitable for use cases
865  that allow negative counts:
866
867  >>> from collections import Counter
868  >>> tally = Counter(dogs=5, cats=3)
869  >>> tally -= Counter(dogs=2, cats=8)    # saturating subtraction
870  >>> tally
871  Counter({'dogs': 3})
872
873  >>> tally = Counter(dogs=5, cats=3)
874  >>> tally.subtract(dogs=2, cats=8)      # regular subtraction
875  >>> tally
876  Counter({'dogs': 3, 'cats': -5})
877
878  (Contributed by Raymond Hettinger.)
879
880* The :class:`collections.OrderedDict` class has a new method
881  :meth:`~collections.OrderedDict.move_to_end` which takes an existing key and
882  moves it to either the first or last position in the ordered sequence.
883
884  The default is to move an item to the last position.  This is equivalent of
885  renewing an entry with ``od[k] = od.pop(k)``.
886
887  A fast move-to-end operation is useful for resequencing entries.  For example,
888  an ordered dictionary can be used to track order of access by aging entries
889  from the oldest to the most recently accessed.
890
891  >>> from collections import OrderedDict
892  >>> d = OrderedDict.fromkeys(['a', 'b', 'X', 'd', 'e'])
893  >>> list(d)
894  ['a', 'b', 'X', 'd', 'e']
895  >>> d.move_to_end('X')
896  >>> list(d)
897  ['a', 'b', 'd', 'e', 'X']
898
899  (Contributed by Raymond Hettinger.)
900
901* The :class:`collections.deque` class grew two new methods
902  :meth:`~collections.deque.count` and :meth:`~collections.deque.reverse` that
903  make them more substitutable for :class:`list` objects:
904
905  >>> from collections import deque
906  >>> d = deque('simsalabim')
907  >>> d.count('s')
908  2
909  >>> d.reverse()
910  >>> d
911  deque(['m', 'i', 'b', 'a', 'l', 'a', 's', 'm', 'i', 's'])
912
913  (Contributed by Raymond Hettinger.)
914
915threading
916---------
917
918The :mod:`threading` module has a new :class:`~threading.Barrier`
919synchronization class for making multiple threads wait until all of them have
920reached a common barrier point.  Barriers are useful for making sure that a task
921with multiple preconditions does not run until all of the predecessor tasks are
922complete.
923
924Barriers can work with an arbitrary number of threads.  This is a generalization
925of a `Rendezvous <https://en.wikipedia.org/wiki/Synchronous_rendezvous>`_ which
926is defined for only two threads.
927
928Implemented as a two-phase cyclic barrier, :class:`~threading.Barrier` objects
929are suitable for use in loops.  The separate *filling* and *draining* phases
930assure that all threads get released (drained) before any one of them can loop
931back and re-enter the barrier.  The barrier fully resets after each cycle.
932
933Example of using barriers::
934
935    from threading import Barrier, Thread
936
937    def get_votes(site):
938        ballots = conduct_election(site)
939        all_polls_closed.wait()        # do not count until all polls are closed
940        totals = summarize(ballots)
941        publish(site, totals)
942
943    all_polls_closed = Barrier(len(sites))
944    for site in sites:
945        Thread(target=get_votes, args=(site,)).start()
946
947In this example, the barrier enforces a rule that votes cannot be counted at any
948polling site until all polls are closed.  Notice how a solution with a barrier
949is similar to one with :meth:`threading.Thread.join`, but the threads stay alive
950and continue to do work (summarizing ballots) after the barrier point is
951crossed.
952
953If any of the predecessor tasks can hang or be delayed, a barrier can be created
954with an optional *timeout* parameter.  Then if the timeout period elapses before
955all the predecessor tasks reach the barrier point, all waiting threads are
956released and a :exc:`~threading.BrokenBarrierError` exception is raised::
957
958    def get_votes(site):
959        ballots = conduct_election(site)
960        try:
961            all_polls_closed.wait(timeout=midnight - time.now())
962        except BrokenBarrierError:
963            lockbox = seal_ballots(ballots)
964            queue.put(lockbox)
965        else:
966            totals = summarize(ballots)
967            publish(site, totals)
968
969In this example, the barrier enforces a more robust rule.  If some election
970sites do not finish before midnight, the barrier times-out and the ballots are
971sealed and deposited in a queue for later handling.
972
973See `Barrier Synchronization Patterns
974<http://osl.cs.illinois.edu/media/papers/karmani-2009-barrier_synchronization_pattern.pdf>`_
975for more examples of how barriers can be used in parallel computing.  Also, there is
976a simple but thorough explanation of barriers in `The Little Book of Semaphores
977<https://greenteapress.com/semaphores/LittleBookOfSemaphores.pdf>`_, *section 3.6*.
978
979(Contributed by Kristján Valur Jónsson with an API review by Jeffrey Yasskin in
980:issue:`8777`.)
981
982datetime and time
983-----------------
984
985* The :mod:`datetime` module has a new type :class:`~datetime.timezone` that
986  implements the :class:`~datetime.tzinfo` interface by returning a fixed UTC
987  offset and timezone name. This makes it easier to create timezone-aware
988  datetime objects::
989
990    >>> from datetime import datetime, timezone
991
992    >>> datetime.now(timezone.utc)
993    datetime.datetime(2010, 12, 8, 21, 4, 2, 923754, tzinfo=datetime.timezone.utc)
994
995    >>> datetime.strptime("01/01/2000 12:00 +0000", "%m/%d/%Y %H:%M %z")
996    datetime.datetime(2000, 1, 1, 12, 0, tzinfo=datetime.timezone.utc)
997
998* Also, :class:`~datetime.timedelta` objects can now be multiplied by
999  :class:`float` and divided by :class:`float` and :class:`int` objects.
1000  And :class:`~datetime.timedelta` objects can now divide one another.
1001
1002* The :meth:`datetime.date.strftime` method is no longer restricted to years
1003  after 1900.  The new supported year range is from 1000 to 9999 inclusive.
1004
1005* Whenever a two-digit year is used in a time tuple, the interpretation has been
1006  governed by :attr:`time.accept2dyear`.  The default is ``True`` which means that
1007  for a two-digit year, the century is guessed according to the POSIX rules
1008  governing the ``%y`` strptime format.
1009
1010  Starting with Py3.2, use of the century guessing heuristic will emit a
1011  :exc:`DeprecationWarning`.  Instead, it is recommended that
1012  :attr:`time.accept2dyear` be set to ``False`` so that large date ranges
1013  can be used without guesswork::
1014
1015    >>> import time, warnings
1016    >>> warnings.resetwarnings()      # remove the default warning filters
1017
1018    >>> time.accept2dyear = True      # guess whether 11 means 11 or 2011
1019    >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
1020    Warning (from warnings module):
1021      ...
1022    DeprecationWarning: Century info guessed for a 2-digit year.
1023    'Fri Jan  1 12:34:56 2011'
1024
1025    >>> time.accept2dyear = False     # use the full range of allowable dates
1026    >>> time.asctime((11, 1, 1, 12, 34, 56, 4, 1, 0))
1027    'Fri Jan  1 12:34:56 11'
1028
1029  Several functions now have significantly expanded date ranges.  When
1030  :attr:`time.accept2dyear` is false, the :func:`time.asctime` function will
1031  accept any year that fits in a C int, while the :func:`time.mktime` and
1032  :func:`time.strftime` functions will accept the full range supported by the
1033  corresponding operating system functions.
1034
1035(Contributed by Alexander Belopolsky and Victor Stinner in :issue:`1289118`,
1036:issue:`5094`, :issue:`6641`, :issue:`2706`, :issue:`1777412`, :issue:`8013`,
1037and :issue:`10827`.)
1038
1039.. XXX https://bugs.python.org/issue?%40search_text=datetime&%40sort=-activity
1040
1041math
1042----
1043
1044The :mod:`math` module has been updated with six new functions inspired by the
1045C99 standard.
1046
1047The :func:`~math.isfinite` function provides a reliable and fast way to detect
1048special values.  It returns ``True`` for regular numbers and ``False`` for *Nan* or
1049*Infinity*:
1050
1051>>> from math import isfinite
1052>>> [isfinite(x) for x in (123, 4.56, float('Nan'), float('Inf'))]
1053[True, True, False, False]
1054
1055The :func:`~math.expm1` function computes ``e**x-1`` for small values of *x*
1056without incurring the loss of precision that usually accompanies the subtraction
1057of nearly equal quantities:
1058
1059>>> from math import expm1
1060>>> expm1(0.013671875)   # more accurate way to compute e**x-1 for a small x
10610.013765762467652909
1062
1063The :func:`~math.erf` function computes a probability integral or `Gaussian
1064error function <https://en.wikipedia.org/wiki/Error_function>`_.  The
1065complementary error function, :func:`~math.erfc`, is ``1 - erf(x)``:
1066
1067.. doctest::
1068   :options: +SKIP
1069
1070   >>> from math import erf, erfc, sqrt
1071   >>> erf(1.0/sqrt(2.0))   # portion of normal distribution within 1 standard deviation
1072   0.682689492137086
1073   >>> erfc(1.0/sqrt(2.0))  # portion of normal distribution outside 1 standard deviation
1074   0.31731050786291404
1075   >>> erf(1.0/sqrt(2.0)) + erfc(1.0/sqrt(2.0))
1076   1.0
1077
1078The :func:`~math.gamma` function is a continuous extension of the factorial
1079function.  See https://en.wikipedia.org/wiki/Gamma_function for details.  Because
1080the function is related to factorials, it grows large even for small values of
1081*x*, so there is also a :func:`~math.lgamma` function for computing the natural
1082logarithm of the gamma function:
1083
1084>>> from math import gamma, lgamma
1085>>> gamma(7.0)           # six factorial
1086720.0
1087>>> lgamma(801.0)        # log(800 factorial)
10884551.950730698041
1089
1090(Contributed by Mark Dickinson.)
1091
1092abc
1093---
1094
1095The :mod:`abc` module now supports :func:`~abc.abstractclassmethod` and
1096:func:`~abc.abstractstaticmethod`.
1097
1098These tools make it possible to define an :term:`abstract base class` that
1099requires a particular :func:`classmethod` or :func:`staticmethod` to be
1100implemented::
1101
1102    class Temperature(metaclass=abc.ABCMeta):
1103        @abc.abstractclassmethod
1104        def from_fahrenheit(cls, t):
1105            ...
1106        @abc.abstractclassmethod
1107        def from_celsius(cls, t):
1108            ...
1109
1110(Patch submitted by Daniel Urban; :issue:`5867`.)
1111
1112io
1113--
1114
1115The :class:`io.BytesIO` has a new method, :meth:`~io.BytesIO.getbuffer`, which
1116provides functionality similar to :func:`memoryview`.  It creates an editable
1117view of the data without making a copy.  The buffer's random access and support
1118for slice notation are well-suited to in-place editing::
1119
1120    >>> REC_LEN, LOC_START, LOC_LEN = 34, 7, 11
1121
1122    >>> def change_location(buffer, record_number, location):
1123    ...     start = record_number * REC_LEN + LOC_START
1124    ...     buffer[start: start+LOC_LEN] = location
1125
1126    >>> import io
1127
1128    >>> byte_stream = io.BytesIO(
1129    ...     b'G3805  storeroom  Main chassis    '
1130    ...     b'X7899  shipping   Reserve cog     '
1131    ...     b'L6988  receiving  Primary sprocket'
1132    ... )
1133    >>> buffer = byte_stream.getbuffer()
1134    >>> change_location(buffer, 1, b'warehouse  ')
1135    >>> change_location(buffer, 0, b'showroom   ')
1136    >>> print(byte_stream.getvalue())
1137    b'G3805  showroom   Main chassis    '
1138    b'X7899  warehouse  Reserve cog     '
1139    b'L6988  receiving  Primary sprocket'
1140
1141(Contributed by Antoine Pitrou in :issue:`5506`.)
1142
1143reprlib
1144-------
1145
1146When writing a :meth:`__repr__` method for a custom container, it is easy to
1147forget to handle the case where a member refers back to the container itself.
1148Python's builtin objects such as :class:`list` and :class:`set` handle
1149self-reference by displaying "..." in the recursive part of the representation
1150string.
1151
1152To help write such :meth:`__repr__` methods, the :mod:`reprlib` module has a new
1153decorator, :func:`~reprlib.recursive_repr`, for detecting recursive calls to
1154:meth:`__repr__` and substituting a placeholder string instead::
1155
1156        >>> class MyList(list):
1157        ...     @recursive_repr()
1158        ...     def __repr__(self):
1159        ...         return '<' + '|'.join(map(repr, self)) + '>'
1160        ...
1161        >>> m = MyList('abc')
1162        >>> m.append(m)
1163        >>> m.append('x')
1164        >>> print(m)
1165        <'a'|'b'|'c'|...|'x'>
1166
1167(Contributed by Raymond Hettinger in :issue:`9826` and :issue:`9840`.)
1168
1169logging
1170-------
1171
1172In addition to dictionary-based configuration described above, the
1173:mod:`logging` package has many other improvements.
1174
1175The logging documentation has been augmented by a :ref:`basic tutorial
1176<logging-basic-tutorial>`\, an :ref:`advanced tutorial
1177<logging-advanced-tutorial>`\, and a :ref:`cookbook <logging-cookbook>` of
1178logging recipes.  These documents are the fastest way to learn about logging.
1179
1180The :func:`logging.basicConfig` set-up function gained a *style* argument to
1181support three different types of string formatting.  It defaults to "%" for
1182traditional %-formatting, can be set to "{" for the new :meth:`str.format` style, or
1183can be set to "$" for the shell-style formatting provided by
1184:class:`string.Template`.  The following three configurations are equivalent::
1185
1186    >>> from logging import basicConfig
1187    >>> basicConfig(style='%', format="%(name)s -> %(levelname)s: %(message)s")
1188    >>> basicConfig(style='{', format="{name} -> {levelname} {message}")
1189    >>> basicConfig(style='$', format="$name -> $levelname: $message")
1190
1191If no configuration is set-up before a logging event occurs, there is now a
1192default configuration using a :class:`~logging.StreamHandler` directed to
1193:attr:`sys.stderr` for events of ``WARNING`` level or higher.  Formerly, an
1194event occurring before a configuration was set-up would either raise an
1195exception or silently drop the event depending on the value of
1196:attr:`logging.raiseExceptions`.  The new default handler is stored in
1197:attr:`logging.lastResort`.
1198
1199The use of filters has been simplified.  Instead of creating a
1200:class:`~logging.Filter` object, the predicate can be any Python callable that
1201returns ``True`` or ``False``.
1202
1203There were a number of other improvements that add flexibility and simplify
1204configuration.  See the module documentation for a full listing of changes in
1205Python 3.2.
1206
1207csv
1208---
1209
1210The :mod:`csv` module now supports a new dialect, :class:`~csv.unix_dialect`,
1211which applies quoting for all fields and a traditional Unix style with ``'\n'`` as
1212the line terminator.  The registered dialect name is ``unix``.
1213
1214The :class:`csv.DictWriter` has a new method,
1215:meth:`~csv.DictWriter.writeheader` for writing-out an initial row to document
1216the field names::
1217
1218    >>> import csv, sys
1219    >>> w = csv.DictWriter(sys.stdout, ['name', 'dept'], dialect='unix')
1220    >>> w.writeheader()
1221    "name","dept"
1222    >>> w.writerows([
1223    ...     {'name': 'tom', 'dept': 'accounting'},
1224    ...     {'name': 'susan', 'dept': 'Salesl'}])
1225    "tom","accounting"
1226    "susan","sales"
1227
1228(New dialect suggested by Jay Talbot in :issue:`5975`, and the new method
1229suggested by Ed Abraham in :issue:`1537721`.)
1230
1231contextlib
1232----------
1233
1234There is a new and slightly mind-blowing tool
1235:class:`~contextlib.ContextDecorator` that is helpful for creating a
1236:term:`context manager` that does double duty as a function decorator.
1237
1238As a convenience, this new functionality is used by
1239:func:`~contextlib.contextmanager` so that no extra effort is needed to support
1240both roles.
1241
1242The basic idea is that both context managers and function decorators can be used
1243for pre-action and post-action wrappers.  Context managers wrap a group of
1244statements using a :keyword:`with` statement, and function decorators wrap a
1245group of statements enclosed in a function.  So, occasionally there is a need to
1246write a pre-action or post-action wrapper that can be used in either role.
1247
1248For example, it is sometimes useful to wrap functions or groups of statements
1249with a logger that can track the time of entry and time of exit.  Rather than
1250writing both a function decorator and a context manager for the task, the
1251:func:`~contextlib.contextmanager` provides both capabilities in a single
1252definition::
1253
1254    from contextlib import contextmanager
1255    import logging
1256
1257    logging.basicConfig(level=logging.INFO)
1258
1259    @contextmanager
1260    def track_entry_and_exit(name):
1261        logging.info('Entering: %s', name)
1262        yield
1263        logging.info('Exiting: %s', name)
1264
1265Formerly, this would have only been usable as a context manager::
1266
1267    with track_entry_and_exit('widget loader'):
1268        print('Some time consuming activity goes here')
1269        load_widget()
1270
1271Now, it can be used as a decorator as well::
1272
1273    @track_entry_and_exit('widget loader')
1274    def activity():
1275        print('Some time consuming activity goes here')
1276        load_widget()
1277
1278Trying to fulfill two roles at once places some limitations on the technique.
1279Context managers normally have the flexibility to return an argument usable by
1280a :keyword:`with` statement, but there is no parallel for function decorators.
1281
1282In the above example, there is not a clean way for the *track_entry_and_exit*
1283context manager to return a logging instance for use in the body of enclosed
1284statements.
1285
1286(Contributed by Michael Foord in :issue:`9110`.)
1287
1288decimal and fractions
1289---------------------
1290
1291Mark Dickinson crafted an elegant and efficient scheme for assuring that
1292different numeric datatypes will have the same hash value whenever their actual
1293values are equal (:issue:`8188`)::
1294
1295   assert hash(Fraction(3, 2)) == hash(1.5) == \
1296          hash(Decimal("1.5")) == hash(complex(1.5, 0))
1297
1298Some of the hashing details are exposed through a new attribute,
1299:attr:`sys.hash_info`, which describes the bit width of the hash value, the
1300prime modulus, the hash values for *infinity* and *nan*, and the multiplier
1301used for the imaginary part of a number:
1302
1303>>> sys.hash_info # doctest: +SKIP
1304sys.hash_info(width=64, modulus=2305843009213693951, inf=314159, nan=0, imag=1000003)
1305
1306An early decision to limit the inter-operability of various numeric types has
1307been relaxed.  It is still unsupported (and ill-advised) to have implicit
1308mixing in arithmetic expressions such as ``Decimal('1.1') + float('1.1')``
1309because the latter loses information in the process of constructing the binary
1310float.  However, since existing floating point value can be converted losslessly
1311to either a decimal or rational representation, it makes sense to add them to
1312the constructor and to support mixed-type comparisons.
1313
1314* The :class:`decimal.Decimal` constructor now accepts :class:`float` objects
1315  directly so there in no longer a need to use the :meth:`~decimal.Decimal.from_float`
1316  method (:issue:`8257`).
1317
1318* Mixed type comparisons are now fully supported so that
1319  :class:`~decimal.Decimal` objects can be directly compared with :class:`float`
1320  and :class:`fractions.Fraction` (:issue:`2531` and :issue:`8188`).
1321
1322Similar changes were made to :class:`fractions.Fraction` so that the
1323:meth:`~fractions.Fraction.from_float()` and :meth:`~fractions.Fraction.from_decimal`
1324methods are no longer needed (:issue:`8294`):
1325
1326>>> from decimal import Decimal
1327>>> from fractions import Fraction
1328>>> Decimal(1.1)
1329Decimal('1.100000000000000088817841970012523233890533447265625')
1330>>> Fraction(1.1)
1331Fraction(2476979795053773, 2251799813685248)
1332
1333Another useful change for the :mod:`decimal` module is that the
1334:attr:`Context.clamp` attribute is now public.  This is useful in creating
1335contexts that correspond to the decimal interchange formats specified in IEEE
1336754 (see :issue:`8540`).
1337
1338(Contributed by Mark Dickinson and Raymond Hettinger.)
1339
1340ftp
1341---
1342
1343The :class:`ftplib.FTP` class now supports the context management protocol to
1344unconditionally consume :exc:`socket.error` exceptions and to close the FTP
1345connection when done::
1346
1347 >>> from ftplib import FTP
1348 >>> with FTP("ftp1.at.proftpd.org") as ftp:
1349         ftp.login()
1350         ftp.dir()
1351
1352 '230 Anonymous login ok, restrictions apply.'
1353 dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 .
1354 dr-xr-xr-x   9 ftp      ftp           154 May  6 10:43 ..
1355 dr-xr-xr-x   5 ftp      ftp          4096 May  6 10:43 CentOS
1356 dr-xr-xr-x   3 ftp      ftp            18 Jul 10  2008 Fedora
1357
1358Other file-like objects such as :class:`mmap.mmap` and :func:`fileinput.input`
1359also grew auto-closing context managers::
1360
1361    with fileinput.input(files=('log1.txt', 'log2.txt')) as f:
1362        for line in f:
1363            process(line)
1364
1365(Contributed by Tarek Ziadé and Giampaolo Rodolà in :issue:`4972`, and
1366by Georg Brandl in :issue:`8046` and :issue:`1286`.)
1367
1368The :class:`~ftplib.FTP_TLS` class now accepts a *context* parameter, which is a
1369:class:`ssl.SSLContext` object allowing bundling SSL configuration options,
1370certificates and private keys into a single (potentially long-lived) structure.
1371
1372(Contributed by Giampaolo Rodolà; :issue:`8806`.)
1373
1374popen
1375-----
1376
1377The :func:`os.popen` and :func:`subprocess.Popen` functions now support
1378:keyword:`with` statements for auto-closing of the file descriptors.
1379
1380(Contributed by Antoine Pitrou and Brian Curtin in :issue:`7461` and
1381:issue:`10554`.)
1382
1383select
1384------
1385
1386The :mod:`select` module now exposes a new, constant attribute,
1387:attr:`~select.PIPE_BUF`, which gives the minimum number of bytes which are
1388guaranteed not to block when :func:`select.select` says a pipe is ready
1389for writing.
1390
1391>>> import select
1392>>> select.PIPE_BUF  # doctest: +SKIP
1393512
1394
1395(Available on Unix systems. Patch by Sébastien Sablé in :issue:`9862`)
1396
1397gzip and zipfile
1398----------------
1399
1400:class:`gzip.GzipFile` now implements the :class:`io.BufferedIOBase`
1401:term:`abstract base class` (except for ``truncate()``).  It also has a
1402:meth:`~gzip.GzipFile.peek` method and supports unseekable as well as
1403zero-padded file objects.
1404
1405The :mod:`gzip` module also gains the :func:`~gzip.compress` and
1406:func:`~gzip.decompress` functions for easier in-memory compression and
1407decompression.  Keep in mind that text needs to be encoded as :class:`bytes`
1408before compressing and decompressing:
1409
1410>>> import gzip
1411>>> s = 'Three shall be the number thou shalt count, '
1412>>> s += 'and the number of the counting shall be three'
1413>>> b = s.encode()                        # convert to utf-8
1414>>> len(b)
141589
1416>>> c = gzip.compress(b)
1417>>> len(c)
141877
1419>>> gzip.decompress(c).decode()[:42]      # decompress and convert to text
1420'Three shall be the number thou shalt count'
1421
1422(Contributed by Anand B. Pillai in :issue:`3488`; and by Antoine Pitrou, Nir
1423Aides and Brian Curtin in :issue:`9962`, :issue:`1675951`, :issue:`7471` and
1424:issue:`2846`.)
1425
1426Also, the :class:`zipfile.ZipExtFile` class was reworked internally to represent
1427files stored inside an archive.  The new implementation is significantly faster
1428and can be wrapped in an :class:`io.BufferedReader` object for more speedups.  It
1429also solves an issue where interleaved calls to *read* and *readline* gave the
1430wrong results.
1431
1432(Patch submitted by Nir Aides in :issue:`7610`.)
1433
1434tarfile
1435-------
1436
1437The :class:`~tarfile.TarFile` class can now be used as a context manager.  In
1438addition, its :meth:`~tarfile.TarFile.add` method has a new option, *filter*,
1439that controls which files are added to the archive and allows the file metadata
1440to be edited.
1441
1442The new *filter* option replaces the older, less flexible *exclude* parameter
1443which is now deprecated.  If specified, the optional *filter* parameter needs to
1444be a :term:`keyword argument`.  The user-supplied filter function accepts a
1445:class:`~tarfile.TarInfo` object and returns an updated
1446:class:`~tarfile.TarInfo` object, or if it wants the file to be excluded, the
1447function can return ``None``::
1448
1449    >>> import tarfile, glob
1450
1451    >>> def myfilter(tarinfo):
1452    ...     if tarinfo.isfile():             # only save real files
1453    ...         tarinfo.uname = 'monty'      # redact the user name
1454    ...         return tarinfo
1455
1456    >>> with tarfile.open(name='myarchive.tar.gz', mode='w:gz') as tf:
1457    ...     for filename in glob.glob('*.txt'):
1458    ...         tf.add(filename, filter=myfilter)
1459    ...     tf.list()
1460    -rw-r--r-- monty/501        902 2011-01-26 17:59:11 annotations.txt
1461    -rw-r--r-- monty/501        123 2011-01-26 17:59:11 general_questions.txt
1462    -rw-r--r-- monty/501       3514 2011-01-26 17:59:11 prion.txt
1463    -rw-r--r-- monty/501        124 2011-01-26 17:59:11 py_todo.txt
1464    -rw-r--r-- monty/501       1399 2011-01-26 17:59:11 semaphore_notes.txt
1465
1466(Proposed by Tarek Ziadé and implemented by Lars Gustäbel in :issue:`6856`.)
1467
1468hashlib
1469-------
1470
1471The :mod:`hashlib` module has two new constant attributes listing the hashing
1472algorithms guaranteed to be present in all implementations and those available
1473on the current implementation::
1474
1475    >>> import hashlib
1476
1477    >>> hashlib.algorithms_guaranteed
1478    {'sha1', 'sha224', 'sha384', 'sha256', 'sha512', 'md5'}
1479
1480    >>> hashlib.algorithms_available
1481    {'md2', 'SHA256', 'SHA512', 'dsaWithSHA', 'mdc2', 'SHA224', 'MD4', 'sha256',
1482    'sha512', 'ripemd160', 'SHA1', 'MDC2', 'SHA', 'SHA384', 'MD2',
1483    'ecdsa-with-SHA1','md4', 'md5', 'sha1', 'DSA-SHA', 'sha224',
1484    'dsaEncryption', 'DSA', 'RIPEMD160', 'sha', 'MD5', 'sha384'}
1485
1486(Suggested by Carl Chenet in :issue:`7418`.)
1487
1488ast
1489---
1490
1491The :mod:`ast` module has a wonderful a general-purpose tool for safely
1492evaluating expression strings using the Python literal
1493syntax.  The :func:`ast.literal_eval` function serves as a secure alternative to
1494the builtin :func:`eval` function which is easily abused.  Python 3.2 adds
1495:class:`bytes` and :class:`set` literals to the list of supported types:
1496strings, bytes, numbers, tuples, lists, dicts, sets, booleans, and ``None``.
1497
1498::
1499
1500    >>> from ast import literal_eval
1501
1502    >>> request = "{'req': 3, 'func': 'pow', 'args': (2, 0.5)}"
1503    >>> literal_eval(request)
1504    {'args': (2, 0.5), 'req': 3, 'func': 'pow'}
1505
1506    >>> request = "os.system('do something harmful')"
1507    >>> literal_eval(request)
1508    Traceback (most recent call last):
1509      ...
1510    ValueError: malformed node or string: <_ast.Call object at 0x101739a10>
1511
1512(Implemented by Benjamin Peterson and Georg Brandl.)
1513
1514os
1515--
1516
1517Different operating systems use various encodings for filenames and environment
1518variables.  The :mod:`os` module provides two new functions,
1519:func:`~os.fsencode` and :func:`~os.fsdecode`, for encoding and decoding
1520filenames:
1521
1522>>> import os
1523>>> filename = 'Sehenswürdigkeiten'
1524>>> os.fsencode(filename)
1525b'Sehensw\xc3\xbcrdigkeiten'
1526
1527Some operating systems allow direct access to encoded bytes in the
1528environment.  If so, the :attr:`os.supports_bytes_environ` constant will be
1529true.
1530
1531For direct access to encoded environment variables (if available),
1532use the new :func:`os.getenvb` function or use :data:`os.environb`
1533which is a bytes version of :data:`os.environ`.
1534
1535(Contributed by Victor Stinner.)
1536
1537shutil
1538------
1539
1540The :func:`shutil.copytree` function has two new options:
1541
1542* *ignore_dangling_symlinks*: when ``symlinks=False`` so that the function
1543  copies a file pointed to by a symlink, not the symlink itself. This option
1544  will silence the error raised if the file doesn't exist.
1545
1546* *copy_function*: is a callable that will be used to copy files.
1547  :func:`shutil.copy2` is used by default.
1548
1549(Contributed by Tarek Ziadé.)
1550
1551In addition, the :mod:`shutil` module now supports :ref:`archiving operations
1552<archiving-operations>` for zipfiles, uncompressed tarfiles, gzipped tarfiles,
1553and bzipped tarfiles.  And there are functions for registering additional
1554archiving file formats (such as xz compressed tarfiles or custom formats).
1555
1556The principal functions are :func:`~shutil.make_archive` and
1557:func:`~shutil.unpack_archive`.  By default, both operate on the current
1558directory (which can be set by :func:`os.chdir`) and on any sub-directories.
1559The archive filename needs to be specified with a full pathname.  The archiving
1560step is non-destructive (the original files are left unchanged).
1561
1562::
1563
1564    >>> import shutil, pprint
1565
1566    >>> os.chdir('mydata')  # change to the source directory
1567    >>> f = shutil.make_archive('/var/backup/mydata',
1568    ...                         'zip')      # archive the current directory
1569    >>> f                                   # show the name of archive
1570    '/var/backup/mydata.zip'
1571    >>> os.chdir('tmp')                     # change to an unpacking
1572    >>> shutil.unpack_archive('/var/backup/mydata.zip')  # recover the data
1573
1574    >>> pprint.pprint(shutil.get_archive_formats())  # display known formats
1575    [('bztar', "bzip2'ed tar-file"),
1576     ('gztar', "gzip'ed tar-file"),
1577     ('tar', 'uncompressed tar file'),
1578     ('zip', 'ZIP file')]
1579
1580    >>> shutil.register_archive_format(     # register a new archive format
1581    ...     name='xz',
1582    ...     function=xz.compress,           # callable archiving function
1583    ...     extra_args=[('level', 8)],      # arguments to the function
1584    ...     description='xz compression'
1585    ... )
1586
1587(Contributed by Tarek Ziadé.)
1588
1589sqlite3
1590-------
1591
1592The :mod:`sqlite3` module was updated to pysqlite version 2.6.0.  It has two new capabilities.
1593
1594* The :attr:`sqlite3.Connection.in_transit` attribute is true if there is an
1595  active transaction for uncommitted changes.
1596
1597* The :meth:`sqlite3.Connection.enable_load_extension` and
1598  :meth:`sqlite3.Connection.load_extension` methods allows you to load SQLite
1599  extensions from ".so" files.  One well-known extension is the fulltext-search
1600  extension distributed with SQLite.
1601
1602(Contributed by R. David Murray and Shashwat Anand; :issue:`8845`.)
1603
1604html
1605----
1606
1607A new :mod:`html` module was introduced with only a single function,
1608:func:`~html.escape`, which is used for escaping reserved characters from HTML
1609markup:
1610
1611>>> import html
1612>>> html.escape('x > 2 && x < 7')
1613'x &gt; 2 &amp;&amp; x &lt; 7'
1614
1615socket
1616------
1617
1618The :mod:`socket` module has two new improvements.
1619
1620* Socket objects now have a :meth:`~socket.socket.detach()` method which puts
1621  the socket into closed state without actually closing the underlying file
1622  descriptor.  The latter can then be reused for other purposes.
1623  (Added by Antoine Pitrou; :issue:`8524`.)
1624
1625* :func:`socket.create_connection` now supports the context management protocol
1626  to unconditionally consume :exc:`socket.error` exceptions and to close the
1627  socket when done.
1628  (Contributed by Giampaolo Rodolà; :issue:`9794`.)
1629
1630ssl
1631---
1632
1633The :mod:`ssl` module added a number of features to satisfy common requirements
1634for secure (encrypted, authenticated) internet connections:
1635
1636* A new class, :class:`~ssl.SSLContext`, serves as a container for persistent
1637  SSL data, such as protocol settings, certificates, private keys, and various
1638  other options. It includes a :meth:`~ssl.SSLContext.wrap_socket` for creating
1639  an SSL socket from an SSL context.
1640
1641* A new function, :func:`ssl.match_hostname`, supports server identity
1642  verification for higher-level protocols by implementing the rules of HTTPS
1643  (from :rfc:`2818`) which are also suitable for other protocols.
1644
1645* The :func:`ssl.wrap_socket` constructor function now takes a *ciphers*
1646  argument.  The *ciphers* string lists the allowed encryption algorithms using
1647  the format described in the `OpenSSL documentation
1648  <https://www.openssl.org/docs/manmaster/man1/ciphers.html#CIPHER-LIST-FORMAT>`__.
1649
1650* When linked against recent versions of OpenSSL, the :mod:`ssl` module now
1651  supports the Server Name Indication extension to the TLS protocol, allowing
1652  multiple "virtual hosts" using different certificates on a single IP port.
1653  This extension is only supported in client mode, and is activated by passing
1654  the *server_hostname* argument to :meth:`ssl.SSLContext.wrap_socket`.
1655
1656* Various options have been added to the :mod:`ssl` module, such as
1657  :data:`~ssl.OP_NO_SSLv2` which disables the insecure and obsolete SSLv2
1658  protocol.
1659
1660* The extension now loads all the OpenSSL ciphers and digest algorithms.  If
1661  some SSL certificates cannot be verified, they are reported as an "unknown
1662  algorithm" error.
1663
1664* The version of OpenSSL being used is now accessible using the module
1665  attributes :data:`ssl.OPENSSL_VERSION` (a string),
1666  :data:`ssl.OPENSSL_VERSION_INFO` (a 5-tuple), and
1667  :data:`ssl.OPENSSL_VERSION_NUMBER` (an integer).
1668
1669(Contributed by Antoine Pitrou in :issue:`8850`, :issue:`1589`, :issue:`8322`,
1670:issue:`5639`, :issue:`4870`, :issue:`8484`, and :issue:`8321`.)
1671
1672nntp
1673----
1674
1675The :mod:`nntplib` module has a revamped implementation with better bytes and
1676text semantics as well as more practical APIs.  These improvements break
1677compatibility with the nntplib version in Python 3.1, which was partly
1678dysfunctional in itself.
1679
1680Support for secure connections through both implicit (using
1681:class:`nntplib.NNTP_SSL`) and explicit (using :meth:`nntplib.NNTP.starttls`)
1682TLS has also been added.
1683
1684(Contributed by Antoine Pitrou in :issue:`9360` and Andrew Vant in :issue:`1926`.)
1685
1686certificates
1687------------
1688
1689:class:`http.client.HTTPSConnection`, :class:`urllib.request.HTTPSHandler`
1690and :func:`urllib.request.urlopen` now take optional arguments to allow for
1691server certificate checking against a set of Certificate Authorities,
1692as recommended in public uses of HTTPS.
1693
1694(Added by Antoine Pitrou, :issue:`9003`.)
1695
1696imaplib
1697-------
1698
1699Support for explicit TLS on standard IMAP4 connections has been added through
1700the new :mod:`imaplib.IMAP4.starttls` method.
1701
1702(Contributed by Lorenzo M. Catucci and Antoine Pitrou, :issue:`4471`.)
1703
1704http.client
1705-----------
1706
1707There were a number of small API improvements in the :mod:`http.client` module.
1708The old-style HTTP 0.9 simple responses are no longer supported and the *strict*
1709parameter is deprecated in all classes.
1710
1711The :class:`~http.client.HTTPConnection` and
1712:class:`~http.client.HTTPSConnection` classes now have a *source_address*
1713parameter for a (host, port) tuple indicating where the HTTP connection is made
1714from.
1715
1716Support for certificate checking and HTTPS virtual hosts were added to
1717:class:`~http.client.HTTPSConnection`.
1718
1719The :meth:`~http.client.HTTPConnection.request` method on connection objects
1720allowed an optional *body* argument so that a :term:`file object` could be used
1721to supply the content of the request.  Conveniently, the *body* argument now
1722also accepts an :term:`iterable` object so long as it includes an explicit
1723``Content-Length`` header.  This extended interface is much more flexible than
1724before.
1725
1726To establish an HTTPS connection through a proxy server, there is a new
1727:meth:`~http.client.HTTPConnection.set_tunnel` method that sets the host and
1728port for HTTP Connect tunneling.
1729
1730To match the behavior of :mod:`http.server`, the HTTP client library now also
1731encodes headers with ISO-8859-1 (Latin-1) encoding.  It was already doing that
1732for incoming headers, so now the behavior is consistent for both incoming and
1733outgoing traffic. (See work by Armin Ronacher in :issue:`10980`.)
1734
1735unittest
1736--------
1737
1738The unittest module has a number of improvements supporting test discovery for
1739packages, easier experimentation at the interactive prompt, new testcase
1740methods, improved diagnostic messages for test failures, and better method
1741names.
1742
1743* The command-line call ``python -m unittest`` can now accept file paths
1744  instead of module names for running specific tests (:issue:`10620`).  The new
1745  test discovery can find tests within packages, locating any test importable
1746  from the top-level directory.  The top-level directory can be specified with
1747  the `-t` option, a pattern for matching files with ``-p``, and a directory to
1748  start discovery with ``-s``:
1749
1750  .. code-block:: shell-session
1751
1752    $ python -m unittest discover -s my_proj_dir -p _test.py
1753
1754  (Contributed by Michael Foord.)
1755
1756* Experimentation at the interactive prompt is now easier because the
1757  :class:`unittest.case.TestCase` class can now be instantiated without
1758  arguments:
1759
1760  >>> from unittest import TestCase
1761  >>> TestCase().assertEqual(pow(2, 3), 8)
1762
1763  (Contributed by Michael Foord.)
1764
1765* The :mod:`unittest` module has two new methods,
1766  :meth:`~unittest.TestCase.assertWarns` and
1767  :meth:`~unittest.TestCase.assertWarnsRegex` to verify that a given warning type
1768  is triggered by the code under test::
1769
1770      with self.assertWarns(DeprecationWarning):
1771          legacy_function('XYZ')
1772
1773  (Contributed by Antoine Pitrou, :issue:`9754`.)
1774
1775  Another new method, :meth:`~unittest.TestCase.assertCountEqual` is used to
1776  compare two iterables to determine if their element counts are equal (whether
1777  the same elements are present with the same number of occurrences regardless
1778  of order)::
1779
1780     def test_anagram(self):
1781         self.assertCountEqual('algorithm', 'logarithm')
1782
1783  (Contributed by Raymond Hettinger.)
1784
1785* A principal feature of the unittest module is an effort to produce meaningful
1786  diagnostics when a test fails.  When possible, the failure is recorded along
1787  with a diff of the output.  This is especially helpful for analyzing log files
1788  of failed test runs. However, since diffs can sometime be voluminous, there is
1789  a new :attr:`~unittest.TestCase.maxDiff` attribute that sets maximum length of
1790  diffs displayed.
1791
1792* In addition, the method names in the module have undergone a number of clean-ups.
1793
1794  For example, :meth:`~unittest.TestCase.assertRegex` is the new name for
1795  :meth:`~unittest.TestCase.assertRegexpMatches` which was misnamed because the
1796  test uses :func:`re.search`, not :func:`re.match`.  Other methods using
1797  regular expressions are now named using short form "Regex" in preference to
1798  "Regexp" -- this matches the names used in other unittest implementations,
1799  matches Python's old name for the :mod:`re` module, and it has unambiguous
1800  camel-casing.
1801
1802  (Contributed by Raymond Hettinger and implemented by Ezio Melotti.)
1803
1804* To improve consistency, some long-standing method aliases are being
1805  deprecated in favor of the preferred names:
1806
1807   ===============================   ==============================
1808   Old Name                          Preferred Name
1809   ===============================   ==============================
1810   :meth:`assert_`                   :meth:`.assertTrue`
1811   :meth:`assertEquals`              :meth:`.assertEqual`
1812   :meth:`assertNotEquals`           :meth:`.assertNotEqual`
1813   :meth:`assertAlmostEquals`        :meth:`.assertAlmostEqual`
1814   :meth:`assertNotAlmostEquals`     :meth:`.assertNotAlmostEqual`
1815   ===============================   ==============================
1816
1817  Likewise, the ``TestCase.fail*`` methods deprecated in Python 3.1 are expected
1818  to be removed in Python 3.3.  Also see the :ref:`deprecated-aliases` section in
1819  the :mod:`unittest` documentation.
1820
1821  (Contributed by Ezio Melotti; :issue:`9424`.)
1822
1823* The :meth:`~unittest.TestCase.assertDictContainsSubset` method was deprecated
1824  because it was misimplemented with the arguments in the wrong order.  This
1825  created hard-to-debug optical illusions where tests like
1826  ``TestCase().assertDictContainsSubset({'a':1, 'b':2}, {'a':1})`` would fail.
1827
1828  (Contributed by Raymond Hettinger.)
1829
1830random
1831------
1832
1833The integer methods in the :mod:`random` module now do a better job of producing
1834uniform distributions.  Previously, they computed selections with
1835``int(n*random())`` which had a slight bias whenever *n* was not a power of two.
1836Now, multiple selections are made from a range up to the next power of two and a
1837selection is kept only when it falls within the range ``0 <= x < n``.  The
1838functions and methods affected are :func:`~random.randrange`,
1839:func:`~random.randint`, :func:`~random.choice`, :func:`~random.shuffle` and
1840:func:`~random.sample`.
1841
1842(Contributed by Raymond Hettinger; :issue:`9025`.)
1843
1844poplib
1845------
1846
1847:class:`~poplib.POP3_SSL` class now accepts a *context* parameter, which is a
1848:class:`ssl.SSLContext` object allowing bundling SSL configuration options,
1849certificates and private keys into a single (potentially long-lived)
1850structure.
1851
1852(Contributed by Giampaolo Rodolà; :issue:`8807`.)
1853
1854asyncore
1855--------
1856
1857:class:`asyncore.dispatcher` now provides a
1858:meth:`~asyncore.dispatcher.handle_accepted()` method
1859returning a `(sock, addr)` pair which is called when a connection has actually
1860been established with a new remote endpoint. This is supposed to be used as a
1861replacement for old :meth:`~asyncore.dispatcher.handle_accept()` and avoids
1862the user  to call :meth:`~asyncore.dispatcher.accept()` directly.
1863
1864(Contributed by Giampaolo Rodolà; :issue:`6706`.)
1865
1866tempfile
1867--------
1868
1869The :mod:`tempfile` module has a new context manager,
1870:class:`~tempfile.TemporaryDirectory` which provides easy deterministic
1871cleanup of temporary directories::
1872
1873    with tempfile.TemporaryDirectory() as tmpdirname:
1874        print('created temporary dir:', tmpdirname)
1875
1876(Contributed by Neil Schemenauer and Nick Coghlan; :issue:`5178`.)
1877
1878inspect
1879-------
1880
1881* The :mod:`inspect` module has a new function
1882  :func:`~inspect.getgeneratorstate` to easily identify the current state of a
1883  generator-iterator::
1884
1885    >>> from inspect import getgeneratorstate
1886    >>> def gen():
1887    ...     yield 'demo'
1888    >>> g = gen()
1889    >>> getgeneratorstate(g)
1890    'GEN_CREATED'
1891    >>> next(g)
1892    'demo'
1893    >>> getgeneratorstate(g)
1894    'GEN_SUSPENDED'
1895    >>> next(g, None)
1896    >>> getgeneratorstate(g)
1897    'GEN_CLOSED'
1898
1899  (Contributed by Rodolpho Eckhardt and Nick Coghlan, :issue:`10220`.)
1900
1901* To support lookups without the possibility of activating a dynamic attribute,
1902  the :mod:`inspect` module has a new function, :func:`~inspect.getattr_static`.
1903  Unlike :func:`hasattr`, this is a true read-only search, guaranteed not to
1904  change state while it is searching::
1905
1906    >>> class A:
1907    ...     @property
1908    ...     def f(self):
1909    ...         print('Running')
1910    ...         return 10
1911    ...
1912    >>> a = A()
1913    >>> getattr(a, 'f')
1914    Running
1915    10
1916    >>> inspect.getattr_static(a, 'f')
1917    <property object at 0x1022bd788>
1918
1919 (Contributed by Michael Foord.)
1920
1921pydoc
1922-----
1923
1924The :mod:`pydoc` module now provides a much-improved web server interface, as
1925well as a new command-line option ``-b`` to automatically open a browser window
1926to display that server:
1927
1928.. code-block:: shell-session
1929
1930    $ pydoc3.2 -b
1931
1932(Contributed by Ron Adam; :issue:`2001`.)
1933
1934dis
1935---
1936
1937The :mod:`dis` module gained two new functions for inspecting code,
1938:func:`~dis.code_info` and :func:`~dis.show_code`.  Both provide detailed code
1939object information for the supplied function, method, source code string or code
1940object.  The former returns a string and the latter prints it::
1941
1942    >>> import dis, random
1943    >>> dis.show_code(random.choice)
1944    Name:              choice
1945    Filename:          /Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/random.py
1946    Argument count:    2
1947    Kw-only arguments: 0
1948    Number of locals:  3
1949    Stack size:        11
1950    Flags:             OPTIMIZED, NEWLOCALS, NOFREE
1951    Constants:
1952       0: 'Choose a random element from a non-empty sequence.'
1953       1: 'Cannot choose from an empty sequence'
1954    Names:
1955       0: _randbelow
1956       1: len
1957       2: ValueError
1958       3: IndexError
1959    Variable names:
1960       0: self
1961       1: seq
1962       2: i
1963
1964In addition, the :func:`~dis.dis` function now accepts string arguments
1965so that the common idiom ``dis(compile(s, '', 'eval'))`` can be shortened
1966to ``dis(s)``::
1967
1968    >>> dis('3*x+1 if x%2==1 else x//2')
1969      1           0 LOAD_NAME                0 (x)
1970                  3 LOAD_CONST               0 (2)
1971                  6 BINARY_MODULO
1972                  7 LOAD_CONST               1 (1)
1973                 10 COMPARE_OP               2 (==)
1974                 13 POP_JUMP_IF_FALSE       28
1975                 16 LOAD_CONST               2 (3)
1976                 19 LOAD_NAME                0 (x)
1977                 22 BINARY_MULTIPLY
1978                 23 LOAD_CONST               1 (1)
1979                 26 BINARY_ADD
1980                 27 RETURN_VALUE
1981            >>   28 LOAD_NAME                0 (x)
1982                 31 LOAD_CONST               0 (2)
1983                 34 BINARY_FLOOR_DIVIDE
1984                 35 RETURN_VALUE
1985
1986Taken together, these improvements make it easier to explore how CPython is
1987implemented and to see for yourself what the language syntax does
1988under-the-hood.
1989
1990(Contributed by Nick Coghlan in :issue:`9147`.)
1991
1992dbm
1993---
1994
1995All database modules now support the :meth:`get` and :meth:`setdefault` methods.
1996
1997(Suggested by Ray Allen in :issue:`9523`.)
1998
1999ctypes
2000------
2001
2002A new type, :class:`ctypes.c_ssize_t` represents the C :c:type:`ssize_t` datatype.
2003
2004site
2005----
2006
2007The :mod:`site` module has three new functions useful for reporting on the
2008details of a given Python installation.
2009
2010* :func:`~site.getsitepackages` lists all global site-packages directories.
2011
2012* :func:`~site.getuserbase` reports on the user's base directory where data can
2013  be stored.
2014
2015* :func:`~site.getusersitepackages` reveals the user-specific site-packages
2016  directory path.
2017
2018::
2019
2020    >>> import site
2021    >>> site.getsitepackages()
2022    ['/Library/Frameworks/Python.framework/Versions/3.2/lib/python3.2/site-packages',
2023     '/Library/Frameworks/Python.framework/Versions/3.2/lib/site-python',
2024     '/Library/Python/3.2/site-packages']
2025    >>> site.getuserbase()
2026    '/Users/raymondhettinger/Library/Python/3.2'
2027    >>> site.getusersitepackages()
2028    '/Users/raymondhettinger/Library/Python/3.2/lib/python/site-packages'
2029
2030Conveniently, some of site's functionality is accessible directly from the
2031command-line:
2032
2033.. code-block:: shell-session
2034
2035    $ python -m site --user-base
2036    /Users/raymondhettinger/.local
2037    $ python -m site --user-site
2038    /Users/raymondhettinger/.local/lib/python3.2/site-packages
2039
2040(Contributed by Tarek Ziadé in :issue:`6693`.)
2041
2042sysconfig
2043---------
2044
2045The new :mod:`sysconfig` module makes it straightforward to discover
2046installation paths and configuration variables that vary across platforms and
2047installations.
2048
2049The module offers access simple access functions for platform and version
2050information:
2051
2052* :func:`~sysconfig.get_platform` returning values like *linux-i586* or
2053  *macosx-10.6-ppc*.
2054* :func:`~sysconfig.get_python_version` returns a Python version string
2055  such as "3.2".
2056
2057It also provides access to the paths and variables corresponding to one of
2058seven named schemes used by :mod:`distutils`.  Those include *posix_prefix*,
2059*posix_home*, *posix_user*, *nt*, *nt_user*, *os2*, *os2_home*:
2060
2061* :func:`~sysconfig.get_paths` makes a dictionary containing installation paths
2062  for the current installation scheme.
2063* :func:`~sysconfig.get_config_vars` returns a dictionary of platform specific
2064  variables.
2065
2066There is also a convenient command-line interface:
2067
2068.. code-block:: doscon
2069
2070  C:\Python32>python -m sysconfig
2071  Platform: "win32"
2072  Python version: "3.2"
2073  Current installation scheme: "nt"
2074
2075  Paths:
2076          data = "C:\Python32"
2077          include = "C:\Python32\Include"
2078          platinclude = "C:\Python32\Include"
2079          platlib = "C:\Python32\Lib\site-packages"
2080          platstdlib = "C:\Python32\Lib"
2081          purelib = "C:\Python32\Lib\site-packages"
2082          scripts = "C:\Python32\Scripts"
2083          stdlib = "C:\Python32\Lib"
2084
2085  Variables:
2086          BINDIR = "C:\Python32"
2087          BINLIBDEST = "C:\Python32\Lib"
2088          EXE = ".exe"
2089          INCLUDEPY = "C:\Python32\Include"
2090          LIBDEST = "C:\Python32\Lib"
2091          SO = ".pyd"
2092          VERSION = "32"
2093          abiflags = ""
2094          base = "C:\Python32"
2095          exec_prefix = "C:\Python32"
2096          platbase = "C:\Python32"
2097          prefix = "C:\Python32"
2098          projectbase = "C:\Python32"
2099          py_version = "3.2"
2100          py_version_nodot = "32"
2101          py_version_short = "3.2"
2102          srcdir = "C:\Python32"
2103          userbase = "C:\Documents and Settings\Raymond\Application Data\Python"
2104
2105(Moved out of Distutils by Tarek Ziadé.)
2106
2107pdb
2108---
2109
2110The :mod:`pdb` debugger module gained a number of usability improvements:
2111
2112* :file:`pdb.py` now has a ``-c`` option that executes commands as given in a
2113  :file:`.pdbrc` script file.
2114* A :file:`.pdbrc` script file can contain ``continue`` and ``next`` commands
2115  that continue debugging.
2116* The :class:`Pdb` class constructor now accepts a *nosigint* argument.
2117* New commands: ``l(list)``, ``ll(long list)`` and ``source`` for
2118  listing source code.
2119* New commands: ``display`` and ``undisplay`` for showing or hiding
2120  the value of an expression if it has changed.
2121* New command: ``interact`` for starting an interactive interpreter containing
2122  the global and local  names found in the current scope.
2123* Breakpoints can be cleared by breakpoint number.
2124
2125(Contributed by Georg Brandl, Antonio Cuni and Ilya Sandler.)
2126
2127configparser
2128------------
2129
2130The :mod:`configparser` module was modified to improve usability and
2131predictability of the default parser and its supported INI syntax.  The old
2132:class:`ConfigParser` class was removed in favor of :class:`SafeConfigParser`
2133which has in turn been renamed to :class:`~configparser.ConfigParser`. Support
2134for inline comments is now turned off by default and section or option
2135duplicates are not allowed in a single configuration source.
2136
2137Config parsers gained a new API based on the mapping protocol::
2138
2139    >>> parser = ConfigParser()
2140    >>> parser.read_string("""
2141    ... [DEFAULT]
2142    ... location = upper left
2143    ... visible = yes
2144    ... editable = no
2145    ... color = blue
2146    ...
2147    ... [main]
2148    ... title = Main Menu
2149    ... color = green
2150    ...
2151    ... [options]
2152    ... title = Options
2153    ... """)
2154    >>> parser['main']['color']
2155    'green'
2156    >>> parser['main']['editable']
2157    'no'
2158    >>> section = parser['options']
2159    >>> section['title']
2160    'Options'
2161    >>> section['title'] = 'Options (editable: %(editable)s)'
2162    >>> section['title']
2163    'Options (editable: no)'
2164
2165The new API is implemented on top of the classical API, so custom parser
2166subclasses should be able to use it without modifications.
2167
2168The INI file structure accepted by config parsers can now be customized. Users
2169can specify alternative option/value delimiters and comment prefixes, change the
2170name of the *DEFAULT* section or switch the interpolation syntax.
2171
2172There is support for pluggable interpolation including an additional interpolation
2173handler :class:`~configparser.ExtendedInterpolation`::
2174
2175  >>> parser = ConfigParser(interpolation=ExtendedInterpolation())
2176  >>> parser.read_dict({'buildout': {'directory': '/home/ambv/zope9'},
2177  ...                   'custom': {'prefix': '/usr/local'}})
2178  >>> parser.read_string("""
2179  ... [buildout]
2180  ... parts =
2181  ...   zope9
2182  ...   instance
2183  ... find-links =
2184  ...   ${buildout:directory}/downloads/dist
2185  ...
2186  ... [zope9]
2187  ... recipe = plone.recipe.zope9install
2188  ... location = /opt/zope
2189  ...
2190  ... [instance]
2191  ... recipe = plone.recipe.zope9instance
2192  ... zope9-location = ${zope9:location}
2193  ... zope-conf = ${custom:prefix}/etc/zope.conf
2194  ... """)
2195  >>> parser['buildout']['find-links']
2196  '\n/home/ambv/zope9/downloads/dist'
2197  >>> parser['instance']['zope-conf']
2198  '/usr/local/etc/zope.conf'
2199  >>> instance = parser['instance']
2200  >>> instance['zope-conf']
2201  '/usr/local/etc/zope.conf'
2202  >>> instance['zope9-location']
2203  '/opt/zope'
2204
2205A number of smaller features were also introduced, like support for specifying
2206encoding in read operations, specifying fallback values for get-functions, or
2207reading directly from dictionaries and strings.
2208
2209(All changes contributed by Łukasz Langa.)
2210
2211.. XXX consider showing a difflib example
2212
2213urllib.parse
2214------------
2215
2216A number of usability improvements were made for the :mod:`urllib.parse` module.
2217
2218The :func:`~urllib.parse.urlparse` function now supports `IPv6
2219<https://en.wikipedia.org/wiki/IPv6>`_ addresses as described in :rfc:`2732`:
2220
2221    >>> import urllib.parse
2222    >>> urllib.parse.urlparse('http://[dead:beef:cafe:5417:affe:8FA3:deaf:feed]/foo/') # doctest: +NORMALIZE_WHITESPACE
2223    ParseResult(scheme='http',
2224                netloc='[dead:beef:cafe:5417:affe:8FA3:deaf:feed]',
2225                path='/foo/',
2226                params='',
2227                query='',
2228                fragment='')
2229
2230The :func:`~urllib.parse.urldefrag` function now returns a :term:`named tuple`::
2231
2232    >>> r = urllib.parse.urldefrag('http://python.org/about/#target')
2233    >>> r
2234    DefragResult(url='http://python.org/about/', fragment='target')
2235    >>> r[0]
2236    'http://python.org/about/'
2237    >>> r.fragment
2238    'target'
2239
2240And, the :func:`~urllib.parse.urlencode` function is now much more flexible,
2241accepting either a string or bytes type for the *query* argument.  If it is a
2242string, then the *safe*, *encoding*, and *error* parameters are sent to
2243:func:`~urllib.parse.quote_plus` for encoding::
2244
2245    >>> urllib.parse.urlencode([
2246    ...      ('type', 'telenovela'),
2247    ...      ('name', '¿Dónde Está Elisa?')],
2248    ...      encoding='latin-1')
2249    'type=telenovela&name=%BFD%F3nde+Est%E1+Elisa%3F'
2250
2251As detailed in :ref:`parsing-ascii-encoded-bytes`, all the :mod:`urllib.parse`
2252functions now accept ASCII-encoded byte strings as input, so long as they are
2253not mixed with regular strings.  If ASCII-encoded byte strings are given as
2254parameters, the return types will also be an ASCII-encoded byte strings:
2255
2256    >>> urllib.parse.urlparse(b'http://www.python.org:80/about/') # doctest: +NORMALIZE_WHITESPACE
2257    ParseResultBytes(scheme=b'http', netloc=b'www.python.org:80',
2258                     path=b'/about/', params=b'', query=b'', fragment=b'')
2259
2260(Work by Nick Coghlan, Dan Mahn, and Senthil Kumaran in :issue:`2987`,
2261:issue:`5468`, and :issue:`9873`.)
2262
2263mailbox
2264-------
2265
2266Thanks to a concerted effort by R. David Murray, the :mod:`mailbox` module has
2267been fixed for Python 3.2.  The challenge was that mailbox had been originally
2268designed with a text interface, but email messages are best represented with
2269:class:`bytes` because various parts of a message may have different encodings.
2270
2271The solution harnessed the :mod:`email` package's binary support for parsing
2272arbitrary email messages.  In addition, the solution required a number of API
2273changes.
2274
2275As expected, the :meth:`~mailbox.Mailbox.add` method for
2276:class:`mailbox.Mailbox` objects now accepts binary input.
2277
2278:class:`~io.StringIO` and text file input are deprecated.  Also, string input
2279will fail early if non-ASCII characters are used.  Previously it would fail when
2280the email was processed in a later step.
2281
2282There is also support for binary output.  The :meth:`~mailbox.Mailbox.get_file`
2283method now returns a file in the binary mode (where it used to incorrectly set
2284the file to text-mode).  There is also a new :meth:`~mailbox.Mailbox.get_bytes`
2285method that returns a :class:`bytes` representation of a message corresponding
2286to a given *key*.
2287
2288It is still possible to get non-binary output using the old API's
2289:meth:`~mailbox.Mailbox.get_string` method, but that approach
2290is not very useful.  Instead, it is best to extract messages from
2291a :class:`~mailbox.Message` object or to load them from binary input.
2292
2293(Contributed by R. David Murray, with efforts from Steffen Daode Nurpmeso and an
2294initial patch by Victor Stinner in :issue:`9124`.)
2295
2296turtledemo
2297----------
2298
2299The demonstration code for the :mod:`turtle` module was moved from the *Demo*
2300directory to main library.  It includes over a dozen sample scripts with
2301lively displays.  Being on :attr:`sys.path`, it can now be run directly
2302from the command-line:
2303
2304.. code-block:: shell-session
2305
2306    $ python -m turtledemo
2307
2308(Moved from the Demo directory by Alexander Belopolsky in :issue:`10199`.)
2309
2310Multi-threading
2311===============
2312
2313* The mechanism for serializing execution of concurrently running Python threads
2314  (generally known as the :term:`GIL` or Global Interpreter Lock) has
2315  been rewritten.  Among the objectives were more predictable switching
2316  intervals and reduced overhead due to lock contention and the number of
2317  ensuing system calls.  The notion of a "check interval" to allow thread
2318  switches has been abandoned and replaced by an absolute duration expressed in
2319  seconds.  This parameter is tunable through :func:`sys.setswitchinterval()`.
2320  It currently defaults to 5 milliseconds.
2321
2322  Additional details about the implementation can be read from a `python-dev
2323  mailing-list message
2324  <https://mail.python.org/pipermail/python-dev/2009-October/093321.html>`_
2325  (however, "priority requests" as exposed in this message have not been kept
2326  for inclusion).
2327
2328  (Contributed by Antoine Pitrou.)
2329
2330* Regular and recursive locks now accept an optional *timeout* argument to their
2331  :meth:`~threading.Lock.acquire` method.  (Contributed by Antoine Pitrou;
2332  :issue:`7316`.)
2333
2334* Similarly, :meth:`threading.Semaphore.acquire` also gained a *timeout*
2335  argument.  (Contributed by Torsten Landschoff; :issue:`850728`.)
2336
2337* Regular and recursive lock acquisitions can now be interrupted by signals on
2338  platforms using Pthreads.  This means that Python programs that deadlock while
2339  acquiring locks can be successfully killed by repeatedly sending SIGINT to the
2340  process (by pressing :kbd:`Ctrl+C` in most shells).
2341  (Contributed by Reid Kleckner; :issue:`8844`.)
2342
2343
2344Optimizations
2345=============
2346
2347A number of small performance enhancements have been added:
2348
2349* Python's peephole optimizer now recognizes patterns such ``x in {1, 2, 3}`` as
2350  being a test for membership in a set of constants.  The optimizer recasts the
2351  :class:`set` as a :class:`frozenset` and stores the pre-built constant.
2352
2353  Now that the speed penalty is gone, it is practical to start writing
2354  membership tests using set-notation.  This style is both semantically clear
2355  and operationally fast::
2356
2357      extension = name.rpartition('.')[2]
2358      if extension in {'xml', 'html', 'xhtml', 'css'}:
2359          handle(name)
2360
2361  (Patch and additional tests contributed by Dave Malcolm; :issue:`6690`).
2362
2363* Serializing and unserializing data using the :mod:`pickle` module is now
2364  several times faster.
2365
2366  (Contributed by Alexandre Vassalotti, Antoine Pitrou
2367  and the Unladen Swallow team in :issue:`9410` and :issue:`3873`.)
2368
2369* The `Timsort algorithm <https://en.wikipedia.org/wiki/Timsort>`_ used in
2370  :meth:`list.sort` and :func:`sorted` now runs faster and uses less memory
2371  when called with a :term:`key function`.  Previously, every element of
2372  a list was wrapped with a temporary object that remembered the key value
2373  associated with each element.  Now, two arrays of keys and values are
2374  sorted in parallel.  This saves the memory consumed by the sort wrappers,
2375  and it saves time lost to delegating comparisons.
2376
2377  (Patch by Daniel Stutzbach in :issue:`9915`.)
2378
2379* JSON decoding performance is improved and memory consumption is reduced
2380  whenever the same string is repeated for multiple keys.  Also, JSON encoding
2381  now uses the C speedups when the ``sort_keys`` argument is true.
2382
2383  (Contributed by Antoine Pitrou in :issue:`7451` and by Raymond Hettinger and
2384  Antoine Pitrou in :issue:`10314`.)
2385
2386* Recursive locks (created with the :func:`threading.RLock` API) now benefit
2387  from a C implementation which makes them as fast as regular locks, and between
2388  10x and 15x faster than their previous pure Python implementation.
2389
2390  (Contributed by Antoine Pitrou; :issue:`3001`.)
2391
2392* The fast-search algorithm in stringlib is now used by the :meth:`split`,
2393  :meth:`rsplit`, :meth:`splitlines` and :meth:`replace` methods on
2394  :class:`bytes`, :class:`bytearray` and :class:`str` objects. Likewise, the
2395  algorithm is also used by :meth:`rfind`, :meth:`rindex`, :meth:`rsplit` and
2396  :meth:`rpartition`.
2397
2398  (Patch by Florent Xicluna in :issue:`7622` and :issue:`7462`.)
2399
2400
2401* Integer to string conversions now work two "digits" at a time, reducing the
2402  number of division and modulo operations.
2403
2404  (:issue:`6713` by Gawain Bolton, Mark Dickinson, and Victor Stinner.)
2405
2406There were several other minor optimizations. Set differencing now runs faster
2407when one operand is much larger than the other (patch by Andress Bennetts in
2408:issue:`8685`).  The :meth:`array.repeat` method has a faster implementation
2409(:issue:`1569291` by Alexander Belopolsky). The :class:`BaseHTTPRequestHandler`
2410has more efficient buffering (:issue:`3709` by Andrew Schaaf).  The
2411:func:`operator.attrgetter` function has been sped-up (:issue:`10160` by
2412Christos Georgiou).  And :class:`ConfigParser` loads multi-line arguments a bit
2413faster (:issue:`7113` by Łukasz Langa).
2414
2415
2416Unicode
2417=======
2418
2419Python has been updated to `Unicode 6.0.0
2420<http://unicode.org/versions/Unicode6.0.0/>`_.  The update to the standard adds
2421over 2,000 new characters including `emoji <https://en.wikipedia.org/wiki/Emoji>`_
2422symbols which are important for mobile phones.
2423
2424In addition, the updated standard has altered the character properties for two
2425Kannada characters (U+0CF1, U+0CF2) and one New Tai Lue numeric character
2426(U+19DA), making the former eligible for use in identifiers while disqualifying
2427the latter.  For more information, see `Unicode Character Database Changes
2428<http://www.unicode.org/versions/Unicode6.0.0/#Database_Changes>`_.
2429
2430
2431Codecs
2432======
2433
2434Support was added for *cp720* Arabic DOS encoding (:issue:`1616979`).
2435
2436MBCS encoding no longer ignores the error handler argument. In the default
2437strict mode, it raises an :exc:`UnicodeDecodeError` when it encounters an
2438undecodable byte sequence and an :exc:`UnicodeEncodeError` for an unencodable
2439character.
2440
2441The MBCS codec supports ``'strict'`` and ``'ignore'`` error handlers for
2442decoding, and ``'strict'`` and ``'replace'`` for encoding.
2443
2444To emulate Python3.1 MBCS encoding, select the ``'ignore'`` handler for decoding
2445and the ``'replace'`` handler for encoding.
2446
2447On Mac OS X, Python decodes command line arguments with ``'utf-8'`` rather than
2448the locale encoding.
2449
2450By default, :mod:`tarfile` uses ``'utf-8'`` encoding on Windows (instead of
2451``'mbcs'``) and the ``'surrogateescape'`` error handler on all operating
2452systems.
2453
2454
2455Documentation
2456=============
2457
2458The documentation continues to be improved.
2459
2460* A table of quick links has been added to the top of lengthy sections such as
2461  :ref:`built-in-funcs`.  In the case of :mod:`itertools`, the links are
2462  accompanied by tables of cheatsheet-style summaries to provide an overview and
2463  memory jog without having to read all of the docs.
2464
2465* In some cases, the pure Python source code can be a helpful adjunct to the
2466  documentation, so now many modules now feature quick links to the latest
2467  version of the source code.  For example, the :mod:`functools` module
2468  documentation has a quick link at the top labeled:
2469
2470    **Source code** :source:`Lib/functools.py`.
2471
2472  (Contributed by Raymond Hettinger; see
2473  `rationale <https://rhettinger.wordpress.com/2011/01/28/open-your-source-more/>`_.)
2474
2475* The docs now contain more examples and recipes.  In particular, :mod:`re`
2476  module has an extensive section, :ref:`re-examples`.  Likewise, the
2477  :mod:`itertools` module continues to be updated with new
2478  :ref:`itertools-recipes`.
2479
2480* The :mod:`datetime` module now has an auxiliary implementation in pure Python.
2481  No functionality was changed.  This just provides an easier-to-read alternate
2482  implementation.
2483
2484  (Contributed by Alexander Belopolsky in :issue:`9528`.)
2485
2486* The unmaintained :file:`Demo` directory has been removed.  Some demos were
2487  integrated into the documentation, some were moved to the :file:`Tools/demo`
2488  directory, and others were removed altogether.
2489
2490  (Contributed by Georg Brandl in :issue:`7962`.)
2491
2492
2493IDLE
2494====
2495
2496* The format menu now has an option to clean source files by stripping
2497  trailing whitespace.
2498
2499  (Contributed by Raymond Hettinger; :issue:`5150`.)
2500
2501* IDLE on Mac OS X now works with both Carbon AquaTk and Cocoa AquaTk.
2502
2503  (Contributed by Kevin Walzer, Ned Deily, and Ronald Oussoren; :issue:`6075`.)
2504
2505Code Repository
2506===============
2507
2508In addition to the existing Subversion code repository at http://svn.python.org
2509there is now a `Mercurial <https://www.mercurial-scm.org/>`_ repository at
2510https://hg.python.org/\ .
2511
2512After the 3.2 release, there are plans to switch to Mercurial as the primary
2513repository.  This distributed version control system should make it easier for
2514members of the community to create and share external changesets.  See
2515:pep:`385` for details.
2516
2517To learn to use the new version control system, see the `Quick Start
2518<https://www.mercurial-scm.org/wiki/QuickStart>`_ or the `Guide to
2519Mercurial Workflows <https://www.mercurial-scm.org/guide>`_.
2520
2521
2522Build and C API Changes
2523=======================
2524
2525Changes to Python's build process and to the C API include:
2526
2527* The *idle*, *pydoc* and *2to3* scripts are now installed with a
2528  version-specific suffix on ``make altinstall`` (:issue:`10679`).
2529
2530* The C functions that access the Unicode Database now accept and return
2531  characters from the full Unicode range, even on narrow unicode builds
2532  (Py_UNICODE_TOLOWER, Py_UNICODE_ISDECIMAL, and others).  A visible difference
2533  in Python is that :func:`unicodedata.numeric` now returns the correct value
2534  for large code points, and :func:`repr` may consider more characters as
2535  printable.
2536
2537  (Reported by Bupjoe Lee and fixed by Amaury Forgeot D'Arc; :issue:`5127`.)
2538
2539* Computed gotos are now enabled by default on supported compilers (which are
2540  detected by the configure script).  They can still be disabled selectively by
2541  specifying ``--without-computed-gotos``.
2542
2543  (Contributed by Antoine Pitrou; :issue:`9203`.)
2544
2545* The option ``--with-wctype-functions`` was removed.  The built-in unicode
2546  database is now used for all functions.
2547
2548  (Contributed by Amaury Forgeot D'Arc; :issue:`9210`.)
2549
2550* Hash values are now values of a new type, :c:type:`Py_hash_t`, which is
2551  defined to be the same size as a pointer.  Previously they were of type long,
2552  which on some 64-bit operating systems is still only 32 bits long.  As a
2553  result of this fix, :class:`set` and :class:`dict` can now hold more than
2554  ``2**32`` entries on builds with 64-bit pointers (previously, they could grow
2555  to that size but their performance degraded catastrophically).
2556
2557  (Suggested by Raymond Hettinger and implemented by Benjamin Peterson;
2558  :issue:`9778`.)
2559
2560* A new macro :c:macro:`Py_VA_COPY` copies the state of the variable argument
2561  list.  It is equivalent to C99 *va_copy* but available on all Python platforms
2562  (:issue:`2443`).
2563
2564* A new C API function :c:func:`PySys_SetArgvEx` allows an embedded interpreter
2565  to set :attr:`sys.argv` without also modifying :attr:`sys.path`
2566  (:issue:`5753`).
2567
2568* :c:macro:`PyEval_CallObject` is now only available in macro form.  The
2569  function declaration, which was kept for backwards compatibility reasons, is
2570  now removed -- the macro was introduced in 1997 (:issue:`8276`).
2571
2572* There is a new function :c:func:`PyLong_AsLongLongAndOverflow` which
2573  is analogous to :c:func:`PyLong_AsLongAndOverflow`.  They both serve to
2574  convert Python :class:`int` into a native fixed-width type while providing
2575  detection of cases where the conversion won't fit (:issue:`7767`).
2576
2577* The :c:func:`PyUnicode_CompareWithASCIIString` function now returns *not
2578  equal* if the Python string is *NUL* terminated.
2579
2580* There is a new function :c:func:`PyErr_NewExceptionWithDoc` that is
2581  like :c:func:`PyErr_NewException` but allows a docstring to be specified.
2582  This lets C exceptions have the same self-documenting capabilities as
2583  their pure Python counterparts (:issue:`7033`).
2584
2585* When compiled with the ``--with-valgrind`` option, the pymalloc
2586  allocator will be automatically disabled when running under Valgrind.  This
2587  gives improved memory leak detection when running under Valgrind, while taking
2588  advantage of pymalloc at other times (:issue:`2422`).
2589
2590* Removed the ``O?`` format from the *PyArg_Parse* functions.  The format is no
2591  longer used and it had never been documented (:issue:`8837`).
2592
2593There were a number of other small changes to the C-API.  See the
2594:source:`Misc/NEWS` file for a complete list.
2595
2596Also, there were a number of updates to the Mac OS X build, see
2597:source:`Mac/BuildScript/README.txt` for details.  For users running a 32/64-bit
2598build, there is a known problem with the default Tcl/Tk on Mac OS X 10.6.
2599Accordingly, we recommend installing an updated alternative such as
2600`ActiveState Tcl/Tk 8.5.9 <https://www.activestate.com/activetcl/downloads>`_\.
2601See https://www.python.org/download/mac/tcltk/ for additional details.
2602
2603Porting to Python 3.2
2604=====================
2605
2606This section lists previously described changes and other bugfixes that may
2607require changes to your code:
2608
2609* The :mod:`configparser` module has a number of clean-ups.  The major change is
2610  to replace the old :class:`ConfigParser` class with long-standing preferred
2611  alternative :class:`SafeConfigParser`.  In addition there are a number of
2612  smaller incompatibilities:
2613
2614  * The interpolation syntax is now validated on
2615    :meth:`~configparser.ConfigParser.get` and
2616    :meth:`~configparser.ConfigParser.set` operations. In the default
2617    interpolation scheme, only two tokens with percent signs are valid: ``%(name)s``
2618    and ``%%``, the latter being an escaped percent sign.
2619
2620  * The :meth:`~configparser.ConfigParser.set` and
2621    :meth:`~configparser.ConfigParser.add_section` methods now verify that
2622    values are actual strings.  Formerly, unsupported types could be introduced
2623    unintentionally.
2624
2625  * Duplicate sections or options from a single source now raise either
2626    :exc:`~configparser.DuplicateSectionError` or
2627    :exc:`~configparser.DuplicateOptionError`.  Formerly, duplicates would
2628    silently overwrite a previous entry.
2629
2630  * Inline comments are now disabled by default so now the **;** character
2631    can be safely used in values.
2632
2633  * Comments now can be indented.  Consequently, for **;** or **#** to appear at
2634    the start of a line in multiline values, it has to be interpolated.  This
2635    keeps comment prefix characters in values from being mistaken as comments.
2636
2637  * ``""`` is now a valid value and is no longer automatically converted to an
2638    empty string. For empty strings, use ``"option ="`` in a line.
2639
2640* The :mod:`nntplib` module was reworked extensively, meaning that its APIs
2641  are often incompatible with the 3.1 APIs.
2642
2643* :class:`bytearray` objects can no longer be used as filenames; instead,
2644  they should be converted to :class:`bytes`.
2645
2646* The :meth:`array.tostring` and :meth:`array.fromstring` have been renamed to
2647  :meth:`array.tobytes` and :meth:`array.frombytes` for clarity.  The old names
2648  have been deprecated. (See :issue:`8990`.)
2649
2650* ``PyArg_Parse*()`` functions:
2651
2652  * "t#" format has been removed: use "s#" or "s*" instead
2653  * "w" and "w#" formats has been removed: use "w*" instead
2654
2655* The :c:type:`PyCObject` type, deprecated in 3.1, has been removed.  To wrap
2656  opaque C pointers in Python objects, the :c:type:`PyCapsule` API should be used
2657  instead; the new type has a well-defined interface for passing typing safety
2658  information and a less complicated signature for calling a destructor.
2659
2660* The :func:`sys.setfilesystemencoding` function was removed because
2661  it had a flawed design.
2662
2663* The :func:`random.seed` function and method now salt string seeds with an
2664  sha512 hash function.  To access the previous version of *seed* in order to
2665  reproduce Python 3.1 sequences, set the *version* argument to *1*,
2666  ``random.seed(s, version=1)``.
2667
2668* The previously deprecated :func:`string.maketrans` function has been removed
2669  in favor of the static methods :meth:`bytes.maketrans` and
2670  :meth:`bytearray.maketrans`.  This change solves the confusion around which
2671  types were supported by the :mod:`string` module.  Now, :class:`str`,
2672  :class:`bytes`, and :class:`bytearray` each have their own **maketrans** and
2673  **translate** methods with intermediate translation tables of the appropriate
2674  type.
2675
2676  (Contributed by Georg Brandl; :issue:`5675`.)
2677
2678* The previously deprecated :func:`contextlib.nested` function has been removed
2679  in favor of a plain :keyword:`with` statement which can accept multiple
2680  context managers.  The latter technique is faster (because it is built-in),
2681  and it does a better job finalizing multiple context managers when one of them
2682  raises an exception::
2683
2684    with open('mylog.txt') as infile, open('a.out', 'w') as outfile:
2685        for line in infile:
2686            if '<critical>' in line:
2687                outfile.write(line)
2688
2689  (Contributed by Georg Brandl and Mattias Brändström;
2690  `appspot issue 53094 <https://codereview.appspot.com/53094>`_.)
2691
2692* :func:`struct.pack` now only allows bytes for the ``s`` string pack code.
2693  Formerly, it would accept text arguments and implicitly encode them to bytes
2694  using UTF-8.  This was problematic because it made assumptions about the
2695  correct encoding and because a variable-length encoding can fail when writing
2696  to fixed length segment of a structure.
2697
2698  Code such as ``struct.pack('<6sHHBBB', 'GIF87a', x, y)`` should be rewritten
2699  with to use bytes instead of text, ``struct.pack('<6sHHBBB', b'GIF87a', x, y)``.
2700
2701  (Discovered by David Beazley and fixed by Victor Stinner; :issue:`10783`.)
2702
2703* The :class:`xml.etree.ElementTree` class now raises an
2704  :exc:`xml.etree.ElementTree.ParseError` when a parse fails. Previously it
2705  raised an :exc:`xml.parsers.expat.ExpatError`.
2706
2707* The new, longer :func:`str` value on floats may break doctests which rely on
2708  the old output format.
2709
2710* In :class:`subprocess.Popen`, the default value for *close_fds* is now
2711  ``True`` under Unix; under Windows, it is ``True`` if the three standard
2712  streams are set to ``None``, ``False`` otherwise.  Previously, *close_fds*
2713  was always ``False`` by default, which produced difficult to solve bugs
2714  or race conditions when open file descriptors would leak into the child
2715  process.
2716
2717* Support for legacy HTTP 0.9 has been removed from :mod:`urllib.request`
2718  and :mod:`http.client`.  Such support is still present on the server side
2719  (in :mod:`http.server`).
2720
2721  (Contributed by Antoine Pitrou, :issue:`10711`.)
2722
2723* SSL sockets in timeout mode now raise :exc:`socket.timeout` when a timeout
2724  occurs, rather than a generic :exc:`~ssl.SSLError`.
2725
2726  (Contributed by Antoine Pitrou, :issue:`10272`.)
2727
2728* The misleading functions :c:func:`PyEval_AcquireLock()` and
2729  :c:func:`PyEval_ReleaseLock()` have been officially deprecated.  The
2730  thread-state aware APIs (such as :c:func:`PyEval_SaveThread()`
2731  and :c:func:`PyEval_RestoreThread()`) should be used instead.
2732
2733* Due to security risks, :func:`asyncore.handle_accept` has been deprecated, and
2734  a new function, :func:`asyncore.handle_accepted`, was added to replace it.
2735
2736  (Contributed by Giampaolo Rodola in :issue:`6706`.)
2737
2738* Due to the new :term:`GIL` implementation, :c:func:`PyEval_InitThreads()`
2739  cannot be called before :c:func:`Py_Initialize()` anymore.
2740