1:mod:`codecs` --- Codec registry and base classes 2================================================= 3 4.. module:: codecs 5 :synopsis: Encode and decode data and streams. 6 7.. moduleauthor:: Marc-André Lemburg <mal@lemburg.com> 8.. sectionauthor:: Marc-André Lemburg <mal@lemburg.com> 9.. sectionauthor:: Martin v. Löwis <martin@v.loewis.de> 10 11**Source code:** :source:`Lib/codecs.py` 12 13.. index:: 14 single: Unicode 15 single: Codecs 16 pair: Codecs; encode 17 pair: Codecs; decode 18 single: streams 19 pair: stackable; streams 20 21-------------- 22 23This module defines base classes for standard Python codecs (encoders and 24decoders) and provides access to the internal Python codec registry, which 25manages the codec and error handling lookup process. Most standard codecs 26are :term:`text encodings <text encoding>`, which encode text to bytes, 27but there are also codecs provided that encode text to text, and bytes to 28bytes. Custom codecs may encode and decode between arbitrary types, but some 29module features are restricted to use specifically with 30:term:`text encodings <text encoding>`, or with codecs that encode to 31:class:`bytes`. 32 33The module defines the following functions for encoding and decoding with 34any codec: 35 36.. function:: encode(obj, encoding='utf-8', errors='strict') 37 38 Encodes *obj* using the codec registered for *encoding*. 39 40 *Errors* may be given to set the desired error handling scheme. The 41 default error handler is ``'strict'`` meaning that encoding errors raise 42 :exc:`ValueError` (or a more codec specific subclass, such as 43 :exc:`UnicodeEncodeError`). Refer to :ref:`codec-base-classes` for more 44 information on codec error handling. 45 46.. function:: decode(obj, encoding='utf-8', errors='strict') 47 48 Decodes *obj* using the codec registered for *encoding*. 49 50 *Errors* may be given to set the desired error handling scheme. The 51 default error handler is ``'strict'`` meaning that decoding errors raise 52 :exc:`ValueError` (or a more codec specific subclass, such as 53 :exc:`UnicodeDecodeError`). Refer to :ref:`codec-base-classes` for more 54 information on codec error handling. 55 56The full details for each codec can also be looked up directly: 57 58.. function:: lookup(encoding) 59 60 Looks up the codec info in the Python codec registry and returns a 61 :class:`CodecInfo` object as defined below. 62 63 Encodings are first looked up in the registry's cache. If not found, the list of 64 registered search functions is scanned. If no :class:`CodecInfo` object is 65 found, a :exc:`LookupError` is raised. Otherwise, the :class:`CodecInfo` object 66 is stored in the cache and returned to the caller. 67 68.. class:: CodecInfo(encode, decode, streamreader=None, streamwriter=None, incrementalencoder=None, incrementaldecoder=None, name=None) 69 70 Codec details when looking up the codec registry. The constructor 71 arguments are stored in attributes of the same name: 72 73 74 .. attribute:: name 75 76 The name of the encoding. 77 78 79 .. attribute:: encode 80 decode 81 82 The stateless encoding and decoding functions. These must be 83 functions or methods which have the same interface as 84 the :meth:`~Codec.encode` and :meth:`~Codec.decode` methods of Codec 85 instances (see :ref:`Codec Interface <codec-objects>`). 86 The functions or methods are expected to work in a stateless mode. 87 88 89 .. attribute:: incrementalencoder 90 incrementaldecoder 91 92 Incremental encoder and decoder classes or factory functions. 93 These have to provide the interface defined by the base classes 94 :class:`IncrementalEncoder` and :class:`IncrementalDecoder`, 95 respectively. Incremental codecs can maintain state. 96 97 98 .. attribute:: streamwriter 99 streamreader 100 101 Stream writer and reader classes or factory functions. These have to 102 provide the interface defined by the base classes 103 :class:`StreamWriter` and :class:`StreamReader`, respectively. 104 Stream codecs can maintain state. 105 106To simplify access to the various codec components, the module provides 107these additional functions which use :func:`lookup` for the codec lookup: 108 109.. function:: getencoder(encoding) 110 111 Look up the codec for the given encoding and return its encoder function. 112 113 Raises a :exc:`LookupError` in case the encoding cannot be found. 114 115 116.. function:: getdecoder(encoding) 117 118 Look up the codec for the given encoding and return its decoder function. 119 120 Raises a :exc:`LookupError` in case the encoding cannot be found. 121 122 123.. function:: getincrementalencoder(encoding) 124 125 Look up the codec for the given encoding and return its incremental encoder 126 class or factory function. 127 128 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec 129 doesn't support an incremental encoder. 130 131 132.. function:: getincrementaldecoder(encoding) 133 134 Look up the codec for the given encoding and return its incremental decoder 135 class or factory function. 136 137 Raises a :exc:`LookupError` in case the encoding cannot be found or the codec 138 doesn't support an incremental decoder. 139 140 141.. function:: getreader(encoding) 142 143 Look up the codec for the given encoding and return its :class:`StreamReader` 144 class or factory function. 145 146 Raises a :exc:`LookupError` in case the encoding cannot be found. 147 148 149.. function:: getwriter(encoding) 150 151 Look up the codec for the given encoding and return its :class:`StreamWriter` 152 class or factory function. 153 154 Raises a :exc:`LookupError` in case the encoding cannot be found. 155 156Custom codecs are made available by registering a suitable codec search 157function: 158 159.. function:: register(search_function) 160 161 Register a codec search function. Search functions are expected to take one 162 argument, being the encoding name in all lower case letters with hyphens 163 and spaces converted to underscores, and return a :class:`CodecInfo` object. 164 In case a search function cannot find a given encoding, it should return 165 ``None``. 166 167 .. versionchanged:: 3.9 168 Hyphens and spaces are converted to underscore. 169 170 171.. function:: unregister(search_function) 172 173 Unregister a codec search function and clear the registry's cache. 174 If the search function is not registered, do nothing. 175 176 .. versionadded:: 3.10 177 178 179While the builtin :func:`open` and the associated :mod:`io` module are the 180recommended approach for working with encoded text files, this module 181provides additional utility functions and classes that allow the use of a 182wider range of codecs when working with binary files: 183 184.. function:: open(filename, mode='r', encoding=None, errors='strict', buffering=-1) 185 186 Open an encoded file using the given *mode* and return an instance of 187 :class:`StreamReaderWriter`, providing transparent encoding/decoding. 188 The default file mode is ``'r'``, meaning to open the file in read mode. 189 190 .. note:: 191 192 Underlying encoded files are always opened in binary mode. 193 No automatic conversion of ``'\n'`` is done on reading and writing. 194 The *mode* argument may be any binary mode acceptable to the built-in 195 :func:`open` function; the ``'b'`` is automatically added. 196 197 *encoding* specifies the encoding which is to be used for the file. 198 Any encoding that encodes to and decodes from bytes is allowed, and 199 the data types supported by the file methods depend on the codec used. 200 201 *errors* may be given to define the error handling. It defaults to ``'strict'`` 202 which causes a :exc:`ValueError` to be raised in case an encoding error occurs. 203 204 *buffering* has the same meaning as for the built-in :func:`open` function. 205 It defaults to -1 which means that the default buffer size will be used. 206 207 208.. function:: EncodedFile(file, data_encoding, file_encoding=None, errors='strict') 209 210 Return a :class:`StreamRecoder` instance, a wrapped version of *file* 211 which provides transparent transcoding. The original file is closed 212 when the wrapped version is closed. 213 214 Data written to the wrapped file is decoded according to the given 215 *data_encoding* and then written to the original file as bytes using 216 *file_encoding*. Bytes read from the original file are decoded 217 according to *file_encoding*, and the result is encoded 218 using *data_encoding*. 219 220 If *file_encoding* is not given, it defaults to *data_encoding*. 221 222 *errors* may be given to define the error handling. It defaults to 223 ``'strict'``, which causes :exc:`ValueError` to be raised in case an encoding 224 error occurs. 225 226 227.. function:: iterencode(iterator, encoding, errors='strict', **kwargs) 228 229 Uses an incremental encoder to iteratively encode the input provided by 230 *iterator*. This function is a :term:`generator`. 231 The *errors* argument (as well as any 232 other keyword argument) is passed through to the incremental encoder. 233 234 This function requires that the codec accept text :class:`str` objects 235 to encode. Therefore it does not support bytes-to-bytes encoders such as 236 ``base64_codec``. 237 238 239.. function:: iterdecode(iterator, encoding, errors='strict', **kwargs) 240 241 Uses an incremental decoder to iteratively decode the input provided by 242 *iterator*. This function is a :term:`generator`. 243 The *errors* argument (as well as any 244 other keyword argument) is passed through to the incremental decoder. 245 246 This function requires that the codec accept :class:`bytes` objects 247 to decode. Therefore it does not support text-to-text encoders such as 248 ``rot_13``, although ``rot_13`` may be used equivalently with 249 :func:`iterencode`. 250 251 252The module also provides the following constants which are useful for reading 253and writing to platform dependent files: 254 255 256.. data:: BOM 257 BOM_BE 258 BOM_LE 259 BOM_UTF8 260 BOM_UTF16 261 BOM_UTF16_BE 262 BOM_UTF16_LE 263 BOM_UTF32 264 BOM_UTF32_BE 265 BOM_UTF32_LE 266 267 These constants define various byte sequences, 268 being Unicode byte order marks (BOMs) for several encodings. They are 269 used in UTF-16 and UTF-32 data streams to indicate the byte order used, 270 and in UTF-8 as a Unicode signature. :const:`BOM_UTF16` is either 271 :const:`BOM_UTF16_BE` or :const:`BOM_UTF16_LE` depending on the platform's 272 native byte order, :const:`BOM` is an alias for :const:`BOM_UTF16`, 273 :const:`BOM_LE` for :const:`BOM_UTF16_LE` and :const:`BOM_BE` for 274 :const:`BOM_UTF16_BE`. The others represent the BOM in UTF-8 and UTF-32 275 encodings. 276 277 278.. _codec-base-classes: 279 280Codec Base Classes 281------------------ 282 283The :mod:`codecs` module defines a set of base classes which define the 284interfaces for working with codec objects, and can also be used as the basis 285for custom codec implementations. 286 287Each codec has to define four interfaces to make it usable as codec in Python: 288stateless encoder, stateless decoder, stream reader and stream writer. The 289stream reader and writers typically reuse the stateless encoder/decoder to 290implement the file protocols. Codec authors also need to define how the 291codec will handle encoding and decoding errors. 292 293 294.. _surrogateescape: 295.. _error-handlers: 296 297Error Handlers 298^^^^^^^^^^^^^^ 299 300To simplify and standardize error handling, 301codecs may implement different error handling schemes by 302accepting the *errors* string argument. The following string values are 303defined and implemented by all standard Python codecs: 304 305.. tabularcolumns:: |l|L| 306 307+-------------------------+-----------------------------------------------+ 308| Value | Meaning | 309+=========================+===============================================+ 310| ``'strict'`` | Raise :exc:`UnicodeError` (or a subclass); | 311| | this is the default. Implemented in | 312| | :func:`strict_errors`. | 313+-------------------------+-----------------------------------------------+ 314| ``'ignore'`` | Ignore the malformed data and continue | 315| | without further notice. Implemented in | 316| | :func:`ignore_errors`. | 317+-------------------------+-----------------------------------------------+ 318 319The following error handlers are only applicable to 320:term:`text encodings <text encoding>`: 321 322.. index:: 323 single: ? (question mark); replacement character 324 single: \ (backslash); escape sequence 325 single: \x; escape sequence 326 single: \u; escape sequence 327 single: \U; escape sequence 328 single: \N; escape sequence 329 330+-------------------------+-----------------------------------------------+ 331| Value | Meaning | 332+=========================+===============================================+ 333| ``'replace'`` | Replace with a suitable replacement | 334| | marker; Python will use the official | 335| | ``U+FFFD`` REPLACEMENT CHARACTER for the | 336| | built-in codecs on decoding, and '?' on | 337| | encoding. Implemented in | 338| | :func:`replace_errors`. | 339+-------------------------+-----------------------------------------------+ 340| ``'xmlcharrefreplace'`` | Replace with the appropriate XML character | 341| | reference (only for encoding). Implemented | 342| | in :func:`xmlcharrefreplace_errors`. | 343+-------------------------+-----------------------------------------------+ 344| ``'backslashreplace'`` | Replace with backslashed escape sequences. | 345| | Implemented in | 346| | :func:`backslashreplace_errors`. | 347+-------------------------+-----------------------------------------------+ 348| ``'namereplace'`` | Replace with ``\N{...}`` escape sequences | 349| | (only for encoding). Implemented in | 350| | :func:`namereplace_errors`. | 351+-------------------------+-----------------------------------------------+ 352| ``'surrogateescape'`` | On decoding, replace byte with individual | 353| | surrogate code ranging from ``U+DC80`` to | 354| | ``U+DCFF``. This code will then be turned | 355| | back into the same byte when the | 356| | ``'surrogateescape'`` error handler is used | 357| | when encoding the data. (See :pep:`383` for | 358| | more.) | 359+-------------------------+-----------------------------------------------+ 360 361In addition, the following error handler is specific to the given codecs: 362 363+-------------------+------------------------+-------------------------------------------+ 364| Value | Codecs | Meaning | 365+===================+========================+===========================================+ 366|``'surrogatepass'``| utf-8, utf-16, utf-32, | Allow encoding and decoding of surrogate | 367| | utf-16-be, utf-16-le, | codes. These codecs normally treat the | 368| | utf-32-be, utf-32-le | presence of surrogates as an error. | 369+-------------------+------------------------+-------------------------------------------+ 370 371.. versionadded:: 3.1 372 The ``'surrogateescape'`` and ``'surrogatepass'`` error handlers. 373 374.. versionchanged:: 3.4 375 The ``'surrogatepass'`` error handlers now works with utf-16\* and utf-32\* codecs. 376 377.. versionadded:: 3.5 378 The ``'namereplace'`` error handler. 379 380.. versionchanged:: 3.5 381 The ``'backslashreplace'`` error handlers now works with decoding and 382 translating. 383 384The set of allowed values can be extended by registering a new named error 385handler: 386 387.. function:: register_error(name, error_handler) 388 389 Register the error handling function *error_handler* under the name *name*. 390 The *error_handler* argument will be called during encoding and decoding 391 in case of an error, when *name* is specified as the errors parameter. 392 393 For encoding, *error_handler* will be called with a :exc:`UnicodeEncodeError` 394 instance, which contains information about the location of the error. The 395 error handler must either raise this or a different exception, or return a 396 tuple with a replacement for the unencodable part of the input and a position 397 where encoding should continue. The replacement may be either :class:`str` or 398 :class:`bytes`. If the replacement is bytes, the encoder will simply copy 399 them into the output buffer. If the replacement is a string, the encoder will 400 encode the replacement. Encoding continues on original input at the 401 specified position. Negative position values will be treated as being 402 relative to the end of the input string. If the resulting position is out of 403 bound an :exc:`IndexError` will be raised. 404 405 Decoding and translating works similarly, except :exc:`UnicodeDecodeError` or 406 :exc:`UnicodeTranslateError` will be passed to the handler and that the 407 replacement from the error handler will be put into the output directly. 408 409 410Previously registered error handlers (including the standard error handlers) 411can be looked up by name: 412 413.. function:: lookup_error(name) 414 415 Return the error handler previously registered under the name *name*. 416 417 Raises a :exc:`LookupError` in case the handler cannot be found. 418 419The following standard error handlers are also made available as module level 420functions: 421 422.. function:: strict_errors(exception) 423 424 Implements the ``'strict'`` error handling: each encoding or 425 decoding error raises a :exc:`UnicodeError`. 426 427 428.. function:: replace_errors(exception) 429 430 Implements the ``'replace'`` error handling (for :term:`text encodings 431 <text encoding>` only): substitutes ``'?'`` for encoding errors 432 (to be encoded by the codec), and ``'\ufffd'`` (the Unicode replacement 433 character) for decoding errors. 434 435 436.. function:: ignore_errors(exception) 437 438 Implements the ``'ignore'`` error handling: malformed data is ignored and 439 encoding or decoding is continued without further notice. 440 441 442.. function:: xmlcharrefreplace_errors(exception) 443 444 Implements the ``'xmlcharrefreplace'`` error handling (for encoding with 445 :term:`text encodings <text encoding>` only): the 446 unencodable character is replaced by an appropriate XML character reference. 447 448 449.. function:: backslashreplace_errors(exception) 450 451 Implements the ``'backslashreplace'`` error handling (for 452 :term:`text encodings <text encoding>` only): malformed data is 453 replaced by a backslashed escape sequence. 454 455.. function:: namereplace_errors(exception) 456 457 Implements the ``'namereplace'`` error handling (for encoding with 458 :term:`text encodings <text encoding>` only): the 459 unencodable character is replaced by a ``\N{...}`` escape sequence. 460 461 .. versionadded:: 3.5 462 463 464.. _codec-objects: 465 466Stateless Encoding and Decoding 467^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 468 469The base :class:`Codec` class defines these methods which also define the 470function interfaces of the stateless encoder and decoder: 471 472 473.. method:: Codec.encode(input[, errors]) 474 475 Encodes the object *input* and returns a tuple (output object, length consumed). 476 For instance, :term:`text encoding` converts 477 a string object to a bytes object using a particular 478 character set encoding (e.g., ``cp1252`` or ``iso-8859-1``). 479 480 The *errors* argument defines the error handling to apply. 481 It defaults to ``'strict'`` handling. 482 483 The method may not store state in the :class:`Codec` instance. Use 484 :class:`StreamWriter` for codecs which have to keep state in order to make 485 encoding efficient. 486 487 The encoder must be able to handle zero length input and return an empty object 488 of the output object type in this situation. 489 490 491.. method:: Codec.decode(input[, errors]) 492 493 Decodes the object *input* and returns a tuple (output object, length 494 consumed). For instance, for a :term:`text encoding`, decoding converts 495 a bytes object encoded using a particular 496 character set encoding to a string object. 497 498 For text encodings and bytes-to-bytes codecs, 499 *input* must be a bytes object or one which provides the read-only 500 buffer interface -- for example, buffer objects and memory mapped files. 501 502 The *errors* argument defines the error handling to apply. 503 It defaults to ``'strict'`` handling. 504 505 The method may not store state in the :class:`Codec` instance. Use 506 :class:`StreamReader` for codecs which have to keep state in order to make 507 decoding efficient. 508 509 The decoder must be able to handle zero length input and return an empty object 510 of the output object type in this situation. 511 512 513Incremental Encoding and Decoding 514^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 515 516The :class:`IncrementalEncoder` and :class:`IncrementalDecoder` classes provide 517the basic interface for incremental encoding and decoding. Encoding/decoding the 518input isn't done with one call to the stateless encoder/decoder function, but 519with multiple calls to the 520:meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method of 521the incremental encoder/decoder. The incremental encoder/decoder keeps track of 522the encoding/decoding process during method calls. 523 524The joined output of calls to the 525:meth:`~IncrementalEncoder.encode`/:meth:`~IncrementalDecoder.decode` method is 526the same as if all the single inputs were joined into one, and this input was 527encoded/decoded with the stateless encoder/decoder. 528 529 530.. _incremental-encoder-objects: 531 532IncrementalEncoder Objects 533~~~~~~~~~~~~~~~~~~~~~~~~~~ 534 535The :class:`IncrementalEncoder` class is used for encoding an input in multiple 536steps. It defines the following methods which every incremental encoder must 537define in order to be compatible with the Python codec registry. 538 539 540.. class:: IncrementalEncoder(errors='strict') 541 542 Constructor for an :class:`IncrementalEncoder` instance. 543 544 All incremental encoders must provide this constructor interface. They are free 545 to add additional keyword arguments, but only the ones defined here are used by 546 the Python codec registry. 547 548 The :class:`IncrementalEncoder` may implement different error handling schemes 549 by providing the *errors* keyword argument. See :ref:`error-handlers` for 550 possible values. 551 552 The *errors* argument will be assigned to an attribute of the same name. 553 Assigning to this attribute makes it possible to switch between different error 554 handling strategies during the lifetime of the :class:`IncrementalEncoder` 555 object. 556 557 558 .. method:: encode(object[, final]) 559 560 Encodes *object* (taking the current state of the encoder into account) 561 and returns the resulting encoded object. If this is the last call to 562 :meth:`encode` *final* must be true (the default is false). 563 564 565 .. method:: reset() 566 567 Reset the encoder to the initial state. The output is discarded: call 568 ``.encode(object, final=True)``, passing an empty byte or text string 569 if necessary, to reset the encoder and to get the output. 570 571 572 .. method:: getstate() 573 574 Return the current state of the encoder which must be an integer. The 575 implementation should make sure that ``0`` is the most common 576 state. (States that are more complicated than integers can be converted 577 into an integer by marshaling/pickling the state and encoding the bytes 578 of the resulting string into an integer.) 579 580 581 .. method:: setstate(state) 582 583 Set the state of the encoder to *state*. *state* must be an encoder state 584 returned by :meth:`getstate`. 585 586 587.. _incremental-decoder-objects: 588 589IncrementalDecoder Objects 590~~~~~~~~~~~~~~~~~~~~~~~~~~ 591 592The :class:`IncrementalDecoder` class is used for decoding an input in multiple 593steps. It defines the following methods which every incremental decoder must 594define in order to be compatible with the Python codec registry. 595 596 597.. class:: IncrementalDecoder(errors='strict') 598 599 Constructor for an :class:`IncrementalDecoder` instance. 600 601 All incremental decoders must provide this constructor interface. They are free 602 to add additional keyword arguments, but only the ones defined here are used by 603 the Python codec registry. 604 605 The :class:`IncrementalDecoder` may implement different error handling schemes 606 by providing the *errors* keyword argument. See :ref:`error-handlers` for 607 possible values. 608 609 The *errors* argument will be assigned to an attribute of the same name. 610 Assigning to this attribute makes it possible to switch between different error 611 handling strategies during the lifetime of the :class:`IncrementalDecoder` 612 object. 613 614 615 .. method:: decode(object[, final]) 616 617 Decodes *object* (taking the current state of the decoder into account) 618 and returns the resulting decoded object. If this is the last call to 619 :meth:`decode` *final* must be true (the default is false). If *final* is 620 true the decoder must decode the input completely and must flush all 621 buffers. If this isn't possible (e.g. because of incomplete byte sequences 622 at the end of the input) it must initiate error handling just like in the 623 stateless case (which might raise an exception). 624 625 626 .. method:: reset() 627 628 Reset the decoder to the initial state. 629 630 631 .. method:: getstate() 632 633 Return the current state of the decoder. This must be a tuple with two 634 items, the first must be the buffer containing the still undecoded 635 input. The second must be an integer and can be additional state 636 info. (The implementation should make sure that ``0`` is the most common 637 additional state info.) If this additional state info is ``0`` it must be 638 possible to set the decoder to the state which has no input buffered and 639 ``0`` as the additional state info, so that feeding the previously 640 buffered input to the decoder returns it to the previous state without 641 producing any output. (Additional state info that is more complicated than 642 integers can be converted into an integer by marshaling/pickling the info 643 and encoding the bytes of the resulting string into an integer.) 644 645 646 .. method:: setstate(state) 647 648 Set the state of the decoder to *state*. *state* must be a decoder state 649 returned by :meth:`getstate`. 650 651 652Stream Encoding and Decoding 653^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 654 655 656The :class:`StreamWriter` and :class:`StreamReader` classes provide generic 657working interfaces which can be used to implement new encoding submodules very 658easily. See :mod:`encodings.utf_8` for an example of how this is done. 659 660 661.. _stream-writer-objects: 662 663StreamWriter Objects 664~~~~~~~~~~~~~~~~~~~~ 665 666The :class:`StreamWriter` class is a subclass of :class:`Codec` and defines the 667following methods which every stream writer must define in order to be 668compatible with the Python codec registry. 669 670 671.. class:: StreamWriter(stream, errors='strict') 672 673 Constructor for a :class:`StreamWriter` instance. 674 675 All stream writers must provide this constructor interface. They are free to add 676 additional keyword arguments, but only the ones defined here are used by the 677 Python codec registry. 678 679 The *stream* argument must be a file-like object open for writing 680 text or binary data, as appropriate for the specific codec. 681 682 The :class:`StreamWriter` may implement different error handling schemes by 683 providing the *errors* keyword argument. See :ref:`error-handlers` for 684 the standard error handlers the underlying stream codec may support. 685 686 The *errors* argument will be assigned to an attribute of the same name. 687 Assigning to this attribute makes it possible to switch between different error 688 handling strategies during the lifetime of the :class:`StreamWriter` object. 689 690 .. method:: write(object) 691 692 Writes the object's contents encoded to the stream. 693 694 695 .. method:: writelines(list) 696 697 Writes the concatenated iterable of strings to the stream (possibly by reusing 698 the :meth:`write` method). Infinite or 699 very large iterables are not supported. The standard bytes-to-bytes codecs 700 do not support this method. 701 702 703 .. method:: reset() 704 705 Resets the codec buffers used for keeping internal state. 706 707 Calling this method should ensure that the data on the output is put into 708 a clean state that allows appending of new fresh data without having to 709 rescan the whole stream to recover state. 710 711 712In addition to the above methods, the :class:`StreamWriter` must also inherit 713all other methods and attributes from the underlying stream. 714 715 716.. _stream-reader-objects: 717 718StreamReader Objects 719~~~~~~~~~~~~~~~~~~~~ 720 721The :class:`StreamReader` class is a subclass of :class:`Codec` and defines the 722following methods which every stream reader must define in order to be 723compatible with the Python codec registry. 724 725 726.. class:: StreamReader(stream, errors='strict') 727 728 Constructor for a :class:`StreamReader` instance. 729 730 All stream readers must provide this constructor interface. They are free to add 731 additional keyword arguments, but only the ones defined here are used by the 732 Python codec registry. 733 734 The *stream* argument must be a file-like object open for reading 735 text or binary data, as appropriate for the specific codec. 736 737 The :class:`StreamReader` may implement different error handling schemes by 738 providing the *errors* keyword argument. See :ref:`error-handlers` for 739 the standard error handlers the underlying stream codec may support. 740 741 The *errors* argument will be assigned to an attribute of the same name. 742 Assigning to this attribute makes it possible to switch between different error 743 handling strategies during the lifetime of the :class:`StreamReader` object. 744 745 The set of allowed values for the *errors* argument can be extended with 746 :func:`register_error`. 747 748 749 .. method:: read([size[, chars, [firstline]]]) 750 751 Decodes data from the stream and returns the resulting object. 752 753 The *chars* argument indicates the number of decoded 754 code points or bytes to return. The :func:`read` method will 755 never return more data than requested, but it might return less, 756 if there is not enough available. 757 758 The *size* argument indicates the approximate maximum 759 number of encoded bytes or code points to read 760 for decoding. The decoder can modify this setting as 761 appropriate. The default value -1 indicates to read and decode as much as 762 possible. This parameter is intended to 763 prevent having to decode huge files in one step. 764 765 The *firstline* flag indicates that 766 it would be sufficient to only return the first 767 line, if there are decoding errors on later lines. 768 769 The method should use a greedy read strategy meaning that it should read 770 as much data as is allowed within the definition of the encoding and the 771 given size, e.g. if optional encoding endings or state markers are 772 available on the stream, these should be read too. 773 774 775 .. method:: readline([size[, keepends]]) 776 777 Read one line from the input stream and return the decoded data. 778 779 *size*, if given, is passed as size argument to the stream's 780 :meth:`read` method. 781 782 If *keepends* is false line-endings will be stripped from the lines 783 returned. 784 785 786 .. method:: readlines([sizehint[, keepends]]) 787 788 Read all lines available on the input stream and return them as a list of 789 lines. 790 791 Line-endings are implemented using the codec's :meth:`decode` method and 792 are included in the list entries if *keepends* is true. 793 794 *sizehint*, if given, is passed as the *size* argument to the stream's 795 :meth:`read` method. 796 797 798 .. method:: reset() 799 800 Resets the codec buffers used for keeping internal state. 801 802 Note that no stream repositioning should take place. This method is 803 primarily intended to be able to recover from decoding errors. 804 805 806In addition to the above methods, the :class:`StreamReader` must also inherit 807all other methods and attributes from the underlying stream. 808 809.. _stream-reader-writer: 810 811StreamReaderWriter Objects 812~~~~~~~~~~~~~~~~~~~~~~~~~~ 813 814The :class:`StreamReaderWriter` is a convenience class that allows wrapping 815streams which work in both read and write modes. 816 817The design is such that one can use the factory functions returned by the 818:func:`lookup` function to construct the instance. 819 820 821.. class:: StreamReaderWriter(stream, Reader, Writer, errors='strict') 822 823 Creates a :class:`StreamReaderWriter` instance. *stream* must be a file-like 824 object. *Reader* and *Writer* must be factory functions or classes providing the 825 :class:`StreamReader` and :class:`StreamWriter` interface resp. Error handling 826 is done in the same way as defined for the stream readers and writers. 827 828:class:`StreamReaderWriter` instances define the combined interfaces of 829:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other 830methods and attributes from the underlying stream. 831 832 833.. _stream-recoder-objects: 834 835StreamRecoder Objects 836~~~~~~~~~~~~~~~~~~~~~ 837 838The :class:`StreamRecoder` translates data from one encoding to another, 839which is sometimes useful when dealing with different encoding environments. 840 841The design is such that one can use the factory functions returned by the 842:func:`lookup` function to construct the instance. 843 844 845.. class:: StreamRecoder(stream, encode, decode, Reader, Writer, errors='strict') 846 847 Creates a :class:`StreamRecoder` instance which implements a two-way conversion: 848 *encode* and *decode* work on the frontend — the data visible to 849 code calling :meth:`read` and :meth:`write`, while *Reader* and *Writer* 850 work on the backend — the data in *stream*. 851 852 You can use these objects to do transparent transcodings, e.g., from Latin-1 853 to UTF-8 and back. 854 855 The *stream* argument must be a file-like object. 856 857 The *encode* and *decode* arguments must 858 adhere to the :class:`Codec` interface. *Reader* and 859 *Writer* must be factory functions or classes providing objects of the 860 :class:`StreamReader` and :class:`StreamWriter` interface respectively. 861 862 Error handling is done in the same way as defined for the stream readers and 863 writers. 864 865 866:class:`StreamRecoder` instances define the combined interfaces of 867:class:`StreamReader` and :class:`StreamWriter` classes. They inherit all other 868methods and attributes from the underlying stream. 869 870 871.. _encodings-overview: 872 873Encodings and Unicode 874--------------------- 875 876Strings are stored internally as sequences of code points in 877range ``0x0``--``0x10FFFF``. (See :pep:`393` for 878more details about the implementation.) 879Once a string object is used outside of CPU and memory, endianness 880and how these arrays are stored as bytes become an issue. As with other 881codecs, serialising a string into a sequence of bytes is known as *encoding*, 882and recreating the string from the sequence of bytes is known as *decoding*. 883There are a variety of different text serialisation codecs, which are 884collectivity referred to as :term:`text encodings <text encoding>`. 885 886The simplest text encoding (called ``'latin-1'`` or ``'iso-8859-1'``) maps 887the code points 0--255 to the bytes ``0x0``--``0xff``, which means that a string 888object that contains code points above ``U+00FF`` can't be encoded with this 889codec. Doing so will raise a :exc:`UnicodeEncodeError` that looks 890like the following (although the details of the error message may differ): 891``UnicodeEncodeError: 'latin-1' codec can't encode character '\u1234' in 892position 3: ordinal not in range(256)``. 893 894There's another group of encodings (the so called charmap encodings) that choose 895a different subset of all Unicode code points and how these code points are 896mapped to the bytes ``0x0``--``0xff``. To see how this is done simply open 897e.g. :file:`encodings/cp1252.py` (which is an encoding that is used primarily on 898Windows). There's a string constant with 256 characters that shows you which 899character is mapped to which byte value. 900 901All of these encodings can only encode 256 of the 1114112 code points 902defined in Unicode. A simple and straightforward way that can store each Unicode 903code point, is to store each code point as four consecutive bytes. There are two 904possibilities: store the bytes in big endian or in little endian order. These 905two encodings are called ``UTF-32-BE`` and ``UTF-32-LE`` respectively. Their 906disadvantage is that if e.g. you use ``UTF-32-BE`` on a little endian machine you 907will always have to swap bytes on encoding and decoding. ``UTF-32`` avoids this 908problem: bytes will always be in natural endianness. When these bytes are read 909by a CPU with a different endianness, then bytes have to be swapped though. To 910be able to detect the endianness of a ``UTF-16`` or ``UTF-32`` byte sequence, 911there's the so called BOM ("Byte Order Mark"). This is the Unicode character 912``U+FEFF``. This character can be prepended to every ``UTF-16`` or ``UTF-32`` 913byte sequence. The byte swapped version of this character (``0xFFFE``) is an 914illegal character that may not appear in a Unicode text. So when the 915first character in an ``UTF-16`` or ``UTF-32`` byte sequence 916appears to be a ``U+FFFE`` the bytes have to be swapped on decoding. 917Unfortunately the character ``U+FEFF`` had a second purpose as 918a ``ZERO WIDTH NO-BREAK SPACE``: a character that has no width and doesn't allow 919a word to be split. It can e.g. be used to give hints to a ligature algorithm. 920With Unicode 4.0 using ``U+FEFF`` as a ``ZERO WIDTH NO-BREAK SPACE`` has been 921deprecated (with ``U+2060`` (``WORD JOINER``) assuming this role). Nevertheless 922Unicode software still must be able to handle ``U+FEFF`` in both roles: as a BOM 923it's a device to determine the storage layout of the encoded bytes, and vanishes 924once the byte sequence has been decoded into a string; as a ``ZERO WIDTH 925NO-BREAK SPACE`` it's a normal character that will be decoded like any other. 926 927There's another encoding that is able to encode the full range of Unicode 928characters: UTF-8. UTF-8 is an 8-bit encoding, which means there are no issues 929with byte order in UTF-8. Each byte in a UTF-8 byte sequence consists of two 930parts: marker bits (the most significant bits) and payload bits. The marker bits 931are a sequence of zero to four ``1`` bits followed by a ``0`` bit. Unicode characters are 932encoded like this (with x being payload bits, which when concatenated give the 933Unicode character): 934 935+-----------------------------------+----------------------------------------------+ 936| Range | Encoding | 937+===================================+==============================================+ 938| ``U-00000000`` ... ``U-0000007F`` | 0xxxxxxx | 939+-----------------------------------+----------------------------------------------+ 940| ``U-00000080`` ... ``U-000007FF`` | 110xxxxx 10xxxxxx | 941+-----------------------------------+----------------------------------------------+ 942| ``U-00000800`` ... ``U-0000FFFF`` | 1110xxxx 10xxxxxx 10xxxxxx | 943+-----------------------------------+----------------------------------------------+ 944| ``U-00010000`` ... ``U-0010FFFF`` | 11110xxx 10xxxxxx 10xxxxxx 10xxxxxx | 945+-----------------------------------+----------------------------------------------+ 946 947The least significant bit of the Unicode character is the rightmost x bit. 948 949As UTF-8 is an 8-bit encoding no BOM is required and any ``U+FEFF`` character in 950the decoded string (even if it's the first character) is treated as a ``ZERO 951WIDTH NO-BREAK SPACE``. 952 953Without external information it's impossible to reliably determine which 954encoding was used for encoding a string. Each charmap encoding can 955decode any random byte sequence. However that's not possible with UTF-8, as 956UTF-8 byte sequences have a structure that doesn't allow arbitrary byte 957sequences. To increase the reliability with which a UTF-8 encoding can be 958detected, Microsoft invented a variant of UTF-8 (that Python 2.5 calls 959``"utf-8-sig"``) for its Notepad program: Before any of the Unicode characters 960is written to the file, a UTF-8 encoded BOM (which looks like this as a byte 961sequence: ``0xef``, ``0xbb``, ``0xbf``) is written. As it's rather improbable 962that any charmap encoded file starts with these byte values (which would e.g. 963map to 964 965 | LATIN SMALL LETTER I WITH DIAERESIS 966 | RIGHT-POINTING DOUBLE ANGLE QUOTATION MARK 967 | INVERTED QUESTION MARK 968 969in iso-8859-1), this increases the probability that a ``utf-8-sig`` encoding can be 970correctly guessed from the byte sequence. So here the BOM is not used to be able 971to determine the byte order used for generating the byte sequence, but as a 972signature that helps in guessing the encoding. On encoding the utf-8-sig codec 973will write ``0xef``, ``0xbb``, ``0xbf`` as the first three bytes to the file. On 974decoding ``utf-8-sig`` will skip those three bytes if they appear as the first 975three bytes in the file. In UTF-8, the use of the BOM is discouraged and 976should generally be avoided. 977 978 979.. _standard-encodings: 980 981Standard Encodings 982------------------ 983 984Python comes with a number of codecs built-in, either implemented as C functions 985or with dictionaries as mapping tables. The following table lists the codecs by 986name, together with a few common aliases, and the languages for which the 987encoding is likely used. Neither the list of aliases nor the list of languages 988is meant to be exhaustive. Notice that spelling alternatives that only differ in 989case or use a hyphen instead of an underscore are also valid aliases; therefore, 990e.g. ``'utf-8'`` is a valid alias for the ``'utf_8'`` codec. 991 992.. impl-detail:: 993 994 Some common encodings can bypass the codecs lookup machinery to 995 improve performance. These optimization opportunities are only 996 recognized by CPython for a limited set of (case insensitive) 997 aliases: utf-8, utf8, latin-1, latin1, iso-8859-1, iso8859-1, mbcs 998 (Windows only), ascii, us-ascii, utf-16, utf16, utf-32, utf32, and 999 the same using underscores instead of dashes. Using alternative 1000 aliases for these encodings may result in slower execution. 1001 1002 .. versionchanged:: 3.6 1003 Optimization opportunity recognized for us-ascii. 1004 1005Many of the character sets support the same languages. They vary in individual 1006characters (e.g. whether the EURO SIGN is supported or not), and in the 1007assignment of characters to code positions. For the European languages in 1008particular, the following variants typically exist: 1009 1010* an ISO 8859 codeset 1011 1012* a Microsoft Windows code page, which is typically derived from an 8859 codeset, 1013 but replaces control characters with additional graphic characters 1014 1015* an IBM EBCDIC code page 1016 1017* an IBM PC code page, which is ASCII compatible 1018 1019.. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| 1020 1021+-----------------+--------------------------------+--------------------------------+ 1022| Codec | Aliases | Languages | 1023+=================+================================+================================+ 1024| ascii | 646, us-ascii | English | 1025+-----------------+--------------------------------+--------------------------------+ 1026| big5 | big5-tw, csbig5 | Traditional Chinese | 1027+-----------------+--------------------------------+--------------------------------+ 1028| big5hkscs | big5-hkscs, hkscs | Traditional Chinese | 1029+-----------------+--------------------------------+--------------------------------+ 1030| cp037 | IBM037, IBM039 | English | 1031+-----------------+--------------------------------+--------------------------------+ 1032| cp273 | 273, IBM273, csIBM273 | German | 1033| | | | 1034| | | .. versionadded:: 3.4 | 1035+-----------------+--------------------------------+--------------------------------+ 1036| cp424 | EBCDIC-CP-HE, IBM424 | Hebrew | 1037+-----------------+--------------------------------+--------------------------------+ 1038| cp437 | 437, IBM437 | English | 1039+-----------------+--------------------------------+--------------------------------+ 1040| cp500 | EBCDIC-CP-BE, EBCDIC-CP-CH, | Western Europe | 1041| | IBM500 | | 1042+-----------------+--------------------------------+--------------------------------+ 1043| cp720 | | Arabic | 1044+-----------------+--------------------------------+--------------------------------+ 1045| cp737 | | Greek | 1046+-----------------+--------------------------------+--------------------------------+ 1047| cp775 | IBM775 | Baltic languages | 1048+-----------------+--------------------------------+--------------------------------+ 1049| cp850 | 850, IBM850 | Western Europe | 1050+-----------------+--------------------------------+--------------------------------+ 1051| cp852 | 852, IBM852 | Central and Eastern Europe | 1052+-----------------+--------------------------------+--------------------------------+ 1053| cp855 | 855, IBM855 | Bulgarian, Byelorussian, | 1054| | | Macedonian, Russian, Serbian | 1055+-----------------+--------------------------------+--------------------------------+ 1056| cp856 | | Hebrew | 1057+-----------------+--------------------------------+--------------------------------+ 1058| cp857 | 857, IBM857 | Turkish | 1059+-----------------+--------------------------------+--------------------------------+ 1060| cp858 | 858, IBM858 | Western Europe | 1061+-----------------+--------------------------------+--------------------------------+ 1062| cp860 | 860, IBM860 | Portuguese | 1063+-----------------+--------------------------------+--------------------------------+ 1064| cp861 | 861, CP-IS, IBM861 | Icelandic | 1065+-----------------+--------------------------------+--------------------------------+ 1066| cp862 | 862, IBM862 | Hebrew | 1067+-----------------+--------------------------------+--------------------------------+ 1068| cp863 | 863, IBM863 | Canadian | 1069+-----------------+--------------------------------+--------------------------------+ 1070| cp864 | IBM864 | Arabic | 1071+-----------------+--------------------------------+--------------------------------+ 1072| cp865 | 865, IBM865 | Danish, Norwegian | 1073+-----------------+--------------------------------+--------------------------------+ 1074| cp866 | 866, IBM866 | Russian | 1075+-----------------+--------------------------------+--------------------------------+ 1076| cp869 | 869, CP-GR, IBM869 | Greek | 1077+-----------------+--------------------------------+--------------------------------+ 1078| cp874 | | Thai | 1079+-----------------+--------------------------------+--------------------------------+ 1080| cp875 | | Greek | 1081+-----------------+--------------------------------+--------------------------------+ 1082| cp932 | 932, ms932, mskanji, ms-kanji | Japanese | 1083+-----------------+--------------------------------+--------------------------------+ 1084| cp949 | 949, ms949, uhc | Korean | 1085+-----------------+--------------------------------+--------------------------------+ 1086| cp950 | 950, ms950 | Traditional Chinese | 1087+-----------------+--------------------------------+--------------------------------+ 1088| cp1006 | | Urdu | 1089+-----------------+--------------------------------+--------------------------------+ 1090| cp1026 | ibm1026 | Turkish | 1091+-----------------+--------------------------------+--------------------------------+ 1092| cp1125 | 1125, ibm1125, cp866u, ruscii | Ukrainian | 1093| | | | 1094| | | .. versionadded:: 3.4 | 1095+-----------------+--------------------------------+--------------------------------+ 1096| cp1140 | ibm1140 | Western Europe | 1097+-----------------+--------------------------------+--------------------------------+ 1098| cp1250 | windows-1250 | Central and Eastern Europe | 1099+-----------------+--------------------------------+--------------------------------+ 1100| cp1251 | windows-1251 | Bulgarian, Byelorussian, | 1101| | | Macedonian, Russian, Serbian | 1102+-----------------+--------------------------------+--------------------------------+ 1103| cp1252 | windows-1252 | Western Europe | 1104+-----------------+--------------------------------+--------------------------------+ 1105| cp1253 | windows-1253 | Greek | 1106+-----------------+--------------------------------+--------------------------------+ 1107| cp1254 | windows-1254 | Turkish | 1108+-----------------+--------------------------------+--------------------------------+ 1109| cp1255 | windows-1255 | Hebrew | 1110+-----------------+--------------------------------+--------------------------------+ 1111| cp1256 | windows-1256 | Arabic | 1112+-----------------+--------------------------------+--------------------------------+ 1113| cp1257 | windows-1257 | Baltic languages | 1114+-----------------+--------------------------------+--------------------------------+ 1115| cp1258 | windows-1258 | Vietnamese | 1116+-----------------+--------------------------------+--------------------------------+ 1117| euc_jp | eucjp, ujis, u-jis | Japanese | 1118+-----------------+--------------------------------+--------------------------------+ 1119| euc_jis_2004 | jisx0213, eucjis2004 | Japanese | 1120+-----------------+--------------------------------+--------------------------------+ 1121| euc_jisx0213 | eucjisx0213 | Japanese | 1122+-----------------+--------------------------------+--------------------------------+ 1123| euc_kr | euckr, korean, ksc5601, | Korean | 1124| | ks_c-5601, ks_c-5601-1987, | | 1125| | ksx1001, ks_x-1001 | | 1126+-----------------+--------------------------------+--------------------------------+ 1127| gb2312 | chinese, csiso58gb231280, | Simplified Chinese | 1128| | euc-cn, euccn, eucgb2312-cn, | | 1129| | gb2312-1980, gb2312-80, | | 1130| | iso-ir-58 | | 1131+-----------------+--------------------------------+--------------------------------+ 1132| gbk | 936, cp936, ms936 | Unified Chinese | 1133+-----------------+--------------------------------+--------------------------------+ 1134| gb18030 | gb18030-2000 | Unified Chinese | 1135+-----------------+--------------------------------+--------------------------------+ 1136| hz | hzgb, hz-gb, hz-gb-2312 | Simplified Chinese | 1137+-----------------+--------------------------------+--------------------------------+ 1138| iso2022_jp | csiso2022jp, iso2022jp, | Japanese | 1139| | iso-2022-jp | | 1140+-----------------+--------------------------------+--------------------------------+ 1141| iso2022_jp_1 | iso2022jp-1, iso-2022-jp-1 | Japanese | 1142+-----------------+--------------------------------+--------------------------------+ 1143| iso2022_jp_2 | iso2022jp-2, iso-2022-jp-2 | Japanese, Korean, Simplified | 1144| | | Chinese, Western Europe, Greek | 1145+-----------------+--------------------------------+--------------------------------+ 1146| iso2022_jp_2004 | iso2022jp-2004, | Japanese | 1147| | iso-2022-jp-2004 | | 1148+-----------------+--------------------------------+--------------------------------+ 1149| iso2022_jp_3 | iso2022jp-3, iso-2022-jp-3 | Japanese | 1150+-----------------+--------------------------------+--------------------------------+ 1151| iso2022_jp_ext | iso2022jp-ext, iso-2022-jp-ext | Japanese | 1152+-----------------+--------------------------------+--------------------------------+ 1153| iso2022_kr | csiso2022kr, iso2022kr, | Korean | 1154| | iso-2022-kr | | 1155+-----------------+--------------------------------+--------------------------------+ 1156| latin_1 | iso-8859-1, iso8859-1, 8859, | Western Europe | 1157| | cp819, latin, latin1, L1 | | 1158+-----------------+--------------------------------+--------------------------------+ 1159| iso8859_2 | iso-8859-2, latin2, L2 | Central and Eastern Europe | 1160+-----------------+--------------------------------+--------------------------------+ 1161| iso8859_3 | iso-8859-3, latin3, L3 | Esperanto, Maltese | 1162+-----------------+--------------------------------+--------------------------------+ 1163| iso8859_4 | iso-8859-4, latin4, L4 | Baltic languages | 1164+-----------------+--------------------------------+--------------------------------+ 1165| iso8859_5 | iso-8859-5, cyrillic | Bulgarian, Byelorussian, | 1166| | | Macedonian, Russian, Serbian | 1167+-----------------+--------------------------------+--------------------------------+ 1168| iso8859_6 | iso-8859-6, arabic | Arabic | 1169+-----------------+--------------------------------+--------------------------------+ 1170| iso8859_7 | iso-8859-7, greek, greek8 | Greek | 1171+-----------------+--------------------------------+--------------------------------+ 1172| iso8859_8 | iso-8859-8, hebrew | Hebrew | 1173+-----------------+--------------------------------+--------------------------------+ 1174| iso8859_9 | iso-8859-9, latin5, L5 | Turkish | 1175+-----------------+--------------------------------+--------------------------------+ 1176| iso8859_10 | iso-8859-10, latin6, L6 | Nordic languages | 1177+-----------------+--------------------------------+--------------------------------+ 1178| iso8859_11 | iso-8859-11, thai | Thai languages | 1179+-----------------+--------------------------------+--------------------------------+ 1180| iso8859_13 | iso-8859-13, latin7, L7 | Baltic languages | 1181+-----------------+--------------------------------+--------------------------------+ 1182| iso8859_14 | iso-8859-14, latin8, L8 | Celtic languages | 1183+-----------------+--------------------------------+--------------------------------+ 1184| iso8859_15 | iso-8859-15, latin9, L9 | Western Europe | 1185+-----------------+--------------------------------+--------------------------------+ 1186| iso8859_16 | iso-8859-16, latin10, L10 | South-Eastern Europe | 1187+-----------------+--------------------------------+--------------------------------+ 1188| johab | cp1361, ms1361 | Korean | 1189+-----------------+--------------------------------+--------------------------------+ 1190| koi8_r | | Russian | 1191+-----------------+--------------------------------+--------------------------------+ 1192| koi8_t | | Tajik | 1193| | | | 1194| | | .. versionadded:: 3.5 | 1195+-----------------+--------------------------------+--------------------------------+ 1196| koi8_u | | Ukrainian | 1197+-----------------+--------------------------------+--------------------------------+ 1198| kz1048 | kz_1048, strk1048_2002, rk1048 | Kazakh | 1199| | | | 1200| | | .. versionadded:: 3.5 | 1201+-----------------+--------------------------------+--------------------------------+ 1202| mac_cyrillic | maccyrillic | Bulgarian, Byelorussian, | 1203| | | Macedonian, Russian, Serbian | 1204+-----------------+--------------------------------+--------------------------------+ 1205| mac_greek | macgreek | Greek | 1206+-----------------+--------------------------------+--------------------------------+ 1207| mac_iceland | maciceland | Icelandic | 1208+-----------------+--------------------------------+--------------------------------+ 1209| mac_latin2 | maclatin2, maccentraleurope, | Central and Eastern Europe | 1210| | mac_centeuro | | 1211+-----------------+--------------------------------+--------------------------------+ 1212| mac_roman | macroman, macintosh | Western Europe | 1213+-----------------+--------------------------------+--------------------------------+ 1214| mac_turkish | macturkish | Turkish | 1215+-----------------+--------------------------------+--------------------------------+ 1216| ptcp154 | csptcp154, pt154, cp154, | Kazakh | 1217| | cyrillic-asian | | 1218+-----------------+--------------------------------+--------------------------------+ 1219| shift_jis | csshiftjis, shiftjis, sjis, | Japanese | 1220| | s_jis | | 1221+-----------------+--------------------------------+--------------------------------+ 1222| shift_jis_2004 | shiftjis2004, sjis_2004, | Japanese | 1223| | sjis2004 | | 1224+-----------------+--------------------------------+--------------------------------+ 1225| shift_jisx0213 | shiftjisx0213, sjisx0213, | Japanese | 1226| | s_jisx0213 | | 1227+-----------------+--------------------------------+--------------------------------+ 1228| utf_32 | U32, utf32 | all languages | 1229+-----------------+--------------------------------+--------------------------------+ 1230| utf_32_be | UTF-32BE | all languages | 1231+-----------------+--------------------------------+--------------------------------+ 1232| utf_32_le | UTF-32LE | all languages | 1233+-----------------+--------------------------------+--------------------------------+ 1234| utf_16 | U16, utf16 | all languages | 1235+-----------------+--------------------------------+--------------------------------+ 1236| utf_16_be | UTF-16BE | all languages | 1237+-----------------+--------------------------------+--------------------------------+ 1238| utf_16_le | UTF-16LE | all languages | 1239+-----------------+--------------------------------+--------------------------------+ 1240| utf_7 | U7, unicode-1-1-utf-7 | all languages | 1241+-----------------+--------------------------------+--------------------------------+ 1242| utf_8 | U8, UTF, utf8, cp65001 | all languages | 1243+-----------------+--------------------------------+--------------------------------+ 1244| utf_8_sig | | all languages | 1245+-----------------+--------------------------------+--------------------------------+ 1246 1247.. versionchanged:: 3.4 1248 The utf-16\* and utf-32\* encoders no longer allow surrogate code points 1249 (``U+D800``--``U+DFFF``) to be encoded. 1250 The utf-32\* decoders no longer decode 1251 byte sequences that correspond to surrogate code points. 1252 1253.. versionchanged:: 3.8 1254 ``cp65001`` is now an alias to ``utf_8``. 1255 1256 1257Python Specific Encodings 1258------------------------- 1259 1260A number of predefined codecs are specific to Python, so their codec names have 1261no meaning outside Python. These are listed in the tables below based on the 1262expected input and output types (note that while text encodings are the most 1263common use case for codecs, the underlying codec infrastructure supports 1264arbitrary data transforms rather than just text encodings). For asymmetric 1265codecs, the stated meaning describes the encoding direction. 1266 1267Text Encodings 1268^^^^^^^^^^^^^^ 1269 1270The following codecs provide :class:`str` to :class:`bytes` encoding and 1271:term:`bytes-like object` to :class:`str` decoding, similar to the Unicode text 1272encodings. 1273 1274.. tabularcolumns:: |l|p{0.3\linewidth}|p{0.3\linewidth}| 1275 1276+--------------------+---------+---------------------------+ 1277| Codec | Aliases | Meaning | 1278+====================+=========+===========================+ 1279| idna | | Implement :rfc:`3490`, | 1280| | | see also | 1281| | | :mod:`encodings.idna`. | 1282| | | Only ``errors='strict'`` | 1283| | | is supported. | 1284+--------------------+---------+---------------------------+ 1285| mbcs | ansi, | Windows only: Encode the | 1286| | dbcs | operand according to the | 1287| | | ANSI codepage (CP_ACP). | 1288+--------------------+---------+---------------------------+ 1289| oem | | Windows only: Encode the | 1290| | | operand according to the | 1291| | | OEM codepage (CP_OEMCP). | 1292| | | | 1293| | | .. versionadded:: 3.6 | 1294+--------------------+---------+---------------------------+ 1295| palmos | | Encoding of PalmOS 3.5. | 1296+--------------------+---------+---------------------------+ 1297| punycode | | Implement :rfc:`3492`. | 1298| | | Stateful codecs are not | 1299| | | supported. | 1300+--------------------+---------+---------------------------+ 1301| raw_unicode_escape | | Latin-1 encoding with | 1302| | | ``\uXXXX`` and | 1303| | | ``\UXXXXXXXX`` for other | 1304| | | code points. Existing | 1305| | | backslashes are not | 1306| | | escaped in any way. | 1307| | | It is used in the Python | 1308| | | pickle protocol. | 1309+--------------------+---------+---------------------------+ 1310| undefined | | Raise an exception for | 1311| | | all conversions, even | 1312| | | empty strings. The error | 1313| | | handler is ignored. | 1314+--------------------+---------+---------------------------+ 1315| unicode_escape | | Encoding suitable as the | 1316| | | contents of a Unicode | 1317| | | literal in ASCII-encoded | 1318| | | Python source code, | 1319| | | except that quotes are | 1320| | | not escaped. Decode | 1321| | | from Latin-1 source code. | 1322| | | Beware that Python source | 1323| | | code actually uses UTF-8 | 1324| | | by default. | 1325+--------------------+---------+---------------------------+ 1326 1327.. versionchanged:: 3.8 1328 "unicode_internal" codec is removed. 1329 1330 1331.. _binary-transforms: 1332 1333Binary Transforms 1334^^^^^^^^^^^^^^^^^ 1335 1336The following codecs provide binary transforms: :term:`bytes-like object` 1337to :class:`bytes` mappings. They are not supported by :meth:`bytes.decode` 1338(which only produces :class:`str` output). 1339 1340 1341.. tabularcolumns:: |l|L|L|L| 1342 1343+----------------------+------------------+------------------------------+------------------------------+ 1344| Codec | Aliases | Meaning | Encoder / decoder | 1345+======================+==================+==============================+==============================+ 1346| base64_codec [#b64]_ | base64, base_64 | Convert the operand to | :meth:`base64.encodebytes` / | 1347| | | multiline MIME base64 (the | :meth:`base64.decodebytes` | 1348| | | result always includes a | | 1349| | | trailing ``'\n'``). | | 1350| | | | | 1351| | | .. versionchanged:: 3.4 | | 1352| | | accepts any | | 1353| | | :term:`bytes-like object` | | 1354| | | as input for encoding and | | 1355| | | decoding | | 1356+----------------------+------------------+------------------------------+------------------------------+ 1357| bz2_codec | bz2 | Compress the operand using | :meth:`bz2.compress` / | 1358| | | bz2. | :meth:`bz2.decompress` | 1359+----------------------+------------------+------------------------------+------------------------------+ 1360| hex_codec | hex | Convert the operand to | :meth:`binascii.b2a_hex` / | 1361| | | hexadecimal | :meth:`binascii.a2b_hex` | 1362| | | representation, with two | | 1363| | | digits per byte. | | 1364+----------------------+------------------+------------------------------+------------------------------+ 1365| quopri_codec | quopri, | Convert the operand to MIME | :meth:`quopri.encode` with | 1366| | quotedprintable, | quoted printable. | ``quotetabs=True`` / | 1367| | quoted_printable | | :meth:`quopri.decode` | 1368+----------------------+------------------+------------------------------+------------------------------+ 1369| uu_codec | uu | Convert the operand using | :meth:`uu.encode` / | 1370| | | uuencode. | :meth:`uu.decode` | 1371+----------------------+------------------+------------------------------+------------------------------+ 1372| zlib_codec | zip, zlib | Compress the operand using | :meth:`zlib.compress` / | 1373| | | gzip. | :meth:`zlib.decompress` | 1374+----------------------+------------------+------------------------------+------------------------------+ 1375 1376.. [#b64] In addition to :term:`bytes-like objects <bytes-like object>`, 1377 ``'base64_codec'`` also accepts ASCII-only instances of :class:`str` for 1378 decoding 1379 1380.. versionadded:: 3.2 1381 Restoration of the binary transforms. 1382 1383.. versionchanged:: 3.4 1384 Restoration of the aliases for the binary transforms. 1385 1386 1387.. _text-transforms: 1388 1389Text Transforms 1390^^^^^^^^^^^^^^^ 1391 1392The following codec provides a text transform: a :class:`str` to :class:`str` 1393mapping. It is not supported by :meth:`str.encode` (which only produces 1394:class:`bytes` output). 1395 1396.. tabularcolumns:: |l|l|L| 1397 1398+--------------------+---------+---------------------------+ 1399| Codec | Aliases | Meaning | 1400+====================+=========+===========================+ 1401| rot_13 | rot13 | Return the Caesar-cypher | 1402| | | encryption of the | 1403| | | operand. | 1404+--------------------+---------+---------------------------+ 1405 1406.. versionadded:: 3.2 1407 Restoration of the ``rot_13`` text transform. 1408 1409.. versionchanged:: 3.4 1410 Restoration of the ``rot13`` alias. 1411 1412 1413:mod:`encodings.idna` --- Internationalized Domain Names in Applications 1414------------------------------------------------------------------------ 1415 1416.. module:: encodings.idna 1417 :synopsis: Internationalized Domain Names implementation 1418.. moduleauthor:: Martin v. Löwis 1419 1420This module implements :rfc:`3490` (Internationalized Domain Names in 1421Applications) and :rfc:`3492` (Nameprep: A Stringprep Profile for 1422Internationalized Domain Names (IDN)). It builds upon the ``punycode`` encoding 1423and :mod:`stringprep`. 1424 1425If you need the IDNA 2008 standard from :rfc:`5891` and :rfc:`5895`, use the 1426third-party `idna module <https://pypi.org/project/idna/>_`. 1427 1428These RFCs together define a protocol to support non-ASCII characters in domain 1429names. A domain name containing non-ASCII characters (such as 1430``www.Alliancefrançaise.nu``) is converted into an ASCII-compatible encoding 1431(ACE, such as ``www.xn--alliancefranaise-npb.nu``). The ACE form of the domain 1432name is then used in all places where arbitrary characters are not allowed by 1433the protocol, such as DNS queries, HTTP :mailheader:`Host` fields, and so 1434on. This conversion is carried out in the application; if possible invisible to 1435the user: The application should transparently convert Unicode domain labels to 1436IDNA on the wire, and convert back ACE labels to Unicode before presenting them 1437to the user. 1438 1439Python supports this conversion in several ways: the ``idna`` codec performs 1440conversion between Unicode and ACE, separating an input string into labels 1441based on the separator characters defined in :rfc:`section 3.1 of RFC 3490 <3490#section-3.1>` 1442and converting each label to ACE as required, and conversely separating an input 1443byte string into labels based on the ``.`` separator and converting any ACE 1444labels found into unicode. Furthermore, the :mod:`socket` module 1445transparently converts Unicode host names to ACE, so that applications need not 1446be concerned about converting host names themselves when they pass them to the 1447socket module. On top of that, modules that have host names as function 1448parameters, such as :mod:`http.client` and :mod:`ftplib`, accept Unicode host 1449names (:mod:`http.client` then also transparently sends an IDNA hostname in the 1450:mailheader:`Host` field if it sends that field at all). 1451 1452When receiving host names from the wire (such as in reverse name lookup), no 1453automatic conversion to Unicode is performed: applications wishing to present 1454such host names to the user should decode them to Unicode. 1455 1456The module :mod:`encodings.idna` also implements the nameprep procedure, which 1457performs certain normalizations on host names, to achieve case-insensitivity of 1458international domain names, and to unify similar characters. The nameprep 1459functions can be used directly if desired. 1460 1461 1462.. function:: nameprep(label) 1463 1464 Return the nameprepped version of *label*. The implementation currently assumes 1465 query strings, so ``AllowUnassigned`` is true. 1466 1467 1468.. function:: ToASCII(label) 1469 1470 Convert a label to ASCII, as specified in :rfc:`3490`. ``UseSTD3ASCIIRules`` is 1471 assumed to be false. 1472 1473 1474.. function:: ToUnicode(label) 1475 1476 Convert a label to Unicode, as specified in :rfc:`3490`. 1477 1478 1479:mod:`encodings.mbcs` --- Windows ANSI codepage 1480----------------------------------------------- 1481 1482.. module:: encodings.mbcs 1483 :synopsis: Windows ANSI codepage 1484 1485This module implements the ANSI codepage (CP_ACP). 1486 1487.. availability:: Windows only. 1488 1489.. versionchanged:: 3.3 1490 Support any error handler. 1491 1492.. versionchanged:: 3.2 1493 Before 3.2, the *errors* argument was ignored; ``'replace'`` was always used 1494 to encode, and ``'ignore'`` to decode. 1495 1496 1497:mod:`encodings.utf_8_sig` --- UTF-8 codec with BOM signature 1498------------------------------------------------------------- 1499 1500.. module:: encodings.utf_8_sig 1501 :synopsis: UTF-8 codec with BOM signature 1502.. moduleauthor:: Walter Dörwald 1503 1504This module implements a variant of the UTF-8 codec. On encoding, a UTF-8 encoded 1505BOM will be prepended to the UTF-8 encoded bytes. For the stateful encoder this 1506is only done once (on the first write to the byte stream). On decoding, an 1507optional UTF-8 encoded BOM at the start of the data will be skipped. 1508