Lines Matching +full:object +full:- +full:copy
1 :mod:`pickle` --- Python object serialization
20 --------------
23 de-serializing a Python object structure. *"Pickling"* is the process
24 whereby a Python object hierarchy is converted into a byte stream, and
26 (from a :term:`binary file` or :term:`bytes-like object`) is converted
27 back into an object hierarchy. Pickling (and unpickling) is alternatively
43 you are processing untrusted data. See :ref:`comparison-with-json`.
47 ------------------------------------
60 so that later references to the same object won't be serialized again.
63 This has implications both for recursive objects and object sharing. Recursive
66 crash your Python interpreter. Object sharing happens when there are multiple
67 references to the same object in different places in the object hierarchy being
69 other references point to the master copy. Shared objects remain shared, which
72 * :mod:`marshal` cannot be used to serialize user-defined classes and their
75 when the object was stored.
80 serialization format in non-backwards compatible ways should the need arise.
87 .. _comparison-with-json:
93 `JSON (JavaScript Object Notation) <https://json.org>`_:
96 most of the time it is then encoded to ``utf-8``), while pickle is
99 * JSON is human-readable, while pickle is not;
102 while pickle is Python-specific;
104 * JSON, by default, can only represent a subset of the Python built-in
108 implementing :ref:`specific object APIs <pickle-inst>`);
118 .. _pickle-protocols:
121 ------------------
126 The data format used by :mod:`pickle` is Python-specific. This has the
129 non-Python programs may not be able to reconstruct pickled Python objects.
143 * Protocol version 0 is the original "human-readable" protocol and is
150 efficient pickling of :term:`new-style classes <new-style class>`. Refer to :pep:`307` for
155 the default protocol in Python 3.0--3.7.
163 * Protocol version 5 was added in Python 3.8. It adds support for out-of-band
164 data and speedup for in-band data. Refer to :pep:`574` for information about
172 object into a byte stream and it can transform the byte stream into an object
177 DBM-style database files.
181 ----------------
183 To serialize an object hierarchy, you simply call the :func:`dumps` function.
184 Similarly, to de-serialize a data stream, you call the :func:`loads` function.
185 However, if you want more control over serialization and de-serialization,
186 you can create a :class:`Pickler` or an :class:`Unpickler` object, respectively.
193 An integer, the highest :ref:`protocol version <pickle-protocols>`
200 An integer, the default :ref:`protocol version <pickle-protocols>` used
218 Write the pickled representation of the object *obj* to the open
219 :term:`file object` *file*. This is equivalent to
230 Return the pickled representation of the object *obj* as a :class:`bytes` object,
241 Read the pickled representation of an object from the open :term:`file object`
242 *file* and return the reconstituted object hierarchy specified therein.
247 of the object are ignored.
257 Return the reconstituted object hierarchy of the pickled representation
258 *data* of an object. *data* must be a :term:`bytes-like object`.
262 of the object are ignored.
280 Error raised when an unpicklable object is encountered by :class:`Pickler`.
283 Refer to :ref:`pickle-picklable` to learn what kinds of objects can be
288 Error raised when there is a problem unpickling an object, such as a data
309 argument. It can thus be an on-disk file opened for binary writing, an
310 :class:`io.BytesIO` instance, or any other custom object that meets this
322 (such as None), the given buffer is :ref:`out-of-band <pickle-oob>`;
323 otherwise the buffer is serialized in-band, i.e. inside the pickle stream.
333 Write the pickled representation of *obj* to the open file object given in
346 See :ref:`pickle-persistent` for details and examples of uses.
350 A pickler object's dispatch table is a registry of *reduction
358 By default, a pickler object will not have a
361 However, to customize the pickling for a specific pickler object
362 one can set the :attr:`dispatch_table` attribute to a dict-like
363 object. Alternatively, if a subclass of :class:`Pickler` has a
367 See :ref:`pickle-dispatch` for usage examples.
377 :attr:`dispatch_table`-registered reducers to pickle ``obj``.
388 self-referential objects, doing otherwise will cause :class:`Pickler` to
404 :class:`io.BufferedIOBase` interface. Thus *file* can be an on-disk file
405 opened for binary reading, an :class:`io.BytesIO` object, or any other
406 custom object that meets this interface.
412 pickle how to decode 8-bit string instances pickled by Python 2;
414 be 'bytes' to read these 8-bit string instances as bytes objects.
424 If *buffers* is not None, it should be an iterable of buffer-enabled
426 an :ref:`out-of-band <pickle-oob>` buffer view. Such buffers have been
427 given in order to the *buffer_callback* of a Pickler object.
434 Read the pickled representation of an object from the open file object
435 given in the constructor, and return the reconstituted object hierarchy
436 specified therein. Bytes past the pickled representation of the object
443 If defined, :meth:`persistent_load` should return the object specified by
447 See :ref:`pickle-persistent` for details and examples of uses.
451 Import *module* if necessary and return the object called *name* from it,
458 :ref:`pickle-restrict` for details.
460 .. audit-event:: pickle.find_class module,name pickle.Unpickler.find_class
465 :ref:`buffer-providing <bufferobjects>` object, such as a
466 :term:`bytes-like object` or a N-dimensional array.
469 possible to pass it to other APIs expecting a buffer-providing object,
474 :ref:`out-of-band serialization <pickle-oob>`.
481 The returned object is a one-dimensional, C-contiguous memoryview
483 the buffer is neither C- nor Fortran-contiguous.
487 Release the underlying buffer exposed by the PickleBuffer object.
490 .. _pickle-picklable:
493 ----------------------------------
499 * integers, floating-point numbers, complex numbers;
505 * functions (built-in and user-defined) accessible from the top level of a
511 is picklable (see section :ref:`pickle-inst` for details).
520 Note that functions (built-in and user-defined) are pickled by fully
525 must contain the named object, otherwise an exception will be raised. [#]_
544 plan to have long-lived objects that will see many versions of a class, it may
549 .. _pickle-inst:
552 ------------------------
577 .. method:: object.__getnewargs_ex__()
584 object. Those will be passed to the :meth:`__new__` method upon
588 class requires keyword-only arguments. Otherwise, it is recommended for
595 .. method:: object.__getnewargs__()
609 .. method:: object.__getstate__()
612 the method :meth:`__getstate__`. It is called and the returned object
616 * For a class that has no instance :attr:`~object.__dict__` and no
617 :attr:`~object.__slots__`, the default state is ``None``.
619 * For a class that has an instance :attr:`~object.__dict__` and no
620 :attr:`~object.__slots__`, the default state is ``self.__dict__``.
622 * For a class that has an instance :attr:`~object.__dict__` and
623 :attr:`~object.__slots__`, the default state is a tuple consisting of two
628 * For a class that has :attr:`~object.__slots__` and no instance
629 :attr:`~object.__dict__`, the default state is a tuple whose first item
635 :class:`object` class.
638 .. method:: object.__setstate__(state)
642 object to be a dictionary. Otherwise, the pickled state must be a dictionary
651 Refer to the section :ref:`pickle-state` for more information about how to use
663 .. index:: pair: copy; protocol
666 fact, these methods are part of the copy protocol which implements the
667 :meth:`__reduce__` special method. The copy protocol provides a unified
672 error prone. For this reason, class designers should use the high-level
678 .. method:: object.__reduce__()
682 returned object is often referred to as the "reduce value").
685 global variable. It should be the object's local name relative to its
687 object's module. This behaviour is typically useful for singletons.
693 .. XXX Mention __newobj__ special-case?
695 * A callable object that will be called to create the initial version of the
696 object.
698 * A tuple of arguments for the callable object. An empty tuple must be given
701 * Optionally, the object's state, which will be passed to the object's
702 :meth:`__setstate__` method as previously described. If the object has no
704 the object's :attr:`~object.__dict__` attribute.
707 These items will be appended to the object either using
715 * Optionally, an iterator (not a sequence) yielding successive key-value
716 pairs. These items will be stored to the object using ``obj[key] =
721 callable allows the user to programmatically control the state-updating
722 behavior of a specific object, instead of using ``obj``'s static
730 .. method:: object.__reduce_ex__(protocol)
737 backwards-compatible reduce values for older Python releases.
741 .. _pickle-persistent:
750 For the benefit of object persistence, the :mod:`pickle` module supports the
751 notion of a reference to an object outside the pickled data stream. Such
753 alphanumeric characters (for protocol 0) [#]_ or just an arbitrary object (for
757 module; it will delegate this resolution to the user-defined methods on the
762 custom :meth:`~Pickler.persistent_id` method that takes an object as an
763 argument and returns either ``None`` or the persistent ID for that object.
764 When ``None`` is returned, the pickler simply pickles the object as normal.
765 When a persistent ID string is returned, the pickler will pickle that object,
769 :meth:`~Unpickler.persistent_load` method that takes a persistent ID object and
770 returns the referenced object.
777 .. _pickle-dispatch:
788 choose to use a modified copy of :data:`copyreg.dispatch_table` as a
795 p.dispatch_table = copyreg.dispatch_table.copy()
803 dispatch_table = copyreg.dispatch_table.copy()
817 .. _pickle-state:
823 single: __getstate__() (copy protocol)
824 single: __setstate__() (copy protocol)
829 :class:`TextReader` instance is pickled, all attributes *except* the file object
848 line = line[:-1]
852 # Copy the object's state from self.__dict__ which contains
853 # all our instance attributes. Always use the dict.copy()
855 state = self.__dict__.copy()
886 --------------------------------------------------------
892 than the object's type, or we may want to customize the pickling of
927 # For any other object, fallback to usual reduction
943 .. _pickle-oob:
945 Out-of-band Buffers
946 -------------------
953 operation of the :mod:`pickle` module, as it transforms a graph-like structure
958 of the object types to be transferred) and the *consumer* (the implementation
959 of the communications system) support the out-of-band transfer facilities
967 :class:`PickleBuffer` instance (instead of e.g. a :class:`bytes` object)
970 A :class:`PickleBuffer` object *signals* that the underlying buffer is
971 eligible for out-of-band data transfer. Those objects remain compatible
973 opt-in to tell :mod:`pickle` that they will handle those buffers by
980 objects generated when serializing an object graph.
985 the object graph. Buffers accumulated by the *buffer_callback* will not
998 is free to implement its own transfer mechanism for out-of-band buffers.
999 Potential optimizations include the use of shared memory or datatype-dependent
1006 able to participate in out-of-band buffer pickling::
1020 # Get a handle over the original buffer object
1023 # Original buffer object is a ZeroCopyByteArray, return it
1024 # as-is.
1030 providing object if it has the right type. This is an easy way to simulate
1031 zero-copy behaviour on this toy example.
1034 when unserialized will give us a copy of the original object::
1040 print(b is new_b) # False: a copy was made
1043 buffers when unserializing, we are able to get back the original object::
1050 print(b is new_b) # True: no copy was made
1054 by another object's memory. However, third-party datatypes such as NumPy
1055 arrays do not have this limitation, and allow use of zero-copy pickling
1059 .. seealso:: :pep:`574` -- Pickle protocol 5 with out-of-band data
1062 .. _pickle-restrict:
1065 -------------------
1073 this hand-crafted pickle data stream does when loaded::
1141 third-party solutions.
1145 -----------
1148 efficient binary encodings for several common features and built-in types.
1152 .. _pickle-example:
1155 --------
1198 Module :mod:`copy`
1199 Shallow and deep object copying.
1202 High-performance serialization of built-in types.
1215 .. [#] The :mod:`copy` module uses this protocol for shallow and deep copying