• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1:mod:`pickle` --- Python object serialization
2=============================================
3
4.. index::
5   single: persistence
6   pair: persistent; objects
7   pair: serializing; objects
8   pair: marshalling; objects
9   pair: flattening; objects
10   pair: pickling; objects
11
12.. module:: pickle
13   :synopsis: Convert Python objects to streams of bytes and back.
14.. sectionauthor:: Jim Kerr <jbkerr@sr.hp.com>.
15.. sectionauthor:: Barry Warsaw <barry@zope.com>
16
17The :mod:`pickle` module implements a fundamental, but powerful algorithm for
18serializing and de-serializing a Python object structure.  "Pickling" is the
19process whereby a Python object hierarchy is converted into a byte stream, and
20"unpickling" is the inverse operation, whereby a byte stream is converted back
21into an object hierarchy.  Pickling (and unpickling) is alternatively known as
22"serialization", "marshalling," [#]_ or "flattening", however, to avoid
23confusion, the terms used here are "pickling" and "unpickling".
24
25This documentation describes both the :mod:`pickle` module and the
26:mod:`cPickle` module.
27
28.. warning::
29
30   The :mod:`pickle` module is not secure against erroneous or maliciously
31   constructed data.  Never unpickle data received from an untrusted or
32   unauthenticated source.
33
34
35Relationship to other Python modules
36------------------------------------
37
38The :mod:`pickle` module has an optimized cousin called the :mod:`cPickle`
39module.  As its name implies, :mod:`cPickle` is written in C, so it can be up to
401000 times faster than :mod:`pickle`.  However it does not support subclassing
41of the :func:`Pickler` and :func:`Unpickler` classes, because in :mod:`cPickle`
42these are functions, not classes.  Most applications have no need for this
43functionality, and can benefit from the improved performance of :mod:`cPickle`.
44Other than that, the interfaces of the two modules are nearly identical; the
45common interface is described in this manual and differences are pointed out
46where necessary.  In the following discussions, we use the term "pickle" to
47collectively describe the :mod:`pickle` and :mod:`cPickle` modules.
48
49The data streams the two modules produce are guaranteed to be interchangeable.
50
51Python has a more primitive serialization module called :mod:`marshal`, but in
52general :mod:`pickle` should always be the preferred way to serialize Python
53objects.  :mod:`marshal` exists primarily to support Python's :file:`.pyc`
54files.
55
56The :mod:`pickle` module differs from :mod:`marshal` in several significant ways:
57
58* The :mod:`pickle` module keeps track of the objects it has already serialized,
59  so that later references to the same object won't be serialized again.
60  :mod:`marshal` doesn't do this.
61
62  This has implications both for recursive objects and object sharing.  Recursive
63  objects are objects that contain references to themselves.  These are not
64  handled by marshal, and in fact, attempting to marshal recursive objects will
65  crash your Python interpreter.  Object sharing happens when there are multiple
66  references to the same object in different places in the object hierarchy being
67  serialized.  :mod:`pickle` stores such objects only once, and ensures that all
68  other references point to the master copy.  Shared objects remain shared, which
69  can be very important for mutable objects.
70
71* :mod:`marshal` cannot be used to serialize user-defined classes and their
72  instances.  :mod:`pickle` can save and restore class instances transparently,
73  however the class definition must be importable and live in the same module as
74  when the object was stored.
75
76* The :mod:`marshal` serialization format is not guaranteed to be portable
77  across Python versions.  Because its primary job in life is to support
78  :file:`.pyc` files, the Python implementers reserve the right to change the
79  serialization format in non-backwards compatible ways should the need arise.
80  The :mod:`pickle` serialization format is guaranteed to be backwards compatible
81  across Python releases.
82
83Note that serialization is a more primitive notion than persistence; although
84:mod:`pickle` reads and writes file objects, it does not handle the issue of
85naming persistent objects, nor the (even more complicated) issue of concurrent
86access to persistent objects.  The :mod:`pickle` module can transform a complex
87object into a byte stream and it can transform the byte stream into an object
88with the same internal structure.  Perhaps the most obvious thing to do with
89these byte streams is to write them onto a file, but it is also conceivable to
90send them across a network or store them in a database.  The module
91:mod:`shelve` provides a simple interface to pickle and unpickle objects on
92DBM-style database files.
93
94
95Data stream format
96------------------
97
98.. index::
99   single: XDR
100   single: External Data Representation
101
102The data format used by :mod:`pickle` is Python-specific.  This has the
103advantage that there are no restrictions imposed by external standards such as
104XDR (which can't represent pointer sharing); however it means that non-Python
105programs may not be able to reconstruct pickled Python objects.
106
107By default, the :mod:`pickle` data format uses a printable ASCII representation.
108This is slightly more voluminous than a binary representation.  The big
109advantage of using printable ASCII (and of some other characteristics of
110:mod:`pickle`'s representation) is that for debugging or recovery purposes it is
111possible for a human to read the pickled file with a standard text editor.
112
113There are currently 3 different protocols which can be used for pickling.
114
115* Protocol version 0 is the original ASCII protocol and is backwards compatible
116  with earlier versions of Python.
117
118* Protocol version 1 is the old binary format which is also compatible with
119  earlier versions of Python.
120
121* Protocol version 2 was introduced in Python 2.3.  It provides much more
122  efficient pickling of :term:`new-style class`\es.
123
124Refer to :pep:`307` for more information.
125
126If a *protocol* is not specified, protocol 0 is used. If *protocol* is specified
127as a negative value or :const:`HIGHEST_PROTOCOL`, the highest protocol version
128available will be used.
129
130.. versionchanged:: 2.3
131   Introduced the *protocol* parameter.
132
133A binary format, which is slightly more efficient, can be chosen by specifying a
134*protocol* version >= 1.
135
136
137Usage
138-----
139
140To serialize an object hierarchy, you first create a pickler, then you call the
141pickler's :meth:`dump` method.  To de-serialize a data stream, you first create
142an unpickler, then you call the unpickler's :meth:`load` method.  The
143:mod:`pickle` module provides the following constant:
144
145
146.. data:: HIGHEST_PROTOCOL
147
148   The highest protocol version available.  This value can be passed as a
149   *protocol* value.
150
151   .. versionadded:: 2.3
152
153.. note::
154
155   Be sure to always open pickle files created with protocols >= 1 in binary mode.
156   For the old ASCII-based pickle protocol 0 you can use either text mode or binary
157   mode as long as you stay consistent.
158
159   A pickle file written with protocol 0 in binary mode will contain lone linefeeds
160   as line terminators and therefore will look "funny" when viewed in Notepad or
161   other editors which do not support this format.
162
163The :mod:`pickle` module provides the following functions to make the pickling
164process more convenient:
165
166
167.. function:: dump(obj, file[, protocol])
168
169   Write a pickled representation of *obj* to the open file object *file*.  This is
170   equivalent to ``Pickler(file, protocol).dump(obj)``.
171
172   If the *protocol* parameter is omitted, protocol 0 is used. If *protocol* is
173   specified as a negative value or :const:`HIGHEST_PROTOCOL`, the highest protocol
174   version will be used.
175
176   .. versionchanged:: 2.3
177      Introduced the *protocol* parameter.
178
179   *file* must have a :meth:`write` method that accepts a single string argument.
180   It can thus be a file object opened for writing, a :mod:`StringIO` object, or
181   any other custom object that meets this interface.
182
183
184.. function:: load(file)
185
186   Read a string from the open file object *file* and interpret it as a pickle data
187   stream, reconstructing and returning the original object hierarchy.  This is
188   equivalent to ``Unpickler(file).load()``.
189
190   *file* must have two methods, a :meth:`read` method that takes an integer
191   argument, and a :meth:`readline` method that requires no arguments.  Both
192   methods should return a string.  Thus *file* can be a file object opened for
193   reading, a :mod:`StringIO` object, or any other custom object that meets this
194   interface.
195
196   This function automatically determines whether the data stream was written in
197   binary mode or not.
198
199
200.. function:: dumps(obj[, protocol])
201
202   Return the pickled representation of the object as a string, instead of writing
203   it to a file.
204
205   If the *protocol* parameter is omitted, protocol 0 is used. If *protocol* is
206   specified as a negative value or :const:`HIGHEST_PROTOCOL`, the highest protocol
207   version will be used.
208
209   .. versionchanged:: 2.3
210      The *protocol* parameter was added.
211
212
213.. function:: loads(string)
214
215   Read a pickled object hierarchy from a string.  Characters in the string past
216   the pickled object's representation are ignored.
217
218The :mod:`pickle` module also defines three exceptions:
219
220
221.. exception:: PickleError
222
223   A common base class for the other exceptions defined below.  This inherits from
224   :exc:`Exception`.
225
226
227.. exception:: PicklingError
228
229   This exception is raised when an unpicklable object is passed to the
230   :meth:`dump` method.
231
232
233.. exception:: UnpicklingError
234
235   This exception is raised when there is a problem unpickling an object. Note that
236   other exceptions may also be raised during unpickling, including (but not
237   necessarily limited to) :exc:`AttributeError`, :exc:`EOFError`,
238   :exc:`ImportError`, and :exc:`IndexError`.
239
240The :mod:`pickle` module also exports two callables [#]_, :class:`Pickler` and
241:class:`Unpickler`:
242
243
244.. class:: Pickler(file[, protocol])
245
246   This takes a file-like object to which it will write a pickle data stream.
247
248   If the *protocol* parameter is omitted, protocol 0 is used. If *protocol* is
249   specified as a negative value or :const:`HIGHEST_PROTOCOL`, the highest
250   protocol version will be used.
251
252   .. versionchanged:: 2.3
253      Introduced the *protocol* parameter.
254
255   *file* must have a :meth:`write` method that accepts a single string argument.
256   It can thus be an open file object, a :mod:`StringIO` object, or any other
257   custom object that meets this interface.
258
259   :class:`Pickler` objects define one (or two) public methods:
260
261
262   .. method:: dump(obj)
263
264      Write a pickled representation of *obj* to the open file object given in the
265      constructor.  Either the binary or ASCII format will be used, depending on the
266      value of the *protocol* argument passed to the constructor.
267
268
269   .. method:: clear_memo()
270
271      Clears the pickler's "memo".  The memo is the data structure that remembers
272      which objects the pickler has already seen, so that shared or recursive objects
273      pickled by reference and not by value.  This method is useful when re-using
274      picklers.
275
276      .. note::
277
278         Prior to Python 2.3, :meth:`clear_memo` was only available on the picklers
279         created by :mod:`cPickle`.  In the :mod:`pickle` module, picklers have an
280         instance variable called :attr:`memo` which is a Python dictionary.  So to clear
281         the memo for a :mod:`pickle` module pickler, you could do the following::
282
283            mypickler.memo.clear()
284
285         Code that does not need to support older versions of Python should simply use
286         :meth:`clear_memo`.
287
288It is possible to make multiple calls to the :meth:`dump` method of the same
289:class:`Pickler` instance.  These must then be matched to the same number of
290calls to the :meth:`load` method of the corresponding :class:`Unpickler`
291instance.  If the same object is pickled by multiple :meth:`dump` calls, the
292:meth:`load` will all yield references to the same object. [#]_
293
294:class:`Unpickler` objects are defined as:
295
296
297.. class:: Unpickler(file)
298
299   This takes a file-like object from which it will read a pickle data stream.
300   This class automatically determines whether the data stream was written in
301   binary mode or not, so it does not need a flag as in the :class:`Pickler`
302   factory.
303
304   *file* must have two methods, a :meth:`read` method that takes an integer
305   argument, and a :meth:`readline` method that requires no arguments.  Both
306   methods should return a string.  Thus *file* can be a file object opened for
307   reading, a :mod:`StringIO` object, or any other custom object that meets this
308   interface.
309
310   :class:`Unpickler` objects have one (or two) public methods:
311
312
313   .. method:: load()
314
315      Read a pickled object representation from the open file object given in
316      the constructor, and return the reconstituted object hierarchy specified
317      therein.
318
319      This method automatically determines whether the data stream was written
320      in binary mode or not.
321
322
323   .. method:: noload()
324
325      This is just like :meth:`load` except that it doesn't actually create any
326      objects.  This is useful primarily for finding what's called "persistent
327      ids" that may be referenced in a pickle data stream.  See section
328      :ref:`pickle-protocol` below for more details.
329
330      **Note:** the :meth:`noload` method is currently only available on
331      :class:`Unpickler` objects created with the :mod:`cPickle` module.
332      :mod:`pickle` module :class:`Unpickler`\ s do not have the :meth:`noload`
333      method.
334
335
336What can be pickled and unpickled?
337----------------------------------
338
339The following types can be pickled:
340
341* ``None``, ``True``, and ``False``
342
343* integers, long integers, floating point numbers, complex numbers
344
345* normal and Unicode strings
346
347* tuples, lists, sets, and dictionaries containing only picklable objects
348
349* functions defined at the top level of a module
350
351* built-in functions defined at the top level of a module
352
353* classes that are defined at the top level of a module
354
355* instances of such classes whose :attr:`~object.__dict__` or the result of
356  calling :meth:`__getstate__` is picklable  (see section :ref:`pickle-protocol`
357  for details).
358
359Attempts to pickle unpicklable objects will raise the :exc:`PicklingError`
360exception; when this happens, an unspecified number of bytes may have already
361been written to the underlying file. Trying to pickle a highly recursive data
362structure may exceed the maximum recursion depth, a :exc:`RuntimeError` will be
363raised in this case. You can carefully raise this limit with
364:func:`sys.setrecursionlimit`.
365
366Note that functions (built-in and user-defined) are pickled by "fully qualified"
367name reference, not by value.  This means that only the function name is
368pickled, along with the name of the module the function is defined in.  Neither
369the function's code, nor any of its function attributes are pickled.  Thus the
370defining module must be importable in the unpickling environment, and the module
371must contain the named object, otherwise an exception will be raised. [#]_
372
373Similarly, classes are pickled by named reference, so the same restrictions in
374the unpickling environment apply.  Note that none of the class's code or data is
375pickled, so in the following example the class attribute ``attr`` is not
376restored in the unpickling environment::
377
378   class Foo:
379       attr = 'a class attr'
380
381   picklestring = pickle.dumps(Foo)
382
383These restrictions are why picklable functions and classes must be defined in
384the top level of a module.
385
386Similarly, when class instances are pickled, their class's code and data are not
387pickled along with them.  Only the instance data are pickled.  This is done on
388purpose, so you can fix bugs in a class or add methods to the class and still
389load objects that were created with an earlier version of the class.  If you
390plan to have long-lived objects that will see many versions of a class, it may
391be worthwhile to put a version number in the objects so that suitable
392conversions can be made by the class's :meth:`__setstate__` method.
393
394
395.. _pickle-protocol:
396
397The pickle protocol
398-------------------
399
400.. currentmodule:: None
401
402This section describes the "pickling protocol" that defines the interface
403between the pickler/unpickler and the objects that are being serialized.  This
404protocol provides a standard way for you to define, customize, and control how
405your objects are serialized and de-serialized.  The description in this section
406doesn't cover specific customizations that you can employ to make the unpickling
407environment slightly safer from untrusted pickle data streams; see section
408:ref:`pickle-sub` for more details.
409
410
411.. _pickle-inst:
412
413Pickling and unpickling normal class instances
414^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
415
416.. method:: object.__getinitargs__()
417
418   When a pickled class instance is unpickled, its :meth:`__init__` method is
419   normally *not* invoked.  If it is desirable that the :meth:`__init__` method
420   be called on unpickling, an old-style class can define a method
421   :meth:`__getinitargs__`, which should return a *tuple* of positional
422   arguments to be passed to the class constructor (:meth:`__init__` for
423   example).  Keyword arguments are not supported.  The :meth:`__getinitargs__`
424   method is called at pickle time; the tuple it returns is incorporated in the
425   pickle for the instance.
426
427.. method:: object.__getnewargs__()
428
429   New-style types can provide a :meth:`__getnewargs__` method that is used for
430   protocol 2.  Implementing this method is needed if the type establishes some
431   internal invariants when the instance is created, or if the memory allocation
432   is affected by the values passed to the :meth:`__new__` method for the type
433   (as it is for tuples and strings).  Instances of a :term:`new-style class`
434   ``C`` are created using ::
435
436      obj = C.__new__(C, *args)
437
438   where *args* is the result of calling :meth:`__getnewargs__` on the original
439   object; if there is no :meth:`__getnewargs__`, an empty tuple is assumed.
440
441.. method:: object.__getstate__()
442
443   Classes can further influence how their instances are pickled; if the class
444   defines the method :meth:`__getstate__`, it is called and the return state is
445   pickled as the contents for the instance, instead of the contents of the
446   instance's dictionary.  If there is no :meth:`__getstate__` method, the
447   instance's :attr:`~object.__dict__` is pickled.
448
449.. method:: object.__setstate__(state)
450
451   Upon unpickling, if the class also defines the method :meth:`__setstate__`,
452   it is called with the unpickled state. [#]_ If there is no
453   :meth:`__setstate__` method, the pickled state must be a dictionary and its
454   items are assigned to the new instance's dictionary.  If a class defines both
455   :meth:`__getstate__` and :meth:`__setstate__`, the state object needn't be a
456   dictionary and these methods can do what they want. [#]_
457
458   .. note::
459
460      For :term:`new-style class`\es, if :meth:`__getstate__` returns a false
461      value, the :meth:`__setstate__` method will not be called.
462
463.. note::
464
465   At unpickling time, some methods like :meth:`__getattr__`,
466   :meth:`__getattribute__`, or :meth:`__setattr__` may be called upon the
467   instance.  In case those methods rely on some internal invariant being
468   true, the type should implement either :meth:`__getinitargs__` or
469   :meth:`__getnewargs__` to establish such an invariant; otherwise, neither
470   :meth:`__new__` nor :meth:`__init__` will be called.
471
472
473Pickling and unpickling extension types
474^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
475
476.. method:: object.__reduce__()
477
478   When the :class:`Pickler` encounters an object of a type it knows nothing
479   about --- such as an extension type --- it looks in two places for a hint of
480   how to pickle it.  One alternative is for the object to implement a
481   :meth:`__reduce__` method.  If provided, at pickling time :meth:`__reduce__`
482   will be called with no arguments, and it must return either a string or a
483   tuple.
484
485   If a string is returned, it names a global variable whose contents are
486   pickled as normal.  The string returned by :meth:`__reduce__` should be the
487   object's local name relative to its module; the pickle module searches the
488   module namespace to determine the object's module.
489
490   When a tuple is returned, it must be between two and five elements long.
491   Optional elements can either be omitted, or ``None`` can be provided as their
492   value.  The contents of this tuple are pickled as normal and used to
493   reconstruct the object at unpickling time.  The semantics of each element
494   are:
495
496   * A callable object that will be called to create the initial version of the
497     object.  The next element of the tuple will provide arguments for this
498     callable, and later elements provide additional state information that will
499     subsequently be used to fully reconstruct the pickled data.
500
501     In the unpickling environment this object must be either a class, a
502     callable registered as a "safe constructor" (see below), or it must have an
503     attribute :attr:`__safe_for_unpickling__` with a true value. Otherwise, an
504     :exc:`UnpicklingError` will be raised in the unpickling environment.  Note
505     that as usual, the callable itself is pickled by name.
506
507   * A tuple of arguments for the callable object.
508
509     .. versionchanged:: 2.5
510        Formerly, this argument could also be ``None``.
511
512   * Optionally, the object's state, which will be passed to the object's
513     :meth:`__setstate__` method as described in section :ref:`pickle-inst`.  If
514     the object has no :meth:`__setstate__` method, then, as above, the value
515     must be a dictionary and it will be added to the object's
516     :attr:`~object.__dict__`.
517
518   * Optionally, an iterator (and not a sequence) yielding successive list
519     items.  These list items will be pickled, and appended to the object using
520     either ``obj.append(item)`` or ``obj.extend(list_of_items)``.  This is
521     primarily used for list subclasses, but may be used by other classes as
522     long as they have :meth:`append` and :meth:`extend` methods with the
523     appropriate signature.  (Whether :meth:`append` or :meth:`extend` is used
524     depends on which pickle protocol version is used as well as the number of
525     items to append, so both must be supported.)
526
527   * Optionally, an iterator (not a sequence) yielding successive dictionary
528     items, which should be tuples of the form ``(key, value)``.  These items
529     will be pickled and stored to the object using ``obj[key] = value``. This
530     is primarily used for dictionary subclasses, but may be used by other
531     classes as long as they implement :meth:`__setitem__`.
532
533.. method:: object.__reduce_ex__(protocol)
534
535   It is sometimes useful to know the protocol version when implementing
536   :meth:`__reduce__`.  This can be done by implementing a method named
537   :meth:`__reduce_ex__` instead of :meth:`__reduce__`. :meth:`__reduce_ex__`,
538   when it exists, is called in preference over :meth:`__reduce__` (you may
539   still provide :meth:`__reduce__` for backwards compatibility).  The
540   :meth:`__reduce_ex__` method will be called with a single integer argument,
541   the protocol version.
542
543   The :class:`object` class implements both :meth:`__reduce__` and
544   :meth:`__reduce_ex__`; however, if a subclass overrides :meth:`__reduce__`
545   but not :meth:`__reduce_ex__`, the :meth:`__reduce_ex__` implementation
546   detects this and calls :meth:`__reduce__`.
547
548An alternative to implementing a :meth:`__reduce__` method on the object to be
549pickled, is to register the callable with the :mod:`copy_reg` module.  This
550module provides a way for programs to register "reduction functions" and
551constructors for user-defined types.   Reduction functions have the same
552semantics and interface as the :meth:`__reduce__` method described above, except
553that they are called with a single argument, the object to be pickled.
554
555The registered constructor is deemed a "safe constructor" for purposes of
556unpickling as described above.
557
558
559Pickling and unpickling external objects
560^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
561
562.. index::
563   single: persistent_id (pickle protocol)
564   single: persistent_load (pickle protocol)
565
566For the benefit of object persistence, the :mod:`pickle` module supports the
567notion of a reference to an object outside the pickled data stream.  Such
568objects are referenced by a "persistent id", which is just an arbitrary string
569of printable ASCII characters. The resolution of such names is not defined by
570the :mod:`pickle` module; it will delegate this resolution to user defined
571functions on the pickler and unpickler. [#]_
572
573To define external persistent id resolution, you need to set the
574:attr:`~Pickler.persistent_id` attribute of the pickler object and the
575:attr:`~Unpickler.persistent_load` attribute of the unpickler object.
576
577To pickle objects that have an external persistent id, the pickler must have a
578custom :func:`~Pickler.persistent_id` method that takes an object as an
579argument and returns either ``None`` or the persistent id for that object.
580When ``None`` is returned, the pickler simply pickles the object as normal.
581When a persistent id string is returned, the pickler will pickle that string,
582along with a marker so that the unpickler will recognize the string as a
583persistent id.
584
585To unpickle external objects, the unpickler must have a custom
586:func:`~Unpickler.persistent_load` function that takes a persistent id string
587and returns the referenced object.
588
589Here's a silly example that *might* shed more light::
590
591   import pickle
592   from cStringIO import StringIO
593
594   src = StringIO()
595   p = pickle.Pickler(src)
596
597   def persistent_id(obj):
598       if hasattr(obj, 'x'):
599           return 'the value %d' % obj.x
600       else:
601           return None
602
603   p.persistent_id = persistent_id
604
605   class Integer:
606       def __init__(self, x):
607           self.x = x
608       def __str__(self):
609           return 'My name is integer %d' % self.x
610
611   i = Integer(7)
612   print i
613   p.dump(i)
614
615   datastream = src.getvalue()
616   print repr(datastream)
617   dst = StringIO(datastream)
618
619   up = pickle.Unpickler(dst)
620
621   class FancyInteger(Integer):
622       def __str__(self):
623           return 'I am the integer %d' % self.x
624
625   def persistent_load(persid):
626       if persid.startswith('the value '):
627           value = int(persid.split()[2])
628           return FancyInteger(value)
629       else:
630           raise pickle.UnpicklingError, 'Invalid persistent id'
631
632   up.persistent_load = persistent_load
633
634   j = up.load()
635   print j
636
637In the :mod:`cPickle` module, the unpickler's :attr:`~Unpickler.persistent_load`
638attribute can also be set to a Python list, in which case, when the unpickler
639reaches a persistent id, the persistent id string will simply be appended to
640this list.  This functionality exists so that a pickle data stream can be
641"sniffed" for object references without actually instantiating all the objects
642in a pickle.
643[#]_  Setting :attr:`~Unpickler.persistent_load` to a list is usually used in
644conjunction with the :meth:`~Unpickler.noload` method on the Unpickler.
645
646.. BAW: Both pickle and cPickle support something called inst_persistent_id()
647   which appears to give unknown types a second shot at producing a persistent
648   id.  Since Jim Fulton can't remember why it was added or what it's for, I'm
649   leaving it undocumented.
650
651
652.. _pickle-sub:
653
654Subclassing Unpicklers
655----------------------
656
657.. index::
658   single: load_global() (pickle protocol)
659   single: find_global() (pickle protocol)
660
661By default, unpickling will import any class that it finds in the pickle data.
662You can control exactly what gets unpickled and what gets called by customizing
663your unpickler.  Unfortunately, exactly how you do this is different depending
664on whether you're using :mod:`pickle` or :mod:`cPickle`. [#]_
665
666In the :mod:`pickle` module, you need to derive a subclass from
667:class:`Unpickler`, overriding the :meth:`load_global` method.
668:meth:`load_global` should read two lines from the pickle data stream where the
669first line will the name of the module containing the class and the second line
670will be the name of the instance's class.  It then looks up the class, possibly
671importing the module and digging out the attribute, then it appends what it
672finds to the unpickler's stack.  Later on, this class will be assigned to the
673:attr:`__class__` attribute of an empty class, as a way of magically creating an
674instance without calling its class's :meth:`__init__`. Your job (should you
675choose to accept it), would be to have :meth:`load_global` push onto the
676unpickler's stack, a known safe version of any class you deem safe to unpickle.
677It is up to you to produce such a class.  Or you could raise an error if you
678want to disallow all unpickling of instances.  If this sounds like a hack,
679you're right.  Refer to the source code to make this work.
680
681Things are a little cleaner with :mod:`cPickle`, but not by much. To control
682what gets unpickled, you can set the unpickler's :attr:`~Unpickler.find_global`
683attribute to a function or ``None``.  If it is ``None`` then any attempts to
684unpickle instances will raise an :exc:`UnpicklingError`.  If it is a function,
685then it should accept a module name and a class name, and return the
686corresponding class object.  It is responsible for looking up the class and
687performing any necessary imports, and it may raise an error to prevent
688instances of the class from being unpickled.
689
690The moral of the story is that you should be really careful about the source of
691the strings your application unpickles.
692
693
694.. _pickle-example:
695
696Example
697-------
698
699For the simplest code, use the :func:`dump` and :func:`load` functions.  Note
700that a self-referencing list is pickled and restored correctly. ::
701
702   import pickle
703
704   data1 = {'a': [1, 2.0, 3, 4+6j],
705            'b': ('string', u'Unicode string'),
706            'c': None}
707
708   selfref_list = [1, 2, 3]
709   selfref_list.append(selfref_list)
710
711   output = open('data.pkl', 'wb')
712
713   # Pickle dictionary using protocol 0.
714   pickle.dump(data1, output)
715
716   # Pickle the list using the highest protocol available.
717   pickle.dump(selfref_list, output, -1)
718
719   output.close()
720
721The following example reads the resulting pickled data.  When reading a
722pickle-containing file, you should open the file in binary mode because you
723can't be sure if the ASCII or binary format was used. ::
724
725   import pprint, pickle
726
727   pkl_file = open('data.pkl', 'rb')
728
729   data1 = pickle.load(pkl_file)
730   pprint.pprint(data1)
731
732   data2 = pickle.load(pkl_file)
733   pprint.pprint(data2)
734
735   pkl_file.close()
736
737Here's a larger example that shows how to modify pickling behavior for a class.
738The :class:`TextReader` class opens a text file, and returns the line number and
739line contents each time its :meth:`!readline` method is called. If a
740:class:`TextReader` instance is pickled, all attributes *except* the file object
741member are saved. When the instance is unpickled, the file is reopened, and
742reading resumes from the last location. The :meth:`__setstate__` and
743:meth:`__getstate__` methods are used to implement this behavior. ::
744
745   #!/usr/local/bin/python
746
747   class TextReader:
748       """Print and number lines in a text file."""
749       def __init__(self, file):
750           self.file = file
751           self.fh = open(file)
752           self.lineno = 0
753
754       def readline(self):
755           self.lineno = self.lineno + 1
756           line = self.fh.readline()
757           if not line:
758               return None
759           if line.endswith("\n"):
760               line = line[:-1]
761           return "%d: %s" % (self.lineno, line)
762
763       def __getstate__(self):
764           odict = self.__dict__.copy() # copy the dict since we change it
765           del odict['fh']              # remove filehandle entry
766           return odict
767
768       def __setstate__(self, dict):
769           fh = open(dict['file'])      # reopen file
770           count = dict['lineno']       # read from file...
771           while count:                 # until line count is restored
772               fh.readline()
773               count = count - 1
774           self.__dict__.update(dict)   # update attributes
775           self.fh = fh                 # save the file object
776
777A sample usage might be something like this::
778
779   >>> import TextReader
780   >>> obj = TextReader.TextReader("TextReader.py")
781   >>> obj.readline()
782   '1: #!/usr/local/bin/python'
783   >>> obj.readline()
784   '2: '
785   >>> obj.readline()
786   '3: class TextReader:'
787   >>> import pickle
788   >>> pickle.dump(obj, open('save.p', 'wb'))
789
790If you want to see that :mod:`pickle` works across Python processes, start
791another Python session, before continuing.  What follows can happen from either
792the same process or a new process. ::
793
794   >>> import pickle
795   >>> reader = pickle.load(open('save.p', 'rb'))
796   >>> reader.readline()
797   '4:     """Print and number lines in a text file."""'
798
799
800.. seealso::
801
802   Module :mod:`copy_reg`
803      Pickle interface constructor registration for extension types.
804
805   Module :mod:`shelve`
806      Indexed databases of objects; uses :mod:`pickle`.
807
808   Module :mod:`copy`
809      Shallow and deep object copying.
810
811   Module :mod:`marshal`
812      High-performance serialization of built-in types.
813
814
815:mod:`cPickle` --- A faster :mod:`pickle`
816=========================================
817
818.. module:: cPickle
819   :synopsis: Faster version of pickle, but not subclassable.
820.. moduleauthor:: Jim Fulton <jim@zope.com>
821.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org>
822
823
824.. index:: module: pickle
825
826The :mod:`cPickle` module supports serialization and de-serialization of Python
827objects, providing an interface and functionality nearly identical to the
828:mod:`pickle` module.  There are several differences, the most important being
829performance and subclassability.
830
831First, :mod:`cPickle` can be up to 1000 times faster than :mod:`pickle` because
832the former is implemented in C.  Second, in the :mod:`cPickle` module the
833callables :func:`Pickler` and :func:`Unpickler` are functions, not classes.
834This means that you cannot use them to derive custom pickling and unpickling
835subclasses.  Most applications have no need for this functionality and should
836benefit from the greatly improved performance of the :mod:`cPickle` module.
837
838The pickle data stream produced by :mod:`pickle` and :mod:`cPickle` are
839identical, so it is possible to use :mod:`pickle` and :mod:`cPickle`
840interchangeably with existing pickles. [#]_
841
842There are additional minor differences in API between :mod:`cPickle` and
843:mod:`pickle`, however for most applications, they are interchangeable.  More
844documentation is provided in the :mod:`pickle` module documentation, which
845includes a list of the documented differences.
846
847.. rubric:: Footnotes
848
849.. [#] Don't confuse this with the :mod:`marshal` module
850
851.. [#] In the :mod:`pickle` module these callables are classes, which you could
852   subclass to customize the behavior.  However, in the :mod:`cPickle` module these
853   callables are factory functions and so cannot be subclassed.  One common reason
854   to subclass is to control what objects can actually be unpickled.  See section
855   :ref:`pickle-sub` for more details.
856
857.. [#] *Warning*: this is intended for pickling multiple objects without intervening
858   modifications to the objects or their parts.  If you modify an object and then
859   pickle it again using the same :class:`Pickler` instance, the object is not
860   pickled again --- a reference to it is pickled and the :class:`Unpickler` will
861   return the old value, not the modified one. There are two problems here: (1)
862   detecting changes, and (2) marshalling a minimal set of changes.  Garbage
863   Collection may also become a problem here.
864
865.. [#] The exception raised will likely be an :exc:`ImportError` or an
866   :exc:`AttributeError` but it could be something else.
867
868.. [#] These methods can also be used to implement copying class instances.
869
870.. [#] This protocol is also used by the shallow and deep copying operations defined in
871   the :mod:`copy` module.
872
873.. [#] The actual mechanism for associating these user defined functions is slightly
874   different for :mod:`pickle` and :mod:`cPickle`.  The description given here
875   works the same for both implementations.  Users of the :mod:`pickle` module
876   could also use subclassing to effect the same results, overriding the
877   :meth:`persistent_id` and :meth:`persistent_load` methods in the derived
878   classes.
879
880.. [#] We'll leave you with the image of Guido and Jim sitting around sniffing pickles
881   in their living rooms.
882
883.. [#] A word of caution: the mechanisms described here use internal attributes and
884   methods, which are subject to change in future versions of Python.  We intend to
885   someday provide a common interface for controlling this behavior, which will
886   work in either :mod:`pickle` or :mod:`cPickle`.
887
888.. [#] Since the pickle data format is actually a tiny stack-oriented programming
889   language, and some freedom is taken in the encodings of certain objects, it is
890   possible that the two modules produce different data streams for the same input
891   objects.  However it is guaranteed that they will always be able to read each
892   other's data streams.
893
894