1:mod:`weakref` --- Weak references 2================================== 3 4.. module:: weakref 5 :synopsis: Support for weak references and weak dictionaries. 6 7.. moduleauthor:: Fred L. Drake, Jr. <fdrake@acm.org> 8.. moduleauthor:: Neil Schemenauer <nas@arctrix.com> 9.. moduleauthor:: Martin von Löwis <martin@loewis.home.cs.tu-berlin.de> 10.. sectionauthor:: Fred L. Drake, Jr. <fdrake@acm.org> 11 12**Source code:** :source:`Lib/weakref.py` 13 14-------------- 15 16The :mod:`weakref` module allows the Python programmer to create :dfn:`weak 17references` to objects. 18 19.. When making changes to the examples in this file, be sure to update 20 Lib/test/test_weakref.py::libreftest too! 21 22In the following, the term :dfn:`referent` means the object which is referred to 23by a weak reference. 24 25A weak reference to an object is not enough to keep the object alive: when the 26only remaining references to a referent are weak references, 27:term:`garbage collection` is free to destroy the referent and reuse its memory 28for something else. However, until the object is actually destroyed the weak 29reference may return the object even if there are no strong references to it. 30 31A primary use for weak references is to implement caches or 32mappings holding large objects, where it's desired that a large object not be 33kept alive solely because it appears in a cache or mapping. 34 35For example, if you have a number of large binary image objects, you may wish to 36associate a name with each. If you used a Python dictionary to map names to 37images, or images to names, the image objects would remain alive just because 38they appeared as values or keys in the dictionaries. The 39:class:`WeakKeyDictionary` and :class:`WeakValueDictionary` classes supplied by 40the :mod:`weakref` module are an alternative, using weak references to construct 41mappings that don't keep objects alive solely because they appear in the mapping 42objects. If, for example, an image object is a value in a 43:class:`WeakValueDictionary`, then when the last remaining references to that 44image object are the weak references held by weak mappings, garbage collection 45can reclaim the object, and its corresponding entries in weak mappings are 46simply deleted. 47 48:class:`WeakKeyDictionary` and :class:`WeakValueDictionary` use weak references 49in their implementation, setting up callback functions on the weak references 50that notify the weak dictionaries when a key or value has been reclaimed by 51garbage collection. :class:`WeakSet` implements the :class:`set` interface, 52but keeps weak references to its elements, just like a 53:class:`WeakKeyDictionary` does. 54 55:class:`finalize` provides a straight forward way to register a 56cleanup function to be called when an object is garbage collected. 57This is simpler to use than setting up a callback function on a raw 58weak reference, since the module automatically ensures that the finalizer 59remains alive until the object is collected. 60 61Most programs should find that using one of these weak container types 62or :class:`finalize` is all they need -- it's not usually necessary to 63create your own weak references directly. The low-level machinery is 64exposed by the :mod:`weakref` module for the benefit of advanced uses. 65 66Not all objects can be weakly referenced; those objects which can include class 67instances, functions written in Python (but not in C), instance methods, sets, 68frozensets, some :term:`file objects <file object>`, :term:`generator`\s, type 69objects, sockets, arrays, deques, regular expression pattern objects, and code 70objects. 71 72.. versionchanged:: 3.2 73 Added support for thread.lock, threading.Lock, and code objects. 74 75Several built-in types such as :class:`list` and :class:`dict` do not directly 76support weak references but can add support through subclassing:: 77 78 class Dict(dict): 79 pass 80 81 obj = Dict(red=1, green=2, blue=3) # this object is weak referenceable 82 83Other built-in types such as :class:`tuple` and :class:`int` do not support weak 84references even when subclassed (This is an implementation detail and may be 85different across various Python implementations.). 86 87Extension types can easily be made to support weak references; see 88:ref:`weakref-support`. 89 90 91.. class:: ref(object[, callback]) 92 93 Return a weak reference to *object*. The original object can be retrieved by 94 calling the reference object if the referent is still alive; if the referent is 95 no longer alive, calling the reference object will cause :const:`None` to be 96 returned. If *callback* is provided and not :const:`None`, and the returned 97 weakref object is still alive, the callback will be called when the object is 98 about to be finalized; the weak reference object will be passed as the only 99 parameter to the callback; the referent will no longer be available. 100 101 It is allowable for many weak references to be constructed for the same object. 102 Callbacks registered for each weak reference will be called from the most 103 recently registered callback to the oldest registered callback. 104 105 Exceptions raised by the callback will be noted on the standard error output, 106 but cannot be propagated; they are handled in exactly the same way as exceptions 107 raised from an object's :meth:`__del__` method. 108 109 Weak references are :term:`hashable` if the *object* is hashable. They will 110 maintain their hash value even after the *object* was deleted. If 111 :func:`hash` is called the first time only after the *object* was deleted, 112 the call will raise :exc:`TypeError`. 113 114 Weak references support tests for equality, but not ordering. If the referents 115 are still alive, two references have the same equality relationship as their 116 referents (regardless of the *callback*). If either referent has been deleted, 117 the references are equal only if the reference objects are the same object. 118 119 This is a subclassable type rather than a factory function. 120 121 .. attribute:: __callback__ 122 123 This read-only attribute returns the callback currently associated to the 124 weakref. If there is no callback or if the referent of the weakref is 125 no longer alive then this attribute will have value ``None``. 126 127 .. versionchanged:: 3.4 128 Added the :attr:`__callback__` attribute. 129 130 131.. function:: proxy(object[, callback]) 132 133 Return a proxy to *object* which uses a weak reference. This supports use of 134 the proxy in most contexts instead of requiring the explicit dereferencing used 135 with weak reference objects. The returned object will have a type of either 136 ``ProxyType`` or ``CallableProxyType``, depending on whether *object* is 137 callable. Proxy objects are not :term:`hashable` regardless of the referent; this 138 avoids a number of problems related to their fundamentally mutable nature, and 139 prevent their use as dictionary keys. *callback* is the same as the parameter 140 of the same name to the :func:`ref` function. 141 142 143.. function:: getweakrefcount(object) 144 145 Return the number of weak references and proxies which refer to *object*. 146 147 148.. function:: getweakrefs(object) 149 150 Return a list of all weak reference and proxy objects which refer to *object*. 151 152 153.. class:: WeakKeyDictionary([dict]) 154 155 Mapping class that references keys weakly. Entries in the dictionary will be 156 discarded when there is no longer a strong reference to the key. This can be 157 used to associate additional data with an object owned by other parts of an 158 application without adding attributes to those objects. This can be especially 159 useful with objects that override attribute accesses. 160 161 .. note:: 162 163 Caution: Because a :class:`WeakKeyDictionary` is built on top of a Python 164 dictionary, it must not change size when iterating over it. This can be 165 difficult to ensure for a :class:`WeakKeyDictionary` because actions 166 performed by the program during iteration may cause items in the 167 dictionary to vanish "by magic" (as a side effect of garbage collection). 168 169:class:`WeakKeyDictionary` objects have an additional method that 170exposes the internal references directly. The references are not guaranteed to 171be "live" at the time they are used, so the result of calling the references 172needs to be checked before being used. This can be used to avoid creating 173references that will cause the garbage collector to keep the keys around longer 174than needed. 175 176 177.. method:: WeakKeyDictionary.keyrefs() 178 179 Return an iterable of the weak references to the keys. 180 181 182.. class:: WeakValueDictionary([dict]) 183 184 Mapping class that references values weakly. Entries in the dictionary will be 185 discarded when no strong reference to the value exists any more. 186 187 .. note:: 188 189 Caution: Because a :class:`WeakValueDictionary` is built on top of a Python 190 dictionary, it must not change size when iterating over it. This can be 191 difficult to ensure for a :class:`WeakValueDictionary` because actions performed 192 by the program during iteration may cause items in the dictionary to vanish "by 193 magic" (as a side effect of garbage collection). 194 195:class:`WeakValueDictionary` objects have an additional method that has the 196same issues as the :meth:`keyrefs` method of :class:`WeakKeyDictionary` 197objects. 198 199 200.. method:: WeakValueDictionary.valuerefs() 201 202 Return an iterable of the weak references to the values. 203 204 205.. class:: WeakSet([elements]) 206 207 Set class that keeps weak references to its elements. An element will be 208 discarded when no strong reference to it exists any more. 209 210 211.. class:: WeakMethod(method) 212 213 A custom :class:`ref` subclass which simulates a weak reference to a bound 214 method (i.e., a method defined on a class and looked up on an instance). 215 Since a bound method is ephemeral, a standard weak reference cannot keep 216 hold of it. :class:`WeakMethod` has special code to recreate the bound 217 method until either the object or the original function dies:: 218 219 >>> class C: 220 ... def method(self): 221 ... print("method called!") 222 ... 223 >>> c = C() 224 >>> r = weakref.ref(c.method) 225 >>> r() 226 >>> r = weakref.WeakMethod(c.method) 227 >>> r() 228 <bound method C.method of <__main__.C object at 0x7fc859830220>> 229 >>> r()() 230 method called! 231 >>> del c 232 >>> gc.collect() 233 0 234 >>> r() 235 >>> 236 237 .. versionadded:: 3.4 238 239.. class:: finalize(obj, func, *args, **kwargs) 240 241 Return a callable finalizer object which will be called when *obj* 242 is garbage collected. Unlike an ordinary weak reference, a finalizer 243 will always survive until the reference object is collected, greatly 244 simplifying lifecycle management. 245 246 A finalizer is considered *alive* until it is called (either explicitly 247 or at garbage collection), and after that it is *dead*. Calling a live 248 finalizer returns the result of evaluating ``func(*arg, **kwargs)``, 249 whereas calling a dead finalizer returns :const:`None`. 250 251 Exceptions raised by finalizer callbacks during garbage collection 252 will be shown on the standard error output, but cannot be 253 propagated. They are handled in the same way as exceptions raised 254 from an object's :meth:`__del__` method or a weak reference's 255 callback. 256 257 When the program exits, each remaining live finalizer is called 258 unless its :attr:`atexit` attribute has been set to false. They 259 are called in reverse order of creation. 260 261 A finalizer will never invoke its callback during the later part of 262 the :term:`interpreter shutdown` when module globals are liable to have 263 been replaced by :const:`None`. 264 265 .. method:: __call__() 266 267 If *self* is alive then mark it as dead and return the result of 268 calling ``func(*args, **kwargs)``. If *self* is dead then return 269 :const:`None`. 270 271 .. method:: detach() 272 273 If *self* is alive then mark it as dead and return the tuple 274 ``(obj, func, args, kwargs)``. If *self* is dead then return 275 :const:`None`. 276 277 .. method:: peek() 278 279 If *self* is alive then return the tuple ``(obj, func, args, 280 kwargs)``. If *self* is dead then return :const:`None`. 281 282 .. attribute:: alive 283 284 Property which is true if the finalizer is alive, false otherwise. 285 286 .. attribute:: atexit 287 288 A writable boolean property which by default is true. When the 289 program exits, it calls all remaining live finalizers for which 290 :attr:`.atexit` is true. They are called in reverse order of 291 creation. 292 293 .. note:: 294 295 It is important to ensure that *func*, *args* and *kwargs* do 296 not own any references to *obj*, either directly or indirectly, 297 since otherwise *obj* will never be garbage collected. In 298 particular, *func* should not be a bound method of *obj*. 299 300 .. versionadded:: 3.4 301 302 303.. data:: ReferenceType 304 305 The type object for weak references objects. 306 307 308.. data:: ProxyType 309 310 The type object for proxies of objects which are not callable. 311 312 313.. data:: CallableProxyType 314 315 The type object for proxies of callable objects. 316 317 318.. data:: ProxyTypes 319 320 Sequence containing all the type objects for proxies. This can make it simpler 321 to test if an object is a proxy without being dependent on naming both proxy 322 types. 323 324 325.. exception:: ReferenceError 326 327 Exception raised when a proxy object is used but the underlying object has been 328 collected. This is the same as the standard :exc:`ReferenceError` exception. 329 330 331.. seealso:: 332 333 :pep:`205` - Weak References 334 The proposal and rationale for this feature, including links to earlier 335 implementations and information about similar features in other languages. 336 337 338.. _weakref-objects: 339 340Weak Reference Objects 341---------------------- 342 343Weak reference objects have no methods and no attributes besides 344:attr:`ref.__callback__`. A weak reference object allows the referent to be 345obtained, if it still exists, by calling it: 346 347 >>> import weakref 348 >>> class Object: 349 ... pass 350 ... 351 >>> o = Object() 352 >>> r = weakref.ref(o) 353 >>> o2 = r() 354 >>> o is o2 355 True 356 357If the referent no longer exists, calling the reference object returns 358:const:`None`: 359 360 >>> del o, o2 361 >>> print(r()) 362 None 363 364Testing that a weak reference object is still live should be done using the 365expression ``ref() is not None``. Normally, application code that needs to use 366a reference object should follow this pattern:: 367 368 # r is a weak reference object 369 o = r() 370 if o is None: 371 # referent has been garbage collected 372 print("Object has been deallocated; can't frobnicate.") 373 else: 374 print("Object is still live!") 375 o.do_something_useful() 376 377Using a separate test for "liveness" creates race conditions in threaded 378applications; another thread can cause a weak reference to become invalidated 379before the weak reference is called; the idiom shown above is safe in threaded 380applications as well as single-threaded applications. 381 382Specialized versions of :class:`ref` objects can be created through subclassing. 383This is used in the implementation of the :class:`WeakValueDictionary` to reduce 384the memory overhead for each entry in the mapping. This may be most useful to 385associate additional information with a reference, but could also be used to 386insert additional processing on calls to retrieve the referent. 387 388This example shows how a subclass of :class:`ref` can be used to store 389additional information about an object and affect the value that's returned when 390the referent is accessed:: 391 392 import weakref 393 394 class ExtendedRef(weakref.ref): 395 def __init__(self, ob, callback=None, **annotations): 396 super(ExtendedRef, self).__init__(ob, callback) 397 self.__counter = 0 398 for k, v in annotations.items(): 399 setattr(self, k, v) 400 401 def __call__(self): 402 """Return a pair containing the referent and the number of 403 times the reference has been called. 404 """ 405 ob = super(ExtendedRef, self).__call__() 406 if ob is not None: 407 self.__counter += 1 408 ob = (ob, self.__counter) 409 return ob 410 411 412.. _weakref-example: 413 414Example 415------- 416 417This simple example shows how an application can use object IDs to retrieve 418objects that it has seen before. The IDs of the objects can then be used in 419other data structures without forcing the objects to remain alive, but the 420objects can still be retrieved by ID if they do. 421 422.. Example contributed by Tim Peters. 423 424:: 425 426 import weakref 427 428 _id2obj_dict = weakref.WeakValueDictionary() 429 430 def remember(obj): 431 oid = id(obj) 432 _id2obj_dict[oid] = obj 433 return oid 434 435 def id2obj(oid): 436 return _id2obj_dict[oid] 437 438 439.. _finalize-examples: 440 441Finalizer Objects 442----------------- 443 444The main benefit of using :class:`finalize` is that it makes it simple 445to register a callback without needing to preserve the returned finalizer 446object. For instance 447 448 >>> import weakref 449 >>> class Object: 450 ... pass 451 ... 452 >>> kenny = Object() 453 >>> weakref.finalize(kenny, print, "You killed Kenny!") #doctest:+ELLIPSIS 454 <finalize object at ...; for 'Object' at ...> 455 >>> del kenny 456 You killed Kenny! 457 458The finalizer can be called directly as well. However the finalizer 459will invoke the callback at most once. 460 461 >>> def callback(x, y, z): 462 ... print("CALLBACK") 463 ... return x + y + z 464 ... 465 >>> obj = Object() 466 >>> f = weakref.finalize(obj, callback, 1, 2, z=3) 467 >>> assert f.alive 468 >>> assert f() == 6 469 CALLBACK 470 >>> assert not f.alive 471 >>> f() # callback not called because finalizer dead 472 >>> del obj # callback not called because finalizer dead 473 474You can unregister a finalizer using its :meth:`~finalize.detach` 475method. This kills the finalizer and returns the arguments passed to 476the constructor when it was created. 477 478 >>> obj = Object() 479 >>> f = weakref.finalize(obj, callback, 1, 2, z=3) 480 >>> f.detach() #doctest:+ELLIPSIS 481 (<...Object object ...>, <function callback ...>, (1, 2), {'z': 3}) 482 >>> newobj, func, args, kwargs = _ 483 >>> assert not f.alive 484 >>> assert newobj is obj 485 >>> assert func(*args, **kwargs) == 6 486 CALLBACK 487 488Unless you set the :attr:`~finalize.atexit` attribute to 489:const:`False`, a finalizer will be called when the program exits if it 490is still alive. For instance 491 492 >>> obj = Object() 493 >>> weakref.finalize(obj, print, "obj dead or exiting") #doctest:+ELLIPSIS 494 <finalize object at ...; for 'Object' at ...> 495 >>> exit() #doctest:+SKIP 496 obj dead or exiting 497 498 499Comparing finalizers with :meth:`__del__` methods 500------------------------------------------------- 501 502Suppose we want to create a class whose instances represent temporary 503directories. The directories should be deleted with their contents 504when the first of the following events occurs: 505 506* the object is garbage collected, 507* the object's :meth:`remove` method is called, or 508* the program exits. 509 510We might try to implement the class using a :meth:`__del__` method as 511follows:: 512 513 class TempDir: 514 def __init__(self): 515 self.name = tempfile.mkdtemp() 516 517 def remove(self): 518 if self.name is not None: 519 shutil.rmtree(self.name) 520 self.name = None 521 522 @property 523 def removed(self): 524 return self.name is None 525 526 def __del__(self): 527 self.remove() 528 529Starting with Python 3.4, :meth:`__del__` methods no longer prevent 530reference cycles from being garbage collected, and module globals are 531no longer forced to :const:`None` during :term:`interpreter shutdown`. 532So this code should work without any issues on CPython. 533 534However, handling of :meth:`__del__` methods is notoriously implementation 535specific, since it depends on internal details of the interpreter's garbage 536collector implementation. 537 538A more robust alternative can be to define a finalizer which only references 539the specific functions and objects that it needs, rather than having access 540to the full state of the object:: 541 542 class TempDir: 543 def __init__(self): 544 self.name = tempfile.mkdtemp() 545 self._finalizer = weakref.finalize(self, shutil.rmtree, self.name) 546 547 def remove(self): 548 self._finalizer() 549 550 @property 551 def removed(self): 552 return not self._finalizer.alive 553 554Defined like this, our finalizer only receives a reference to the details 555it needs to clean up the directory appropriately. If the object never gets 556garbage collected the finalizer will still be called at exit. 557 558The other advantage of weakref based finalizers is that they can be used to 559register finalizers for classes where the definition is controlled by a 560third party, such as running code when a module is unloaded:: 561 562 import weakref, sys 563 def unloading_module(): 564 # implicit reference to the module globals from the function body 565 weakref.finalize(sys.modules[__name__], unloading_module) 566 567 568.. note:: 569 570 If you create a finalizer object in a daemonic thread just as the program 571 exits then there is the possibility that the finalizer 572 does not get called at exit. However, in a daemonic thread 573 :func:`atexit.register`, ``try: ... finally: ...`` and ``with: ...`` 574 do not guarantee that cleanup occurs either. 575