Lines Matching +full:is +full:- +full:module
10 ``static`` variables, which have process-wide scope. This document
11 describes problems of such per-process state and shows a safer way:
12 per-module state.
14 The document also describes how to switch to per-module state where
17 importantly—accessing per-module state from code.
23 This guide is written for maintainers of :ref:`C-API <c-api-index>` extensions
25 Python itself is used as a library.
31 An *interpreter* is the context in which Python code runs. It contains
38 - in sequence, with several :c:func:`Py_InitializeEx`/:c:func:`Py_FinalizeEx`
40 - in parallel, managing "sub-interpreters" using
46 assuming a process-wide "main Python interpreter".
49 Many extension modules (and even some stdlib modules) use *per-process*
52 between interpreters. Unless the extension developer is careful, it is very
53 easy to introduce edge cases that lead to crashes when a module is loaded in
56 Unfortunately, *per-interpreter* state is not easy to achieve. Extension
58 and it is currently cumbersome to test the behavior.
60 Enter Per-Module State
61 ----------------------
63 Instead of focusing on per-interpreter state, Python's C API is evolving
64 to better support the more granular *per-module* state.
65 This means that C-level data is be attached to a *module object*.
66 Each interpreter creates its own module object, keeping the data separate.
67 For testing the isolation, multiple module objects corresponding to a single
70 Per-module state provides an easy way to think about lifetime and
71 resource ownership: the extension module will initialize when a
72 module object is created, and clean up when it's freed. In this regard,
73 a module is just like any other :c:expr:`PyObject *`; there are no "on
77 per-process, per-interpreter, per-thread or per-task state.
78 With per-module state as the default, these are still possible,
84 Isolated Module Objects
85 -----------------------
87 The key point to keep in mind when developing an extension module is
88 that several module objects can be created from a single shared library.
91 .. code-block:: pycon
97 >>> import binascii # create a new module object
102 All objects and state specific to the module should be encapsulated
103 within the module object, not shared with other module objects, and
104 cleaned up when the module object is deallocated.
105 Since this just is a rule of thumb, exceptions are possible
115 ---------------------
118 notably, each module object will typically not share its classes and
120 `example above <Isolated Module Objects_>`__,
122 separate objects. In the following code, the exception is *not* caught:
124 .. code-block:: pycon
134 File "<stdin>", line 2, in <module>
135 binascii.Error: Non-hexadecimal digit found
137 This is expected. Notice that pure-Python modules behave the same way:
138 it is a part of how Python works.
140 The goal is to make extension modules safe at the C level, not to make
150 ---------------------
152 Sometimes, the state associated with a Python module is not specific to that module, but
153 to the entire process (or something else "more global" than a module).
156 - The ``readline`` module manages *the* terminal.
157 - A module running on a circuit board wants to control *the* on-board
160 In these cases, the Python module should provide *access* to the global
161 state, rather than *own* it. If possible, write the module so that
163 other libraries, whether for Python or other languages). If that is not
166 If it is necessary to use process-global state, the simplest way to
167 avoid issues with multiple interpreters is to explicitly prevent a
168 module from being loaded more than once per process—see
169 `Opt-Out: Limiting to One Module Object per Process`_.
172 Managing Per-Module State
173 -------------------------
175 To use per-module state, use
176 :ref:`multi-phase extension module initialization <multi-phase-initialization>`.
177 This signals that your module supports multiple interpreters correctly.
180 bytes of storage local to the module. Usually, this will be set to the
181 size of some module-specific ``struct``, which can store all of the
182 module's C-level state. In particular, it is where you should put
188 Another option is to store state in the module's ``__dict__``,
190 Python code. This usually means error- and type-checking at the C level,
191 which is easy to get wrong and hard to test sufficiently.
193 However, if module state is not needed in C code, storing it in
194 ``__dict__`` only is a good idea.
196 If the module state includes ``PyObject`` pointers, the module object
197 must hold references to those objects and implement the module-level hooks
200 require some work and make the code longer; this is the price for
203 An example of a module with per-module state is currently available as
205 example module initialization shown at the bottom of the file.
208 Opt-Out: Limiting to One Module Object per Process
209 --------------------------------------------------
211 A non-negative ``PyModuleDef.m_size`` signals that a module supports
212 multiple interpreters correctly. If this is not yet the case for your
213 module, you can explicitly make your module loadable only once per
219 exec_module(PyObject* module)
223 "cannot load module more than once per process");
224 return -1;
231 Module State Access from Functions
232 ----------------------------------
234 Accessing the state from module-level functions is straightforward.
235 Functions get the module object as their first argument; for extracting
239 func(PyObject *module, PyObject *args)
241 my_struct *state = (my_struct*)PyModule_GetState(module);
250 exception if there is no module state, i.e. ``PyModuleDef.m_size`` was
251 zero. In your own module, you're in control of ``m_size``, so this is
258 Traditionally, types defined in C code are *static*; that is,
263 between module objects requires paying attention to any state they own
267 .. impl-detail::
268 Sharing truly immutable objects between interpreters is fine,
273 depends on CPython's current, process-wide GIL.
275 Because they are immutable and process-global, static types cannot access
276 "their" module state.
277 If any method of such a type requires access to module state,
278 the type must be converted to a *heap-allocated type*, or *heap type*
282 For new modules, using heap types by default is a good rule of thumb.
286 -----------------------------------
290 from static types—that is, creating a type that works exactly like a given
297 Watch out for the following two points in particular (but note that this is not
308 -------------------
316 heap types, but :c:func:`PyType_FromModuleAndSpec` associates the module
317 with the class, allowing access to the module state from methods.
319 The class should generally be stored in *both* the module state (for
320 safe access from C) and the module's ``__dict__`` (for access from
324 Garbage-Collection Protocol
325 ---------------------------
334 That is, heap types should:
336 - Have the :c:data:`Py_TPFLAGS_HAVE_GC` flag.
337 - Define a traverse function using ``Py_tp_traverse``, which
340 Please refer to the :ref:`the documentation <type-structs>` of
345 (or another type), ensure that ``Py_TYPE(self)`` is visited only once.
350 base->tp_traverse(self, visit, arg)
354 if (base->tp_flags & Py_TPFLAGS_HEAPTYPE) {
360 It is not necessary to handle the type's reference count in ``tp_new``
364 Module State Access from Classes
365 --------------------------------
368 you can call :c:func:`PyType_GetModule` to get the associated module, and then
369 :c:func:`PyModule_GetState` to get the module's state.
371 To save a some tedious error-handling boilerplate code, you can combine
380 Module State Access from Regular Methods
381 ----------------------------------------
383 Accessing the module-level state from methods of a class is somewhat more
384 complicated, but is possible thanks to API introduced in Python 3.9.
386 get the module state from it.
388 The largest roadblock is getting *the class a method was defined in*, or
390 reference to the module it is part of.
393 is called on a *subclass* of your type, ``Py_TYPE(self)`` will refer to
394 that subclass, which may be defined in different module than yours.
401 .. code-block:: python
426 the state of its associated module.
455 Module State Access from Slot Methods, Getters and Setters
456 ----------------------------------------------------------
460 This is new in Python 3.11.
475 To access the module state in these cases, use the
476 :c:func:`PyType_GetModuleByDef` function, and pass in the module definition.
477 Once you have the module, call :c:func:`PyModule_GetState`
480 PyObject *module = PyType_GetModuleByDef(Py_TYPE(self), &module_def);
481 my_struct *state = (my_struct*)PyModule_GetState(module);
488 superclass that has a corresponding module.
494 return the module of the true defining class. However, it will always
495 return a module with the same definition, ensuring a compatible
499 Lifetime of the Module State
500 ----------------------------
502 When a module object is garbage-collected, its module state is freed.
503 For each pointer to (a part of) the module state, you must hold a reference
504 to the module object.
506 Usually this is not an issue, because types created with
508 to the module.
510 module state from other places, such as callbacks for external
517 Several issues around per-module state and heap types are still open.
519 Discussions about improving the situation are best held on the `capi-sig
520 mailing list <https://mail.python.org/mailman3/lists/capi-sig.python.org/>`__.
523 Per-Class Scope
524 ---------------
526 It is currently (as of Python 3.11) not possible to attach state to individual
529 per-class scope).
533 ---------------------------------
536 that is, creating a type that works exactly like a given static type.