Lines Matching full:memory
7 Memory Management
19 Memory management in Python involves a private heap containing all Python
21 internally by the *Python memory manager*. The Python memory manager has
25 At the lowest level, a raw memory allocator ensures that there is enough room in
27 memory manager of the operating system. On top of the raw memory allocator,
29 distinct memory management policies adapted to the peculiarities of every object
32 requirements and speed/space tradeoffs. The Python memory manager thus delegates
38 even if they regularly manipulate object pointers to memory blocks inside that
40 buffers is performed on demand by the Python memory manager through the Python/C
49 To avoid memory corruption, extension writers should never try to operate on
52 calls between the C allocator and the Python memory manager with fatal
54 different heaps. However, one may safely allocate and release memory blocks
68 In this example, the memory request for the I/O buffer is handled by the C
69 library allocator. The Python memory manager is involved only in the allocation
72 In most situations, however, it is recommended to allocate memory from the
74 memory manager. For example, this is required when the interpreter is extended
76 the desire to *inform* the Python memory manager about the memory needs of the
77 extension module. Even when the requested memory is used exclusively for
78 internal, highly-specific purposes, delegating all memory requests to the Python
79 memory manager causes the interpreter to have a more accurate image of its
80 memory footprint as a whole. Consequently, under certain circumstances, the
81 Python memory manager may or may not trigger appropriate actions, like garbage
82 collection, memory compaction or other preventive procedures. Note that by using
83 the C library allocator as shown in the previous example, the allocated memory
84 for the I/O buffer escapes completely the Python memory manager.
89 the memory allocators used by Python.
92 statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a
101 how every domain allocates memory or what internal functions each domain calls
103 table can be found at :ref:`here <default-memory-allocators>`. There is no hard
104 requirement to use the memory returned by the allocation functions belonging to
106 recommended practice). For example, one could use the memory returned by
107 :c:func:`PyMem_RawMalloc` for allocating Python objects or the memory returned
108 by :c:func:`PyObject_Malloc` for allocating memory for buffers.
112 * Raw domain: intended for allocating memory for general-purpose memory
114 allocator can operate without the :term:`GIL`. The memory is requested directly
117 * "Mem" domain: intended for allocating memory for Python buffers and
118 general-purpose memory buffers where the allocation must be performed with
119 the :term:`GIL` held. The memory is taken from the Python private heap.
121 * Object domain: intended for allocating memory belonging to Python objects. The
122 memory is taken from the Python private heap.
124 When freeing memory previously allocated by the allocating functions belonging to a
126 :c:func:`PyMem_Free` must be used to free memory allocated using :c:func:`PyMem_Malloc`.
128 Raw Memory Interface
135 The :ref:`default raw memory allocator <default-memory-allocators>` uses
145 allocated memory, or ``NULL`` if the request fails.
148 if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
155 a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the
156 request fails. The memory is initialized to zeros.
167 Resizes the memory block pointed to by *p* to *n* bytes. The contents will
171 *n* is equal to zero, the memory block is resized but is not freed, and the
179 remains a valid pointer to the previous memory area.
184 Frees the memory block pointed to by *p*, which must have been returned by a
194 Memory Interface
199 memory from the Python heap.
201 The :ref:`default memory allocator <default-memory-allocators>` uses the
202 :ref:`pymalloc memory allocator <pymalloc>`.
216 allocated memory, or ``NULL`` if the request fails.
219 if ``PyMem_Malloc(1)`` had been called instead. The memory will not have
226 a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the
227 request fails. The memory is initialized to zeros.
238 Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
242 is equal to zero, the memory block is resized but is not freed, and the
249 a valid pointer to the previous memory area.
254 Frees the memory block pointed to by *p*, which must have been returned by a
268 memory. Returns a pointer cast to :c:type:`TYPE*`. The memory will not have
274 Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n *
276 *p* will be a pointer to the new memory area, or ``NULL`` in the event of
280 value of *p* to avoid losing memory when handling errors.
287 In addition, the following macro sets are provided for calling the Python memory
305 memory from the Python heap.
308 There is no guarantee that the memory returned by these allocators can be
311 the :ref:`Customize Memory Allocators <customize-memory-allocators>` section.
313 The :ref:`default object allocator <default-memory-allocators>` uses the
314 :ref:`pymalloc memory allocator <pymalloc>`.
324 allocated memory, or ``NULL`` if the request fails.
327 if ``PyObject_Malloc(1)`` had been called instead. The memory will not have
334 a pointer of type :c:type:`void*` to the allocated memory, or ``NULL`` if the
335 request fails. The memory is initialized to zeros.
346 Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
350 is equal to zero, the memory block is resized but is not freed, and the
357 a valid pointer to the previous memory area.
362 Frees the memory block pointed to by *p*, which must have been returned by a
370 .. _default-memory-allocators:
372 Default Memory Allocators
375 Default memory allocators:
391 * ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
392 * "+ debug": with :ref:`debug hooks on the Python memory allocators
396 .. _customize-memory-allocators:
398 Customize Memory Allocators
405 Structure used to describe a memory block allocator. The structure has
413 …| ``void* malloc(void *ctx, size_t size)`` | allocate a memory block …
415 …| ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized …
418 …| ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block …
420 …| ``void free(void *ctx, void *ptr)`` | free a memory block …
461 Get the memory block allocator of the specified domain.
466 Set the memory block allocator of the specified domain.
482 Setup :ref:`debug hooks in the Python memory allocators <pymem-debug-hooks>`
483 to detect memory errors.
488 Debug hooks on the Python memory allocators
493 preinitialization <c-preinit>` to setup debug hooks on Python memory allocators
494 to detect memory errors.
502 These debug hooks fill dynamically allocated memory blocks with special,
503 recognizable bit patterns. Newly allocated memory is filled with the byte
504 ``0xCD`` (``PYMEM_CLEANBYTE``), freed memory is filled with the byte ``0xDD``
505 (``PYMEM_DEADBYTE``). Memory blocks are surrounded by "forbidden bytes"
512 called on a memory block allocated by :c:func:`PyMem_Malloc`.
521 traceback where a memory block was allocated. The traceback is only displayed
522 if :mod:`tracemalloc` is tracing Python memory allocations and the memory block
526 of *N* bytes requested. The memory layout is like so, where p represents the
533 to read in a memory dump).
545 The requested memory, filled with copies of PYMEM_CLEANBYTE, used to catch
546 reference to uninitialized memory. When a realloc-like function is called
547 requesting a larger memory block, the new excess bytes are also filled with
549 overwritten with PYMEM_DEADBYTE, to catch reference to freed memory. When
550 a realloc- like function is called requesting a smaller memory block, the
561 realloc-like function. Big-endian ``size_t``. If "bad memory" is detected
570 main failure mode is provoking a memory error when a program reads up one of
573 filled with PYMEM_DEADBYTE (meaning freed memory is getting used) or
574 PYMEM_CLEANBYTE (meaning uninitialized memory is getting used).
579 :mod:`tracemalloc` to get the traceback where a memory block was allocated.
597 to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
601 *pymalloc* is the :ref:`default allocator <default-memory-allocators>` of the
651 Track an allocated memory block in the :mod:`tracemalloc` module.
653 Return ``0`` on success, return ``-1`` on error (failed to allocate memory to
656 If memory block is already tracked, update the existing trace.
660 Untrack an allocated memory block in the :mod:`tracemalloc` module.
698 memory API family for a given memory block, so that the risk of mixing different
711 In addition to the functions aimed at handling raw memory blocks from the Python