Lines Matching full:memory
7 Memory Management
19 Memory management in Python involves a private heap containing all Python
21 internally by the *Python memory manager*. The Python memory manager has
25 At the lowest level, a raw memory allocator ensures that there is enough room in
27 memory manager of the operating system. On top of the raw memory allocator,
29 distinct memory management policies adapted to the peculiarities of every object
32 requirements and speed/space tradeoffs. The Python memory manager thus delegates
38 even if they regularly manipulate object pointers to memory blocks inside that
40 buffers is performed on demand by the Python memory manager through the Python/C
49 To avoid memory corruption, extension writers should never try to operate on
52 calls between the C allocator and the Python memory manager with fatal
54 different heaps. However, one may safely allocate and release memory blocks
68 In this example, the memory request for the I/O buffer is handled by the C
69 library allocator. The Python memory manager is involved only in the allocation
72 In most situations, however, it is recommended to allocate memory from the
74 memory manager. For example, this is required when the interpreter is extended
76 the desire to *inform* the Python memory manager about the memory needs of the
77 extension module. Even when the requested memory is used exclusively for
78 internal, highly specific purposes, delegating all memory requests to the Python
79 memory manager causes the interpreter to have a more accurate image of its
80 memory footprint as a whole. Consequently, under certain circumstances, the
81 Python memory manager may or may not trigger appropriate actions, like garbage
82 collection, memory compaction or other preventive procedures. Note that by using
83 the C library allocator as shown in the previous example, the allocated memory
84 for the I/O buffer escapes completely the Python memory manager.
89 the memory allocators used by Python.
92 statistics of the :ref:`pymalloc memory allocator <pymalloc>` every time a
103 how every domain allocates memory or what internal functions each domain calls
105 table can be found at :ref:`here <default-memory-allocators>`. There is no hard
106 requirement to use the memory returned by the allocation functions belonging to
108 recommended practice). For example, one could use the memory returned by
109 :c:func:`PyMem_RawMalloc` for allocating Python objects or the memory returned
110 by :c:func:`PyObject_Malloc` for allocating memory for buffers.
114 * Raw domain: intended for allocating memory for general-purpose memory
116 allocator can operate without the :term:`GIL`. The memory is requested directly
119 * "Mem" domain: intended for allocating memory for Python buffers and
120 general-purpose memory buffers where the allocation must be performed with
121 the :term:`GIL` held. The memory is taken from the Python private heap.
123 * Object domain: intended for allocating memory belonging to Python objects. The
124 memory is taken from the Python private heap.
126 When freeing memory previously allocated by the allocating functions belonging to a
128 :c:func:`PyMem_Free` must be used to free memory allocated using :c:func:`PyMem_Malloc`.
130 Raw Memory Interface
137 The :ref:`default raw memory allocator <default-memory-allocators>` uses
147 allocated memory, or ``NULL`` if the request fails.
150 if ``PyMem_RawMalloc(1)`` had been called instead. The memory will not have
157 a pointer of type :c:expr:`void*` to the allocated memory, or ``NULL`` if the
158 request fails. The memory is initialized to zeros.
169 Resizes the memory block pointed to by *p* to *n* bytes. The contents will
173 *n* is equal to zero, the memory block is resized but is not freed, and the
181 remains a valid pointer to the previous memory area.
186 Frees the memory block pointed to by *p*, which must have been returned by a
196 Memory Interface
201 memory from the Python heap.
203 The :ref:`default memory allocator <default-memory-allocators>` uses the
204 :ref:`pymalloc memory allocator <pymalloc>`.
218 allocated memory, or ``NULL`` if the request fails.
221 if ``PyMem_Malloc(1)`` had been called instead. The memory will not have
228 a pointer of type :c:expr:`void*` to the allocated memory, or ``NULL`` if the
229 request fails. The memory is initialized to zeros.
240 Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
244 is equal to zero, the memory block is resized but is not freed, and the
251 a valid pointer to the previous memory area.
256 Frees the memory block pointed to by *p*, which must have been returned by a
270 memory. Returns a pointer cast to :c:expr:`TYPE*`. The memory will not have
276 Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n *
278 *p* will be a pointer to the new memory area, or ``NULL`` in the event of
282 value of *p* to avoid losing memory when handling errors.
289 In addition, the following macro sets are provided for calling the Python memory
307 memory from the Python heap.
310 There is no guarantee that the memory returned by these allocators can be
313 the :ref:`Customize Memory Allocators <customize-memory-allocators>` section.
315 The :ref:`default object allocator <default-memory-allocators>` uses the
316 :ref:`pymalloc memory allocator <pymalloc>`.
326 allocated memory, or ``NULL`` if the request fails.
329 if ``PyObject_Malloc(1)`` had been called instead. The memory will not have
336 a pointer of type :c:expr:`void*` to the allocated memory, or ``NULL`` if the
337 request fails. The memory is initialized to zeros.
348 Resizes the memory block pointed to by *p* to *n* bytes. The contents will be
352 is equal to zero, the memory block is resized but is not freed, and the
359 a valid pointer to the previous memory area.
364 Frees the memory block pointed to by *p*, which must have been returned by a
372 .. _default-memory-allocators:
374 Default Memory Allocators
377 Default memory allocators:
393 * ``pymalloc``: :ref:`pymalloc memory allocator <pymalloc>`.
394 * "+ debug": with :ref:`debug hooks on the Python memory allocators
398 .. _customize-memory-allocators:
400 Customize Memory Allocators
407 Structure used to describe a memory block allocator. The structure has
415 …| ``void* malloc(void *ctx, size_t size)`` | allocate a memory block …
417 …| ``void* calloc(void *ctx, size_t nelem, size_t elsize)`` | allocate a memory block initialized …
420 …| ``void* realloc(void *ctx, void *ptr, size_t new_size)`` | allocate or resize a memory block …
422 …| ``void free(void *ctx, void *ptr)`` | free a memory block …
463 Get the memory block allocator of the specified domain.
468 Set the memory block allocator of the specified domain.
489 :c:func:`Py_InitializeFromConfig` to install a custom memory
505 Setup :ref:`debug hooks in the Python memory allocators <pymem-debug-hooks>`
506 to detect memory errors.
511 Debug hooks on the Python memory allocators
516 preinitialization <c-preinit>` to setup debug hooks on Python memory allocators
517 to detect memory errors.
525 These debug hooks fill dynamically allocated memory blocks with special,
526 recognizable bit patterns. Newly allocated memory is filled with the byte
527 ``0xCD`` (``PYMEM_CLEANBYTE``), freed memory is filled with the byte ``0xDD``
528 (``PYMEM_DEADBYTE``). Memory blocks are surrounded by "forbidden bytes"
535 called on a memory block allocated by :c:func:`PyMem_Malloc`.
544 traceback where a memory block was allocated. The traceback is only displayed
545 if :mod:`tracemalloc` is tracing Python memory allocations and the memory block
549 of *N* bytes requested. The memory layout is like so, where p represents the
556 to read in a memory dump).
568 The requested memory, filled with copies of PYMEM_CLEANBYTE, used to catch
569 reference to uninitialized memory. When a realloc-like function is called
570 requesting a larger memory block, the new excess bytes are also filled with
572 overwritten with PYMEM_DEADBYTE, to catch reference to freed memory. When
573 a realloc- like function is called requesting a smaller memory block, the
584 realloc-like function. Big-endian ``size_t``. If "bad memory" is detected
593 main failure mode is provoking a memory error when a program reads up one of
596 filled with PYMEM_DEADBYTE (meaning freed memory is getting used) or
597 PYMEM_CLEANBYTE (meaning uninitialized memory is getting used).
602 :mod:`tracemalloc` to get the traceback where a memory block was allocated.
620 to 512 bytes) with a short lifetime. It uses memory mappings called "arenas"
624 *pymalloc* is the :ref:`default allocator <default-memory-allocators>` of the
674 Track an allocated memory block in the :mod:`tracemalloc` module.
676 Return ``0`` on success, return ``-1`` on error (failed to allocate memory to
679 If memory block is already tracked, update the existing trace.
683 Untrack an allocated memory block in the :mod:`tracemalloc` module.
721 memory API family for a given memory block, so that the risk of mixing different
734 In addition to the functions aimed at handling raw memory blocks from the Python