1.. highlightlang:: c 2 3 4.. _memory: 5 6***************** 7Memory Management 8***************** 9 10.. sectionauthor:: Vladimir Marangozov <Vladimir.Marangozov@inrialpes.fr> 11 12 13 14.. _memoryoverview: 15 16Overview 17======== 18 19Memory management in Python involves a private heap containing all Python 20objects and data structures. The management of this private heap is ensured 21internally by the *Python memory manager*. The Python memory manager has 22different components which deal with various dynamic storage management aspects, 23like sharing, segmentation, preallocation or caching. 24 25At the lowest level, a raw memory allocator ensures that there is enough room in 26the private heap for storing all Python-related data by interacting with the 27memory manager of the operating system. On top of the raw memory allocator, 28several object-specific allocators operate on the same heap and implement 29distinct memory management policies adapted to the peculiarities of every object 30type. For example, integer objects are managed differently within the heap than 31strings, tuples or dictionaries because integers imply different storage 32requirements and speed/space tradeoffs. The Python memory manager thus delegates 33some of the work to the object-specific allocators, but ensures that the latter 34operate within the bounds of the private heap. 35 36It is important to understand that the management of the Python heap is 37performed by the interpreter itself and that the user has no control over it, 38even if she regularly manipulates object pointers to memory blocks inside that 39heap. The allocation of heap space for Python objects and other internal 40buffers is performed on demand by the Python memory manager through the Python/C 41API functions listed in this document. 42 43.. index:: 44 single: malloc() 45 single: calloc() 46 single: realloc() 47 single: free() 48 49To avoid memory corruption, extension writers should never try to operate on 50Python objects with the functions exported by the C library: :c:func:`malloc`, 51:c:func:`calloc`, :c:func:`realloc` and :c:func:`free`. This will result in mixed 52calls between the C allocator and the Python memory manager with fatal 53consequences, because they implement different algorithms and operate on 54different heaps. However, one may safely allocate and release memory blocks 55with the C library allocator for individual purposes, as shown in the following 56example:: 57 58 PyObject *res; 59 char *buf = (char *) malloc(BUFSIZ); /* for I/O */ 60 61 if (buf == NULL) 62 return PyErr_NoMemory(); 63 ...Do some I/O operation involving buf... 64 res = PyString_FromString(buf); 65 free(buf); /* malloc'ed */ 66 return res; 67 68In this example, the memory request for the I/O buffer is handled by the C 69library allocator. The Python memory manager is involved only in the allocation 70of the string object returned as a result. 71 72In most situations, however, it is recommended to allocate memory from the 73Python heap specifically because the latter is under control of the Python 74memory manager. For example, this is required when the interpreter is extended 75with new object types written in C. Another reason for using the Python heap is 76the desire to *inform* the Python memory manager about the memory needs of the 77extension module. Even when the requested memory is used exclusively for 78internal, highly-specific purposes, delegating all memory requests to the Python 79memory manager causes the interpreter to have a more accurate image of its 80memory footprint as a whole. Consequently, under certain circumstances, the 81Python memory manager may or may not trigger appropriate actions, like garbage 82collection, memory compaction or other preventive procedures. Note that by using 83the C library allocator as shown in the previous example, the allocated memory 84for the I/O buffer escapes completely the Python memory manager. 85 86 87.. _memoryinterface: 88 89Memory Interface 90================ 91 92The following function sets, modeled after the ANSI C standard, but specifying 93behavior when requesting zero bytes, are available for allocating and releasing 94memory from the Python heap: 95 96 97.. c:function:: void* PyMem_Malloc(size_t n) 98 99 Allocates *n* bytes and returns a pointer of type :c:type:`void\*` to the 100 allocated memory, or *NULL* if the request fails. Requesting zero bytes returns 101 a distinct non-*NULL* pointer if possible, as if ``PyMem_Malloc(1)`` had 102 been called instead. The memory will not have been initialized in any way. 103 104 105.. c:function:: void* PyMem_Realloc(void *p, size_t n) 106 107 Resizes the memory block pointed to by *p* to *n* bytes. The contents will be 108 unchanged to the minimum of the old and the new sizes. If *p* is *NULL*, the 109 call is equivalent to ``PyMem_Malloc(n)``; else if *n* is equal to zero, 110 the memory block is resized but is not freed, and the returned pointer is 111 non-*NULL*. Unless *p* is *NULL*, it must have been returned by a previous call 112 to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. If the request fails, 113 :c:func:`PyMem_Realloc` returns *NULL* and *p* remains a valid pointer to the 114 previous memory area. 115 116 117.. c:function:: void PyMem_Free(void *p) 118 119 Frees the memory block pointed to by *p*, which must have been returned by a 120 previous call to :c:func:`PyMem_Malloc` or :c:func:`PyMem_Realloc`. Otherwise, or 121 if ``PyMem_Free(p)`` has been called before, undefined behavior occurs. If 122 *p* is *NULL*, no operation is performed. 123 124The following type-oriented macros are provided for convenience. Note that 125*TYPE* refers to any C type. 126 127 128.. c:function:: TYPE* PyMem_New(TYPE, size_t n) 129 130 Same as :c:func:`PyMem_Malloc`, but allocates ``(n * sizeof(TYPE))`` bytes of 131 memory. Returns a pointer cast to :c:type:`TYPE\*`. The memory will not have 132 been initialized in any way. 133 134 135.. c:function:: TYPE* PyMem_Resize(void *p, TYPE, size_t n) 136 137 Same as :c:func:`PyMem_Realloc`, but the memory block is resized to ``(n * 138 sizeof(TYPE))`` bytes. Returns a pointer cast to :c:type:`TYPE\*`. On return, 139 *p* will be a pointer to the new memory area, or *NULL* in the event of 140 failure. This is a C preprocessor macro; p is always reassigned. Save 141 the original value of p to avoid losing memory when handling errors. 142 143 144.. c:function:: void PyMem_Del(void *p) 145 146 Same as :c:func:`PyMem_Free`. 147 148In addition, the following macro sets are provided for calling the Python memory 149allocator directly, without involving the C API functions listed above. However, 150note that their use does not preserve binary compatibility across Python 151versions and is therefore deprecated in extension modules. 152 153:c:func:`PyMem_MALLOC`, :c:func:`PyMem_REALLOC`, :c:func:`PyMem_FREE`. 154 155:c:func:`PyMem_NEW`, :c:func:`PyMem_RESIZE`, :c:func:`PyMem_DEL`. 156 157 158.. _memoryexamples: 159 160Examples 161======== 162 163Here is the example from section :ref:`memoryoverview`, rewritten so that the 164I/O buffer is allocated from the Python heap by using the first function set:: 165 166 PyObject *res; 167 char *buf = (char *) PyMem_Malloc(BUFSIZ); /* for I/O */ 168 169 if (buf == NULL) 170 return PyErr_NoMemory(); 171 /* ...Do some I/O operation involving buf... */ 172 res = PyString_FromString(buf); 173 PyMem_Free(buf); /* allocated with PyMem_Malloc */ 174 return res; 175 176The same code using the type-oriented function set:: 177 178 PyObject *res; 179 char *buf = PyMem_New(char, BUFSIZ); /* for I/O */ 180 181 if (buf == NULL) 182 return PyErr_NoMemory(); 183 /* ...Do some I/O operation involving buf... */ 184 res = PyString_FromString(buf); 185 PyMem_Del(buf); /* allocated with PyMem_New */ 186 return res; 187 188Note that in the two examples above, the buffer is always manipulated via 189functions belonging to the same set. Indeed, it is required to use the same 190memory API family for a given memory block, so that the risk of mixing different 191allocators is reduced to a minimum. The following code sequence contains two 192errors, one of which is labeled as *fatal* because it mixes two different 193allocators operating on different heaps. :: 194 195 char *buf1 = PyMem_New(char, BUFSIZ); 196 char *buf2 = (char *) malloc(BUFSIZ); 197 char *buf3 = (char *) PyMem_Malloc(BUFSIZ); 198 ... 199 PyMem_Del(buf3); /* Wrong -- should be PyMem_Free() */ 200 free(buf2); /* Right -- allocated via malloc() */ 201 free(buf1); /* Fatal -- should be PyMem_Del() */ 202 203In addition to the functions aimed at handling raw memory blocks from the Python 204heap, objects in Python are allocated and released with :c:func:`PyObject_New`, 205:c:func:`PyObject_NewVar` and :c:func:`PyObject_Del`. 206 207These will be explained in the next chapter on defining and implementing new 208object types in C. 209 210