1.. highlightlang:: c 2 3.. _bufferobjects: 4 5Buffers and Memoryview Objects 6------------------------------ 7 8.. sectionauthor:: Greg Stein <gstein@lyra.org> 9.. sectionauthor:: Benjamin Peterson 10 11 12.. index:: 13 object: buffer 14 single: buffer interface 15 16Python objects implemented in C can export a group of functions called the 17"buffer interface." These functions can be used by an object to expose its 18data in a raw, byte-oriented format. Clients of the object can use the buffer 19interface to access the object data directly, without needing to copy it 20first. 21 22Two examples of objects that support the buffer interface are strings and 23arrays. The string object exposes the character contents in the buffer 24interface's byte-oriented form. An array can only expose its contents via the 25old-style buffer interface. This limitation does not apply to Python 3, 26where :class:`memoryview` objects can be constructed from arrays, too. 27Array elements may be multi-byte values. 28 29An example user of the buffer interface is the file object's :meth:`write` 30method. Any object that can export a series of bytes through the buffer 31interface can be written to a file. There are a number of format codes to 32:c:func:`PyArg_ParseTuple` that operate against an object's buffer interface, 33returning data from the target object. 34 35Starting from version 1.6, Python has been providing Python-level buffer 36objects and a C-level buffer API so that any built-in or used-defined type can 37expose its characteristics. Both, however, have been deprecated because of 38various shortcomings, and have been officially removed in Python 3 in favour 39of a new C-level buffer API and a new Python-level object named 40:class:`memoryview`. 41 42The new buffer API has been backported to Python 2.6, and the 43:class:`memoryview` object has been backported to Python 2.7. It is strongly 44advised to use them rather than the old APIs, unless you are blocked from 45doing so for compatibility reasons. 46 47 48The new-style Py_buffer struct 49============================== 50 51 52.. c:type:: Py_buffer 53 54 .. c:member:: void *buf 55 56 A pointer to the start of the memory for the object. 57 58 .. c:member:: Py_ssize_t len 59 :noindex: 60 61 The total length of the memory in bytes. 62 63 .. c:member:: int readonly 64 65 An indicator of whether the buffer is read only. 66 67 .. c:member:: const char *format 68 :noindex: 69 70 A *NULL* terminated string in :mod:`struct` module style syntax giving 71 the contents of the elements available through the buffer. If this is 72 *NULL*, ``"B"`` (unsigned bytes) is assumed. 73 74 .. c:member:: int ndim 75 76 The number of dimensions the memory represents as a multi-dimensional 77 array. If it is ``0``, :c:data:`strides` and :c:data:`suboffsets` must be 78 *NULL*. 79 80 .. c:member:: Py_ssize_t *shape 81 82 An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the 83 shape of the memory as a multi-dimensional array. Note that 84 ``((*shape)[0] * ... * (*shape)[ndims-1])*itemsize`` should be equal to 85 :c:data:`len`. 86 87 .. c:member:: Py_ssize_t *strides 88 89 An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim` giving the 90 number of bytes to skip to get to a new element in each dimension. 91 92 .. c:member:: Py_ssize_t *suboffsets 93 94 An array of :c:type:`Py_ssize_t`\s the length of :c:data:`ndim`. If these 95 suboffset numbers are greater than or equal to 0, then the value stored 96 along the indicated dimension is a pointer and the suboffset value 97 dictates how many bytes to add to the pointer after de-referencing. A 98 suboffset value that it negative indicates that no de-referencing should 99 occur (striding in a contiguous memory block). 100 101 If all suboffsets are negative (i.e. no de-referencing is needed, then 102 this field must be NULL (the default value). 103 104 Here is a function that returns a pointer to the element in an N-D array 105 pointed to by an N-dimensional index when there are both non-NULL strides 106 and suboffsets:: 107 108 void *get_item_pointer(int ndim, void *buf, Py_ssize_t *strides, 109 Py_ssize_t *suboffsets, Py_ssize_t *indices) { 110 char *pointer = (char*)buf; 111 int i; 112 for (i = 0; i < ndim; i++) { 113 pointer += strides[i] * indices[i]; 114 if (suboffsets[i] >=0 ) { 115 pointer = *((char**)pointer) + suboffsets[i]; 116 } 117 } 118 return (void*)pointer; 119 } 120 121 122 .. c:member:: Py_ssize_t itemsize 123 124 This is a storage for the itemsize (in bytes) of each element of the 125 shared memory. It is technically un-necessary as it can be obtained 126 using :c:func:`PyBuffer_SizeFromFormat`, however an exporter may know 127 this information without parsing the format string and it is necessary 128 to know the itemsize for proper interpretation of striding. Therefore, 129 storing it is more convenient and faster. 130 131 .. c:member:: void *internal 132 133 This is for use internally by the exporting object. For example, this 134 might be re-cast as an integer by the exporter and used to store flags 135 about whether or not the shape, strides, and suboffsets arrays must be 136 freed when the buffer is released. The consumer should never alter this 137 value. 138 139 140Buffer related functions 141======================== 142 143 144.. c:function:: int PyObject_CheckBuffer(PyObject *obj) 145 146 Return ``1`` if *obj* supports the buffer interface otherwise ``0``. 147 148 149.. c:function:: int PyObject_GetBuffer(PyObject *obj, Py_buffer *view, int flags) 150 151 Export *obj* into a :c:type:`Py_buffer`, *view*. These arguments must 152 never be *NULL*. The *flags* argument is a bit field indicating what 153 kind of buffer the caller is prepared to deal with and therefore what 154 kind of buffer the exporter is allowed to return. The buffer interface 155 allows for complicated memory sharing possibilities, but some caller may 156 not be able to handle all the complexity but may want to see if the 157 exporter will let them take a simpler view to its memory. 158 159 Some exporters may not be able to share memory in every possible way and 160 may need to raise errors to signal to some consumers that something is 161 just not possible. These errors should be a :exc:`BufferError` unless 162 there is another error that is actually causing the problem. The 163 exporter can use flags information to simplify how much of the 164 :c:data:`Py_buffer` structure is filled in with non-default values and/or 165 raise an error if the object can't support a simpler view of its memory. 166 167 ``0`` is returned on success and ``-1`` on error. 168 169 The following table gives possible values to the *flags* arguments. 170 171 +-------------------------------+---------------------------------------------------+ 172 | Flag | Description | 173 +===============================+===================================================+ 174 | :c:macro:`PyBUF_SIMPLE` | This is the default flag state. The returned | 175 | | buffer may or may not have writable memory. The | 176 | | format of the data will be assumed to be unsigned | 177 | | bytes. This is a "stand-alone" flag constant. It | 178 | | never needs to be '|'d to the others. The exporter| 179 | | will raise an error if it cannot provide such a | 180 | | contiguous buffer of bytes. | 181 | | | 182 +-------------------------------+---------------------------------------------------+ 183 | :c:macro:`PyBUF_WRITABLE` | The returned buffer must be writable. If it is | 184 | | not writable, then raise an error. | 185 +-------------------------------+---------------------------------------------------+ 186 | :c:macro:`PyBUF_STRIDES` | This implies :c:macro:`PyBUF_ND`. The returned | 187 | | buffer must provide strides information (i.e. the | 188 | | strides cannot be NULL). This would be used when | 189 | | the consumer can handle strided, discontiguous | 190 | | arrays. Handling strides automatically assumes | 191 | | you can handle shape. The exporter can raise an | 192 | | error if a strided representation of the data is | 193 | | not possible (i.e. without the suboffsets). | 194 | | | 195 +-------------------------------+---------------------------------------------------+ 196 | :c:macro:`PyBUF_ND` | The returned buffer must provide shape | 197 | | information. The memory will be assumed C-style | 198 | | contiguous (last dimension varies the | 199 | | fastest). The exporter may raise an error if it | 200 | | cannot provide this kind of contiguous buffer. If | 201 | | this is not given then shape will be *NULL*. | 202 | | | 203 | | | 204 | | | 205 +-------------------------------+---------------------------------------------------+ 206 |:c:macro:`PyBUF_C_CONTIGUOUS` | These flags indicate that the contiguity returned | 207 |:c:macro:`PyBUF_F_CONTIGUOUS` | buffer must be respectively, C-contiguous (last | 208 |:c:macro:`PyBUF_ANY_CONTIGUOUS`| dimension varies the fastest), Fortran contiguous | 209 | | (first dimension varies the fastest) or either | 210 | | one. All of these flags imply | 211 | | :c:macro:`PyBUF_STRIDES` and guarantee that the | 212 | | strides buffer info structure will be filled in | 213 | | correctly. | 214 | | | 215 +-------------------------------+---------------------------------------------------+ 216 | :c:macro:`PyBUF_INDIRECT` | This flag indicates the returned buffer must have | 217 | | suboffsets information (which can be NULL if no | 218 | | suboffsets are needed). This can be used when | 219 | | the consumer can handle indirect array | 220 | | referencing implied by these suboffsets. This | 221 | | implies :c:macro:`PyBUF_STRIDES`. | 222 | | | 223 | | | 224 | | | 225 +-------------------------------+---------------------------------------------------+ 226 | :c:macro:`PyBUF_FORMAT` | The returned buffer must have true format | 227 | | information if this flag is provided. This would | 228 | | be used when the consumer is going to be checking | 229 | | for what 'kind' of data is actually stored. An | 230 | | exporter should always be able to provide this | 231 | | information if requested. If format is not | 232 | | explicitly requested then the format must be | 233 | | returned as *NULL* (which means ``'B'``, or | 234 | | unsigned bytes) | 235 +-------------------------------+---------------------------------------------------+ 236 | :c:macro:`PyBUF_STRIDED` | This is equivalent to ``(PyBUF_STRIDES | | 237 | | PyBUF_WRITABLE)``. | 238 +-------------------------------+---------------------------------------------------+ 239 | :c:macro:`PyBUF_STRIDED_RO` | This is equivalent to ``(PyBUF_STRIDES)``. | 240 | | | 241 +-------------------------------+---------------------------------------------------+ 242 | :c:macro:`PyBUF_RECORDS` | This is equivalent to ``(PyBUF_STRIDES | | 243 | | PyBUF_FORMAT | PyBUF_WRITABLE)``. | 244 +-------------------------------+---------------------------------------------------+ 245 | :c:macro:`PyBUF_RECORDS_RO` | This is equivalent to ``(PyBUF_STRIDES | | 246 | | PyBUF_FORMAT)``. | 247 +-------------------------------+---------------------------------------------------+ 248 | :c:macro:`PyBUF_FULL` | This is equivalent to ``(PyBUF_INDIRECT | | 249 | | PyBUF_FORMAT | PyBUF_WRITABLE)``. | 250 +-------------------------------+---------------------------------------------------+ 251 | :c:macro:`PyBUF_FULL_RO` | This is equivalent to ``(PyBUF_INDIRECT | | 252 | | PyBUF_FORMAT)``. | 253 +-------------------------------+---------------------------------------------------+ 254 | :c:macro:`PyBUF_CONTIG` | This is equivalent to ``(PyBUF_ND | | 255 | | PyBUF_WRITABLE)``. | 256 +-------------------------------+---------------------------------------------------+ 257 | :c:macro:`PyBUF_CONTIG_RO` | This is equivalent to ``(PyBUF_ND)``. | 258 | | | 259 +-------------------------------+---------------------------------------------------+ 260 261 262.. c:function:: void PyBuffer_Release(Py_buffer *view) 263 264 Release the buffer *view*. This should be called when the buffer 265 is no longer being used as it may free memory from it. 266 267 268.. c:function:: Py_ssize_t PyBuffer_SizeFromFormat(const char *) 269 270 Return the implied :c:data:`~Py_buffer.itemsize` from the struct-stype 271 :c:data:`~Py_buffer.format`. 272 273 274.. c:function:: int PyBuffer_IsContiguous(Py_buffer *view, char fortran) 275 276 Return ``1`` if the memory defined by the *view* is C-style (*fortran* is 277 ``'C'``) or Fortran-style (*fortran* is ``'F'``) contiguous or either one 278 (*fortran* is ``'A'``). Return ``0`` otherwise. 279 280 281.. c:function:: void PyBuffer_FillContiguousStrides(int ndim, Py_ssize_t *shape, Py_ssize_t *strides, Py_ssize_t itemsize, char fortran) 282 283 Fill the *strides* array with byte-strides of a contiguous (C-style if 284 *fortran* is ``'C'`` or Fortran-style if *fortran* is ``'F'``) array of the 285 given shape with the given number of bytes per element. 286 287 288.. c:function:: int PyBuffer_FillInfo(Py_buffer *view, PyObject *obj, void *buf, Py_ssize_t len, int readonly, int infoflags) 289 290 Fill in a buffer-info structure, *view*, correctly for an exporter that can 291 only share a contiguous chunk of memory of "unsigned bytes" of the given 292 length. Return ``0`` on success and ``-1`` (with raising an error) on error. 293 294 295MemoryView objects 296================== 297 298.. versionadded:: 2.7 299 300A :class:`memoryview` object exposes the new C level buffer interface as a 301Python object which can then be passed around like any other object. 302 303.. c:function:: PyObject *PyMemoryView_FromObject(PyObject *obj) 304 305 Create a memoryview object from an object that defines the new buffer 306 interface. 307 308 309.. c:function:: PyObject *PyMemoryView_FromBuffer(Py_buffer *view) 310 311 Create a memoryview object wrapping the given buffer-info structure *view*. 312 The memoryview object then owns the buffer, which means you shouldn't 313 try to release it yourself: it will be released on deallocation of the 314 memoryview object. 315 316 317.. c:function:: PyObject *PyMemoryView_GetContiguous(PyObject *obj, int buffertype, char order) 318 319 Create a memoryview object to a contiguous chunk of memory (in either 320 'C' or 'F'ortran *order*) from an object that defines the buffer 321 interface. If memory is contiguous, the memoryview object points to the 322 original memory. Otherwise copy is made and the memoryview points to a 323 new bytes object. 324 325 326.. c:function:: int PyMemoryView_Check(PyObject *obj) 327 328 Return true if the object *obj* is a memoryview object. It is not 329 currently allowed to create subclasses of :class:`memoryview`. 330 331 332.. c:function:: Py_buffer *PyMemoryView_GET_BUFFER(PyObject *obj) 333 334 Return a pointer to the buffer-info structure wrapped by the given 335 object. The object **must** be a memoryview instance; this macro doesn't 336 check its type, you must do it yourself or you will risk crashes. 337 338 339Old-style buffer objects 340======================== 341 342.. index:: single: PyBufferProcs 343 344More information on the old buffer interface is provided in the section 345:ref:`buffer-structs`, under the description for :c:type:`PyBufferProcs`. 346 347A "buffer object" is defined in the :file:`bufferobject.h` header (included by 348:file:`Python.h`). These objects look very similar to string objects at the 349Python programming level: they support slicing, indexing, concatenation, and 350some other standard string operations. However, their data can come from one 351of two sources: from a block of memory, or from another object which exports 352the buffer interface. 353 354Buffer objects are useful as a way to expose the data from another object's 355buffer interface to the Python programmer. They can also be used as a 356zero-copy slicing mechanism. Using their ability to reference a block of 357memory, it is possible to expose any data to the Python programmer quite 358easily. The memory could be a large, constant array in a C extension, it could 359be a raw block of memory for manipulation before passing to an operating 360system library, or it could be used to pass around structured data in its 361native, in-memory format. 362 363 364.. c:type:: PyBufferObject 365 366 This subtype of :c:type:`PyObject` represents a buffer object. 367 368 369.. c:var:: PyTypeObject PyBuffer_Type 370 371 .. index:: single: BufferType (in module types) 372 373 The instance of :c:type:`PyTypeObject` which represents the Python buffer type; 374 it is the same object as ``buffer`` and ``types.BufferType`` in the Python 375 layer. . 376 377 378.. c:var:: int Py_END_OF_BUFFER 379 380 This constant may be passed as the *size* parameter to 381 :c:func:`PyBuffer_FromObject` or :c:func:`PyBuffer_FromReadWriteObject`. It 382 indicates that the new :c:type:`PyBufferObject` should refer to *base* 383 object from the specified *offset* to the end of its exported buffer. 384 Using this enables the caller to avoid querying the *base* object for its 385 length. 386 387 388.. c:function:: int PyBuffer_Check(PyObject *p) 389 390 Return true if the argument has type :c:data:`PyBuffer_Type`. 391 392 393.. c:function:: PyObject* PyBuffer_FromObject(PyObject *base, Py_ssize_t offset, Py_ssize_t size) 394 395 Return a new read-only buffer object. This raises :exc:`TypeError` if 396 *base* doesn't support the read-only buffer protocol or doesn't provide 397 exactly one buffer segment, or it raises :exc:`ValueError` if *offset* is 398 less than zero. The buffer will hold a reference to the *base* object, and 399 the buffer's contents will refer to the *base* object's buffer interface, 400 starting as position *offset* and extending for *size* bytes. If *size* is 401 :const:`Py_END_OF_BUFFER`, then the new buffer's contents extend to the 402 length of the *base* object's exported buffer data. 403 404 .. versionchanged:: 2.5 405 This function used an :c:type:`int` type for *offset* and *size*. This 406 might require changes in your code for properly supporting 64-bit 407 systems. 408 409 410.. c:function:: PyObject* PyBuffer_FromReadWriteObject(PyObject *base, Py_ssize_t offset, Py_ssize_t size) 411 412 Return a new writable buffer object. Parameters and exceptions are similar 413 to those for :c:func:`PyBuffer_FromObject`. If the *base* object does not 414 export the writeable buffer protocol, then :exc:`TypeError` is raised. 415 416 .. versionchanged:: 2.5 417 This function used an :c:type:`int` type for *offset* and *size*. This 418 might require changes in your code for properly supporting 64-bit 419 systems. 420 421 422.. c:function:: PyObject* PyBuffer_FromMemory(void *ptr, Py_ssize_t size) 423 424 Return a new read-only buffer object that reads from a specified location 425 in memory, with a specified size. The caller is responsible for ensuring 426 that the memory buffer, passed in as *ptr*, is not deallocated while the 427 returned buffer object exists. Raises :exc:`ValueError` if *size* is less 428 than zero. Note that :const:`Py_END_OF_BUFFER` may *not* be passed for the 429 *size* parameter; :exc:`ValueError` will be raised in that case. 430 431 .. versionchanged:: 2.5 432 This function used an :c:type:`int` type for *size*. This might require 433 changes in your code for properly supporting 64-bit systems. 434 435 436.. c:function:: PyObject* PyBuffer_FromReadWriteMemory(void *ptr, Py_ssize_t size) 437 438 Similar to :c:func:`PyBuffer_FromMemory`, but the returned buffer is 439 writable. 440 441 .. versionchanged:: 2.5 442 This function used an :c:type:`int` type for *size*. This might require 443 changes in your code for properly supporting 64-bit systems. 444 445 446.. c:function:: PyObject* PyBuffer_New(Py_ssize_t size) 447 448 Return a new writable buffer object that maintains its own memory buffer of 449 *size* bytes. :exc:`ValueError` is returned if *size* is not zero or 450 positive. Note that the memory buffer (as returned by 451 :c:func:`PyObject_AsWriteBuffer`) is not specifically aligned. 452 453 .. versionchanged:: 2.5 454 This function used an :c:type:`int` type for *size*. This might require 455 changes in your code for properly supporting 64-bit systems. 456