1:mod:`multiprocessing.shared_memory` --- Provides shared memory for direct access across processes 2=================================================================================================== 3 4.. module:: multiprocessing.shared_memory 5 :synopsis: Provides shared memory for direct access across processes. 6 7**Source code:** :source:`Lib/multiprocessing/shared_memory.py` 8 9.. versionadded:: 3.8 10 11.. index:: 12 single: Shared Memory 13 single: POSIX Shared Memory 14 single: Named Shared Memory 15 16-------------- 17 18This module provides a class, :class:`SharedMemory`, for the allocation 19and management of shared memory to be accessed by one or more processes 20on a multicore or symmetric multiprocessor (SMP) machine. To assist with 21the life-cycle management of shared memory especially across distinct 22processes, a :class:`~multiprocessing.managers.BaseManager` subclass, 23:class:`SharedMemoryManager`, is also provided in the 24``multiprocessing.managers`` module. 25 26In this module, shared memory refers to "System V style" shared memory blocks 27(though is not necessarily implemented explicitly as such) and does not refer 28to "distributed shared memory". This style of shared memory permits distinct 29processes to potentially read and write to a common (or shared) region of 30volatile memory. Processes are conventionally limited to only have access to 31their own process memory space but shared memory permits the sharing 32of data between processes, avoiding the need to instead send messages between 33processes containing that data. Sharing data directly via memory can provide 34significant performance benefits compared to sharing data via disk or socket 35or other communications requiring the serialization/deserialization and 36copying of data. 37 38 39.. class:: SharedMemory(name=None, create=False, size=0) 40 41 Creates a new shared memory block or attaches to an existing shared 42 memory block. Each shared memory block is assigned a unique name. 43 In this way, one process can create a shared memory block with a 44 particular name and a different process can attach to that same shared 45 memory block using that same name. 46 47 As a resource for sharing data across processes, shared memory blocks 48 may outlive the original process that created them. When one process 49 no longer needs access to a shared memory block that might still be 50 needed by other processes, the :meth:`close()` method should be called. 51 When a shared memory block is no longer needed by any process, the 52 :meth:`unlink()` method should be called to ensure proper cleanup. 53 54 *name* is the unique name for the requested shared memory, specified as 55 a string. When creating a new shared memory block, if ``None`` (the 56 default) is supplied for the name, a novel name will be generated. 57 58 *create* controls whether a new shared memory block is created (``True``) 59 or an existing shared memory block is attached (``False``). 60 61 *size* specifies the requested number of bytes when creating a new shared 62 memory block. Because some platforms choose to allocate chunks of memory 63 based upon that platform's memory page size, the exact size of the shared 64 memory block may be larger or equal to the size requested. When attaching 65 to an existing shared memory block, the ``size`` parameter is ignored. 66 67 .. method:: close() 68 69 Closes access to the shared memory from this instance. In order to 70 ensure proper cleanup of resources, all instances should call 71 ``close()`` once the instance is no longer needed. Note that calling 72 ``close()`` does not cause the shared memory block itself to be 73 destroyed. 74 75 .. method:: unlink() 76 77 Requests that the underlying shared memory block be destroyed. In 78 order to ensure proper cleanup of resources, ``unlink()`` should be 79 called once (and only once) across all processes which have need 80 for the shared memory block. After requesting its destruction, a 81 shared memory block may or may not be immediately destroyed and 82 this behavior may differ across platforms. Attempts to access data 83 inside the shared memory block after ``unlink()`` has been called may 84 result in memory access errors. Note: the last process relinquishing 85 its hold on a shared memory block may call ``unlink()`` and 86 :meth:`close()` in either order. 87 88 .. attribute:: buf 89 90 A memoryview of contents of the shared memory block. 91 92 .. attribute:: name 93 94 Read-only access to the unique name of the shared memory block. 95 96 .. attribute:: size 97 98 Read-only access to size in bytes of the shared memory block. 99 100 101The following example demonstrates low-level use of :class:`SharedMemory` 102instances:: 103 104 >>> from multiprocessing import shared_memory 105 >>> shm_a = shared_memory.SharedMemory(create=True, size=10) 106 >>> type(shm_a.buf) 107 <class 'memoryview'> 108 >>> buffer = shm_a.buf 109 >>> len(buffer) 110 10 111 >>> buffer[:4] = bytearray([22, 33, 44, 55]) # Modify multiple at once 112 >>> buffer[4] = 100 # Modify single byte at a time 113 >>> # Attach to an existing shared memory block 114 >>> shm_b = shared_memory.SharedMemory(shm_a.name) 115 >>> import array 116 >>> array.array('b', shm_b.buf[:5]) # Copy the data into a new array.array 117 array('b', [22, 33, 44, 55, 100]) 118 >>> shm_b.buf[:5] = b'howdy' # Modify via shm_b using bytes 119 >>> bytes(shm_a.buf[:5]) # Access via shm_a 120 b'howdy' 121 >>> shm_b.close() # Close each SharedMemory instance 122 >>> shm_a.close() 123 >>> shm_a.unlink() # Call unlink only once to release the shared memory 124 125 126 127The following example demonstrates a practical use of the :class:`SharedMemory` 128class with `NumPy arrays <https://www.numpy.org/>`_, accessing the 129same ``numpy.ndarray`` from two distinct Python shells: 130 131.. doctest:: 132 :options: +SKIP 133 134 >>> # In the first Python interactive shell 135 >>> import numpy as np 136 >>> a = np.array([1, 1, 2, 3, 5, 8]) # Start with an existing NumPy array 137 >>> from multiprocessing import shared_memory 138 >>> shm = shared_memory.SharedMemory(create=True, size=a.nbytes) 139 >>> # Now create a NumPy array backed by shared memory 140 >>> b = np.ndarray(a.shape, dtype=a.dtype, buffer=shm.buf) 141 >>> b[:] = a[:] # Copy the original data into shared memory 142 >>> b 143 array([1, 1, 2, 3, 5, 8]) 144 >>> type(b) 145 <class 'numpy.ndarray'> 146 >>> type(a) 147 <class 'numpy.ndarray'> 148 >>> shm.name # We did not specify a name so one was chosen for us 149 'psm_21467_46075' 150 151 >>> # In either the same shell or a new Python shell on the same machine 152 >>> import numpy as np 153 >>> from multiprocessing import shared_memory 154 >>> # Attach to the existing shared memory block 155 >>> existing_shm = shared_memory.SharedMemory(name='psm_21467_46075') 156 >>> # Note that a.shape is (6,) and a.dtype is np.int64 in this example 157 >>> c = np.ndarray((6,), dtype=np.int64, buffer=existing_shm.buf) 158 >>> c 159 array([1, 1, 2, 3, 5, 8]) 160 >>> c[-1] = 888 161 >>> c 162 array([ 1, 1, 2, 3, 5, 888]) 163 164 >>> # Back in the first Python interactive shell, b reflects this change 165 >>> b 166 array([ 1, 1, 2, 3, 5, 888]) 167 168 >>> # Clean up from within the second Python shell 169 >>> del c # Unnecessary; merely emphasizing the array is no longer used 170 >>> existing_shm.close() 171 172 >>> # Clean up from within the first Python shell 173 >>> del b # Unnecessary; merely emphasizing the array is no longer used 174 >>> shm.close() 175 >>> shm.unlink() # Free and release the shared memory block at the very end 176 177 178.. class:: SharedMemoryManager([address[, authkey]]) 179 :module: multiprocessing.managers 180 181 A subclass of :class:`~multiprocessing.managers.BaseManager` which can be 182 used for the management of shared memory blocks across processes. 183 184 A call to :meth:`~multiprocessing.managers.BaseManager.start` on a 185 :class:`SharedMemoryManager` instance causes a new process to be started. 186 This new process's sole purpose is to manage the life cycle 187 of all shared memory blocks created through it. To trigger the release 188 of all shared memory blocks managed by that process, call 189 :meth:`~multiprocessing.managers.BaseManager.shutdown()` on the instance. 190 This triggers a :meth:`SharedMemory.unlink()` call on all of the 191 :class:`SharedMemory` objects managed by that process and then 192 stops the process itself. By creating ``SharedMemory`` instances 193 through a ``SharedMemoryManager``, we avoid the need to manually track 194 and trigger the freeing of shared memory resources. 195 196 This class provides methods for creating and returning :class:`SharedMemory` 197 instances and for creating a list-like object (:class:`ShareableList`) 198 backed by shared memory. 199 200 Refer to :class:`multiprocessing.managers.BaseManager` for a description 201 of the inherited *address* and *authkey* optional input arguments and how 202 they may be used to connect to an existing ``SharedMemoryManager`` service 203 from other processes. 204 205 .. method:: SharedMemory(size) 206 207 Create and return a new :class:`SharedMemory` object with the 208 specified ``size`` in bytes. 209 210 .. method:: ShareableList(sequence) 211 212 Create and return a new :class:`ShareableList` object, initialized 213 by the values from the input ``sequence``. 214 215 216The following example demonstrates the basic mechanisms of a 217:class:`SharedMemoryManager`: 218 219.. doctest:: 220 :options: +SKIP 221 222 >>> from multiprocessing.managers import SharedMemoryManager 223 >>> smm = SharedMemoryManager() 224 >>> smm.start() # Start the process that manages the shared memory blocks 225 >>> sl = smm.ShareableList(range(4)) 226 >>> sl 227 ShareableList([0, 1, 2, 3], name='psm_6572_7512') 228 >>> raw_shm = smm.SharedMemory(size=128) 229 >>> another_sl = smm.ShareableList('alpha') 230 >>> another_sl 231 ShareableList(['a', 'l', 'p', 'h', 'a'], name='psm_6572_12221') 232 >>> smm.shutdown() # Calls unlink() on sl, raw_shm, and another_sl 233 234The following example depicts a potentially more convenient pattern for using 235:class:`SharedMemoryManager` objects via the :keyword:`with` statement to 236ensure that all shared memory blocks are released after they are no longer 237needed: 238 239.. doctest:: 240 :options: +SKIP 241 242 >>> with SharedMemoryManager() as smm: 243 ... sl = smm.ShareableList(range(2000)) 244 ... # Divide the work among two processes, storing partial results in sl 245 ... p1 = Process(target=do_work, args=(sl, 0, 1000)) 246 ... p2 = Process(target=do_work, args=(sl, 1000, 2000)) 247 ... p1.start() 248 ... p2.start() # A multiprocessing.Pool might be more efficient 249 ... p1.join() 250 ... p2.join() # Wait for all work to complete in both processes 251 ... total_result = sum(sl) # Consolidate the partial results now in sl 252 253When using a :class:`SharedMemoryManager` in a :keyword:`with` statement, the 254shared memory blocks created using that manager are all released when the 255:keyword:`with` statement's code block finishes execution. 256 257 258.. class:: ShareableList(sequence=None, *, name=None) 259 260 Provides a mutable list-like object where all values stored within are 261 stored in a shared memory block. This constrains storable values to 262 only the ``int``, ``float``, ``bool``, ``str`` (less than 10M bytes each), 263 ``bytes`` (less than 10M bytes each), and ``None`` built-in data types. 264 It also notably differs from the built-in ``list`` type in that these 265 lists can not change their overall length (i.e. no append, insert, etc.) 266 and do not support the dynamic creation of new :class:`ShareableList` 267 instances via slicing. 268 269 *sequence* is used in populating a new ``ShareableList`` full of values. 270 Set to ``None`` to instead attach to an already existing 271 ``ShareableList`` by its unique shared memory name. 272 273 *name* is the unique name for the requested shared memory, as described 274 in the definition for :class:`SharedMemory`. When attaching to an 275 existing ``ShareableList``, specify its shared memory block's unique 276 name while leaving ``sequence`` set to ``None``. 277 278 .. method:: count(value) 279 280 Returns the number of occurrences of ``value``. 281 282 .. method:: index(value) 283 284 Returns first index position of ``value``. Raises :exc:`ValueError` if 285 ``value`` is not present. 286 287 .. attribute:: format 288 289 Read-only attribute containing the :mod:`struct` packing format used by 290 all currently stored values. 291 292 .. attribute:: shm 293 294 The :class:`SharedMemory` instance where the values are stored. 295 296 297The following example demonstrates basic use of a :class:`ShareableList` 298instance: 299 300 >>> from multiprocessing import shared_memory 301 >>> a = shared_memory.ShareableList(['howdy', b'HoWdY', -273.154, 100, None, True, 42]) 302 >>> [ type(entry) for entry in a ] 303 [<class 'str'>, <class 'bytes'>, <class 'float'>, <class 'int'>, <class 'NoneType'>, <class 'bool'>, <class 'int'>] 304 >>> a[2] 305 -273.154 306 >>> a[2] = -78.5 307 >>> a[2] 308 -78.5 309 >>> a[2] = 'dry ice' # Changing data types is supported as well 310 >>> a[2] 311 'dry ice' 312 >>> a[2] = 'larger than previously allocated storage space' 313 Traceback (most recent call last): 314 ... 315 ValueError: exceeds available storage for existing str 316 >>> a[2] 317 'dry ice' 318 >>> len(a) 319 7 320 >>> a.index(42) 321 6 322 >>> a.count(b'howdy') 323 0 324 >>> a.count(b'HoWdY') 325 1 326 >>> a.shm.close() 327 >>> a.shm.unlink() 328 >>> del a # Use of a ShareableList after call to unlink() is unsupported 329 330The following example depicts how one, two, or many processes may access the 331same :class:`ShareableList` by supplying the name of the shared memory block 332behind it: 333 334 >>> b = shared_memory.ShareableList(range(5)) # In a first process 335 >>> c = shared_memory.ShareableList(name=b.shm.name) # In a second process 336 >>> c 337 ShareableList([0, 1, 2, 3, 4], name='...') 338 >>> c[-1] = -999 339 >>> b[-1] 340 -999 341 >>> b.shm.close() 342 >>> c.shm.close() 343 >>> c.shm.unlink() 344 345The following examples demonstrates that ``ShareableList`` 346(and underlying ``SharedMemory``) objects 347can be pickled and unpickled if needed. 348Note, that it will still be the same shared object. 349This happens, because the deserialized object has 350the same unique name and is just attached to an existing 351object with the same name (if the object is still alive): 352 353 >>> import pickle 354 >>> from multiprocessing import shared_memory 355 >>> sl = shared_memory.ShareableList(range(10)) 356 >>> list(sl) 357 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 358 359 >>> deserialized_sl = pickle.loads(pickle.dumps(sl)) 360 >>> list(deserialized_sl) 361 [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] 362 363 >>> sl[0] = -1 364 >>> deserialized_sl[1] = -2 365 >>> list(sl) 366 [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] 367 >>> list(deserialized_sl) 368 [-1, -2, 2, 3, 4, 5, 6, 7, 8, 9] 369 370 >>> sl.shm.close() 371 >>> sl.shm.unlink() 372