Lines Matching +full:in +full:- +full:process
1 :mod:`multiprocessing` --- Process-based parallelism
5 :synopsis: Process-based parallelism.
9 --------------
11 .. include:: ../includes/wasm-notavail.rst
14 ------------
18 offers both local and remote concurrency, effectively side-stepping the
26 analogs in the :mod:`threading` module. A prime example of this is the
30 example demonstrates the common practice of defining such functions in a module
51 to push tasks to a background process without blocking execution of the
52 calling process. Compared to using the :class:`~multiprocessing.pool.Pool`
54 the submission of work to the underlying process pool to be separated from
58 The :class:`Process` class
61 In :mod:`multiprocessing`, processes are spawned by creating a :class:`Process`
62 object and then calling its :meth:`~Process.start` method. :class:`Process`
66 from multiprocessing import Process
72 p = Process(target=f, args=('bob',))
76 To show the individual process IDs involved, here is an expanded example::
78 from multiprocessing import Process
84 print('parent process:', os.getppid())
85 print('process id:', os.getpid())
93 p = Process(target=f, args=('bob',))
98 necessary, see :ref:`multiprocessing-programming`.
105 .. _multiprocessing-start-methods:
108 to start a process. These *start methods* are
111 The parent process starts a fresh Python interpreter process. The
112 child process will only inherit those resources necessary to run
113 the process object's :meth:`~Process.run` method. In particular,
114 unnecessary file descriptors and handles from the parent process
115 will not be inherited. Starting a process using this method is
121 The parent process uses :func:`os.fork` to fork the Python
122 interpreter. The child process, when it begins, is effectively
123 identical to the parent process. All resources of the parent are
124 inherited by the child process. Note that safely forking a
125 multithreaded process is problematic.
131 a server process is started. From then on, whenever a new process
132 is needed, the parent process connects to the server and requests
133 that it fork a new process. The fork server process is single
153 start a *resource tracker* process which tracks the unlinked named
158 Usually there should be none, but if a process was killed by a signal
162 named semaphores, and shared memory segments occupy some space in the main
165 To select a start method you use the :func:`set_start_method` in
177 p = mp.Process(target=foo, args=(q,))
182 :func:`set_start_method` should not be used more than once in the
187 module, and allow one to use multiple start methods in the same
198 p = ctx.Process(target=foo, args=(q,))
204 processes for a different context. In particular, locks created using
231 from multiprocessing import Process, Queue
238 p = Process(target=f, args=(q,))
243 Queues are thread and process safe.
248 pipe which by default is duplex (two-way). For example::
250 from multiprocessing import Process, Pipe
258 p = Process(target=f, args=(child_conn,))
265 :meth:`~Connection.recv` methods (among others). Note that data in a pipe
277 that only one process prints to standard output at a time::
279 from multiprocessing import Process, Lock
291 for num in range(10):
292 Process(target=f, args=(lock, num)).start()
310 Data can be stored in a shared memory map using :class:`Value` or
313 from multiprocessing import Process, Value, Array
317 for i in range(len(a)):
318 a[i] = -a[i]
324 p = Process(target=f, args=(num, arr))
334 [0, -1, -2, -3, -4, -5, -6, -7, -8, -9]
339 objects will be process and thread-safe.
341 For more flexibility in using shared memory one can use the
345 **Server process**
347 A manager object returned by :func:`Manager` controls a server process which
357 from multiprocessing import Process, Manager
370 p = Process(target=f, args=(d, l))
382 Server process managers are more flexible than using shared memory objects
393 processes in a few different ways.
411 # print same numbers in arbitrary order
412 for i in pool.imap_unordered(f, range(10)):
416 res = pool.apply_async(f, (20,)) # runs in *only* one process
420 res = pool.apply_async(os.getpid, ()) # runs in *only* one process
421 print(res.get(timeout=1)) # prints the PID of that process
424 multiple_results = [pool.apply_async(os.getpid, ()) for i in range(4)]
425 print([res.get(timeout=1) for res in multiple_results])
436 # exiting the 'with'-block has stopped the pool
440 process which created it.
445 importable by the children. This is covered in :ref:`multiprocessing-programming`
447 as the :class:`multiprocessing.pool.Pool` examples will not work in the
457 Process PoolWorker-1:
458 Process PoolWorker-2:
459 Process PoolWorker-3:
468 interleaved in a semi-random fashion, and then you may have to
469 stop the parent process somehow.)
473 ---------
479 :class:`Process` and exceptions
482 .. class:: Process(group=None, target=None, name=None, args=(), kwargs={}, \
485 Process objects represent activity that is run in a separate process. The
486 :class:`Process` class has equivalents of all the methods of
493 called. *name* is the process name (see :attr:`name` for more details).
496 the keyword-only *daemon* argument sets the process :attr:`daemon` flag
498 inherited from the creating process.
505 base class constructor (:meth:`Process.__init__`) before doing anything else
506 to the process.
513 Method representing the process's activity.
515 You may override this method in a subclass. The standard :meth:`run`
520 Using a list or tuple as the *args* argument passed to :class:`Process`
525 >>> from multiprocessing import Process
526 >>> p = Process(target=print, args=[1])
529 >>> p = Process(target=print, args=(1,))
535 Start the process's activity.
537 This must be called at most once per process object. It arranges for the
538 object's :meth:`run` method to be invoked in a separate process.
543 blocks until the process whose :meth:`join` method is called terminates.
545 Note that the method returns ``None`` if its process terminates or if the
546 method times out. Check the process's :attr:`exitcode` to determine if
549 A process can be joined many times.
551 A process cannot join itself because this would cause a deadlock. It is
552 an error to attempt to join a process before it has been started.
556 The process's name. The name is a string used for identification purposes
562 'Process-N\ :sub:`1`:N\ :sub:`2`:...:N\ :sub:`k`' is constructed, where
563 each N\ :sub:`k` is the N-th child of its parent.
567 Return whether the process is alive.
569 Roughly, a process object is alive from the moment the :meth:`start`
570 method returns until the child process terminates.
574 The process's daemon flag, a Boolean value. This must be set before
577 The initial value is inherited from the creating process.
579 When a process exits, it attempts to terminate all of its daemonic child
582 Note that a daemonic process is not allowed to create child processes.
583 Otherwise a daemonic process would leave its children orphaned if it gets
584 terminated when its parent process exits. Additionally, these are **not**
586 terminated (and not joined) if non-daemonic processes have exited.
588 In addition to the :class:`threading.Thread` API, :class:`Process` objects
593 Return the process ID. Before the process is spawned, this will be
598 The child's exit code. This will be ``None`` if the process has not yet
607 signal *N*, the exit code will be the negative value *-N*.
611 The process's authentication key (a byte string).
613 When :mod:`multiprocessing` is initialized the main process is assigned a
616 When a :class:`Process` object is created, it will inherit the
617 authentication key of its parent process, although this may be changed by
620 See :ref:`multiprocessing-auth-keys`.
625 the process ends.
639 Terminate the process. On Unix this is done using the ``SIGTERM`` signal;
643 Note that descendant processes of the process will *not* be terminated --
648 If this method is used when the associated process is using a pipe or
650 become unusable by other process. Similarly, if the process has
662 Close the :class:`Process` object, releasing all resources associated
663 with it. :exc:`ValueError` is raised if the underlying process
665 other methods and attributes of the :class:`Process` object will
672 the process that created the process object.
674 Example usage of some of the methods of :class:`Process`:
679 >>> p = multiprocessing.Process(target=time.sleep, args=(1000,))
681 <Process ... initial> False
684 <Process ... started> True
688 <Process ... stopped exitcode=-SIGTERM> False
689 >>> p.exitcode == -signal.SIGTERM
723 are multi-producer, multi-consumer :abbr:`FIFO (first-in, first-out)`
724 queues modelled on the :class:`queue.Queue` class in the
725 standard library. They differ in that :class:`Queue` lacks the
734 Note that one can also create a shared queue by using a manager object -- see
735 :ref:`multiprocessing-managers`.
740 :exc:`queue.Full` exceptions to signal a timeout. They are not available in
749 but should not cause any practical difficulties -- if they really
751 :ref:`manager <multiprocessing-managers>`.
759 the objects to be received at the other end out-of-order.
760 However, objects enqueued by the same process will always be in
765 If a process is killed using :meth:`Process.terminate` or :func:`os.kill`
766 while it is trying to use a :class:`Queue`, then the data in the queue is
767 likely to become corrupted. This may cause any other process to get an
772 As mentioned above, if a child process has put items on a queue (and it has
774 <multiprocessing.Queue.cancel_join_thread>`), then that process will
777 This means that if you try joining that process you may get a deadlock unless
779 consumed. Similarly, if the child process is non-daemonic then the parent
780 process may hang on exit when it tries to join all its non-daemonic children.
783 :ref:`multiprocessing-programming`.
786 :ref:`multiprocessing-examples`.
803 Returns a process shared queue implemented using a pipe and a few
804 locks/semaphores. When a process first puts an item on the queue a feeder
840 ignored in that case).
858 :exc:`queue.Empty` exception (*timeout* is ignored in that case).
868 :class:`multiprocessing.Queue` has a few additional methods not found in
875 process. The background thread will quit once it has flushed all buffered
883 all data in the buffer has been flushed to the pipe.
885 By default if a process is not the creator of the queue then on exit it
886 will attempt to join the queue's background thread. The process can call
891 Prevent :meth:`join_thread` from blocking. In particular, this prevents
892 the background thread from being joined automatically when the process
893 exits -- see :meth:`join_thread`.
898 It is really only there if you need the current process to exit
906 functionality in this class will be disabled, and attempts to
907 instantiate a :class:`Queue` will result in an :exc:`ImportError`. See
955 placed in the queue.
960 Block until all items in the queue have been gotten and processed.
974 Return list of all live children of the current process.
981 Return the number of CPUs in the system.
983 This number is not equivalent to the number of CPUs the current process can
995 Return the :class:`Process` object corresponding to the current process.
1001 Return the :class:`Process` object corresponding to the parent process of
1002 the :func:`current_process`. For the main process, ``parent_process`` will
1016 from multiprocessing import Process, freeze_support
1023 Process(target=f).start()
1029 system other than Windows. In addition, if the module is being run
1078 Set the path of the Python interpreter to use when starting a child process.
1090 Accepts a :term:`path-like object`.
1124 :func:`Pipe <multiprocessing.Pipe>` -- see also
1125 :ref:`multiprocessing-listeners-clients`.
1159 *timeout* is a number then this specifies the maximum time in seconds to
1167 Send byte data from a :term:`bytes-like object` as a complete message.
1169 If *offset* is given then data is read from that position in *buffer*. If
1193 of the connection and return the number of bytes in the message. Blocks
1198 *buffer* must be a writable :term:`bytes-like object`. If
1200 that position. Offset must be a non-negative integer less than the
1201 length of *buffer* (in bytes).
1212 Connection objects now support the context management protocol -- see
1237 .. _multiprocessing-recv-pickle-security:
1242 receives, which can be a security risk unless you can trust the process
1248 :ref:`multiprocessing-auth-keys`.
1252 If a process is killed while it is trying to read or write to a pipe then
1253 the data in the pipe is likely to become corrupted, because it may become
1262 Generally synchronization primitives are not as necessary in a multiprocess
1263 program as they are in a multithreaded program. See the documentation for
1267 object -- see :ref:`multiprocessing-managers`.
1304 A non-recursive lock object: a close analog of :class:`threading.Lock`.
1305 Once a process or thread has acquired a lock, subsequent attempts to
1306 acquire it from any process or thread will block until it is released;
1307 any process or thread may release it. The concepts and behaviors of
1308 :class:`threading.Lock` as it applies to threads are replicated here in
1317 used in :keyword:`with` statements.
1321 Acquire a lock, blocking or non-blocking.
1324 will block until the lock is in an unlocked state, then set it to locked
1326 from that in :meth:`threading.Lock.acquire`.
1329 block. If the lock is currently in a locked state, return ``False``;
1332 When invoked with a positive, floating-point value for *timeout*, block
1338 *timeout* differs from the implemented behavior in
1347 Release a lock. This can be called from any process or thread, not only
1348 the process or thread which originally acquired the lock.
1350 Behavior is the same as in :meth:`threading.Lock.release` except that
1357 recursive lock must be released by the process or thread that acquired it.
1358 Once a process or thread has acquired a recursive lock, the same process
1359 or thread may acquire it again without blocking; that process or thread
1367 used in :keyword:`with` statements.
1372 Acquire a lock, blocking or non-blocking.
1375 lock is in an unlocked state (not owned by any process or thread) unless
1376 the lock is already owned by the current process or thread. The current
1377 process or thread then takes ownership of the lock (if it does not
1379 by one, resulting in a return value of ``True``. Note that there are
1380 several differences in this first argument's behavior compared to the
1386 process or thread, the current process or thread does not take ownership
1387 and the recursion level within the lock is not changed, resulting in
1388 a return value of ``False``. If the lock is in an unlocked state, the
1389 current process or thread takes ownership and the recursion level is
1390 incremented, resulting in a return value of ``True``.
1392 Use and behaviors of the *timeout* argument are the same as in
1394 differ from the implemented behaviors in :meth:`threading.RLock.acquire`.
1401 owned by any process or thread) and if any other processes or threads
1404 nonzero, the lock remains locked and owned by the calling process or
1407 Only call this method when the calling process or thread owns the lock.
1408 An :exc:`AssertionError` is raised if this method is called by a process
1409 or thread other than the owner or if the lock is in an unlocked (unowned)
1410 state. Note that the type of exception raised in this situation
1411 differs from the implemented behavior in :meth:`threading.RLock.release`.
1428 If the SIGINT signal generated by :kbd:`Ctrl-C` arrives while the main thread is
1435 ignored while the equivalent blocking calls are in progress.
1442 import it will result in an :exc:`ImportError`. See
1467 by a lock, so it will not necessarily be "process-safe".
1481 Note that *lock* is a keyword-only argument.
1500 "process-safe".
1520 Although it is possible to store a pointer in shared memory remember that
1521 this will refer to a location in the address space of a specific process.
1522 However, the pointer is quite likely to be invalid in the context of a second
1523 process and trying to dereference the pointer from the second process may
1537 Note that setting and getting an element is potentially non-atomic -- use
1549 Note that setting and getting the value is potentially non-atomic -- use
1554 attributes which allow one to use it to store and retrieve strings -- see
1560 process-safe synchronization wrapper may be returned instead of a raw ctypes
1569 "process-safe".
1571 Note that *lock* is a keyword-only argument.
1576 process-safe synchronization wrapper may be returned instead of a raw ctypes
1584 "process-safe".
1586 Note that *lock* is a keyword-only argument.
1595 Return a process-safe wrapper object for a ctypes object which uses *lock* to
1599 A synchronized wrapper will have two methods in addition to those of the
1611 shared memory with the normal ctypes syntax. (In the table ``MyStruct`` is some
1625 process::
1627 from multiprocessing import Process, Lock
1638 for a in A:
1648 A = Array(Point, [(1.875,-6.25), (-5.75,2.0), (2.375,9.5)], lock=lock)
1650 p = Process(target=modify, args=(n, x, s, A))
1657 print([(a.x, a.y) for a in A])
1672 .. _multiprocessing-managers:
1679 different machines. A manager object controls a server process which manages
1688 object corresponds to a spawned child process and has methods which will
1692 :synopsis: Share data between process with shared objects.
1695 their parent process exits. The manager classes are defined in the
1703 that the manager object refers to a started manager process.
1705 *address* is the address on which the manager process listens for new
1709 validity of incoming connections to the server process. If
1719 *shutdown_timeout* is a timeout in seconds used to wait until the process
1720 used by the manager completes in the :meth:`shutdown` method. If the
1721 shutdown times out, the process is terminated. If terminating the process
1722 also times out, the process is killed.
1747 Connect a local manager object to a remote manager process::
1755 Stop the process used by the manager. This is only available if
1756 :meth:`start` has been used to start the server process.
1781 :attr:`proxytype._exposed_` is used instead if it exists.) In the case
1795 *typeid* which can be used to tell the server process to create a new
1798 :class:`BaseManager` instances also have one read-only property:
1805 Manager objects support the context management protocol -- see
1807 server process (if it has not already started) and then returns the
1810 In previous versions :meth:`~contextmanager.__enter__` did not start the
1811 manager's server process if it was not already started.
1819 Its methods create and return :ref:`multiprocessing-proxy_objects` for a
1987 >>> from multiprocessing import Process, Queue
1989 >>> class Worker(Process):
2006 .. _multiprocessing-proxy_objects:
2012 in a different process. The shared object is said to be the *referent* of the
2017 the proxy). In this way, a proxy can be used just like its referent can:
2023 >>> l = manager.list([i*i for i in range(10)])
2039 :ref:`multiprocessing-proxy_objects`. This permits nesting of these managed
2040 lists, dicts, and other :ref:`multiprocessing-proxy_objects`:
2055 >>> l_outer = manager.list([ manager.dict() for i in range(2) ])
2066 If standard (non-proxy) :class:`list` or :class:`dict` objects are contained
2067 in a referent, modifications to those mutable values will not be propagated
2069 contained within are modified. However, storing a value in a container proxy
2071 the manager and so to effectively modify such an item, one could re-assign the
2086 :ref:`multiprocessing-proxy_objects` for most use cases but also
2091 The proxy types in :mod:`multiprocessing` do nothing to support comparisons
2117 in the manager's process.
2120 a new shared object -- see documentation for the *method_to_typeid*
2123 If an exception is raised by the call, then is re-raised by
2124 :meth:`_callmethod`. If some other exception is raised in the manager's
2125 process then this is converted into a :exc:`RemoteError` exception and is
2128 Note in particular that an exception will be raised if *methodname* has
2166 A shared object gets deleted from the manager process when there are no longer
2170 Process Pools
2181 A process pool object which controls a pool of worker processes to which jobs
2188 If *initializer* is not ``None`` then each worker process will call
2191 *maxtasksperchild* is the number of tasks a worker process can complete
2192 before it will exit and be replaced with a fresh worker process, to enable
2199 of a context object. In both cases *context* is set
2203 the process which created the pool.
2209 can lead to the process hanging on finalization.
2224 duration of the Pool's work queue. A frequent pattern found in other
2228 process spawned to replace the old one. The *maxtasksperchild*
2235 better suited for performing work in parallel. Additionally, *func*
2236 is only executed in one of the workers of the pool.
2245 it, that is unless the call failed, in which case the *error_callback*
2257 A parallel equivalent of the :func:`map` built-in function (it supports only
2262 the process pool as separate tasks. The (approximate) size of these
2276 it, that is unless the call failed, in which case the *error_callback*
2304 only one worker process is the order guaranteed to be "correct".)
2312 Hence an *iterable* of ``[(1,2), (3, 4)]`` results in ``[func(1,2),
2342 Pool objects now support the context management protocol -- see
2386 result = pool.apply_async(f, (10,)) # evaluate "f(10)" asynchronously in a single process
2400 .. _multiprocessing-listeners-clients:
2443 *address*. (See :ref:`multiprocessing-address-formats`)
2446 used as the secret key for an HMAC-based authentication challenge. No
2449 See :ref:`multiprocessing-auth-keys`.
2462 end point on Windows. If you require a connectable end-point,
2472 :ref:`multiprocessing-address-formats`. Note that if *family* is
2473 ``'AF_UNIX'`` and address is ``None`` then the socket will be created in a
2481 used as the secret key for an HMAC-based authentication challenge. No
2484 See :ref:`multiprocessing-auth-keys`.
2499 Listener objects have the following read-only properties:
2511 Listener objects now support the context management protocol -- see
2517 Wait till an object in *object_list* is ready. Returns the list of
2518 those objects in *object_list* which are ready. If *timeout* is a
2523 For both Unix and Windows, an object can appear in *object_list* if
2528 * the :attr:`~multiprocessing.Process.sentinel` attribute of a
2529 :class:`~multiprocessing.Process` object.
2540 **Windows**: An item in *object_list* must either be an integer
2592 from multiprocessing import Process, Pipe, current_process
2596 for i in range(10):
2603 for i in range(4):
2606 p = Process(target=foo, args=(w,))
2609 # p is the only process which owns a handle for it. This
2615 for r in wait(readers):
2624 .. _multiprocessing-address-formats:
2644 .. _multiprocessing-auth-keys:
2663 :class:`~multiprocessing.Process`). This value will be automatically inherited by
2664 any :class:`~multiprocessing.Process` object that the current process creates.
2665 This means that (by default) all processes of a multi-process program will share
2676 package does not use process shared locks so it is possible (depending on the
2690 parent process's logger -- any other customization of the logger will not be
2696 This function performs a call to :func:`get_logger` but in addition to
2710 [INFO/SyncManager-...] child process calling self.run()
2711 [INFO/SyncManager-...] created temp directory /.../pymp-...
2712 [INFO/SyncManager-...] manager serving at '/.../listener-...'
2715 [INFO/SyncManager-...] manager exiting with exitcode 0
2731 In particular, the ``Pool`` function provided by :mod:`multiprocessing.dummy`
2749 If *initializer* is not ``None`` then each worker process will call
2770 .. _multiprocessing-programming:
2773 ----------------------
2806 On Unix when a process finishes but has not been joined it becomes a zombie.
2807 There should never be very many because each time a new process starts (or
2810 process's :meth:`Process.is_alive <multiprocessing.Process.is_alive>` will
2811 join the process. Even so it is probably good
2820 Instead you should arrange the program so that a process which
2822 from an ancestor process.
2826 Using the :meth:`Process.terminate <multiprocessing.Process.terminate>`
2827 method to stop a process is liable to
2829 currently being used by the process to become broken or unavailable to other
2833 :meth:`Process.terminate <multiprocessing.Process.terminate>` on processes
2838 Bear in mind that a process that has put items in a queue will wait before
2840 the underlying pipe. (The child process can call the
2846 process is joined. Otherwise you cannot be sure that processes which have
2847 put items on the queue will terminate. Remember also that non-daemonic
2852 from multiprocessing import Process, Queue
2859 p = Process(target=f, args=(queue,))
2869 On Unix using the *fork* start method, a child process can make
2870 use of a shared resource created in a parent process using a
2872 argument to the constructor for the child process.
2876 child process is still alive the object will not be garbage
2877 collected in the parent process. This might be important if some
2878 resource is freed when the object is garbage collected in the
2879 parent process.
2883 from multiprocessing import Process, Lock
2890 for i in range(10):
2891 Process(target=f).start()
2895 from multiprocessing import Process, Lock
2902 for i in range(10):
2903 Process(target=f, args=(lock,)).start()
2911 in the :meth:`multiprocessing.Process._bootstrap` method --- this resulted
2912 in issues with processes-in-processes. This has been changed to::
2918 resulting in a bad file descriptor error, but introduces a potential danger
2919 to applications which replace :func:`sys.stdin` with a "file-like object"
2921 :meth:`~io.IOBase.close()` on this file-like object, it could result in the same
2922 data being flushed to the object multiple times, resulting in corruption.
2924 If you write a file-like object and implement your own caching, you can
2925 make it fork-safe by storing the pid whenever you append to the cache,
2946 Ensure that all arguments to :meth:`Process.__init__` are picklable.
2947 Also, if you subclass :class:`~multiprocessing.Process` then make sure that
2948 instances will be picklable when the :meth:`Process.start
2949 <multiprocessing.Process.start>` method is called.
2953 Bear in mind that if code run in a child process tries to access a global
2955 in the parent process at the time that :meth:`Process.start
2956 <multiprocessing.Process.start>` was called.
2961 .. _multiprocessing-safe-main-import:
2967 process).
2973 from multiprocessing import Process
2978 p = Process(target=foo)
2984 from multiprocessing import Process, freeze_support, set_start_method
2992 p = Process(target=foo)
3001 Similar restrictions apply if a pool or manager is created in the main
3005 .. _multiprocessing-examples:
3008 --------