1.. currentmodule:: asyncio 2 3.. _asyncio-subprocess: 4 5============ 6Subprocesses 7============ 8 9**Source code:** :source:`Lib/asyncio/subprocess.py`, 10:source:`Lib/asyncio/base_subprocess.py` 11 12---------------------------------------- 13 14This section describes high-level async/await asyncio APIs to 15create and manage subprocesses. 16 17.. _asyncio_example_subprocess_shell: 18 19Here's an example of how asyncio can run a shell command and 20obtain its result:: 21 22 import asyncio 23 24 async def run(cmd): 25 proc = await asyncio.create_subprocess_shell( 26 cmd, 27 stdout=asyncio.subprocess.PIPE, 28 stderr=asyncio.subprocess.PIPE) 29 30 stdout, stderr = await proc.communicate() 31 32 print(f'[{cmd!r} exited with {proc.returncode}]') 33 if stdout: 34 print(f'[stdout]\n{stdout.decode()}') 35 if stderr: 36 print(f'[stderr]\n{stderr.decode()}') 37 38 asyncio.run(run('ls /zzz')) 39 40will print:: 41 42 ['ls /zzz' exited with 1] 43 [stderr] 44 ls: /zzz: No such file or directory 45 46Because all asyncio subprocess functions are asynchronous and asyncio 47provides many tools to work with such functions, it is easy to execute 48and monitor multiple subprocesses in parallel. It is indeed trivial 49to modify the above example to run several commands simultaneously:: 50 51 async def main(): 52 await asyncio.gather( 53 run('ls /zzz'), 54 run('sleep 1; echo "hello"')) 55 56 asyncio.run(main()) 57 58See also the `Examples`_ subsection. 59 60 61Creating Subprocesses 62===================== 63 64.. coroutinefunction:: create_subprocess_exec(program, *args, stdin=None, \ 65 stdout=None, stderr=None, limit=None, **kwds) 66 67 Create a subprocess. 68 69 The *limit* argument sets the buffer limit for :class:`StreamReader` 70 wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` 71 (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). 72 73 Return a :class:`~asyncio.subprocess.Process` instance. 74 75 See the documentation of :meth:`loop.subprocess_exec` for other 76 parameters. 77 78 .. deprecated-removed:: 3.8 3.10 79 80 The ``loop`` parameter. This function has been implicitly getting the 81 current running loop since 3.7. See 82 :ref:`What's New in 3.10's Removed section <whatsnew310-removed>` 83 for more information. 84 85 86.. coroutinefunction:: create_subprocess_shell(cmd, stdin=None, \ 87 stdout=None, stderr=None, limit=None, **kwds) 88 89 Run the *cmd* shell command. 90 91 The *limit* argument sets the buffer limit for :class:`StreamReader` 92 wrappers for :attr:`Process.stdout` and :attr:`Process.stderr` 93 (if :attr:`subprocess.PIPE` is passed to *stdout* and *stderr* arguments). 94 95 Return a :class:`~asyncio.subprocess.Process` instance. 96 97 See the documentation of :meth:`loop.subprocess_shell` for other 98 parameters. 99 100 .. important:: 101 102 It is the application's responsibility to ensure that all whitespace and 103 special characters are quoted appropriately to avoid `shell injection 104 <https://en.wikipedia.org/wiki/Shell_injection#Shell_injection>`_ 105 vulnerabilities. The :func:`shlex.quote` function can be used to properly 106 escape whitespace and special shell characters in strings that are going 107 to be used to construct shell commands. 108 109 .. deprecated-removed:: 3.8 3.10 110 111 The ``loop`` parameter. This function has been implicitly getting the 112 current running loop since 3.7. See 113 :ref:`What's New in 3.10's Removed section <whatsnew310-removed>` 114 for more information. 115 116.. note:: 117 118 Subprocesses are available for Windows if a :class:`ProactorEventLoop` is 119 used. See :ref:`Subprocess Support on Windows <asyncio-windows-subprocess>` 120 for details. 121 122.. seealso:: 123 124 asyncio also has the following *low-level* APIs to work with subprocesses: 125 :meth:`loop.subprocess_exec`, :meth:`loop.subprocess_shell`, 126 :meth:`loop.connect_read_pipe`, :meth:`loop.connect_write_pipe`, 127 as well as the :ref:`Subprocess Transports <asyncio-subprocess-transports>` 128 and :ref:`Subprocess Protocols <asyncio-subprocess-protocols>`. 129 130 131Constants 132========= 133 134.. data:: asyncio.subprocess.PIPE 135 136 Can be passed to the *stdin*, *stdout* or *stderr* parameters. 137 138 If *PIPE* is passed to *stdin* argument, the 139 :attr:`Process.stdin <asyncio.subprocess.Process.stdin>` attribute 140 will point to a :class:`StreamWriter` instance. 141 142 If *PIPE* is passed to *stdout* or *stderr* arguments, the 143 :attr:`Process.stdout <asyncio.subprocess.Process.stdout>` and 144 :attr:`Process.stderr <asyncio.subprocess.Process.stderr>` 145 attributes will point to :class:`StreamReader` instances. 146 147.. data:: asyncio.subprocess.STDOUT 148 149 Special value that can be used as the *stderr* argument and indicates 150 that standard error should be redirected into standard output. 151 152.. data:: asyncio.subprocess.DEVNULL 153 154 Special value that can be used as the *stdin*, *stdout* or *stderr* argument 155 to process creation functions. It indicates that the special file 156 :data:`os.devnull` will be used for the corresponding subprocess stream. 157 158 159Interacting with Subprocesses 160============================= 161 162Both :func:`create_subprocess_exec` and :func:`create_subprocess_shell` 163functions return instances of the *Process* class. *Process* is a high-level 164wrapper that allows communicating with subprocesses and watching for 165their completion. 166 167.. class:: asyncio.subprocess.Process 168 169 An object that wraps OS processes created by the 170 :func:`create_subprocess_exec` and :func:`create_subprocess_shell` 171 functions. 172 173 This class is designed to have a similar API to the 174 :class:`subprocess.Popen` class, but there are some 175 notable differences: 176 177 * unlike Popen, Process instances do not have an equivalent to 178 the :meth:`~subprocess.Popen.poll` method; 179 180 * the :meth:`~asyncio.subprocess.Process.communicate` and 181 :meth:`~asyncio.subprocess.Process.wait` methods don't have a 182 *timeout* parameter: use the :func:`wait_for` function; 183 184 * the :meth:`Process.wait() <asyncio.subprocess.Process.wait>` method 185 is asynchronous, whereas :meth:`subprocess.Popen.wait` method 186 is implemented as a blocking busy loop; 187 188 * the *universal_newlines* parameter is not supported. 189 190 This class is :ref:`not thread safe <asyncio-multithreading>`. 191 192 See also the :ref:`Subprocess and Threads <asyncio-subprocess-threads>` 193 section. 194 195 .. coroutinemethod:: wait() 196 197 Wait for the child process to terminate. 198 199 Set and return the :attr:`returncode` attribute. 200 201 .. note:: 202 203 This method can deadlock when using ``stdout=PIPE`` or 204 ``stderr=PIPE`` and the child process generates so much output 205 that it blocks waiting for the OS pipe buffer to accept 206 more data. Use the :meth:`communicate` method when using pipes 207 to avoid this condition. 208 209 .. coroutinemethod:: communicate(input=None) 210 211 Interact with process: 212 213 1. send data to *stdin* (if *input* is not ``None``); 214 2. read data from *stdout* and *stderr*, until EOF is reached; 215 3. wait for process to terminate. 216 217 The optional *input* argument is the data (:class:`bytes` object) 218 that will be sent to the child process. 219 220 Return a tuple ``(stdout_data, stderr_data)``. 221 222 If either :exc:`BrokenPipeError` or :exc:`ConnectionResetError` 223 exception is raised when writing *input* into *stdin*, the 224 exception is ignored. This condition occurs when the process 225 exits before all data are written into *stdin*. 226 227 If it is desired to send data to the process' *stdin*, 228 the process needs to be created with ``stdin=PIPE``. Similarly, 229 to get anything other than ``None`` in the result tuple, the 230 process has to be created with ``stdout=PIPE`` and/or 231 ``stderr=PIPE`` arguments. 232 233 Note, that the data read is buffered in memory, so do not use 234 this method if the data size is large or unlimited. 235 236 .. method:: send_signal(signal) 237 238 Sends the signal *signal* to the child process. 239 240 .. note:: 241 242 On Windows, :py:data:`SIGTERM` is an alias for :meth:`terminate`. 243 ``CTRL_C_EVENT`` and ``CTRL_BREAK_EVENT`` can be sent to processes 244 started with a *creationflags* parameter which includes 245 ``CREATE_NEW_PROCESS_GROUP``. 246 247 .. method:: terminate() 248 249 Stop the child process. 250 251 On POSIX systems this method sends :py:data:`signal.SIGTERM` to the 252 child process. 253 254 On Windows the Win32 API function :c:func:`TerminateProcess` is 255 called to stop the child process. 256 257 .. method:: kill() 258 259 Kill the child process. 260 261 On POSIX systems this method sends :py:data:`SIGKILL` to the child 262 process. 263 264 On Windows this method is an alias for :meth:`terminate`. 265 266 .. attribute:: stdin 267 268 Standard input stream (:class:`StreamWriter`) or ``None`` 269 if the process was created with ``stdin=None``. 270 271 .. attribute:: stdout 272 273 Standard output stream (:class:`StreamReader`) or ``None`` 274 if the process was created with ``stdout=None``. 275 276 .. attribute:: stderr 277 278 Standard error stream (:class:`StreamReader`) or ``None`` 279 if the process was created with ``stderr=None``. 280 281 .. warning:: 282 283 Use the :meth:`communicate` method rather than 284 :attr:`process.stdin.write() <stdin>`, 285 :attr:`await process.stdout.read() <stdout>` or 286 :attr:`await process.stderr.read <stderr>`. 287 This avoids deadlocks due to streams pausing reading or writing 288 and blocking the child process. 289 290 .. attribute:: pid 291 292 Process identification number (PID). 293 294 Note that for processes created by the :func:`create_subprocess_shell` 295 function, this attribute is the PID of the spawned shell. 296 297 .. attribute:: returncode 298 299 Return code of the process when it exits. 300 301 A ``None`` value indicates that the process has not terminated yet. 302 303 A negative value ``-N`` indicates that the child was terminated 304 by signal ``N`` (POSIX only). 305 306 307.. _asyncio-subprocess-threads: 308 309Subprocess and Threads 310---------------------- 311 312Standard asyncio event loop supports running subprocesses from different threads by 313default. 314 315On Windows subprocesses are provided by :class:`ProactorEventLoop` only (default), 316:class:`SelectorEventLoop` has no subprocess support. 317 318On UNIX *child watchers* are used for subprocess finish waiting, see 319:ref:`asyncio-watchers` for more info. 320 321 322.. versionchanged:: 3.8 323 324 UNIX switched to use :class:`ThreadedChildWatcher` for spawning subprocesses from 325 different threads without any limitation. 326 327 Spawning a subprocess with *inactive* current child watcher raises 328 :exc:`RuntimeError`. 329 330Note that alternative event loop implementations might have own limitations; 331please refer to their documentation. 332 333.. seealso:: 334 335 The :ref:`Concurrency and multithreading in asyncio 336 <asyncio-multithreading>` section. 337 338 339Examples 340-------- 341 342An example using the :class:`~asyncio.subprocess.Process` class to 343control a subprocess and the :class:`StreamReader` class to read from 344its standard output. 345 346.. _asyncio_example_create_subprocess_exec: 347 348The subprocess is created by the :func:`create_subprocess_exec` 349function:: 350 351 import asyncio 352 import sys 353 354 async def get_date(): 355 code = 'import datetime; print(datetime.datetime.now())' 356 357 # Create the subprocess; redirect the standard output 358 # into a pipe. 359 proc = await asyncio.create_subprocess_exec( 360 sys.executable, '-c', code, 361 stdout=asyncio.subprocess.PIPE) 362 363 # Read one line of output. 364 data = await proc.stdout.readline() 365 line = data.decode('ascii').rstrip() 366 367 # Wait for the subprocess exit. 368 await proc.wait() 369 return line 370 371 date = asyncio.run(get_date()) 372 print(f"Current date: {date}") 373 374 375See also the :ref:`same example <asyncio_example_subprocess_proto>` 376written using low-level APIs. 377