• Home
  • Line#
  • Scopes#
  • Navigate#
  • Raw
  • Download
1.. currentmodule:: asyncio
2
3Tasks and coroutines
4====================
5
6.. _coroutine:
7
8Coroutines
9----------
10
11Coroutines used with :mod:`asyncio` may be implemented using the
12:keyword:`async def` statement, or by using :term:`generators <generator>`.
13The :keyword:`async def` type of coroutine was added in Python 3.5, and
14is recommended if there is no need to support older Python versions.
15
16Generator-based coroutines should be decorated with :func:`@asyncio.coroutine
17<asyncio.coroutine>`, although this is not strictly enforced.
18The decorator enables compatibility with :keyword:`async def` coroutines,
19and also serves as documentation.  Generator-based
20coroutines use the ``yield from`` syntax introduced in :pep:`380`,
21instead of the original ``yield`` syntax.
22
23The word "coroutine", like the word "generator", is used for two
24different (though related) concepts:
25
26- The function that defines a coroutine
27  (a function definition using :keyword:`async def` or
28  decorated with ``@asyncio.coroutine``).  If disambiguation is needed
29  we will call this a *coroutine function* (:func:`iscoroutinefunction`
30  returns ``True``).
31
32- The object obtained by calling a coroutine function.  This object
33  represents a computation or an I/O operation (usually a combination)
34  that will complete eventually.  If disambiguation is needed we will
35  call it a *coroutine object* (:func:`iscoroutine` returns ``True``).
36
37Things a coroutine can do:
38
39- ``result = await future`` or ``result = yield from future`` --
40  suspends the coroutine until the
41  future is done, then returns the future's result, or raises an
42  exception, which will be propagated.  (If the future is cancelled,
43  it will raise a ``CancelledError`` exception.)  Note that tasks are
44  futures, and everything said about futures also applies to tasks.
45
46- ``result = await coroutine`` or ``result = yield from coroutine`` --
47  wait for another coroutine to
48  produce a result (or raise an exception, which will be propagated).
49  The ``coroutine`` expression must be a *call* to another coroutine.
50
51- ``return expression`` -- produce a result to the coroutine that is
52  waiting for this one using :keyword:`await` or ``yield from``.
53
54- ``raise exception`` -- raise an exception in the coroutine that is
55  waiting for this one using :keyword:`await` or ``yield from``.
56
57Calling a coroutine does not start its code running --
58the coroutine object returned by the call doesn't do anything until you
59schedule its execution.  There are two basic ways to start it running:
60call ``await coroutine`` or ``yield from coroutine`` from another coroutine
61(assuming the other coroutine is already running!), or schedule its execution
62using the :func:`ensure_future` function or the :meth:`AbstractEventLoop.create_task`
63method.
64
65
66Coroutines (and tasks) can only run when the event loop is running.
67
68.. decorator:: coroutine
69
70    Decorator to mark generator-based coroutines.  This enables
71    the generator use :keyword:`!yield from` to call :keyword:`async
72    def` coroutines, and also enables the generator to be called by
73    :keyword:`async def` coroutines, for instance using an
74    :keyword:`await` expression.
75
76    There is no need to decorate :keyword:`async def` coroutines themselves.
77
78    If the generator is not yielded from before it is destroyed, an error
79    message is logged. See :ref:`Detect coroutines never scheduled
80    <asyncio-coroutine-not-scheduled>`.
81
82.. note::
83
84    In this documentation, some methods are documented as coroutines,
85    even if they are plain Python functions returning a :class:`Future`.
86    This is intentional to have a freedom of tweaking the implementation
87    of these functions in the future. If such a function is needed to be
88    used in a callback-style code, wrap its result with :func:`ensure_future`.
89
90
91.. _asyncio-hello-world-coroutine:
92
93Example: Hello World coroutine
94^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
95
96Example of coroutine displaying ``"Hello World"``::
97
98    import asyncio
99
100    async def hello_world():
101        print("Hello World!")
102
103    loop = asyncio.get_event_loop()
104    # Blocking call which returns when the hello_world() coroutine is done
105    loop.run_until_complete(hello_world())
106    loop.close()
107
108.. seealso::
109
110   The :ref:`Hello World with call_soon() <asyncio-hello-world-callback>`
111   example uses the :meth:`AbstractEventLoop.call_soon` method to schedule a
112   callback.
113
114
115.. _asyncio-date-coroutine:
116
117Example: Coroutine displaying the current date
118^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
119
120Example of coroutine displaying the current date every second during 5 seconds
121using the :meth:`sleep` function::
122
123    import asyncio
124    import datetime
125
126    async def display_date(loop):
127        end_time = loop.time() + 5.0
128        while True:
129            print(datetime.datetime.now())
130            if (loop.time() + 1.0) >= end_time:
131                break
132            await asyncio.sleep(1)
133
134    loop = asyncio.get_event_loop()
135    # Blocking call which returns when the display_date() coroutine is done
136    loop.run_until_complete(display_date(loop))
137    loop.close()
138
139.. seealso::
140
141   The :ref:`display the current date with call_later()
142   <asyncio-date-callback>` example uses a callback with the
143   :meth:`AbstractEventLoop.call_later` method.
144
145
146Example: Chain coroutines
147^^^^^^^^^^^^^^^^^^^^^^^^^
148
149Example chaining coroutines::
150
151    import asyncio
152
153    async def compute(x, y):
154        print("Compute %s + %s ..." % (x, y))
155        await asyncio.sleep(1.0)
156        return x + y
157
158    async def print_sum(x, y):
159        result = await compute(x, y)
160        print("%s + %s = %s" % (x, y, result))
161
162    loop = asyncio.get_event_loop()
163    loop.run_until_complete(print_sum(1, 2))
164    loop.close()
165
166``compute()`` is chained to ``print_sum()``: ``print_sum()`` coroutine waits
167until ``compute()`` is completed before returning its result.
168
169Sequence diagram of the example:
170
171.. image:: tulip_coro.png
172   :align: center
173
174The "Task" is created by the :meth:`AbstractEventLoop.run_until_complete` method
175when it gets a coroutine object instead of a task.
176
177The diagram shows the control flow, it does not describe exactly how things
178work internally. For example, the sleep coroutine creates an internal future
179which uses :meth:`AbstractEventLoop.call_later` to wake up the task in 1 second.
180
181
182InvalidStateError
183-----------------
184
185.. exception:: InvalidStateError
186
187   The operation is not allowed in this state.
188
189
190TimeoutError
191------------
192
193.. exception:: TimeoutError
194
195   The operation exceeded the given deadline.
196
197.. note::
198
199   This exception is different from the builtin :exc:`TimeoutError` exception!
200
201
202Future
203------
204
205.. class:: Future(\*, loop=None)
206
207   This class is *almost* compatible with :class:`concurrent.futures.Future`.
208
209   Differences:
210
211   - :meth:`result` and :meth:`exception` do not take a timeout argument and
212     raise an exception when the future isn't done yet.
213
214   - Callbacks registered with :meth:`add_done_callback` are always called
215     via the event loop's :meth:`~AbstractEventLoop.call_soon_threadsafe`.
216
217   - This class is not compatible with the :func:`~concurrent.futures.wait` and
218     :func:`~concurrent.futures.as_completed` functions in the
219     :mod:`concurrent.futures` package.
220
221   This class is :ref:`not thread safe <asyncio-multithreading>`.
222
223   .. method:: cancel()
224
225      Cancel the future and schedule callbacks.
226
227      If the future is already done or cancelled, return ``False``. Otherwise,
228      change the future's state to cancelled, schedule the callbacks and return
229      ``True``.
230
231   .. method:: cancelled()
232
233      Return ``True`` if the future was cancelled.
234
235   .. method:: done()
236
237      Return ``True`` if the future is done.
238
239      Done means either that a result / exception are available, or that the
240      future was cancelled.
241
242   .. method:: result()
243
244      Return the result this future represents.
245
246      If the future has been cancelled, raises :exc:`CancelledError`. If the
247      future's result isn't yet available, raises :exc:`InvalidStateError`. If
248      the future is done and has an exception set, this exception is raised.
249
250   .. method:: exception()
251
252      Return the exception that was set on this future.
253
254      The exception (or ``None`` if no exception was set) is returned only if
255      the future is done. If the future has been cancelled, raises
256      :exc:`CancelledError`. If the future isn't done yet, raises
257      :exc:`InvalidStateError`.
258
259   .. method:: add_done_callback(fn)
260
261      Add a callback to be run when the future becomes done.
262
263      The callback is called with a single argument - the future object. If the
264      future is already done when this is called, the callback is scheduled
265      with :meth:`~AbstractEventLoop.call_soon`.
266
267      :ref:`Use functools.partial to pass parameters to the callback
268      <asyncio-pass-keywords>`. For example,
269      ``fut.add_done_callback(functools.partial(print, "Future:",
270      flush=True))`` will call ``print("Future:", fut, flush=True)``.
271
272   .. method:: remove_done_callback(fn)
273
274      Remove all instances of a callback from the "call when done" list.
275
276      Returns the number of callbacks removed.
277
278   .. method:: set_result(result)
279
280      Mark the future done and set its result.
281
282      If the future is already done when this method is called, raises
283      :exc:`InvalidStateError`.
284
285   .. method:: set_exception(exception)
286
287      Mark the future done and set an exception.
288
289      If the future is already done when this method is called, raises
290      :exc:`InvalidStateError`.
291
292
293Example: Future with run_until_complete()
294^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
295
296Example combining a :class:`Future` and a :ref:`coroutine function
297<coroutine>`::
298
299    import asyncio
300
301    async def slow_operation(future):
302        await asyncio.sleep(1)
303        future.set_result('Future is done!')
304
305    loop = asyncio.get_event_loop()
306    future = asyncio.Future()
307    asyncio.ensure_future(slow_operation(future))
308    loop.run_until_complete(future)
309    print(future.result())
310    loop.close()
311
312The coroutine function is responsible for the computation (which takes 1 second)
313and it stores the result into the future. The
314:meth:`~AbstractEventLoop.run_until_complete` method waits for the completion of
315the future.
316
317.. note::
318   The :meth:`~AbstractEventLoop.run_until_complete` method uses internally the
319   :meth:`~Future.add_done_callback` method to be notified when the future is
320   done.
321
322
323Example: Future with run_forever()
324^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
325
326The previous example can be written differently using the
327:meth:`Future.add_done_callback` method to describe explicitly the control
328flow::
329
330    import asyncio
331
332    async def slow_operation(future):
333        await asyncio.sleep(1)
334        future.set_result('Future is done!')
335
336    def got_result(future):
337        print(future.result())
338        loop.stop()
339
340    loop = asyncio.get_event_loop()
341    future = asyncio.Future()
342    asyncio.ensure_future(slow_operation(future))
343    future.add_done_callback(got_result)
344    try:
345        loop.run_forever()
346    finally:
347        loop.close()
348
349In this example, the future is used to link ``slow_operation()`` to
350``got_result()``: when ``slow_operation()`` is done, ``got_result()`` is called
351with the result.
352
353
354Task
355----
356
357.. class:: Task(coro, \*, loop=None)
358
359   Schedule the execution of a :ref:`coroutine <coroutine>`: wrap it in a
360   future. A task is a subclass of :class:`Future`.
361
362   A task is responsible for executing a coroutine object in an event loop.  If
363   the wrapped coroutine yields from a future, the task suspends the execution
364   of the wrapped coroutine and waits for the completion of the future. When
365   the future is done, the execution of the wrapped coroutine restarts with the
366   result or the exception of the future.
367
368   Event loops use cooperative scheduling: an event loop only runs one task at
369   a time. Other tasks may run in parallel if other event loops are
370   running in different threads. While a task waits for the completion of a
371   future, the event loop executes a new task.
372
373   The cancellation of a task is different from the cancelation of a
374   future. Calling :meth:`cancel` will throw a
375   :exc:`~concurrent.futures.CancelledError` to the wrapped
376   coroutine. :meth:`~Future.cancelled` only returns ``True`` if the
377   wrapped coroutine did not catch the
378   :exc:`~concurrent.futures.CancelledError` exception, or raised a
379   :exc:`~concurrent.futures.CancelledError` exception.
380
381   If a pending task is destroyed, the execution of its wrapped :ref:`coroutine
382   <coroutine>` did not complete. It is probably a bug and a warning is
383   logged: see :ref:`Pending task destroyed <asyncio-pending-task-destroyed>`.
384
385   Don't directly create :class:`Task` instances: use the :func:`ensure_future`
386   function or the :meth:`AbstractEventLoop.create_task` method.
387
388   This class is :ref:`not thread safe <asyncio-multithreading>`.
389
390   .. classmethod:: all_tasks(loop=None)
391
392      Return a set of all tasks for an event loop.
393
394      By default all tasks for the current event loop are returned.
395
396   .. classmethod:: current_task(loop=None)
397
398      Return the currently running task in an event loop or ``None``.
399
400      By default the current task for the current event loop is returned.
401
402      ``None`` is returned when called not in the context of a :class:`Task`.
403
404   .. method:: cancel()
405
406      Request that this task cancel itself.
407
408      This arranges for a :exc:`~concurrent.futures.CancelledError` to be
409      thrown into the wrapped coroutine on the next cycle through the event
410      loop. The coroutine then has a chance to clean up or even deny the
411      request using try/except/finally.
412
413      Unlike :meth:`Future.cancel`, this does not guarantee that the task
414      will be cancelled: the exception might be caught and acted upon, delaying
415      cancellation of the task or preventing cancellation completely. The task
416      may also return a value or raise a different exception.
417
418      Immediately after this method is called, :meth:`~Future.cancelled` will
419      not return ``True`` (unless the task was already cancelled). A task will
420      be marked as cancelled when the wrapped coroutine terminates with a
421      :exc:`~concurrent.futures.CancelledError` exception (even if
422      :meth:`cancel` was not called).
423
424   .. method:: get_stack(\*, limit=None)
425
426      Return the list of stack frames for this task's coroutine.
427
428      If the coroutine is not done, this returns the stack where it is
429      suspended.  If the coroutine has completed successfully or was
430      cancelled, this returns an empty list.  If the coroutine was
431      terminated by an exception, this returns the list of traceback
432      frames.
433
434      The frames are always ordered from oldest to newest.
435
436      The optional limit gives the maximum number of frames to return; by
437      default all available frames are returned.  Its meaning differs depending
438      on whether a stack or a traceback is returned: the newest frames of a
439      stack are returned, but the oldest frames of a traceback are returned.
440      (This matches the behavior of the traceback module.)
441
442      For reasons beyond our control, only one stack frame is returned for a
443      suspended coroutine.
444
445   .. method:: print_stack(\*, limit=None, file=None)
446
447      Print the stack or traceback for this task's coroutine.
448
449      This produces output similar to that of the traceback module, for the
450      frames retrieved by get_stack().  The limit argument is passed to
451      get_stack().  The file argument is an I/O stream to which the output
452      is written; by default output is written to sys.stderr.
453
454
455Example: Parallel execution of tasks
456^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
457
458Example executing 3 tasks (A, B, C) in parallel::
459
460    import asyncio
461
462    async def factorial(name, number):
463        f = 1
464        for i in range(2, number+1):
465            print("Task %s: Compute factorial(%s)..." % (name, i))
466            await asyncio.sleep(1)
467            f *= i
468        print("Task %s: factorial(%s) = %s" % (name, number, f))
469
470    loop = asyncio.get_event_loop()
471    loop.run_until_complete(asyncio.gather(
472        factorial("A", 2),
473        factorial("B", 3),
474        factorial("C", 4),
475    ))
476    loop.close()
477
478Output::
479
480    Task A: Compute factorial(2)...
481    Task B: Compute factorial(2)...
482    Task C: Compute factorial(2)...
483    Task A: factorial(2) = 2
484    Task B: Compute factorial(3)...
485    Task C: Compute factorial(3)...
486    Task B: factorial(3) = 6
487    Task C: Compute factorial(4)...
488    Task C: factorial(4) = 24
489
490A task is automatically scheduled for execution when it is created. The event
491loop stops when all tasks are done.
492
493
494Task functions
495--------------
496
497.. note::
498
499   In the functions below, the optional *loop* argument allows explicitly setting
500   the event loop object used by the underlying task or coroutine.  If it's
501   not provided, the default event loop is used.
502
503.. function:: as_completed(fs, \*, loop=None, timeout=None)
504
505   Return an iterator whose values, when waited for, are :class:`Future`
506   instances.
507
508   Raises :exc:`asyncio.TimeoutError` if the timeout occurs before all Futures
509   are done.
510
511   Example::
512
513       for f in as_completed(fs):
514           result = yield from f  # The 'yield from' may raise
515           # Use result
516
517   .. note::
518
519      The futures ``f`` are not necessarily members of fs.
520
521.. function:: ensure_future(coro_or_future, \*, loop=None)
522
523   Schedule the execution of a :ref:`coroutine object <coroutine>`: wrap it in
524   a future. Return a :class:`Task` object.
525
526   If the argument is a :class:`Future`, it is returned directly.
527
528   .. versionadded:: 3.4.4
529
530   .. versionchanged:: 3.5.1
531      The function accepts any :term:`awaitable` object.
532
533   .. seealso::
534
535      The :meth:`AbstractEventLoop.create_task` method.
536
537.. function:: async(coro_or_future, \*, loop=None)
538
539   A deprecated alias to :func:`ensure_future`.
540
541   .. deprecated:: 3.4.4
542
543.. function:: gather(\*coros_or_futures, loop=None, return_exceptions=False)
544
545   Return a future aggregating results from the given coroutine objects or
546   futures.
547
548   All futures must share the same event loop.  If all the tasks are done
549   successfully, the returned future's result is the list of results (in the
550   order of the original sequence, not necessarily the order of results
551   arrival).  If *return_exceptions* is true, exceptions in the tasks are
552   treated the same as successful results, and gathered in the result list;
553   otherwise, the first raised exception will be immediately propagated to the
554   returned future.
555
556   Cancellation: if the outer Future is cancelled, all children (that have not
557   completed yet) are also cancelled.  If any child is cancelled, this is
558   treated as if it raised :exc:`~concurrent.futures.CancelledError` -- the
559   outer Future is *not* cancelled in this case.  (This is to prevent the
560   cancellation of one child to cause other children to be cancelled.)
561
562.. function:: iscoroutine(obj)
563
564   Return ``True`` if *obj* is a :ref:`coroutine object <coroutine>`,
565   which may be based on a generator or an :keyword:`async def` coroutine.
566
567.. function:: iscoroutinefunction(func)
568
569   Return ``True`` if *func* is determined to be a :ref:`coroutine function
570   <coroutine>`, which may be a decorated generator function or an
571   :keyword:`async def` function.
572
573.. function:: run_coroutine_threadsafe(coro, loop)
574
575   Submit a :ref:`coroutine object <coroutine>` to a given event loop.
576
577   Return a :class:`concurrent.futures.Future` to access the result.
578
579   This function is meant to be called from a different thread than the one
580   where the event loop is running. Usage::
581
582     # Create a coroutine
583     coro = asyncio.sleep(1, result=3)
584     # Submit the coroutine to a given loop
585     future = asyncio.run_coroutine_threadsafe(coro, loop)
586     # Wait for the result with an optional timeout argument
587     assert future.result(timeout) == 3
588
589   If an exception is raised in the coroutine, the returned future will be
590   notified. It can also be used to cancel the task in the event loop::
591
592     try:
593         result = future.result(timeout)
594     except asyncio.TimeoutError:
595         print('The coroutine took too long, cancelling the task...')
596         future.cancel()
597     except Exception as exc:
598         print('The coroutine raised an exception: {!r}'.format(exc))
599     else:
600         print('The coroutine returned: {!r}'.format(result))
601
602   See the :ref:`concurrency and multithreading <asyncio-multithreading>`
603   section of the documentation.
604
605   .. note::
606
607      Unlike other functions from the module,
608      :func:`run_coroutine_threadsafe` requires the *loop* argument to
609      be passed explicitly.
610
611   .. versionadded:: 3.5.1
612
613.. coroutinefunction:: sleep(delay, result=None, \*, loop=None)
614
615   Create a :ref:`coroutine <coroutine>` that completes after a given
616   time (in seconds).  If *result* is provided, it is produced to the caller
617   when the coroutine completes.
618
619   The resolution of the sleep depends on the :ref:`granularity of the event
620   loop <asyncio-delayed-calls>`.
621
622   This function is a :ref:`coroutine <coroutine>`.
623
624.. function:: shield(arg, \*, loop=None)
625
626   Wait for a future, shielding it from cancellation.
627
628   The statement::
629
630       res = yield from shield(something())
631
632   is exactly equivalent to the statement::
633
634       res = yield from something()
635
636   *except* that if the coroutine containing it is cancelled, the task running
637   in ``something()`` is not cancelled.  From the point of view of
638   ``something()``, the cancellation did not happen.  But its caller is still
639   cancelled, so the yield-from expression still raises
640   :exc:`~concurrent.futures.CancelledError`.  Note: If ``something()`` is
641   cancelled by other means this will still cancel ``shield()``.
642
643   If you want to completely ignore cancellation (not recommended) you can
644   combine ``shield()`` with a try/except clause, as follows::
645
646       try:
647           res = yield from shield(something())
648       except CancelledError:
649           res = None
650
651
652.. coroutinefunction:: wait(futures, \*, loop=None, timeout=None,\
653                            return_when=ALL_COMPLETED)
654
655   Wait for the Futures and coroutine objects given by the sequence *futures*
656   to complete.  Coroutines will be wrapped in Tasks. Returns two sets of
657   :class:`Future`: (done, pending).
658
659   The sequence *futures* must not be empty.
660
661   *timeout* can be used to control the maximum number of seconds to wait before
662   returning.  *timeout* can be an int or float.  If *timeout* is not specified
663   or ``None``, there is no limit to the wait time.
664
665   *return_when* indicates when this function should return.  It must be one of
666   the following constants of the :mod:`concurrent.futures` module:
667
668   .. tabularcolumns:: |l|L|
669
670   +-----------------------------+----------------------------------------+
671   | Constant                    | Description                            |
672   +=============================+========================================+
673   | :const:`FIRST_COMPLETED`    | The function will return when any      |
674   |                             | future finishes or is cancelled.       |
675   +-----------------------------+----------------------------------------+
676   | :const:`FIRST_EXCEPTION`    | The function will return when any      |
677   |                             | future finishes by raising an          |
678   |                             | exception.  If no future raises an     |
679   |                             | exception then it is equivalent to     |
680   |                             | :const:`ALL_COMPLETED`.                |
681   +-----------------------------+----------------------------------------+
682   | :const:`ALL_COMPLETED`      | The function will return when all      |
683   |                             | futures finish or are cancelled.       |
684   +-----------------------------+----------------------------------------+
685
686   This function is a :ref:`coroutine <coroutine>`.
687
688   Usage::
689
690        done, pending = yield from asyncio.wait(fs)
691
692   .. note::
693
694      This does not raise :exc:`asyncio.TimeoutError`! Futures that aren't done
695      when the timeout occurs are returned in the second set.
696
697
698.. coroutinefunction:: wait_for(fut, timeout, \*, loop=None)
699
700   Wait for the single :class:`Future` or :ref:`coroutine object <coroutine>`
701   to complete with timeout. If *timeout* is ``None``, block until the future
702   completes.
703
704   Coroutine will be wrapped in :class:`Task`.
705
706   Returns result of the Future or coroutine.  When a timeout occurs, it
707   cancels the task and raises :exc:`asyncio.TimeoutError`. To avoid the task
708   cancellation, wrap it in :func:`shield`.
709
710   If the wait is cancelled, the future *fut* is also cancelled.
711
712   This function is a :ref:`coroutine <coroutine>`, usage::
713
714       result = yield from asyncio.wait_for(fut, 60.0)
715
716   .. versionchanged:: 3.4.3
717      If the wait is cancelled, the future *fut* is now also cancelled.
718