:mod:`futures` --- Asynchronous computation =========================================== .. module:: futures :synopsis: Execute computations asynchronously using threads or (experimentally) processes. The :mod:`futures` module provides a high-level interface for asynchronously executing functions and methods. The asynchronous execution can be be performed by threads, using :class:`ThreadPoolExecutor`, or seperate processes, using :class:`ProcessPoolExecutor`. Both implement the same interface, which is defined by the abstract :class:`Executor` class. Executor Objects ---------------- :class:`Executor` is an abstract class that provides methods to execute calls asynchronously. It should not be used directly, but through its two subclasses: :class:`ThreadPoolExecutor` and (experimental) :class:`ProcessPoolExecutor`. .. method:: Executor.run_to_futures(calls, timeout=None, return_when=ALL_COMPLETED) Schedule the given calls for execution and return a :class:`FutureList` containing a :class:`Future` for each call. This method should always be called using keyword arguments, which are: *calls* must be a sequence of callables that take no arguments. *timeout* can be used to control the maximum number of seconds to wait before returning. If *timeout* is not specified or ``None`` then there is no limit to the wait time. *return_when* indicates when the method should return. It must be one of the following constants: +-----------------------------+----------------------------------------+ | Constant | Description | +=============================+========================================+ | :const:`FIRST_COMPLETED` | The method will return when any call | | | finishes. | +-----------------------------+----------------------------------------+ | :const:`FIRST_EXCEPTION` | The method will return when any call | | | raises an exception or when all calls | | | finish. | +-----------------------------+----------------------------------------+ | :const:`ALL_COMPLETED` | The method will return when all calls | | | finish. | +-----------------------------+----------------------------------------+ | :const:`RETURN_IMMEDIATELY` | The method will return immediately. | +-----------------------------+----------------------------------------+ .. method:: Executor.run_to_results(calls, timeout=None) Schedule the given calls for execution and return an iterator over their results. Raises a :exc:`TimeoutError` if the calls do not complete before *timeout* seconds. If *timeout* is not specified or ``None`` then there is no limit to the wait time. .. method:: Executor.map(func, *iterables, timeout=None) Equivalent to map(*func*, *\*iterables*) but executed asynchronously. Raises a :exc:`TimeoutError` if the map cannot be generated before *timeout* seconds. If *timeout* is not specified or ``None`` then there is no limit to the wait time. .. method:: Executor.shutdown() Signal the executor that it should free any resources that it is using when the currently pending futures are done executing. Calls to :meth:`Executor.run_to_futures`, :meth:`Executor.run_to_results` and :meth:`Executor.map` made after shutdown will raise :exc:`RuntimeError`. ThreadPoolExecutor Objects -------------------------- The :class:`ThreadPoolExecutor` class is an :class:`Executor` subclass that uses a pool of threads to execute calls asynchronously. .. class:: ThreadPoolExecutor(max_threads) Executes calls asynchronously using at pool of at most *max_threads* threads. .. _threadpoolexecutor-example: ThreadPoolExecutor Example ^^^^^^^^^^^^^^^^^^^^^^^^^^ :: import functools import urllib.request import futures URLS = ['http://www.foxnews.com/', 'http://www.cnn.com/', 'http://europe.wsj.com/', 'http://www.bbc.co.uk/', 'http://some-made-up-domain.com/'] def load_url(url, timeout): return urllib.request.urlopen(url, timeout=timeout).read() with futures.ThreadPoolExecutor(50) as executor: future_list = executor.run_to_futures( [functools.partial(load_url, url, 30) for url in URLS]) for url, future in zip(URLS, future_list): if future.exception() is not None: print('%r generated an exception: %s' % (url, future.exception())) else: print('%r page is %d bytes' % (url, len(future.result()))) ProcessPoolExecutor Objects --------------------------- The :class:`ProcessPoolExecutor` class is an **experimental** :class:`Executor` subclass that uses a pool of processes to execute calls asynchronously. There are situations where it can deadlock. :class:`ProcessPoolExecutor` uses the :mod:`multiprocessing` module, which allows it to side-step the :term:`Global Interpreter Lock` but also means that only picklable objects can be executed and returned. .. class:: ProcessPoolExecutor(max_processes=None) Executes calls asynchronously using a pool of at most *max_processes* processes. If *max_processes* is ``None`` or not given then as many worker processes will be created as the machine has processors. ProcessPoolExecutor Example ^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: PRIMES = [ 112272535095293, 112582705942171, 112272535095293, 115280095190773, 115797848077099, 1099726899285419] def is_prime(n): if n % 2 == 0: return False sqrt_n = int(math.floor(math.sqrt(n))) for i in range(3, sqrt_n + 1, 2): if n % i == 0: return False return True with futures.ProcessPoolExecutor() as executor: for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): print('%d is prime: %s' % (number, is_prime)) FutureList Objects ------------------ The :class:`FutureList` class is an immutable container for :class:`Future` instances and should only be instantiated by :meth:`Executor.run_to_futures`. .. method:: FutureList.wait(timeout=None, return_when=ALL_COMPLETED) Wait until the given conditions are met. This method should always be called using keyword arguments, which are: *timeout* can be used to control the maximum number of seconds to wait before returning. If *timeout* is not specified or ``None`` then there is no limit to the wait time. *return_when* indicates when the method should return. It must be one of the following constants: +-----------------------------+----------------------------------------+ | Constant | Description | +=============================+========================================+ | :const:`FIRST_COMPLETED` | The method will return when any call | | | finishes. | +-----------------------------+----------------------------------------+ | :const:`FIRST_EXCEPTION` | The method will return when any call | | | raises an exception or when all calls | | | finish. | +-----------------------------+----------------------------------------+ | :const:`ALL_COMPLETED` | The method will return when all calls | | | finish. | +-----------------------------+----------------------------------------+ | :const:`RETURN_IMMEDIATELY` | The method will return immediately. | | | This option is only available for | | | consistency with | | | :meth:`Executor.run_to_results` and is | | | not likely to be useful. | +-----------------------------+----------------------------------------+ .. method:: FutureList.cancel(timeout=None) Cancel every :class:`Future` in the list and wait up to *timeout* seconds for them to be cancelled or, if any are already running, to finish. Raises a :exc:`TimeoutError` if the running calls do not complete before the timeout. If *timeout* is not specified or ``None`` then there is no limit to the wait time. .. method:: FutureList.has_running_futures() Return true if any :class:`Future` in the list is currently executing. .. method:: FutureList.has_cancelled_futures() Return true if any :class:`Future` in the list was successfully cancelled. .. method:: FutureList.has_done_futures() Return true if any :class:`Future` in the list has completed or was successfully cancelled. .. method:: FutureList.has_successful_futures() Return true if any :class:`Future` in the list has completed without raising an exception. .. method:: FutureList.has_exception_futures() Return true if any :class:`Future` in the list completed by raising an exception. .. method:: FutureList.cancelled_futures() Return an iterator over all :class:`Future` instances that were successfully cancelled. .. method:: FutureList.done_futures() Return an iterator over all :class:`Future` instances that completed are were cancelled. .. method:: FutureList.successful_futures() Return an iterator over all :class:`Future` instances that completed without raising an exception. .. method:: FutureList.exception_futures() Return an iterator over all :class:`Future` instances that completed by raising an exception. .. method:: FutureList.running_futures() Return an iterator over all :class:`Future` instances that are currently executing. .. method:: FutureList.__len__() Return the number of futures in the :class:`FutureList`. .. method:: FutureList.__getitem__(i) Return the ith :class:`Future` in the list. The order of the futures in the :class:`FutureList` matches the order of the class passed to :meth:`Executor.run_to_futures` .. method:: FutureList.__contains__(future) Return true if *future* is in the list. Future Objects -------------- The :class:`Future` class encapulates the asynchronous execution of a function or method call. :class:`Future` instances are created by the :meth:`Executor.run_to_futures` and bundled into a :class:`FutureList`. .. method:: Future.cancel() Attempt to cancel the call. If the call is currently being executed then it cannot be cancelled and the method will return false, otherwise the call will be cancelled and the method will return true. .. method:: Future.cancelled() Return true if the call was successfully cancelled. .. method:: Future.done() Return true if the call was successfully cancelled or finished running. .. method:: Future.result(timeout=None) Return the value returned by the call. If the call hasn't yet completed then this method will wait up to *timeout* seconds. If the call hasn't completed in *timeout* seconds then a :exc:`TimeoutError` will be raised. If *timeout* is not specified or ``None`` then there is no limit to the wait time. If the future is cancelled before completing then :exc:`CancelledError` will be raised. If the call raised then this method will raise the same exception. .. method:: Future.exception(timeout=None) Return the exception raised by the call. If the call hasn't yet completed then this method will wait up to *timeout* seconds. If the call hasn't completed in *timeout* seconds then a :exc:`TimeoutError` will be raised. If *timeout* is not specified or ``None`` then there is no limit to the wait time. If the future is cancelled before completing then :exc:`CancelledError` will be raised. If the call completed without raising then ``None`` is returned. .. attribute:: Future.index int indicating the index of the future in its :class:`FutureList`.