diff options
Diffstat (limited to 'docs/index.rst')
| -rw-r--r-- | docs/index.rst | 413 |
1 files changed, 221 insertions, 192 deletions
diff --git a/docs/index.rst b/docs/index.rst index 1a80434..ac006a8 100644 --- a/docs/index.rst +++ b/docs/index.rst @@ -5,10 +5,10 @@ :synopsis: Execute computations asynchronously using threads or processes. The :mod:`futures` module provides a high-level interface for asynchronously -executing functions and methods. +executing callables. -The asynchronous execution can be be performed by threads, using -:class:`ThreadPoolExecutor`, or seperate processes, using +The asynchronous execution can be be performed by threads using +:class:`ThreadPoolExecutor` or seperate processes using :class:`ProcessPoolExecutor`. Both implement the same interface, which is defined by the abstract :class:`Executor` class. @@ -19,63 +19,56 @@ Executor Objects asynchronously. It should not be used directly, but through its two subclasses: :class:`ThreadPoolExecutor` and :class:`ProcessPoolExecutor`. -.. method:: Executor.run_to_futures(calls, timeout=None, return_when=ALL_COMPLETED) +.. method:: Executor.submit(fn, *args, **kwargs) - Schedule the given calls for execution and return a :class:`FutureList` - containing a :class:`Future` for each call. This method should always be - called using keyword arguments, which are: + Schedules the callable to be executed as *fn*(*\*args*, *\*\*kwargs*) and + returns a :class:`Future` representing the execution of the callable. - *calls* must be a sequence of callables that take no arguments. +:: - *timeout* can be used to control the maximum number of seconds to wait before - returning. If *timeout* is not specified or ``None`` then there is no limit - to the wait time. + with ThreadPoolExecutor(max_workers=1) as executor: + future = executor.submit(pow, 323, 1235) + print(future.result()) - *return_when* indicates when the method should return. It must be one of the - following constants: +.. method:: Executor.map(func, *iterables, timeout=None) - +-----------------------------+----------------------------------------+ - | Constant | Description | - +=============================+========================================+ - | :const:`FIRST_COMPLETED` | The method will return when any call | - | | finishes. | - +-----------------------------+----------------------------------------+ - | :const:`FIRST_EXCEPTION` | The method will return when any call | - | | raises an exception or when all calls | - | | finish. | - +-----------------------------+----------------------------------------+ - | :const:`ALL_COMPLETED` | The method will return when all calls | - | | finish. | - +-----------------------------+----------------------------------------+ - | :const:`RETURN_IMMEDIATELY` | The method will return immediately. | - +-----------------------------+----------------------------------------+ + Equivalent to map(*func*, *\*iterables*) but func is executed asynchronously + and several calls to *func* may be made concurrently. The returned iterator + raises a :exc:`TimeoutError` if :meth:`__next__()` is called and the result + isn't available after *timeout* seconds from the original call to + :meth:`map()`. *timeout* can be an int or float. If *timeout* is not + specified or ``None`` then there is no limit to the wait time. If a call + raises an exception then that exception will be raised when its value is + retrieved from the iterator. -.. method:: Executor.run_to_results(calls, timeout=None) +.. method:: Executor.shutdown(wait=True) - Schedule the given calls for execution and return an iterator over their - results. The returned iterator raises a :exc:`TimeoutError` if - :meth:`__next__()` is called and the result isn't available after - *timeout* seconds from the original call to :meth:`run_to_results()`. If - *timeout* is not specified or ``None`` then there is no limit to the wait - time. If a call raises an exception then that exception will be raised when - its value is retrieved from the iterator. + Signal the executor that it should free any resources that it is using when + the currently pending futures are done executing. Calls to + :meth:`Executor.submit` and :meth:`Executor.map` made after shutdown will + raise :exc:`RuntimeError`. -.. method:: Executor.map(func, *iterables, timeout=None) + If *wait* is `True` then this method will not return until all the pending + futures are done executing and the resources associated with the executor + have been freed. If *wait* is `False` then this method will return + immediately and the resources associated with the executor will be freed + when all pending futures are done executing. Regardless of the value of + *wait*, the entire Python program will not exit until all pending futures + are done executing. - Equivalent to map(*func*, *\*iterables*) but executed asynchronously and - possibly out-of-order. The returned iterator raises a :exc:`TimeoutError` if - :meth:`__next__()` is called and the result isn't available after - *timeout* seconds from the original call to :meth:`run_to_results()`. If - *timeout* is not specified or ``None`` then there is no limit to the wait - time. If a call raises an exception then that exception will be raised when - its value is retrieved from the iterator. + You can avoid having to call this method explicitly if you use the `with` + statement, which will shutdown the `Executor` (waiting as if + `Executor.shutdown` were called with *wait* set to `True`): -.. method:: Executor.shutdown() +:: + + import shutil + with ThreadPoolExecutor(max_workers=4) as e: + e.submit(shutil.copy, 'src1.txt', 'dest1.txt') + e.submit(shutil.copy, 'src2.txt', 'dest2.txt') + e.submit(shutil.copy, 'src3.txt', 'dest3.txt') + e.submit(shutil.copy, 'src3.txt', 'dest4.txt') - Signal the executor that it should free any resources that it is using when - the currently pending futures are done executing. Calls to - :meth:`Executor.run_to_futures`, :meth:`Executor.run_to_results` and - :meth:`Executor.map` made after shutdown will raise :exc:`RuntimeError`. ThreadPoolExecutor Objects -------------------------- @@ -83,9 +76,43 @@ ThreadPoolExecutor Objects The :class:`ThreadPoolExecutor` class is an :class:`Executor` subclass that uses a pool of threads to execute calls asynchronously. -.. class:: ThreadPoolExecutor(max_threads) +Deadlock can occur when the callable associated with a :class:`Future` waits on +the results of another :class:`Future`. For example: + +:: + + import time + def wait_on_b(): + time.sleep(5) + print(b.result()) # b will never complete because it is waiting on a. + return 5 + + def wait_on_a(): + time.sleep(5) + print(a.result()) # a will never complete because it is waiting on b. + return 6 + + + executor = ThreadPoolExecutor(max_workers=2) + a = executor.submit(wait_on_b) + b = executor.submit(wait_on_a) + +And: + +:: + + def wait_on_future(): + f = executor.submit(pow, 5, 2) + # This will never complete because there is only one worker thread and + # it is executing this function. + print(f.result()) + + executor = ThreadPoolExecutor(max_workers=1) + executor.submit(wait_on_future) + +.. class:: ThreadPoolExecutor(max_workers) - Executes calls asynchronously using at pool of at most *max_threads* threads. + Executes calls asynchronously using at pool of at most *max_workers* threads. .. _threadpoolexecutor-example: @@ -93,28 +120,29 @@ ThreadPoolExecutor Example ^^^^^^^^^^^^^^^^^^^^^^^^^^ :: - import functools - import urllib.request - import futures - - URLS = ['http://www.foxnews.com/', - 'http://www.cnn.com/', - 'http://europe.wsj.com/', - 'http://www.bbc.co.uk/', - 'http://some-made-up-domain.com/'] - - def load_url(url, timeout): - return urllib.request.urlopen(url, timeout=timeout).read() - - with futures.ThreadPoolExecutor(50) as executor: - future_list = executor.run_to_futures( - [functools.partial(load_url, url, 30) for url in URLS]) - - for url, future in zip(URLS, future_list): - if future.exception() is not None: - print('%r generated an exception: %s' % (url, future.exception())) - else: - print('%r page is %d bytes' % (url, len(future.result()))) + import futures + import urllib.request + + URLS = ['http://www.foxnews.com/', + 'http://www.cnn.com/', + 'http://europe.wsj.com/', + 'http://www.bbc.co.uk/', + 'http://some-made-up-domain.com/'] + + def load_url(url, timeout): + return urllib.request.urlopen(url, timeout=timeout).read() + + with futures.ThreadPoolExecutor(max_workers=5) as executor: + future_to_url = dict((executor.submit(load_url, url, 60), url) + for url in URLS) + + for future in futures.as_completed(future_to_url): + url = future_to_url[future] + if future.exception() is not None: + print('%r generated an exception: %s' % (url, + future.exception())) + else: + print('%r page is %d bytes' % (url, len(future.result()))) ProcessPoolExecutor Objects --------------------------- @@ -125,16 +153,23 @@ uses a pool of processes to execute calls asynchronously. allows it to side-step the :term:`Global Interpreter Lock` but also means that only picklable objects can be executed and returned. -.. class:: ProcessPoolExecutor(max_processes=None) +Calling :class:`Executor` or :class:`Future` methods from a callable submitted +to a :class:`ProcessPoolExecutor` will result in deadlock. - Executes calls asynchronously using a pool of at most *max_processes* - processes. If *max_processes* is ``None`` or not given then as many worker +.. class:: ProcessPoolExecutor(max_workers=None) + + Executes calls asynchronously using a pool of at most *max_workers* + processes. If *max_workers* is ``None`` or not given then as many worker processes will be created as the machine has processors. +.. _processpoolexecutor-example: + ProcessPoolExecutor Example ^^^^^^^^^^^^^^^^^^^^^^^^^^^ :: + import math + PRIMES = [ 112272535095293, 112582705942171, @@ -153,164 +188,158 @@ ProcessPoolExecutor Example return False return True - with futures.ProcessPoolExecutor() as executor: - for number, is_prime in zip(PRIMES, executor.map(is_prime, PRIMES)): - print('%d is prime: %s' % (number, is_prime)) - -FutureList Objects ------------------- + def main(): + with futures.ProcessPoolExecutor() as executor: + for number, prime in zip(PRIMES, executor.map(is_prime, PRIMES)): + print('%d is prime: %s' % (number, prime)) -The :class:`FutureList` class is an immutable container for :class:`Future` -instances and should only be instantiated by :meth:`Executor.run_to_futures`. + if __name__ == '__main__': + main() -.. method:: FutureList.wait(timeout=None, return_when=ALL_COMPLETED) +Future Objects +-------------- - Wait until the given conditions are met. This method should always be - called using keyword arguments, which are: +The :class:`Future` class encapulates the asynchronous execution of a callable. +:class:`Future` instances are created by :meth:`Executor.submit`. - *timeout* can be used to control the maximum number of seconds to wait before - returning. If *timeout* is not specified or ``None`` then there is no limit - to the wait time. +.. method:: Future.cancel() - *return_when* indicates when the method should return. It must be one of the - following constants: + Attempt to cancel the call. If the call is currently being executed then + it cannot be cancelled and the method will return `False`, otherwise the call + will be cancelled and the method will return `True`. - +-----------------------------+----------------------------------------+ - | Constant | Description | - +=============================+========================================+ - | :const:`FIRST_COMPLETED` | The method will return when any call | - | | finishes. | - +-----------------------------+----------------------------------------+ - | :const:`FIRST_EXCEPTION` | The method will return when any call | - | | raises an exception or when all calls | - | | finish. | - +-----------------------------+----------------------------------------+ - | :const:`ALL_COMPLETED` | The method will return when all calls | - | | finish. | - +-----------------------------+----------------------------------------+ - | :const:`RETURN_IMMEDIATELY` | The method will return immediately. | - | | This option is only available for | - | | consistency with | - | | :meth:`Executor.run_to_results` and is | - | | not likely to be useful. | - +-----------------------------+----------------------------------------+ +.. method:: Future.cancelled() -.. method:: FutureList.cancel(timeout=None) + Return `True` if the call was successfully cancelled. - Cancel every :class:`Future` in the list and wait up to *timeout* seconds for - them to be cancelled or, if any are already running, to finish. Raises a - :exc:`TimeoutError` if the running calls do not complete before the timeout. - If *timeout* is not specified or ``None`` then there is no limit to the wait - time. +.. method:: Future.running() -.. method:: FutureList.has_running_futures() + Return `True` if the call is currently being executed and cannot be + cancelled. - Return `True` if any :class:`Future` in the list is currently executing. +.. method:: Future.done() -.. method:: FutureList.has_cancelled_futures() + Return `True` if the call was successfully cancelled or finished running. - Return `True` if any :class:`Future` in the list was successfully cancelled. +.. method:: Future.result(timeout=None) -.. method:: FutureList.has_done_futures() + Return the value returned by the call. If the call hasn't yet completed then + this method will wait up to *timeout* seconds. If the call hasn't completed + in *timeout* seconds then a :exc:`TimeoutError` will be raised. *timeout* can + be an int or float.If *timeout* is not specified or ``None`` then there is no + limit to the wait time. - Return `True` if any :class:`Future` in the list has completed or was - successfully cancelled. + If the future is cancelled before completing then :exc:`CancelledError` will + be raised. -.. method:: FutureList.has_successful_futures() + If the call raised then this method will raise the same exception. - Return `True` if any :class:`Future` in the list has completed without raising - an exception. +.. method:: Future.exception(timeout=None) -.. method:: FutureList.has_exception_futures() + Return the exception raised by the call. If the call hasn't yet completed + then this method will wait up to *timeout* seconds. If the call hasn't + completed in *timeout* seconds then a :exc:`TimeoutError` will be raised. + *timeout* can be an int or float. If *timeout* is not specified or ``None`` + then there is no limit to the wait time. - Return `True` if any :class:`Future` in the list completed by raising an - exception. + If the future is cancelled before completing then :exc:`CancelledError` will + be raised. -.. method:: FutureList.cancelled_futures() + If the call completed without raising then ``None`` is returned. - Return an iterator over all :class:`Future` instances that were successfully - cancelled. +.. method:: Future.add_done_callback(fn) -.. method:: FutureList.done_futures() + Attaches the callable *fn* to the future. *fn* will be called, with the + future as its only argument, when the future is cancelled or finishes + running. - Return an iterator over all :class:`Future` instances that completed or - were cancelled. + Added callables are called in the order that they were added and are always + called in a thread belonging to the process that added them. If the callable + raises an :exc:`Exception` then it will be logged and ignored. If the + callable raises another :exc:`BaseException` then the behavior is not + defined. -.. method:: FutureList.successful_futures() + If the future has already completed or been cancelled then *fn* will be + called immediately. - Return an iterator over all :class:`Future` instances that completed without - raising an exception. +Internal Future Methods +^^^^^^^^^^^^^^^^^^^^^^^ -.. method:: FutureList.exception_futures() +The following :class:`Future` methods are meant for use in unit tests and +:class:`Executor` implementations. - Return an iterator over all :class:`Future` instances that completed by - raising an exception. +.. method:: Future.set_running_or_notify_cancel() -.. method:: FutureList.running_futures() + This method should only be called by :class:`Executor` implementations before + executing the work associated with the :class:`Future` and by unit tests. - Return an iterator over all :class:`Future` instances that are currently - executing. + If the method returns `False` then the :class:`Future` was cancelled i.e. + :meth:`Future.cancel` was called and returned `True`. Any threads waiting + on the :class:`Future` completing (i.e. through :func:`as_completed` or + :func:`wait`) will be woken up. -.. method:: FutureList.__len__() + If the method returns `True` then the :class:`Future` was not cancelled + and has been put in the running state i.e. calls to + :meth:`Future.running` will return `True`. - Return the number of futures in the :class:`FutureList`. + This method can only be called once and cannot be called after + :meth:`Future.set_result` or :meth:`Future.set_exception` have been + called. -.. method:: FutureList.__getitem__(i) +.. method:: Future.set_result(result) - Return the ith :class:`Future` in the list. The order of the futures in the - :class:`FutureList` matches the order of the class passed to - :meth:`Executor.run_to_futures` + Sets the result of the work associated with the :class:`Future` to *result*. -.. method:: FutureList.__contains__(future) + This method should only be used by Executor implementations and unit tests. - Return `True` if *future* is in the :class:`FutureList`. +.. method:: Future.set_exception(exception) -Future Objects --------------- + Sets the result of the work associated with the :class:`Future` to the + :class:`Exception` *exception*. -The :class:`Future` class encapulates the asynchronous execution of a function -or method call. :class:`Future` instances are created by -:meth:`Executor.run_to_futures` and bundled into a :class:`FutureList`. + This method should only be used by Executor implementations and unit tests. -.. method:: Future.cancel() +Module Functions +---------------- - Attempt to cancel the call. If the call is currently being executed then - it cannot be cancelled and the method will return `False`, otherwise the call - will be cancelled and the method will return `True`. +.. function:: wait(fs, timeout=None, return_when=ALL_COMPLETED) -.. method:: Future.cancelled() + Wait for the :class:`Future` instances (possibly created by different + :class:`Executor` instances) given by *fs* to complete. Returns a named + 2-tuple of sets. The first set, named "done", contains the futures that + completed (finished or were cancelled) before the wait completed. The second + set, named "not_done", contains uncompleted futures. - Return `True` if the call was successfully cancelled. + *timeout* can be used to control the maximum number of seconds to wait before + returning. *timeout* can be an int or float. If *timeout* is not specified or + ``None`` then there is no limit to the wait time. -.. method:: Future.done() + *return_when* indicates when this function should return. It must be one of + the following constants: - Return `True` if the call was successfully cancelled or finished running. + +-----------------------------+----------------------------------------+ + | Constant | Description | + +=============================+========================================+ + | :const:`FIRST_COMPLETED` | The function will return when any | + | | future finishes or is cancelled. | + +-----------------------------+----------------------------------------+ + | :const:`FIRST_EXCEPTION` | The function will return when any | + | | future finishes by raising an | + | | exception. If no future raises an | + | | exception then it is equivalent to | + | | `ALL_COMPLETED`. | + +-----------------------------+----------------------------------------+ + | :const:`ALL_COMPLETED` | The function will return when all | + | | futures finish or are cancelled. | + +-----------------------------+----------------------------------------+ -.. method:: Future.result(timeout=None) +.. function:: as_completed(fs, timeout=None) - Return the value returned by the call. If the call hasn't yet completed then - this method will wait up to *timeout* seconds. If the call hasn't completed - in *timeout* seconds then a :exc:`TimeoutError` will be raised. If *timeout* + Returns an iterator over the :class:`Future` instances (possibly created + by different :class:`Executor` instances) given by *fs* that yields futures + as they complete (finished or were cancelled). Any futures that completed + before :func:`as_completed()` was called will be yielded first. The returned + iterator raises a :exc:`TimeoutError` if :meth:`__next__()` is called and + the result isn't available after *timeout* seconds from the original call + to :func:`as_completed()`. *timeout* can be an int or float. If *timeout* is not specified or ``None`` then there is no limit to the wait time. - - If the future is cancelled before completing then :exc:`CancelledError` will - be raised. - - If the call raised then this method will raise the same exception. - -.. method:: Future.exception(timeout=None) - - Return the exception raised by the call. If the call hasn't yet completed - then this method will wait up to *timeout* seconds. If the call hasn't - completed in *timeout* seconds then a :exc:`TimeoutError` will be raised. - If *timeout* is not specified or ``None`` then there is no limit to the wait - time. - - If the future is cancelled before completing then :exc:`CancelledError` will - be raised. - - If the call completed without raising then ``None`` is returned. - -.. attribute:: Future.index - - int indicating the index of the future in its :class:`FutureList`.
\ No newline at end of file |