summaryrefslogtreecommitdiff
path: root/taskflow/utils/threading_utils.py
Commit message (Collapse)AuthorAgeFilesLines
* Add a executor backed conductor and have existing impl. use itJoshua Harlow2015-11-121-0/+12
| | | | | | | | | | | | | | | | | | | This adds a executor backed job dispatching base class and has the existing blocking executor use it by running jobs and dispatching jobs into a sync executor. It also allows for dispatching jobs into a thread executor, or other executor via a new '_executor_factory' method that can generate executors (it can be overriden in the non-blocking executor to provide your own executors instances). This does alter the behavior in that now that jobs are dispatched into an executor we no longer can immediatly know if a job was dispatched and raised an exception or whether it will raise an exception in the future, so we now alter the 'local_dispatched' to just be a boolean that is used to determine if any dispatches happened (failure or not). Change-Id: I485770e8f4c85d3833892a453c9fb5168d8f0407
* Integrate futurist (and **remove** taskflow originating code)Joshua Harlow2015-07-091-12/+0
| | | | Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
* Remove 2.6 classifier + 2.6 compatibility codeJoshua Harlow2015-06-211-20/+0
| | | | | | | | Fixes bug 1445827 Depends-On: I02e3c9aacef0b295a2f823a5cbaf11768a90cb82 Change-Id: I1db681803598ac1bc917fd74a99458bc61edf3f1
* Set a no-op functor when none is providedJoshua Harlow2015-03-071-19/+23
| | | | | | | | | | | Instead of having later checks to check for none when calling the thread bundle callback just initially set it to a no-op functor that does nothing at binding time instead; this avoids the need to do repeated checks at start/stop time. Change-Id: I5ab4f801bf4767c802ea607fdf864d4852e6c84d
* Ensure the thread bundle stops in last to first orderJoshua Harlow2015-03-041-2/+6
| | | | | | | | | | Instead of stopping in first started to last started which is typically not desired (this is the starting order) we should do the reverse and stop in last to first instead (which is typically the expected stopping order) by default. Change-Id: Ic579438bc549d380c62c4d56c55c168de425adeb
* Add a thread bundle helper utility + testsJoshua Harlow2015-01-241-0/+104
| | | | | | | | | | | | To make it easier to create a bunch of threads in a single call (and stop them in a single call) create a concept of a thread bundle (similar to a thread group) that will call into a provided set of factories to get a thread, activate callbacks to notify others that a thread is about to start or stop and then perform the start or stop of the bound threads in a orderly manner. Change-Id: I7d233cccb230b716af41243ad27220b988eec14c
* Get the basics of a process executor workingJoshua Harlow2014-12-191-0/+7
| | | | | | | | | | | | | | | | | Since we support various executors (threaded and distributed) the next best executor when a threaded executor will not perform and a distributed one requires to much setup is a local process based one so it would be great to support this where we can. Things that are currently (likely never) not going to work: * Non-pickleable/non-copyable tasks * Tasks that return non-pickleable/non-copyable results * Tasks that use non-pickleable/non-copyable args/kwargs Part of blueprint process-executor Change-Id: I966ae01d390c7217b858db3feb2db949ce5c08d1
* Use and verify event and latch wait() return using timeoutsJoshua Harlow2014-10-181-0/+20
| | | | | | | | | | | | | Instead of blocking up the whole test suite when a latch or event was not decremented to its desired value (or not set for an event) we should use a reasonably high value that we use when waiting for those actions to occur and verify that when those wait() functions return that we have reached the desired state and if not either raise an exception or stop further testing. Fixes bug 1363739 Change-Id: I8b40282ac2db9cabd48b0b65c8a2a49610d77c4f
* Remove functions created for pre-six 1.7.0Joshua Harlow2014-06-181-6/+5
| | | | | | | | | With the new six version we can remove some of the utility functions that we created now that upstream six has equivalent or better functionality. Change-Id: I637fcf1475ca1af02608e736ca920c3e116c5529
* Move the daemon thread helper functionJoshua Harlow2014-04-101-0/+11
| | | | | | This function seems better suited in the threading_utils module. Change-Id: Iddd438b57973c7c6c26bd7b6239630656530bd1b
* Fix few spelling and grammar errorsIvan A. Melnikov2014-02-201-1/+1
| | | | Change-Id: I8e678a9f76cba3d90053d28baf4ff42d34b4e390
* Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
| | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Use reader/writer locks in storageJoshua Harlow2014-02-051-23/+5
| | | | | | | | | | | | | | Switch to using a reader/writer lock scheme to protect against simultaneous storage mutations, typically seen when running in a multi-threaded mode. For the single-threaded mode provide a dummy reader/writer lock which will mimic the locking api but not actually lock anything. Closes-Bug: 1273146 Change-Id: I954f542d9ab34b693e8da71c9fc913f823e869ba
* Doc strings and comments clean-upskudriashev2014-01-261-1/+1
| | | | | | | | * Added missing period for doc strings * Correct syntax errors * Remove H402 from flake8 ignore list Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
* Move six to the right locationJoshua Harlow2014-01-091-1/+2
| | | | | | | | | Third party imports are supposed to be after standard library imports so make six be after. Some newlines are also added to separate standard library imports from all the others. Co-authored-by: Ivan A. Melnikov <imelnikov@griddynamics.com> Change-Id: Ied067e9367612758666da726df195ed390215e1b
* Removed unused utilitiesIvan A. Melnikov2013-10-141-57/+0
| | | | Change-Id: Ifa3e7b820d594303c6c02eae12f55bce2bd1dacc
* Fix several python3 incompatibilitiesIvan A. Melnikov2013-10-111-1/+2
| | | | Change-Id: Ibf5dd6c0b6bcd161364daf35d618641f6079acf5
* Remove decorators and move to utilsJoshua Harlow2013-10-071-42/+2
| | | | | | | | | | | | | In order to avoid the circular import in threading utils move the decorators functionality to utils/misc and move the locking functionality to utils/lock_utils and then use these functions from the threading util (and elsewhere). Fixes bug: 1236080 Change-Id: I9e71c2ba15782cbb6dd5ab7e1264b77ed47bc29e
* Flow utils and adding commentsJoshua Harlow2013-10-051-0/+1
| | | | | | | | | Make the graph duplicate node names be a post flatten function and add a bunch of mostly useful comments to the various utils functions/modules. Change-Id: Iea7afc2f1b6e96a87da0aa2ca83dfbe4b76ee2db
* Simpler API to load flows into enginesIvan A. Melnikov2013-10-031-0/+12
| | | | | | | | | | | | | | | | | | | | | Previously to run a flow client code had to put together the flow, an engine, logbook, flowdetail, and storage backend. This commit adds two helper functions, run() and load(), so that simplest usecase now looks like taskflow.engines.run(flow) Client code may also provide configuration for storage and engine if needed, but if not needed it just works with defaults. Engines are loaded via stevedore, as drivers in 'taskflow.engines' backend. Now three entry points are defined in that namespace: - 'default', for SingleThreadedActionEngine, used by default; - 'serial', as another synonym for SingleThreadedActionEngine; - 'parallel', for MultiThreadedActionEngine. Closes-bug: #1224726 Change-Id: I7f4cb5c8ff7f5f12831ddd0952c202d2fd8cd6ef
* Remove unused utility classesIvan A. Melnikov2013-09-241-27/+0
| | | | | | | | Removing ThreadGroupExecutor allows to get rid of threading2 dependency. LastFedIter is also not used, so there is no reason to keep it. Change-Id: Id396ab59ebf1e2314ef620417d7cac09e1ab8d3b
* MultiThreaded engine and parallel actionAnastasia Karpinska2013-09-041-0/+19
| | | | | | | | MultiThreaded engine was implemented to execute tasks in parallel: - added parallel action that executes and reverts tasks in parallel; - added thread-safe storage. Change-Id: I4a1f78c95ae5d38660bd32ce21d2b3fb1b2af8ad
* Split utils moduleIvan A. Melnikov2013-09-031-0/+146
In this commit we split utils module into several parts: - flow_utils, with code used in running flows; - threading_utils, with code that helps in working with threads; - reflection, with code that inspects python objects metadata; - misc, with all the other code that used to live in utils.py. We also move graph_utils into taskflow.utils package. This commit just moves code around. It should not change any logic (with exception of complex_graph example). Change-Id: Iebfe45395f0ff502bc00fc7ae14829130b2c6abe
View Lorry Controller Status