summaryrefslogtreecommitdiff
path: root/taskflow/tests/utils.py
Commit message (Collapse)AuthorAgeFilesLines
* adding check for str/unicode type in requiresMin Pae2015-02-151-0/+6
| | | | | | | | | | | | When the requires argument for an Atom is passed in as a string, each character of the string is iterated over to build up a requirement list. This works for simple one letter argument names but not for long argument names. Added check for str and unicode types to prevent iterating over a string. Change-Id: Ida584221b48966d26935fb2ede0075aabb7ce972
* Improve upon/adjust/move around new optional exampleJoshua Harlow2015-02-111-0/+6
| | | | | | | | | | Instead of having the optional requirements example be a example that is itself a unittest just move the example to be an actual unit test that gets tested using the various engine types and change the example to be something slightly different (but shows the same kind of usage information). Change-Id: Ia03a81a6be636c501a35e7e290f587f7d05f8b30
* Add a capturing listener (for test or other usage)Joshua Harlow2015-02-031-47/+16
| | | | | | | | | | | To enable those that are using taskflow to easily attach to an engine (say in a unit test) and capture all the transitions, details and state changes in a accumulating 'values' list provide a capturing listener that does just this (and change the one used for internal testing to use it) so that others may also benefit from the same functionality. Change-Id: I7a35e282dd4a6b3e14277bea2af1f275615bd212
* Remove 'SaveOrderTask' and test state in class variablesJoshua Harlow2015-01-151-28/+54
| | | | | | | | | | | | | | | | | | | Instead of saving task state in a class variable that is later introspected by further test code just remove that concept (which doesn't work in multiprocessing or worker engines which can not have access those types of shared/globally available concepts due to how they run) and use a specialized listener that can gather the same information in a more decoupled manner (and it will work in multiprocessing and worker engines correctly). This allows our engine test cases to work in those engine types which increases those engines test coverage (and future coverage and engine tests that are added). Fixes a bunch of occurrences of bug 1357117 as well that were removed during this cleanup and adjustment process... Change-Id: Ic9901de2902ac28ec255bef146be5846d18f9bfb
* Remove direct usage of the deprecated failure locationJoshua Harlow2014-10-211-2/+2
| | | | | | | | Internally we should be using the new location and not the deprecated location wherever possible. This avoids emitting warnings messages on our own code, which is a dirty habit. Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
* Use and verify event and latch wait() return using timeoutsJoshua Harlow2014-10-181-9/+9
| | | | | | | | | | | | | Instead of blocking up the whole test suite when a latch or event was not decremented to its desired value (or not set for an event) we should use a reasonably high value that we use when waiting for those actions to occur and verify that when those wait() functions return that we have reached the desired state and if not either raise an exception or stop further testing. Fixes bug 1363739 Change-Id: I8b40282ac2db9cabd48b0b65c8a2a49610d77c4f
* Merge "Expose only `ensure_atom` from storage"Jenkins2014-10-181-0/+10
|\
| * Expose only `ensure_atom` from storageJoshua Harlow2014-09-271-0/+10
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Move the storage ensuring logic from being split across the engine and the storage layer and expose only a single `ensure_atom` function that does the work instead. This also removes the access to the `ensure_task` and `ensure_retry` methods as the internals of the `ensure_atom` function is now the only location that needs to use these two functions. This reduces the need to do type specific atom checks in the non-storage components (which we want to reduce overall). Breaking change: removes the public methods named `ensure_task` and `ensure_retry` (which should not be used externally anyway) from the storage object and makes those internal/private methods instead. Change-Id: I3a0f1f0dd777a1633b4937e16b50030275c84d1d
* | Switch to a custom NotImplementedError derivativeJoshua Harlow2014-10-081-1/+3
|/ | | | | | | | | | When a feature or method is not implemented it's useful to throw our own derivative of a NotImplementedError error so that we can distingush these types of errors vs actual usage of NotImplementedError which could be thrown from driver or user code. Change-Id: I8d5dfb56a254f315c5509dc600a078cfef2dde0b
* Tweak engine iteration 'close-up shop' runtime pathJoshua Harlow2014-09-041-0/+14
| | | | | | | | | | | | | | | | | | | | | | | 1. Have the runner yield the final set of failures instead of raising them, this allows the same yield syntax to be used for all exit points that the runner run_iter() produces and now raise failures from the main engine run loop to match this change. 2. Use a context manager instead of try/finally to start and stop the action engines task executor (teenie niceness...) 3. When the engine run_iter() is used and the generator that is returned is closed, instead of breaking from the run loop, which can leave running tasks incomplete instead continue running and signal to the runner that the engine has suspended itself. This ensures that the running atoms are not lost when the generator from run_iter() is closed (for whatever reason) before finishing. Also adds a bunch of useful tests that directly test the runner instead of the indirect testing that we were doing before. Fixes bug 1361013 Change-Id: I1b598e26f0b3877c8f7004f87bacdb7f5e9c9897
* Cleanup zookeeper integration testingJoshua Harlow2014-05-091-0/+25
| | | | | | | | | | | | In the persistence tests only use zake when zookeeper is not available (and of the right version). When zookeeper is available skip running zake. In the jobboard tests split out the tests which are not specific for zake into a base class (allowing for a future commit to add a zookeeper integration test). Change-Id: I50d51639a7f6c03c29d559c485676fddb9a7cf20
* Implement run iterationsJoshua Harlow2014-05-071-1/+1
| | | | | | | | | | | | | Instead of blocking the caller when they call run() allow there to be a new api run_iter() that will yield back the engine state transitions while running. This allows for a engine user to do alternate work while an engine is running (and come back to yield on there own time). Implements blueprint iterable-execution Change-Id: Ibb48c6c5618c97c59a6ab170dab5233ed47e5554
* Revert "Move taskflow.utils.misc.Failure to its own module"Jeremy Stanley2014-03-311-0/+16
| | | | | | | | | | | | This reverts commit 42ca240e8157b840c298d14fbf478ae570376633 which was a breaking change in a library consumed by other OpenStack projects with no deprecation or backwards compatibility considerations. It was able to merge because openstack/taskflow is apparently not yet part of the integrated gate via the proposed I202f4809afd689155e2cc4a00fc704fd772a0e92 change. Change-Id: I96cf36dc317499df91e43502efc85221f8177395 Closes-Bug: #1300161
* Move taskflow.utils.misc.Failure to its own moduleIvan A. Melnikov2014-03-311-16/+0
| | | | | | | | | | Failure class is important part of TaskFlow API, so it should be more visible and accessible. Breaking change: any client that used taskflow.utils.misc.Failure should be updated. Change-Id: Ib30000c9246bbcb227b34dfb0aba4d0b950bf926
* Add timeout to WaitForOneFromTaskIvan A. Melnikov2014-03-261-1/+7
| | | | | | We should never wait forever for anything. Change-Id: Icaa61febbf4745206abbe5bf9255c9401f341e71
* Merge "Drop obsolete test_unordered_flow"Jenkins2014-03-241-20/+0
|\
| * Drop obsolete test_unordered_flowIvan A. Melnikov2014-03-211-20/+0
| | | | | | | | | | | | Action engine tests were modified to keep few cases covered. Change-Id: Ibfcdd5378605532a4eb3fb1a43a70532a771ea16
* | tests: Fix WaitForOneFromTask constructor parameter introspectionIvan A. Melnikov2014-03-221-4/+3
|/ | | | | | | | Strings are iterable, so the check in WaitForOneFromTask was effectively useless. This lead to TypeErrors (caught by transition notifier) when it tried to find None in 'task1'. Change-Id: I3e91029f7b7cb960d6ca84490de76b5e044bc238
* Flow smart revert with retry controllerAnastasia Karpinska2014-03-181-1/+58
| | | | | | | | | | | | | | | | | | | - Remove flow REVERTING state. Now flow can be running and reverting simultaneously. Until flow isn't finished it is in RUNNING state. - Add RETRYING state for the retry controller. - Implement smart revertion and flow retries and retries resumption. - Default retry controllers: Times, ForEach and ParameterizedForEach. - Example and unit tests. Implements: blueprint subgraph-execution Implements: blueprint reversion-strategies Implements: blueprint smart-revert Change-Id: Ifa600bcad1edf2910f02ac36783cd458afbd880c
* Add retry action to execute retriesAnastasia Karpinska2014-03-141-0/+10
| | | | Change-Id: I093a49f88d9eecea5eb88fcecad9f7825cd7cb33
* Merge "Run action-engine tests with worker-based engine"Jenkins2014-02-201-0/+16
|\
| * Run action-engine tests with worker-based engineStanislav Kudriashev2014-02-191-0/+16
| | | | | | | | Change-Id: I00398fd2387b261c577f5d3eba9e2aebae3ba165
* | Merge "Message-oriented worker-based flow with kombu"Jenkins2014-02-201-0/+35
|\ \ | |/
| * Message-oriented worker-based flow with kombuStanislav Kudriashev2014-02-191-0/+35
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * Implemented Worker to be started on remote host for handling tasks request. * Implemented WorkerTaskExecutor that proxies tasks requests to remote workers. * Implemented Proxy that is used for consuming and publishing messages by Worker and Executor. * Added worker-based engine and worker task executor. * Added kombu dependency to requirements. * Added worker-based flow example. * Added unit-tests for worker-based flow components. Implements: blueprint worker-based-engine Change-Id: I8c6859ba4a1a56c2592e3d67cdfb8968b13ee99c
* | Remove extraneous vim configuration commentsyangxurong2014-02-141-2/+0
|/ | | | | | | | | Remove line containing comment - # vim: tabstop=4 shiftwidth=4 softtabstop=4 Change-Id: I7581cc88b8de433d5609ed06c6570b0b45c13573 Closes-Bug:#1229324
* Engine tests refactoringIvan A. Melnikov2014-01-301-10/+6
| | | | | | | | | | | | This change makes tests from test_action_engine.py more focused and deterministic: - replace assertIsSubset with assertIsSuperAndSubsequence, which checks order, too; - remove all sleeping from test tasks, it does not help anything anyway; - refactor tests that verify behaviour in case of task failing in nested subflow. Change-Id: I87ce1e5eed4a3a93c3c6593b618e82cfdd68204f
* Tests: don't pass 'values' to task constructorIvan A. Melnikov2014-01-301-8/+7
| | | | | | This might make it easier for worker-based engine to run it. Change-Id: I94d7b8e0a447708e448c8c5d3dd7232f67c7adda
* Move six to the right locationJoshua Harlow2014-01-091-1/+2
| | | | | | | | | Third party imports are supposed to be after standard library imports so make six be after. Some newlines are also added to separate standard library imports from all the others. Co-authored-by: Ivan A. Melnikov <imelnikov@griddynamics.com> Change-Id: Ied067e9367612758666da726df195ed390215e1b
* Enable H202 warning for flake8Ivan A. Melnikov2014-01-021-1/+1
| | | | Change-Id: Iaeb6d4ab1af5063558ef1a7b9a85e46c6cf0ccb7
* Unit tests refactoringAnastasia Karpinska2013-10-171-16/+163
| | | | | | | * duplicated tests were removed * common tasks moved to utils Change-Id: I69c91a264ec668b1333db8fd907298262af098cb
* Removed unused utilitiesIvan A. Melnikov2013-10-141-16/+0
| | | | Change-Id: Ifa3e7b820d594303c6c02eae12f55bce2bd1dacc
* No module name for builtin type and exception namesIvan A. Melnikov2013-10-101-8/+12
| | | | | | | | Make utils.reflection.get_class_name() return just type name (without module) for builtin exceptions and types. This makes output less clumsy and makes this function return same strings for python 2 and 3. Change-Id: Ibc599ed34c804cf2cf2ab67e7466a48ebe7f6df3
* Task decorator was removed and examples updatedAnastasia Karpinska2013-09-231-5/+3
| | | | Change-Id: Ie49fe6c2f48a18130d1fd2a3aa5485cd8cee4ed4
* Add basic sanity tests for unordered flowJoshua Harlow2013-09-221-0/+22
| | | | | | Fixes: bug 1224194 Change-Id: Id5af06a2f10e357e8d042df6f41795cc9328469b
* Add provides and requires properties to FlowAnastasia Karpinska2013-09-111-1/+1
| | | | | | | | | | | Provides and requires properties are used to browse all required and provided values for the whole Flow. The same properties were added to the Task. Appropriate Task properties were renamed to rebind and save_as. Change-Id: I02eb02303a9701a13f1a54f06f20bbf9aebd1d04
* Allow provides to be a set and results to be a dictionaryJoshua Harlow2013-09-071-6/+17
| | | | | | | | | | | | Instead of forcing task provides to be a tuple or list and using the corresponding numerical index to determine how to interpret the tasks results we should also allow for the task provides to be a set and then allow for the tasks result to be interpreted as a dictionary. Fixes bug 1221998 Change-Id: Ibae689e7975d6782aa248d2b6e3691c44a89ef9f
* Engine, task, linear_flow unificationJoshua Harlow2013-09-051-12/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | In order to move away from the existing flows having their own implementation of running, start moving the existing flows to be patterns that only structure tasks (and impose constraints about how the group of tasks can run) in useful ways. Let the concept of running those patterns be handled by an engine instead of being handled by the flow itself. This will allow for varying engines to be able to run flows in whichever way the engine chooses (as long as the constraints set up by the flow are observed). Currently threaded flow and graph flow are broken by this commit, since they have not been converted to being a structure of tasks + constraints. The existing engine has not yet been modified to run those structures either, work is underway to remediate this. Part of: blueprint patterns-and-engines Followup bugs that must be addressed: Bug: 1221448 Bug: 1221505 Change-Id: I3a8b96179f336d1defe269728ebae0caa3d832d7
* Rename Task.__call__ to Task.executeIvan A. Melnikov2013-09-031-2/+2
| | | | | | | | | We'd like library interface to be nice and simple, which implies that it does not involve too many underscores. Also 'execute' matches with 'revert' nicely. Suggested-by: Anastasia Karpinska <akarpinska@griddynamics.com> Change-Id: Iea72e6230a32665b4236a1e7d1dcc41bd56d0418
* File movementsJoshua Harlow2013-08-171-1/+1
| | | | | | | | | | | | In order to rework some of the persistence layer I would like to move around some of the files first, keeping job specifics in a jobs folder. Having some of the items which are root level taskflow items (flow, task) be at the root of the hiearchy. Also for now until the celery work is commited move that, since it doesn't make sense in backends anyway. Change-Id: If6c149710b40f70d4ec69ee8e523defe8f5e766d
* Added Memory Persistence API and Generic Datatypeskchenweijie2013-08-091-1/+14
| | | | | | | | | | Added a generic persistence API for JobBoards, Jobs, LogBooks, Workflows, FlowDetails, Tasks, and TaskDetails. The currently implemented backends are in-memory. Also implemented generic return types (listed above) for the generic API. Change-Id: I6d09860ee08a900faf2c213a230429bf9e0dec01
* Fix the current flake8 errorsAngus Salkeld2013-06-261-1/+1
| | | | | | | | The test pip ordering was resulting in the pep8 rules not really running so this fixes that up and the resulting errors so we can get a clean run. Change-Id: Icc27be05cd513c4f04135c7e768007dbf9d07b10
* Continue work on decorator usage.Joshua Harlow2013-05-291-2/+2
| | | | | | | | Remove the need for wrappers.py now that there is an advanced decorator provided that can provide all the same features as the functor wrapper task class. This new decorator is now used in the tests instead of the previous usage of the functor task class.
* Instead of apply use __call__Joshua Harlow2013-05-261-1/+1
| | | | | | | | Instead of requiring a apply() function just use the built-in one provided by objects or functions implementing __call__. Also change how requires/provides may not be found if functors are just passed in (since functors implement __call__).
* Use rollback accumulator and remove requires()/provides() from being functionsJoshua Harlow2013-05-231-9/+3
| | | | | | Use a new rollback accumulator to collect which tasks need to be rolled back and use that in the ordered workflow code. Move the usage of provides/requires as functions and just let them be attributes of the flow objects.
* More pylint cleanupsJoshua Harlow2013-05-221-0/+1
|
* Make more tests for linear and shuffle test utils to common file.Joshua Harlow2013-05-221-0/+58
View Lorry Controller Status