| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
| |
Change-Id: I9ca92fdcec388e02462332e04fe7c1bf8b5f64b8
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Let's dive into what the problem is here.
First a description of what happens to a task that
is to be executed in a external (but local) process
via the process executor mechanism.
When a task is about to be sent to execute in the
external (but local) process its first cloned, this
is mainly done so that its notification callbacks can
be altered in a safe manner (ie not altering the
original task object to do this) and that clone has
its notifier emptied out.
What replaces the clone's notifier callbacks though
is a new object (that has a __call__ method so it
looks like just another callback) that will send
messages to the parent process (the one that has
the engine in it) over a secure(ish) channel whenever
the local task triggers its notifier notify() method.
This allows for callbacks in the parent process to
get triggered because once the messages recieved the
original tasks notifier object has its notify() method
called (therefore those callbacks do not really know
the task they are getting messages from is executing out
of process).
The issue though is that if the ANY(*) event type is registered
due to how it works in the notifier is that if the child/cloned
notifier has the ANY event type registered and the cloned task
calls notify() with a specific event this will cause the ANY
callback (in the clone) to transmit a message *and* it will
cause the *specific* event callback to also transmit a message
back to the parent process.
On the engine process side it will get 2 messages and trigger
the callbacks 3 times (twice for the specific event callback
because how the local notifier has the ANY callback registered
and one more time when the local process also sends the same
event based on its registration of the ANY event in the child
process).
This is not what is expected (the message rcved on the engine
process should only trigger one callback to get triggered
if the engine process task has no ANY callback registered or two
engine process callbacks to get triggered if the engine process
task has the ANY callback registered).
Closes-Bug: #1537948
Change-Id: I271bf1f23ad73df6c177cf00fd902c4881ba44ae
|
| |
|
|
|
|
|
|
|
|
| |
assertRaisesRegexp was renamed to assertRaisesRegex in Py3.2
For more details, please check:
https://docs.python.org/3/library/
unittest.html#unittest.TestCase.assertRaisesRegex
Change-Id: I89cce19e80b04074aab9f49a76c7652acace78b3
Closes-Bug: #1436957
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a local process based executor usage currently to ensure
that task emitted notifications are proxied we use the multi
processing library and use its queue concept. This sadly creates
a proxy process that gets associated, and this proxy process
handles the queue and messages sent to and from it. Instead of
doing this we can instead just create a temporary local socket
using a random socket and have tasks (which are running in
different processes) use that to communicate back any emitted
notifications instead (and we can use the asyncore module to handle
the emitted notifications since it handles the lower level socket
reading, polling and dispatching).
To ensure that the socket created is somewhat secure we use a
similar process as the multi-processing library uses where we
sign all messages with a hmac that uses a one time key that only
the main process and the child process know about (and reject
any messages that do not validate using this key).
Change-Id: Iff9180054bf14495e5667af00ae2fafbdbc23791
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Currently when a string is passed to the parallel engine
it will only know how to create a process or a native thread
based executor. The futurist library also supports making
a green thread based executor, so support creating it.
This will save glance some code that they have to create
a executor based on different options (one of those is a
green option).
Change-Id: I15c164a38b4445d28eb6062aed6c56cce0e0364b
|
| |\ |
|
| | |
| |
| |
| | |
Change-Id: Ic16c854d285398c688f132697c3bb7e637feb9a8
|
| |\ \
| |/
|/| |
|
| | |
| |
| |
| | |
Change-Id: I1d9b13b6f5cd48b7ac98f5c34ef9cb837f9ca7d1
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Christmas came early.
Closes-Bug: #1479466
Change-Id: I931d826690c925f022dbfffe9afb7bf41345b1d0
|
| | |
| |
| |
| |
| |
| |
| |
| | |
If someone really wants to provide a flow to run that is
empty that is there prerogative so it doesn't seem that valuable
to blow up if they do this.
Change-Id: I0ad89b0ade85a64f6ec107e2686454ef6dc97353
|
| |\ \ |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
In situations where many atoms can execute at the same
time it is sometimes useful to denote that when this situation
happens that certain atoms should execute/revert before other
atoms (or at least an attempt should be made to do this) instead
of being nearly arbitrary.
This adds a priority class attribute to the atom class (which
can be overridden or changed as needed) which is then used in
the runtime state machine to sort on so that atoms with higher
priority get submitted (and therefore executed/reverted) first.
Closes-Bug: #1507755
Change-Id: I3dcc705959085cba167883c85278e394b5cb1d2b
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
When a flow is added to another flow and there is
a decider placed on that link we need to make sure
that we retain that decider.
This also ensures that any predecessor of that flow
that has a decider that affects the flow nodes execution
also gets used in the decision process.
Closes-Bug: #1515748
Change-Id: Ifdf20378b26cdd13e0a3ff87cec8990fe89c0661
|
| |/ /
| |
| |
| |
| |
| |
| | |
First parameter should be the expected value.
Change-Id: I2941296e38c3245da298cc64aeb5636fbc4b2eb6
Partial-Bug: #1357117
|
| |/
|
|
|
|
|
|
|
|
| |
If an engines work was previously partially completed and
it is desired to reset it (and re-run) so that partially
completed or ignored (or other) work inside of it can run
again make that possible by exposing and documenting
a new `reset` method (and use it internally as well).
Change-Id: I47f82010a2108d5d8fd5e42ca9f7e5f165e65488
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
When a atom is reverted it can be useful to retain the
result of that 'revert' method being called, so that it
can be later analyzed (or used for various purposes) so
adjust the storage, and actions to enable it to be stored.
Change-Id: I38a9a5f3bf7550e924468bb4a86652cb8beb306c
|
| |\ \
| |/
|/| |
|
| | |
| |
| |
| | |
Change-Id: If89baa042695f19e42b6368034f3ccf22c2cf0aa
|
| |\ \
| |/
|/| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since this module contains more than the logbook
class and really is a our generic models that are used
to hold the runtime structure it is more appropriate to
place it under a models module and deprecate the usage
of the old module by placing a warning there (so that
when it is imported that warning is triggered).
Change-Id: I79def5ee08f560d38f2c9dcefd0b33becc2a4d36
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To make it possible to alter the runtime flow via a simple
conditional like structure make it possible to have the graph
flow link function take a decider that is expected to be some
callable that will decide (via a boolean return) whether the
edge should actually be traversed when running. When a decider
returns false; the affected + successors will be set into the
IGNORE state and they will be exempt from future runtime and
scheduling decisions.
Part of blueprint taskflow-conditional-execution
Change-Id: Iab0ee46f86d6b8e747911174d54a7295b3fa404d
|
| |
|
|
|
|
|
|
|
|
| |
Now that a task can provide the same thing that it
requires, having a nice little test that ensures what
is gotten is expected in a nice manner should be done
to make sure what a task will be getting for its arguments
respect the scoping that is defined by the flow ordering.
Change-Id: I1cdc5c3bd3f56b39f20f28f51c36017405150cea
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This change removes the DependencyFailure that is raised when a task
requires the same dependency as it provides. Taskflow returns a
frozenset([]) instead of a frozenset() when more than one value with the
same name is in the store.
This prevents the need for an inconvenient rename function when you want
to update a store variable with a new value.
Example case:
class Inc(task.Task):
def execute(self, a):
return a + 1
class AwkwardRename(task.Task):
def execute(self, b):
return b
store = {
'a': 1
}
f = linear_flow.Flow('inc-flow')
f.add(
Inc('t1',
provides='b',
requires='a',
),
AwkwardRename('t2',
provides='a',
requires='b'))
e = engines.load(f, store=store)
e.run()
print e.storage.fetch('a', many_handler=lambda x: x[-1])
Now with ability to have the same provides as requires:
class Inc(task.Task):
def execute(self, a):
return a + 1
store = {
'a': 1
}
f = linear_flow.Flow('inc-flow')
f.add(
Inc('t3',
provides='a',
requires='a'),
)
e = engines.load(f, store=store)
e.run()
print e.storage.fetch('a', many_handler=lambda x: x[-1])
Change-Id: I421e1ab33508c25baf78bd76df158bb6116d6fb0
|
| |
|
|
| |
Change-Id: I170bedb5c667aa30764ad29634910dda40c6cd49
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When look-up is occurring the possible provider
ordering is used instead of the scope returned provider
ordering. This causes incorrect matches when look-up
argument names that are produced by multiple providers (at
the same scope level) all providing the same requirement
name.
The scope order should be enforced as the de-facto order
and not the order that the storage unit finds (which is
hash based and varies depending on hash ordering).
Closes-Bug: #1425326
Change-Id: I15f1ee5515758bdc470c0f7dd7a2f616923e5628
|
| |
|
|
|
|
|
|
|
|
| |
Instead of having the optional requirements example be a
example that is itself a unittest just move the example to be an
actual unit test that gets tested using the various engine types
and change the example to be something slightly different (but
shows the same kind of usage information).
Change-Id: Ia03a81a6be636c501a35e7e290f587f7d05f8b30
|
| |
|
|
|
|
|
|
|
|
|
| |
Recreate a very simple eventlet utility module that
has only a few features; one function checks if eventlet is
available and if not raise an exception; and a constant that
can be used by calling code (such as tests or other optional
functionality) to check if eventlet is useable before
proceeding.
Change-Id: I32df0702eeae7c7c78972c9796156dd824b2f123
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of saving task state in a class variable that is later
introspected by further test code just remove that concept (which
doesn't work in multiprocessing or worker engines which can not
have access those types of shared/globally available concepts due
to how they run) and use a specialized listener that can gather the same
information in a more decoupled manner (and it will work in multiprocessing
and worker engines correctly).
This allows our engine test cases to work in those engine types which
increases those engines test coverage (and future coverage and engine tests
that are added).
Fixes a bunch of occurrences of bug 1357117 as well that were removed
during this cleanup and adjustment process...
Change-Id: Ic9901de2902ac28ec255bef146be5846d18f9bfb
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since we support various executors (threaded and distributed)
the next best executor when a threaded executor will not perform
and a distributed one requires to much setup is a local process
based one so it would be great to support this where we can.
Things that are currently (likely never) not going to work:
* Non-pickleable/non-copyable tasks
* Tasks that return non-pickleable/non-copyable results
* Tasks that use non-pickleable/non-copyable args/kwargs
Part of blueprint process-executor
Change-Id: I966ae01d390c7217b858db3feb2db949ce5c08d1
|
| |
|
|
|
|
|
|
|
|
|
| |
Instead of creating daemon threads using the threads module directly
use our small utility file to create the daemon thread on our behalf
and set the appropriate attributes to ensure it's a daemon thread.
This change replaces the existing locations where we were doing this
manually and uses the threading_utils helper function uniformly instead.
Change-Id: I535cee8a63407f753cf812df53c4f5bc83e0c9ae
|
| |\ |
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Internally we should be using the new location and not the
deprecated location wherever possible. This avoids emitting
warnings messages on our own code, which is a dirty habit.
Change-Id: Idac5a772eca7529d92542ada3be1cea092880e25
|
| |\ \
| |/
|/| |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This adjustment tweaks how workers and an engine use
there configuration (making it so they share the majority
of configuration). This also allows both engine and worker
to share the same configuration, which allows them to share
the same polling interval and so-on.
This has the effect of reducing the test time on a local
machine that I am running them on (one of many that I run
them on).
Before: Ran 1022 tests in ~130s (same machine)
After: Ran 1022 tests in ~60s (same machine)
Change-Id: I2150d26fb52cb86d61682124d710ba15c7ff27a6
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Move the currently existing green future executor and associated
code to a new futures types module so that it can be accessed from
this new location (TODO: deprecate the old location and link the
old to the new for one release so that we can remove the old link
in N + 1 release).
This unifies the API that the existing pool (thread or process) future
executors and the green thread pool future executor, and the newly added
synchronous executor (replacing the previous `make_completed_future`
function) provide so there usage is as seamless as possible.
Part of blueprint top-level-types
Change-Id: Ie5500eaa7f4425edb604b2dd13a15f82909a673b
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
To avoid having one set of options coming from `engine_conf`
and another set of options coming from `kwargs` and another set
coming from `engine_conf` if it is a URI just start to shift
toward `engine_conf` being deprecated and `engine` being a string
type only (or a URI with additional query parameters) and having
any additional **kwargs that are provided just get merged into the
final engine options.
This adds a new helper function that handles all these various
options and adds in a keyword argument `engine` that will be shifted
to in a future version (in that future version we can also then
remove the `engine_conf` and just stick to a smaller set of option
mechanisms).
It also adjusts all examples to use this new and more easier to
understand format and adjusts tests, conductor interface to use
this new more easily understandable style of getting an engine.
Change-Id: Ic7617057338e0c63775cf38a24643cff6e454950
|
| |
|
|
|
|
|
|
| |
Rename the single threaded engine to be the serial engine which
better matches its entrypoint, do the same for the multithreaded
engine (renaming it to the parallel engine).
Change-Id: I6174b4f1936858c13eeee416bfa3836cf20a1350
|
|
|
To match how we have a worker_based subdirectory for
its specific tests lets start moving pieces of the action
engine specific tests to its own directory as well and
move more in the future as well.
Change-Id: I003b07a95259ba18b961834515121243e27d7456
|
