| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
| |
It turns out we don't need to use multiprocessing.Manager() queues when
using the 'spawn' method - the regular multiprocessing queues are also
picklable, if passed as parameters to the new process.
Thanks to @BenjaminSchubert for pointing this out.
|
| |
|
|
|
| |
Calling disable_fork() will prevent the scheduler from running but will
allow communication with casd in the main process.
|
| |
|
|
| |
Cache size will be tracked by buildbox-casd.
|
| |
|
|
| |
Cache expiry will be managed by buildbox-casd.
|
| |
|
|
|
|
| |
App shouldn't need to inspect Elements directly when handling
failures, and after frontend process separation pickling the
object is not plausible.
|
| |
|
|
|
|
|
|
| |
Add support for using `multiprocessing.Manager` and the associated
queues. Downgrade the queue event callback guarantees accordingly. In
later work we may be able to support callbacks in all scenarios.
Pickle and unpickle the child job if the platform requires it.
|
| |
|
|
|
|
|
| |
Pave the way to supporting starting processes by the 'spawn' method, by
abstracting our usage of `multiprocessing.Queue`. This means we can
easily switch to using a multiprocessing.Manager() and associated queues
instead when necessary.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Adding the element full name and display key into all element related
messages removes the need to look up the plugintable via a plugin
unique_id just to retrieve the same values for logging and widget
frontend display. Relying on plugintable state is also incompatible
if the frontend will be running in a different process, as it will
exist in multiple states.
The element full name is now displayed instead of the unique_id,
such as in the debugging widget. It is also displayed in place of
'name' (i.e including any junction prepend) to be more informative.
|
| |
|
|
|
| |
The exception was incorrectly marked as 'KeyError', but enum throw
'ValueError' instead.
|
| |
|
|
|
|
|
| |
'Enum' has a big performance impact on the running code. Replacing
it with a safe subset of functionality removes lots of this overhead
without removing the benefits of using enums (safe comparisions,
uniqueness)
|
| |
|
|
|
|
|
|
| |
If we're running BuildStream tests then pickle child jobs.
This ensures that we keep things picklable, whilst we work towards being
able to support platforms that need to use the 'spawn' method of
starting processes.
|
| |
|
|
|
|
|
|
|
| |
Pave the way toward supporting the 'spawn' method of creating jobs, by
adding support for pickling ChildJobs. Introduce a new 'jobpickler'
module that provides an entrypoint for this functionality.
This also makes replays of jobs possible, which has made the debugging
of plugins much easier for me.
|
| |
|
|
| |
The {,un}register_exclusive_interest() mechanism is used for this now.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It was possible for multiple Queues of the same type to exist.
Currently, there is no desired reason for this to happen.
These changes add an explicit function call to the Scheduler that
destroys the queues, to be used before constructing the next list of
queues to pass into the Scheduler.
It also calls this in all the places before we construct the queues.
Further, it alters Stream.fetch_subprojects because there is currently
no reason why we'd want to preserve the Stream's list of queues before
running.
|
| |
|
|
|
|
|
|
| |
We only seen to generate the list so we can get its length, so it is
more efficient to only store a count of skipped/processed elements.
failed_elements needs to remain a list for the moment, as it's used to
retry a failed element job.
|
| |
|
|
|
|
|
|
|
| |
Reduce the amount of context shared with child jobs, by only sending the
messenger portion of it rather than the whole thing. Also send the
logdir.
This also means that we will need to pickle less stuff when using the
'spawn' method of multi-processing, as opposed to the 'fork' method.
|
| |
|
|
|
|
| |
Instead of having methods in Context forward calls on to the Messenger,
have folks call the Messenger directly. Remove the forwarding methods in
Context.
|
| |
|
|
|
|
|
|
| |
Remove the need to pass the Context object to message handlers, by
passing what is usually requested from the context instead.
This paves the way to sharing less information with some child jobs -
they won't need the whole context object, just the messenger.
|
| |
|
|
|
|
|
|
|
|
|
| |
Stop passing the scheduler's job queue's across to child jobs, via the
'action_cb' parameter. Instead pass a module-level function, which will
pickle nicely.
This isn't much of a problem while we are in the 'fork' multiprocessing
model. As we move towards supporting the 'spawn' model for win32, then
we need to consider what we will be pickling and unpickling, to cross
the process boundary.
|
| | |
|
| |
|
|
|
|
|
| |
Instead of passing the whole scheduler to the ChildJob, only pass the
part that is used - the context. Reducing the amount of shared state
makes it easier to follow what's going on, and will make it more
economical to move to away from the 'fork' model later.
|
| |
|
|
|
|
| |
Both run() and the App callback _ticker_callback() call & return
elapsed_time(), which is the unused by the respective callers and
as such is unnecessary.
|
| |
|
|
|
|
|
| |
This patch includes setting a _depth to each element
once the pipeline has been sorted. This is necessary
as we need to store elements in the heapq sorted by
their depth.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Queue.enqueue() and Queue.harvest_jobs() now exhibit
push-based behaviour. Although most of the logic from
Queue.enqueue() has been moved to Queue._enqueue_element()
* QueueStatus.WAIT has been replaced with QueueStatus.PENDING
to reflect the new push-based nature of the queues
* There now exists a virtual method in Queue:
register_pending_element which is used to register am
element which is not immediately ready to be processed
in the queue with specific callbacks. These
callbacks will enqueue the element when called.
|
| |
|
|
|
|
| |
Since JobStatus is an enum, it's clearer to compare using 'is' -
equality comparison will fail in the same cases, but might lull folks
into thinking that comparison with integer would also work.
|
| |
|
|
|
|
|
|
| |
This provides some minor guards against mistakes, and we'll be able to
do type-checking later.
This does open the possibility of problems if folks mistakenly try to
pass off an integer as a JobStatus.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Avoid confusion by not referring to starting another process as
'spawning'. Note that 'spawn' is a process creation method, which is an
alternative to forking.
Say 'create child process' instead of 'fork' where it doesn't harm
understanding. Although we currently only use the 'fork' method for
creating subprocesses, there are reasons for us to support 'spawn' in
the future.
More information on forking and spawning:
https://docs.python.org/3/library/multiprocessing.html#contexts-and-start-methods
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Simplify the custom 'handle_message' / 'send_message' protocol by not
requiring a message_type. These message types share a namespace with the
base Job implementation, which could cause trouble.
Introduce a new private '_send_message' to implement the old
functionality.
Subclasses are free to pack a message type into their own messages, this
isn't necessary at present and simplifies existing subclass code.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
Make it clearer what happens in which process by splitting out a
'ChildJob', which encapsulates the work that happens in the child
process.
This also makes it possible to control what is transferred to the
child process. This is very useful for adding support for the 'spawn'
method of creating child processes as opposed to the 'fork' method.
|
| |
|
|
| |
This just does the default behaviour, clearer to remove it.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
Ease the burden on subclasses of Job slightly, by providing a new
set_message_unique_id() method. It ensures that created Message
instances will use that id.
This removes the need to override the message() method, so it is no
longer in the 'abstract method' section.
Enable callers to Job's message() method to override the 'unique_id'.
|
| |
|
|
|
|
|
|
| |
Move the parent-specific methods above the child-specific methods.
This makes slightly more sense chronologically, as the parent creates
the child. It will also make diffs cleaner when splitting parent and
child into separate classes.
|
|
|
This was discussed in #1008.
Fixes #1009.
|
