| Commit message (Collapse) | Author | Age | Files | Lines |
| |
|
|
|
|
| |
To cancel jobs cleanly we need to know when a job has failed.
|
| |
|
| |
|
|
|
|
| |
add_initiator() isn't necessary given lists have a remove method.
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| |
| |
| | |
Reviewed by:
Sam Thursfield
Richard Maw
Lars Wirzenius
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| | |
The contents of the message has changed for several events,
event messages that need to be sent to several initiators have a list
of ids instead of a single id.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There are two new messages:
WorkerBuildStepAlreadyStarted tells the initiator that the artifact
they want to build is already being built, e.g.
'eglibc-misc is already building on 172.17.1.37:3434'
WorkerBuildWaiting tells the initiator that the artifact they want
to build can't be built yet because there aren't any workers free, e.g.
'Ready to build eglibc-misc: waiting for a worker to become available'
|
| | |
|
| |
| |
| |
| |
| |
| |
| | |
Put our _exec_response_msg into WorkerBuildFinished event,
it's essentially the same as _finished_msg, just a different name
Get our artifact's cache key from the job
|
| |
| |
| |
| | |
Now we just get everything from the job object
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The exec_response_msg also needs to be sent to a number of initiators,
so we give it a list of ids not just one.
The exec_response_msg will be sent to the controller once the artifacts
have been cached successfully.
There's no longer any need to use a route map to retrieve
the id of the initiator, since this is stored with the job
|
| |
| |
| |
| |
| | |
msg now contains a list of initiator ids rather than a single one,
since BuiltOutput needs to be sent to a number of initiators
|
| |
| |
| |
| |
| |
| | |
Each job is given a unique id, so we don't need to generate
an id for each exec request this means we can remove use of route map
since we can use the job's id for the exec request
|
| |
| |
| |
| | |
This method no longer works, we will replace it soon.
|
| |
| |
| |
| |
| | |
The name change from BuildFailed -> JobFailed etc
was unintentionally merged into master, undo this.
|
| |
| |
| |
| |
| |
| | |
_job is the job this worker is carrying out
_exec_response_msg will contain the response the worker sends back to us
when it finishes the build.
|
| | |
|
| | |
|
| |
| |
| |
| | |
We need to be able to send this message to a number of initiators
|
| | |
|
|/ |
|
|
|
|
|
|
| |
Most of the time knowing the state of the build
graph isn't that useful for debugging distbuild,
but it may be useful in some situations.
|
|
|
|
|
|
| |
The controller should check the response event is
actually the response it was waiting for before
logging that there was a cache response
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
distbuild/build_controller.py
Reviewed by:
Lars Wirzenius
Daniel Silverstone
Sam Thursfield
|
| |
| |
| |
| | |
We now get the state of all artifacts with a single request.
|
| |
| |
| |
| | |
body and headers must now be specified for http-request message.
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There are cases where a state machine handles an event but stays in the
same state. A callback is registered which filters messages further
before possibly taking action. There have been bugs caused by this
pattern being incorrectly implemented (where the callback is expected to
filter the message, but a transition takes place anyway). Hopefully a
consistent naming convention will make the pattern clearer.
|
| |
| |
| |
| |
| |
| | |
There is always one BuildController object per InitiatorConnection.
By coupling the objects slightly closer we can simplify some transitions
in BuildController.
|
| |
| |
| |
| |
| | |
This is similar to the issue fixed by commit
c38b77bed86acc8b90f253ce354f3ecf98e475e7.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Each BuildController instance sets itself up to receive all messages from
all workers (via the WorkerConnection instances). In the case of a build
failure, all BuildController objects would transition to 'None' state
(causing them all to be destroyed) on any WorkerBuildFailed message.
This meant that if any one build failed on a distbuild network:
- the user whose build actually failed would receive the error
messages correctly
- any concurrent users would see no further build messages from the
controller, making it look like their builds had hung.
Ctrl+C from the 'hung' users would still be correctly handled by the
controller, as their InitiatorConnection instance would still be alive.
|
| | |
|
| |
| |
| |
| | |
Makes it easier to see what they mean at a glance.
|
| | |
|
| |
| |
| |
| | |
This makes tracking multiple builds through one log file a bit easier.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
New DistbuildSocket class that wraps socket.socket(), providing a
descriptive repr() handler showing where the socket is connected, and
providing a couple of helper methods for fetching local and remote
endpoint names.
This commit also adds a descriptive repr() handler to a few other
objects (mostly giving socket connection details).
|
| |
| |
| |
| |
| | |
We still log all messages sent to workers, which include the output of the serialise-artifact
code in full. There's no need for these status messages.
|
|/ |
|
| |
|
|
|
|
|
| |
We want to be able to transfer all
source artifacts in a single transaction
|
|
|
|
|
|
|
|
|
|
| |
Whenever the controller finds a source artifact
it wants to build, it changes its state to BUILDING.
We build all a chunk's source artifacts in one go.
So for any chunk artifact, we change the state of
all chunk artifacts that are built from the
same source to BUILDING
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Serialisation was simple when we only had 1 artifact per source.
However, to allow smaller systems, we need artifact splitting to produce
multiple artifacts per chunk source.
So now the new serialisation format has a separate list of artifacts
and sources, rather than the Source being generated from the artifact's
serialisation.
Python's id() function is used to encode the references between the
various Sources and Artifacts, these are replaced with a reference to
the new object after deserialisation.
Previously the cache-key was used, but this is no longer sufficient to
uniquely identify an Artifact.
The resultant build graph after deserialisation is a little different
to what went in: Strata end up with a different Source per Artifact,
so it _is_ a 1 to 1 mapping, as opposed to Chunks, where it's many to 1.
We serialise strata and chunks differently because stratum artifacts
from the same source can have different dependencies, for example
core-devel can have different dependencies to core-runtime.
Without intervention we would serialise core-devel and core-devel's
dependencies without including core-runtime's dependencies.
To solve this we've decided to encode stratum artifacts completely
indepedently: each stratum artifact has its own source. This is safe
because stratum artifacts can be constructed independently,
as opposed to Chunks where all the Artifacts for a Source
are produced together.
This is a little hacky in its current form, but it simplifies matters
later in distbuild with regards to how it handles expressing that
every Artifact that shares a Source is built together.
Arguably, this should be the output of producing the build graph
anyway, since it more helpfully represents which Artifacts are built
together than checking the morphology kind all the time, but more
assumptions need checking in morph before it's safe to make this
change across the whole of the morph codebase.
|