summaryrefslogtreecommitdiff
path: root/distbuild
Commit message (Collapse)AuthorAgeFilesLines
* Merge branch 'baserock/richardipsum/distbuild_jobs'Richard Ipsum2014-04-246-103/+262
|\ | | | | | | | | | | | | Reviewed by: Sam Thursfield Richard Maw Lars Wirzenius
| * Import all the thingsRichard Ipsum2014-04-231-1/+4
| |
| * Initiator: Handle step-already-started messageRichard Ipsum2014-04-231-0/+7
| |
| * InitiatorConnection: Handle _step_already_startedRichard Ipsum2014-04-231-1/+21
| |
| * Check whether request id is in the event's id listRichard Ipsum2014-04-231-6/+8
| | | | | | | | | | | | The contents of the message has changed for several events, event messages that need to be sent to several initiators have a list of ids instead of a single id.
| * Add handling for new messages to BuildControllerRichard Ipsum2014-04-231-1/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two new messages: WorkerBuildStepAlreadyStarted tells the initiator that the artifact they want to build is already being built, e.g. 'eglibc-misc is already building on 172.17.1.37:3434' WorkerBuildWaiting tells the initiator that the artifact they want to build can't be built yet because there aren't any workers free, e.g. 'Ready to build eglibc-misc: waiting for a worker to become available'
| * Add new step-already-started message to protocolRichard Ipsum2014-04-231-0/+5
| |
| * WorkerConnection: _maybe_handle_helper_resultRichard Ipsum2014-04-231-9/+4
| | | | | | | | | | | | | | Put our _exec_response_msg into WorkerBuildFinished event, it's essentially the same as _finished_msg, just a different name Get our artifact's cache key from the job
| * WorkerConnection: _request_cachingRichard Ipsum2014-04-231-9/+6
| | | | | | | | Now we just get everything from the job object
| * WorkerConection: _handle_exec_responseRichard Ipsum2014-04-231-9/+8
| | | | | | | | | | | | | | | | | | | | | | The exec_response_msg also needs to be sent to a number of initiators, so we give it a list of ids not just one. The exec_response_msg will be sent to the controller once the artifacts have been cached successfully. There's no longer any need to use a route map to retrieve the id of the initiator, since this is stored with the job
| * WorkerConnection: _handle_exec_outputRichard Ipsum2014-04-231-2/+2
| | | | | | | | | | msg now contains a list of initiator ids rather than a single one, since BuiltOutput needs to be sent to a number of initiators
| * WorkerBuildQueuer: Use job's artifact and idRichard Ipsum2014-04-231-14/+17
| | | | | | | | | | | | Each job is given a unique id, so we don't need to generate an id for each exec request this means we can remove use of route map since we can use the job's id for the exec request
| * Remove cancelRichard Ipsum2014-04-231-6/+1
| | | | | | | | This method no longer works, we will replace it soon.
| * Change event names backRichard Ipsum2014-04-231-3/+3
| | | | | | | | | | The name change from BuildFailed -> JobFailed etc was unintentionally merged into master, undo this.
| * WorkerConnection: misc attributesRichard Ipsum2014-04-231-0/+6
| | | | | | | | | | | | _job is the job this worker is carrying out _exec_response_msg will contain the response the worker sends back to us when it finishes the build.
| * WorkerBuildQueuer: replace request queue with jobsRichard Ipsum2014-04-231-25/+67
| |
| * Add Jobs and Job classesRichard Ipsum2014-04-231-10/+49
| |
| * Make WorkerBuildCaching carry a list of idsRichard Ipsum2014-04-231-5/+2
| | | | | | | | We need to be able to send this message to a number of initiators
| * Add new build messages to worker build schedulerRichard Ipsum2014-04-231-2/+12
| |
* | Remove duplicated function _artifact_filenameRichard Ipsum2014-04-161-8/+2
|/
* Add flag for debugging state of build graphRichard Ipsum2014-04-161-8/+10
| | | | | | Most of the time knowing the state of the build graph isn't that useful for debugging distbuild, but it may be useful in some situations.
* Don't log a cache response till we're sureRichard Ipsum2014-04-161-1/+2
| | | | | | The controller should check the response event is actually the response it was waiting for before logging that there was a cache response
* Merge branch 'baserock/richardipsum/distbuild_improve_annotation3'Richard Ipsum2014-04-153-55/+75
|\ | | | | | | | | | | | | | | | | | | Conflicts: distbuild/build_controller.py Reviewed by: Lars Wirzenius Daniel Silverstone Sam Thursfield
| * Annotation using single post requestRichard Ipsum2014-04-111-53/+70
| | | | | | | | We now get the state of all artifacts with a single request.
| * Set body and headers in messageRichard Ipsum2014-04-111-1/+2
| | | | | | | | body and headers must now be specified for http-request message.
| * Add body and headers to http-request messageRichard Ipsum2014-04-081-0/+2
| |
* | Fix lines longer than 80 characters (my fault)Sam Thursfield2014-04-143-5/+8
| |
* | distbuild: Consistently name all maybe_ message callbacksSam Thursfield2014-04-142-25/+25
| | | | | | | | | | | | | | | | | | There are cases where a state machine handles an event but stays in the same state. A callback is registered which filters messages further before possibly taking action. There have been bugs caused by this pattern being incorrectly implemented (where the callback is expected to filter the message, but a transition takes place anyway). Hopefully a consistent naming convention will make the pattern clearer.
* | distbuild: Make some transitions more specificSam Thursfield2014-04-142-23/+18
| | | | | | | | | | | | There is always one BuildController object per InitiatorConnection. By coupling the objects slightly closer we can simplify some transitions in BuildController.
* | distbuild: Fix possible 'one user disconnecting breaks all builds' issueSam Thursfield2014-04-141-2/+2
| | | | | | | | | | This is similar to the issue fixed by commit c38b77bed86acc8b90f253ce354f3ecf98e475e7.
* | distbuild: Fix 'one build failure causes all concurrent builds to hang'Sam Thursfield2014-04-141-8/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Each BuildController instance sets itself up to receive all messages from all workers (via the WorkerConnection instances). In the case of a build failure, all BuildController objects would transition to 'None' state (causing them all to be destroyed) on any WorkerBuildFailed message. This meant that if any one build failed on a distbuild network: - the user whose build actually failed would receive the error messages correctly - any concurrent users would see no further build messages from the controller, making it look like their builds had hung. Ctrl+C from the 'hung' users would still be correctly handled by the controller, as their InitiatorConnection instance would still be alive.
* | distbuild: Clarify InitiatorConnection docstringSam Thursfield2014-04-141-3/+6
| |
* | distbuild: Label state machine transition tablesSam Thursfield2014-04-1410-0/+11
| | | | | | | | Makes it easier to see what they mean at a glance.
* | distbuild: worker-build failure does not indicate an error in the controllerSam Thursfield2014-04-141-1/+1
| |
* | distbuild: Log IPs of initiators and workers in more placesSam Thursfield2014-04-102-32/+31
| | | | | | | | This makes tracking multiple builds through one log file a bit easier.
* | distbuild: Improve logging of connections and objectsSam Thursfield2014-04-109-7/+93
| | | | | | | | | | | | | | | | | | | | New DistbuildSocket class that wraps socket.socket(), providing a descriptive repr() handler showing where the socket is connected, and providing a couple of helper methods for fetching local and remote endpoint names. This commit also adds a descriptive repr() handler to a few other objects (mostly giving socket connection details).
* | distbuild: Remove debug logging from serialise-artifact codeSam Thursfield2014-04-101-7/+0
| | | | | | | | | | We still log all messages sent to workers, which include the output of the serialise-artifact code in full. There's no need for these status messages.
* | distbuild: Disable extra debugging by defaultSam Thursfield2014-04-102-2/+2
|/
* Fix _maybe_abort in distbuildRichard Ipsum2014-03-281-1/+1
|
* Transfer all chunk artifacts after they are builtRichard Ipsum2014-03-261-10/+15
| | | | | We want to be able to transfer all source artifacts in a single transaction
* Fix build controllerRichard Ipsum2014-03-261-6/+34
| | | | | | | | | | Whenever the controller finds a source artifact it wants to build, it changes its state to BUILDING. We build all a chunk's source artifacts in one go. So for any chunk artifact, we change the state of all chunk artifacts that are built from the same source to BUILDING
* Make serialise work with artifact splittingRichard Ipsum2014-03-261-26/+118
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Serialisation was simple when we only had 1 artifact per source. However, to allow smaller systems, we need artifact splitting to produce multiple artifacts per chunk source. So now the new serialisation format has a separate list of artifacts and sources, rather than the Source being generated from the artifact's serialisation. Python's id() function is used to encode the references between the various Sources and Artifacts, these are replaced with a reference to the new object after deserialisation. Previously the cache-key was used, but this is no longer sufficient to uniquely identify an Artifact. The resultant build graph after deserialisation is a little different to what went in: Strata end up with a different Source per Artifact, so it _is_ a 1 to 1 mapping, as opposed to Chunks, where it's many to 1. We serialise strata and chunks differently because stratum artifacts from the same source can have different dependencies, for example core-devel can have different dependencies to core-runtime. Without intervention we would serialise core-devel and core-devel's dependencies without including core-runtime's dependencies. To solve this we've decided to encode stratum artifacts completely indepedently: each stratum artifact has its own source. This is safe because stratum artifacts can be constructed independently, as opposed to Chunks where all the Artifacts for a Source are produced together. This is a little hacky in its current form, but it simplifies matters later in distbuild with regards to how it handles expressing that every Artifact that shares a Source is built together. Arguably, this should be the output of producing the build graph anyway, since it more helpfully represents which Artifacts are built together than checking the morphology kind all the time, but more assumptions need checking in morph before it's safe to make this change across the whole of the morph codebase.
* Don't print worker in initiatorRichard Ipsum2014-03-261-2/+0
| | | | Worker name is not sent in message
* Add the distbuild libsRichard Ipsum2014-03-2128-0/+3886
View Lorry Controller Status