summaryrefslogtreecommitdiff
path: root/distbuild/build_controller.py
Commit message (Collapse)AuthorAgeFilesLines
* Fix partial distbuilds of non-existant componentsAdam Coldrick2015-04-301-8/+9
| | | | | | | | | | Currently, attempting to distbuild a component which is not in the given system or doesn't exist at all will cause the full system to be built, rather than an error raised. This is because the logic which checks that all components were found is completely nonsensical. This commit makes it actually check the right thing. Change-Id: Ide4d7e3fa5f71e433f3a7b7c8c387fe594c92e43
* Remove n_state_machines_of_type functionRichard Ipsum2015-04-291-1/+2
| | | | Change-Id: Icfc3d1aa125196e208d7ac35f43f06c5f5a21ba4
* distbuild: Add distbuild status commandLauren Perry2015-04-291-0/+12
| | | | | | | | | Adds a command to get the status of all recently ran distbuilds for a given server (e.g. Running, Finished, Failed, Cancelled), so as to tell if a build running via distbuild-start has finished or otherwise exited without going through the server's log files Change-Id: I5ce9fe54ae7b1bd8fe3e0d629f615042be8827ed
* distbuild: Add distbuild start and cancel functionalityLauren Perry2015-04-291-1/+41
| | | | | | | | | | | Add command for distbuild-start to build_plugin in morphlib, and create a boolean parameter to inform the initiator whether to disconnect the controller and leave the build running remotely. Add distbuild-cancel command to parse currently-running distbuild build-request IDs and cancel the one matching the given argument Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
* Make distbuild use an ArtifactReference not an Artifact internally when buildingAdam Coldrick2015-04-241-15/+14
| | | | | | | | We no longer serialise entire artifacts, so the output of deserialise_artifact is an ArtifactReference. This commit changes stuff in distbuild to know how to deal with that rather than an Artifact. Change-Id: I79b40d041700a85c25980e3bd70cd34dedd2a113
* distbuild: Add distbuild-list-jobs functionLauren Perry2015-04-171-2/+8
| | | | | | | | | | Add InitiatorListJobs class and list-jobs message template, add distbuild-list-jobs to morph commandlist, send running job information back to initiator, split out handling of build request and list-jobs messages to separate functions and change generating a random integer to UUID for message identification Change-Id: Id02604f2c1201dbc10f6bbd7f501b8ce1ce0deae
* distbuild: Fix issues in build cancellationSam Thursfield2015-04-021-8/+13
| | | | | | | | | A cancel during the 'graphing' or 'annotating' stages would be ignored as the BuildController was listening for the InitiatorDisconnect message from the wrong event source. In 'building' state the actual build would be stopped, but the BuildController instance would stick around due to sending the message class instead of an instance of the message. Change-Id: I222a8aa39bf7fffab4d89e12997ffd18cd1b54fc
* Implement partial distbuildsAdam Coldrick2015-04-021-23/+85
| | | | | | | | | | | | | | In addition to partial builds we also want to be able to do partial distbuilds, and distbuild uses a different codepath. This commit updates the distbuild code to know what to do if a partial build is requested. It only builds up to the latest chunk/stratum that was requested, and displays where to find the artifacts for each of the chunks/strata requested upon completion of the build. The usage is the same as for local builds. Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-1/+1
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* distbuild: Remove the build-steps messageSam Thursfield2015-02-181-16/+4
| | | | | | | | | | | | | | | | | This message was hundreds of kilobytes in size, as it contained a recursive list of dependencies for each artifact in the build graph. It was used in the initiator only to print this message: Build steps in total: 592 This message is now gone. The 'Need to build %d artifacts' build-progress message now indicates the total build steps instead: Need to build 300 artifacts, of 592 total This is a compatible change to the distbuild protocol: old initiators will continue to work as normal with new controllers that don't send the build-steps message.
* distbuild: Use source name, not artifact name, for build step logsSam Thursfield2015-02-181-1/+1
| | | | | | | | Users build sources, not artifacts. So the log files should be called build-step-systemd.log and not build-step-systemd-misc.log. Note strata are a kind of special case so you will still see build-step-foundation-runtime.log, build-step-foundation-devel.log etc.
* Fix copyright yearsSam Thursfield2015-02-111-1/+1
|
* distbuild: Give more detail when requests to cache-server failSam Thursfield2015-02-111-2/+3
| | | | | | Let the end-user see the URL that distbuild was attempting to talk to, so they can more easily spot configuration errors. It's kind of silly to say 'HTTP request failed' without saying where the request was going.
* distbuild: Simplify error when computing build graph failsSam Thursfield2015-02-111-2/+1
| | | | | | | | | | | | | | | | | | The previous error looked like this by the time it had reached the initiator's console: ERROR: Failed to build baserock:baserock/definitions c7292b7c81cdd7e5b9e85722406371748453c44f systems/base-system-x86_64-generic.morph.frodsham: Failed to compute build graph. Problem with serialise-artifact: ERROR: Couldn't find morphology: systems/base-system-x86_64-generic.morph.frodsham New message is at least a bit simpler: ERROR: Failed to build baserock:baserock/definitions c7292b7c81cdd7e5b9e85722406371748453c44f systems/base-system-x86_64-generic.morph.frodsham: ERROR: Couldn't find morphology: systems/base-system-x86_64-generic.morph.frodsham
* distbuild: Fix case where 'computing build graph' would hang foreverSam Thursfield2015-02-111-0/+3
| | | | | | If there's no distbuild-helper process running on the controller then the controller would hang forever. This situation is unlikely, but it's important to give the user feedback instead of silently hanging forever.
* distbuild: Simplify failure cases in BuildControllerSam Thursfield2015-02-111-43/+19
| | | | | | | There's no need to handle failure differently at each stage of the build. Simpler to use the BuildFailed message for all errors. This then allows us to have a single self.fail() function that can be used everywhere.
* distbuild: Rearrange code that sends exec-request messageSam Thursfield2015-02-111-8/+13
|
* Fix distbuild to allow passing a commit instead of a named ref to be builtSam Thursfield2014-10-271-0/+2
| | | | | | | | | | | The recent changes to the BuildCommand.build() function caused distbuild to break, because I didn't make the same change to the InitiatorBuildCommand.build() function but did change how it was called. This commit adds the ability to have optional fields in distbuild messages. This is used to add an optional 'original_ref' field, which will get passed to `morph serialise-artifact` by new distbuild controllers, and will be ignored by older ones.
* Fix issues with distbuild caused by moving to building per-sourceRichard Maw2014-10-081-5/+6
|
* Fix copyright years of distbuild code.Sam Thursfield2014-09-111-1/+1
|
* Revert distbuild parts of "Make our use of json binary path safe"baserock/richardmaw/bugfix/stop-decoding-distbuild-commsRichard Maw2014-07-181-2/+2
| | | | | | | | | | | | | | | | | | | | | | | The "unicode fix" worked for the subset of cases relevant, and only broke distbuild because its tests have not been integrated with ./check, so the fact that it broke for any string ending with a \ escaped notice, if you will excuse the pun. During json.load, the encode option is for specifying the character encoding of the file or string that is being loaded. During json.dump, the encode option is for the encoding of `str` keys and values. The fact that it worked for the set of cases we cared about is a small mystery, probably caused by the strings we happened to give it being valid unicode-escape encoded `str`ings. A full fix would require either converting all these cases to a different format, such as YAML, which will handle input data not being valid Unicode, or pre-processing the data that is passed to `json.dump` to convert all `str` instances to an appropriately escaped `unicode`, and converting back on `json.load`, but this is a quick fix to get the distbuild code working again.
* Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-111-2/+2
| | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* Improve error message in _notify_annotation_failedRichard Ipsum2014-06-041-4/+6
|
* Add cancelling to build controllerRichard Ipsum2014-05-051-7/+19
|
* Merge branch 'baserock/richardipsum/distbuild_jobs'Richard Ipsum2014-04-241-7/+48
|\ | | | | | | | | | | | | Reviewed by: Sam Thursfield Richard Maw Lars Wirzenius
| * Check whether request id is in the event's id listRichard Ipsum2014-04-231-6/+8
| | | | | | | | | | | | The contents of the message has changed for several events, event messages that need to be sent to several initiators have a list of ids instead of a single id.
| * Add handling for new messages to BuildControllerRichard Ipsum2014-04-231-1/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two new messages: WorkerBuildStepAlreadyStarted tells the initiator that the artifact they want to build is already being built, e.g. 'eglibc-misc is already building on 172.17.1.37:3434' WorkerBuildWaiting tells the initiator that the artifact they want to build can't be built yet because there aren't any workers free, e.g. 'Ready to build eglibc-misc: waiting for a worker to become available'
* | Remove duplicated function _artifact_filenameRichard Ipsum2014-04-161-8/+2
|/
* Add flag for debugging state of build graphRichard Ipsum2014-04-161-8/+10
| | | | | | Most of the time knowing the state of the build graph isn't that useful for debugging distbuild, but it may be useful in some situations.
* Don't log a cache response till we're sureRichard Ipsum2014-04-161-1/+2
| | | | | | The controller should check the response event is actually the response it was waiting for before logging that there was a cache response
* Merge branch 'baserock/richardipsum/distbuild_improve_annotation3'Richard Ipsum2014-04-151-54/+71
|\ | | | | | | | | | | | | | | | | | | Conflicts: distbuild/build_controller.py Reviewed by: Lars Wirzenius Daniel Silverstone Sam Thursfield
| * Annotation using single post requestRichard Ipsum2014-04-111-53/+70
| | | | | | | | We now get the state of all artifacts with a single request.
* | distbuild: Consistently name all maybe_ message callbacksSam Thursfield2014-04-141-23/+23
| | | | | | | | | | | | | | | | | | There are cases where a state machine handles an event but stays in the same state. A callback is registered which filters messages further before possibly taking action. There have been bugs caused by this pattern being incorrectly implemented (where the callback is expected to filter the message, but a transition takes place anyway). Hopefully a consistent naming convention will make the pattern clearer.
* | distbuild: Make some transitions more specificSam Thursfield2014-04-141-22/+17
| | | | | | | | | | | | There is always one BuildController object per InitiatorConnection. By coupling the objects slightly closer we can simplify some transitions in BuildController.
* | distbuild: Fix possible 'one user disconnecting breaks all builds' issueSam Thursfield2014-04-141-2/+2
| | | | | | | | | | This is similar to the issue fixed by commit c38b77bed86acc8b90f253ce354f3ecf98e475e7.
* | distbuild: Fix 'one build failure causes all concurrent builds to hang'Sam Thursfield2014-04-141-8/+25
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Each BuildController instance sets itself up to receive all messages from all workers (via the WorkerConnection instances). In the case of a build failure, all BuildController objects would transition to 'None' state (causing them all to be destroyed) on any WorkerBuildFailed message. This meant that if any one build failed on a distbuild network: - the user whose build actually failed would receive the error messages correctly - any concurrent users would see no further build messages from the controller, making it look like their builds had hung. Ctrl+C from the 'hung' users would still be correctly handled by the controller, as their InitiatorConnection instance would still be alive.
* | distbuild: Label state machine transition tablesSam Thursfield2014-04-141-0/+1
| | | | | | | | Makes it easier to see what they mean at a glance.
* | distbuild: worker-build failure does not indicate an error in the controllerSam Thursfield2014-04-141-1/+1
| |
* | distbuild: Improve logging of connections and objectsSam Thursfield2014-04-101-0/+4
| | | | | | | | | | | | | | | | | | | | New DistbuildSocket class that wraps socket.socket(), providing a descriptive repr() handler showing where the socket is connected, and providing a couple of helper methods for fetching local and remote endpoint names. This commit also adds a descriptive repr() handler to a few other objects (mostly giving socket connection details).
* | distbuild: Disable extra debugging by defaultSam Thursfield2014-04-101-1/+1
|/
* Fix _maybe_abort in distbuildRichard Ipsum2014-03-281-1/+1
|
* Fix build controllerRichard Ipsum2014-03-261-6/+34
| | | | | | | | | | Whenever the controller finds a source artifact it wants to build, it changes its state to BUILDING. We build all a chunk's source artifacts in one go. So for any chunk artifact, we change the state of all chunk artifacts that are built from the same source to BUILDING
* Add the distbuild libsRichard Ipsum2014-03-211-0/+531
View Lorry Controller Status