summaryrefslogtreecommitdiff
path: root/distbuild
Commit message (Collapse)AuthorAgeFilesLines
* Fix lines longer than 79 charactersSam Thursfield2014-10-291-1/+2
|
* Fix distbuild to allow passing a commit instead of a named ref to be builtSam Thursfield2014-10-273-6/+21
| | | | | | | | | | | The recent changes to the BuildCommand.build() function caused distbuild to break, because I didn't make the same change to the InitiatorBuildCommand.build() function but did change how it was called. This commit adds the ability to have optional fields in distbuild messages. This is used to add an optional 'original_ref' field, which will get passed to `morph serialise-artifact` by new distbuild controllers, and will be ignored by older ones.
* distbuild: serialize dependent sources of graphRichard Maw2014-10-082-24/+46
|
* Allow distbuilds to choose where to put logsRichard Maw2014-10-081-1/+7
|
* distbuild: yaml-encode messages before json encodingRichard Maw2014-10-081-2/+8
| | | | | | JSON can only handle unicode strings, but commands can write anything to stdout/stderr, so we do the same trick as for the serialise, and json encode yaml.
* Allow distbuilding morphologies with binary data embeddedRichard Maw2014-10-081-9/+10
| | | | | | The horrible json.dumped, yaml dump is because we need it to be both binary safe (which yaml gives us) and one line per message (which json gives us).
* Fix issues with distbuild caused by moving to building per-sourceRichard Maw2014-10-084-22/+26
|
* Allow ephemeral ports for distbuild servicesRichard Maw2014-10-021-1/+1
|
* distbuild: allow daemons to bind to ephemeral portsRichard Maw2014-10-011-1/+6
| | | | | | | | | | | | | You can bind to an ephemeral port by passing 0 as the port number. To work out which port you actually got, you need to call getsockname(). To facilitate being able to spawn multiple copies of the daemons for testing environments, you can pass a -file option, which will make the daemon write which port it actually bound to. If this path is a fifo, reading from it in the spawner process will allow synchronisation of only spawning services that require that port to be ready after it is.
* Fix and integrate distbuild unit testsRichard Maw2014-10-012-127/+113
|
* Fix copyright years of distbuild code.Sam Thursfield2014-09-1129-29/+29
|
* Fix all distbuild code to be GPLv2 licensed.Sam Thursfield2014-09-1011-21/+141
|
* Rename for consistencyRichard Ipsum2014-08-141-10/+9
|
* Remove dead functionsRichard Ipsum2014-08-141-20/+0
|
* Revert distbuild parts of "Make our use of json binary path safe"baserock/richardmaw/bugfix/stop-decoding-distbuild-commsRichard Maw2014-07-183-6/+6
| | | | | | | | | | | | | | | | | | | | | | | The "unicode fix" worked for the subset of cases relevant, and only broke distbuild because its tests have not been integrated with ./check, so the fact that it broke for any string ending with a \ escaped notice, if you will excuse the pun. During json.load, the encode option is for specifying the character encoding of the file or string that is being loaded. During json.dump, the encode option is for the encoding of `str` keys and values. The fact that it worked for the set of cases we cared about is a small mystery, probably caused by the strings we happened to give it being valid unicode-escape encoded `str`ings. A full fix would require either converting all these cases to a different format, such as YAML, which will handle input data not being valid Unicode, or pre-processing the data that is passed to `json.dump` to convert all `str` instances to an appropriately escaped `unicode`, and converting back on `json.load`, but this is a quick fix to get the distbuild code working again.
* Fix JSON file handling to be binary filename safeLars Wirzenius2014-07-153-6/+6
|\ | | | | | | | | Reviewed-by: Lars Wirzenius Reviewed-by: Pedro Alvarez
| * Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-113-6/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* | Log the address we attempt to bindRichard Ipsum2014-07-111-0/+1
|/ | | | | This will make it easier to determine what is wrong if the controller daemon is run with a bad controller host address.
* distbuild: Log a message when a socket error is receivedSam Thursfield2014-06-191-0/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I found an issue in distbuild where the controller was stuck in a busy loop where it was continually writing to a closed socket. With 'strace' I saw write(), SIGPIPE, write(), SIGPIPE, ad infinitum. I got this much of a Python backtrace using GDB: distbuild.socketsrc.SocketEventSource.write() distbuild.sockbuf.SocketBuffer._flush() distbuild.sm.StateMachine.handle_event() I didn't manage to get further. However, I suspect one of the state machine transitions may be creating an event loop instead of correctly handling the error. The log file was quiet at this point, the last entries were: 2014-06-19 08:57:36 INFO There seems to be nothing to build 2014-06-19 08:57:36 INFO Requested artifact is built 2014-06-19 08:57:36 DEBUG InitiatorConnection: sent to 10.24.1.215:53818: {'mess age': 'Need to build 0 artifacts', 'type': 'build-progress', 'id': 790629564} 2014-06-19 08:57:36 DEBUG Notifying initiator of successful build 2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <BuildController at 0xb 6c554c, request-id InitiatorConnection-93> 2014-06-19 08:57:36 DEBUG InitiatorConnection: sent to 10.24.1.215:53818: {'type ': 'build-finished', 'id': 790629564, 'urls': [u'http://hawkdevtrove:8080/1.0/ar tifacts?filename=861f640923494ca3626bbd65655b350ce1bebea4c0bf7a57693bc06ed122cef 4.system.devel-system-x86_32-chroot-rootfs']} 2014-06-19 08:57:36 DEBUG InitiatorConnection: 10.24.1.215:53818: closing: <Json Machine at 0xc6cb22c: socket 10.24.1.164:7878 -> 10.24.1.215:53818, max_buffer 1 6384> 2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <InitiatorConnection at 0xc6cbcec: remote 10.24.1.215:53818> 2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <JsonMachine at 0xc6cb22c: socket 10.24.1.164:7878 -> 10.24.1.215:53818, max_buffer 16384> 2014-06-19 08:57:36 DEBUG MainLoop.remove_state_machine: <SocketBuffer at 0xc6cbe2c: socket None max_buffer 16384> This commit should improve matters a little: in future the log file will show the ID of the SocketEventSource object and error we hit when calling its write() function.
* Import InitiatorConnectionMachineRichard Ipsum2014-06-111-1/+2
|
* Use superRichard Ipsum2014-06-111-1/+1
| | | | This change is made just for consistency.
* Add InitiatorConnectionMachineRichard Ipsum2014-06-111-5/+36
| | | | | | The InitiatorConnectionMachine wraps the ConnectionMachine, so we can continue to use ConnectionMachine without providing it with an app.
* Add max_retries optionRichard Ipsum2014-06-111-5/+14
| | | | | | | By default there is no limit on the number of reconnection attempts. We make the reconnect_interval a parameter, but the default interval remains 1 second.
* Fix: force select to wait on timeoutRichard Ipsum2014-06-101-0/+3
| | | | | | | | | | | | | Currently the mainloop unintentionally busy-waits if we can't connect to the controller. We want the mainloop's select call to wait for the timeout not for this socket's descriptors (which are always ready). We could just call stop_reading() and stop_writing() but since we won't be needing this socket again we may as well close the entire event source, which calls stop_reading(), stop_writing() and then closes the socket.
* Note future improvement for fetching artifacts from remote cacheSam Thursfield2014-06-101-0/+3
|
* Move presence check for job into remove functionRichard Ipsum2014-06-041-6/+6
| | | | We always want to warn if we attempt to remove a job that's not present
* Add comment to explain the use of _JobFailed eventRichard Ipsum2014-06-041-0/+6
|
* Merge branch 'baserock/richardipsum/distbuild_state_fix'Richard Ipsum2014-06-041-23/+28
|\ | | | | | | | | | | | | Reviewed by: Sam Thursfield Adam Coldrick Richard Maw
| * Make Jobs finish when caching is completeRichard Ipsum2014-06-031-24/+24
| | | | | | | | | | | | | | | | | | | | If a new build request makes a request for an artifact that is currently being cached then the artifact will be needlessly rebuilt. To avoid this the new build request should wait for caching to finish. We rename _ExecStarted, _ExecEnded, _ExecFailed to _JobStarted, _JobFinished, _JobFailed and Job's is_building attribute is renamed to running.
| * Make job fail if caching failsRichard Ipsum2014-06-031-0/+5
| | | | | | | | | | This fixes the bug that causes the distbuild controller to crash when population of the artifact cache fails.
* | Improve error message in _notify_annotation_failedRichard Ipsum2014-06-041-4/+6
|/
* Merge remote-tracking branch 'origin/sam/distbuild-logs-2'Sam Thursfield2014-05-141-0/+1
|\ | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * distbuild: Include .build-log when copying chunk artifacts to the TroveSam Thursfield2014-05-141-0/+1
| | | | | | | | | | Users need to be able to see logs of all builds, not just those that failed.
* | Make distbuild put worker logs onto stdoutRichard Ipsum2014-05-141-0/+1
|/
* Add _ExecFailed eventRichard Ipsum2014-05-061-4/+24
| | | | To cancel jobs cleanly we need to know when a job has failed.
* Use messages to update job stateRichard Ipsum2014-05-061-3/+24
|
* Add cancelling to WorkerBuildSchedulerRichard Ipsum2014-05-061-13/+93
|
* Remove unused import and methodRichard Ipsum2014-05-061-3/+0
| | | | add_initiator() isn't necessary given lists have a remove method.
* Rename vars in dequeue_eventsRichard Ipsum2014-05-061-2/+2
|
* Add cancelling to build controllerRichard Ipsum2014-05-051-7/+19
|
* Remove route mapRichard Ipsum2014-04-241-1/+0
|
* Merge branch 'baserock/richardipsum/distbuild_jobs'Richard Ipsum2014-04-246-103/+262
|\ | | | | | | | | | | | | Reviewed by: Sam Thursfield Richard Maw Lars Wirzenius
| * Import all the thingsRichard Ipsum2014-04-231-1/+4
| |
| * Initiator: Handle step-already-started messageRichard Ipsum2014-04-231-0/+7
| |
| * InitiatorConnection: Handle _step_already_startedRichard Ipsum2014-04-231-1/+21
| |
| * Check whether request id is in the event's id listRichard Ipsum2014-04-231-6/+8
| | | | | | | | | | | | The contents of the message has changed for several events, event messages that need to be sent to several initiators have a list of ids instead of a single id.
| * Add handling for new messages to BuildControllerRichard Ipsum2014-04-231-1/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two new messages: WorkerBuildStepAlreadyStarted tells the initiator that the artifact they want to build is already being built, e.g. 'eglibc-misc is already building on 172.17.1.37:3434' WorkerBuildWaiting tells the initiator that the artifact they want to build can't be built yet because there aren't any workers free, e.g. 'Ready to build eglibc-misc: waiting for a worker to become available'
| * Add new step-already-started message to protocolRichard Ipsum2014-04-231-0/+5
| |
| * WorkerConnection: _maybe_handle_helper_resultRichard Ipsum2014-04-231-9/+4
| | | | | | | | | | | | | | Put our _exec_response_msg into WorkerBuildFinished event, it's essentially the same as _finished_msg, just a different name Get our artifact's cache key from the job
| * WorkerConnection: _request_cachingRichard Ipsum2014-04-231-9/+6
| | | | | | | | Now we just get everything from the job object
View Lorry Controller Status