summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Add command to validate the local artifact cachesam/validationSam Thursfield2014-05-083-10/+179
| | | | | | | | | | | | | | | This tests the checksums of artifacts and tests any unpacked chunks. The unpacked chunks are tested by turning them back into tarfiles and calculating a checksum, which we can rely on being identical to the chunk artifact in the artifact cache if there is no corruption. That is pretty cool! This code doesn't work because the zlib.adler32() code can't be used as a rolling checksum. I'm going to switch to using hashlib.md5() which should still be fast enough. Alternately we could go for sha256 right away, so that when we /do/ have bit-for-bit reproducibility from source, we can say that everything is totally secure.
* Fix LocalArtifactCache.list_contents() returning bad valuesSam Thursfield2014-05-081-2/+3
| | | | | | | | | | The OSFS.walkfiles() method returns file paths inside the cachedir, but LocalArtifactCache.list_contents() expected file names (with no trailing slash). The last digit of the cache key was also being removed. TODO: check if `morph gc` still works
* squash! Add 'morph validate' command to run 'git fsck' in all cached reposSam Thursfield2014-05-081-2/+4
|
* Just let me code and stop whining about long linesSam Thursfield2014-05-081-1/+1
|
* Fix crossbootstrap plugin for checksum changes to builder.pySam Thursfield2014-05-081-2/+4
|
* Write checksums alongside artifacts in cache.Sam Thursfield2014-05-083-15/+58
|
* Change cache_check_plugin a bitSam Thursfield2014-05-081-46/+0
|
* Make artifact creation more exception-safeSam Thursfield2014-05-081-16/+14
|
* Rework previous patchSam Thursfield2014-05-082-13/+11
|
* Bring code path for system artifact creation to join with that for chunk ↵Sam Thursfield2014-05-081-0/+10
| | | | artifacts
* Add 'morph validate' command to run 'git fsck' in all cached reposSam Thursfield2014-05-085-1/+154
|
* Merge branch 'baserock/richardipsum/distbuild_cancel2'Richard Ipsum2014-05-073-29/+158
|\ | | | | | | | | | | | | Reviewed by: Richard Maw Lars Wirzenius Daniel Silverstone
| * Add _ExecFailed eventRichard Ipsum2014-05-061-4/+24
| | | | | | | | To cancel jobs cleanly we need to know when a job has failed.
| * Use messages to update job stateRichard Ipsum2014-05-061-3/+24
| |
| * Add cancelling to WorkerBuildSchedulerRichard Ipsum2014-05-061-13/+93
| |
| * Remove unused import and methodRichard Ipsum2014-05-061-3/+0
| | | | | | | | add_initiator() isn't necessary given lists have a remove method.
| * Rename vars in dequeue_eventsRichard Ipsum2014-05-061-2/+2
| |
| * Add cancelling to build controllerRichard Ipsum2014-05-051-7/+19
| |
| * Remove route mapRichard Ipsum2014-04-241-1/+0
| |
* | Fix bad indentationRichard Ipsum2014-05-071-2/+2
| |
* | A more useful morphology syntax error messageRichard Ipsum2014-05-071-4/+3
| | | | | | | | | | There's no longer any need to log the exception separately, it will be logged with the MorphologySyntaxError
* | Merge branch 'baserock/richardipsum/improve_copyright_check'Richard Ipsum2014-04-282-15/+11
|\ \ | |/ |/| | | | | | | Reviewed by: Daniel Silverstone Richard Maw
| * Exit check as soon as we have an errorRichard Ipsum2014-04-251-6/+2
| |
| * Improve check-copyright-year error reportingRichard Ipsum2014-04-251-9/+9
|/ | | | Be more verbose and exit with an error not an exception.
* Merge branch 'baserock/richardipsum/distbuild_jobs'Richard Ipsum2014-04-246-103/+262
|\ | | | | | | | | | | | | Reviewed by: Sam Thursfield Richard Maw Lars Wirzenius
| * Import all the thingsRichard Ipsum2014-04-231-1/+4
| |
| * Initiator: Handle step-already-started messageRichard Ipsum2014-04-231-0/+7
| |
| * InitiatorConnection: Handle _step_already_startedRichard Ipsum2014-04-231-1/+21
| |
| * Check whether request id is in the event's id listRichard Ipsum2014-04-231-6/+8
| | | | | | | | | | | | The contents of the message has changed for several events, event messages that need to be sent to several initiators have a list of ids instead of a single id.
| * Add handling for new messages to BuildControllerRichard Ipsum2014-04-231-1/+40
| | | | | | | | | | | | | | | | | | | | | | | | | | | | There are two new messages: WorkerBuildStepAlreadyStarted tells the initiator that the artifact they want to build is already being built, e.g. 'eglibc-misc is already building on 172.17.1.37:3434' WorkerBuildWaiting tells the initiator that the artifact they want to build can't be built yet because there aren't any workers free, e.g. 'Ready to build eglibc-misc: waiting for a worker to become available'
| * Add new step-already-started message to protocolRichard Ipsum2014-04-231-0/+5
| |
| * WorkerConnection: _maybe_handle_helper_resultRichard Ipsum2014-04-231-9/+4
| | | | | | | | | | | | | | Put our _exec_response_msg into WorkerBuildFinished event, it's essentially the same as _finished_msg, just a different name Get our artifact's cache key from the job
| * WorkerConnection: _request_cachingRichard Ipsum2014-04-231-9/+6
| | | | | | | | Now we just get everything from the job object
| * WorkerConection: _handle_exec_responseRichard Ipsum2014-04-231-9/+8
| | | | | | | | | | | | | | | | | | | | | | The exec_response_msg also needs to be sent to a number of initiators, so we give it a list of ids not just one. The exec_response_msg will be sent to the controller once the artifacts have been cached successfully. There's no longer any need to use a route map to retrieve the id of the initiator, since this is stored with the job
| * WorkerConnection: _handle_exec_outputRichard Ipsum2014-04-231-2/+2
| | | | | | | | | | msg now contains a list of initiator ids rather than a single one, since BuiltOutput needs to be sent to a number of initiators
| * WorkerBuildQueuer: Use job's artifact and idRichard Ipsum2014-04-231-14/+17
| | | | | | | | | | | | Each job is given a unique id, so we don't need to generate an id for each exec request this means we can remove use of route map since we can use the job's id for the exec request
| * Remove cancelRichard Ipsum2014-04-231-6/+1
| | | | | | | | This method no longer works, we will replace it soon.
| * Change event names backRichard Ipsum2014-04-231-3/+3
| | | | | | | | | | The name change from BuildFailed -> JobFailed etc was unintentionally merged into master, undo this.
| * WorkerConnection: misc attributesRichard Ipsum2014-04-231-0/+6
| | | | | | | | | | | | _job is the job this worker is carrying out _exec_response_msg will contain the response the worker sends back to us when it finishes the build.
| * WorkerBuildQueuer: replace request queue with jobsRichard Ipsum2014-04-231-25/+67
| |
| * Add Jobs and Job classesRichard Ipsum2014-04-231-10/+49
| |
| * Make WorkerBuildCaching carry a list of idsRichard Ipsum2014-04-231-5/+2
| | | | | | | | We need to be able to send this message to a number of initiators
| * Add new build messages to worker build schedulerRichard Ipsum2014-04-231-2/+12
| |
* | Merge remote-tracking branch ↵Richard Maw2014-04-221-2/+2
|\ \ | | | | | | | | | | | | | | | | | | 'origin/baserock/ps/fix-virtualbox-version-checking' Reviewed-by: Richard Maw Reviewed-by: Pedro Alvarez
| * | fix the Virtualbox version checkingPaul Sherwood2014-04-201-2/+2
| | |
* | | Update copyright header in check-silliness scriptRichard Maw2014-04-221-1/+1
|/ / | | | | | | | | Reviewed-by: Pedro Alvarez Rubber-stamped-by: Richard Maw
* | Merge remote-tracking branch 'origin/baserock/richardmaw/style-checker'Richard Maw2014-04-172-5/+23
|\ \ | | | | | | | | | | | | Reviewed-by: Lars Wirzenius Reviewed-by: Daniel Silverstone
| * | style check: Include newly added files in checksRichard Maw2014-03-191-2/+3
| | |
| * | check-copyright-year: Use current date for uncommitted changesRichard Maw2014-03-191-3/+20
| | |
* | | Fix check-silliness' use of awk length functionLars Wirzenius2014-04-171-2/+2
| | | | | | | | | | | | | | | | | | | | | Posix took away using length without (). Reviewed-by: Richard Maw Reviewed-by: Daniel Silverstone
View Lorry Controller Status