summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* 6. Write lots of output to the log filessam/distbuild-build-logs-scienceSam Thursfield2015-02-172-1/+7
| | | | | THIS CAUSES THE CROSSED WIRES BUG TO OCCUR!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!1
* 5: build once system, while repeatedly starting and cancelling anotherSam Thursfield2015-02-171-11/+21
| | | | No errors.
* fixup! distbuild: Make JsonRouter code less genericSam Thursfield2015-02-171-2/+2
|
* distbuild: Fix logged exception when an initiator disconnectsSam Thursfield2015-02-171-2/+2
| | | | | | | | | | | | | | | | | IDs are strings, but we were using '%d' to format IDs. The following exception would appear in the controller log file. Traceback (most recent call last): File "/usr/lib/python2.7/logging/__init__.py", line 859, in emit msg = self.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 732, in format return fmt.format(record) File "/usr/lib/python2.7/logging/__init__.py", line 471, in format record.message = record.getMessage() File "/usr/lib/python2.7/logging/__init__.py", line 335, in getMessage msg = msg % self.args TypeError: %d format: a number is required, not str Logged from file build_controller.py, line 446
* distbuild: Use clearer IDs in JsonRouterSam Thursfield2015-02-171-1/+1
| | | | | The JsonRouter creates unique IDs for the messages. In practice, this is unnecessary as there is only ever one controller process connected
* distbuild: Make JsonRouter code less genericSam Thursfield2015-02-171-37/+81
| | | | | | | | | | | The JsonRouter object is actually only used in the `morph worker-daemon` process. I find it hard to understand what's going on in this class because the names are generic (e.g. 'client' could refer to anything, but in fact the controller-daemon is the only 'client' there will ever be). This commit adds docstrings and cleans up some function and variable names to hopefully make the code easier to reason about.
* distbuild: remove old routes from route map in worker-daemonSam Thursfield2015-02-171-0/+11
| | | | | | | | | Also, warn if process or response messages are received for requests that have already completed or been cancelled. This won't gain much performance-wise, and I don't think that this kind of error is taking place at the moment, but should make it easier to spot if things aren't working as expected in future.
* 4: build 2 different systems and two different commitsSam Thursfield2015-02-171-2/+11
| | | | No crossed wires.
* Ensure writing to cache .pickle files is atomicSam Thursfield2015-02-171-1/+1
| | | | | | | | Otherwise, if one Morph process reads from the file while another Morph process writes to the file, the reader will see a corrupted file and will raise ValueError('insecure string pickle'). This is much less likely to happen if morphlib.savefile.SaveFile is used, because the data is updated in a single 'move' operation.
* 3: sharing artifacts, 30 seconds between initiators, and 4 of themSam Thursfield2015-02-171-2/+2
| | | | Still no crossed wires.
* scripts/distbuild: Fix artifacts not being sharedSam Thursfield2015-02-171-0/+1
| | | | | | | | | We need to pass --artifact-cache-server as well as --writable-cache-server to the `morph controller-daemon` process, otherwise it uses git.baserock.org as the shared artifact cache. Using git.baserock.org results in each build starting from scratch instead of reusing the already-build artifacts in the artifact cache set up by the test harness.
* 2: 10 seconds between initiators starting. No errors.Sam Thursfield2015-02-171-1/+2
|
* 1. Two initiators at the same time, 1 line of logs.Sam Thursfield2015-02-171-16/+19
| | | | No errors.
* squash! WIP: remove artifact serialisation from distbuildSam Thursfield2015-02-171-1/+1
|
* squash! WIP: remove artifact serialisation from distbuildSam Thursfield2015-02-163-23/+25
| | | | | | Conflicts: morphlib/buildcommand.py morphlib/plugins/distbuild_plugin.py
* WIP: remove artifact serialisation from distbuildSam Thursfield2015-02-164-20/+78
| | | | | Conflicts: morphlib/plugins/distbuild_plugin.py
* TEMPSam Thursfield2015-02-162-16/+70
|
* scripts/distbuild: Ensure process names are uniqueSam Thursfield2015-02-161-0/+3
| | | | | Previously, duplicate names would cause the ProcessMonitor class to forget about processes, and leave them running after the process exits.
* scripts/distbuild: Capture stderr of morph-cache-server processesSam Thursfield2015-02-161-1/+4
| | | | | All requests are written to stderr (and not to the log file) by Bottle, so it's useful to capture this.
* scripts/distbuild: Add the ability to manually start a processSam Thursfield2015-02-161-5/+20
| | | | | This allows you to run one of the processes under an interactive debugger.
* distbuild: Add a timeout to HTTP requestsSam Thursfield2015-02-161-1/+1
| | | | | | | | It's better to fail with an error after 10 seconds than to get stuck forever. This code path is used for telling the shared artifact cache server to fetch artifacts from a worker, and previously if the HTTP request didn't get a response for whatever reason, the build would hang until the end of time.
* morph-cache-server: Log errors as errors, not debug messagesSam Thursfield2015-02-161-8/+8
|
* Merge branch 'sam/distbuild-test-harness' into sam/distbuild-build-logsSam Thursfield2015-02-162-132/+164
|\ | | | | | | | | Conflicts: scripts/distbuild
| * Use distbuild test harness in the Yarn distbuild scenariosam/distbuild-test-harnessSam Thursfield2015-02-133-125/+25
| |
| * Add script to run a distbuild network on the local machineSam Thursfield2015-02-131-0/+402
| | | | | | | | | | | | | | This extends the work Richard Maw did on getting distbuild to run locally. It's now possible to interact with the local distbuild network, where before it could only be used as part of Yarn tests. It's also easy to have more than a single worker now.
| * Merge branch 'sam/distbuild-errors'Sam Thursfield2015-02-134-71/+81
| |\ | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
* | | Use scripts/distbuild for the Yarn distbuild scenarioSam Thursfield2015-02-134-139/+101
| | |
* | | TEMP: stub buildsSam Thursfield2015-02-121-5/+2
| | |
* | | Fix mistake in distbuild artifact transferringSam Thursfield2015-02-121-1/+2
| | |
* | | Lots of changes to distbuild test harnessSam Thursfield2015-02-121-123/+234
| | |
* | | Add initial distbuild test harnessSam Thursfield2015-02-111-0/+197
| | |
* | | Merge branch 'sam/fix-fetch-errors' into sam/distbuild-build-logsSam Thursfield2015-02-1119-533/+609
|\ \ \ | | | | | | | | | | | | | | | | Conflicts: without-test-modules
| * | | Add test_utils module (oops)Sam Thursfield2015-02-111-0/+57
| | | |
| * | | remoteartifactcache: Avoid crashing on connection errorsSam Thursfield2015-02-111-4/+4
| | | |
| * | | Add a timeout to remote artifact cache connectionsSam Thursfield2015-02-112-14/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This means that builds won't hang forever if network connectivity drops. Instead, Morph will show an error to the user, then try to do a local build of the current artifact. (This may still break if the local git cache does not contain the necessary SHA1s).
| * | | Retry requests to the remote cache server 3 times before failingSam Thursfield2015-02-112-24/+62
| | | | | | | | | | | | | | | | | | | | This makes it less likely that a build running overnight will fail due to network gremlins.
| * | | yarns: Explicitly specify there is no remote cache serverSam Thursfield2015-02-111-0/+1
| | | | | | | | | | | | | | | | | | | | Previously the value of trove-host would be used, resulting in requests to http://[]:8080/ that are a bit hard to reason about.
| * | | Fix to source.files()Sam Thursfield2015-02-111-1/+1
| | | |
| * | | Fix the really cheesy progress meter for fetching artifactsSam Thursfield2015-02-111-8/+10
| | | |
| * | | builder: Fix some places that assume there is always a remote cacheSam Thursfield2015-02-111-7/+11
| | | |
| * | | buildcommand: Think per-source not per-artifactSam Thursfield2015-02-111-11/+12
| | | |
| * | | Remove an unnecessary caching stepSam Thursfield2015-02-111-8/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | The BuildCommand code walks the build graph and fetches or builds each artifact in order. So there's no need to fetch the dependencies of something once again when we come to building it: everything before it in the build graph has already been fetched or built.
| * | | distbuild: Use source.files() to work out what artifacts to transferSam Thursfield2015-02-111-16/+6
| | | |
| * | | WhateverSam Thursfield2015-02-114-28/+84
| | | |
| * | | WIPOSam Thursfield2015-02-115-16/+25
| | | |
| * | | Rework artifact fetchingSam Thursfield2015-02-117-88/+72
| | | | | | | | | | | | | | | | | | | | We now fetch sources, not artifacts. This means that built artifacts of a source are either all available, or all missing.
| * | | Use Requests instead of urllib2 for remote HTTP requestsSam Thursfield2015-02-115-31/+39
| | | | | | | | | | | | | | | | Requests is a much better library for this purpose.
| * | | check: Only check for lines > 79 characters in .py filesSam Thursfield2015-02-111-4/+2
| | | |
| * | | Rewrite tests for remoteartifactcache moduleSam Thursfield2015-02-115-231/+198
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The old test suite was closely tied with the implementation of the module, which makes it very hard to rewrite any of the module. The new test suite actually sets up a remote HTTP server and exercises the real code paths. I added a new 'testutils' module, because there seem to be quite a few mock classes duplicated in different test modules. Better to collect them when they are really needed.
| * | | Consolidate code for fetching artifacts from remote cacheSam Thursfield2015-02-113-132/+90
| | | |
View Lorry Controller Status