summaryrefslogtreecommitdiff
path: root/morphlib/buildcommand.py
Commit message (Collapse)AuthorAgeFilesLines
* Honour --quietRichard Ipsum2016-04-251-2/+3
| | | | Change-Id: Ifec7f4274079d9c8f2a75bdedc8a253568a6ec78
* Add support for definitions version 8Pedro Alvarez2016-03-291-12/+21
| | | | | | | | | This code is a rework from changes done by: - Tiago Gomes <tiago.gomes@codethink.co.uk> https://storyboard.baserock.org/#!/story/86 Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
* Unify local and remote repo cache modulesSam Thursfield2016-03-161-11/+7
| | | | | | | | | | | | | | | | | | | | | There's not really any reason you'd want to use the RemoteRepoCache class except as a workaround for the slow speed of some LocalRepoCache operations, so I can't see this ruining anyone's day. The main reason for doing this is so we can simply the sourceresolver code. One reason that the sourceresolver class is so hopelessly complicated is that it right now has to use two incompatible interfaces for Git repo caches. I've taken the opportunity to detangle the RepoCache class from the App class. Now all of the configuration for the RepoCache class is passed into the constructor explicitly. This makes the class usable from outside Morph: resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[]) repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver) Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
* Remove old cruftTiago Gomes2015-11-201-2/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Use app.status() to log ldconfig messagesTiago Gomes2015-09-281-1/+1
| | | | | | | Using app.status() produces nicer logs as the messages will have the status prefix prepended. Change-Id: I0573d7066784afdbfb878bed85af704e82bb9797
* Display progress bar when fetching to local cacheRichard Ipsum2015-09-231-3/+31
| | | | | | | | | Looks like, 2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc stage1-binutils-misc[##################### ] 51.9/73.0 MB Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
* Simplify StagingArea classTiago Gomes2015-09-221-5/+6
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Stop moving staging areas of failed buildsTiago Gomes2015-09-011-5/+1
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Fix prototype of create_source_pool()Sam Thursfield2015-05-141-7/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* Explain how to cancel a distbuildAdam Coldrick2015-05-121-0/+5
| | | | | | | | | | | | | | | | Cancelling a distbuild with ctrl+c no longer cancels the build itself. This commit adds some output explaining what should be done to cancel the build as well as the local process. This commit also fixes a bug where the BuildStarted event would be sent each time a chunk finished building, since it was being sent in _queue_worker_builds. This is fixed by adding a new function to be called when the build graph annotation is complete which sends BuildStarted and then calls _queue_worker_builds, which no longer sends the BuildStarted event. Change-Id: I26ddea2c9080887f449e87004411ddffe4e583b7
* SourceResolver: Allow the resolution of multiple systemsRichard Maw2015-05-111-3/+6
| | | | | | | | | | | | | | | | The existing Source resolution code handles resolution of multiple systems sufficiently. It is not appropriate to then take this source pool and attempt to create a build graph from it though, as the logical structure of the input of what we want to build, and the logical structure of what we will produce are conflated in the Source object. If we do not intend to create a build graph from the Source Pool we generate, then it is an appropriate data structure that may be used to analyse changes in the input to a build. Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
* distbuild: Add distbuild start and cancel functionalityLauren Perry2015-04-291-2/+8
| | | | | | | | | | | Add command for distbuild-start to build_plugin in morphlib, and create a boolean parameter to inform the initiator whether to disconnect the controller and leave the build running remotely. Add distbuild-cancel command to parse currently-running distbuild build-request IDs and cancel the one matching the given argument Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
* Add armv5l to list of valid architecturesJavier Jardón2015-04-281-0/+6
| | | | | | | | Also add support to allow building compatible architectures on armv7 machines, as per the rationale in the comment in _validate_architecture(). Change-Id: Ie0fe4002523f1f92f576bac8b654d4ea6fad5cf2
* Implement partial distbuildsAdam Coldrick2015-04-021-2/+4
| | | | | | | | | | | | | | In addition to partial builds we also want to be able to do partial distbuilds, and distbuild uses a different codepath. This commit updates the distbuild code to know what to do if a partial build is requested. It only builds up to the latest chunk/stratum that was requested, and displays where to find the artifacts for each of the chunks/strata requested upon completion of the build. The usage is the same as for local builds. Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
* Improve build starting outputAdam Coldrick2015-04-011-1/+2
| | | | Change-Id: Ica2a270b254b01b9fb15956556efb74ee641e1b4
* Implement partial buildsAdam Coldrick2015-04-011-1/+4
| | | | | | | | | | | | | | | | | | This commit allows you to build only a single chunk or stratum and its dependencies, instead of a whole system. This will be useful for developers who want to quickly check the build commands of a chunk for example. You can give more than one chunk or stratum to the command, and the build will run up to the latest one, building the dependencies of both. An example of the usage: morph build systems/devel-system-x86_64-generic.morph \ strata/build-essential.morph strata/core/curl.morph This will only build up to curl in the core stratum. Change-Id: Id55b8335fc61e8fdf847f5e383cfedd13d07e6d2
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Merge branch 'sam/distbuild-build-logs'Sam Thursfield2015-03-111-1/+9
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * distbuild: Log in build-step-xx.log files when initiator cancels buildSam Thursfield2015-02-181-1/+9
| | | | | | | | | | This makes it easier to spot if an incomplete build was due to the user cancelling, or if it represents a dropped connection or internal error.
* | Merge branch 'sam/fewer-submodule-updates'Sam Thursfield2015-03-031-33/+2
|\ \ | | | | | | | | | | | | Reviewed-By: Javier Jardón <javier.jardon@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | Only update Git submodules in cache when necessarySam Thursfield2015-02-101-33/+2
| |/ | | | | | | | | | | | | | | | | | | | | | | | | | | This saves a duplicate `git remote update origin` that was being run as part of each chunk build. For any repos that have submodules, it also avoids updating repos if the SHA1 we need to build is already present locally. As well as speeding up builds slightly, this means Morph can now build without being connected to a network, as long as the local Git cache all of the necessary repos and commits in the build, without needing the '--no-git-update' option. The code is also now in a more logical place than before.
* | Show a message when the build is startingAdam Coldrick2015-02-181-1/+1
| |
* | Add an LRU cache for resolved tree refsSam Thursfield2015-02-181-0/+1
|/ | | | | | | | | | | | This uses the PyLRU module, from: <https://pypi.python.org/pypi/pylru/1.0.6>. Python 3.2 and newer provide a built-in LRU cache, but this is specifically for in-memory use. See <http://bugs.python.org/issue17528>. Git commits are immutable, so caching information about their contents is fairly easy and trouble-free. There's no danger of the cache becoming stale.
* Merge branch 'sam/cached-repo-cleanup'Sam Thursfield2015-01-121-2/+2
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-031-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
* | Merge remote-tracking branch 'origin/sam/rename-builder2'Sam Thursfield2015-01-051-3/+3
|\ \ | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
| * | Rename builder2 module to builderSam Thursfield2014-12-191-2/+2
| |/
* | Support armv8l, armv8b, aarch64, and aarch64bEmmet Hikory2014-12-191-5/+15
|/ | | | | | Also add support to allow building compatible architectures on armv8 machines, as per the rationale in the comment in _validate_architecture().
* Move create_source_pool code into new 'sourceresolver' moduleSam Thursfield2014-11-101-3/+4
| | | | | | | | This code is an essential part of 'morph build'. It's quite complex and really shouldn't be mixed in with the base Application class. Given a dedicated class we can store some state in the object and avoid functions with seven parameters, too.
* Fix distbuild to allow passing a commit instead of a named ref to be builtSam Thursfield2014-10-271-5/+2
| | | | | | | | | | | The recent changes to the BuildCommand.build() function caused distbuild to break, because I didn't make the same change to the InitiatorBuildCommand.build() function but did change how it was called. This commit adds the ability to have optional fields in distbuild messages. This is used to add an optional 'original_ref' field, which will get passed to `morph serialise-artifact` by new distbuild controllers, and will be ignored by older ones.
* build: Allow passing in the name of the original ref to create_source_pool()Sam Thursfield2014-10-241-4/+6
| | | | | This means that we can force the building of a specific commit without losing the original branch name in the metadata of the resulting system.
* build: Restructure how BuildCommand.build() is calledSam Thursfield2014-10-241-12/+14
| | | | | | | | | Rather than take a list of triplets to build, the BuildCommand.build() function now takes a single repo/ref/morph triplet. Iterating through multiple sets of triplets is now done in the build plugin. There are a couple of cosmetic changes to the status output at the start and end of a build as a result.
* Give clearer error when the same 'name' is used by two strataSam Thursfield2014-10-161-5/+4
|
* Don't print the ref of each stratum in the buildSam Thursfield2014-10-161-3/+0
| | | | | | | There's no need because all strata live in the same Git repository now. This message was added back when there could be multiple versions of strata coming from different repos or different refs in the same repo and life was very confusing.
* Merge remote-tracking branch 'origin/baserock/ps/log-chunk-build-times'Richard Maw2014-10-091-0/+8
|\ | | | | | | | | Reviewed-by: Pedro Alvarez Reviewed-by: Richard Maw
| * Log elapsed time for each chunk buildbaserock/ps/log-chunk-build-timesPaul Sherwood2014-09-271-0/+7
| |
* | Tidy up artifact resolvingRichard Ipsum2014-10-091-33/+19
|/ | | | | | | | This patch started off as an attempt to address the comment in find_root_artifacts, though this patch doesn't impose an ordering on the list of artifacts as the comment suggested. The artifact resolver now returns a list of root artifacts which may make it easier to add multi-system builds to morph.
* Remove Artifact compatibility methodsRichard Maw2014-09-191-4/+4
|
* Build per-source rather than per-artifactRichard Maw2014-09-191-62/+97
|
* Create multiple sources per stratum morphologyRichard Maw2014-09-191-13/+13
| | | | | | | | | | | | | | | | | Building per-artifact results in undesirable behaviour, as multiple artifacts are produced for every chunk build. It therefore makes more sense to build per-source. This implies that actually, the model of one source per morphology is wrong and we should move the dependencies into the source. Unlike chunks however, where every chunk artifact has the same dependencies, stratum artifacts can have different dependencies. So before we can move the dependencies into the Source, we need to have as many Sources as Stratum Artifacts.
* Rename BuildCommand.get_sources to fetch_sourcesRichard Maw2014-09-191-2/+2
| | | | | There's other methods called get_sources in other modules, and fetch_sources explains more about what it does in the context.
* Rewrite traverse_morphs to be able to load chunk morphologies from definitionsRichard Maw2014-07-221-1/+1
| | | | | | | | | | | | | | It will now load the morphology from the definitions repository if the "morph" field is present in the chunk spec. Rather than adapting the loop to fit yet-more changing circumstances, it has been partially rewritten, so there is one loop for loading systems and strata from definitions.git, another for chunks from definitions.git, and a third for chunks in the source repository. This is tidier than attempting to fit the logic in the main loop, as it makes it easier to remove afterwards when we no longer need to load chunk morphologies from the source repository.
* Don't set 'morph' field by default, and don't expect to find itAdam Coldrick2014-07-221-1/+2
|
* Use exact filenames to refer to morphology filesRichard Maw2014-07-101-1/+1
| | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
* Make buildcommand use InitiatorConnectionMachineRichard Ipsum2014-06-111-2/+11
| | | | | | After a failed attempt to connect to the controller node the initiator will wait 30 seconds before attempting a reconnect, if this reconnect fails the initiator gives up.
* Note future improvement for fetching artifacts from remote cacheSam Thursfield2014-06-101-0/+4
|
* Fix up sam/remote-artifact-cache-failure branch after reviewSam Thursfield2014-06-101-3/+10
|
* Ensure that transferring an artifact from the remote cache is atomicSam Thursfield2014-06-031-15/+29
| | | | | | | | | | Artifacts can have multiple parts; while this may not be an ideal design, changing the format of artifacts has implications for backwards compatibility. We should transfer all parts at once and delete them all if we encounter any errors, to reduce the change of getting the local artifact cache into an inconsistent state.
* Improve robustness when fetching artifacts from remote artifact cacheSam Thursfield2014-06-031-18/+16
| | | | | | | | | | | Previously Morph would check if an artifact is present in the remote artifact cache, then fetch the necessary files. If an error occured during fetching, it would raise an error and abort. Instead, we should just try and fetch the files and if anything fails, move on to building locally. This avoids the situation where an error in the remote cache makes local building impossible, which we experienced recently.
* Merge branch 'baserock/richardipsum/merge_distbuild'Richard Ipsum2014-03-271-0/+30
|\ | | | | | | | | | | | | | | | | | | Conflicts: morphlib/plugins/deploy_plugin.py without-test-modules Reviewed by: Richard Maw Lars Wirzenius
View Lorry Controller Status