summaryrefslogtreecommitdiff
path: root/morphlib/plugins
Commit message (Collapse)AuthorAgeFilesLines
* Add '--repo' and '--ref' options to cross-bootstrap commandPedro Alvarez2016-07-271-3/+19
| | | | Change-Id: I5d8c2f0979f086f6abeec4421dded440b5948706
* Add '--repo' and '--ref' options to [dist]build commandsPedro Alvarez2016-07-271-9/+61
| | | | Change-Id: I1f41d544a202aa3e54c5d6fba62a629dddcf6c1c
* Add support for '--repo' and '--ref' in list-artifactsbaserock/pedroalvarez/repo-ref-optionsPedro Alvarez2016-07-271-1/+26
| | | | Change-Id: I48cebf0007e602c0262b667643c3290d41c7b485
* Rename get-repo's option '--ref' to '--checkout-ref'Pedro Alvarez2016-07-261-1/+1
| | | | | | | The option '--ref' is going to be used for other things in the tool. Change-Id: Icbee89c4693ca53618b9c039120fbb0c6636285d
* Fix generate-manifest-csv options helpPedro Alvarez2016-07-261-1/+1
| | | | Change-Id: I3a87c82b0c50a648edd7c6487e0269116da35c20
* Make `morph list-artifacts` work from definitions checkoutPedro Alvarez2016-05-171-7/+33
| | | | Change-Id: I711953b829c786911c46e413d2b7af3427b7ba26
* Make `morph cross-bootstrap` work from definitions checkoutPedro Alvarez2016-05-171-27/+35
| | | | Change-Id: I9ae5a846e20c0d32047a3d4fd5892e2f632f0cbe
* Make `morph get-repo` use submodules from definitionsPedro Alvarez2016-04-121-14/+46
| | | | Change-Id: I8d9330eb5f72a8f3f106aae4387eb8321f6126e5
* Add support for definitions version 8Pedro Alvarez2016-03-293-8/+17
| | | | | | | | | This code is a rework from changes done by: - Tiago Gomes <tiago.gomes@codethink.co.uk> https://storyboard.baserock.org/#!/story/86 Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
* Remove support for definitions version 6Tiago Gomes2016-03-251-2/+0
| | | | Change-Id: I603b5172902b5249874ea3bb90de59db5a908185
* morphloader: simplify APITiago Gomes2016-03-251-2/+2
| | | | | | | Use an optional parameter to control whether the defaults should be set instead of a separate function. Change-Id: Idbbd5a08e8b16d8e01bb9539274092978b64f6f0
* Unify local and remote repo cache modulesSam Thursfield2016-03-169-80/+50
| | | | | | | | | | | | | | | | | | | | | There's not really any reason you'd want to use the RemoteRepoCache class except as a workaround for the slow speed of some LocalRepoCache operations, so I can't see this ruining anyone's day. The main reason for doing this is so we can simply the sourceresolver code. One reason that the sourceresolver class is so hopelessly complicated is that it right now has to use two incompatible interfaces for Git repo caches. I've taken the opportunity to detangle the RepoCache class from the App class. Now all of the configuration for the RepoCache class is passed into the constructor explicitly. This makes the class usable from outside Morph: resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[]) repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver) Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
* Get rid of the CachedRepo class (almost)Sam Thursfield2016-03-035-13/+13
| | | | | | | | | | | | | | | | | | For a long time the CachedRepo class has basically been a wrapper around the GitDir class, but with a few extra methods that don't really even belong there. It is now a tiny class in the localrepocache module which just keeps track of a few extra attributes. All other functionality is provided by the gitdir module. This commit also removes the `git clone` approach for copying repos out of the cache. The alternative approach implemented by git.copy_repository() was slightly faster when I tested, so for now we should use that everywhere. Longer term we should find out why this is quicker than `git clone`, and fix Git itself to be fast. Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
* Hide get_repo() and cache_repo() functions, always use get_updated_repo()Sam Thursfield2016-03-034-23/+8
| | | | | | | | | | This allows us to simplify a couple of places. I cannot think of a single situation where'd you want to get the cached copy of a repo, but not update it. Think about it -- the repo might be *years* behind the upstream remote. Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
* deploy: Fix an unlikely crash at deploy-timeSam Thursfield2016-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some reason I had a deployment .morph file that specified: deploy-defaults: This resulted in the deploy_defaults variable having value None, which lead to a crash further down: 2016-02-19 12:03:52 Deciding on task order Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 574, in upgrade self.deploy(args) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 370, in deploy env_vars, deployments) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 403, in deploy_cluster parent_location='') File "/src/morph/morphlib/plugins/deploy_plugin.py", line 462, in deploy_system env_vars, deployment_filter, parent_location) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 483, in deploy_system_with_source_pool system_id, env_vars, deploy_defaults, deploy_params) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 59, in configuration_for_system deploy_params.items() + AttributeError: 'NoneType' object has no attribute 'items' This change makes the code work as expected. Change-Id: I667f4142667be31797cac7c8994d35a404119cca
* Remove (dist)?build-morphology commandsTiago Gomes2015-12-021-79/+0
| | | | Change-Id: Ifb2bc2e2084806fe70b6db96828a390d3082288f
* Prepare the removal of workspace and system branches codeTiago Gomes2015-11-255-14/+9
| | | | Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
* Remove branch-and-merge pluginTiago Gomes2015-11-241-482/+0
| | | | | | RIP. Change-Id: I6aac995415c5d67c60687367697173be52cd2bde
* Remove old cruftTiago Gomes2015-11-201-20/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Improve get-repo error messageRichard Ipsum2015-10-221-1/+5
| | | | | | | | | | | | | | When attempting to get a repo with no commit at the ref we want, we get a 'No such file or directory' error, which is confusing, this is because we attempt to remove the directory we plan to clone into, but if there's no commit at the ref we want then we don't get as far as running git clone, so the directory doesn't exist, and we fail to remove it. This commit improves the error message by catching the InvalidRefError separately. Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
* Change how MorphologyLoader instances are createdSam Thursfield2015-09-284-7/+5
| | | | | | | | | | | | | | Loading .morph files is becoming a bit more complicated, as we need to deal with the VERSION file, and possibly soon with a DEFAULTS file as well. The logic of loading and parsing .morph files is done either in the sourceresolver module, or the morphloader module. This change means that all users of the latter module can use the get hold of a MorphologyLoader instance with VERSION already parsed. If DEFAULTS is added then it is also simple to parse DEFAULTS. Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
* Remove edit commandRichard Ipsum2015-09-251-94/+0
| | | | Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
* Simplify StagingArea classTiago Gomes2015-09-221-2/+2
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Remove foreach commandRichard Ipsum2015-09-171-59/+0
| | | | | | | | We're beginning a transition away from workspaces, foreach in particular is possibly one of the least used commands, and is only mentioned in tutorials explicitly marked as 'old'. Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
* Stop moving staging areas of failed buildsTiago Gomes2015-09-011-1/+27
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Fix crash in `morph distbuild` commandSam Thursfield2015-06-231-1/+1
| | | | | | | | This was causing: UnboundLocalError: local variable 'original_ref' referenced before assignment Change-Id: I46e4b5d527d9ac5480a3d1fe5e6d631f0b7279dc
* Make `morph show-build-log` look in local repo cache for build logsSam Thursfield2015-06-221-21/+33
| | | | | | | This makes it more useful and saves people from poking around in the cache in order to dig up build logs. Change-Id: I8e062c5c32b01aca0df54e1974ead3c3b3134cc3
* Make more commands work outside a system-branch/workspaceSam Thursfield2015-06-223-54/+56
| | | | | | | | | | | | | | | | This makes `morph get-chunk-details`, `morph get-repo` and `morph show-build-log` work in any Git checkout of definitions.git. These are all of the commands outside of the deprecated branch_and_merge_plugin that take notice of system branches. The DefinitionsRepo.relative_path_to_chunk() function is changed a bit. It actually only existed for the `get-repo` command to use, but it turns out that it did the wrong thing by returning a long path for the repo (e.g. baserock/baserock/fhs-dirs) instead of a short one (fhs-dirs). The latter is less typing, and is the behaviour expected by the `get-repo` yarn tests, so it now does that. Change-Id: I430b540b3b0f309cf7018e0b8236f0e8a9042d89
* Determine whether to include_local_changes from app.settingsSam Thursfield2015-06-222-6/+2
| | | | | | | Previously callers to DefinitionsRepoWithApp.source_pool() could override the user's value, but that's not actually used anywhere. Change-Id: I43b53e7b9fc937886c8e6e95947e5e2b6776d085
* Raise exception if show-build-log doesn't find the build logSam Thursfield2015-06-191-5/+14
| | | | | | | | | | | | | Previously it would print nothing and then exit with success. Better to tell the user that an error occurred. Example of the new error: ERROR: No build log for artifact 6fb39673b8f9a1c9848063f5132aa958ffa75f2be61a8dde68ebb11f5a3c4a5f found on cache server http://cache.baserock.org:8080/ Change-Id: Icf8ceef60eb497ff90e00391d442a394e3d76f10
* Fix `morph show-build-log` for chunks of a different architectureSam Thursfield2015-06-191-0/+6
| | | | | | | | Previously if you tried to view the log on an x86_32 machine for a chunk that was build on ARM, it'd give a spurious error saying "Are you trying to cross-build?" Change-Id: I19c3781e0951ecb4161f06fd8b3a0c8ea5bfef9d
* Use DefinitionsRepo class in deploy pluginSam Thursfield2015-06-161-74/+64
| | | | | | | | | | This allows you to run `morph deploy` from any Git checkout of definitions.git. This also changes `morph help-extensions` to use the DefinitionsRepo class, as the code overlaps. Change-Id: I64c48f59c8ec5aebc7169f3b4b4abbb759bd0b9a
* Use DefinitionsRepo in build commandSam Thursfield2015-06-161-143/+74
| | | | | | | | | This allows using `morph build`, `morph distbuild` and `morph distbuild-start` from any Git checkout of a definitions.git repo, so nobody needs to use `morph checkout` or `morph branch` if they don't want to. Change-Id: I5fdfae0f8bec1953893e26f0d227e289da11fa84
* Set PYTHONPATH when running deployment extensionsAdam Coldrick2015-06-121-0/+8
| | | | | | | | | | | | | After moving deployment extensions into the definitions repository, but they need access to shared code, so deployment extensions need to import code which lives in definitions. However morph runs a copy of them in a temporary directory. Version 5 will allow deployment extensions to depend only on code which is also in definitions and the Python standard library, though they may also include code from elsewhere as described in PYTHONPATH. Change-Id: I998f01830656a5118bba1e579f649ec98f5f82c6
* morphlib: Use new temp_dir helper context managerRichard Maw2015-06-053-54/+38
| | | | Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
* deploy: Don't imply --upgrade in subsystemsRichard Maw2015-05-271-2/+5
| | | | | | | | | | | --upgrade being passed on to subsystem deployments broke attempting to perfom an ssh-rsync upgrade with a system that had a sysroot in it. Therefore it is appropriate to prevent --upgrade being inherited like this, as it is seldom what is wanted, and if needed, it can be set in the cluster, or on the command-line separately. Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
* morph deploy: Allow overriding UPGRADE in definition or on command lineRichard Maw2015-05-271-4/+11
| | | | | | | | | | | | | | | | | | | Concievably, you may want to force a cluster to behave as an upgrade regardless of whether --upgrade is set or not. This would allow two use cases that are not currently possible: 1. To force a cluster to be upgrade-only, which would make our instructions for the upgrade-devel.morph cluster simpler. 2. To allow a cluster to deploy then immediately upgrade. One potential use for this would be to create a disk image with multiple versions on it. This also works as a work-around for --upgrade implying UPGRADE=yes for subsystem deployments, but that should be fixed separately. Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
* Ignore --upgrade flag rather than exitingRichard Ipsum2015-05-271-4/+0
| | | | | | | | Specifying --upgrade with the upgrade command is redundant, but it's not a contradiction, so we can safely ignore --upgrade if it happens to be specified with the upgrade command. Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
* generate-manifest-csv: Catch NoRemote error when cloning lorries repoAdam Coldrick2015-05-201-9/+20
| | | | | | | | | | | If there is no $trove-id/local-config/lorries repository, and similarly if there is no baserock/local-config/lorries repository on the Trove defined by 'trove-host', the generate-manifest-csv command will fail with a NoRemote error from trying to clone a non-existent repository. This commit fixes this and instead outputs a warning if either of the repositories cannot be found. Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
* Give exact filename when an artifact is not found in the cache.Sam Thursfield2015-05-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Old message: ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. New message: ERROR: Deployment failed as 45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
* distbuild: Condense Initiator class to remove unnecessary duplicationLauren Perry2015-05-151-4/+16
| | | | | | | | | Create an InitiatorCommand class that accepts message_type and status_text parameters to be used by the distbuild-list-jobs, distbuild-status and distbuild-cancel commands to send request messages to the distbuild network Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
* Fix prototype of create_source_pool()Sam Thursfield2015-05-1411-12/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* Clean up artifact serialisationAdam Coldrick2015-05-121-9/+10
| | | | | | | We no longer serialise whole artifacts, so it doesn't make sense for things to still refer to serialise-artifact and similar. Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
* Move duplicate fix_chunk_build_mode function to a common locationAdam Coldrick2015-05-122-52/+5
| | | | Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
* Enable native-bootstrap to continue build after recovered from fault.Kejia Hu (Terry)2015-05-121-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | The previous script creates new directory for the chunk it is going to build without checking whether the directory exists or not. It will fail back if the directory it attempted to creat exists. So if build failed, you always need to remove all .inst directories and let the native-bootstrap script build from the beginning. This patch improves this, and you can run the native bootstrap script direct after resumed without loss previous progress. A condition was added to determine whether previous native bootstrap script was terminated when it was building current chunk. As .build directory for certain chunk only exists during building phase of itself, it was created when started build, and cleaned up after building finished. If .inst for certain chunk exists, the .build directory doesn't, the building of the chunk should be succeed in previous build. The second go of native-bootstrap will skip all successful chunks and start where it left off. Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
* morph: Add morph diff subcommandRichard Maw2015-05-111-0/+147
| | | | Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
* morph anchor: Handle updating refsRichard Maw2015-05-111-5/+6
| | | | | | | | | | Previously it would not attempt to make commits it needed locally available if the commit was available on the remote repo cache. Now it will do the update if the commit is not available locally, and will obey --no-git-update. Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
* Fix distbuild-morphologyAdam Coldrick2015-05-071-1/+1
| | | | | | This fixes an error caused by not enough parameters being given to the InitiatorBuildCommand constructor in distbuild-morphology. Change-Id: I133bd2f267fd06cfe88a1cbf4711cc79ad00d209
* Add `morph anchor` commandRichard Maw2015-05-011-0/+226
| | | | Change-Id: If9d92d7c75b9c4276b69c482c076c6fc1d4ccbbf
* Allow the deployment of individual chunks/strata from systemsAdam Coldrick2015-04-301-40/+215
| | | | | | | | | | | | | This commit allows the specification of one or more strata/chunks in a deployment entry in a cluster morphology to deploy instead of the full system if --partial is set. These are listed in a 'partial-deploy-components' field in each deployment definition. The components must be in the system, and this only works for tarball or sysroot deployments. It SHOULD NOT be used when deploying production systems, as it has a number of limitations. Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714