summaryrefslogtreecommitdiff
path: root/morphlib/plugins
Commit message (Collapse)AuthorAgeFilesLines
* Remove (dist)?build-morphology commandsTiago Gomes2015-12-021-79/+0
| | | | Change-Id: Ifb2bc2e2084806fe70b6db96828a390d3082288f
* Prepare the removal of workspace and system branches codeTiago Gomes2015-11-255-14/+9
| | | | Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
* Remove branch-and-merge pluginTiago Gomes2015-11-241-482/+0
| | | | | | RIP. Change-Id: I6aac995415c5d67c60687367697173be52cd2bde
* Remove old cruftTiago Gomes2015-11-201-20/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Improve get-repo error messageRichard Ipsum2015-10-221-1/+5
| | | | | | | | | | | | | | When attempting to get a repo with no commit at the ref we want, we get a 'No such file or directory' error, which is confusing, this is because we attempt to remove the directory we plan to clone into, but if there's no commit at the ref we want then we don't get as far as running git clone, so the directory doesn't exist, and we fail to remove it. This commit improves the error message by catching the InvalidRefError separately. Change-Id: Ibca20dd995d858713e56ab834daa97a8297cafc8
* Change how MorphologyLoader instances are createdSam Thursfield2015-09-284-7/+5
| | | | | | | | | | | | | | Loading .morph files is becoming a bit more complicated, as we need to deal with the VERSION file, and possibly soon with a DEFAULTS file as well. The logic of loading and parsing .morph files is done either in the sourceresolver module, or the morphloader module. This change means that all users of the latter module can use the get hold of a MorphologyLoader instance with VERSION already parsed. If DEFAULTS is added then it is also simple to parse DEFAULTS. Change-Id: Ib33756e9dbd078e38f12dd7f776c89584b178959
* Remove edit commandRichard Ipsum2015-09-251-94/+0
| | | | Change-Id: I276a00bb9117aaa2d4fcc7e187b846eedae65591
* Simplify StagingArea classTiago Gomes2015-09-221-2/+2
| | | | | | | | | | | | Pass the Source to the staging area constructor so that we don't need to pass it as a parameter when we call some StagingArea methods. Also move the creation of the build and destdir directories to the constructor so that we can get rid of the chroot_open() and chroot_close() methods. Also, provide API to retrieve the relative locations for buildir and destdir. Change-Id: I6e8085392e19ff3d8df807f260acf90eec9e0901
* Remove foreach commandRichard Ipsum2015-09-171-59/+0
| | | | | | | | We're beginning a transition away from workspaces, foreach in particular is possibly one of the least used commands, and is only mentioned in tutorials explicitly marked as 'old'. Change-Id: I2c95ce689bb5550bf50feb6b55be6c58671c4c4a
* Stop moving staging areas of failed buildsTiago Gomes2015-09-011-1/+27
| | | | | | | | | | | | | | | | | | | | | Stop moving staging areas of failed builds from the 'staging' directory to the 'failed' directory. Moving staging areas make it very difficult to debug build failures on the build essential chunks, as the paths set on the configure scripts and some environment variables (e.g. STAGE2_SYSROOT, DESTDIR) will be invalid after moving the staging area. This change will also make it easier to create scripts that chroot n environment similiar to the one where the build failure occurred. To make it still possible to safely do a build an run `morph gc` in parallel, we use flock(2) to control access to the staging area directory. Also, move the `test_supports_non_isolated_mode` test into a different class, as it requires a different SetUp() routine (the staging area is contructed with different parameters). Change-Id: I06c3c435ad05c12afabc0adc2a9d4f8a284ccc02
* Fix crash in `morph distbuild` commandSam Thursfield2015-06-231-1/+1
| | | | | | | | This was causing: UnboundLocalError: local variable 'original_ref' referenced before assignment Change-Id: I46e4b5d527d9ac5480a3d1fe5e6d631f0b7279dc
* Make `morph show-build-log` look in local repo cache for build logsSam Thursfield2015-06-221-21/+33
| | | | | | | This makes it more useful and saves people from poking around in the cache in order to dig up build logs. Change-Id: I8e062c5c32b01aca0df54e1974ead3c3b3134cc3
* Make more commands work outside a system-branch/workspaceSam Thursfield2015-06-223-54/+56
| | | | | | | | | | | | | | | | This makes `morph get-chunk-details`, `morph get-repo` and `morph show-build-log` work in any Git checkout of definitions.git. These are all of the commands outside of the deprecated branch_and_merge_plugin that take notice of system branches. The DefinitionsRepo.relative_path_to_chunk() function is changed a bit. It actually only existed for the `get-repo` command to use, but it turns out that it did the wrong thing by returning a long path for the repo (e.g. baserock/baserock/fhs-dirs) instead of a short one (fhs-dirs). The latter is less typing, and is the behaviour expected by the `get-repo` yarn tests, so it now does that. Change-Id: I430b540b3b0f309cf7018e0b8236f0e8a9042d89
* Determine whether to include_local_changes from app.settingsSam Thursfield2015-06-222-6/+2
| | | | | | | Previously callers to DefinitionsRepoWithApp.source_pool() could override the user's value, but that's not actually used anywhere. Change-Id: I43b53e7b9fc937886c8e6e95947e5e2b6776d085
* Raise exception if show-build-log doesn't find the build logSam Thursfield2015-06-191-5/+14
| | | | | | | | | | | | | Previously it would print nothing and then exit with success. Better to tell the user that an error occurred. Example of the new error: ERROR: No build log for artifact 6fb39673b8f9a1c9848063f5132aa958ffa75f2be61a8dde68ebb11f5a3c4a5f found on cache server http://cache.baserock.org:8080/ Change-Id: Icf8ceef60eb497ff90e00391d442a394e3d76f10
* Fix `morph show-build-log` for chunks of a different architectureSam Thursfield2015-06-191-0/+6
| | | | | | | | Previously if you tried to view the log on an x86_32 machine for a chunk that was build on ARM, it'd give a spurious error saying "Are you trying to cross-build?" Change-Id: I19c3781e0951ecb4161f06fd8b3a0c8ea5bfef9d
* Use DefinitionsRepo class in deploy pluginSam Thursfield2015-06-161-74/+64
| | | | | | | | | | This allows you to run `morph deploy` from any Git checkout of definitions.git. This also changes `morph help-extensions` to use the DefinitionsRepo class, as the code overlaps. Change-Id: I64c48f59c8ec5aebc7169f3b4b4abbb759bd0b9a
* Use DefinitionsRepo in build commandSam Thursfield2015-06-161-143/+74
| | | | | | | | | This allows using `morph build`, `morph distbuild` and `morph distbuild-start` from any Git checkout of a definitions.git repo, so nobody needs to use `morph checkout` or `morph branch` if they don't want to. Change-Id: I5fdfae0f8bec1953893e26f0d227e289da11fa84
* Set PYTHONPATH when running deployment extensionsAdam Coldrick2015-06-121-0/+8
| | | | | | | | | | | | | After moving deployment extensions into the definitions repository, but they need access to shared code, so deployment extensions need to import code which lives in definitions. However morph runs a copy of them in a temporary directory. Version 5 will allow deployment extensions to depend only on code which is also in definitions and the Python standard library, though they may also include code from elsewhere as described in PYTHONPATH. Change-Id: I998f01830656a5118bba1e579f649ec98f5f82c6
* morphlib: Use new temp_dir helper context managerRichard Maw2015-06-053-54/+38
| | | | Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
* deploy: Don't imply --upgrade in subsystemsRichard Maw2015-05-271-2/+5
| | | | | | | | | | | --upgrade being passed on to subsystem deployments broke attempting to perfom an ssh-rsync upgrade with a system that had a sysroot in it. Therefore it is appropriate to prevent --upgrade being inherited like this, as it is seldom what is wanted, and if needed, it can be set in the cluster, or on the command-line separately. Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
* morph deploy: Allow overriding UPGRADE in definition or on command lineRichard Maw2015-05-271-4/+11
| | | | | | | | | | | | | | | | | | | Concievably, you may want to force a cluster to behave as an upgrade regardless of whether --upgrade is set or not. This would allow two use cases that are not currently possible: 1. To force a cluster to be upgrade-only, which would make our instructions for the upgrade-devel.morph cluster simpler. 2. To allow a cluster to deploy then immediately upgrade. One potential use for this would be to create a disk image with multiple versions on it. This also works as a work-around for --upgrade implying UPGRADE=yes for subsystem deployments, but that should be fixed separately. Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
* Ignore --upgrade flag rather than exitingRichard Ipsum2015-05-271-4/+0
| | | | | | | | Specifying --upgrade with the upgrade command is redundant, but it's not a contradiction, so we can safely ignore --upgrade if it happens to be specified with the upgrade command. Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
* generate-manifest-csv: Catch NoRemote error when cloning lorries repoAdam Coldrick2015-05-201-9/+20
| | | | | | | | | | | If there is no $trove-id/local-config/lorries repository, and similarly if there is no baserock/local-config/lorries repository on the Trove defined by 'trove-host', the generate-manifest-csv command will fail with a NoRemote error from trying to clone a non-existent repository. This commit fixes this and instead outputs a warning if either of the repositories cannot be found. Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
* Give exact filename when an artifact is not found in the cache.Sam Thursfield2015-05-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Old message: ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. New message: ERROR: Deployment failed as 45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
* distbuild: Condense Initiator class to remove unnecessary duplicationLauren Perry2015-05-151-4/+16
| | | | | | | | | Create an InitiatorCommand class that accepts message_type and status_text parameters to be used by the distbuild-list-jobs, distbuild-status and distbuild-cancel commands to send request messages to the distbuild network Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
* Fix prototype of create_source_pool()Sam Thursfield2015-05-1411-12/+14
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* Clean up artifact serialisationAdam Coldrick2015-05-121-9/+10
| | | | | | | We no longer serialise whole artifacts, so it doesn't make sense for things to still refer to serialise-artifact and similar. Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
* Move duplicate fix_chunk_build_mode function to a common locationAdam Coldrick2015-05-122-52/+5
| | | | Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
* Enable native-bootstrap to continue build after recovered from fault.Kejia Hu (Terry)2015-05-121-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | The previous script creates new directory for the chunk it is going to build without checking whether the directory exists or not. It will fail back if the directory it attempted to creat exists. So if build failed, you always need to remove all .inst directories and let the native-bootstrap script build from the beginning. This patch improves this, and you can run the native bootstrap script direct after resumed without loss previous progress. A condition was added to determine whether previous native bootstrap script was terminated when it was building current chunk. As .build directory for certain chunk only exists during building phase of itself, it was created when started build, and cleaned up after building finished. If .inst for certain chunk exists, the .build directory doesn't, the building of the chunk should be succeed in previous build. The second go of native-bootstrap will skip all successful chunks and start where it left off. Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
* morph: Add morph diff subcommandRichard Maw2015-05-111-0/+147
| | | | Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
* morph anchor: Handle updating refsRichard Maw2015-05-111-5/+6
| | | | | | | | | | Previously it would not attempt to make commits it needed locally available if the commit was available on the remote repo cache. Now it will do the update if the commit is not available locally, and will obey --no-git-update. Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
* Fix distbuild-morphologyAdam Coldrick2015-05-071-1/+1
| | | | | | This fixes an error caused by not enough parameters being given to the InitiatorBuildCommand constructor in distbuild-morphology. Change-Id: I133bd2f267fd06cfe88a1cbf4711cc79ad00d209
* Add `morph anchor` commandRichard Maw2015-05-011-0/+226
| | | | Change-Id: If9d92d7c75b9c4276b69c482c076c6fc1d4ccbbf
* Allow the deployment of individual chunks/strata from systemsAdam Coldrick2015-04-301-40/+215
| | | | | | | | | | | | | This commit allows the specification of one or more strata/chunks in a deployment entry in a cluster morphology to deploy instead of the full system if --partial is set. These are listed in a 'partial-deploy-components' field in each deployment definition. The components must be in the system, and this only works for tarball or sysroot deployments. It SHOULD NOT be used when deploying production systems, as it has a number of limitations. Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
* morphlib: Add command to get build-log for a given chunk and system fileLauren Perry2015-04-301-0/+90
| | | | Change-Id: I09e9b17ef2e0fb94dbf5a96dca91062d64433add
* Add a CSV manifest generator.Adam Coldrick2015-04-302-2/+338
| | | | | | | | | | | | | | | | This adds a `morph generate-manifest-csv` command which creates a CSV manifest for a system containing license information, upstream URL and a guess at the version of each component in a system. The existing `morph generate-manifest` is also renamed to `morph generate-manifest-genivi` to improve the distinction between the two commands which generate a manifest. The basis of this commit was done by Michael Drake <michael.drake@codethink.co.uk>. Change-Id: I712de11eaa6e1564c5bae5f5f396e5e77484972f
* Check number of args given to build commandsRichard Ipsum2015-04-301-5/+38
| | | | Change-Id: I501632c4d7acde06e391df3752275a3afc334ba6
* Improve 'Deploy failed as system is not yet build' error.Sam Thursfield2015-04-301-4/+5
| | | | | | | | | | If a user had the wrong setting for artifact-cache-server then they could build a system with 'morph distbuild', then be told it wasn't built by 'morph deploy'. The logs do not show any info about which cache server was used, so it looks like a Morph bug. This change should make such a configuration problem much easier to spot. Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
* distbuild: Add distbuild status commandLauren Perry2015-04-291-0/+47
| | | | | | | | | Adds a command to get the status of all recently ran distbuilds for a given server (e.g. Running, Finished, Failed, Cancelled), so as to tell if a build running via distbuild-start has finished or otherwise exited without going through the server's log files Change-Id: I5ce9fe54ae7b1bd8fe3e0d629f615042be8827ed
* distbuild: Add distbuild start and cancel functionalityLauren Perry2015-04-292-1/+72
| | | | | | | | | | | Add command for distbuild-start to build_plugin in morphlib, and create a boolean parameter to inform the initiator whether to disconnect the controller and leave the build running remotely. Add distbuild-cancel command to parse currently-running distbuild build-request IDs and cancel the one matching the given argument Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
* get-repo: Delete partially cloned repo on errorsSam Thursfield2015-04-281-2/+9
| | | | | | | | | If a `morph get-repo` command hits an error or is interrupted while the 'git clone' operation is in progress, the partial clone is left around. When the user tries to run the command again, they hit an error saying the directory already exists. Change-Id: Id4bb57f2136049ede0ea900e7e359255617427ec
* Add a `morph get-repo` command to clone repositories from the cacheAdam Coldrick2015-04-271-0/+149
| | | | Change-Id: I0c175649bd98067ea69d471ebe6880aac5a25598
* Fix regression that caused original_ref to be lost in some buildsSam Thursfield2015-04-271-2/+3
| | | | | | | | | | Commit bd788c7219d8b ("Implement partial builds") caused the original_ref metadata to be lost in some cases. This meant that the metadata for built systems would have original_ref set to a SHA1 hex string, instead of the original named ref that the system was built from. Change-Id: Id31654ad60b411206ccc7ee70b46f005baa209ef
* morphlib: Add example use to morph certify and fix spelling errorLauren Perry2015-04-271-1/+6
| | | | Change-Id: I3c4b4f687703ab665841a3527044d0f669135513
* Calculate the build graph in worker-buildAdam Coldrick2015-04-241-5/+25
| | | | | | | Calculate the build graph and find the artifact which is referred to by the ArtifactReference which was deserialised. Change-Id: Icbef90932979fc1a18daf0dedac16f4429f56349
* distbuild: Don't check if there is free disk space on the initiatorSam Thursfield2015-04-221-6/+7
| | | | | | | This check is intended for local builds, not distbuilds. Much less disk space is needed to distbuild. Change-Id: Iaff77805119129cbf6584de418ef97c7f1b82ad8
* deploy: Clean up `morph deploy` documentation a bitSam Thursfield2015-04-211-133/+30
| | | | | | | | | | | | Remove documentation specific to certain .configure and .write extensions, and point the user to `morph help-extensions` and `morph help xxx.write` instead. Don't mention the deprecated nfsboot.write extension. Other small cleanups. Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
* deploy: Deploy and upgrade systems from the same 'cluster' definitionSam Thursfield2015-04-211-17/+83
| | | | | | | | | | | | | | | | | | | This patch adds two fields to deployment (cluster) .morph files: 'upgrade-type' and 'upgrade-location'. The `morph deploy` command ignores these. The `morph upgrade` command will honour them if present, instead of the existing 'type' and 'location' fields. If they are not present, `morph upgrade` will give a warning, and will use the 'type' and 'location' fields as before. This avoids the need to edit the deployment .morph file after deploying a system. Small detail: the 'type' and 'location' variables are no longer removed from the environment that is passed to the .configure and .write extensions. This shouldn't affect anything. Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
* deploy: Factor out a bit of code into its own functionSam Thursfield2015-04-211-10/+37
| | | | | | | This behaviour is complex, it shouldn't be hidden away in a larger function. Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
View Lorry Controller Status