summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
* Fix crash in `morph distbuild` commandSam Thursfield2015-06-231-1/+1
| | | | | | | | This was causing: UnboundLocalError: local variable 'original_ref' referenced before assignment Change-Id: I46e4b5d527d9ac5480a3d1fe5e6d631f0b7279dc
* Make `morph show-build-log` look in local repo cache for build logsSam Thursfield2015-06-221-21/+33
| | | | | | | This makes it more useful and saves people from poking around in the cache in order to dig up build logs. Change-Id: I8e062c5c32b01aca0df54e1974ead3c3b3134cc3
* Make more commands work outside a system-branch/workspaceSam Thursfield2015-06-224-67/+61
| | | | | | | | | | | | | | | | This makes `morph get-chunk-details`, `morph get-repo` and `morph show-build-log` work in any Git checkout of definitions.git. These are all of the commands outside of the deprecated branch_and_merge_plugin that take notice of system branches. The DefinitionsRepo.relative_path_to_chunk() function is changed a bit. It actually only existed for the `get-repo` command to use, but it turns out that it did the wrong thing by returning a long path for the repo (e.g. baserock/baserock/fhs-dirs) instead of a short one (fhs-dirs). The latter is less typing, and is the behaviour expected by the `get-repo` yarn tests, so it now does that. Change-Id: I430b540b3b0f309cf7018e0b8236f0e8a9042d89
* Determine whether to include_local_changes from app.settingsSam Thursfield2015-06-223-9/+5
| | | | | | | Previously callers to DefinitionsRepoWithApp.source_pool() could override the user's value, but that's not actually used anywhere. Change-Id: I43b53e7b9fc937886c8e6e95947e5e2b6776d085
* Raise exception if show-build-log doesn't find the build logSam Thursfield2015-06-191-5/+14
| | | | | | | | | | | | | Previously it would print nothing and then exit with success. Better to tell the user that an error occurred. Example of the new error: ERROR: No build log for artifact 6fb39673b8f9a1c9848063f5132aa958ffa75f2be61a8dde68ebb11f5a3c4a5f found on cache server http://cache.baserock.org:8080/ Change-Id: Icf8ceef60eb497ff90e00391d442a394e3d76f10
* Fix `morph show-build-log` for chunks of a different architectureSam Thursfield2015-06-191-0/+6
| | | | | | | | Previously if you tried to view the log on an x86_32 machine for a chunk that was build on ARM, it'd give a spurious error saying "Are you trying to cross-build?" Change-Id: I19c3781e0951ecb4161f06fd8b3a0c8ea5bfef9d
* localrepocache: Count freshly cloned repositories as updatedRichard Maw2015-06-171-1/+4
| | | | | | | This fixes requires_update_for_ref when it is given a non-fixed ref and it is retrieved from a call to get_updated_repo. Change-Id: Icbcf2e4ec45bd559c6bcb450ceb4a8861b56f654
* Don't ignore the 'cwd' parameterSam Thursfield2015-06-161-1/+1
| | | | Thanks to Richard Maw for suggesting this fixup.
* Expose reset_workdir in GitDirectory classRichard Ipsum2015-06-161-0/+6
| | | | Change-Id: Ib2d7bd31bea49c052a59582524382df6b931a31f
* Swap repr() for str() in ref add errmsgRichard Ipsum2015-06-161-1/+1
| | | | | | | | | | | | | | | | | | | | Old message: ERROR: Adding ref refs/heads/baserock/builds/review/javier_jard_n/foundation_core/d3401783ea054fc18ee56ccdb0030d0a with commit 1c60a89a98c5ee532fefd363240b06870ada02f5 failed in git repository located at /src/definitions: AppException() New message: ERROR: Adding ref refs/heads/baserock/builds/review/javier_jard_n/foundation_core/d3401783ea054fc18ee56ccdb0030d0a with commit 1c60a89a98c5ee532fefd363240b06870ada02f5 failed in git repository located at /src/definitions: Command failed: git update-ref refs/heads/baserock/builds/review/javier_jard_n/foundation_core/d3401783ea054fc18ee56ccdb0030d0a 1c60a89a98c5ee532fefd363240b06870ada02f5... Change-Id: Idc6a47388f53d358b7dfc2c0f7fa82eefbc92630
* Improve error when local HEAD of definitions repo is not found in remoteSam Thursfield2015-06-162-6/+29
| | | | | | | | | | | | | | | | | | Old error: ERROR: Git directory /build/cache/gits/git___git_baserock_org_baserock_baserock_definitions has no commit at ref 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca^{commit}. New error: ERROR: Commit 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca wasn't found in the "origin" remote git://git.baserock.org/baserock/baserock/definitions. You either need to push your local commits on branch xxx to "origin", or use the --local-changes=include feature. Change-Id: I0c3658e9cd27c23f40653662ba7e4ba58b7892de
* Use DefinitionsRepo class in deploy pluginSam Thursfield2015-06-162-92/+77
| | | | | | | | | | This allows you to run `morph deploy` from any Git checkout of definitions.git. This also changes `morph help-extensions` to use the DefinitionsRepo class, as the code overlaps. Change-Id: I64c48f59c8ec5aebc7169f3b4b4abbb759bd0b9a
* Use DefinitionsRepo in build commandSam Thursfield2015-06-163-147/+81
| | | | | | | | | This allows using `morph build`, `morph distbuild` and `morph distbuild-start` from any Git checkout of a definitions.git repo, so nobody needs to use `morph checkout` or `morph branch` if they don't want to. Change-Id: I5fdfae0f8bec1953893e26f0d227e289da11fa84
* Add DefinitionsRepo classSam Thursfield2015-06-165-39/+520
| | | | | | | | | | | | | | The intention is for this class to take over the from the Workspace and SystemBranch classes. It allows Morph to load and parse definitions from a Git repo, without requiring the user to run `morph checkout` or `morph branch`: it can operate from any normal Git repository. The class behaves differently when the Git repository is inside a Morph system-branch checkout made with `morph branch` or `morph checkout`, to avoid changing things under the feet of people who are used to those commands. Change-Id: I52a898efb9f6fb7f7e94c65b9ed38516bd51f49d
* Treat True and False as actual commands, rather than booleans.Jim MacArthur2015-06-161-0/+3
| | | | | | | | | | | It's quite easy to forget that "true" and "false" are special words in YAML, and write "false" to halt the build process while debugging. This was treated as a Boolean instead of a string, so causes an unhelpful error in morph. Since there is no use for booleans in morphologies, they should be treated as strings. Change-Id: I7c872f9696611920febec5f375b599eee89d040e
* buildsystem: Comment strip commandRichard Maw2015-06-121-0/+7
| | | | Change-Id: Ib8ed113731590b3c9bc3559fb07820bd25c462b8
* sourceresolver: Support version 5Richard Maw2015-06-121-1/+1
| | | | | | | | This adds support for strip commands, and deployment extensions that rely on python libraries being available alongside the extensions in the definitions repository. Change-Id: I23dafd4968002037d182507762b09147f87bf469
* Set PYTHONPATH when running deployment extensionsAdam Coldrick2015-06-121-0/+8
| | | | | | | | | | | | | After moving deployment extensions into the definitions repository, but they need access to shared code, so deployment extensions need to import code which lives in definitions. However morph runs a copy of them in a temporary directory. Version 5 will allow deployment extensions to depend only on code which is also in definitions and the Python standard library, though they may also include code from elsewhere as described in PYTHONPATH. Change-Id: I998f01830656a5118bba1e579f649ec98f5f82c6
* cachekey/builder: Use and cache strip commandsRichard Maw2015-06-122-6/+12
| | | | Change-Id: I19a6c31979aa36ff3c03f41e16e2d25ef407533b
* sourceresolver: Use versioned morphology loaderRichard Maw2015-06-121-1/+2
| | | | | | | | | | | | We don't use this globally, since other users of the MorphologyLoader don't need them to be aware of strip commands yet, and a more significant rework would be required to allow the injection of this context, because previously the MorpohologyLoader did not need any, so code which used the MorphologyLoader just created one as they needed it, rather than allowing one to be passed in where the context was available. Change-Id: Iea707a1cb2c631dd6f251b5b989d31e192f2387d
* morphloader: Smoke test strip command inclusionRichard Maw2015-06-121-0/+10
| | | | Change-Id: Icd8858dbdbb45a636641f7c5b8ab540a8376acc7
* morphloader: Allow build-system lookup to be customisedRichard Maw2015-06-121-3/+5
| | | | | | | This defaults to the default, and only useful behaviour at this point. It is there so that tests may mock it out. Change-Id: I17499fc1bb0a1ad39da2ac836a2946b1a0ed5abe
* morphloader: When given definitions v5, include strip commandsRichard Maw2015-06-121-0/+9
| | | | Change-Id: Ib64e4f823de129a3a5a78eb1db6a4e0f6da03928
* buildsystems: Add strip commandsRichard Maw2015-06-121-0/+21
| | | | | | | This adds strip commands to all build systems, that will be hooked up into the cache and build logic. Change-Id: I37ad3e43311a417a949e7dfef860a50fdf6b8c43
* sourceresolver: Re-order method definitions in use orderRichard Maw2015-06-121-74/+74
| | | | Change-Id: I66560dfb0e97c1d55a4e2399f81747644eb6d74e
* sourceresolver: Pass all hidden stateRichard Maw2015-06-122-112/+164
| | | | Change-Id: I72327a2fdb091fc742369af68c8ebfe16f34be5e
* Fix test suiteSam Thursfield2015-06-101-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The test suite was failing in my Baserock 15.19 chroot with: FAILURE: test_commit_tree (gitdir_tests.GitDirectoryContentsTests) Traceback (most recent call last): File "morphlib/gitdir_tests.py", line 294, in test_commit_tree self.assertEqual(expected, gd.get_commit_contents(commit).split('\n')) AssertionError: Lists differ: ['tree 01d830ae9bae1a9970a9f14... != ['tree 01d830ae9bae1a9970a9f14... First differing element 2: author Author Name <author@email> 683074800 +0000 author Author Name <author@email> 683074800 +0100 ['tree 01d830ae9bae1a9970a9f1491c914ad177f8afce', 'parent 1240834484cebc4ba8cc40cd5b76f6dffd937a50', - 'author Author Name <author@email> 683074800 +0000', ? ^ + 'author Author Name <author@email> 683074800 +0100', ? ^ - 'committer Committer Name <committer@email> 683074800 +0000', ? ^ + 'committer Committer Name <committer@email> 683074800 +0100', ? ^ '', 'MESSAGE', ''] 1 failures, 0 errors 1543 excluded statements 57 excluded modules Time: 8.4 s Change-Id: If03e7ccbb5e3415eead9dcbcf908d3d2717e8fe0
* CacheKeyComputer: Only attempt to include commands present in morphologyRichard Maw2015-06-051-1/+2
| | | | | | | | The set of commands is potentially extendable, and it's easier to accept optional keys than get the list of potential fields from the build system. Change-Id: I0d876112db43637661e9548472a9fb191ea88268
* morphloader: Hoist buildsystem lookup outside of loopsRichard Maw2015-06-051-6/+6
| | | | Change-Id: If548a9962d5e70480f7eaba6ab6596465975e894
* morphloader: Add test-commands to key orderRichard Maw2015-06-051-0/+3
| | | | Change-Id: I01b3132c9fb27f1c0277ffe5eb0166e489312e38
* sourceresolver: bind util.sanitise_morphology_pathRichard Maw2015-06-051-3/+4
| | | | | | It's too much of a mouthful to use a qualified import. Change-Id: Ide5695e49fd6188b520fecce9d6b076fd7280b8e
* sourceresolver: Remove cache_repo_locallyRichard Maw2015-06-052-20/+9
| | | | Change-Id: Ib8281c4030c42b61e298d6cc52f150a02384ed96
* sourceresolver: Use context managers for cleanup and cache writebackRichard Maw2015-06-051-17/+17
| | | | Change-Id: I895776bfd57ca6a14997c5535355b259d827b3a6
* morphlib: Use new temp_dir helper context managerRichard Maw2015-06-054-59/+40
| | | | Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
* morphlib.util: Add temp_dir context managerRichard Maw2015-06-031-0/+16
| | | | | | | We commonly create a temporary directory and clean it up after we're done. This can be encapsulated into a context manager. Change-Id: Ie50de5c3954141a3e7fd40e4627fb743287ef61f
* sourceresolver: Pass build system directly to morphology creationRichard Maw2015-06-032-18/+19
| | | | | | | | We cache based on the name, but to get the name we acutally loaded the build system, so it's a waste of effort to cache by the name only to reload it later. Change-Id: Ia77e1c73d27005cd4cc6a3bb5553b44e8dfc75c1
* Add separate_mount_namespace kwarg to ext runnerRichard Ipsum2015-06-031-2/+8
| | | | | | Sometimes we don't want a separate mount namespace Change-Id: I9f2645f552c69090612aecd45104a0320cd436c2
* Do not call str() on None types in sanitize_envRichard Ipsum2015-05-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For a cluster morph, - morph: systems/devel-system-x86_64-generic.morph deploy: self: type: ssh-rsync location: root@127.0.0.1 ATTACK_KITTENS: The yaml parser will assign the value None to the element with key 'ATTACK_KITTENS', sanitize_environment blindly calls str() on each value in the dictionary, the result is that our extensions receive an 'ATTACK_KITTENS' environment variable with the string value 'None', this is not useful and confuses anyone trying to do basic validation in their extension. This patch makes it so that any None types are replaced with the empty string, which seems a more sensible value for this case. Change-Id: I734d33e9304f0b7d4d590e464e350d2e5b4720d3
* Ignore the configuration extensions when generating the cache-keyPedro Alvarez2015-05-291-0/+3
| | | | | | | | System artifacts won't change because of changes in the configuration extensions. These extensions run at deployment time, therefore, they are ignored for generating the cache-key. Change-Id: I64f28605630bda4c27748197aef84809b4b1f86b
* Use keystoneclient python api in openstack.checkRichard Ipsum2015-05-291-29/+28
| | | | | | | Switching to the keystoneclient python api gives us a more reliable means of detecting auth failure. Change-Id: I5f734bbfe5568c855f524a3448357f7cf46ab254
* deploy: Don't imply --upgrade in subsystemsRichard Maw2015-05-271-2/+5
| | | | | | | | | | | --upgrade being passed on to subsystem deployments broke attempting to perfom an ssh-rsync upgrade with a system that had a sysroot in it. Therefore it is appropriate to prevent --upgrade being inherited like this, as it is seldom what is wanted, and if needed, it can be set in the cluster, or on the command-line separately. Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
* morph deploy: Allow overriding UPGRADE in definition or on command lineRichard Maw2015-05-271-4/+11
| | | | | | | | | | | | | | | | | | | Concievably, you may want to force a cluster to behave as an upgrade regardless of whether --upgrade is set or not. This would allow two use cases that are not currently possible: 1. To force a cluster to be upgrade-only, which would make our instructions for the upgrade-devel.morph cluster simpler. 2. To allow a cluster to deploy then immediately upgrade. One potential use for this would be to create a disk image with multiple versions on it. This also works as a work-around for --upgrade implying UPGRADE=yes for subsystem deployments, but that should be fixed separately. Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
* Ignore --upgrade flag rather than exitingRichard Ipsum2015-05-271-4/+0
| | | | | | | | Specifying --upgrade with the upgrade command is redundant, but it's not a contradiction, so we can safely ignore --upgrade if it happens to be specified with the upgrade command. Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
* Move 'git config' wrapper into its own classSam Thursfield2015-05-223-18/+75
| | | | | | | | | | We use 'git config' format config files outside Git repos, so it's useful to have a helper class independent of the GitDirectory class. This allows us to use it in the sysbranchdir.open() function to remove a hack. Change-Id: Ifa5e87f404d10666c98b9469079b7925d16becf6
* generate-manifest-csv: Catch NoRemote error when cloning lorries repoAdam Coldrick2015-05-201-9/+20
| | | | | | | | | | | If there is no $trove-id/local-config/lorries repository, and similarly if there is no baserock/local-config/lorries repository on the Trove defined by 'trove-host', the generate-manifest-csv command will fail with a NoRemote error from trying to clone a non-existent repository. This commit fixes this and instead outputs a warning if either of the repositories cannot be found. Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
* Give exact filename when an artifact is not found in the cache.Sam Thursfield2015-05-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Old message: ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. New message: ERROR: Deployment failed as 45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
* Bump supported definitions version to version 4Javier Jardón2015-05-181-1/+1
| | | | | | | | | | | | | A fix for the install-files extension has been merged in commit 93b034f045a2c7443122d7b082ad4460339907d0 This effectively make a new version of definitions if we want to use this (be able overwrite symlinks), in our definitions We will use this to create a symlink to /usr/lib/os-release from /etc/os-release using the install-essential-files extension Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
* distbuild: Condense Initiator class to remove unnecessary duplicationLauren Perry2015-05-151-4/+16
| | | | | | | | | Create an InitiatorCommand class that accepts message_type and status_text parameters to be used by the distbuild-list-jobs, distbuild-status and distbuild-cancel commands to send request messages to the distbuild network Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
* Fix crash in handling of recursive submodulesSam Thursfield2015-05-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out I completely broke Morph's handling of recursive submodules in commit 0855c357e74e6dd7a. We didn't notice because nothing in the reference systems actually uses recursive submodules. However, building the baserock/tlsa/mason2 branch of definitions.git triggered the bug due to the delta:python-packages/gitpython chunk, which contains delta:python-packages/gitdb which in turn contains delta:python-packages/smmap. The error is this: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build self._build(source_pool, filename, component_names=component_names) File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build bc.build_in_order(component) File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order self.cache_or_build_source(s, build_env) File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source self.build_source(source, build_env) File "/src/morph/morphlib/buildcommand.py", line 336, in build_source self.fetch_sources(source) File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources self.lrc.ensure_submodules(source.repo, source.sha1) File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules if (submod.url, submod.commit) not in done: AttributeError: 'tuple' object has no attribute 'url' This commit fixes the breakage and the baserock/tlsa/mason2 branch of definitions.git now builds again. Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
* Fix prototype of create_source_pool()Sam Thursfield2015-05-1413-23/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5