summaryrefslogtreecommitdiff
path: root/morphlib
Commit message (Collapse)AuthorAgeFilesLines
...
* sourceresolver: Pass all hidden stateRichard Maw2015-06-122-112/+164
| | | | Change-Id: I72327a2fdb091fc742369af68c8ebfe16f34be5e
* Fix test suiteSam Thursfield2015-06-101-2/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The test suite was failing in my Baserock 15.19 chroot with: FAILURE: test_commit_tree (gitdir_tests.GitDirectoryContentsTests) Traceback (most recent call last): File "morphlib/gitdir_tests.py", line 294, in test_commit_tree self.assertEqual(expected, gd.get_commit_contents(commit).split('\n')) AssertionError: Lists differ: ['tree 01d830ae9bae1a9970a9f14... != ['tree 01d830ae9bae1a9970a9f14... First differing element 2: author Author Name <author@email> 683074800 +0000 author Author Name <author@email> 683074800 +0100 ['tree 01d830ae9bae1a9970a9f1491c914ad177f8afce', 'parent 1240834484cebc4ba8cc40cd5b76f6dffd937a50', - 'author Author Name <author@email> 683074800 +0000', ? ^ + 'author Author Name <author@email> 683074800 +0100', ? ^ - 'committer Committer Name <committer@email> 683074800 +0000', ? ^ + 'committer Committer Name <committer@email> 683074800 +0100', ? ^ '', 'MESSAGE', ''] 1 failures, 0 errors 1543 excluded statements 57 excluded modules Time: 8.4 s Change-Id: If03e7ccbb5e3415eead9dcbcf908d3d2717e8fe0
* CacheKeyComputer: Only attempt to include commands present in morphologyRichard Maw2015-06-051-1/+2
| | | | | | | | The set of commands is potentially extendable, and it's easier to accept optional keys than get the list of potential fields from the build system. Change-Id: I0d876112db43637661e9548472a9fb191ea88268
* morphloader: Hoist buildsystem lookup outside of loopsRichard Maw2015-06-051-6/+6
| | | | Change-Id: If548a9962d5e70480f7eaba6ab6596465975e894
* morphloader: Add test-commands to key orderRichard Maw2015-06-051-0/+3
| | | | Change-Id: I01b3132c9fb27f1c0277ffe5eb0166e489312e38
* sourceresolver: bind util.sanitise_morphology_pathRichard Maw2015-06-051-3/+4
| | | | | | It's too much of a mouthful to use a qualified import. Change-Id: Ide5695e49fd6188b520fecce9d6b076fd7280b8e
* sourceresolver: Remove cache_repo_locallyRichard Maw2015-06-052-20/+9
| | | | Change-Id: Ib8281c4030c42b61e298d6cc52f150a02384ed96
* sourceresolver: Use context managers for cleanup and cache writebackRichard Maw2015-06-051-17/+17
| | | | Change-Id: I895776bfd57ca6a14997c5535355b259d827b3a6
* morphlib: Use new temp_dir helper context managerRichard Maw2015-06-054-59/+40
| | | | Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
* morphlib.util: Add temp_dir context managerRichard Maw2015-06-031-0/+16
| | | | | | | We commonly create a temporary directory and clean it up after we're done. This can be encapsulated into a context manager. Change-Id: Ie50de5c3954141a3e7fd40e4627fb743287ef61f
* sourceresolver: Pass build system directly to morphology creationRichard Maw2015-06-032-18/+19
| | | | | | | | We cache based on the name, but to get the name we acutally loaded the build system, so it's a waste of effort to cache by the name only to reload it later. Change-Id: Ia77e1c73d27005cd4cc6a3bb5553b44e8dfc75c1
* Add separate_mount_namespace kwarg to ext runnerRichard Ipsum2015-06-031-2/+8
| | | | | | Sometimes we don't want a separate mount namespace Change-Id: I9f2645f552c69090612aecd45104a0320cd436c2
* Do not call str() on None types in sanitize_envRichard Ipsum2015-05-291-1/+1
| | | | | | | | | | | | | | | | | | | | | | | For a cluster morph, - morph: systems/devel-system-x86_64-generic.morph deploy: self: type: ssh-rsync location: root@127.0.0.1 ATTACK_KITTENS: The yaml parser will assign the value None to the element with key 'ATTACK_KITTENS', sanitize_environment blindly calls str() on each value in the dictionary, the result is that our extensions receive an 'ATTACK_KITTENS' environment variable with the string value 'None', this is not useful and confuses anyone trying to do basic validation in their extension. This patch makes it so that any None types are replaced with the empty string, which seems a more sensible value for this case. Change-Id: I734d33e9304f0b7d4d590e464e350d2e5b4720d3
* Ignore the configuration extensions when generating the cache-keyPedro Alvarez2015-05-291-0/+3
| | | | | | | | System artifacts won't change because of changes in the configuration extensions. These extensions run at deployment time, therefore, they are ignored for generating the cache-key. Change-Id: I64f28605630bda4c27748197aef84809b4b1f86b
* Use keystoneclient python api in openstack.checkRichard Ipsum2015-05-291-29/+28
| | | | | | | Switching to the keystoneclient python api gives us a more reliable means of detecting auth failure. Change-Id: I5f734bbfe5568c855f524a3448357f7cf46ab254
* deploy: Don't imply --upgrade in subsystemsRichard Maw2015-05-271-2/+5
| | | | | | | | | | | --upgrade being passed on to subsystem deployments broke attempting to perfom an ssh-rsync upgrade with a system that had a sysroot in it. Therefore it is appropriate to prevent --upgrade being inherited like this, as it is seldom what is wanted, and if needed, it can be set in the cluster, or on the command-line separately. Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
* morph deploy: Allow overriding UPGRADE in definition or on command lineRichard Maw2015-05-271-4/+11
| | | | | | | | | | | | | | | | | | | Concievably, you may want to force a cluster to behave as an upgrade regardless of whether --upgrade is set or not. This would allow two use cases that are not currently possible: 1. To force a cluster to be upgrade-only, which would make our instructions for the upgrade-devel.morph cluster simpler. 2. To allow a cluster to deploy then immediately upgrade. One potential use for this would be to create a disk image with multiple versions on it. This also works as a work-around for --upgrade implying UPGRADE=yes for subsystem deployments, but that should be fixed separately. Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
* Ignore --upgrade flag rather than exitingRichard Ipsum2015-05-271-4/+0
| | | | | | | | Specifying --upgrade with the upgrade command is redundant, but it's not a contradiction, so we can safely ignore --upgrade if it happens to be specified with the upgrade command. Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
* Move 'git config' wrapper into its own classSam Thursfield2015-05-223-18/+75
| | | | | | | | | | We use 'git config' format config files outside Git repos, so it's useful to have a helper class independent of the GitDirectory class. This allows us to use it in the sysbranchdir.open() function to remove a hack. Change-Id: Ifa5e87f404d10666c98b9469079b7925d16becf6
* generate-manifest-csv: Catch NoRemote error when cloning lorries repoAdam Coldrick2015-05-201-9/+20
| | | | | | | | | | | If there is no $trove-id/local-config/lorries repository, and similarly if there is no baserock/local-config/lorries repository on the Trove defined by 'trove-host', the generate-manifest-csv command will fail with a NoRemote error from trying to clone a non-existent repository. This commit fixes this and instead outputs a warning if either of the repositories cannot be found. Change-Id: I8edd8484e45d8fc91021a9be34966c13d2beb705
* Give exact filename when an artifact is not found in the cache.Sam Thursfield2015-05-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Old message: ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. New message: ERROR: Deployment failed as 45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
* Bump supported definitions version to version 4Javier Jardón2015-05-181-1/+1
| | | | | | | | | | | | | A fix for the install-files extension has been merged in commit 93b034f045a2c7443122d7b082ad4460339907d0 This effectively make a new version of definitions if we want to use this (be able overwrite symlinks), in our definitions We will use this to create a symlink to /usr/lib/os-release from /etc/os-release using the install-essential-files extension Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
* distbuild: Condense Initiator class to remove unnecessary duplicationLauren Perry2015-05-151-4/+16
| | | | | | | | | Create an InitiatorCommand class that accepts message_type and status_text parameters to be used by the distbuild-list-jobs, distbuild-status and distbuild-cancel commands to send request messages to the distbuild network Change-Id: Ib686dcd7c370d802b612e9aaa1e3df76f0275fae
* Fix crash in handling of recursive submodulesSam Thursfield2015-05-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out I completely broke Morph's handling of recursive submodules in commit 0855c357e74e6dd7a. We didn't notice because nothing in the reference systems actually uses recursive submodules. However, building the baserock/tlsa/mason2 branch of definitions.git triggered the bug due to the delta:python-packages/gitpython chunk, which contains delta:python-packages/gitdb which in turn contains delta:python-packages/smmap. The error is this: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build self._build(source_pool, filename, component_names=component_names) File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build bc.build_in_order(component) File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order self.cache_or_build_source(s, build_env) File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source self.build_source(source, build_env) File "/src/morph/morphlib/buildcommand.py", line 336, in build_source self.fetch_sources(source) File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources self.lrc.ensure_submodules(source.repo, source.sha1) File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules if (submod.url, submod.commit) not in done: AttributeError: 'tuple' object has no attribute 'url' This commit fixes the breakage and the baserock/tlsa/mason2 branch of definitions.git now builds again. Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
* Fix prototype of create_source_pool()Sam Thursfield2015-05-1413-23/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* Explain how to cancel a distbuildAdam Coldrick2015-05-121-0/+5
| | | | | | | | | | | | | | | | Cancelling a distbuild with ctrl+c no longer cancels the build itself. This commit adds some output explaining what should be done to cancel the build as well as the local process. This commit also fixes a bug where the BuildStarted event would be sent each time a chunk finished building, since it was being sent in _queue_worker_builds. This is fixed by adding a new function to be called when the build graph annotation is complete which sends BuildStarted and then calls _queue_worker_builds, which no longer sends the BuildStarted event. Change-Id: I26ddea2c9080887f449e87004411ddffe4e583b7
* Remove mention of MorphologyFactory in the unit testsAdam Coldrick2015-05-121-2/+0
| | | | Change-Id: I27f5319721aa3e996c186f92a3c2296d6df4bedb
* Clean up artifact serialisationAdam Coldrick2015-05-121-9/+10
| | | | | | | We no longer serialise whole artifacts, so it doesn't make sense for things to still refer to serialise-artifact and similar. Change-Id: Id4d563a07041bbce77f13ac71dc3f7de39df5e23
* Move duplicate fix_chunk_build_mode function to a common locationAdam Coldrick2015-05-123-52/+33
| | | | Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
* Enable native-bootstrap to continue build after recovered from fault.Kejia Hu (Terry)2015-05-121-6/+13
| | | | | | | | | | | | | | | | | | | | | | | | The previous script creates new directory for the chunk it is going to build without checking whether the directory exists or not. It will fail back if the directory it attempted to creat exists. So if build failed, you always need to remove all .inst directories and let the native-bootstrap script build from the beginning. This patch improves this, and you can run the native bootstrap script direct after resumed without loss previous progress. A condition was added to determine whether previous native bootstrap script was terminated when it was building current chunk. As .build directory for certain chunk only exists during building phase of itself, it was created when started build, and cleaned up after building finished. If .inst for certain chunk exists, the .build directory doesn't, the building of the chunk should be succeed in previous build. The second go of native-bootstrap will skip all successful chunks and start where it left off. Change-Id: I91ae213ecc8c98808efdfd969624291e70f7e0fe
* morph: Add morph diff subcommandRichard Maw2015-05-111-0/+147
| | | | Change-Id: If74c97ccd81aa4d92ef247d2be59282f9552d4a1
* morphlib: Add utility module for parsing argv into lists of systemsRichard Maw2015-05-112-0/+142
| | | | | | | | | | | | | | The `morph anchor`, `morph build-morphology` and a potential `morph diff` command would all benefit from having a unified way to parse the argv for the systems it must operate on, especially in the case of the potential `morph diff`, which needs to be able to handle being given two sets of systems. `morph anchor` may make use of it now by passing the list of systems to the Source resolver, but `morph build-morphology` would have to iterate over the systems and graph each independently. Change-Id: I91ab4764ffca3aa16f144f89f68f37cc21b9f643
* SourceResolver: Allow the resolution of multiple systemsRichard Maw2015-05-112-5/+10
| | | | | | | | | | | | | | | | The existing Source resolution code handles resolution of multiple systems sufficiently. It is not appropriate to then take this source pool and attempt to create a build graph from it though, as the logical structure of the input of what we want to build, and the logical structure of what we will produce are conflated in the Source object. If we do not intend to create a build graph from the Source Pool we generate, then it is an appropriate data structure that may be used to analyse changes in the input to a build. Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
* LRC: Make get_updated_repo handle multiple refsRichard Maw2015-05-111-16/+27
| | | | | | | | | | Passing a single ref is still accepted, but if you have multiple refs you need to check from the same repository, it is more appropriate to do it in one call to get_updated_repo, as otherwise there will be unnecessary output about it not needing to be updated in multiple places. Change-Id: I194d7c0e3e84c4917518ba37672b508505c71b8e
* MorphologyLoader: Set filename attribute at parse timeRichard Maw2015-05-111-2/+3
| | | | Change-Id: I0e0b8d352eb4ef1ab6c50e0ba0162263d9bac09d
* morph anchor: Handle updating refsRichard Maw2015-05-111-5/+6
| | | | | | | | | | Previously it would not attempt to make commits it needed locally available if the commit was available on the remote repo cache. Now it will do the update if the commit is not available locally, and will obey --no-git-update. Change-Id: I80f1e351ce334641e2ef733fa4c9a6ab967f9b67
* morphlib.util: add word_join_listRichard Maw2015-05-111-0/+10
| | | | | | | This is useful for representing lists of items in status or exception messages. Change-Id: I530eecdc311ac77fca9922dab063f550ea810c06
* Fix mistake in sysroot.writeSam Thursfield2015-05-081-1/+1
| | | | | | The * should not be in quotes. Change-Id: Ieebdc7532ba1bff5ba9742f72440ed00b0c0de2a
* Raise an error if a stratum build-depends on itselfAdam Coldrick2015-05-082-0/+25
| | | | | | | | | | | | If a stratum build-depends on itself, the build graph calculation gets stuck in an infinite loop as it adds the same stratum to the queue of morphologies to inspect over and over again. This commit causes MorphologyLoader.validate_stratum to raise an error if a stratum contains itself in it's build-depends, as depending on itself makes no sense and will cause the above problem. Change-Id: I76df5b7d63d010ae3b17f72bfa39b273e74279dd
* Fix distbuild-morphologyAdam Coldrick2015-05-071-1/+1
| | | | | | This fixes an error caused by not enough parameters being given to the InitiatorBuildCommand constructor in distbuild-morphology. Change-Id: I133bd2f267fd06cfe88a1cbf4711cc79ad00d209
* Fix sysroot.write trying to overwrite existing filesSam Thursfield2015-05-072-11/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Commit 807e6a90876c5469d242 changed the behaviour of sysroot.write to avoid deleting the contents of the sysroot. This was done so if you accidentally set 'sysroot=/' it wouldn't delete your whole system. It turns out that SDK deployments like clusters/sdk-example-cluster.morph depended on the contents of the directory being deleted. The system armv7lhf-cross-toolchain-system-x86_64.morph has a bunch of files installed by the cross-toolchain in /usr/armv7lhf-baserock-linux-gnueabi/sys-root. Previously sysroot.write would delete these, but since commit 807e6a90876c5469d242 it would fail with several errors like: mv: can't rename '/src/tmp/deployments/usr/armv7l.../sys-root/sbin' If we use 'cp -a' instead of 'mv' then it is slower to deploy, but there are no errors. I am still unsure why files from the cross-toolchain system are installed and then deleted. Although this patch fixes the immediate issue, I don't know if it's the right thing to do. It seems better to not install those files in the first place, if we do not need them. This commit also removes the check for the sysroot target location being empty. This doesn't work, because it runs /before/ the system being deployed is unpacked. Change-Id: I10671c2f3b2060cfb36f880675b83351c6cdd807
* Make str() of a GitDirectory return its location.Sam Thursfield2015-05-071-0/+3
| | | | | | Handy for log messages. Change-Id: I4336866c456a6225a6f3ecbfef10dfc7b864ac59
* Make listing contents of local tarball cache more robustSam Thursfield2015-05-071-1/+12
| | | | | | | | | | | Previously it would crash with a backtrace if there were unexpected filenames in the directory. It's still not amazingly robust, but I don't have time to rewrite the whole thing now. This code seems to have ignored that cachefs.walkfiles() returns filenames with a preceeding '/', which I have fixed now. Change-Id: I98b3094bd6c82b26984513ee81a1eab9bf253a34
* Show progress of downloads when --verbose is passed, not --debugSam Thursfield2015-05-064-4/+5
| | | | | | | This commit undoes behaviour changes from commit aa6dfcbb70c03dfeb3f9af02. Change-Id: Ie677fb9c4e6bcd6edeba2cdd87f4f6125dcae7a4
* GitDir: Fix setting fetch url when push url is already on-diskRichard Maw2015-05-061-2/+2
| | | | | | | | | | | | 290483010cfc7945cd4483fadd1d98c3b83efb3 broke morph checkout, which uses set_fetch_url on a repository that has been cloned, hence has its origin remote url config already on-disk. The fix prevents it changing the push_url when the fetch_url is set, unless it is an unnamed remote, as if the config is on-disk, this already does the right thing. Change-Id: I6204f664407bab3d7f8ecf0fcca72f5015dee55e
* Add `morph anchor` commandRichard Maw2015-05-011-0/+226
| | | | Change-Id: If9d92d7c75b9c4276b69c482c076c6fc1d4ccbbf
* RemoteRefManager: Fail all ref updates when one failsRichard Maw2015-04-291-1/+27
| | | | | | | There's no API to do it in one push yet, but we can send a delete for all the branches that *did* commit. Change-Id: I671e9384b84657a3e9034d62818caa0ac0d8de1e
* GitDir: Set the fetch or push url when the other is setRichard Maw2015-04-291-0/+4
| | | | Change-Id: I500cb81fd0f133bd9f4e76d46bc0ff8a4f57fe50
* CachedRepo: Fix reference to _gitdirRichard Maw2015-04-291-1/+1
| | | | | | | 87f8dbefda89bf6cb9e4b88f23a5317b054da0d4 added a method that used _gitdir, but the patch to change it to gitdir was merged afterwards. Change-Id: Ibd9bff73a0fe69b3c1c2ff6acd02df6cea4a13de
* install-files.configure: make possible to overwrite symlinksJavier Jardón2015-04-301-1/+10
| | | | | | os.symlink will fail if the origin file/link already exist Change-Id: I8175c8dce699e55c3e39e35dfd45c0c19b8bd96d
View Lorry Controller Status