summaryrefslogtreecommitdiff
path: root/morphlib/sourceresolver.py
Commit message (Collapse)AuthorAgeFilesLines
* Remove old cruftTiago Gomes2015-11-201-19/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Error on duplicate chunksRichard Ipsum2015-10-151-0/+32
| | | | | | | | | | | | | | This commit makes it an error for a system to contain duplicate chunk sources, example error message below, ERROR: Multiple `syslinux' chunks detected: upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux Multiple `nasm' chunks detected: upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
* Add support for Baserock definitions version 7Sam Thursfield2015-10-141-14/+55
| | | | | | | | | | | | | | | | | | | | | | This adds a new 'Defaults' class to represent definitions defaults The Python 'jsonschema' module is used to validate the contents of the Defaults file. This module is already included in Baserock 'build' and 'devel' reference systems by way of the 'openstack-common' stratum. This commit embeds a copy of the JSON-Schema schema for the DEFAULTS file. I think the canonical location of this schema should be in the reference definitions.git, for now. In future, the schemas should maybe have their own repos. Either way, Morph should embed a copy for the time being so that we are sure the schema matches how Morph expects to parse the file. Morph's automated tests are all updated to use definitions version 7. I removed most of the tests for built-in build systems, because the built-ins themselves are no longer part of Morph. Only the mechanism for defining them needs to be tested now. Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
* Remove support for Baserock definitions format versions 3, 4 and 5Sam Thursfield2015-09-181-250/+58
| | | | Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
* Move definitions VERSION parsing code into its own definitions_version moduleSam Thursfield2015-08-181-47/+3
| | | | | | | I need to use this outside of the 'sourceresolver' module. Also, the 'sourceresolver' module is too big. Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
* Log a message if a file isn't found in definitionsSam Thursfield2015-08-181-0/+1
| | | | | | Just for future debuggers. Change-Id: Ie915a9f47fbf3ca56d6e75b816eaa5a4fb36bc88
* Add support for Baserock definitions version 6Sam Thursfield2015-07-291-10/+34
| | | | Change-Id: I891d1b13ed0581b293fe6b09b3cc73af8fd81d67
* Remove support for Baserock definitions format versions 0, 1 and 2Sam Thursfield2015-07-231-11/+4
| | | | | | | | | | | The previous release of the Baserock reference systems (baserock-15.25 tag) used Baserock definitions version 3, so this version of Morph can still build the last release of the Baserock reference systems. In the course of fixing up the cmdtest tests to use version 5 definitions, I removed a couple of bits of redundant code. Change-Id: If9d480aa966d2ea5939cf04351f3d488cccea67f
* Improve error when local HEAD of definitions repo is not found in remoteSam Thursfield2015-06-161-2/+18
| | | | | | | | | | | | | | | | | | Old error: ERROR: Git directory /build/cache/gits/git___git_baserock_org_baserock_baserock_definitions has no commit at ref 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca^{commit}. New error: ERROR: Commit 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca wasn't found in the "origin" remote git://git.baserock.org/baserock/baserock/definitions. You either need to push your local commits on branch xxx to "origin", or use the --local-changes=include feature. Change-Id: I0c3658e9cd27c23f40653662ba7e4ba58b7892de
* sourceresolver: Support version 5Richard Maw2015-06-121-1/+1
| | | | | | | | This adds support for strip commands, and deployment extensions that rely on python libraries being available alongside the extensions in the definitions repository. Change-Id: I23dafd4968002037d182507762b09147f87bf469
* sourceresolver: Use versioned morphology loaderRichard Maw2015-06-121-1/+2
| | | | | | | | | | | | We don't use this globally, since other users of the MorphologyLoader don't need them to be aware of strip commands yet, and a more significant rework would be required to allow the injection of this context, because previously the MorpohologyLoader did not need any, so code which used the MorphologyLoader just created one as they needed it, rather than allowing one to be passed in where the context was available. Change-Id: Iea707a1cb2c631dd6f251b5b989d31e192f2387d
* sourceresolver: Re-order method definitions in use orderRichard Maw2015-06-121-74/+74
| | | | Change-Id: I66560dfb0e97c1d55a4e2399f81747644eb6d74e
* sourceresolver: Pass all hidden stateRichard Maw2015-06-121-84/+112
| | | | Change-Id: I72327a2fdb091fc742369af68c8ebfe16f34be5e
* sourceresolver: bind util.sanitise_morphology_pathRichard Maw2015-06-051-3/+4
| | | | | | It's too much of a mouthful to use a qualified import. Change-Id: Ide5695e49fd6188b520fecce9d6b076fd7280b8e
* sourceresolver: Remove cache_repo_locallyRichard Maw2015-06-051-20/+6
| | | | Change-Id: Ib8281c4030c42b61e298d6cc52f150a02384ed96
* sourceresolver: Use context managers for cleanup and cache writebackRichard Maw2015-06-051-17/+17
| | | | Change-Id: I895776bfd57ca6a14997c5535355b259d827b3a6
* sourceresolver: Pass build system directly to morphology creationRichard Maw2015-06-031-6/+7
| | | | | | | | We cache based on the name, but to get the name we acutally loaded the build system, so it's a waste of effort to cache by the name only to reload it later. Change-Id: Ia77e1c73d27005cd4cc6a3bb5553b44e8dfc75c1
* Bump supported definitions version to version 4Javier Jardón2015-05-181-1/+1
| | | | | | | | | | | | | A fix for the install-files extension has been merged in commit 93b034f045a2c7443122d7b082ad4460339907d0 This effectively make a new version of definitions if we want to use this (be able overwrite symlinks), in our definitions We will use this to create a symlink to /usr/lib/os-release from /etc/os-release using the install-essential-files extension Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
* Fix prototype of create_source_pool()Sam Thursfield2015-05-141-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* SourceResolver: Allow the resolution of multiple systemsRichard Maw2015-05-111-2/+4
| | | | | | | | | | | | | | | | The existing Source resolution code handles resolution of multiple systems sufficiently. It is not appropriate to then take this source pool and attempt to create a build graph from it though, as the logical structure of the input of what we want to build, and the logical structure of what we will produce are conflated in the Source object. If we do not intend to create a build graph from the Source Pool we generate, then it is an appropriate data structure that may be used to analyse changes in the input to a build. Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
* Bump supported definitions format version to 3baserock-definitions-v3Javier Jardón2015-04-291-1/+1
| | | | | | | | | | | | Since commit 154a760fb884cee14c2604b8bfbe52b0e7c0d4b1 morph suports a new architecture (armv5) This is effectivly a change in the definitions format, as old morph doesnt recognizse this architecture and will fail if a system with this architecture is added to definitions Change-Id: I2adc3a4b103b35c8ca5ec6d4034b797fd10f2485
* _parse_version_file tweakRichard Ipsum2015-04-131-8/+6
| | | | | | | | Swap nested ifs for and Also replace type() with isinstance() Also fix doc string Change-Id: I58177566f19b601bc734f7542c6a5dd52938063a
* Fix: check that path to chunk morph existsbaserock-definitions-v2Richard Ipsum2015-04-111-5/+33
| | | | | | | | | | | | | | | | | | | | | This fixes a bug that allows stratum definitions to carry paths to non-existent chunk morphs. This fix changes the interpretation of definitions so necessarily introduces a new definitions version (version 2) Assuming a morph field defined in a stratum: "morph: strata/cats/xattr.morph" and assuming "strata/cats/xattr.morph" does not exist, this patch will not alter current behaviour for definitions versions 0 and 1, besides producing a warning message like this: "Warning! `strata/cats/xattr.morph' referenced in `strata/swift.morph' does not exist" for definitions version 2 and greater the following error will be raised: "ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph" Change-Id: I4def5e92741cce25168f1038136503022ab27ffd
* Make _check_version_file return versionRichard Ipsum2015-04-111-6/+16
| | | | | | Also add a distinct error message for an invalid version file Change-Id: I674e6af284b33705ae8235054906f56b8cd8a130
* Make source resolver pass filenameRichard Ipsum2015-04-041-1/+1
| | | | | | | | | | | | | Passing the filename when loading a morphology lets us give a more useful error message: e.g. ERROR: Missing field name from morphology strata/build-essential.morph instead of ERROR: Missing field name from morphology string Change-Id: I2e5b33e8154567469aa56adbf942fb7b6e9dd1ac
* morphlib/sourceresolver.py: parse VERSION file in a functionJavier Jardón2015-03-171-6/+22
| | | | Change-Id: I6e714d1994632875a5a15f840fe8ab3a66dddc77
* morphlib/sourceresolver.py: Fail if morph doesnt support the version of ↵baserock-definitions-v1Javier Jardón2015-03-171-8/+12
| | | | | | | | | | | | | | | | | | | | | definitions format This patch will add the following restriction: if VERSION exist and its a YAML file and its a dict and has the key 'version' and the type stored in the 'version' key is an int and that int is not in the supported format, then fail. So, if someone is using 'version: 4' in VERSION, morph will fail (as opposed to current morph, that will not fail in the check but will likely fail when it tries to compile) Change-Id: I555f7e6018b9bdf18c80039df92a253acbd51c8c
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-1/+1
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Merge "Use the modern way of the GPL copyright header: URL instead real address"Javier Jardón2015-03-161-2/+1
|\
| * Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* | Fix call to status() in commit 364455436aded34ce6f843dfa68a32c14aaf01eaSam Thursfield2015-03-161-1/+2
| | | | | | | | I was meant to do this at merge time but didn't.
* | Minimise repo cache fetchesRichard Ipsum2015-03-071-21/+49
| | | | | | | | | | | | | | If we know the build system for a chunk is cached then we also know that it doesn't have a chunk morph, so we don't need to look for one. (We only cache the build systems of chunks that don't have chunk morphs)
* | Fix bug in build system cache accessRichard Ipsum2015-03-071-3/+11
|/ | | | | | | | | This fixes a bug that causes morph to run build system detection even though the build system has already been cached. The cache is accessed before the key has been computed (absref isn't known until after we've called resolve_ref) so it always misses.
* morphlib/sourceresolver.py: Check and parse VERSION filejjardon/versioning_supportJavier Jardón2015-03-031-0/+25
| | | | | | | | | | | Only fail if: - VERSION file exists - and its a yaml file - and its a dict - and has the key 'version' - and the contents of the key 'version' is an int - and that int is in the list of non_supported_versions (empty at the moment)
* morphlib/sourceresolver.py: Remove not used functionsJavier Jardón2015-02-271-16/+0
| | | | _get_morphology_from_definitions() and _get_morphology_from_repo()
* morphlib/sourceresolver.py: Do not duplicate what is already done in ↵Javier Jardón2015-02-271-8/+3
| | | | load_from_string()
* morphlib/sourceresolver.py: Add _get_file_contents()Javier Jardón2015-02-271-12/+23
| | | | And make _get_morphology() use it
* morphlib/sourceresolver.py: Add _get_file_contents_from_definitions()Javier Jardón2015-02-271-1/+10
| | | | And make _get_morphology_from_definitions() use it
* morphlib/sourceresolver.py: Add _get_file_contents_from_repo()Javier Jardón2015-02-271-5/+11
| | | | And make _get_morphology_from_repo() use it
* sourceresolver: Never assume that a given ref is present locallySam Thursfield2015-02-201-6/+8
| | | | | | | | | | | | | Checking that a given ref exists using `git rev-parse --verify 1234^{commit}` is a reasonably quick operation. As a rough guide, 1000 invocations took 1.6 seconds on my PC. The code is too fragile and hard to reason about if we assume that one function has been called before another so the repo will already be up to date. This should fix any spurious InvalidRefError exceptions that the build graph speedups branch has introduced.
* sourceresolver: Factor out 'cache repo locally' code into a functionSam Thursfield2015-02-201-25/+19
| | | | | | | Also, move the repo.update() call into the 'fetch from tarball' code path in the localrepocache module -- there's no need to update straight after doing a `git clone`, but we do need to do one if we got the repo from a `git archive` tarball that may be out of date.
* sourceresolver: Simplify some code pathsSam Thursfield2015-02-201-33/+35
| | | | | It turns out that always looking for the chunk morph in the definitions repo allows us to make the code a little less nasty. Bonus!
* sourceresolver: Fix InvalidRefError when local git cache is out of dateSam Thursfield2015-02-201-7/+11
| | | | | | | | | | | | | Code in sourceresolver.py assumed that resolve_ref() would be called for the chunk repo before get_morphology() was called, so the repo would always be up to date. This wasn't actually true. If your local repo cache was out of date, you might see the following sort of error: InvalidRefError: Git directory /src/cache/gits/git___git_baserock_org_delta_usbutils has no commit at ref c37f146eb2c6642c600f1b025a6d56996b0697ff^{tree}.
* sourceresolver: Always try to read chunk morph files from definitionsSam Thursfield2015-02-201-29/+40
| | | | | | | | | | | | | | | | | | | | | This fixes the 'Morph ignores the chunk morph file I added to definitions' regression introduced in the recent build-graph speedups branch. The new 'detected-chunk-buildsystems' cache associates a (chunk repo, chunk ref) pair with the auto-detected build system. When calculating the build graph, if the is an auto-detected build system known already, that will be used instead of looking for a chunk morphology. There's a flaw here: if the user added or changed the chunk morphology in the definitions repo, the *chunk* ref won't necessarily have changed -- so Morph will ignore the user's changes, if it had already cached an autodetected buildsystem for that repo/ref pair. This doesn't cost much in terms of speed because we're just reading a file from disk. We can still avoid looking in the chunk repo for a chunk morph, because the chunk ref would be different if something had changed in the chunk repo.
* sourceresolver: Fix LsTreeError raised when constructing build graphSam Thursfield2015-02-201-7/+19
| | | | | | | | | | | | | | | | If you had a repo that was not in your local or remote cache in your definitions, Morph might raise LsTreeError and abort. The code was assuming the repo would already be cached in the local cache by the time it got to detecting build systems, but that's not always true -- if the (reponame, ref) pair was already in the 'resolved_trees' cache but the repo had been deleted from the local Git cache, to name one possibility. Also, the remoterepocache.ls_tree operation can fail if the repo is not hosted in the Trove, so we shouldn't abort the program in that case. Finally, this patch removes an unused variable.
* Add tests for sourceresolverAdam Coldrick2015-02-181-10/+10
| | | | | | | | | This only adds tests for the bits which were moved from morphologyfactory into sourceresolver, namely detection of build systems and the '_get_morphology()' function. These are just the morphologyfactory tests reworked slightly to work properly with the modified API.
* Read files from a local clone of definitions where possibleSam Thursfield2015-02-181-1/+30
| | | | | | | | | | | | | | | | | Most morphologies involved in a build are in the definitions repo these days. Currently we read each of them using `git cat-file`, which is slow. It's quicker to check out all the files in one go to a temporary directory and then read them from there. With the current workflow users often have definitions.git checked out on disk. It seems strange to not just read the files from there. There are two reasons why I don't want to do that yet: - there are commands which don't run inside a system branch, which would be broken if we expected to always be in a system branch - there may be local changes in the checked-out repo, and it takes around 5 seconds on each build to check if there aren't any local changes. It actually seems faster to just check out a known clean version from the cache.
* Split up traverse_morphs to improve readabilitySam Thursfield2015-02-181-61/+79
|
* Add an LRU cache for detected build-systemsSam Thursfield2015-02-181-19/+101
| | | | | | | This will speed up builds of chunks which don't have a chunk morph. It won't have much (if any) effect on the speed of the first build, but subsequent builds will be much faster as we won't have to query the git cache.
* Add an LRU cache for resolved tree refsSam Thursfield2015-02-181-29/+72
| | | | | | | | | | | | This uses the PyLRU module, from: <https://pypi.python.org/pypi/pylru/1.0.6>. Python 3.2 and newer provide a built-in LRU cache, but this is specifically for in-memory use. See <http://bugs.python.org/issue17528>. Git commits are immutable, so caching information about their contents is fairly easy and trouble-free. There's no danger of the cache becoming stale.
View Lorry Controller Status