summaryrefslogtreecommitdiff
path: root/morphlib/sourceresolver.py
Commit message (Collapse)AuthorAgeFilesLines
* WIP Add support for multiple sources per chunkbaserock/pedroalvarez/defv8Tiago Gomes2016-03-251-15/+73
| | | | | | | | | TODO: - Add API to the cache server to retrieve a submodule commit from a given path. - Fix cross-bootstrap command. Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
* Use jsonschema to validate the morphsTiago Gomes2016-03-241-1/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Use jsonschema to do a first stage of validation of the morphologies. Example error messages: ERROR: strata/core.morph: core->build-depends: 'foo' is not of type 'array' ERROR: strata/core.morph: core: 'name' is a required property ERROR: strata/core.morph: core: Additional properties are not allowed ('extra-field' was unexpected) ERROR: strata/core.morph: core->chunks[0]->name: 1111 is not of type 'string' ERROR: strata/core.morph: core->chunks[0]: 'repo' is a required property on {'morph': 'strata/core/gdbm.morph', 'ref': 'e5faeaaf75ecfb705a9b643b3e4cb881ebb69f48', 'unpetrify-ref': 'gdbm-1.11', 'name': 'gdbm'} ERROR: strata/core.morph: core->chunks[0]: Additional properties are not allowed ('extra-field' was unexpected) on {'repo': 'upstream:gdbm-tarball', 'extra-field': None, 'name': 'gdbm', 'morph': 'strata/core/gdbm.morph', 'ref': 'e5faeaaf75ecfb705a9b643b3e4cb881ebb69f48', 'unpetrify-ref': 'gdbm-1.11'} Change-Id: If9fd488e16db35130d074492a93754a447ea98e1
* Remove support for definitions version 6Tiago Gomes2016-03-241-9/+2
| | | | Change-Id: I603b5172902b5249874ea3bb90de59db5a908185
* Unify local and remote repo cache modulesSam Thursfield2016-03-161-52/+28
| | | | | | | | | | | | | | | | | | | | | There's not really any reason you'd want to use the RemoteRepoCache class except as a workaround for the slow speed of some LocalRepoCache operations, so I can't see this ruining anyone's day. The main reason for doing this is so we can simply the sourceresolver code. One reason that the sourceresolver class is so hopelessly complicated is that it right now has to use two incompatible interfaces for Git repo caches. I've taken the opportunity to detangle the RepoCache class from the App class. Now all of the configuration for the RepoCache class is passed into the constructor explicitly. This makes the class usable from outside Morph: resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[]) repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver) Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
* Hide get_repo() and cache_repo() functions, always use get_updated_repo()Sam Thursfield2016-03-031-5/+1
| | | | | | | | | | This allows us to simplify a couple of places. I cannot think of a single situation where'd you want to get the cached copy of a repo, but not update it. Think about it -- the repo might be *years* behind the upstream remote. Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
* Error when different names in system/strataPedro Alvarez2016-02-251-6/+25
| | | | | | | | | | | | | | Now is an error when the name of a stratum in a system is different from the name of the stratum in the stratum file. Example of the error message: ERROR: Name 'build-essential-foo' doesn't match 'build-essential' in morphology: strata/build-essential.morph We are forcing the error here instead of a warning because this problem makes Morph create broken systems. https://storyboard.baserock.org/#!/story/15 Change-Id: Ifbe8d98e64a768e4e238d2213ffaf575789dd9e2
* Warning when different names in strata/chunksPedro Alvarez2016-02-251-0/+4
| | | | | | | | | Now is an warning when the name of a chunk in a stratum is different from the name of the chunk in the chunk file. Example of the warning message: WARNING: Name 'binutils' doesn't match 'binutils-foo' in morphology: strata/build-essential/binutils.morph Change-Id: I38f0ccff57fc25db12c572fdc8a01dbd6434f01c
* Avoid stack overflow in cachekeycomputerSam Thursfield2016-02-241-0/+12
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | I managed to trigger a stack overflow in Morph, by having two chunks with the same name and same build instructions in different strata (by mistake, obviously). The error before was this: .... File "/src/morph/morphlib/cachekeycomputer.py", line 77, in get_cache_id cacheid = self._calculate(source) File "/src/morph/morphlib/cachekeycomputer.py", line 86, in _calculate for a in source.dependencies], File "/src/morph/morphlib/cachekeycomputer.py", line 38, in compute_key ret = self._hash_id(self.get_cache_id(source)) File "/src/morph/morphlib/cachekeycomputer.py", line 77, in get_cache_id cacheid = self._calculate(source) File "/src/morph/morphlib/cachekeycomputer.py", line 83, in _calculate 'env': self._filterenv(self._build_env.env), File "/src/morph/morphlib/cachekeycomputer.py", line 32, in _filterenv return dict([(k, env[k]) for k in keys]) RuntimeError: maximum recursion depth exceeded while calling a Python object The error afterwards is this: 2016-02-18 13:53:14 Deciding on task order ERROR: There are multiple versions of component 'cython' Change-Id: I96a41cabe4d9aa81bddc5186c5248a636a0843c2
* sourceresolver: Use real name for sources without chunk filePedro Alvarez2016-02-161-11/+12
| | | | | | | | | | | | | | | | | | The filename was being used instead. I noticed this bug after seeing cache artifacts named like: de31c7bcac0240315bc4e0994b34f3757d37d35e4179cdbe0a8b2e60e26449f8.chunk.xorg-proto-x11proto.morph-bins de7b9f6f7a10b8dc04b52a08e74987df7da310b59dc25c6cf083b0381726e15f.chunk.networkx.morph-bins def910ed756be94c626d59d73f0770a445d62278a2eb46e031af42563691c6ed.chunk.pv.morph-bins e003da7f8c697286fa4d550299920160a6e76df449b7e434b270cd760c0c9304.chunk.cython.morph-bins e193576471c957387c9684c385a8c03db1e70618890210f6272c362ccc3e259b.chunk.xorg-proto-xcmiscproto.morph-bins e3b0b0dbb7b3de4bb8c72cd064408956296e290fc4b56acbbee241e2d78bf533.chunk.greenlet.morph-bins e3f09aa095e5839aa9bf7774720951c8b03d78300b056f7bbdeacdf4c5d5944d.chunk.libvorbis.morph-bins e437810247200713da17937181cc914e080ac7d6030451ccfaa86ddccf7a395b.chunk.python-requests.morph-bins e4b7de417c5691794f4ad6a6481cacc16242af072db39bcadede6ca2bf48241e.chunk.flac.morph-bins e4f1e6dbaeed869b445f9b45b98033b6e6a2952674b76b8465e40a67e675333c.chunk.Sub-Name.morph-bins Change-Id: I8a8697a12cf7e4c7891c93e4f774819fe2f8ea86
* Fix breakage when new build systems are added to DEFAULTSSam Thursfield2016-02-111-4/+8
| | | | | | | | | | | | | It turns out that one part of Morph was still using the predefined build systems that were built into the 'morphlib.buildsystem' Python module, instead of using the ones from the DEFAULTS file. This meant that Morph wasn't actually implementing Baserock definitions version 7 properly. https://storyboard.baserock.org/#!/story/72 Change-Id: I440a8f7455874a1b8491b48b0b122792aa8189ae
* Remove old cruftTiago Gomes2015-11-201-19/+0
| | | | | | | Static analysis showed this code is not being used, and we can always bring it back if necessary through git. Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
* Error on duplicate chunksRichard Ipsum2015-10-151-0/+32
| | | | | | | | | | | | | | This commit makes it an error for a system to contain duplicate chunk sources, example error message below, ERROR: Multiple `syslinux' chunks detected: upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux Multiple `nasm' chunks detected: upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
* Add support for Baserock definitions version 7Sam Thursfield2015-10-141-14/+55
| | | | | | | | | | | | | | | | | | | | | | This adds a new 'Defaults' class to represent definitions defaults The Python 'jsonschema' module is used to validate the contents of the Defaults file. This module is already included in Baserock 'build' and 'devel' reference systems by way of the 'openstack-common' stratum. This commit embeds a copy of the JSON-Schema schema for the DEFAULTS file. I think the canonical location of this schema should be in the reference definitions.git, for now. In future, the schemas should maybe have their own repos. Either way, Morph should embed a copy for the time being so that we are sure the schema matches how Morph expects to parse the file. Morph's automated tests are all updated to use definitions version 7. I removed most of the tests for built-in build systems, because the built-ins themselves are no longer part of Morph. Only the mechanism for defining them needs to be tested now. Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
* Remove support for Baserock definitions format versions 3, 4 and 5Sam Thursfield2015-09-181-250/+58
| | | | Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
* Move definitions VERSION parsing code into its own definitions_version moduleSam Thursfield2015-08-181-47/+3
| | | | | | | I need to use this outside of the 'sourceresolver' module. Also, the 'sourceresolver' module is too big. Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
* Log a message if a file isn't found in definitionsSam Thursfield2015-08-181-0/+1
| | | | | | Just for future debuggers. Change-Id: Ie915a9f47fbf3ca56d6e75b816eaa5a4fb36bc88
* Add support for Baserock definitions version 6Sam Thursfield2015-07-291-10/+34
| | | | Change-Id: I891d1b13ed0581b293fe6b09b3cc73af8fd81d67
* Remove support for Baserock definitions format versions 0, 1 and 2Sam Thursfield2015-07-231-11/+4
| | | | | | | | | | | The previous release of the Baserock reference systems (baserock-15.25 tag) used Baserock definitions version 3, so this version of Morph can still build the last release of the Baserock reference systems. In the course of fixing up the cmdtest tests to use version 5 definitions, I removed a couple of bits of redundant code. Change-Id: If9d480aa966d2ea5939cf04351f3d488cccea67f
* Improve error when local HEAD of definitions repo is not found in remoteSam Thursfield2015-06-161-2/+18
| | | | | | | | | | | | | | | | | | Old error: ERROR: Git directory /build/cache/gits/git___git_baserock_org_baserock_baserock_definitions has no commit at ref 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca^{commit}. New error: ERROR: Commit 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca wasn't found in the "origin" remote git://git.baserock.org/baserock/baserock/definitions. You either need to push your local commits on branch xxx to "origin", or use the --local-changes=include feature. Change-Id: I0c3658e9cd27c23f40653662ba7e4ba58b7892de
* sourceresolver: Support version 5Richard Maw2015-06-121-1/+1
| | | | | | | | This adds support for strip commands, and deployment extensions that rely on python libraries being available alongside the extensions in the definitions repository. Change-Id: I23dafd4968002037d182507762b09147f87bf469
* sourceresolver: Use versioned morphology loaderRichard Maw2015-06-121-1/+2
| | | | | | | | | | | | We don't use this globally, since other users of the MorphologyLoader don't need them to be aware of strip commands yet, and a more significant rework would be required to allow the injection of this context, because previously the MorpohologyLoader did not need any, so code which used the MorphologyLoader just created one as they needed it, rather than allowing one to be passed in where the context was available. Change-Id: Iea707a1cb2c631dd6f251b5b989d31e192f2387d
* sourceresolver: Re-order method definitions in use orderRichard Maw2015-06-121-74/+74
| | | | Change-Id: I66560dfb0e97c1d55a4e2399f81747644eb6d74e
* sourceresolver: Pass all hidden stateRichard Maw2015-06-121-84/+112
| | | | Change-Id: I72327a2fdb091fc742369af68c8ebfe16f34be5e
* sourceresolver: bind util.sanitise_morphology_pathRichard Maw2015-06-051-3/+4
| | | | | | It's too much of a mouthful to use a qualified import. Change-Id: Ide5695e49fd6188b520fecce9d6b076fd7280b8e
* sourceresolver: Remove cache_repo_locallyRichard Maw2015-06-051-20/+6
| | | | Change-Id: Ib8281c4030c42b61e298d6cc52f150a02384ed96
* sourceresolver: Use context managers for cleanup and cache writebackRichard Maw2015-06-051-17/+17
| | | | Change-Id: I895776bfd57ca6a14997c5535355b259d827b3a6
* sourceresolver: Pass build system directly to morphology creationRichard Maw2015-06-031-6/+7
| | | | | | | | We cache based on the name, but to get the name we acutally loaded the build system, so it's a waste of effort to cache by the name only to reload it later. Change-Id: Ia77e1c73d27005cd4cc6a3bb5553b44e8dfc75c1
* Bump supported definitions version to version 4Javier Jardón2015-05-181-1/+1
| | | | | | | | | | | | | A fix for the install-files extension has been merged in commit 93b034f045a2c7443122d7b082ad4460339907d0 This effectively make a new version of definitions if we want to use this (be able overwrite symlinks), in our definitions We will use this to create a symlink to /usr/lib/os-release from /etc/os-release using the install-essential-files extension Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
* Fix prototype of create_source_pool()Sam Thursfield2015-05-141-4/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* SourceResolver: Allow the resolution of multiple systemsRichard Maw2015-05-111-2/+4
| | | | | | | | | | | | | | | | The existing Source resolution code handles resolution of multiple systems sufficiently. It is not appropriate to then take this source pool and attempt to create a build graph from it though, as the logical structure of the input of what we want to build, and the logical structure of what we will produce are conflated in the Source object. If we do not intend to create a build graph from the Source Pool we generate, then it is an appropriate data structure that may be used to analyse changes in the input to a build. Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
* Bump supported definitions format version to 3baserock-definitions-v3Javier Jardón2015-04-291-1/+1
| | | | | | | | | | | | Since commit 154a760fb884cee14c2604b8bfbe52b0e7c0d4b1 morph suports a new architecture (armv5) This is effectivly a change in the definitions format, as old morph doesnt recognizse this architecture and will fail if a system with this architecture is added to definitions Change-Id: I2adc3a4b103b35c8ca5ec6d4034b797fd10f2485
* _parse_version_file tweakRichard Ipsum2015-04-131-8/+6
| | | | | | | | Swap nested ifs for and Also replace type() with isinstance() Also fix doc string Change-Id: I58177566f19b601bc734f7542c6a5dd52938063a
* Fix: check that path to chunk morph existsbaserock-definitions-v2Richard Ipsum2015-04-111-5/+33
| | | | | | | | | | | | | | | | | | | | | This fixes a bug that allows stratum definitions to carry paths to non-existent chunk morphs. This fix changes the interpretation of definitions so necessarily introduces a new definitions version (version 2) Assuming a morph field defined in a stratum: "morph: strata/cats/xattr.morph" and assuming "strata/cats/xattr.morph" does not exist, this patch will not alter current behaviour for definitions versions 0 and 1, besides producing a warning message like this: "Warning! `strata/cats/xattr.morph' referenced in `strata/swift.morph' does not exist" for definitions version 2 and greater the following error will be raised: "ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph" Change-Id: I4def5e92741cce25168f1038136503022ab27ffd
* Make _check_version_file return versionRichard Ipsum2015-04-111-6/+16
| | | | | | Also add a distinct error message for an invalid version file Change-Id: I674e6af284b33705ae8235054906f56b8cd8a130
* Make source resolver pass filenameRichard Ipsum2015-04-041-1/+1
| | | | | | | | | | | | | Passing the filename when loading a morphology lets us give a more useful error message: e.g. ERROR: Missing field name from morphology strata/build-essential.morph instead of ERROR: Missing field name from morphology string Change-Id: I2e5b33e8154567469aa56adbf942fb7b6e9dd1ac
* morphlib/sourceresolver.py: parse VERSION file in a functionJavier Jardón2015-03-171-6/+22
| | | | Change-Id: I6e714d1994632875a5a15f840fe8ab3a66dddc77
* morphlib/sourceresolver.py: Fail if morph doesnt support the version of ↵baserock-definitions-v1Javier Jardón2015-03-171-8/+12
| | | | | | | | | | | | | | | | | | | | | definitions format This patch will add the following restriction: if VERSION exist and its a YAML file and its a dict and has the key 'version' and the type stored in the 'version' key is an int and that int is not in the supported format, then fail. So, if someone is using 'version: 4' in VERSION, morph will fail (as opposed to current morph, that will not fail in the check but will likely fail when it tries to compile) Change-Id: I555f7e6018b9bdf18c80039df92a253acbd51c8c
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-1/+1
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Merge "Use the modern way of the GPL copyright header: URL instead real address"Javier Jardón2015-03-161-2/+1
|\
| * Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* | Fix call to status() in commit 364455436aded34ce6f843dfa68a32c14aaf01eaSam Thursfield2015-03-161-1/+2
| | | | | | | | I was meant to do this at merge time but didn't.
* | Minimise repo cache fetchesRichard Ipsum2015-03-071-21/+49
| | | | | | | | | | | | | | If we know the build system for a chunk is cached then we also know that it doesn't have a chunk morph, so we don't need to look for one. (We only cache the build systems of chunks that don't have chunk morphs)
* | Fix bug in build system cache accessRichard Ipsum2015-03-071-3/+11
|/ | | | | | | | | This fixes a bug that causes morph to run build system detection even though the build system has already been cached. The cache is accessed before the key has been computed (absref isn't known until after we've called resolve_ref) so it always misses.
* morphlib/sourceresolver.py: Check and parse VERSION filejjardon/versioning_supportJavier Jardón2015-03-031-0/+25
| | | | | | | | | | | Only fail if: - VERSION file exists - and its a yaml file - and its a dict - and has the key 'version' - and the contents of the key 'version' is an int - and that int is in the list of non_supported_versions (empty at the moment)
* morphlib/sourceresolver.py: Remove not used functionsJavier Jardón2015-02-271-16/+0
| | | | _get_morphology_from_definitions() and _get_morphology_from_repo()
* morphlib/sourceresolver.py: Do not duplicate what is already done in ↵Javier Jardón2015-02-271-8/+3
| | | | load_from_string()
* morphlib/sourceresolver.py: Add _get_file_contents()Javier Jardón2015-02-271-12/+23
| | | | And make _get_morphology() use it
* morphlib/sourceresolver.py: Add _get_file_contents_from_definitions()Javier Jardón2015-02-271-1/+10
| | | | And make _get_morphology_from_definitions() use it
* morphlib/sourceresolver.py: Add _get_file_contents_from_repo()Javier Jardón2015-02-271-5/+11
| | | | And make _get_morphology_from_repo() use it
* sourceresolver: Never assume that a given ref is present locallySam Thursfield2015-02-201-6/+8
| | | | | | | | | | | | | Checking that a given ref exists using `git rev-parse --verify 1234^{commit}` is a reasonably quick operation. As a rough guide, 1000 invocations took 1.6 seconds on my PC. The code is too fragile and hard to reason about if we assume that one function has been called before another so the repo will already be up to date. This should fix any spurious InvalidRefError exceptions that the build graph speedups branch has introduced.
View Lorry Controller Status