| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Recently something regressed which meant that when building a missing
file, the result was this long unhelpful traceback:
2016-04-27 12:23:36 Deciding on task order
Traceback (most recent call last):
...
File "/src/morph/morphlib/sourceresolver.py", line 506, in create_source_pool
definitions_original_ref=original_ref)
File "/src/morph/morphlib/sourceresolver.py", line 440, in add_morphs_to_source_pool
add_to_pool, predefined_split_rules)
File "/src/morph/morphlib/sourceresolver.py", line 287, in _process_definitions_with_children
morphology = get_morphology(filename)
File "/src/morph/morphlib/sourceresolver.py", line 282, in get_morphology
filename)
File "/src/morph/morphlib/sourceresolver.py", line 255, in _get_morphology
morph = morph_loader.load_from_string(text, filename)
File "/src/morph/morphlib/morphloader.py", line 395, in load_from_string
obj = yaml.safe_load(string)
...
File "/usr/lib/python2.7/site-packages/yaml/reader.py", line 178, in update_raw
data = self.stream.read(size)
AttributeError: 'NoneType' object has no attribute 'read'
Now the error is:
ERROR: Couldn't find definition file to build: missing-file.morph
Change-Id: Iabb315c9d0ac5c7c50003db36a1b21e86fb23223
|
|
|
|
|
|
|
|
|
| |
This code is a rework from changes done by:
- Tiago Gomes <tiago.gomes@codethink.co.uk>
https://storyboard.baserock.org/#!/story/86
Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
|
|
|
|
| |
Change-Id: I603b5172902b5249874ea3bb90de59db5a908185
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's not really any reason you'd want to use the RemoteRepoCache
class except as a workaround for the slow speed of some LocalRepoCache
operations, so I can't see this ruining anyone's day.
The main reason for doing this is so we can simply the sourceresolver
code. One reason that the sourceresolver class is so hopelessly
complicated is that it right now has to use two incompatible interfaces
for Git repo caches.
I've taken the opportunity to detangle the RepoCache class from the
App class. Now all of the configuration for the RepoCache class is
passed into the constructor explicitly. This makes the class usable from
outside Morph:
resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[])
repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver)
Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
|
|
|
|
|
|
|
|
|
|
| |
This allows us to simplify a couple of places.
I cannot think of a single situation where'd you want to get the cached
copy of a repo, but not update it. Think about it -- the repo might be
*years* behind the upstream remote.
Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now is an error when the name of a stratum in a system is different from
the name of the stratum in the stratum file. Example of the error message:
ERROR: Name 'build-essential-foo' doesn't match 'build-essential' in morphology: strata/build-essential.morph
We are forcing the error here instead of a warning because this problem
makes Morph create broken systems.
https://storyboard.baserock.org/#!/story/15
Change-Id: Ifbe8d98e64a768e4e238d2213ffaf575789dd9e2
|
|
|
|
|
|
|
|
|
| |
Now is an warning when the name of a chunk in a stratum is different from
the name of the chunk in the chunk file. Example of the warning message:
WARNING: Name 'binutils' doesn't match 'binutils-foo' in morphology: strata/build-essential/binutils.morph
Change-Id: I38f0ccff57fc25db12c572fdc8a01dbd6434f01c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I managed to trigger a stack overflow in Morph, by having two chunks with
the same name and same build instructions in different strata (by mistake,
obviously).
The error before was this:
....
File "/src/morph/morphlib/cachekeycomputer.py", line 77, in get_cache_id
cacheid = self._calculate(source)
File "/src/morph/morphlib/cachekeycomputer.py", line 86, in _calculate
for a in source.dependencies],
File "/src/morph/morphlib/cachekeycomputer.py", line 38, in compute_key
ret = self._hash_id(self.get_cache_id(source))
File "/src/morph/morphlib/cachekeycomputer.py", line 77, in get_cache_id
cacheid = self._calculate(source)
File "/src/morph/morphlib/cachekeycomputer.py", line 83, in _calculate
'env': self._filterenv(self._build_env.env),
File "/src/morph/morphlib/cachekeycomputer.py", line 32, in _filterenv
return dict([(k, env[k]) for k in keys])
RuntimeError: maximum recursion depth exceeded while calling a Python object
The error afterwards is this:
2016-02-18 13:53:14 Deciding on task order
ERROR: There are multiple versions of component 'cython'
Change-Id: I96a41cabe4d9aa81bddc5186c5248a636a0843c2
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The filename was being used instead. I noticed this bug after seeing
cache artifacts named like:
de31c7bcac0240315bc4e0994b34f3757d37d35e4179cdbe0a8b2e60e26449f8.chunk.xorg-proto-x11proto.morph-bins
de7b9f6f7a10b8dc04b52a08e74987df7da310b59dc25c6cf083b0381726e15f.chunk.networkx.morph-bins
def910ed756be94c626d59d73f0770a445d62278a2eb46e031af42563691c6ed.chunk.pv.morph-bins
e003da7f8c697286fa4d550299920160a6e76df449b7e434b270cd760c0c9304.chunk.cython.morph-bins
e193576471c957387c9684c385a8c03db1e70618890210f6272c362ccc3e259b.chunk.xorg-proto-xcmiscproto.morph-bins
e3b0b0dbb7b3de4bb8c72cd064408956296e290fc4b56acbbee241e2d78bf533.chunk.greenlet.morph-bins
e3f09aa095e5839aa9bf7774720951c8b03d78300b056f7bbdeacdf4c5d5944d.chunk.libvorbis.morph-bins
e437810247200713da17937181cc914e080ac7d6030451ccfaa86ddccf7a395b.chunk.python-requests.morph-bins
e4b7de417c5691794f4ad6a6481cacc16242af072db39bcadede6ca2bf48241e.chunk.flac.morph-bins
e4f1e6dbaeed869b445f9b45b98033b6e6a2952674b76b8465e40a67e675333c.chunk.Sub-Name.morph-bins
Change-Id: I8a8697a12cf7e4c7891c93e4f774819fe2f8ea86
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out that one part of Morph was still using the predefined build
systems that were built into the 'morphlib.buildsystem' Python module,
instead of using the ones from the DEFAULTS file.
This meant that Morph wasn't actually implementing Baserock definitions
version 7 properly.
https://storyboard.baserock.org/#!/story/72
Change-Id: I440a8f7455874a1b8491b48b0b122792aa8189ae
|
|
|
|
|
|
|
| |
Static analysis showed this code is not being used, and we can always
bring it back if necessary through git.
Change-Id: Id8bf7d73436b5c3d0dfe050befaae034a05afc86
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit makes it an error for a system to contain duplicate chunk sources,
example error message below,
ERROR: Multiple `syslinux' chunks detected:
upstream:syslinux|d715b39c0801ecea5e52f9029cea7c76320f93cf|strata/bsp-x86_both-tools/syslinux.morph|syslinux
upstream:syslinux|2aab8555987b547b617cbb887e61083fece01541|strata/bsp-x86_64-generic/syslinux.morph|syslinux
Multiple `nasm' chunks detected:
upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_64-generic/nasm.morph|nasm
upstream:nasm|78bdad3d14fb875d5f2062957e326ba2a9e4ccb0|strata/bsp-x86_both-tools/nasm.morph|nasm
Change-Id: I1d1539a46ce6eb098d3a559295ab9a08d6d2865c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new 'Defaults' class to represent definitions defaults
The Python 'jsonschema' module is used to validate the contents of the
Defaults file. This module is already included in Baserock 'build' and
'devel' reference systems by way of the 'openstack-common' stratum.
This commit embeds a copy of the JSON-Schema schema for the DEFAULTS
file. I think the canonical location of this schema should be in the
reference definitions.git, for now. In future, the schemas should maybe
have their own repos. Either way, Morph should embed a copy for the time
being so that we are sure the schema matches how Morph expects to parse
the file.
Morph's automated tests are all updated to use definitions version 7.
I removed most of the tests for built-in build systems, because the
built-ins themselves are no longer part of Morph. Only the mechanism for
defining them needs to be tested now.
Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
|
|
|
|
| |
Change-Id: Iad95af65bd5c528d2e72f5b2ffa80a01152f50ff
|
|
|
|
|
|
|
| |
I need to use this outside of the 'sourceresolver' module. Also, the
'sourceresolver' module is too big.
Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
|
|
|
|
|
|
| |
Just for future debuggers.
Change-Id: Ie915a9f47fbf3ca56d6e75b816eaa5a4fb36bc88
|
|
|
|
| |
Change-Id: I891d1b13ed0581b293fe6b09b3cc73af8fd81d67
|
|
|
|
|
|
|
|
|
|
|
| |
The previous release of the Baserock reference systems (baserock-15.25
tag) used Baserock definitions version 3, so this version of Morph can
still build the last release of the Baserock reference systems.
In the course of fixing up the cmdtest tests to use version 5
definitions, I removed a couple of bits of redundant code.
Change-Id: If9d480aa966d2ea5939cf04351f3d488cccea67f
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Old error:
ERROR: Git directory
/build/cache/gits/git___git_baserock_org_baserock_baserock_definitions
has no commit at ref 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca^{commit}.
New error:
ERROR: Commit 5046c3a0e4fd587c23f7c6a9e0d0b50d1914dcca wasn't found
in the "origin" remote
git://git.baserock.org/baserock/baserock/definitions. You either need
to push your local commits on branch xxx to "origin", or use the
--local-changes=include feature.
Change-Id: I0c3658e9cd27c23f40653662ba7e4ba58b7892de
|
|
|
|
|
|
|
|
| |
This adds support for strip commands, and deployment extensions that
rely on python libraries being available alongside the extensions in the
definitions repository.
Change-Id: I23dafd4968002037d182507762b09147f87bf469
|
|
|
|
|
|
|
|
|
|
|
|
| |
We don't use this globally, since other users of the MorphologyLoader
don't need them to be aware of strip commands yet, and a more
significant rework would be required to allow the injection of this
context, because previously the MorpohologyLoader did not need any, so
code which used the MorphologyLoader just created one as they needed it,
rather than allowing one to be passed in where the context was
available.
Change-Id: Iea707a1cb2c631dd6f251b5b989d31e192f2387d
|
|
|
|
| |
Change-Id: I66560dfb0e97c1d55a4e2399f81747644eb6d74e
|
|
|
|
| |
Change-Id: I72327a2fdb091fc742369af68c8ebfe16f34be5e
|
|
|
|
|
|
| |
It's too much of a mouthful to use a qualified import.
Change-Id: Ide5695e49fd6188b520fecce9d6b076fd7280b8e
|
|
|
|
| |
Change-Id: Ib8281c4030c42b61e298d6cc52f150a02384ed96
|
|
|
|
| |
Change-Id: I895776bfd57ca6a14997c5535355b259d827b3a6
|
|
|
|
|
|
|
|
| |
We cache based on the name, but to get the name we acutally loaded the
build system, so it's a waste of effort to cache by the name only to
reload it later.
Change-Id: Ia77e1c73d27005cd4cc6a3bb5553b44e8dfc75c1
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
A fix for the install-files extension has been merged in
commit 93b034f045a2c7443122d7b082ad4460339907d0
This effectively make a new version of definitions if we want to
use this (be able overwrite symlinks), in our definitions
We will use this to create a symlink to /usr/lib/os-release from
/etc/os-release using the install-essential-files extension
Change-Id: I615df922f2e720da411221e61bb838c8c5c63fef
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes an issue in `morph list-artifacts` and probably other commands:
ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts
repo, ref, system_filename)
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system
status_cb=self.app.status)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool
definitions_original_ref=original_ref)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs
definitions_absref, definitions_tree, visit)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children
definitions_queue = collections.deque(system_filenames)
TypeError: 'NoneType' object is not iterable
The issue is that the sourceresolver.create_source_pool() function did
not work correctly if passed a single filename. This was a regression from
commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0.
In order to have a logical function prototype, the create_source_pool()
function now only takes a list of systems, instead of taking either a
single system or a list.
Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The existing Source resolution code handles resolution of multiple
systems sufficiently.
It is not appropriate to then take this source pool and attempt to
create a build graph from it though, as the logical structure of the
input of what we want to build, and the logical structure of what we
will produce are conflated in the Source object.
If we do not intend to create a build graph from the Source Pool we
generate, then it is an appropriate data structure that may be used to
analyse changes in the input to a build.
Change-Id: If8e4a726f16f8aca000e59ecbbeb7d926cc08391
|
|
|
|
|
|
|
|
|
|
|
|
| |
Since commit 154a760fb884cee14c2604b8bfbe52b0e7c0d4b1
morph suports a new architecture (armv5)
This is effectivly a change in the definitions format,
as old morph doesnt recognizse this architecture and
will fail if a system with this architecture is added
to definitions
Change-Id: I2adc3a4b103b35c8ca5ec6d4034b797fd10f2485
|
|
|
|
|
|
|
|
| |
Swap nested ifs for and
Also replace type() with isinstance()
Also fix doc string
Change-Id: I58177566f19b601bc734f7542c6a5dd52938063a
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes a bug that allows stratum definitions
to carry paths to non-existent chunk morphs.
This fix changes the interpretation of definitions
so necessarily introduces a new definitions version (version 2)
Assuming a morph field defined in a stratum: "morph: strata/cats/xattr.morph"
and assuming "strata/cats/xattr.morph" does not exist, this patch will not
alter current behaviour for definitions versions 0 and 1,
besides producing a warning message like this:
"Warning! `strata/cats/xattr.morph' referenced in `strata/swift.morph' does not exist"
for definitions version 2 and greater the following error will be raised:
"ERROR: Couldn't find morphology: strata/cats/xattr.morph referenced in strata/swift.morph"
Change-Id: I4def5e92741cce25168f1038136503022ab27ffd
|
|
|
|
|
|
| |
Also add a distinct error message for an invalid version file
Change-Id: I674e6af284b33705ae8235054906f56b8cd8a130
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Passing the filename when loading a morphology
lets us give a more useful error message:
e.g. ERROR: Missing field name from morphology strata/build-essential.morph
instead of
ERROR: Missing field name from morphology string
Change-Id: I2e5b33e8154567469aa56adbf942fb7b6e9dd1ac
|
|
|
|
| |
Change-Id: I6e714d1994632875a5a15f840fe8ab3a66dddc77
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
definitions format
This patch will add the following restriction:
if VERSION exist
and its a YAML file
and its a dict
and has the key 'version'
and the type stored in the 'version' key is an int
and that int is not in the supported format,
then fail.
So, if someone is using 'version: 4' in VERSION, morph will fail (as
opposed to current morph, that will not fail in the check but will
likely fail when it tries to compile)
Change-Id: I555f7e6018b9bdf18c80039df92a253acbd51c8c
|
|
|
|
| |
Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
|
|\ |
|
| |
| |
| |
| | |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
| |
| |
| |
| | |
I was meant to do this at merge time but didn't.
|
| |
| |
| |
| |
| |
| |
| | |
If we know the build system for a chunk is cached then we also know that
it doesn't have a chunk morph, so we don't need to look for one.
(We only cache the build systems of chunks that don't have chunk morphs)
|
|/
|
|
|
|
|
|
|
| |
This fixes a bug that causes morph to run build system detection
even though the build system has already been cached.
The cache is accessed before the key has been computed
(absref isn't known until after we've called resolve_ref)
so it always misses.
|
|
|
|
|
|
|
|
|
|
|
| |
Only fail if:
- VERSION file exists
- and its a yaml file
- and its a dict
- and has the key 'version'
- and the contents of the key 'version' is an int
- and that int is in the list of non_supported_versions (empty
at the moment)
|
|
|
|
| |
_get_morphology_from_definitions() and _get_morphology_from_repo()
|
|
|
|
| |
load_from_string()
|
|
|
|
| |
And make _get_morphology() use it
|
|
|
|
| |
And make _get_morphology_from_definitions() use it
|
|
|
|
| |
And make _get_morphology_from_repo() use it
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Checking that a given ref exists using `git rev-parse --verify
1234^{commit}` is a reasonably quick operation. As a rough guide, 1000
invocations took 1.6 seconds on my PC.
The code is too fragile and hard to reason about if we assume that one
function has been called before another so the repo will already be up
to date.
This should fix any spurious InvalidRefError exceptions that the
build graph speedups branch has introduced.
|