| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
| |
Change-Id: I2d1da2c094a80b14d80e8b461072ec4097f946b9
|
|
|
|
| |
Change-Id: I0e42ee2fa7545aadd8483ffd9f23ebeaf1d8f2c9
|
|
|
|
| |
Change-Id: Id35fe6b9b21ac0e98f5a2675e3fa8f5fb6f8e669
|
|
|
|
| |
Change-Id: I800edd478b1ea0afa495e765583989094fb106f0
|
|
|
|
| |
Change-Id: I4b03c2d118c046ca9795a8462d4dd2cafcbba196
|
|
|
|
| |
Change-Id: I853be29a778a198751f7ab948009122a633aa899
|
|
|
|
| |
Change-Id: Ia0315f76cb993d29e7cf20a89d13485f9d122024
|
|
|
|
| |
Change-Id: I0453930690b04eeba808b6bf46375c2a06df227e
|
|
|
|
| |
Change-Id: Ib0af85cd3529d865a8b17a5461acdeecccb5f52d
|
|
|
|
|
|
|
|
|
|
| |
Since we build chunks using the '-e' flag in `sh`, we might have
different artifacts as result of building a chunk. It's better
we change the cache-key of these so that we ensure that everything
rebuilds, and we don't find the case where something that doesn't
build, is already cached.
Change-Id: Ic0fe8f9cf36b7787ebb08bd8200e0f1ea4cb01ef
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some failing commands in a yaml block were being ignored. For
example the following commands...
pre-configure-commands:
- |
cp nonexistent foo
echo foo
... where giving the following result without making the build
fail:
### PRE-CONFIGURE-COMMANDS ###
+ cp nonexistent foo
cp: can't stat 'nonexistent': No such file or directory
+ echo foo
foo
This commit also adjusts the test-shell.c to allow the use of '-e'
flag. No other changes because the shell was already failing after
the first failure found.
Change-Id: Ie78b25a99f1b5d6d0d26be74ddc377025dff69fd
|
|
|
|
| |
Change-Id: I5d8c2f0979f086f6abeec4421dded440b5948706
|
|
|
|
| |
Change-Id: I1f41d544a202aa3e54c5d6fba62a629dddcf6c1c
|
|
|
|
| |
Change-Id: I48cebf0007e602c0262b667643c3290d41c7b485
|
|
|
|
| |
Change-Id: Ie905724b0e77a7f9b3a2ea1af348aa0f53ae26b8
|
|
|
|
|
|
|
|
| |
These new options will be used by subcommands that use
a morphology file to allow them to use remote files without
having to clone the repository.
Change-Id: I0f35068627b7f7ba766ac015f0e1debff94abc57
|
|
|
|
|
|
|
| |
The option '--ref' is going to be used for other things
in the tool.
Change-Id: Icbee89c4693ca53618b9c039120fbb0c6636285d
|
|
|
|
| |
Change-Id: I3a87c82b0c50a648edd7c6487e0269116da35c20
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
In WorkerBuildQueuer._handle_request we were only considering
running jobs when creating a new one to not create duplicates.
This was making `morph distbuild` build some components more than
one time.
In this commit we also start considering active jobs when cancelling
them.
Change-Id: Ib0a7296d453ccd0b8c636c7506d9f1da82acc462
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is done by adding an status update when a command fails. An example
of the new output is:
2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Creating staging area
2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Starting actual build: stage2-libstdc++ fc4868e
2016-06-03 10:35:04 [Build 5/133] [stage2-libstdc++] Extracting /src/gcc-tarball into /src/tmp/staging/tmpE5b84y/stage2-libstdc++.build
2016-06-03 10:35:38 [Build 5/133] [stage2-libstdc++] Running configure-commands
2016-06-03 10:35:55 [Build 5/133] [stage2-libstdc++] Running build-commands
2016-06-03 10:36:14 [Build 5/133] [stage2-libstdc++] Error when running build-commands
Change-Id: Iac94794311e649d13c534f453d3473ac8003e8f4
|
|
|
|
| |
Change-Id: I711953b829c786911c46e413d2b7af3427b7ba26
|
|
|
|
| |
Change-Id: I9ae5a846e20c0d32047a3d4fd5892e2f632f0cbe
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Recently something regressed which meant that when building a missing
file, the result was this long unhelpful traceback:
2016-04-27 12:23:36 Deciding on task order
Traceback (most recent call last):
...
File "/src/morph/morphlib/sourceresolver.py", line 506, in create_source_pool
definitions_original_ref=original_ref)
File "/src/morph/morphlib/sourceresolver.py", line 440, in add_morphs_to_source_pool
add_to_pool, predefined_split_rules)
File "/src/morph/morphlib/sourceresolver.py", line 287, in _process_definitions_with_children
morphology = get_morphology(filename)
File "/src/morph/morphlib/sourceresolver.py", line 282, in get_morphology
filename)
File "/src/morph/morphlib/sourceresolver.py", line 255, in _get_morphology
morph = morph_loader.load_from_string(text, filename)
File "/src/morph/morphlib/morphloader.py", line 395, in load_from_string
obj = yaml.safe_load(string)
...
File "/usr/lib/python2.7/site-packages/yaml/reader.py", line 178, in update_raw
data = self.stream.read(size)
AttributeError: 'NoneType' object has no attribute 'read'
Now the error is:
ERROR: Couldn't find definition file to build: missing-file.morph
Change-Id: Iabb315c9d0ac5c7c50003db36a1b21e86fb23223
|
|
|
|
| |
Change-Id: Ifec7f4274079d9c8f2a75bdedc8a253568a6ec78
|
|
|
|
|
|
| |
Fixes https://storyboard.baserock.org/#!/story/83
Change-Id: Iae3f99b4189f513fa9b02d904c4fcdc0f52bbfb6
|
|
|
|
| |
Change-Id: I8d9330eb5f72a8f3f106aae4387eb8321f6126e5
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It seems to be broken since d58d8e8f7a4ec03ff14021a4515c8283dad52573
and it's making `morph get-repo` fail with:
/src/definitions # morph get-repo ansible
2016-04-06 11:36:31 Loading in all morphologies
2016-04-06 11:36:32 Checking out ref 0a7124541247cc613352054f4bc0c3e116e0b657 of ansible in ansible stratum
Traceback (most recent call last):
File "cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/get_repo_plugin.py", line 142, in get_repo
chunk_dirname = checkout_chunk(morph, chunk)
File "/src/morph/morphlib/plugins/get_repo_plugin.py", line 110, in checkout_chunk
ref or chunk_spec['ref'])
File "/src/morph/morphlib/plugins/get_repo_plugin.py", line 60, in _clone_repo
gd.update_submodules(self.app)
File "/src/morph/morphlib/gitdir.py", line 527, in update_submodules
morphlib.git.update_submodules(app, self.dirname)
File "/src/morph/morphlib/git.py", line 150, in update_submodules
submodules.load()
File "/src/morph/morphlib/git.py", line 69, in load
content = self._read_gitmodules_file()
File "/src/morph/morphlib/git.py", line 82, in _read_gitmodules_file
ignore_fail=True)
File "/src/morph/morphlib/git.py", line 306, in gitcmd
return runcmd(cmdline, **kwargs)
TypeError: 'str' object is not callable
Change-Id: I8d08eb71a2b0a773bc4322ed46222bea0adbbee1
|
|
|
|
|
|
|
|
|
|
| |
In commit 3a429d0 of git, the casing of certain error messages was
changed, convert the message to lower case before comparison.
ERROR: Command failed: git rev-parse --abbrev-ref benbrown/foo@{upstream}
fatal: no upstream configured for branch 'benbrown/foo'
Change-Id: I5e0659b69f3be84820592b7fe26e50203272da22
|
|
|
|
|
|
| |
It was removed by mistake in 014a029ade9a045a839ca86c35690b218098ea33
Change-Id: I766f39f09444a66234ce3bdcb9134e592d5363ce
|
|
|
|
|
|
|
|
|
|
|
|
| |
Some repositories may have submodules listed in .gitmodules that
are not pointing to a commit object in the tree. This was causing
the following error:
ERROR: /srv/distbuild/gits/git___git_foo_bar_git:c36652c7dd9f1687c87ce04ad3d4af2beb52dc25:.gitmodules: No commit object found for submodule "codegen"
Instead of raising that error, capture it and ignore the submodule.
Change-Id: Iba0428feea7605f797dc78e3e250f3d66c003670
|
|
|
|
|
|
|
|
| |
Update README with details on how to run morph independently from Baserock.
https://storyboard.baserock.org/#!/story/88
Change-Id: Ic1cb58dddb43599017b41fa1126a49edb243a89a
|
|
|
|
|
|
|
|
| |
Warn if 'Yarn' isn't available, since this means most tests don't run.
Also fix some issue with missing .coverage file that I hit.
Change-Id: I675354799bbdefe1ced0f5a0c5221b103de29a1c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Add a copy of the specific version of the cliapp module required by morph, a
known version of which can't otherwise be guaranteed to be available on
general Linux systems, or via PyPi. Including the module directly allows for
continued development in line with morph, until a time when it may become
desirable to move away from dependence on this module.
With a local copy of cliapp added to morph, it has been shown that morph is
capable of building Baserock images without depending on an existing Baserock
system.
Change-Id: Iab1f7037c7afff054a2404c0552d9b5e4d2d141f
|
|
|
|
|
|
|
|
|
| |
This code is a rework from changes done by:
- Tiago Gomes <tiago.gomes@codethink.co.uk>
https://storyboard.baserock.org/#!/story/86
Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
|
|
|
|
| |
Change-Id: I603b5172902b5249874ea3bb90de59db5a908185
|
|
|
|
|
|
|
|
| |
With version 7, less dependencies are shown as the DEFAULTS file used
for testing has a reduced set of artifact splitting rules, compared with
the built-in split rules.
Change-Id: I1e9d5390126bad007d39729faabf83cfb60a5092
|
|
|
|
|
|
|
| |
Use an optional parameter to control whether the defaults should be set
instead of a separate function.
Change-Id: Idbbd5a08e8b16d8e01bb9539274092978b64f6f0
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's not really any reason you'd want to use the RemoteRepoCache
class except as a workaround for the slow speed of some LocalRepoCache
operations, so I can't see this ruining anyone's day.
The main reason for doing this is so we can simply the sourceresolver
code. One reason that the sourceresolver class is so hopelessly
complicated is that it right now has to use two incompatible interfaces
for Git repo caches.
I've taken the opportunity to detangle the RepoCache class from the
App class. Now all of the configuration for the RepoCache class is
passed into the constructor explicitly. This makes the class usable from
outside Morph:
resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[])
repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver)
Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a long time the CachedRepo class has basically been a wrapper around
the GitDir class, but with a few extra methods that don't really even
belong there.
It is now a tiny class in the localrepocache module which just keeps
track of a few extra attributes. All other functionality is provided
by the gitdir module.
This commit also removes the `git clone` approach for copying repos out
of the cache. The alternative approach implemented by
git.copy_repository() was slightly faster when I tested, so for now we
should use that everywhere. Longer term we should find out why this is
quicker than `git clone`, and fix Git itself to be fast.
Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
|
|
|
|
|
|
|
|
|
|
| |
This allows us to simplify a couple of places.
I cannot think of a single situation where'd you want to get the cached
copy of a repo, but not update it. Think about it -- the repo might be
*years* behind the upstream remote.
Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
|
|
|
|
|
|
| |
These seem to have been introduced by commit fb6dd955cbc7081f47dc4391.
Change-Id: I2fb0be5f8829722264f3509f4db9473d19ea2939
|
|
|
|
|
|
| |
File was modified in 472dbbbbfa07e67c02a98e9f4ee50fc86ad32b7e
Change-Id: If0ffa0aae76ff9f0093e5eb7ff7d74d66c26119d
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Now is an error when the name of a stratum in a system is different from
the name of the stratum in the stratum file. Example of the error message:
ERROR: Name 'build-essential-foo' doesn't match 'build-essential' in morphology: strata/build-essential.morph
We are forcing the error here instead of a warning because this problem
makes Morph create broken systems.
https://storyboard.baserock.org/#!/story/15
Change-Id: Ifbe8d98e64a768e4e238d2213ffaf575789dd9e2
|
|
|
|
|
|
|
|
|
| |
Now is an warning when the name of a chunk in a stratum is different from
the name of the chunk in the chunk file. Example of the warning message:
WARNING: Name 'binutils' doesn't match 'binutils-foo' in morphology: strata/build-essential/binutils.morph
Change-Id: I38f0ccff57fc25db12c572fdc8a01dbd6434f01c
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If a command in a chunk is evaluated to None when a morphology is loaded,
this would previously lead to the subprocess module throwing a 'Coercing to
Unicode' error when morph attempts to run the command during the chunk build.
This change throws a more helpful error if any commands are not valid strings
when the chunk is validated, for example:
ERROR: Field configure-commands[217] expected type <type 'str'>, got <type
'NoneType'> in morphology linux-jetson-tk1
Change-Id: I45220e7deab8c7cd9351507bc998f7ff12797442
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I managed to trigger a stack overflow in Morph, by having two chunks with
the same name and same build instructions in different strata (by mistake,
obviously).
The error before was this:
....
File "/src/morph/morphlib/cachekeycomputer.py", line 77, in get_cache_id
cacheid = self._calculate(source)
File "/src/morph/morphlib/cachekeycomputer.py", line 86, in _calculate
for a in source.dependencies],
File "/src/morph/morphlib/cachekeycomputer.py", line 38, in compute_key
ret = self._hash_id(self.get_cache_id(source))
File "/src/morph/morphlib/cachekeycomputer.py", line 77, in get_cache_id
cacheid = self._calculate(source)
File "/src/morph/morphlib/cachekeycomputer.py", line 83, in _calculate
'env': self._filterenv(self._build_env.env),
File "/src/morph/morphlib/cachekeycomputer.py", line 32, in _filterenv
return dict([(k, env[k]) for k in keys])
RuntimeError: maximum recursion depth exceeded while calling a Python object
The error afterwards is this:
2016-02-18 13:53:14 Deciding on task order
ERROR: There are multiple versions of component 'cython'
Change-Id: I96a41cabe4d9aa81bddc5186c5248a636a0843c2
|
|
|
|
|
|
|
| |
This is the counterpart fix to b3ecd02236e58386ac4d7566ef70e751ff0d7e26,
which had broken the Morph test suite, as it turns out.
Change-Id: I5392c2c762c733d7d88cd20898970ec314525d89
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some reason I had a deployment .morph file that specified:
deploy-defaults:
This resulted in the deploy_defaults variable having value None, which
lead to a crash further down:
2016-02-19 12:03:52 Deciding on task order
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 574, in upgrade
self.deploy(args)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 370, in deploy
env_vars, deployments)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 403, in deploy_cluster
parent_location='')
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 462, in deploy_system
env_vars, deployment_filter, parent_location)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 483, in deploy_system_with_source_pool
system_id, env_vars, deploy_defaults, deploy_params)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 59, in configuration_for_system
deploy_params.items() +
AttributeError: 'NoneType' object has no attribute 'items'
This change makes the code work as expected.
Change-Id: I667f4142667be31797cac7c8994d35a404119cca
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The filename was being used instead. I noticed this bug after seeing
cache artifacts named like:
de31c7bcac0240315bc4e0994b34f3757d37d35e4179cdbe0a8b2e60e26449f8.chunk.xorg-proto-x11proto.morph-bins
de7b9f6f7a10b8dc04b52a08e74987df7da310b59dc25c6cf083b0381726e15f.chunk.networkx.morph-bins
def910ed756be94c626d59d73f0770a445d62278a2eb46e031af42563691c6ed.chunk.pv.morph-bins
e003da7f8c697286fa4d550299920160a6e76df449b7e434b270cd760c0c9304.chunk.cython.morph-bins
e193576471c957387c9684c385a8c03db1e70618890210f6272c362ccc3e259b.chunk.xorg-proto-xcmiscproto.morph-bins
e3b0b0dbb7b3de4bb8c72cd064408956296e290fc4b56acbbee241e2d78bf533.chunk.greenlet.morph-bins
e3f09aa095e5839aa9bf7774720951c8b03d78300b056f7bbdeacdf4c5d5944d.chunk.libvorbis.morph-bins
e437810247200713da17937181cc914e080ac7d6030451ccfaa86ddccf7a395b.chunk.python-requests.morph-bins
e4b7de417c5691794f4ad6a6481cacc16242af072db39bcadede6ca2bf48241e.chunk.flac.morph-bins
e4f1e6dbaeed869b445f9b45b98033b6e6a2952674b76b8465e40a67e675333c.chunk.Sub-Name.morph-bins
Change-Id: I8a8697a12cf7e4c7891c93e4f774819fe2f8ea86
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's no longer any need to look up the buildsystem in builder,
since the morphloader now sets commands directly in the morphology.
Removing this unused variable also fixes a bug where morph can
crash if the chunk being built is being built with a buildsystem
that is not hardcoded in morph, error below:
2016-02-15 15:29:46 Starting build of devel-system-x86_64-generic
2016-02-15 15:29:46 [Build 70/407] [cython.morph] Building chunk cython.morph
2016-02-15 15:29:46 [Build 70/407] [cython.morph] Creating staging area
2016-02-15 15:29:47 [Build 70/407] [cython.morph] Starting actual build: cython.morph 4dd8e76
2016-02-15 15:29:47 [Build 70/407] [cython.morph] Extracting upstream:cython into /src/tmp/staging/tmpLI9TUj/cython.morph.build
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/build_plugin.py", line 208, in build
self._build(source_pool, filename, component_names=component_names)
File "/src/morph/morphlib/plugins/build_plugin.py", line 237, in _build
bc.build_in_order(component)
File "/src/morph/morphlib/buildcommand.py", line 297, in build_in_order
self.cache_or_build_source(s, build_env)
File "/src/morph/morphlib/buildcommand.py", line 317, in cache_or_build_source
self.build_source(source, build_env)
File "/src/morph/morphlib/buildcommand.py", line 374, in build_source
self.build_and_cache(staging_area, source, setup_mounts)
File "/src/morph/morphlib/buildcommand.py", line 545, in build_and_cache
return builder.build_and_cache(source)
File "/src/morph/morphlib/builder.py", line 748, in build_and_cache
built_artifacts = o.build_and_cache()
File "/src/morph/morphlib/builder.py", line 288, in build_and_cache
self.run_commands(temppath, stdout)
File "/src/morph/morphlib/builder.py", line 314, in run_commands
bs = morphlib.buildsystem.lookup_build_system(m['build-system'])
File "/src/morph/morphlib/buildsystem.py", line 300, in lookup_build_system
raise KeyError('Unknown build system: %s' % name)
KeyError: 'Unknown build system: python3-distutils'
Change-Id: Iddf651b408c612c7c80296012cf1c97ab99e8ccc
|