| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's not really any reason you'd want to use the RemoteRepoCache
class except as a workaround for the slow speed of some LocalRepoCache
operations, so I can't see this ruining anyone's day.
The main reason for doing this is so we can simply the sourceresolver
code. One reason that the sourceresolver class is so hopelessly
complicated is that it right now has to use two incompatible interfaces
for Git repo caches.
I've taken the opportunity to detangle the RepoCache class from the
App class. Now all of the configuration for the RepoCache class is
passed into the constructor explicitly. This makes the class usable from
outside Morph:
resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[])
repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver)
Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When starting various distbuilds at the same time we were
having randomly errors like:
2016-01-06 12:14:03 Starting distributed build
2016-01-06 12:14:03 Connecting to mason-x86-64:7878
2016-01-06 12:14:03 Requesting build of git://192.168.222.58/baserock/baserock/definitions 28b92192c00a36395acd6a960959d3b4468f9894 systems/openstack-system-x86_64.morph
ERROR: /srv/distbuild/artifacts: File exists
ERROR: /srv/distbuild/artifacts: File exists
ERROR: /srv/distbuild/artifacts: File exists
ERROR: /srv/distbuild/artifacts: File exists
This has been happening in Mason since we remove every folder
used by Morph after Mason finishes.
Change-Id: I175de7282302d9c1e2fb1b0872f7eb72c742f28e
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently the logs of our continuous builder
(https://mason-x86-64.baserock.org) are filled with lots of this sort of
crap:
stage1-binutils-misc[ ] 2.4/73.0 MB
stage1-binutils-misc[ ] 2.4/73.0 MB
stage1-binutils-misc[ ] 2.4/73.0 MB
stage1-binutils-misc[ ] 2.4/73.0 MB
stage1-binutils-misc[# ] 2.4/73.0 MB
stage1-binutils-misc[# ] 2.5/73.0 MB
stage1-binutils-misc[# ] 2.5/73.0 MB
...
Change-Id: I0b7a4c8421c3ecdd141fc8323d3001ae4fb44d9b
|
|
|
|
| |
Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
|
|
|
|
|
|
|
| |
YBD has been doing this for a while and seems like it is faster as a
result.
Change-Id: I8f95a53195cdbc2c75c06a8abe9eb089a84b1c1b
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new 'Defaults' class to represent definitions defaults
The Python 'jsonschema' module is used to validate the contents of the
Defaults file. This module is already included in Baserock 'build' and
'devel' reference systems by way of the 'openstack-common' stratum.
This commit embeds a copy of the JSON-Schema schema for the DEFAULTS
file. I think the canonical location of this schema should be in the
reference definitions.git, for now. In future, the schemas should maybe
have their own repos. Either way, Morph should embed a copy for the time
being so that we are sure the schema matches how Morph expects to parse
the file.
Morph's automated tests are all updated to use definitions version 7.
I removed most of the tests for built-in build systems, because the
built-ins themselves are no longer part of Morph. Only the mechanism for
defining them needs to be tested now.
Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
|
|
|
|
|
|
|
|
|
| |
Looks like,
2015-07-05 16:08:10 [Build 1/304] [stage1-binutils] Fetching to local cache: artifact stage1-binutils-misc
stage1-binutils-misc[##################### ] 51.9/73.0 MB
Change-Id: Ib10f1cfaa0c1df80ae605ecfeb5b706c8d46c4a4
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Improve error messages and remove redundant information
* Stop displaying the containerised settings, as they are not very
helpful to debug a build failure
* Display the error code
* Change system integration commands logging to consistent with build
commands logging.
* Provide a log file in the staging directory for each chunk build
failure or system integration failure.
* Provide a chroot script in the staging directory for each chunk build
failure or system integration build failure. This script allows the
user to enter an environment similiar to the one where the build
failed.
* Fix a bug where the system integration commands stdout was being
omitted when these commands failed.
* Fix a bug where the error_message_for_containerised_commandline() was
being called with an empty error message, as the stderr was being
directed to stdout.
Chunk build failure output before the this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:55:21 [Build 258/267] [test] Running build-commands
build failed
# configure
# # echo "configure-commands stdout"
configure-commands stdout
# # echo "configure-commands stderr" >&2
configure-commands stderr
# build
# # echo "build commands stdout"
build commands stdout
# # echo "build commands stderr" >&2
build commands stderr
# # echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
some more sdout
some more stderr
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
sh: foo: not found
ERROR: In staging area /src/tmp/failed/tmpglc5Sz: Command failed: sh -c
'echo "some more sdout"
echo "some more stderr" >&2
echo $PATH
foo
':
Containerisation settings: {'binds': (('/src/cache/ccache/nano-tarball',
'/src/tmp/staging/tmpglc5Sz/tmp/ccache'),), 'mount_proc': True,
'mounts': (('dev/shm', 'tmpfs', 'none'),), 'writable_paths':
['/src/tmp/staging/tmpglc5Sz/test.build',
'/src/tmp/staging/tmpglc5Sz/test.inst',
'/src/tmp/staging/tmpglc5Sz/dev', '/src/tmp/staging/tmpglc5Sz/proc',
'/src/tmp/staging/tmpglc5Sz/tmp'], 'root': '/src/tmp/staging/tmpglc5Sz',
'cwd': '/test.build'}
Error output:
$>
Chunk build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:53:59 [Build 258/267] [test] Running build-commands
### CONFIGURE-COMMANDS ###
+ echo configure-commands stdout
configure-commands stdout
+ echo configure-commands stderr
configure-commands stderr
### BUILD-COMMANDS ###
+ echo build commands stdout
build commands stdout
+ echo build commands stderr
build commands stderr
+ echo some more sdout
some more sdout
+ echo some more stderr
some more stderr
+ echo /tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
/tools/bin:/usr/bin:/usr/lib/ccache:/sbin:/usr/sbin:/bin:/usr/bin
+ foo
sh: foo: not found
ERROR: In staging area /src/tmp/staging/tmpjES5_2: build failed (exi
t_code=127)
$>
System integration build failure output before this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Running the
system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
running system integration commands
2015-08-03 15:58:25 [Build 267/267] [build-system-x86_64] Error while
building system
ERROR: Command failed:
baserock/system-integration/00-test-test-misc-0002:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'),
('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root':
'/src/tmp/staging/tmpvpwB4l/build-system-x86_64.inst'}
Error output:
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
$>
System integration build failure output after this change and change
I16ebe9ba4ac46fef82e37d0b3e05f42d14249de8:
2015-08-03 16:00:06 [Build 267/267] [build-system-x86_64] Running the
system integration commands
+ echo stdout
stdout
+ echo stderr
stderr
+ foo
baserock/system-integration/00-test-test-misc-0002: line 1: foo: not
found
ERROR: In staging area /src/tmp/staging/tmpl9VNzf: system integration
commands failed (exit_code=127)
$>
Change-Id: Id992f707f69f3fa761b4c21e9904c4e5328e1c77
|
|
|
|
|
|
|
| |
We commonly create a temporary directory and clean it up after we're
done. This can be encapsulated into a context manager.
Change-Id: Ie50de5c3954141a3e7fd40e4627fb743287ef61f
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a cluster morph,
- morph: systems/devel-system-x86_64-generic.morph
deploy:
self:
type: ssh-rsync
location: root@127.0.0.1
ATTACK_KITTENS:
The yaml parser will assign the value None
to the element with key 'ATTACK_KITTENS',
sanitize_environment blindly calls str() on each value in the dictionary,
the result is that our extensions receive an 'ATTACK_KITTENS' environment
variable with the string value 'None', this is not useful and confuses anyone
trying to do basic validation in their extension.
This patch makes it so that any None types are replaced with the empty string,
which seems a more sensible value for this case.
Change-Id: I734d33e9304f0b7d4d590e464e350d2e5b4720d3
|
|
|
|
| |
Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
|
|
|
|
|
|
|
| |
This is useful for representing lists of items in status or exception
messages.
Change-Id: I530eecdc311ac77fca9922dab063f550ea810c06
|
|
|
|
|
|
|
|
| |
Also add support to allow building compatible architectures on armv7
machines, as per the rationale in the comment in
_validate_architecture().
Change-Id: Ie0fe4002523f1f92f576bac8b654d4ea6fad5cf2
|
|
|
|
| |
Change-Id: I9344b9b80a6ec008715559390b63c9003f34bf90
|
|
|
|
|
|
|
| |
Let morph recognise the architecture string reported by
a big-endian ARMv8 kernel.
Change-Id: I64323e6c2a4675a8cc34f185b10fa04754ebe8c8
|
|
|
|
| |
A function to read lines from a dictionary and append them to a file
|
|
|
|
| |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
| |
|
|
|
|
|
|
| |
Also add support to allow building compatible architectures on armv8
machines, as per the rationale in the comment in
_validate_architecture().
|
|\
| |
| |
| |
| |
| |
| | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
|
| | |
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This affects errors encountered at build time and at system-integration time.
New errors look like this:
ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'}
Error output:
+ install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644
install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
Previously the error message would have been this:
Command failed: unshare --mount -- sh -ec.
mount --make-rprivate /
root="$1"
shift
while true; do
case "$1" in
--)
shift
break
;;
*)
mount_point="$1"
mount_type="$2"
mount_source="$3"
shift 3
path="$root/$mount_point"
mount -t "$mount_type" "$mount_source" "$path"
;;
esac
done
exec "$@"
- /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc
/var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000
+ install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644
install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
|
|
|
|
|
| |
This way the build commands, system integration commands and deployment
extension commands can all share the logic.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously if the user included 'baserock' in their list of trove-ids,
the keyed URL expansions in the repo-alias field would change such that
reading from git.baserock.org required SSH access. This isn't good and
in particular breaks Mason, which by default isn't set up with a Gitano
account on the Trove that it reads from, and therefore doesn't have SSH
access to that Trove.
With this change, the trove-id field will not cause the default
'baserock:' and 'upstream:' prefixes to be overridden, so setting
'trove-id = baserock' in morph.conf will not Morph's behaviour to
change.
|
|
|
|
|
|
|
|
|
| |
The msg parameter to status is a format string. If we pass a string
directly to it, then we have to be careful to escape any formatting
characters.
However, we can just do the interpolation directly in the status call
instead, which is less code.
|
|
|
|
|
| |
The upstream cliapp project is not interested in this functionality
right now.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This involved rewriting the util.log_dict_diff() function. It has been
renamed to log_environment_changes() to better reflect its purpose.
It no longer logs both the old and new values in the event of an
environment variable changing. It now just logs the new value. This makes
the code simpler and seems like it should not be a big problem.
Some projects recommend passing credentials through the environment.
OpenStack does this, for example, see:
<http://docs.openstack.org/user-guide/content/cli_openrc.html>
It's unlikely that users would be happy about applications saving
these passwords in log files all over their system.
I do not recommend ever storing valuable passwords in the environment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The KVM and VirtualBox deployments use sparse files for raw disk
images. This means they can store a large disk (say, tens or hundreds
of gigabytes) without using more disk space than is required for the
actual content (e.g., a gigabyte or so for the files in the root
filesystem). The kernel and filesystem make the unwritten parts of the
disk image look as if they are filled with zero bytes. This is good.
However, during deployment those sparse files get transferred as if
there really are a lot of zeroes. Those zeroes take a lot of time to
transfer. rsync, for example, does not handle large holes efficiently.
This change introduces a couple of helper tools (morphlib/xfer-hole
and morphlib/recv-hole), which transfer the holes more efficiently.
The xfer-hole program reads a file and outputs records like these:
DATA
123
binary data (exaclyt 123 bytes and no newline at the end)
HOLE
3245
xfer-hole can do this efficiently, without having to read through all
the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments
to lseek.
Using this, the holes take only take a few bytes each, making it
possible to transfer a disk image faster. In my benchmarks,
transferring a 100G byte disk image took about 100 seconds for KVM,
and 220 seconds for VirtualBox (which needs to more work at the
receiver to convert the raw disk to a VDI). Both benchmarks were from
a VM on my laptop to the laptop itself.
The interesting bit here is that the receiver (recv-hole) is simple
enough that it can be implemented in a bit of shell script, and the
text of the shell script can be run on the remote end by giving it to
ssh as a command line argument. This means there is no need to install
any special tools on the receiver, which makes using this improvement
much simpler.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This was previously a private method of the branch and merge
plugin, but it's useful to other plugins, so has been moved to the
SystemBranchDirectory class, where everything else can get to it.
It has an unpleasant amount of coupling to other classes, but in a *good*
object oriented design it would either be a tiny module on its own,
or not exist and leave all its users to re-implement the same logic
multiple ways, so we've opted for a less clean, but more useful design.
It is left un-covered by the unit tests, since it requires a great deal
of instrumentation to test, at which point it may be best to leave it
to integration tests.
|
| |
|
|
|
|
|
|
|
|
| |
This is used to split a large iterable into more manageable chunks.
This is important when a consumer can only handle so much input at once
e.g. A program can take a large, but finite number of arguments, xargs
exists to handle this for shell programs.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This bug was triggered by the fix to CachedRepo.resolve_ref and without
this fix, the resolve_ref fix will break the test suite.
The bug is that log_dict_diff calls the status method with an msg
keyword argument that may contain percentage characters. status
interprets the value of msg as a format string, and the percentage
characters trigger formatting to happen.
The fix for that is to not interpolate the value of key and dictA[key]
and dictB[key] into msg before calling status, but letting status
do that. Thus the msg values are changed to reference %(key)s instead
and passing in a value for key as a separate argument. Ditto for
dictA[key] and dictB[key].
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The workspace needs to be able to list all its system branches, and the
system branches need to be able to list all their git repositories.
This is broadly the same thing, just with a different directory to look
out for, so provide that utility in morphlib.util.
find_leaf() is rewritten to use find_leaves(), this is less efficient
since it waits until every leaf is found.
I felt it was better to reduce the code than maintain a slightly more
optimal algorithm.
The find_leaf() algorithm could become more optimal if it could lazily
check for at least one result in a generator.
|
|
|
|
|
|
|
|
|
| |
This will be useful in the next commit, where we want
to construct an environment to run a command from a
dictionary.
The cliapp runcmd method expects that the values in this
dictionary are strings, so we need to convert them before.
|
| |
|
|
|
|
| |
It is now a tested helper function in morphlib.util
|
|
|
|
|
|
|
| |
These will be used to find workspace and system branch root directories.
Also accidentally removed some whitespace from ends of lines. Too lazy
to split that into a separate commit.
|
|\
| |
| |
| |
| |
| | |
'origin/baserock/richarddale/check-repo-alias-option'
Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| | |
|
| | |
|
|/ |
|
| |
|