| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Also add support to allow building compatible architectures on armv8
machines, as per the rationale in the comment in
_validate_architecture().
|
| |\
| |
| |
| |
| |
| |
| | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This affects errors encountered at build time and at system-integration time.
New errors look like this:
ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000:
Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'}
Error output:
+ install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644
install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
Previously the error message would have been this:
Command failed: unshare --mount -- sh -ec.
mount --make-rprivate /
root="$1"
shift
while true; do
case "$1" in
--)
shift
break
;;
*)
mount_point="$1"
mount_type="$2"
mount_source="$3"
shift 3
path="$root/$mount_point"
mount -t "$mount_type" "$mount_source" "$path"
;;
esac
done
exec "$@"
- /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc
/var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000
+ install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644
install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
|
| |
|
|
|
| |
This way the build commands, system integration commands and deployment
extension commands can all share the logic.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously if the user included 'baserock' in their list of trove-ids,
the keyed URL expansions in the repo-alias field would change such that
reading from git.baserock.org required SSH access. This isn't good and
in particular breaks Mason, which by default isn't set up with a Gitano
account on the Trove that it reads from, and therefore doesn't have SSH
access to that Trove.
With this change, the trove-id field will not cause the default
'baserock:' and 'upstream:' prefixes to be overridden, so setting
'trove-id = baserock' in morph.conf will not Morph's behaviour to
change.
|
| |
|
|
|
|
|
|
|
| |
The msg parameter to status is a format string. If we pass a string
directly to it, then we have to be careful to escape any formatting
characters.
However, we can just do the interpolation directly in the status call
instead, which is less code.
|
| |
|
|
|
| |
The upstream cliapp project is not interested in this functionality
right now.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This involved rewriting the util.log_dict_diff() function. It has been
renamed to log_environment_changes() to better reflect its purpose.
It no longer logs both the old and new values in the event of an
environment variable changing. It now just logs the new value. This makes
the code simpler and seems like it should not be a big problem.
Some projects recommend passing credentials through the environment.
OpenStack does this, for example, see:
<http://docs.openstack.org/user-guide/content/cli_openrc.html>
It's unlikely that users would be happy about applications saving
these passwords in log files all over their system.
I do not recommend ever storing valuable passwords in the environment.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The KVM and VirtualBox deployments use sparse files for raw disk
images. This means they can store a large disk (say, tens or hundreds
of gigabytes) without using more disk space than is required for the
actual content (e.g., a gigabyte or so for the files in the root
filesystem). The kernel and filesystem make the unwritten parts of the
disk image look as if they are filled with zero bytes. This is good.
However, during deployment those sparse files get transferred as if
there really are a lot of zeroes. Those zeroes take a lot of time to
transfer. rsync, for example, does not handle large holes efficiently.
This change introduces a couple of helper tools (morphlib/xfer-hole
and morphlib/recv-hole), which transfer the holes more efficiently.
The xfer-hole program reads a file and outputs records like these:
DATA
123
binary data (exaclyt 123 bytes and no newline at the end)
HOLE
3245
xfer-hole can do this efficiently, without having to read through all
the zeroes in the holes, using the SEEK_DATA and SEEK_HOLE arguments
to lseek.
Using this, the holes take only take a few bytes each, making it
possible to transfer a disk image faster. In my benchmarks,
transferring a 100G byte disk image took about 100 seconds for KVM,
and 220 seconds for VirtualBox (which needs to more work at the
receiver to convert the raw disk to a VDI). Both benchmarks were from
a VM on my laptop to the laptop itself.
The interesting bit here is that the receiver (recv-hole) is simple
enough that it can be implemented in a bit of shell script, and the
text of the shell script can be run on the remote end by giving it to
ssh as a command line argument. This means there is no need to install
any special tools on the receiver, which makes using this improvement
much simpler.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This was previously a private method of the branch and merge
plugin, but it's useful to other plugins, so has been moved to the
SystemBranchDirectory class, where everything else can get to it.
It has an unpleasant amount of coupling to other classes, but in a *good*
object oriented design it would either be a tiny module on its own,
or not exist and leave all its users to re-implement the same logic
multiple ways, so we've opted for a less clean, but more useful design.
It is left un-covered by the unit tests, since it requires a great deal
of instrumentation to test, at which point it may be best to leave it
to integration tests.
|
| | |
|
| |
|
|
|
|
|
|
| |
This is used to split a large iterable into more manageable chunks.
This is important when a consumer can only handle so much input at once
e.g. A program can take a large, but finite number of arguments, xargs
exists to handle this for shell programs.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This bug was triggered by the fix to CachedRepo.resolve_ref and without
this fix, the resolve_ref fix will break the test suite.
The bug is that log_dict_diff calls the status method with an msg
keyword argument that may contain percentage characters. status
interprets the value of msg as a format string, and the percentage
characters trigger formatting to happen.
The fix for that is to not interpolate the value of key and dictA[key]
and dictB[key] into msg before calling status, but letting status
do that. Thus the msg values are changed to reference %(key)s instead
and passing in a value for key as a separate argument. Ditto for
dictA[key] and dictB[key].
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The workspace needs to be able to list all its system branches, and the
system branches need to be able to list all their git repositories.
This is broadly the same thing, just with a different directory to look
out for, so provide that utility in morphlib.util.
find_leaf() is rewritten to use find_leaves(), this is less efficient
since it waits until every leaf is found.
I felt it was better to reduce the code than maintain a slightly more
optimal algorithm.
The find_leaf() algorithm could become more optimal if it could lazily
check for at least one result in a generator.
|
| |
|
|
|
|
|
|
|
| |
This will be useful in the next commit, where we want
to construct an environment to run a command from a
dictionary.
The cliapp runcmd method expects that the values in this
dictionary are strings, so we need to convert them before.
|
| | |
|
| |
|
|
| |
It is now a tested helper function in morphlib.util
|
| |
|
|
|
|
|
| |
These will be used to find workspace and system branch root directories.
Also accidentally removed some whitespace from ends of lines. Too lazy
to split that into a separate commit.
|
| |\
| |
| |
| |
| |
| | |
'origin/baserock/richarddale/check-repo-alias-option'
Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| | | |
|
| | | |
|
| |/ |
|
| | |
|
| | |
|
| |
|
|
|
| |
This has no tests, as the ability to run them requires root privilidges
or a more recent kernel and user namespace support.
|
| |
|
|
| |
ciapp is not imported, but morphlib.Error is an appropriate exception
|
| |
|
|
|
| |
Ensure that trove-id meets the same requirements as repo aliases in terms of
naming.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Define a specific set of 4 architectures that Morph supports, and only
expose that value to morphologies.
Since GNU triplets are very common we also expose a GNU triplet. Other
morphologies should work out their configuration based on MORPH_ARCH.
This commit also removes the morphlib.util.arch() function, which
detected the machine Morph is running on via 'uname -m'. Morph's
architecture names do not necessarily map to the output of 'uname -m'
so we should not rely on it anywhere.
|
| |
|
|
|
|
| |
In the future we will allow this to be modified to provide a
cross-bootstrap mode, but for now we use the same target as
the host compiler.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reorganise the build_artifact() and build_artifacts() functions to
allow more complex work when setting up chunk builds in
build_artifact().
The staging area now holds the BuildEnvironment object (the
environment variables that should be set during build). This makes sense
because all build commands should be run inside the staging area and
therefore through the StagingArea object.
The BuildEnvironment object is now considered immutable after it is
created. The environment is used in cache key computation when
computing what artifacts are required; if it changes after that point
we risk either computing different artifact keys for the same artifact
or missing data in the cache key that should be included in the hash.
Better to force changes into a separate 'extra_env' variable.
|
| |\
| |
| |
| |
| |
| | |
'origin/baserock/richardholland/environment-logging' into staging
Added a "pragma: no cover" to make the test suite pass.
|
| |/
|
|
|
|
|
|
| |
Added function log_dict_diff to identify and record changes in dicts
to the debug log
This new function was then implemented in app.py to log changes in the
environment
|
| | |
|
| |
|
|
|
|
|
|
| |
This way everyone who needs json or an OrderedDict doesn't have to worry
about it.
Currently morph2 is the only one that needs this complex behaviour,
other users of the json module don't need object_pairs_hook.
|
| | |
|
| |
|
|
|
|
|
| |
This makes a non-backward-compatible change to morph which switches it to using
tarballs instead of bundles when initialising cached git repositories. This is
faster because it doesn't require index-pack --fix-thin operations on the
machine running morph.
|
| |
|
|
|
|
|
|
|
|
|
| |
Trove has a fixed structure for repositories and prefixes. By taking advantage
of this, we can simplify the repo-alias configuration in morph somewhat and
thus make it easier for others to add their own aliases and prefixes later,
without worrying that they'll miss out on default ones created by the Baserock
team.
We override process_args in the application class to achieve this because
cliapp's setup() hook is not quite right for our purposes.
|
| |
|
|
|
|
|
|
|
|
|
| |
Code for creating caches is repeated through the codebase because
there wasn't a useful function for creating caches.
Now the functions in BuildCommand have been moved into morphlib.util,
where they will be more usefully reachable.
Stubs have been left in BuildCommand so they can be overridden by
subclasses of BuildCommand and not break external plugins.
|
| |
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| | |
|
