| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Overlayfs is new in version 3.18 of the kernel, so add support for a
different implementation of a union/overlay filesystem in order to
allow morph to work on older kernels.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When deploying, configuration extensions are run against the unpacked
tarball of a system created by a build. If we are to use OSTree to
store systems (rather than tarballs) this will not be possible as the
result of `ostree checkout` would be read-only.
To solve this, we use overlayfs to mount the unpacked tarball
underneath a temporary directory somewhere, and run the configuration
extensions on that mount point. This means that the changes are
made in the temporary directory rather than directly on the tarball.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Not everyone is a fan of the `morph build` magic that collects up your
changes and puts them in a temporary branch. Now you can disable it by
setting 'local-changes=ignore' in your morph.conf file.
This speeds up `morph build` and `morph deploy` by 5-10 seconds on my
machine.
I looked an option to make `morph build` warn if there are uncommitted
changes. I found that with a cold cache, it takes about 5 seconds on my
machine to verify that there are no uncommitted changes to a checkout of
definitions.git. That defeats the main purpose of this patch for me, so
I didn't include the option.
|
| |
|
|
|
|
|
|
| |
The user should be aware of this because if they aren't building
baserock:baserock/definitions or a repo forked from it, those extensions
won't be available.
Also fix some long lines that I seem to have failed to commit already.
|
| | |
|
| | |
|
| |
|
|
|
| |
- Document different ways of calling parameters
- Allowed values for boolean parameters
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit c63f4e61f69820c71c2d8e9b96ce9bdec0d476a1.
This change was introduced back when the Baserock system definitions
used named Git refs rather than exact SHA1s. Morph would thus need to
update each repo involved in a build and resolve each ref before it
could build. Now that the system definitions use SHA1s, the 'updating
gits' phase is much less slow.
The problem with forcing 'no-git-update' is that it makes `morph deploy`
depend on the remote repo cache. In the case that all repos involved in
the build are avaiable in the remote repo cache (Trove), there is no
need for them to have been cached locally just to construct a build
graph. But if the network connection to the Trove is unreliable Morph
will need to fall back to the local repo cache. Unless Morph has done a
full local build of the system being built, it might not have every repo
cached locally. Often, cached artifacts from the Trove or distbuild may
be used instead of locally building everything.
In this case the user sees `morph deploy` failing with errors such as:
Repository upstream:binutils-redhat is not cached yet
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Most usefully, this patch means that Morph no longer updates its cached
copy of definitions.git every time you run `morph build`.
Also, it prevents confusion in the following situation. Imagine I have
run:
morph checkout baserock:baserock/definitions master
I then wait a while, during which time someone pushes to 'master' in the
definitions.git repo that I cloned from. Now I run:
cd master
morph build systems/whatever.morph
Which commit does it build, the local head of 'master' or the remote
head of 'master'?
The answer, both before and after this patch, is that it builds the
local version of master. But previously, this only happened because of
the magic that we have to detect local changes. With this patch, the
local change detection could be disabled and `morph build` would still
build what the user had checked out as 'master' locally, not whatever
'master' pointed to in the remote repo.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This involved rewriting the util.log_dict_diff() function. It has been
renamed to log_environment_changes() to better reflect its purpose.
It no longer logs both the old and new values in the event of an
environment variable changing. It now just logs the new value. This makes
the code simpler and seems like it should not be a big problem.
Some projects recommend passing credentials through the environment.
OpenStack does this, for example, see:
<http://docs.openstack.org/user-guide/content/cli_openrc.html>
It's unlikely that users would be happy about applications saving
these passwords in log files all over their system.
I do not recommend ever storing valuable passwords in the environment.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is achieved by passing it the write end of a pipe, so that the
extension has somewhere to write debug messages without clobbering either
stdout or stderr.
Previously deployment extensions could only display status messages on
stdout, which is good for telling the user what is happening but is not
useful when trying to do post-mortem debugging, when more information is
usually required.
This uses the asyncore asynchronous event framework, which is rather
specific to sockets, but can be made to work with file descriptors, and
has the advantage that it's part of the python standard library.
It relies on polling file descriptors, but there's a trick with pipes to
use them as a notification that a process has exited:
1. Ensure the write-end of the pipe is only open in the process you
want to know when it exits
2. Make sure the pipe's FDs are set in non-blocking read/write mode
3. Call select or poll on the read-end of the file descriptor
4. If select/poll says you can read from that FD, but you get an EOF,
then the process has closed it, and if you know it doesn't do that
in normal operation, then the process has terminated.
It took a few weird hacks to get the async_chat module to unregister
its event handlers on EOF, but the result is an event loop that is
asleep until the kernel tells it that it has to do something.
|
| |
|
|
|
|
| |
The arguments to `morph deploy` can get quite long, any way we can make it
shorter and clearer is useful. We can also avoid having the strange
--no-upgrade flag in future.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sorry about the big lump, I can split it into a nicer set of changes,
but they didn't naturally emerge in a nice series.
This creates a pushed_build_branch context manager, to eliminate code
duplication between build and deploy.
Rather than the build branch being constructed knowing whether it needs
to push the branch, it infers that from the state of the repositories,
and whether a local build would be possible.
If there are no uncommitted changes and all local branches are pushed,
then it doesn't create temporary branches or push them, and instead uses
what it already has.
It will currently create and use temporary build branches even for
chunks that have no local changes, but it's pretty cheap, and doesn't
require re-working the build-ref injection code to check whether there
are local changes.
|
| |
|
|
|
|
|
|
|
|
|
| |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously they were generator functions, which yielded interesting
context at interesting times so that the caller could respond by
printing status messages.
The only benefits this had over callbacks were:
1. 1 fewer function scope to worry about. I don't have data on the
amount of memory used for a function scope vs a generator, but it
could be less cognitive load for determining which variables are
defined in the callback's body.
2. It is possible to yield in the caller, so you could make that into a
coroutine too, however this wasn't required in this case, as the
yielded value was intended to be informational.
The downsides to this are:
1. It's a rather peculiar construct, so can be difficult to understand
what's going on, and the implications, which led to
2. If you call the function, but don't use the iterator it returned,
then it won't do anything, which is very confusing indeed, if you're
not used to how generator functions work.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We want to use file paths to locate morphologies now, so the old model
of get a list of names and hand it those back to get the contents
doesn't really make sense any more.
This abstraction initially came about as one idea I had for moving
morphologies out of the source tree was to put them in something like
git notes, where it's possible to look up information for one commit in
another ref in the repository, at which point this abstraction would
have been flexible enough to handle that as well as in the
However, moving the chunk morphologies into the definitions repository
has other benefits too, so it makes more sense to be honest about using
filenames in the API.
It remains as a single point where we can put the logic for knowing which
files in a repository look like morphologies, but if we need to remove
any further functionality, it should be replaced by a single function.
|
| | |
|
| |
|
|
|
|
|
|
| |
Instead of taking the name of a cluster morphology and zero or more
parameters for overriding the cluster morphology, morph deploy should
take the name of a cluster morphology and the names of zero or more
system deployments that are defined in the cluster morphology. If no
deployment names are given then all deployments are deployed.
|
| |
|
|
|
|
|
|
| |
The deploy plugin is relying on existing code which respects the
'no-git-update' setting and updates gits if it is not set.
Since deploy can only work for built systems anyway, we can force this
True for deployments, to avoid wasting the user/caller's time.
|
| |
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
tarfile's open needs the file-like object
to have a tell() method, objects returned
from sockets don't have this method.
So instead we fetch the artifact from the
remote and cache it locally
|
| | | |
|
| | |
| |
| |
| | |
This also makes it obey status prefix
|
| | | |
|
| | |
| |
| |
| |
| | |
Check is now separate from setup, which is now separate from running
the commands.
|
| | |
| |
| |
| |
| |
| | |
We don't need to remove the whole thing every time, and for nested
deployments, we want to keep the directories around, since there will
be multiple deploys happening in it concurrently.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit introduces a new requirement: USERS MUST NOT HAVE SENSITIVE
DATA IN THEIR ENVIRONMENT. Otherwise it will be leaked into the system.
Note that configuration fields with 'PASSWORD' in their name are
stripped before writing the /baserock/deployment.meta file, so the
OpenStack OS_PASSWORD field is not leaked.
We want this so that we can run hooks at upgrade-time in the future.
These hooks might need to know how the system was configured and what
releaseuu it was. I'm not quite sure how we will define 'release' yet,
but by using `git tag` and `git describe` we are able to textually label
a time period in the history of the system's source code. We already
have the specific SHA1 of definitions.git stored in the system metadata,
so this should give us enough to be able to implement specific hooks
that work around any awkward upgrade complications we encounter in the
future.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
A write extension will have various kinds of sanity checks to do before
actually performing the write. The current architecture of 'morph
deploy' means that several minutes pass between the user starting the
command and the write extension actually executing. It would be
rage-inducing watching `morph deploy` spend 3 minutes unpacking a
system only to then abort due to a silly error such as forgetting the
--upgrade switch. Therefore it's better for now to split the sanity
checks out into separate extensions that can be run as soon as possible
and abort if the write extension is not going to be able to operate.
For now this will just be used to validate usage of the --upgrade flag
but in future checking connectivity to remote servers and the like
should be done here too.
|
| | | |
|
| |/
|
|
|
|
| |
After this, "./check --full" works.
Reviewed-by: Daniel Silverstone (on IRC)
|
| |
|
|
| |
This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
It now does not push branches as this is not necessary to locate the
artifact.
It still makes temporary build branches, since it is assumed that if
you have changes in your workspace, it's preferable for the deploy to
fail, rather than think you've deployed something you haven't.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
From now on, `morph deploy` will work to only accept a cluster
morphology as argument. A cluster morphology defines a list of
systems to built, and for each system a list of ways to deploy
them. We will not support the old syntax anymore.
- Update `morph deploy` docstring with the new syntax, including
an explanation of cluster morphologies (also document `tar`
deployments).
- Fix tests that used the old syntax.
- Add tests for cluster deployments.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a configuration extension or write extension fails, then it should
abort then, not continue to run other configuration extensions.
There was a period where this didn't happen, due to a missing feature
of cliapp that was assumed to be there, so failure to run these
extensions was not noticed.
This has since been fixed, but this would cause deploy to fail to
clean up its temporary directories.
Now it will cleanup the contents of the temporary directory after
any failures after it has been created.
A small amount of re-ordering was performed to make this easier.
|
| |
|
|
|
|
|
|
| |
It's a waste of time to unpack the rootfs, only to have to clean it up
again when you find out that you messed up the command line arguments.
This also has the benefit of reducing the amount of resources that have
to be considered for cleanup.
|
| |
|
|
| |
It is now a tested helper function in morphlib.util
|
| |
|
|
|
|
|
|
|
| |
We can't assume an extension cleans up after itself, as they can be
arbitrary shell scripts, and the best shell has to offer for cleanup is
`trap`, which is difficult to use.
So now, anything created with `mktemp` will get automatically cleaned
up by morph.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
