| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We want to use file paths to locate morphologies now, so the old model
of get a list of names and hand it those back to get the contents
doesn't really make sense any more.
This abstraction initially came about as one idea I had for moving
morphologies out of the source tree was to put them in something like
git notes, where it's possible to look up information for one commit in
another ref in the repository, at which point this abstraction would
have been flexible enough to handle that as well as in the
However, moving the chunk morphologies into the definitions repository
has other benefits too, so it makes more sense to be honest about using
filenames in the API.
It remains as a single point where we can put the logic for knowing which
files in a repository look like morphologies, but if we need to remove
any further functionality, it should be replaced by a single function.
|
| |
|
|
|
|
|
|
|
|
| |
Instead of taking the name of a cluster morphology and zero or more
parameters for overriding the cluster morphology, morph deploy should
take the name of a cluster morphology and the names of zero or more
system deployments that are defined in the cluster morphology. If no
deployment names are given then all deployments are deployed.
|
|
|
|
|
|
|
|
| |
The deploy plugin is relying on existing code which respects the
'no-git-update' setting and updates gits if it is not set.
Since deploy can only work for built systems anyway, we can force this
True for deployments, to avoid wasting the user/caller's time.
|
|
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
tarfile's open needs the file-like object
to have a tell() method, objects returned
from sockets don't have this method.
So instead we fetch the artifact from the
remote and cache it locally
|
| | |
|
| |
| |
| |
| | |
This also makes it obey status prefix
|
| | |
|
| |
| |
| |
| |
| | |
Check is now separate from setup, which is now separate from running
the commands.
|
| |
| |
| |
| |
| |
| | |
We don't need to remove the whole thing every time, and for nested
deployments, we want to keep the directories around, since there will
be multiple deploys happening in it concurrently.
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit introduces a new requirement: USERS MUST NOT HAVE SENSITIVE
DATA IN THEIR ENVIRONMENT. Otherwise it will be leaked into the system.
Note that configuration fields with 'PASSWORD' in their name are
stripped before writing the /baserock/deployment.meta file, so the
OpenStack OS_PASSWORD field is not leaked.
We want this so that we can run hooks at upgrade-time in the future.
These hooks might need to know how the system was configured and what
releaseuu it was. I'm not quite sure how we will define 'release' yet,
but by using `git tag` and `git describe` we are able to textually label
a time period in the history of the system's source code. We already
have the specific SHA1 of definitions.git stored in the system metadata,
so this should give us enough to be able to implement specific hooks
that work around any awkward upgrade complications we encounter in the
future.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
A write extension will have various kinds of sanity checks to do before
actually performing the write. The current architecture of 'morph
deploy' means that several minutes pass between the user starting the
command and the write extension actually executing. It would be
rage-inducing watching `morph deploy` spend 3 minutes unpacking a
system only to then abort due to a silly error such as forgetting the
--upgrade switch. Therefore it's better for now to split the sanity
checks out into separate extensions that can be run as soon as possible
and abort if the write extension is not going to be able to operate.
For now this will just be used to validate usage of the --upgrade flag
but in future checking connectivity to remote servers and the like
should be done here too.
|
| | |
|
|/
|
|
|
|
| |
After this, "./check --full" works.
Reviewed-by: Daniel Silverstone (on IRC)
|
|
|
|
| |
This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
It now does not push branches as this is not necessary to locate the
artifact.
It still makes temporary build branches, since it is assumed that if
you have changes in your workspace, it's preferable for the deploy to
fail, rather than think you've deployed something you haven't.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
From now on, `morph deploy` will work to only accept a cluster
morphology as argument. A cluster morphology defines a list of
systems to built, and for each system a list of ways to deploy
them. We will not support the old syntax anymore.
- Update `morph deploy` docstring with the new syntax, including
an explanation of cluster morphologies (also document `tar`
deployments).
- Fix tests that used the old syntax.
- Add tests for cluster deployments.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When a configuration extension or write extension fails, then it should
abort then, not continue to run other configuration extensions.
There was a period where this didn't happen, due to a missing feature
of cliapp that was assumed to be there, so failure to run these
extensions was not noticed.
This has since been fixed, but this would cause deploy to fail to
clean up its temporary directories.
Now it will cleanup the contents of the temporary directory after
any failures after it has been created.
A small amount of re-ordering was performed to make this easier.
|
|
|
|
|
|
|
|
| |
It's a waste of time to unpack the rootfs, only to have to clean it up
again when you find out that you messed up the command line arguments.
This also has the benefit of reducing the amount of resources that have
to be considered for cleanup.
|
|
|
|
| |
It is now a tested helper function in morphlib.util
|
|
|
|
|
|
|
|
|
| |
We can't assume an extension cleans up after itself, as they can be
arbitrary shell scripts, and the best shell has to offer for cleanup is
`trap`, which is difficult to use.
So now, anything created with `mktemp` will get automatically cleaned
up by morph.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
| |
|
|
|
|
|
| |
This was left over from experimentation, it works with sh, and is
faster.
|
|
|
|
|
| |
This requires a patch to cliapp to actually print the output, as there
is a bug that causes it to always output to a pipe.
|
|
|
|
|
| |
I find it easier to read without them, since there are less variables
to remember.
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
git://git.baserock.org/baserock/baserock/morph
I had fixed an conflict and change to use morph_tmp instead of
morph as default temp dir.
Reviewed by Lars Wirzenius
Conflicts:
morphlib/app.py
|
| |
| |
| |
| |
| |
| | |
Now, inside the temporary directory we will have the following
subdirectories: chunks, staging, failed and deployments. The failed
directory will contain the staging areas of failed builds.
|
|/
|
|
|
|
|
| |
The same check that cachedir and tempdir are large enough is used
for both build and build-morphology.
Deploy only checks for tempdir being large enough.
|
| |
|
|\ |
|
| |
| |
| |
| |
| | |
Terminal will let user know which extension (write or config) is
being run.
|
|\ \
| | |
| | |
| | |
| | |
| | | |
'remotes/origin/baserock/richardholland/expand-morph-deploy-help'
Reformatted the text a bit.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
When too few arguments are given to morph deploy it points you to the
help page, which contains no information about the number of arguments
needed. This patch adds some information from the morph reference manual
regarding the arguments needed for morph deploy.
|
| |/
|/|
| |
| |
| |
| |
| | |
This adds TMPDIR to the environment of deployment extensions.
It uses the --tempdir setting from morph in the case that TMPDIR
is not already set.
|
|/
|
|
|
|
| |
Previously we used the system branch top directory, when we need
the directory where the git repo for the system morphology is
checked out.
|
| |
|
|\ |
|
| | |
|