| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For some reason I had a deployment .morph file that specified:
deploy-defaults:
This resulted in the deploy_defaults variable having value None, which
lead to a crash further down:
2016-02-19 12:03:52 Deciding on task order
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 299, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 574, in upgrade
self.deploy(args)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 370, in deploy
env_vars, deployments)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 403, in deploy_cluster
parent_location='')
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 462, in deploy_system
env_vars, deployment_filter, parent_location)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 483, in deploy_system_with_source_pool
system_id, env_vars, deploy_defaults, deploy_params)
File "/src/morph/morphlib/plugins/deploy_plugin.py", line 59, in configuration_for_system
deploy_params.items() +
AttributeError: 'NoneType' object has no attribute 'items'
This change makes the code work as expected.
Change-Id: I667f4142667be31797cac7c8994d35a404119cca
|
|
|
|
| |
Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
|
|
|
|
|
|
|
| |
Previously callers to DefinitionsRepoWithApp.source_pool() could
override the user's value, but that's not actually used anywhere.
Change-Id: I43b53e7b9fc937886c8e6e95947e5e2b6776d085
|
|
|
|
|
|
|
|
|
|
| |
This allows you to run `morph deploy` from any Git checkout of
definitions.git.
This also changes `morph help-extensions` to use the DefinitionsRepo
class, as the code overlaps.
Change-Id: I64c48f59c8ec5aebc7169f3b4b4abbb759bd0b9a
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
After moving deployment extensions into the definitions repository, but
they need access to shared code, so deployment extensions need to import
code which lives in definitions. However morph runs a copy of them in a
temporary directory.
Version 5 will allow deployment extensions to depend only on code which
is also in definitions and the Python standard library, though they may
also include code from elsewhere as described in PYTHONPATH.
Change-Id: I998f01830656a5118bba1e579f649ec98f5f82c6
|
|
|
|
| |
Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
|
|
|
|
|
|
|
|
|
|
|
| |
--upgrade being passed on to subsystem deployments broke attempting to
perfom an ssh-rsync upgrade with a system that had a sysroot in it.
Therefore it is appropriate to prevent --upgrade being inherited like
this, as it is seldom what is wanted, and if needed, it can be set in
the cluster, or on the command-line separately.
Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Concievably, you may want to force a cluster to behave as an upgrade
regardless of whether --upgrade is set or not.
This would allow two use cases that are not currently possible:
1. To force a cluster to be upgrade-only, which would make our
instructions for the upgrade-devel.morph cluster simpler.
2. To allow a cluster to deploy then immediately upgrade.
One potential use for this would be to create a disk image with
multiple versions on it.
This also works as a work-around for --upgrade implying UPGRADE=yes for
subsystem deployments, but that should be fixed separately.
Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
|
|
|
|
|
|
|
|
| |
Specifying --upgrade with the upgrade command is redundant,
but it's not a contradiction, so we can safely ignore --upgrade
if it happens to be specified with the upgrade command.
Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Old message:
ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not
present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
New message:
ERROR: Deployment failed as
45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs
is not present in the artifact cache.
Please ensure that mason-system-x86_64-generic-rootfs is built before
deployment, and the artifact-cache-server
(http://cache.baserock.org:8080/) is the correct one.
Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This fixes an issue in `morph list-artifacts` and probably other commands:
ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts
repo, ref, system_filename)
File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system
status_cb=self.app.status)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool
definitions_original_ref=original_ref)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs
definitions_absref, definitions_tree, visit)
File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children
definitions_queue = collections.deque(system_filenames)
TypeError: 'NoneType' object is not iterable
The issue is that the sourceresolver.create_source_pool() function did
not work correctly if passed a single filename. This was a regression from
commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0.
In order to have a logical function prototype, the create_source_pool()
function now only takes a list of systems, instead of taking either a
single system or a list.
Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
|
|
|
|
| |
Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit allows the specification of one or more strata/chunks
in a deployment entry in a cluster morphology to deploy instead of
the full system if --partial is set. These are listed in a
'partial-deploy-components' field in each deployment definition.
The components must be in the system, and this only works for
tarball or sysroot deployments. It SHOULD NOT be used when
deploying production systems, as it has a number of limitations.
Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
|
|
|
|
|
|
|
|
|
|
| |
If a user had the wrong setting for artifact-cache-server then they
could build a system with 'morph distbuild', then be told it wasn't
built by 'morph deploy'. The logs do not show any info about which
cache server was used, so it looks like a Morph bug. This change should
make such a configuration problem much easier to spot.
Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
|
|
|
|
|
|
|
|
|
|
|
|
| |
Remove documentation specific to certain .configure and .write
extensions, and point the user to `morph help-extensions` and `morph
help xxx.write` instead.
Don't mention the deprecated nfsboot.write extension.
Other small cleanups.
Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch adds two fields to deployment (cluster) .morph files:
'upgrade-type' and 'upgrade-location'.
The `morph deploy` command ignores these. The `morph upgrade` command
will honour them if present, instead of the existing 'type' and
'location' fields. If they are not present, `morph upgrade` will give
a warning, and will use the 'type' and 'location' fields as before.
This avoids the need to edit the deployment .morph file after deploying
a system.
Small detail: the 'type' and 'location' variables are no longer removed
from the environment that is passed to the .configure and .write
extensions. This shouldn't affect anything.
Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
|
|
|
|
|
|
|
| |
This behaviour is complex, it shouldn't be hidden away in a larger
function.
Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
|
| |
|
|
|
|
| |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Not everyone is a fan of the `morph build` magic that collects up your
changes and puts them in a temporary branch. Now you can disable it by
setting 'local-changes=ignore' in your morph.conf file.
This speeds up `morph build` and `morph deploy` by 5-10 seconds on my
machine.
I looked an option to make `morph build` warn if there are uncommitted
changes. I found that with a cold cache, it takes about 5 seconds on my
machine to verify that there are no uncommitted changes to a checkout of
definitions.git. That defeats the main purpose of this patch for me, so
I didn't include the option.
|
|
|
|
|
|
|
|
| |
The user should be aware of this because if they aren't building
baserock:baserock/definitions or a repo forked from it, those extensions
won't be available.
Also fix some long lines that I seem to have failed to commit already.
|
| |
|
| |
|
|
|
|
|
| |
- Document different ways of calling parameters
- Allowed values for boolean parameters
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit c63f4e61f69820c71c2d8e9b96ce9bdec0d476a1.
This change was introduced back when the Baserock system definitions
used named Git refs rather than exact SHA1s. Morph would thus need to
update each repo involved in a build and resolve each ref before it
could build. Now that the system definitions use SHA1s, the 'updating
gits' phase is much less slow.
The problem with forcing 'no-git-update' is that it makes `morph deploy`
depend on the remote repo cache. In the case that all repos involved in
the build are avaiable in the remote repo cache (Trove), there is no
need for them to have been cached locally just to construct a build
graph. But if the network connection to the Trove is unreliable Morph
will need to fall back to the local repo cache. Unless Morph has done a
full local build of the system being built, it might not have every repo
cached locally. Often, cached artifacts from the Trove or distbuild may
be used instead of locally building everything.
In this case the user sees `morph deploy` failing with errors such as:
Repository upstream:binutils-redhat is not cached yet
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Most usefully, this patch means that Morph no longer updates its cached
copy of definitions.git every time you run `morph build`.
Also, it prevents confusion in the following situation. Imagine I have
run:
morph checkout baserock:baserock/definitions master
I then wait a while, during which time someone pushes to 'master' in the
definitions.git repo that I cloned from. Now I run:
cd master
morph build systems/whatever.morph
Which commit does it build, the local head of 'master' or the remote
head of 'master'?
The answer, both before and after this patch, is that it builds the
local version of master. But previously, this only happened because of
the magic that we have to detect local changes. With this patch, the
local change detection could be disabled and `morph build` would still
build what the user had checked out as 'master' locally, not whatever
'master' pointed to in the remote repo.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This involved rewriting the util.log_dict_diff() function. It has been
renamed to log_environment_changes() to better reflect its purpose.
It no longer logs both the old and new values in the event of an
environment variable changing. It now just logs the new value. This makes
the code simpler and seems like it should not be a big problem.
Some projects recommend passing credentials through the environment.
OpenStack does this, for example, see:
<http://docs.openstack.org/user-guide/content/cli_openrc.html>
It's unlikely that users would be happy about applications saving
these passwords in log files all over their system.
I do not recommend ever storing valuable passwords in the environment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is achieved by passing it the write end of a pipe, so that the
extension has somewhere to write debug messages without clobbering either
stdout or stderr.
Previously deployment extensions could only display status messages on
stdout, which is good for telling the user what is happening but is not
useful when trying to do post-mortem debugging, when more information is
usually required.
This uses the asyncore asynchronous event framework, which is rather
specific to sockets, but can be made to work with file descriptors, and
has the advantage that it's part of the python standard library.
It relies on polling file descriptors, but there's a trick with pipes to
use them as a notification that a process has exited:
1. Ensure the write-end of the pipe is only open in the process you
want to know when it exits
2. Make sure the pipe's FDs are set in non-blocking read/write mode
3. Call select or poll on the read-end of the file descriptor
4. If select/poll says you can read from that FD, but you get an EOF,
then the process has closed it, and if you know it doesn't do that
in normal operation, then the process has terminated.
It took a few weird hacks to get the async_chat module to unregister
its event handlers on EOF, but the result is an event loop that is
asleep until the kernel tells it that it has to do something.
|
|
|
|
|
|
| |
The arguments to `morph deploy` can get quite long, any way we can make it
shorter and clearer is useful. We can also avoid having the strange
--no-upgrade flag in future.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sorry about the big lump, I can split it into a nicer set of changes,
but they didn't naturally emerge in a nice series.
This creates a pushed_build_branch context manager, to eliminate code
duplication between build and deploy.
Rather than the build branch being constructed knowing whether it needs
to push the branch, it infers that from the state of the repositories,
and whether a local build would be possible.
If there are no uncommitted changes and all local branches are pushed,
then it doesn't create temporary branches or push them, and instead uses
what it already has.
It will currently create and use temporary build branches even for
chunks that have no local changes, but it's pretty cheap, and doesn't
require re-working the build-ref injection code to check whether there
are local changes.
|
|
|
|
|
|
|
|
|
|
|
| |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously they were generator functions, which yielded interesting
context at interesting times so that the caller could respond by
printing status messages.
The only benefits this had over callbacks were:
1. 1 fewer function scope to worry about. I don't have data on the
amount of memory used for a function scope vs a generator, but it
could be less cognitive load for determining which variables are
defined in the callback's body.
2. It is possible to yield in the caller, so you could make that into a
coroutine too, however this wasn't required in this case, as the
yielded value was intended to be informational.
The downsides to this are:
1. It's a rather peculiar construct, so can be difficult to understand
what's going on, and the implications, which led to
2. If you call the function, but don't use the iterator it returned,
then it won't do anything, which is very confusing indeed, if you're
not used to how generator functions work.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We want to use file paths to locate morphologies now, so the old model
of get a list of names and hand it those back to get the contents
doesn't really make sense any more.
This abstraction initially came about as one idea I had for moving
morphologies out of the source tree was to put them in something like
git notes, where it's possible to look up information for one commit in
another ref in the repository, at which point this abstraction would
have been flexible enough to handle that as well as in the
However, moving the chunk morphologies into the definitions repository
has other benefits too, so it makes more sense to be honest about using
filenames in the API.
It remains as a single point where we can put the logic for knowing which
files in a repository look like morphologies, but if we need to remove
any further functionality, it should be replaced by a single function.
|
| |
|
|
|
|
|
|
|
|
| |
Instead of taking the name of a cluster morphology and zero or more
parameters for overriding the cluster morphology, morph deploy should
take the name of a cluster morphology and the names of zero or more
system deployments that are defined in the cluster morphology. If no
deployment names are given then all deployments are deployed.
|
|
|
|
|
|
|
|
| |
The deploy plugin is relying on existing code which respects the
'no-git-update' setting and updates gits if it is not set.
Since deploy can only work for built systems anyway, we can force this
True for deployments, to avoid wasting the user/caller's time.
|
|
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
tarfile's open needs the file-like object
to have a tell() method, objects returned
from sockets don't have this method.
So instead we fetch the artifact from the
remote and cache it locally
|
| | |
|
| |
| |
| |
| | |
This also makes it obey status prefix
|
| | |
|
| |
| |
| |
| |
| | |
Check is now separate from setup, which is now separate from running
the commands.
|
| |
| |
| |
| |
| |
| | |
We don't need to remove the whole thing every time, and for nested
deployments, we want to keep the directories around, since there will
be multiple deploys happening in it concurrently.
|
| | |
|