| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
| |
This uses the PyLRU module, from:
<https://pypi.python.org/pypi/pylru/1.0.6>.
Python 3.2 and newer provide a built-in LRU cache, but this is
specifically for in-memory use. See <http://bugs.python.org/issue17528>.
Git commits are immutable, so caching information about their contents
is fairly easy and trouble-free. There's no danger of the cache becoming
stale.
|
| |
|
|
|
|
|
|
|
|
| |
This is another situation where builds could hang forever if the server
is misconfigured.
Longer term, workers should be able to come and go dynamically without
needing to reconfigure and restart the controller process.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Currently the build commands treat the system argument
as a path relative to the root repo.
This means that regardless of your working directory you must run
morph build systems/foo-system.morph
This behaviour can be confusing, for example when your working
directory is $systembranch/definitions/systems you might expect
to be able to run
morph build foo-system.morph
especially since most shells would tab-complete the filename for you.
At the moment running the above command from $systembranch/definitions/systems
would result in an error, because morph would look for
$systembranch/definitions/foo-system.morph rather than
$systembranch/definitions/systems/foo-system.morph
This behaviour also means you can't give the morph build commands
an absolute path to a system morph.
This patch changes the treatment of the system arg so that it is interpreted
relative to the current working directory.
|
|\
| |
| |
| |
| | |
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Not everyone is a fan of the `morph build` magic that collects up your
changes and puts them in a temporary branch. Now you can disable it by
setting 'local-changes=ignore' in your morph.conf file.
This speeds up `morph build` and `morph deploy` by 5-10 seconds on my
machine.
I looked an option to make `morph build` warn if there are uncommitted
changes. I found that with a cold cache, it takes about 5 seconds on my
machine to verify that there are no uncommitted changes to a checkout of
definitions.git. That defeats the main purpose of this patch for me, so
I didn't include the option.
|
|\ \
| |/
|/|
| |
| | |
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This consolidates a bunch of code paths that were previously duplicated.
This also changes the API for local cached repos to match the
function names GitDirectory uses. Note that the remote repo cache still
uses the old names, and should be fixed when time permits.
Some unit tests that use the CachedRepo module required a bit of
inelegant monkey-patching in order that they continue to work. A better
way to do this would be with the 'mock' library (which would need to be
added to Baserock 'build' and 'devel' systems before we could use it).
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| | |
The user should be aware of this because if they aren't building
baserock:baserock/definitions or a repo forked from it, those extensions
won't be available.
Also fix some long lines that I seem to have failed to commit already.
|
|/ |
|
| |
|
|
|
|
|
| |
- Document different ways of calling parameters
- Allowed values for boolean parameters
|
|
|
|
|
| |
Silly mistake caused this to be broken, and there are no tests for this
command inside the morph.git source tree.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This reverts commit c63f4e61f69820c71c2d8e9b96ce9bdec0d476a1.
This change was introduced back when the Baserock system definitions
used named Git refs rather than exact SHA1s. Morph would thus need to
update each repo involved in a build and resolve each ref before it
could build. Now that the system definitions use SHA1s, the 'updating
gits' phase is much less slow.
The problem with forcing 'no-git-update' is that it makes `morph deploy`
depend on the remote repo cache. In the case that all repos involved in
the build are avaiable in the remote repo cache (Trove), there is no
need for them to have been cached locally just to construct a build
graph. But if the network connection to the Trove is unreliable Morph
will need to fall back to the local repo cache. Unless Morph has done a
full local build of the system being built, it might not have every repo
cached locally. Often, cached artifacts from the Trove or distbuild may
be used instead of locally building everything.
In this case the user sees `morph deploy` failing with errors such as:
Repository upstream:binutils-redhat is not cached yet
|
|
|
|
|
|
|
|
| |
This code is an essential part of 'morph build'. It's quite complex and
really shouldn't be mixed in with the base Application class.
Given a dedicated class we can store some state in the object and avoid
functions with seven parameters, too.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The GitDirectory() constructor, if passed a 'dirname' that doesn't
contain a '.git' subdirectory, can search upwards to find the real root
of the repository. This is used by the `add-binary`, `push`, and `pull`
commands.
This causes very confusing behaviour in the case that 'dirname' points to
a directory that should be a Git repository, but isn't, and that directory
is a path inside the working tree of another Git repository. Rather than
raising an error, in this case the GitDirectory class would perform
operations on a different repository to the one the caller expected.
This 'search_for_root' behaviour is now opt-in, to avoid confusion.
|
| |
|
|\
| |
| |
| |
| | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
|/
|
|
|
|
|
|
|
|
|
| |
The recent changes to the BuildCommand.build() function caused distbuild
to break, because I didn't make the same change to the
InitiatorBuildCommand.build() function but did change how it was called.
This commit adds the ability to have optional fields in distbuild
messages. This is used to add an optional 'original_ref' field, which
will get passed to `morph serialise-artifact` by new distbuild
controllers, and will be ignored by older ones.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Most usefully, this patch means that Morph no longer updates its cached
copy of definitions.git every time you run `morph build`.
Also, it prevents confusion in the following situation. Imagine I have
run:
morph checkout baserock:baserock/definitions master
I then wait a while, during which time someone pushes to 'master' in the
definitions.git repo that I cloned from. Now I run:
cd master
morph build systems/whatever.morph
Which commit does it build, the local head of 'master' or the remote
head of 'master'?
The answer, both before and after this patch, is that it builds the
local version of master. But previously, this only happened because of
the magic that we have to detect local changes. With this patch, the
local change detection could be disabled and `morph build` would still
build what the user had checked out as 'master' locally, not whatever
'master' pointed to in the remote repo.
|
|
|
|
|
|
|
|
|
| |
Rather than take a list of triplets to build, the BuildCommand.build()
function now takes a single repo/ref/morph triplet. Iterating through
multiple sets of triplets is now done in the build plugin.
There are a couple of cosmetic changes to the status output at the start
and end of a build as a result.
|
|
|
|
|
| |
Previously it was incorrect in the `morph help` output and in Morph's
man page.
|
|
|
|
| |
It was broken by the per-source building changes.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
You can bind to an ephemeral port by passing 0 as the port number.
To work out which port you actually got, you need to call getsockname().
To facilitate being able to spawn multiple copies of the daemons for
testing environments, you can pass a -file option, which will make the
daemon write which port it actually bound to.
If this path is a fifo, reading from it in the spawner process will
allow synchronisation of only spawning services that require that port to
be ready after it is.
|
|
|
|
|
|
|
| |
I've rarely needed to use it, and on those rare occasions, it would have
been easy enough to calculate it.
Let's get rid of this step, and save everyone some time in future.
|
|
|
|
| |
It's easy enough to deploy the image.
|
|\
| |
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius (+2 to misc fixups)
Reviewed-by: Sam Thursfield (+1 to per-source building)
Reviewed-by: Paul Sherwood (+1 to per-source building)
|
| | |
|
| | |
|
| |
| |
| |
| |
| | |
This was used before the Artifact splitting code landed to determine
which artifacts should be produced.
|
| |
| |
| |
| |
| | |
There's other methods called get_sources in other modules, and
fetch_sources explains more about what it does in the context.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This involved rewriting the util.log_dict_diff() function. It has been
renamed to log_environment_changes() to better reflect its purpose.
It no longer logs both the old and new values in the event of an
environment variable changing. It now just logs the new value. This makes
the code simpler and seems like it should not be a big problem.
Some projects recommend passing credentials through the environment.
OpenStack does this, for example, see:
<http://docs.openstack.org/user-guide/content/cli_openrc.html>
It's unlikely that users would be happy about applications saving
these passwords in log files all over their system.
I do not recommend ever storing valuable passwords in the environment.
|
|
|
|
|
|
| |
It broke when we added /baserock/deployment.meta.
We didn't notice this because our test suite was looking at the artifact
produced by morph build, and listed on the terminal.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This is achieved by passing it the write end of a pipe, so that the
extension has somewhere to write debug messages without clobbering either
stdout or stderr.
Previously deployment extensions could only display status messages on
stdout, which is good for telling the user what is happening but is not
useful when trying to do post-mortem debugging, when more information is
usually required.
This uses the asyncore asynchronous event framework, which is rather
specific to sockets, but can be made to work with file descriptors, and
has the advantage that it's part of the python standard library.
It relies on polling file descriptors, but there's a trick with pipes to
use them as a notification that a process has exited:
1. Ensure the write-end of the pipe is only open in the process you
want to know when it exits
2. Make sure the pipe's FDs are set in non-blocking read/write mode
3. Call select or poll on the read-end of the file descriptor
4. If select/poll says you can read from that FD, but you get an EOF,
then the process has closed it, and if you know it doesn't do that
in normal operation, then the process has terminated.
It took a few weird hacks to get the async_chat module to unregister
its event handlers on EOF, but the result is an event loop that is
asleep until the kernel tells it that it has to do something.
|
|
|
|
|
|
| |
The arguments to `morph deploy` can get quite long, any way we can make it
shorter and clearer is useful. We can also avoid having the strange
--no-upgrade flag in future.
|
| |
|
|
|
|
|
|
| |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
| |
|
|\
| |
| |
| |
| | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Pedro Alvarez
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There was a check in it to see whether it needed to do the git branch
and git checkout based on whether the name of the branch matched that in
the morphology.
This had a couple of problems:
1. Now that we aren't always building from HEAD, we need to be able to
roll its commit back, so using the existing branch isn't always the
best idea.
2. It only checks the "ref" field, not "unpetrify-ref", so even though
we clone the right ref in there, it's checking the commit id against
the system branch name, so would always try to re-create the branch,
and fail when it already exists.
So now, we remove the original ref and re-create it with our preferred
HEAD.
A better solution might be to change the clone logic to not
automatically checkout HEAD, and instead require an explicit branch then
checkout, but the initial clone logic is shared with build, and I didn't
feel like tracking down all the different places that it was used.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The help for the show-branch-root command said it returns a path, but
the command and the yarns just showed the aliased url it was cloned
from.
Given I found myself needing the path in some scripts, not the repo url,
I think it's more useful to reconcile the difference this way.
|
|/
|
|
|
|
|
| |
We don't use this any more, and instead prefer to always keep
definitions.git petrified, and update the refs ourselves.
branch-from-image still uses some of the remaining petrify code.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sorry about the big lump, I can split it into a nicer set of changes,
but they didn't naturally emerge in a nice series.
This creates a pushed_build_branch context manager, to eliminate code
duplication between build and deploy.
Rather than the build branch being constructed knowing whether it needs
to push the branch, it infers that from the state of the repositories,
and whether a local build would be possible.
If there are no uncommitted changes and all local branches are pushed,
then it doesn't create temporary branches or push them, and instead uses
what it already has.
It will currently create and use temporary build branches even for
chunks that have no local changes, but it's pretty cheap, and doesn't
require re-working the build-ref injection code to check whether there
are local changes.
|
|
|
|
|
|
|
|
|
|
|
| |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously they were generator functions, which yielded interesting
context at interesting times so that the caller could respond by
printing status messages.
The only benefits this had over callbacks were:
1. 1 fewer function scope to worry about. I don't have data on the
amount of memory used for a function scope vs a generator, but it
could be less cognitive load for determining which variables are
defined in the callback's body.
2. It is possible to yield in the caller, so you could make that into a
coroutine too, however this wasn't required in this case, as the
yielded value was intended to be informational.
The downsides to this are:
1. It's a rather peculiar construct, so can be difficult to understand
what's going on, and the implications, which led to
2. If you call the function, but don't use the iterator it returned,
then it won't do anything, which is very confusing indeed, if you're
not used to how generator functions work.
|