| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
| | |
|
| |\
| |
| |
| |
| | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Pedro Alvarez
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There was a check in it to see whether it needed to do the git branch
and git checkout based on whether the name of the branch matched that in
the morphology.
This had a couple of problems:
1. Now that we aren't always building from HEAD, we need to be able to
roll its commit back, so using the existing branch isn't always the
best idea.
2. It only checks the "ref" field, not "unpetrify-ref", so even though
we clone the right ref in there, it's checking the commit id against
the system branch name, so would always try to re-create the branch,
and fail when it already exists.
So now, we remove the original ref and re-create it with our preferred
HEAD.
A better solution might be to change the clone logic to not
automatically checkout HEAD, and instead require an explicit branch then
checkout, but the initial clone logic is shared with build, and I didn't
feel like tracking down all the different places that it was used.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
The help for the show-branch-root command said it returns a path, but
the command and the yarns just showed the aliased url it was cloned
from.
Given I found myself needing the path in some scripts, not the repo url,
I think it's more useful to reconcile the difference this way.
|
| |/
|
|
|
|
|
| |
We don't use this any more, and instead prefer to always keep
definitions.git petrified, and update the refs ourselves.
branch-from-image still uses some of the remaining petrify code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sorry about the big lump, I can split it into a nicer set of changes,
but they didn't naturally emerge in a nice series.
This creates a pushed_build_branch context manager, to eliminate code
duplication between build and deploy.
Rather than the build branch being constructed knowing whether it needs
to push the branch, it infers that from the state of the repositories,
and whether a local build would be possible.
If there are no uncommitted changes and all local branches are pushed,
then it doesn't create temporary branches or push them, and instead uses
what it already has.
It will currently create and use temporary build branches even for
chunks that have no local changes, but it's pretty cheap, and doesn't
require re-working the build-ref injection code to check whether there
are local changes.
|
| |
|
|
|
|
|
|
|
|
|
| |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously they were generator functions, which yielded interesting
context at interesting times so that the caller could respond by
printing status messages.
The only benefits this had over callbacks were:
1. 1 fewer function scope to worry about. I don't have data on the
amount of memory used for a function scope vs a generator, but it
could be less cognitive load for determining which variables are
defined in the callback's body.
2. It is possible to yield in the caller, so you could make that into a
coroutine too, however this wasn't required in this case, as the
yielded value was intended to be informational.
The downsides to this are:
1. It's a rather peculiar construct, so can be difficult to understand
what's going on, and the implications, which led to
2. If you call the function, but don't use the iterator it returned,
then it won't do anything, which is very confusing indeed, if you're
not used to how generator functions work.
|
| | |
|
| |
|
|
|
|
|
| |
The chunks in definitions change changed the api of create_source_pool,
and because list artifacts is not covered by any tests, and the
create_source_pool method was confused with a function of the same name
that did not need changing, it failed to be fixed.
|
| |
|
|
|
|
|
|
|
| |
This hasn't been used in a long time to my knowledge, its API completely
misses the point, and its implementation relies on old APIs that need to
change.
Until we discover we need it, and work out what it should look like, I
think the best thing to do is get rid of it.
|
| | |
|
| |\
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius
Reviewed-by: Pedro Alvarez
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| |/ |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We want to use file paths to locate morphologies now, so the old model
of get a list of names and hand it those back to get the contents
doesn't really make sense any more.
This abstraction initially came about as one idea I had for moving
morphologies out of the source tree was to put them in something like
git notes, where it's possible to look up information for one commit in
another ref in the repository, at which point this abstraction would
have been flexible enough to handle that as well as in the
However, moving the chunk morphologies into the definitions repository
has other benefits too, so it makes more sense to be honest about using
filenames in the API.
It remains as a single point where we can put the logic for knowing which
files in a repository look like morphologies, but if we need to remove
any further functionality, it should be replaced by a single function.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
`morph merge` only worked for a small subset of cases, and has been left
to bit-rot, since we don't use it.
`morph tag` is just a `git tag` when we have petrified definitions
repository. We don't use it, nor do we need it, so it can go away rather
than take up valuable development time fixing it when requirements
change.
`old-foo` have all been superceded by newer versions and are no-longer
used.
|
| |
|
|
|
|
|
| |
This was the wrong response to the problem of accidentally checking-out
morph when trying to check out morphs. Now that it's called definitions,
this is irrelevent, and aborting a checkout when this check fails is the
wrong thing to do.
|
| |
|
|
|
|
| |
This was originally used by `morph edit`, but since we removed the need
to run `morph edit system stratum` and could shorten it to `morph edit
chunk`, this function is no longer used.
|
| | |
|
| |
|
|
|
|
|
|
| |
Instead of taking the name of a cluster morphology and zero or more
parameters for overriding the cluster morphology, morph deploy should
take the name of a cluster morphology and the names of zero or more
system deployments that are defined in the cluster morphology. If no
deployment names are given then all deployments are deployed.
|
| |\
| |
| |
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| | |
There is code that assumes these exist in at least one place:
StagingArea.abort(). That code should be fixed, but we should also stop
deleting them every time we run 'morph gc'.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Previously Morph would check if an artifact is present in the remote
artifact cache, then fetch the necessary files. If an error occured
during fetching, it would raise an error and abort.
Instead, we should just try and fetch the files and if anything fails,
move on to building locally. This avoids the situation where an error in
the remote cache makes local building impossible, which we experienced
recently.
|
| | |/
|/|
| |
| |
| |
| |
| |
| | |
The deploy plugin is relying on existing code which respects the
'no-git-update' setting and updates gits if it is not set.
Since deploy can only work for built systems anyway, we can force this
True for deployments, to avoid wasting the user/caller's time.
|
| | |
| |
| |
| |
| |
| |
| | |
- All tests now pass
- The odd case of chunks with the same name but different repo URLs now
correctly informs the user of the multiple checkouts that were done.
- Tidyups
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since we removed ref: fields from the system morph files, some of the logic
in morph edit is no longer needed. In particular, running
morph edit <system> <stratum>
is a no-op. This is because the version of <system> and <stratum> are now
implicit from the context of what we are doing. In other words we're always
working with the current version of <system> and <stratum> from the system
branch we are in.
Because of the complexity of morph's logic, we didn't notice this when
dropping the ref: fields, and we missed the opportunity to simplify our
logic for 'morph edit'
This patch aims to provide the simplest possible implementation of
morph edit:
morph edit <chunk>
It checks all strata for instances of <chunk>, and does what morph edit should
do for the instances it finds.
A later patch can add warnings to help users deal with situations where <chunk>
is not found, or is found more than once.
Also since this changes the syntax of morph, it breaks many of our tests. Later
patches will address that.
|
| | |
| |
| |
| |
| |
| |
| |
| | |
Generally deployment temp dirs are removed by morph during deployment,
but in some cases deployment dirs may not be cleared up,
for example if morph gets a SIGKILL or something unexpected
happens that causes morph to terminate without having a chance
to cleanup.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The copy-artifacts and list-artifacts commands are mainly useful for
making releases. As part of the release process we copy artifacts for
the entire build graph of the release to the artifact cache on
trove.baserock.org, to provide Baserock users with ready-built
artifacts.
This part of the release process is now automated, and the automation
require the list-artifacts command to function as a 'plumbing' command.
The copy-artifacts command is no longer required. It can be replaced
with:
morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET
The previous version of this plugin looked in the system artifact's
metadata for the list of artifacts. This is flawed as the final system
does not necessarily contain every build dependency. The new version of
the plugin calculates the build graph from source, using the same
process as the 'buildcommand' module. It also required looking in
Morph's artifact cache for the system artifact file to analyse.
|
| |
|
|
|
|
|
|
|
| |
This is useful to build releases using distbuild. It avoids having the
SHA1 fields in the artifact metadata files pointing to commits that
exist only on temporary build branches. It also avoids file:// URLs in
the repo fields. Note that the repo URL still points to the Trove used
by the distbuild network, rather than being an upstream URL pointing to
git.baserock.org.
|
| |
|
|
| |
Makes it easier to see what they mean at a glance.
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| | |
Signed-off by: Adam Coldrick
Reviewed by:
Daniel Silverstone <daniel.silverstone@codethink.co.uk>
Richard Maw <richard.maw@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since git's parsing of .gitattributes can't deal with spaces, or
even with quotes, replace any whitespace with [[:space:]]. Also,
make sure to escape any glob metacharacters to avoid accidental
pattern matches.
Note:
This approach could cause a file "foo<tab>bar" to be treated
as a binary if a file "foo bar" has already been added with
morph add-binary. However this is as good of a workaround as
I can find.
Signed-off by: Adam Coldrick
|
| |/
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
tarfile's open needs the file-like object
to have a tell() method, objects returned
from sockets don't have this method.
So instead we fetch the artifact from the
remote and cache it locally
|
| | |
| |
| |
| |
| |
| |
| | |
The constructor for BuildCommand sets up the caches,
we want the caches to be set up for distbuild too
so that we can deploy a system (system artifact will be
fetched from the artifact cache)
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Add a plugin to implement both `morph push` and `morph pull`.
These commands are wrappers around the corresponding git commands
push and pull, which also implement the functionality of pushing
and pulling large files provided by git-fat.
For example, running `morph pull` will pull any commits from the
remote branch not on your local branch, and then pull any large
files from the separate git-fat/rsync store on the Trove.
|
| | |
| |
| |
| |
| |
| |
| | |
Add a plugin which implements the morph add-binary command. This
command is used to add large files to a git repository. It sets
up the files needed to use git-fat, and then runs `git add` with
git-fat initiated.
|
| | |
| |
| |
| |
| |
| | |
When cloning a repository, the files stored using git-fat need to
be pulled. This situation occurs in `morph branch`, `morph edit`,
and `morph checkout`.
|
| | | |
|
| | |
| |
| |
| | |
This also makes it obey status prefix
|
