| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |\
| |
| |
| |
| | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Pedro Alvarez
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
There was a check in it to see whether it needed to do the git branch
and git checkout based on whether the name of the branch matched that in
the morphology.
This had a couple of problems:
1. Now that we aren't always building from HEAD, we need to be able to
roll its commit back, so using the existing branch isn't always the
best idea.
2. It only checks the "ref" field, not "unpetrify-ref", so even though
we clone the right ref in there, it's checking the commit id against
the system branch name, so would always try to re-create the branch,
and fail when it already exists.
So now, we remove the original ref and re-create it with our preferred
HEAD.
A better solution might be to change the clone logic to not
automatically checkout HEAD, and instead require an explicit branch then
checkout, but the initial clone logic is shared with build, and I didn't
feel like tracking down all the different places that it was used.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
The help for the show-branch-root command said it returns a path, but
the command and the yarns just showed the aliased url it was cloned
from.
Given I found myself needing the path in some scripts, not the repo url,
I think it's more useful to reconcile the difference this way.
|
| |/
|
|
|
|
|
| |
We don't use this any more, and instead prefer to always keep
definitions.git petrified, and update the refs ourselves.
branch-from-image still uses some of the remaining petrify code.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Sorry about the big lump, I can split it into a nicer set of changes,
but they didn't naturally emerge in a nice series.
This creates a pushed_build_branch context manager, to eliminate code
duplication between build and deploy.
Rather than the build branch being constructed knowing whether it needs
to push the branch, it infers that from the state of the repositories,
and whether a local build would be possible.
If there are no uncommitted changes and all local branches are pushed,
then it doesn't create temporary branches or push them, and instead uses
what it already has.
It will currently create and use temporary build branches even for
chunks that have no local changes, but it's pretty cheap, and doesn't
require re-working the build-ref injection code to check whether there
are local changes.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We used to coincidentally include morphologies because we later include
every morphology in the build graph into our temporary build branches.
Since we're now checking whether there's any uncommitted changes before
attempting to create a temporary build branch, this means that we can no
longer build uncommitted morphologies if they aren't reported as
changes.
So this patch makes an exception to the untracked changes rule for
anything that ends with .morph.
It's still confusing that some files aren't included in temporary build
branches, but that would cause performance regressions, so we'll limit
it to just morphologies for now, until we make a decision on what
uncomitted content we care about.
|
| |
|
|
|
|
|
|
|
|
|
| |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously they were generator functions, which yielded interesting
context at interesting times so that the caller could respond by
printing status messages.
The only benefits this had over callbacks were:
1. 1 fewer function scope to worry about. I don't have data on the
amount of memory used for a function scope vs a generator, but it
could be less cognitive load for determining which variables are
defined in the callback's body.
2. It is possible to yield in the caller, so you could make that into a
coroutine too, however this wasn't required in this case, as the
yielded value was intended to be informational.
The downsides to this are:
1. It's a rather peculiar construct, so can be difficult to understand
what's going on, and the implications, which led to
2. If you call the function, but don't use the iterator it returned,
then it won't do anything, which is very confusing indeed, if you're
not used to how generator functions work.
|
| |\
| |
| |
| |
| |
| |
| |
| | |
This fixes a bug where distbuild was calculating a different cache
local build. It will no longer be needed once morph2 is removed.
Reviewed-by: Sam Thursfield
Reviewed-by: Richard Maw
|
| | |
| |
| |
| |
| |
| |
| |
| | |
morph2.Morphology adds a number of fields beginning with _orig_, which
contain default values. There is no reason for these values to be in
the cache key, and this fixes a bug where distbuild created systems with
a different cache key than local builds did when the configuration
extensions list was empty.
|
| |\ \
| | |
| | |
| | | |
'remotes/origin/baserock/richardmaw/bugfix/distbuild-eglibc'
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
If the destdir path returned when creating a staging area is a unicode
string, then when attempting to `os.walk(destdir)`, it will encounter
unicode errors if there are file paths in the destdir that are not
representable as unicode strings.
For various as-yet unknown reasons, when building stage-2 eglibc it
produces file paths that are not unicode compatible.
There was previously a patch to fix this issue with regards to creating
the metadata files, but it did not fix all the issues, because the build
at the time was local rather than distributed.
This is failing during a distributed build because morphologies are
serialised into json, and during deserialisation their string values are
left as unicode.
Rather than doing the byte-string conversion during deserialisation, I
have chosen to do it when the contents of the morphology are used,
because it's only at the point where it's used to create a file path,
that it matters whether it's unicode or not.
|
| | | | |
|
| | |/
|/|
| |
| |
| |
| |
| |
| | |
This prevents the description fields of morphologies being mangled.
This does not preserve the original formatting, so much as happen to
dump it in the same way we wrote it, but given we chose that form
because we think it looks the nicest, that's not a problem.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Adam Coldrick
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
The chunks in definitions change changed the api of create_source_pool,
and because list artifacts is not covered by any tests, and the
create_source_pool method was confused with a function of the same name
that did not need changing, it failed to be fixed.
|
| |/ /
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
With os.walk, if the target of the link doesn't exist, or it is a link
to a file, it ends up in the basenames list.
If it is a link to a directory, it goes in the subdirs list.
There's a bug in the subdirsymlinks check, in that it checks if the
wrong file is a symlink, so it never returns them.
This was missed, since we did not have the cross bootstrap in CI.
This is not eligible for our yarn tests, since to trigger this would
require changes to the host system's rootfs, so it's a system-level test.
To test this properly requires putting the cross bootstrap in CI.
|
| |\ \
| | |
| | |
| | | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It was referring to the remote git cache as 'remote artifact cache'.
These are usually the same server (a Trove, defaulting to
git.baserock.org) but can be configured independently, so they should
not be considered the same.
Indentally, the configuration / commandline options in question are
'git-resolve-cache-server' and 'artifact-cache-server'. Both default
to 'cache-server' if not set, which in turn defaults to
http://<trove-host>:8080/.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It will now load the morphology from the definitions repository if the
"morph" field is present in the chunk spec.
Rather than adapting the loop to fit yet-more changing circumstances, it
has been partially rewritten, so there is one loop for loading
systems and strata from definitions.git, another for chunks from
definitions.git, and a third for chunks in the source repository.
This is tidier than attempting to fit the logic in the main loop, as it
makes it easier to remove afterwards when we no longer need to load
chunk morphologies from the source repository.
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously the contents of the morphology would be included by virtue of
the fact that it came from the source repository, so would be included
in the "tree" field.
Now that chunk morphologies can come from the definitions repository, it
is not always included in the "tree" field, so the logical contents of
the morphology need to be included in the cache key computation.
Build commands are included after looking them up in the build-system,
so that in future, we don't need to change the chunk morphology
compatibility version when we change how build-systems work.
Since we may be moving the morphologies about in the definitions
repository, it would suck if we had to do a full rebuild after we move
things, so I dropped the filename from the cache key.
This also tweaks the system and stratum cache keys to include the
contents directly, rather than hashed in the "morphology-sha1" field.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
This hasn't been used in a long time to my knowledge, its API completely
misses the point, and its implementation relies on old APIs that need to
change.
Until we discover we need it, and work out what it should look like, I
think the best thing to do is get rid of it.
|
| |/ |
|
| |\
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius
Reviewed-by: Pedro Alvarez
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Parts of the morphology go into the name of the staging area, so it
helps to convert them into a str, so later attempts to join it with
another string don't result in a unicode string.
pyfilesystem insists that file paths must be unicode. It is incorrect,
but we passed something unicode compatible in in the first place, so we
can get away with converting it back to a bytestring.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| |/ |
|
| |\
| |
| |
| |
| |
| |
| | |
'origin/baserock/richardmaw/S11284/morphologies-by-path-v4'
Reviewed-by: Sam Thursfield
Reviewed-by: Lars Wirzenius
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We used to check whether a file existed before trying to read it. We
used to be able to get away with only looking at the top-level
directory, which made using ls-files before trying to cat-file it
better.
Unfortunately, we need to look at the files in subdirectories now, so
this no longer works.
We could make it include files in subdirectories, but for repositories
with many files, you would end up reading a file listing longer than
the morphology, so even in the slow case of needing to read the entire
morphology file, it would be faster to attempt to read the file first.
So now we beg forgiveness rather than asking permission.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This was noticed because our definitions.git had a dangling symlink,
so it failed to construct the temporary build branch.
We shouldn't process symlinks as morphologies either, since either we
make symlinked morphologies a first-class behaviour, and validate that
the link points inside the repository, which is a lot of work for
something we don't and probably won't need; or we can just assume that
we deal with the morphology this is linked to correctly anyway.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We want to use file paths to locate morphologies now, so the old model
of get a list of names and hand it those back to get the contents
doesn't really make sense any more.
This abstraction initially came about as one idea I had for moving
morphologies out of the source tree was to put them in something like
git notes, where it's possible to look up information for one commit in
another ref in the repository, at which point this abstraction would
have been flexible enough to handle that as well as in the
However, moving the chunk morphologies into the definitions repository
has other benefits too, so it makes more sense to be honest about using
filenames in the API.
It remains as a single point where we can put the logic for knowing which
files in a repository look like morphologies, but if we need to remove
any further functionality, it should be replaced by a single function.
|
| | |
| |
| |
| |
| | |
There's a lot of them, it's too much of a pain to enumerate them all, so
it's convenient to provide a hierachy and catch the base exceptions.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
`morph merge` only worked for a small subset of cases, and has been left
to bit-rot, since we don't use it.
`morph tag` is just a `git tag` when we have petrified definitions
repository. We don't use it, nor do we need it, so it can go away rather
than take up valuable development time fixing it when requirements
change.
`old-foo` have all been superceded by newer versions and are no-longer
used.
|
| | |
| |
| |
| |
| |
| |
| | |
This was the wrong response to the problem of accidentally checking-out
morph when trying to check out morphs. Now that it's called definitions,
this is irrelevent, and aborting a checkout when this check fails is the
wrong thing to do.
|
| | |
| |
| |
| |
| |
| | |
This was originally used by `morph edit`, but since we removed the need
to run `morph edit system stratum` and could shorten it to `morph edit
chunk`, this function is no longer used.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | | | |
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When overriding a default chunk splitting rule, the use 'break' would
mean that some of the other default chunk split rules might be ignored.
This isn't ever what you want.
I also clarified the current behaviour in a comment. I think in future
we should add a mechanism to extend the default rules, as well as the
current mechanism which allows only replacing them. But that is a
separate issue.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I was getting the following error when running the 'do-release.py'
script:
ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic
ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux
The commit that it wants did actually exist in git.baserock.org and the
logs show that Morph hadn't done a `git remote update` to get the commit
locally.
This turned out to be because it'd had already looked up a different ref
in linux.git. It hadn't needed to run 'git remote update', but it *had*
added linux.git to a "don't need to update this repo again" list. Oops!
I've moved the code in question to the cachedrepo module so that the repo
object keeps that of whether it should be updated. The bug is now fixed.
As a side effect this patch fixes the spurious 'Updating repo
file:///...' messages, which were printed even though repos with
file:/// URLs are never updated.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The error message:
ERROR: Failed to determine the build system of repo file://foo/bar/baz at
ref 59713cf997385a094091443fdcce9d5c17313f39: was looking for
distbuild-system-x86-64.morph
is confusing since our system morph has nothing to do with build systems,
the fact that build system autodetection is executed when looking for
distbuild-system-x86-64.morph is an implementation detail that shouldn't be
exposed to the user.
This patch replaces this error message with:
ERROR: Couldn't find morphology: distbuild-system-x86-64.morph
This is still not ideal, since there are cases where we may not be able
find the morph because build system autodection has failed, but out of the
two user typos/mistakes are probably more likely.
Differentiating between user error and build system detection failure would
require more substantial changes.
This patch also splits the get_morphology function into two functions
_get_morphology_text, that actually fetches the morph text from the cache
or otherwise infers it from the build system,
and get_morphology which constructs a Morphology object from the text.
|
| | |
|
| | |
|
| |
|
|
|
| |
This error message didn't report the cluster morph at fault.
We also fix a minor formatting issue.
|
| |\
| |
| |
| |
| | |
Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
When the given ref points to a specific commit, and it's already available in the
locally cached copy of repo, there's no need to update the repo.
If the ref points to a branch or tag, and the user didn't pass --no-git-update, the
locally cached copy of the repo will still be updated.
This should speed up many Morph commands.
|
| | | |
|
