| Commit message (Collapse) | Author | Age | Files | Lines |
|\
| |
| |
| |
| | |
Reviewed-by: Francisco Redondo Marchena
Reviewed-by: Sam Thursfield
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The MorphologyFactory class will use a RemoteRepoCache to see if a
morphology file exists, and if it doesn't, uses a file listing to see
if it can detect what build-system is uses, hence what the default
morphology should be.
However, it was overly generic in what error cases it would accept as
the morphology not being found, so if the RemoteRepoCache was suddenly
un-resolvable for a brief period, then it would assume the morphology
didn't exist, and use the default one.
This happened to a user, and the result was a full rebuild.
So we now fix this by only raising the exception that means the file
didn't exist, if we got a HTTP 404.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The clone_into function is non-functional when you pass it a sha1 ref.
If you have a file:// URI then this doesn't get used, which is how it
slipped past the tests.
Reviewed-by: Lars Wirzenius +2
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We assumed that the sha1 of the tree of the commit of the ref we care
about was sufficient to cache, but `git replace` means that you need to
know the state of other branches too, since anything in refs/replace can
completely change what the tree you check-out is.
This behaviour can be disabled globally by setting
GIT_NO_REPLACE_OBJECTS, so we're going to do that.
If we need to integrate a project that uses git-replace to change the
contents of their git trees then we could support that by:
if any(refs/replace/*):
potentially_replacable_objects = [
`git rev-parse HEAD`,
`git rev-parse HEAD^{commit}`,
`git rev-parse HEAD^{tree}`]
potentially_replacable_objects.extend(
`git ls-tree -r HEAD | awk '{print $3}'`)
# NOTE: doesn't handle submodules, and you'd need to expand this
# set whenever you process a replacement
for object in refs/replace/*:
if basename(object) not in potentially_replacable_objects: continue
cache_key['replacements'][basename(object)] = `git rev-parse $object`
If we were to support this would need to traverse the tree anyway, doing
replacements, so we may as well use libgit to do the checkout anyway,
and list which replacements were used.
However, since the expected use-case of `git replace` is as a better way
to do history grafting, we're unlikely to need it, as it would only have
any effect if it replaced the commit we were using with a different one.
Rubber-stamped-by: Daniel Silverstone
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This avoids confusion when the user expected to be doing an initial
deployment and wasn't aware that a file with the same name as the
target already existed. Previously rawdisk.write would try to mount
the file and upgrade it.
Now we require the user to pass '--upgrade' when they intend to
upgrade, as with other deployment extensions.
|
|/
|
|
|
|
|
|
| |
This reduces the vast number of 'git config -z core.bare' which
we used to get a lot of, to one or two per run. It doesn't
save much time, but it does make logs less full of confusion.
Reviewed-By: Richard maw <richard.maw@codethink.co.uk> +2
|
| |
|
| |
|
| |
|
| |
|
| |
|
|\
| |
| |
| |
| | |
Reviewed by: Richard Maw <richard.maw@codethink.co.uk>
Merged by: James Thomas <james.thomas@codethink.co.uk>
|
| |\ |
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Remove the BOOTLOADER environment variable and instead favour
BOOTLOADER_CONFIG_FORMAT to set the desired bootloader format, and
BOOTLOADER_INSTALL to set the type of bootloader to install.
For example, since u-boot can boot using extlinux.conf files, it's
conceivable that someone might want to do CONFIG_FORMAT=extlinux.conf,
INSTALL=u-boot.
However, for most platforms you would want to set INSTALL to "none"
|
| |/ |
|
| |
| |
| |
| |
| |
| | |
Instead of leaving morph3 with a potentially confusing name, rename it
to `morphology` since it is now the only implementation of the
Morphology class.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This reverts commit 4124c50b8dc3dfb0ffb933153d0fe6385edf389c.
This commit is no longer needed now that morph2 is gone.
Conflicts:
morphlib/cachekeycomputer.py
|
| |
| |
| |
| | |
This commit removes the now unneeded morph2 and its associated tests.
|
| |
| |
| |
| |
| | |
This commit removes all use of morph2 from the unittests, replacing it
with morphloader/morph3. It also converts the test morphologies to YAML.
|
| |
| |
| |
| |
| | |
Rather than generating the text of a morphology which is later loaded,
generate a Morphology object and return that.
|
| |
| |
| |
| |
| |
| |
| |
| | |
This commit reworks morphologyfactory to stop using morph2. It removes
the validation that is already handled by morphloader, and uses
morphloader to load the morphology rather than using morph2. The unit
tests are also changed to turn the example morphologies into YAML, and
also to check for the correct exceptions.
|
| |
| |
| |
| |
| |
| | |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This commit stops morphloader setting the `morph` field in chunk specs,
and also makes it set defaults for the prefix and build-mode fields. Not
setting the `morph` field is necessary as its presence in chunk specs is
used by `traverse_morphs` to mean that the morphology file is in the
definitions repository, not the chunk source repository. If we set a
default value here, we end up looking for files which do not exist.
|
| |
| |
| |
| |
| |
| |
| |
| | |
Currently, morphloader sets the morph field in chunk specs by default.
A later commit will remove this behaviour, so morphset needs to stop
expecting the `morph` field to exist in morphology specs. However,
stratum specs don't have a `name` field so we can't expect to be able to
find that either.
|
| | |
|
| | |
|
|\ \
| | |
| | |
| | |
| | | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Pedro Alvarez
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
There was a check in it to see whether it needed to do the git branch
and git checkout based on whether the name of the branch matched that in
the morphology.
This had a couple of problems:
1. Now that we aren't always building from HEAD, we need to be able to
roll its commit back, so using the existing branch isn't always the
best idea.
2. It only checks the "ref" field, not "unpetrify-ref", so even though
we clone the right ref in there, it's checking the commit id against
the system branch name, so would always try to re-create the branch,
and fail when it already exists.
So now, we remove the original ref and re-create it with our preferred
HEAD.
A better solution might be to change the clone logic to not
automatically checkout HEAD, and instead require an explicit branch then
checkout, but the initial clone logic is shared with build, and I didn't
feel like tracking down all the different places that it was used.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
The help for the show-branch-root command said it returns a path, but
the command and the yarns just showed the aliased url it was cloned
from.
Given I found myself needing the path in some scripts, not the repo url,
I think it's more useful to reconcile the difference this way.
|
|/ /
| |
| |
| |
| |
| |
| | |
We don't use this any more, and instead prefer to always keep
definitions.git petrified, and update the refs ourselves.
branch-from-image still uses some of the remaining petrify code.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Sorry about the big lump, I can split it into a nicer set of changes,
but they didn't naturally emerge in a nice series.
This creates a pushed_build_branch context manager, to eliminate code
duplication between build and deploy.
Rather than the build branch being constructed knowing whether it needs
to push the branch, it infers that from the state of the repositories,
and whether a local build would be possible.
If there are no uncommitted changes and all local branches are pushed,
then it doesn't create temporary branches or push them, and instead uses
what it already has.
It will currently create and use temporary build branches even for
chunks that have no local changes, but it's pretty cheap, and doesn't
require re-working the build-ref injection code to check whether there
are local changes.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We used to coincidentally include morphologies because we later include
every morphology in the build graph into our temporary build branches.
Since we're now checking whether there's any uncommitted changes before
attempting to create a temporary build branch, this means that we can no
longer build uncommitted morphologies if they aren't reported as
changes.
So this patch makes an exception to the untracked changes rule for
anything that ends with .morph.
It's still confusing that some files aren't included in temporary build
branches, but that would cause performance regressions, so we'll limit
it to just morphologies for now, until we make a decision on what
uncomitted content we care about.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This was previously used just so it could get the right repo and ref to
read the file out of.
However, there was a subtle bug in this behaviour, as if we had not
previously used morph build in that branch, it would attempt to read the
extensions from a branch which didn't exist.
So now it reads it from the working tree, which always exists.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously they were generator functions, which yielded interesting
context at interesting times so that the caller could respond by
printing status messages.
The only benefits this had over callbacks were:
1. 1 fewer function scope to worry about. I don't have data on the
amount of memory used for a function scope vs a generator, but it
could be less cognitive load for determining which variables are
defined in the callback's body.
2. It is possible to yield in the caller, so you could make that into a
coroutine too, however this wasn't required in this case, as the
yielded value was intended to be informational.
The downsides to this are:
1. It's a rather peculiar construct, so can be difficult to understand
what's going on, and the implications, which led to
2. If you call the function, but don't use the iterator it returned,
then it won't do anything, which is very confusing indeed, if you're
not used to how generator functions work.
|
|\
| |
| |
| |
| |
| |
| |
| | |
This fixes a bug where distbuild was calculating a different cache
local build. It will no longer be needed once morph2 is removed.
Reviewed-by: Sam Thursfield
Reviewed-by: Richard Maw
|
| |
| |
| |
| |
| |
| |
| |
| | |
morph2.Morphology adds a number of fields beginning with _orig_, which
contain default values. There is no reason for these values to be in
the cache key, and this fixes a bug where distbuild created systems with
a different cache key than local builds did when the configuration
extensions list was empty.
|
|\ \
| | |
| | |
| | | |
'remotes/origin/baserock/richardmaw/bugfix/distbuild-eglibc'
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
If the destdir path returned when creating a staging area is a unicode
string, then when attempting to `os.walk(destdir)`, it will encounter
unicode errors if there are file paths in the destdir that are not
representable as unicode strings.
For various as-yet unknown reasons, when building stage-2 eglibc it
produces file paths that are not unicode compatible.
There was previously a patch to fix this issue with regards to creating
the metadata files, but it did not fix all the issues, because the build
at the time was local rather than distributed.
This is failing during a distributed build because morphologies are
serialised into json, and during deserialisation their string values are
left as unicode.
Rather than doing the byte-string conversion during deserialisation, I
have chosen to do it when the contents of the morphology are used,
because it's only at the point where it's used to create a file path,
that it matters whether it's unicode or not.
|
| | | |
|
| |/
|/|
| |
| |
| |
| |
| |
| | |
This prevents the description fields of morphologies being mangled.
This does not preserve the original formatting, so much as happen to
dump it in the same way we wrote it, but given we chose that form
because we think it looks the nicest, that's not a problem.
|
|\ \
| | |
| | |
| | |
| | | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Adam Coldrick
|
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
The chunks in definitions change changed the api of create_source_pool,
and because list artifacts is not covered by any tests, and the
create_source_pool method was confused with a function of the same name
that did not need changing, it failed to be fixed.
|
|/ /
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
With os.walk, if the target of the link doesn't exist, or it is a link
to a file, it ends up in the basenames list.
If it is a link to a directory, it goes in the subdirs list.
There's a bug in the subdirsymlinks check, in that it checks if the
wrong file is a symlink, so it never returns them.
This was missed, since we did not have the cross bootstrap in CI.
This is not eligible for our yarn tests, since to trigger this would
require changes to the host system's rootfs, so it's a system-level test.
To test this properly requires putting the cross bootstrap in CI.
|
|\ \
| | |
| | |
| | | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It was referring to the remote git cache as 'remote artifact cache'.
These are usually the same server (a Trove, defaulting to
git.baserock.org) but can be configured independently, so they should
not be considered the same.
Indentally, the configuration / commandline options in question are
'git-resolve-cache-server' and 'artifact-cache-server'. Both default
to 'cache-server' if not set, which in turn defaults to
http://<trove-host>:8080/.
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
It will now load the morphology from the definitions repository if the
"morph" field is present in the chunk spec.
Rather than adapting the loop to fit yet-more changing circumstances, it
has been partially rewritten, so there is one loop for loading
systems and strata from definitions.git, another for chunks from
definitions.git, and a third for chunks in the source repository.
This is tidier than attempting to fit the logic in the main loop, as it
makes it easier to remove afterwards when we no longer need to load
chunk morphologies from the source repository.
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Previously the contents of the morphology would be included by virtue of
the fact that it came from the source repository, so would be included
in the "tree" field.
Now that chunk morphologies can come from the definitions repository, it
is not always included in the "tree" field, so the logical contents of
the morphology need to be included in the cache key computation.
Build commands are included after looking them up in the build-system,
so that in future, we don't need to change the chunk morphology
compatibility version when we change how build-systems work.
Since we may be moving the morphologies about in the definitions
repository, it would suck if we had to do a full rebuild after we move
things, so I dropped the filename from the cache key.
This also tweaks the system and stratum cache keys to include the
contents directly, rather than hashed in the "morphology-sha1" field.
|