| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Instead of leaving morph3 with a potentially confusing name, rename it
to `morphology` since it is now the only implementation of the
Morphology class.
|
| |
|
|
|
|
|
|
|
| |
This reverts commit 4124c50b8dc3dfb0ffb933153d0fe6385edf389c.
This commit is no longer needed now that morph2 is gone.
Conflicts:
morphlib/cachekeycomputer.py
|
| |
|
|
| |
This commit removes the now unneeded morph2 and its associated tests.
|
| |
|
|
|
|
| |
Update the edit-morph script used in the test suite to use morphloader
for saving/loading morphologies rather than morph2. Also remove some
unused code.
|
| | |
|
| | |
|
| |
|
|
|
| |
This commit removes all use of morph2 from the unittests, replacing it
with morphloader/morph3. It also converts the test morphologies to YAML.
|
| |
|
|
|
| |
Rather than generating the text of a morphology which is later loaded,
generate a Morphology object and return that.
|
| |
|
|
|
|
|
|
| |
This commit reworks morphologyfactory to stop using morph2. It removes
the validation that is already handled by morphloader, and uses
morphloader to load the morphology rather than using morph2. The unit
tests are also changed to turn the example morphologies into YAML, and
also to check for the correct exceptions.
|
| |
|
|
|
|
| |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
| |
|
|
|
|
|
|
|
| |
This commit stops morphloader setting the `morph` field in chunk specs,
and also makes it set defaults for the prefix and build-mode fields. Not
setting the `morph` field is necessary as its presence in chunk specs is
used by `traverse_morphs` to mean that the morphology file is in the
definitions repository, not the chunk source repository. If we set a
default value here, we end up looking for files which do not exist.
|
| |
|
|
|
|
|
|
| |
Currently, morphloader sets the morph field in chunk specs by default.
A later commit will remove this behaviour, so morphset needs to stop
expecting the `morph` field to exist in morphology specs. However,
stratum specs don't have a `name` field so we can't expect to be able to
find that either.
|
| | |
|
| | |
|
| |\
| |
| |
| |
| |
| |
| |
| | |
This fixes a bug where distbuild was calculating a different cache
local build. It will no longer be needed once morph2 is removed.
Reviewed-by: Sam Thursfield
Reviewed-by: Richard Maw
|
| | |
| |
| |
| |
| |
| |
| |
| | |
morph2.Morphology adds a number of fields beginning with _orig_, which
contain default values. There is no reason for these values to be in
the cache key, and this fixes a bug where distbuild created systems with
a different cache key than local builds did when the configuration
extensions list was empty.
|
| |\ \
| | |
| | |
| | | |
'remotes/origin/baserock/richardmaw/bugfix/distbuild-eglibc'
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
If the destdir path returned when creating a staging area is a unicode
string, then when attempting to `os.walk(destdir)`, it will encounter
unicode errors if there are file paths in the destdir that are not
representable as unicode strings.
For various as-yet unknown reasons, when building stage-2 eglibc it
produces file paths that are not unicode compatible.
There was previously a patch to fix this issue with regards to creating
the metadata files, but it did not fix all the issues, because the build
at the time was local rather than distributed.
This is failing during a distributed build because morphologies are
serialised into json, and during deserialisation their string values are
left as unicode.
Rather than doing the byte-string conversion during deserialisation, I
have chosen to do it when the contents of the morphology are used,
because it's only at the point where it's used to create a file path,
that it matters whether it's unicode or not.
|
| |\ \ \
| | | |
| | | |
| | | | |
Reviewed-by: Richard Maw (+2)
|
| |/ / / |
|
| |\ \ \
| |_|/
|/| |
| | | |
'origin/baserock/richardmaw/bugfix/yaml-multi-line-dump'
|
| |/ /
| |
| |
| |
| |
| |
| |
| | |
This prevents the description fields of morphologies being mangled.
This does not preserve the original formatting, so much as happen to
dump it in the same way we wrote it, but given we chose that form
because we think it looks the nicest, that's not a problem.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Adam Coldrick
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
The chunks in definitions change changed the api of create_source_pool,
and because list artifacts is not covered by any tests, and the
create_source_pool method was confused with a function of the same name
that did not need changing, it failed to be fixed.
|
| |\ \ \
| |/ /
|/| |
| | | |
Reviewed-by: Daniel Silverstone
|
| |/ /
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
With os.walk, if the target of the link doesn't exist, or it is a link
to a file, it ends up in the basenames list.
If it is a link to a directory, it goes in the subdirs list.
There's a bug in the subdirsymlinks check, in that it checks if the
wrong file is a symlink, so it never returns them.
This was missed, since we did not have the cross bootstrap in CI.
This is not eligible for our yarn tests, since to trigger this would
require changes to the host system's rootfs, so it's a system-level test.
To test this properly requires putting the cross bootstrap in CI.
|
| |\ \
| | |
| | |
| | | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
It was referring to the remote git cache as 'remote artifact cache'.
These are usually the same server (a Trove, defaulting to
git.baserock.org) but can be configured independently, so they should
not be considered the same.
Indentally, the configuration / commandline options in question are
'git-resolve-cache-server' and 'artifact-cache-server'. Both default
to 'cache-server' if not set, which in turn defaults to
http://<trove-host>:8080/.
|
| |\ \ \
| |/ /
|/| |
| | |
| | | |
Reviewed-by: Daniel Silverstone
Reviewed-by: Sam Thursfield
|
| | | |
| | |
| | |
| | |
| | |
| | | |
It is entirely possible that we could accidentally give chunks that use
morphologies from the definitions repository, the definitions repository
to build from, rather than the source repository.
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
It will now load the morphology from the definitions repository if the
"morph" field is present in the chunk spec.
Rather than adapting the loop to fit yet-more changing circumstances, it
has been partially rewritten, so there is one loop for loading
systems and strata from definitions.git, another for chunks from
definitions.git, and a third for chunks in the source repository.
This is tidier than attempting to fit the logic in the main loop, as it
makes it easier to remove afterwards when we no longer need to load
chunk morphologies from the source repository.
|
| | | | |
|
| | | | |
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Previously the contents of the morphology would be included by virtue of
the fact that it came from the source repository, so would be included
in the "tree" field.
Now that chunk morphologies can come from the definitions repository, it
is not always included in the "tree" field, so the logical contents of
the morphology need to be included in the cache key computation.
Build commands are included after looking them up in the build-system,
so that in future, we don't need to change the chunk morphology
compatibility version when we change how build-systems work.
Since we may be moving the morphologies about in the definitions
repository, it would suck if we had to do a full rebuild after we move
things, so I dropped the filename from the cache key.
This also tweaks the system and stratum cache keys to include the
contents directly, rather than hashed in the "morphology-sha1" field.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This hasn't been used in a long time to my knowledge, its API completely
misses the point, and its implementation relies on old APIs that need to
change.
Until we discover we need it, and work out what it should look like, I
think the best thing to do is get rid of it.
|
| | | |
| | |
| | |
| | |
| | |
| | | |
We don't use this in definitions.git, and we're going to change its
semantics, so the test suite would break until we introduced the new
semantics, unless we remove its use of the old semantics first.
|
| | | | |
|
| |/ /
| |
| |
| |
| |
| | |
This is required to ensure the right version of morph is used. I have
a .bashrc that causes `morph` to be "$HOME/morph/morph", so it fails to
find morph, because HOME is set to a directory inside DATADIR.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The "unicode fix" worked for the subset of cases relevant, and only
broke distbuild because its tests have not been integrated with ./check,
so the fact that it broke for any string ending with a \ escaped notice,
if you will excuse the pun.
During json.load, the encode option is for specifying the character
encoding of the file or string that is being loaded.
During json.dump, the encode option is for the encoding of `str` keys
and values.
The fact that it worked for the set of cases we cared about is a small
mystery, probably caused by the strings we happened to give it being
valid unicode-escape encoded `str`ings.
A full fix would require either converting all these cases to a
different format, such as YAML, which will handle input data not being
valid Unicode, or pre-processing the data that is passed to `json.dump`
to convert all `str` instances to an appropriately escaped `unicode`,
and converting back on `json.load`, but this is a quick fix to get the
distbuild code working again.
|
| |\
| |
| |
| |
| | |
Reviewed-by: Lars Wirzenius
Reviewed-by: Pedro Alvarez
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Parts of the morphology go into the name of the staging area, so it
helps to convert them into a str, so later attempts to join it with
another string don't result in a unicode string.
pyfilesystem insists that file paths must be unicode. It is incorrect,
but we passed something unicode compatible in in the first place, so we
can get away with converting it back to a bytestring.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
json only accepts unicode. Various APIs such as file paths and environment
variables allow binary data, so we need to support this properly.
This patch changes every[1] use of json.load or json.dump to escape
non-unicode data strings. This appears exactly as it used to if the
input was valid unicode, if it isn't it will insert \xabcd escapes in
the place of non-unicode data.
When loading back in, if json.load is told to unescape it with
`encoding='unicode-escape'` then it will convert it back correctly.
This change was primarily to support file paths that weren't valid
unicode, where this would choke and die. Now it works, but any tools
that parsed the metadata need to unescape the paths.
[1]: The interface to the remote repo cache uses json data, but I haven't
changes its json.load calls to unescape the data, since the repo
caches haven't been made to escape the data.
|
| | | |
|
| |/
|
|
|
| |
This will make it easier to determine what is wrong if the controller
daemon is run with a bad controller host address.
|
| |\
| |
| |
| |
| |
| |
| | |
'origin/baserock/richardmaw/S11284/morphologies-by-path-v4'
Reviewed-by: Sam Thursfield
Reviewed-by: Lars Wirzenius
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We want to move our morphologies in our definitions repository into
subdirectories, so they're more organised.
We'd prefer to only refer to morphologies by file path, rather than a
name that loosely corresponds to the file path, but we need to support
that for backwards compatibility until we can move all of our
morphologies into the definitions repository.
However, since we want to eventually remove this, and we want to ensure
that file paths work, we change the yarn tests to use file paths.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
We used to check whether a file existed before trying to read it. We
used to be able to get away with only looking at the top-level
directory, which made using ls-files before trying to cat-file it
better.
Unfortunately, we need to look at the files in subdirectories now, so
this no longer works.
We could make it include files in subdirectories, but for repositories
with many files, you would end up reading a file listing longer than
the morphology, so even in the slow case of needing to read the entire
morphology file, it would be faster to attempt to read the file first.
So now we beg forgiveness rather than asking permission.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
This is an inversion of the previous logic, which would strip the .morph
extension and use the "name" internally.
The exception to this rule of always using the filename, is that `morph
edit CHUNK` uses the name of the morphology as-defined in the stratum.
This is based off Adam Coldrick's inital patch, but this version will
allow the old style of providing the "name" by converting it into a path
if it does not have either a / or a . in it.
An unfortunate consequence of this change is that the show-dependencies
command's output changed, so the test needed updating.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This was noticed because our definitions.git had a dangling symlink,
so it failed to construct the temporary build branch.
We shouldn't process symlinks as morphologies either, since either we
make symlinked morphologies a first-class behaviour, and validate that
the link points inside the repository, which is a lot of work for
something we don't and probably won't need; or we can just assume that
we deal with the morphology this is linked to correctly anyway.
|
