| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
| |
Rather than generating the text of a morphology which is later loaded,
generate a Morphology object and return that.
|
| |
|
|
|
|
|
|
| |
This commit reworks morphologyfactory to stop using morph2. It removes
the validation that is already handled by morphloader, and uses
morphloader to load the morphology rather than using morph2. The unit
tests are also changed to turn the example morphologies into YAML, and
also to check for the correct exceptions.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
It was referring to the remote git cache as 'remote artifact cache'.
These are usually the same server (a Trove, defaulting to
git.baserock.org) but can be configured independently, so they should
not be considered the same.
Indentally, the configuration / commandline options in question are
'git-resolve-cache-server' and 'artifact-cache-server'. Both default
to 'cache-server' if not set, which in turn defaults to
http://<trove-host>:8080/.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We used to check whether a file existed before trying to read it. We
used to be able to get away with only looking at the top-level
directory, which made using ls-files before trying to cat-file it
better.
Unfortunately, we need to look at the files in subdirectories now, so
this no longer works.
We could make it include files in subdirectories, but for repositories
with many files, you would end up reading a file listing longer than
the morphology, so even in the slow case of needing to read the entire
morphology file, it would be faster to attempt to read the file first.
So now we beg forgiveness rather than asking permission.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The error message:
ERROR: Failed to determine the build system of repo file://foo/bar/baz at
ref 59713cf997385a094091443fdcce9d5c17313f39: was looking for
distbuild-system-x86-64.morph
is confusing since our system morph has nothing to do with build systems,
the fact that build system autodetection is executed when looking for
distbuild-system-x86-64.morph is an implementation detail that shouldn't be
exposed to the user.
This patch replaces this error message with:
ERROR: Couldn't find morphology: distbuild-system-x86-64.morph
This is still not ideal, since there are cases where we may not be able
find the morph because build system autodection has failed, but out of the
two user typos/mistakes are probably more likely.
Differentiating between user error and build system detection failure would
require more substantial changes.
This patch also splits the get_morphology function into two functions
_get_morphology_text, that actually fetches the morph text from the cache
or otherwise infers it from the build system,
and get_morphology which constructs a Morphology object from the text.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This now means that the system morphology is not altered when chunks are
altered, so some tests had to change.
Since this uses the python warnings API, these warnings can be ignored
by running
python -W ignore:"stratum morphology" \
-W ignore:"system morphology" \
"$(which morph)" ...`
or turned into errors with
python -W error:"stratum morphology" \
-W error:"system morphology" \
"$(which morph)" ...`
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
I think that it's confusing for both strata and chunk morphologies to
have a 'chunks' field, with the former listing sources and the latter
listing rules for splitting this source into artifacts.
The design for splitting strata has roughly the same idea, but operating
on chunk artifact names, rather than file names, so a name that can be
used for both was chosen.
Splits and artifacts weren't satisfactory names, so they're now called
'products'.
It was decided to break backwards compatibility of chunk morphologies
being able to specify 'chunks', since the format has changed, so extra
code would be required to translate the format, and the only users of
the 'chunks' field was the test suite, since there was no way to select
from the system, which chunk artifacts were included.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
A stratum morphology must either have stratum build-depends, or
have some bootstrap chunks, otherwise there's no way to have the
required set of commands to be able to build chunks.
A concession has been made to also allow strata that contain chunks
built in test mode, but this opens a reproducibility hole.
Unit tests for these failures have been added, and the stratum
used by other unit tests has been fixed.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It is better to test whether a straum is empty here, since it will be
noticed earlier in the build, as soon as the morphologies are loaded,
rather than after they have all been parsed.
It is also conceptually nicer to put it here, since the morphologyfactory
was written to perform this kind of validation.
On a more practical note, the validation is moved here so that the test
for this error isn't masked by the test for no build dependencies.
To ensure tests still pass, we alter the stratum morphology used by
other unit tests to no longer be empty, and add an empty one to test.
|
| |
|
|
|
|
|
|
|
| |
This creates a small exception hierarchy for failures to validate
stratum morphologies. This is currently a line rather than a tree,
but it will be expanded later in the patch series.
This also adds test coverage for chunk build dependencies being
omitted.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
| | |
|
| |\
| |
| |
| | |
Reviewed-By: concensus
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Define a specific set of 4 architectures that Morph supports, and only
expose that value to morphologies.
Since GNU triplets are very common we also expose a GNU triplet. Other
morphologies should work out their configuration based on MORPH_ARCH.
This commit also removes the morphlib.util.arch() function, which
detected the machine Morph is running on via 'uname -m'. Morph's
architecture names do not necessarily map to the output of 'uname -m'
so we should not rely on it anywhere.
|
| |\ \
| |/
| |
| |
| |
| |
| | |
Conflicts:
morphlib/bins_tests.py
Reviewed-By: consensus
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Allowed values:
staging: build with a staging chroot (default)
test: build with the host's tools
bootstrap: build with the host's tools, and do not include this
chunk in the final stratum artifact
In the past, 'normal mode' has been used to describe building a chunk
with the host's tools. We don't want that mode to ever be used,
because it is a huge hole in reproducability, but we need to keep it
around to avoid making Morph's cmdtest suite depend on Baserock.
Hopefully naming it 'test' should discourage potential abusers.
It is unfortunate that the build tests now take a separate code path
compared to real-world usage of Morph. However, this is necessary to
avoid a circular dependency between Morph's test suite and the
build-essential stratum in Baserock.
We do whole-build testing of Baserock, too, so the 'staging' code path
is still tested outside of Morph. However, testing a staging area
requires populating it with at minimum a working shell, and this is a
bit too complex to go in Morph's test suite.
|
| |/ |
|
| | |
|
| |
|
|
|
|
|
| |
Allowing differences causes only confusion. Whether or not the 'name'
field should surive at all is up for debate: it's very useful when
seeing the contents of the morphology out of context (e.g. in
documentation or pastebins), but is that worth the extra effort?
|
| |
|
|
|
| |
This can go away when we have made a release with yaml in it, and its
staging filler.
|
| |
|
|
|
| |
Tests are currently broken, one because invalid JSON
can be valid YAML, and coverage is incomplete.
|
| |
|
|
|
|
|
| |
This simply increases the verbosity of the operations which
are done against the remote artifact cache. It should result in some
progress messages if you build with --verbose set and you're mostly
getting stuff from your remote artifact cache (Trove).
|
| |
|
|
|
| |
armv5 and armv7 are different enough to be a problem, so specify
which particular arm sub-architecture we currently support.
|
| |
|
|
|
| |
This allows a few more diagnostics of what went wrong when the error
is inside the baserock:morphs repository.
|
| |
|
|
| |
Rename "sources" field of stratum morphologies to "chunks".
|
| |
|
|
|
| |
This provides much better performance in cases where most of the
repos do not have a .morph included.
|
| |
|
|
|
|
|
| |
The cost of one git ls-tree call is roughly the same as one
git cat-file call. Therefore, when autodetecting the build system,
it is much faster to list the tree once and then search for the
required files than to call git cat-file for every possible one.
|
| |
|
|
|
|
| |
It turns out that BaseException.message is deprecated, and str(e) is
a better way of achieving that. This makes the unit tests not spew out
a deprecation warning.
|
| |\
| |
| |
| |
| |
| | |
Conflicts:
morphlib/morphologyfactory.py
morphlib/morphologyfactory_tests.py
|
| | |\
| | |
| | |
| | |
| | |
| | |
| | | |
git://roadtrain.codethink.co.uk/baserock/morph
Merged with a tweak to the layout, since the code style's display
width is 79 characters, rather than 80.
|
| | |/ |
|
| |/
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
This way we can have one place in the code where we determine what
artifacts get built from a specific morphology, rather than spreading
the information around the code base.
From now on, everything is supposed to use the builds_artifacts attribute
to get the list of artifacts. ArtifactResolver has been changed to do that.
Some of the tests are now a bit messier, and should really be changed
to create Morphology objects using MorphologyFactory, but that's a change
for another day.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
| |
I think morphologies should be verified closer to where they are
created, not when they are traversed, since this verification would
be useful in more code paths.
This is in morphologyfactory, rather than Morphology itself, since
it may be useful to be able to create morphologies without checking,
such as in unit tests.
|
| |
|
|
|
|
|
|
|
|
| |
Detecting the build system is managed by it asking if any files exist
detecting if the file exists is done with a callback function.
This callback can use cat-file.
If list_files existed this could be more efficient as it would not
require the files to be read from the remote server and it only needs
to be one round-trip
|
| |
|
|
|
| |
There are cases where we would not need a remote cache, so it
should be able to operate without one.
|
| |
|
