| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
| |
We were previously memoising the computation of the dictionaries
but this patch adds support for memoising the computation of the
cache key itself (the SHA string). This massively improves cache
key computation performance.
Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|
|
|
|
|
|
|
| |
This messes up the baserock-system-config-sync tool. Systemd does not
require /etc/fstab to exist in any case.
I have bumped the 'system-compatibility-version' field in this commit
to trigger rebuilding all system artifacts.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If two systems with the same name (e.g. different repo/ref) depend on the
same strata, then it will collide with systems which depend on different
artifacts from that stratum, but the same number of artifacts.
For example, if you checkout an existing branch and change the artifacts
used by one of its strata, then your local changes won't be built.
This is because the 'kids' field lists artifacts it depends on by their
cache-key, which is now no longer sufficient to uniquely identify
artifacts. The same number of artifacts issue is from it listing cache
keys multiple times.
The fix for this is to include the artifact name, so the 'kids' field is
now a list of dicts, with artifact name and cache key.
This is a dict rather than a tuple so that the generated /baserock
metadata is more readable.
|
|
|
|
|
|
| |
For chunks the products field doesn't need to be hashed, since the
split-rules field is used instead, and includes the default rule set,
and for strata and systems it is handled by hashing the dependencies.
|
|
|
|
|
|
|
|
|
|
|
|
| |
Chunk artifacts need the [(artifact_name, [regular_expression])] so that
if the default split rules change, or the blending rule changes, then
an extra version field doesn't need to be added to the cache key computer.
This is for future plans to allow the split rules to be configurable
and allow us to more easily change them.
System and Stratum artifact computations don't need this, since those
splitting rules are already expressed in the dependencies information.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Define a specific set of 4 architectures that Morph supports, and only
expose that value to morphologies.
Since GNU triplets are very common we also expose a GNU triplet. Other
morphologies should work out their configuration based on MORPH_ARCH.
This commit also removes the morphlib.util.arch() function, which
detected the machine Morph is running on via 'uname -m'. Morph's
architecture names do not necessarily map to the output of 'uname -m'
so we should not rely on it anywhere.
|
|
|
|
|
|
|
|
|
| |
Morph no longer supports setting the prefix using the --prefix
argument / setting. This was only used in tests and during bootstrap.
If a chunk build-depends on a chunk within a stratum which has
a custom prefix, that prefix is appended to the PATH in the build
environment.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allowed values:
staging: build with a staging chroot (default)
test: build with the host's tools
bootstrap: build with the host's tools, and do not include this
chunk in the final stratum artifact
In the past, 'normal mode' has been used to describe building a chunk
with the host's tools. We don't want that mode to ever be used,
because it is a huge hole in reproducability, but we need to keep it
around to avoid making Morph's cmdtest suite depend on Baserock.
Hopefully naming it 'test' should discourage potential abusers.
It is unfortunate that the build tests now take a separate code path
compared to real-world usage of Morph. However, this is necessary to
avoid a circular dependency between Morph's test suite and the
build-essential stratum in Baserock.
We do whole-build testing of Baserock, too, so the 'staging' code path
is still tested outside of Morph. However, testing a staging area
requires populating it with at minimum a working shell, and this is a
bit too complex to go in Morph's test suite.
|
|
|
|
|
|
| |
In the future we will allow this to be modified to provide a
cross-bootstrap mode, but for now we use the same target as
the host compiler.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reorganise the build_artifact() and build_artifacts() functions to
allow more complex work when setting up chunk builds in
build_artifact().
The staging area now holds the BuildEnvironment object (the
environment variables that should be set during build). This makes sense
because all build commands should be run inside the staging area and
therefore through the StagingArea object.
The BuildEnvironment object is now considered immutable after it is
created. The environment is used in cache key computation when
computing what artifacts are required; if it changes after that point
we risk either computing different artifact keys for the same artifact
or missing data in the cache key that should be included in the hash.
Better to force changes into a separate 'extra_env' variable.
|
|
|
|
|
|
|
|
| |
This changes the cache key generation so that it will ignore unimportant
fields of the morphology, e.g. description, build-depends, chunks.
description is unimportant because it does not affect building, and
build-depends/chunks are unimportant because they are already considered
|
|
|
|
|
| |
This avoids rebuilding things when commits are made that do not
change the actual source code or the morphologies.
|
|
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
|
|
|
|
|
|
|
|
|
| |
This is required for systemd's journald to start.
This is probably a bad dependency in systemd, trying to start the
journal before it has mounted everything properly.
This required a compat change, it is a string to make it more
noticeable that it's a temporary version.
|
|
|
|
|
|
|
|
|
| |
This will prevent systems that don't have the right mount options
being cache hits.
Having fstab be in some System configuration morphology, which is also
included in the cache key would prevent this being needed if the fstab
format changes again.
|
|
|
|
|
|
| |
Since strata aren't tarballs any more, it's simpler to make
tarball strata cache misses than have the system build code
behave differently for tarballs and json chunk lists.
|
|
|
|
|
|
| |
CacheKeyComputer already re-invents the wheel for hashing a dict,
so re-use that logic rather than re-implement it as creating a
huge string.
|
|
|
|
|
|
|
| |
Previously we were stringifying complex values (lists, dicts) by
using whatever Python produces, but different runs and versions of
Python may convert them differently, particularly dicts. We now do
it manually.
|
|
|
|
|
|
|
|
|
|
| |
commit
This avoids a problem where we make a change to one system morphology in
the "morphs" repository, and then we have to rebuild all system and
stratum artifacts, because their commit sha1 also changed. With this
change, we don't care about the commit sha1 for systems and strata,
just the contents of the morphologies.
|
|
|
|
| |
This makes the dicts smaller, so should be faster and more manageable
|
|
|
|
| |
get_cache_id is called recursively, so it would help to log that too
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
With this commit, the ArtifactResolver no longer computes the cache keys
when creating Artifact objects. This will have to happen as a
post-resolving step (e.g. prior to building or checking whether a local
or remote artifact cache has any of the resolved artifacts).
The CacheKeyComputer now takes an Artifact object and computes the cache
keys using its dependencies.
BuildGraph is no longer needed for the CacheKeyComputer unit tests.
|
|
|
|
|
| |
Also amend tests to operate on a dependency chain starting from
a system morph
|
|
|
|
| |
Duplicate data less, so it should be more manageable
|
|
|
|
|
|
| |
CacheKeyComputer needs to know some stuff that is needed elsewhere
as well. Rather than duplicate the storage, have a BuildEnvironment
class to handle that.
|
|
|