| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
metadata_version is for when the format of the /baserock metadata files
changes.
This means it would make sense for it to either live with the code for
generating the metadata, or the cache key code so it lives with the rest
of the compatibility values.
Since the code for generating the metadata isn't in a nice module
anywhere, I've put it in the cachekeycomputer module.
|
| |
|
|
|
|
|
|
|
| |
There's no need to log every time we look something up in a dict. This just
makes log files huge.
The CacheKeyComputer.compute_key() function still logs the first time it
calculates a cache key for an Artifact instance. This message now includes
the hex string that is used to identify the artifact.
|
| |
|
|
|
|
|
|
|
| |
This reverts commit 4124c50b8dc3dfb0ffb933153d0fe6385edf389c.
This commit is no longer needed now that morph2 is gone.
Conflicts:
morphlib/cachekeycomputer.py
|
| |
|
|
|
|
| |
Rather than having a `get_commands` method to obtain missing commands
from the build system when they are needed, get the commands when
loading a morphology.
|
| |\
| |
| |
| |
| |
| |
| |
| | |
This fixes a bug where distbuild was calculating a different cache
local build. It will no longer be needed once morph2 is removed.
Reviewed-by: Sam Thursfield
Reviewed-by: Richard Maw
|
| |/
|
|
|
|
|
|
| |
morph2.Morphology adds a number of fields beginning with _orig_, which
contain default values. There is no reason for these values to be in
the cache key, and this fixes a bug where distbuild created systems with
a different cache key than local builds did when the configuration
extensions list was empty.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Previously the contents of the morphology would be included by virtue of
the fact that it came from the source repository, so would be included
in the "tree" field.
Now that chunk morphologies can come from the definitions repository, it
is not always included in the "tree" field, so the logical contents of
the morphology need to be included in the cache key computation.
Build commands are included after looking them up in the build-system,
so that in future, we don't need to change the chunk morphology
compatibility version when we change how build-systems work.
Since we may be moving the morphologies about in the definitions
repository, it would suck if we had to do a full rebuild after we move
things, so I dropped the filename from the cache key.
This also tweaks the system and stratum cache keys to include the
contents directly, rather than hashed in the "morphology-sha1" field.
|
| |
|
|
|
|
|
|
|
| |
We were previously memoising the computation of the dictionaries
but this patch adds support for memoising the computation of the
cache key itself (the SHA string). This massively improves cache
key computation performance.
Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |
|
|
|
|
|
|
| |
This messes up the baserock-system-config-sync tool. Systemd does not
require /etc/fstab to exist in any case.
I have bumped the 'system-compatibility-version' field in this commit
to trigger rebuilding all system artifacts.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
If two systems with the same name (e.g. different repo/ref) depend on the
same strata, then it will collide with systems which depend on different
artifacts from that stratum, but the same number of artifacts.
For example, if you checkout an existing branch and change the artifacts
used by one of its strata, then your local changes won't be built.
This is because the 'kids' field lists artifacts it depends on by their
cache-key, which is now no longer sufficient to uniquely identify
artifacts. The same number of artifacts issue is from it listing cache
keys multiple times.
The fix for this is to include the artifact name, so the 'kids' field is
now a list of dicts, with artifact name and cache key.
This is a dict rather than a tuple so that the generated /baserock
metadata is more readable.
|
| |
|
|
|
|
| |
For chunks the products field doesn't need to be hashed, since the
split-rules field is used instead, and includes the default rule set,
and for strata and systems it is handled by hashing the dependencies.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
Chunk artifacts need the [(artifact_name, [regular_expression])] so that
if the default split rules change, or the blending rule changes, then
an extra version field doesn't need to be added to the cache key computer.
This is for future plans to allow the split rules to be configurable
and allow us to more easily change them.
System and Stratum artifact computations don't need this, since those
splitting rules are already expressed in the dependencies information.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Define a specific set of 4 architectures that Morph supports, and only
expose that value to morphologies.
Since GNU triplets are very common we also expose a GNU triplet. Other
morphologies should work out their configuration based on MORPH_ARCH.
This commit also removes the morphlib.util.arch() function, which
detected the machine Morph is running on via 'uname -m'. Morph's
architecture names do not necessarily map to the output of 'uname -m'
so we should not rely on it anywhere.
|
| |
|
|
|
|
|
|
|
| |
Morph no longer supports setting the prefix using the --prefix
argument / setting. This was only used in tests and during bootstrap.
If a chunk build-depends on a chunk within a stratum which has
a custom prefix, that prefix is appended to the PATH in the build
environment.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Allowed values:
staging: build with a staging chroot (default)
test: build with the host's tools
bootstrap: build with the host's tools, and do not include this
chunk in the final stratum artifact
In the past, 'normal mode' has been used to describe building a chunk
with the host's tools. We don't want that mode to ever be used,
because it is a huge hole in reproducability, but we need to keep it
around to avoid making Morph's cmdtest suite depend on Baserock.
Hopefully naming it 'test' should discourage potential abusers.
It is unfortunate that the build tests now take a separate code path
compared to real-world usage of Morph. However, this is necessary to
avoid a circular dependency between Morph's test suite and the
build-essential stratum in Baserock.
We do whole-build testing of Baserock, too, so the 'staging' code path
is still tested outside of Morph. However, testing a staging area
requires populating it with at minimum a working shell, and this is a
bit too complex to go in Morph's test suite.
|
| |
|
|
|
|
| |
In the future we will allow this to be modified to provide a
cross-bootstrap mode, but for now we use the same target as
the host compiler.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Reorganise the build_artifact() and build_artifacts() functions to
allow more complex work when setting up chunk builds in
build_artifact().
The staging area now holds the BuildEnvironment object (the
environment variables that should be set during build). This makes sense
because all build commands should be run inside the staging area and
therefore through the StagingArea object.
The BuildEnvironment object is now considered immutable after it is
created. The environment is used in cache key computation when
computing what artifacts are required; if it changes after that point
we risk either computing different artifact keys for the same artifact
or missing data in the cache key that should be included in the hash.
Better to force changes into a separate 'extra_env' variable.
|
| |
|
|
|
|
|
|
| |
This changes the cache key generation so that it will ignore unimportant
fields of the morphology, e.g. description, build-depends, chunks.
description is unimportant because it does not affect building, and
build-depends/chunks are unimportant because they are already considered
|
| |
|
|
|
| |
This avoids rebuilding things when commits are made that do not
change the actual source code or the morphologies.
|
| |
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| |
|
|
|
|
|
|
|
|
| |
This is required for systemd's journald to start.
This is probably a bad dependency in systemd, trying to start the
journal before it has mounted everything properly.
This required a compat change, it is a string to make it more
noticeable that it's a temporary version.
|
| |
|
|
|
|
|
|
|
| |
This will prevent systems that don't have the right mount options
being cache hits.
Having fstab be in some System configuration morphology, which is also
included in the cache key would prevent this being needed if the fstab
format changes again.
|
| |
|
|
|
|
| |
Since strata aren't tarballs any more, it's simpler to make
tarball strata cache misses than have the system build code
behave differently for tarballs and json chunk lists.
|
| |
|
|
|
|
| |
CacheKeyComputer already re-invents the wheel for hashing a dict,
so re-use that logic rather than re-implement it as creating a
huge string.
|
| |
|
|
|
|
|
| |
Previously we were stringifying complex values (lists, dicts) by
using whatever Python produces, but different runs and versions of
Python may convert them differently, particularly dicts. We now do
it manually.
|
| |
|
|
|
|
|
|
|
|
| |
commit
This avoids a problem where we make a change to one system morphology in
the "morphs" repository, and then we have to rebuild all system and
stratum artifacts, because their commit sha1 also changed. With this
change, we don't care about the commit sha1 for systems and strata,
just the contents of the morphologies.
|
| |
|
|
| |
This makes the dicts smaller, so should be faster and more manageable
|
| |
|
|
| |
get_cache_id is called recursively, so it would help to log that too
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
With this commit, the ArtifactResolver no longer computes the cache keys
when creating Artifact objects. This will have to happen as a
post-resolving step (e.g. prior to building or checking whether a local
or remote artifact cache has any of the resolved artifacts).
The CacheKeyComputer now takes an Artifact object and computes the cache
keys using its dependencies.
BuildGraph is no longer needed for the CacheKeyComputer unit tests.
|
| |
|
|
|
| |
Also amend tests to operate on a dependency chain starting from
a system morph
|
| |
|
|
| |
Duplicate data less, so it should be more manageable
|
| |
|
|
|
|
| |
CacheKeyComputer needs to know some stuff that is needed elsewhere
as well. Rather than duplicate the storage, have a BuildEnvironment
class to handle that.
|
| |
|
