| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
Its much harder to look at the contents of the artifact cache now that
they aren't stored by chunk name (just cache key and artifact suffix).
All of these were failing because they were trying to extract chunk
tarballs or locate and inspect artifacts, so just make sure the builds
themselves don't fail. The build-system tests are a lot of work to make
into yarns, as we'll need to build systems with actual tools in as far
as I can tell.
Also disable the cross-bootstrap test for now.
|
| |
|
|
|
|
|
| |
The cross-bootstrap plugin is currently full of hacks, so it makes sense
to rewrite it properly rather than extending these hacks to work with
OSTree. I don't have time to do this right now, so disable the
cross-bootstrap yarn.
|
| |
|
|
|
|
|
|
|
| |
OSTree can easily pull over HTTP. All that is necessary is to expose the
repo directory. This commit adds a simple HTTP server to do this in the
test suite. Actual implementations should use something better, like
lighttpd.
Also add some logging of the cache servers in this yarn to help debug.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Now that we have an OSTree artifact cache, the deploy plugin needs to use
that to get the system to be deployed. Due to the changes in how we store
systems, we need to get the contents of each stratum then put the system
delta on top of that.
This is still much quicker than unpacking stuff from tarballs.
|
| |
|
|
|
|
|
| |
The API of the OSTree artifact cache is slightly different to that of the
old tarball cache, so adjust things accordingly. Also, only store the
files changed at system-construction-time rather than everything in
system artifacts.
|
| | |
|
| |
|
|
|
| |
This commit updates RemoteArtifactCache to enable it to interact with
a remote OSTree artifact cache.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
Change create_chunk to put the contents of an artifact into a directory
rather than storing them in a tarball, as we want to store chunks as
directory trees in OSTree rather than tarballs now.
|
| |
|
|
|
|
| |
This avoids needing to pass the cache to the staging area since
lac.get returns a path to a directory tree rather than a file
handle now, so we also don't need to do any unpacking.
|
| |
|
|
|
|
|
|
|
| |
We can't store devices nodes in OSTree, so we can't create them at
artifact build time and store them in the chunk artifacts anymore.
Instead, we should create them in the staging area when installing
an artifact with a source which has a morphology which defines
them into the staging area.
|
| |
|
|
|
|
| |
Overlayfs is new in version 3.18 of the kernel, so add support for a
different implementation of a union/overlay filesystem in order to
allow morph to work on older kernels.
|
| |
|
|
|
|
|
|
|
|
|
|
| |
When deploying, configuration extensions are run against the unpacked
tarball of a system created by a build. If we are to use OSTree to
store systems (rather than tarballs) this will not be possible as the
result of `ostree checkout` would be read-only.
To solve this, we use overlayfs to mount the unpacked tarball
underneath a temporary directory somewhere, and run the configuration
extensions on that mount point. This means that the changes are
made in the temporary directory rather than directly on the tarball.
|
| |
|
|
|
| |
This will allow us to cache systems as a list of chunks and a small
filesystem delta, rather than a massive tarball.
|
| |
|
|
|
| |
In order to mount using overlayfs, fsutils.mount needs to take a
string of options to pass to the mount command.
|
| |\
| |
| |
| |
| |
| | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
The logic to handle a worker disconnecting was broken. The
WorkerConnection object would remove itself from the main loop as soon
as the worker disconnected. But it would not get removed from the list
of available workers that the WorkerBuildQueue maintains. So the
controller would continue sending messages to this dead connection, and
the builds it sent would hang forever for a response.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Javier Jardón <javier.jardon@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| | |/
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This saves a duplicate `git remote update origin` that was being run as
part of each chunk build. For any repos that have submodules, it also
avoids updating repos if the SHA1 we need to build is already present
locally.
As well as speeding up builds slightly, this means Morph can now build
without being connected to a network, as long as the local Git cache
all of the necessary repos and commits in the build, without needing the
'--no-git-update' option.
The code is also now in a more logical place than before.
|
| | | |
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk>
Reviewed-By: Zara Zaimeche <zara.zaimeche@codethink.co.uk>
|
| |/ /
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since the version of btrfs-progs in the Baserock reference system
definitions was updated to v3.18.2, Morph has produced unbootable
x86 systems. This is down to lack of support for new Btrfs features
in SYSLINUX.
This patch disables the new features so that deployed systems will boot.
Although I generally don't want to have compatibility code in Morph,
this patch keeps support for the old mkfs.btrfs (which doesn't support
the commandline options that we need to pass to new mkfs.btrfs). This
will hopefully save people from suffering 'I need to use new Morph to
upgrade my devel system, but new Morph doesn't run on my devel system'.
|
| |\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Paul Martin <paul.martin@codethink.co.uk>
Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Checking that a given ref exists using `git rev-parse --verify
1234^{commit}` is a reasonably quick operation. As a rough guide, 1000
invocations took 1.6 seconds on my PC.
The code is too fragile and hard to reason about if we assume that one
function has been called before another so the repo will already be up
to date.
This should fix any spurious InvalidRefError exceptions that the
build graph speedups branch has introduced.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | | |
Also, move the repo.update() call into the 'fetch from tarball' code
path in the localrepocache module -- there's no need to update straight
after doing a `git clone`, but we do need to do one if we got the repo
from a `git archive` tarball that may be out of date.
|
| | | |
| | |
| | |
| | |
| | | |
It turns out that always looking for the chunk morph in the definitions
repo allows us to make the code a little less nasty. Bonus!
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Code in sourceresolver.py assumed that resolve_ref() would be called for
the chunk repo before get_morphology() was called, so the repo would
always be up to date. This wasn't actually true.
If your local repo cache was out of date, you might see the following
sort of error:
InvalidRefError: Git directory
/src/cache/gits/git___git_baserock_org_delta_usbutils has no commit at
ref c37f146eb2c6642c600f1b025a6d56996b0697ff^{tree}.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This fixes the 'Morph ignores the chunk morph file I added to
definitions' regression introduced in the recent build-graph speedups
branch.
The new 'detected-chunk-buildsystems' cache associates a (chunk repo,
chunk ref) pair with the auto-detected build system. When calculating
the build graph, if the is an auto-detected build system known already,
that will be used instead of looking for a chunk morphology.
There's a flaw here: if the user added or changed the chunk morphology
in the definitions repo, the *chunk* ref won't necessarily have changed
-- so Morph will ignore the user's changes, if it had already cached an
autodetected buildsystem for that repo/ref pair.
This doesn't cost much in terms of speed because we're just reading a
file from disk. We can still avoid looking in the chunk repo for a chunk
morph, because the chunk ref would be different if something had changed
in the chunk repo.
|
| | | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
If you had a repo that was not in your local or remote cache in your
definitions, Morph might raise LsTreeError and abort.
The code was assuming the repo would already be cached in the local
cache by the time it got to detecting build systems, but that's not
always true -- if the (reponame, ref) pair was already in the
'resolved_trees' cache but the repo had been deleted from the local
Git cache, to name one possibility.
Also, the remoterepocache.ls_tree operation can fail if the repo is
not hosted in the Trove, so we shouldn't abort the program in that case.
Finally, this patch removes an unused variable.
|
| |\ \ \
| |/ /
|/| |
| | |
| | | |
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
|
| |/ / |
|
| |\ \
| | |
| | |
| | |
| | |
| | |
| | | |
'lauren/baserock/lauren/distbuild-invalid-input-crash'
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
|
| | | | |
|
| | | | |
|
| |\ \ \
| | | |
| | | |
| | | |
| | | | |
Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This only adds tests for the bits which were moved from
morphologyfactory into sourceresolver, namely detection
of build systems and the '_get_morphology()' function.
These are just the morphologyfactory tests reworked
slightly to work properly with the modified API.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
Most morphologies involved in a build are in the definitions repo these
days. Currently we read each of them using `git cat-file`, which is
slow. It's quicker to check out all the files in one go to a temporary
directory and then read them from there.
With the current workflow users often have definitions.git checked out
on disk. It seems strange to not just read the files from there. There
are two reasons why I don't want to do that yet:
- there are commands which don't run inside a system branch, which
would be broken if we expected to always be in a system branch
- there may be local changes in the checked-out repo, and it takes
around 5 seconds on each build to check if there aren't any local
changes. It actually seems faster to just check out a known clean
version from the cache.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This is nice because it's fast. We don't have to copy all the Git
history along with it like we do with a clone. And it doesn't touch
any files in the cached repo.
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This will speed up builds of chunks which don't have a chunk
morph. It won't have much (if any) effect on the speed of the
first build, but subsequent builds will be much faster as we
won't have to query the git cache.
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
This uses the PyLRU module, from:
<https://pypi.python.org/pypi/pylru/1.0.6>.
Python 3.2 and newer provide a built-in LRU cache, but this is
specifically for in-memory use. See <http://bugs.python.org/issue17528>.
Git commits are immutable, so caching information about their contents
is fairly easy and trouble-free. There's no danger of the cache becoming
stale.
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | | |
There's no need for this stuff to be in a separate class. This allows
integrating it with the caching in the SourceResolver class.
|
| | | | | |
|
| | | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
We will use MorphologyLoader.load_from_file() to load morphologies
from a temporary file, so the output will be unpredictable. That
will break this test, so we may as well convert it into a yarn
to fix it.
|
| |/ / /
| | |
| | |
| | |
| | |
| | | |
This commit fixes an issue where running morph's yarn tests would
leave behind a number of scripts/git-daemon-wrap processes due to
some tests not stopping their git server properly.
|
