| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
| |
Begin adding help documentation for configuration
and write extensions starting with nfsboot.write,
rawdisk.write and tar.write.
|
|
|
|
|
|
| |
Add a command 'help-extensions' to list all extensions.
Add the ability to find help on an extension by calling
'morph help [extension name]'.
|
|
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
| |
|
|\
| |
| |
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Explicitly test that we get the same result twice when computing
ids and keys for the same artifact.
This has the effect of verifying that the memoisation code is
correctly operating.
Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|/
|
|
|
|
|
|
|
| |
We were previously memoising the computation of the dictionaries
but this patch adds support for memoising the computation of the
cache key itself (the SHA string). This massively improves cache
key computation performance.
Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
|\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| |
| |
| |
| |
| | |
tarfile's open needs the file-like object
to have a tell() method, objects returned
from sockets don't have this method.
So instead we fetch the artifact from the
remote and cache it locally
|
| | |
|
| |
| |
| |
| |
| |
| |
| | |
The constructor for BuildCommand sets up the caches,
we want the caches to be set up for distbuild too
so that we can deploy a system (system artifact will be
fetched from the artifact cache)
|
| |
| |
| |
| |
| | |
We want to be able to transfer all
source artifacts in a single transaction
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Whenever the controller finds a source artifact
it wants to build, it changes its state to BUILDING.
We build all a chunk's source artifacts in one go.
So for any chunk artifact, we change the state of
all chunk artifacts that are built from the
same source to BUILDING
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Serialisation was simple when we only had 1 artifact per source.
However, to allow smaller systems, we need artifact splitting to produce
multiple artifacts per chunk source.
So now the new serialisation format has a separate list of artifacts
and sources, rather than the Source being generated from the artifact's
serialisation.
Python's id() function is used to encode the references between the
various Sources and Artifacts, these are replaced with a reference to
the new object after deserialisation.
Previously the cache-key was used, but this is no longer sufficient to
uniquely identify an Artifact.
The resultant build graph after deserialisation is a little different
to what went in: Strata end up with a different Source per Artifact,
so it _is_ a 1 to 1 mapping, as opposed to Chunks, where it's many to 1.
We serialise strata and chunks differently because stratum artifacts
from the same source can have different dependencies, for example
core-devel can have different dependencies to core-runtime.
Without intervention we would serialise core-devel and core-devel's
dependencies without including core-runtime's dependencies.
To solve this we've decided to encode stratum artifacts completely
indepedently: each stratum artifact has its own source. This is safe
because stratum artifacts can be constructed independently,
as opposed to Chunks where all the Artifacts for a Source
are produced together.
This is a little hacky in its current form, but it simplifies matters
later in distbuild with regards to how it handles expressing that
every Artifact that shares a Source is built together.
Arguably, this should be the output of producing the build graph
anyway, since it more helpfully represents which Artifacts are built
together than checking the morphology kind all the time, but more
assumptions need checking in morph before it's safe to make this
change across the whole of the morph codebase.
|
| |
| |
| |
| | |
Worker name is not sent in message
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
A semantic error in the BuildBranch class meant that it was not possible
to push temporary branches.
This escaped testing since BuildBranch interacts too tightly with
other components to be easily unit-tested, so testing was deferred to
a yarn test.
However, coverage isn't measured in yarn tests, so this code path was
forgotten.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
| |
| |
| |
| | |
This function causes a UnicodeDecodeError for some repositories
when building. Use os.path.isfile() when looking for .gitfat instead.
|
|\ \
| | |
| | |
| | |
| | | |
Reviewed-By: Daniel Silverstone (on IRC)
Reviewed-By: Richard Maw (on IRC)
|
| | | |
|
|/ /
| |
| |
| |
| | |
We were building it from a variable in some places and hardcoding it in
others; now we build it from a variable everywhere.
|
|\ \
| | |
| | |
| | |
| | |
| | | |
Reviewed by:
Richard Maw
Lars Wirzenius
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Move some code to the '.check' extension to verify that the deployment
can happen *before* spending 5 minutes unpacking and configuring the
rootfs.
This is not a perfect solution yet because when multiple systems are
being deployed in a cluster, we do not check all systems and then
deploy them all. Instead, we check one, then deploy it, then check the
second, etc.
|
| | | |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
There is no longer a default /etc/fstab in the Baserock fhs-dirs chunk,
and the nfsboot.write extension does not use the default Btrfs system
layout so no entry is added for / to /etc/fstab at deploy-time.
We cannot have / in /etc/fstab for nfsboot deployments because it
causes systemd to remount / during bootup, which breaks everything.
|
|\ \ \
| | | |
| | | |
| | | | |
Reviewed-by: Richard Maw
|
| | | | |
|
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | |
| | | | |
The scripts will be created into the folder:
/baserock/sytem-integration/
The system integration commands have the following
syntax:
name: linux
kind: chunk
...
install-commands:
...
system-integration:
linux-libs:
00-depmod:
- depmod -a
70-more-integration:
- touch /baserock/FILE
- |
for FOLDER in $(ls /)
do
echo "$FOLDER"
done
linux-misc:
70-more-integration:
- echo "Hello world"
In this concrete example, the following files will be created:
$DESTDIR/baserock/system-integration/
00-depmod-linux-libs-0000:
#!/bin/sh
set -xev
depmod -a
70-more-integration-linux-libs-0000
#!/bin/sh
set -xev
touch /baserock/FILE
70-more-integration-linux-libs-0001
#!/bin/sh
set -xev
for FOLDER in $(ls /)
do
echo "$FOLDER"
done
70-more-integration-linux-misc-0000
#!/bin/sh
set -xev
echo "Hello world"
|
| | | | |
|
| |/ / |
|
|\ \ \
| |/ /
|/| |
| | |
| | |
| | |
| | |
| | |
| | | |
Author: Adam Coldrick <adam.coldrick@codethink.co.uk>
Reviewed by:
* Richard Maw <richard.maw@codethink.co.uk>
* Lars Wirzenius <lars.wirzenius@codethink.co.uk>
* Richard Ipsum <richard.ipsum@codethink.co.uk>
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Add a plugin to implement both `morph push` and `morph pull`.
These commands are wrappers around the corresponding git commands
push and pull, which also implement the functionality of pushing
and pulling large files provided by git-fat.
For example, running `morph pull` will pull any commits from the
remote branch not on your local branch, and then pull any large
files from the separate git-fat/rsync store on the Trove.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
Add a plugin which implements the morph add-binary command. This
command is used to add large files to a git repository. It sets
up the files needed to use git-fat, and then runs `git add` with
git-fat initiated.
|
| | |
| | |
| | |
| | |
| | |
| | | |
When cloning a repository, the files stored using git-fat need to
be pulled. This situation occurs in `morph branch`, `morph edit`,
and `morph checkout`.
|
|/ /
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
These commands are:
* git fat <init|push|pull>
* git pull
They are required for the morph add-binary and push/pull plugins. Also
make sure that GitDirectory is working in the root directory of the
specified git repository, and add some helper functions for handling
paths of files in the working tree.
|
|\ \ |
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This hard-coded the expected artifact key, which is dependent on
architecture.
Since we don't care about the output, so much as it failing, let's just
discard it.
|
| | |
| | |
| | |
| | |
| | | |
This is tested in yarns already, and this test breaks the test suite on
any architecture but x86_64.
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
They hard-code the expected sha1, so break the test suite for anything
but x86_64.
If we decide tag has a place in the new world order, yarns can be made,
but until then, we're better off without these tests.
|
| | |
| | |
| | |
| | | |
Older versions of btrfs fail with just 10M.
|
| | |
| | |
| | |
| | |
| | | |
uname tends to only give us a valid morph architecture on x86_64,
this makes it work on other architectures.
|
|/ /
| |
| |
| |
| |
| |
| |
| |
| |
| | |
./check with no arguments is as-before, similarly ./check --full,
but now you may also specify individual tests to run.
So just the style check is `./check --style`. Everything but style
is `./check --full --no-style`.
I found this convenient when working on the test suite.
|
|\ \
| | |
| | |
| | | |
Reviewed-by: Richard Ipsum and Lars Wirzenius
|
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | |
| | | |
This reverts commit f366960273b026322f7e7cc3c1eb0cd632ebc73e.
These changes break building on x86_64, which is our main development
platform.
Better patches will be forthcoming later.
|