| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
| |
Make morph look for system morphologies in subdirectories of the system
branch root repository checkout, in case the repository contains all its
system morphologies in a specific subdirectory.
|
| | |
|
| |
|
|
|
|
| |
When loading a morphology from the repo cache, don't bother doing
ls-tree to check if the file exists. Instead just try to cat the file
and raise an error if it is empty.
|
| |
|
|
|
|
|
|
|
|
| |
Rather than repeatedly stripping and appending an optional .morph extension
morphology names, instead always use the file path of the morphology
relative to the definitions repository.
Also, make the repo and ref of chunk morphology triplets be some branch
of the definitions repository, rather than the repo/ref given in the
morphology itself (which is the location of the chunk source).
|
| | |
|
| |
|
|
|
| |
To edit a chunk, do `morph edit path/to/chunk.morph` where the path is
relative to the root of the definitions repository checkout.
|
| | |
|
| |
|
|
|
|
| |
Rather than traversing chunk triplets using the source repo and ref, use
the definitions repository and add the correct triplet to the source pool
using the repo and ref from the chunk morphology.
|
| |\
| |
| |
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
There are cases where a state machine handles an event but stays in the
same state. A callback is registered which filters messages further
before possibly taking action. There have been bugs caused by this
pattern being incorrectly implemented (where the callback is expected to
filter the message, but a transition takes place anyway). Hopefully a
consistent naming convention will make the pattern clearer.
|
| | |
| |
| |
| |
| |
| | |
There is always one BuildController object per InitiatorConnection.
By coupling the objects slightly closer we can simplify some transitions
in BuildController.
|
| | |
| |
| |
| |
| | |
This is similar to the issue fixed by commit
c38b77bed86acc8b90f253ce354f3ecf98e475e7.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Each BuildController instance sets itself up to receive all messages from
all workers (via the WorkerConnection instances). In the case of a build
failure, all BuildController objects would transition to 'None' state
(causing them all to be destroyed) on any WorkerBuildFailed message.
This meant that if any one build failed on a distbuild network:
- the user whose build actually failed would receive the error
messages correctly
- any concurrent users would see no further build messages from the
controller, making it look like their builds had hung.
Ctrl+C from the 'hung' users would still be correctly handled by the
controller, as their InitiatorConnection instance would still be alive.
|
| | | |
|
| | |
| |
| |
| | |
Makes it easier to see what they mean at a glance.
|
| |/ |
|
| |\
| |
| |
| |
| | |
Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
| |
| |
| | |
This makes tracking multiple builds through one log file a bit easier.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
New DistbuildSocket class that wraps socket.socket(), providing a
descriptive repr() handler showing where the socket is connected, and
providing a couple of helper methods for fetching local and remote
endpoint names.
This commit also adds a descriptive repr() handler to a few other
objects (mostly giving socket connection details).
|
| | |
| |
| |
| |
| | |
We still log all messages sent to workers, which include the output of the serialise-artifact
code in full. There's no need for these status messages.
|
| | | |
|
| |\ \
| |/
|/|
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |/ |
|
| |\
| |
| |
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The default Python __repr__() is useless for both developers and users,
so avoid using it in error messages.
The following message:
ERROR: Push to remote <morphlib.gitdir.Remote object at 0x905096c>,
push url None with refspecs (<morphlib.gitdir.RefSpec object at 0xf72fec2c>,)
failed with exit code 128
Becomes:
ERROR: Push to remote "origin", push url None failed with exit code 128
|
| |\ |
|
| |/ |
|
| |\
| |
| |
| |
| |
| |
| |
| | |
Signed-off by: Adam Coldrick
Reviewed by:
Daniel Silverstone <daniel.silverstone@codethink.co.uk>
Richard Maw <richard.maw@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Since git's parsing of .gitattributes can't deal with spaces, or
even with quotes, replace any whitespace with [[:space:]]. Also,
make sure to escape any glob metacharacters to avoid accidental
pattern matches.
Note:
This approach could cause a file "foo<tab>bar" to be treated
as a binary if a file "foo bar" has already been added with
morph add-binary. However this is as good of a workaround as
I can find.
Signed-off by: Adam Coldrick
|
| | | |
|
| |\ \
| | |
| | |
| | | |
'origin/baserock/markdoffman/s10617/add-yaml-help-option'
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Begin adding help documentation for configuration
and write extensions starting with nfsboot.write,
rawdisk.write and tar.write.
|
| | | |
| | |
| | |
| | |
| | |
| | | |
Add a command 'help-extensions' to list all extensions.
Add the ability to find help on an extension by calling
'morph help [extension name]'.
|
| |/ /
| |
| |
| |
| |
| |
| |
| |
| | |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
| | | |
|
| |\ \
| |/
|/|
| | |
Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Explicitly test that we get the same result twice when computing
ids and keys for the same artifact.
This has the effect of verifying that the memoisation code is
correctly operating.
Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |/
|
|
|
|
|
|
|
| |
We were previously memoising the computation of the dictionaries
but this patch adds support for memoising the computation of the
cache key itself (the SHA string). This massively improves cache
key computation performance.
Signed-Off-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| | | |
|
| | | |
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| | |
tarfile's open needs the file-like object
to have a tell() method, objects returned
from sockets don't have this method.
So instead we fetch the artifact from the
remote and cache it locally
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| | |
The constructor for BuildCommand sets up the caches,
we want the caches to be set up for distbuild too
so that we can deploy a system (system artifact will be
fetched from the artifact cache)
|
| | |
| |
| |
| |
| | |
We want to be able to transfer all
source artifacts in a single transaction
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Whenever the controller finds a source artifact
it wants to build, it changes its state to BUILDING.
We build all a chunk's source artifacts in one go.
So for any chunk artifact, we change the state of
all chunk artifacts that are built from the
same source to BUILDING
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Serialisation was simple when we only had 1 artifact per source.
However, to allow smaller systems, we need artifact splitting to produce
multiple artifacts per chunk source.
So now the new serialisation format has a separate list of artifacts
and sources, rather than the Source being generated from the artifact's
serialisation.
Python's id() function is used to encode the references between the
various Sources and Artifacts, these are replaced with a reference to
the new object after deserialisation.
Previously the cache-key was used, but this is no longer sufficient to
uniquely identify an Artifact.
The resultant build graph after deserialisation is a little different
to what went in: Strata end up with a different Source per Artifact,
so it _is_ a 1 to 1 mapping, as opposed to Chunks, where it's many to 1.
We serialise strata and chunks differently because stratum artifacts
from the same source can have different dependencies, for example
core-devel can have different dependencies to core-runtime.
Without intervention we would serialise core-devel and core-devel's
dependencies without including core-runtime's dependencies.
To solve this we've decided to encode stratum artifacts completely
indepedently: each stratum artifact has its own source. This is safe
because stratum artifacts can be constructed independently,
as opposed to Chunks where all the Artifacts for a Source
are produced together.
This is a little hacky in its current form, but it simplifies matters
later in distbuild with regards to how it handles expressing that
every Artifact that shares a Source is built together.
Arguably, this should be the output of producing the build graph
anyway, since it more helpfully represents which Artifacts are built
together than checking the morphology kind all the time, but more
assumptions need checking in morph before it's safe to make this
change across the whole of the morph codebase.
|
| | |
| |
| |
| | |
Worker name is not sent in message
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
A semantic error in the BuildBranch class meant that it was not possible
to push temporary branches.
This escaped testing since BuildBranch interacts too tightly with
other components to be easily unit-tested, so testing was deferred to
a yarn test.
However, coverage isn't measured in yarn tests, so this code path was
forgotten.
|
