| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
| |
This class would be pretty useful in other cases where a system image
needs to be mounted and inspected.
Also updates the Trebuchet plugin to use this class.
|
| | |
|
| | |
|
| |
|
|
|
| |
This can go away when we have made a release with yaml in it, and its
staging filler.
|
| |\ |
|
| | |
| |
| |
| |
| | |
Tests are currently broken, one because invalid JSON
can be valid YAML, and coverage is incomplete.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When morph is built, it writes various version information from git
into morphlib.
When morphlib is loaded it attempts to read this version information.
if it cannot be found then it checks whether morphlib is being run from
inside a git checkout, if it is then it reads the information that way.
If it isn't in a git checkout then it raises an exception as builds made
in such a fashion are not reproducible.
The git version information retained is:
1. The output of git describe
This is a relatively human-friendly way of knowing a version and
gives a reasonably short output string.
This will end with `-unreproducible` if there were uncommitted changes.
2. The commit sha1, so the exact part of Morph's history can be found
3. The tree sha1, so if the branch has been rebased rather than
merged such that the commit is lost, you may still be able to find
it, though it requires a git-wizard to check it out
4. The branch of morph, so that it's easier to see if the
Further possible changes to increase reproducibility include:
1. Not allowing `python setup.py build` if there are uncommitted changes
2. Failing to run with uncommitted changes (recommended against since it
will just annoy developers who are making changes to morph, and make
them commit just to shut it up, then destroy the history later)
Requiring an extra flag to build in this case may work better.
3. Reading the uncommitted changes into a tree object and including
that would allow it to be recovered if the tree was later committed.
4. Checking whether the commit has been pushed upstream as well.
Too annoying to work.
|
| |
|
|
| |
Reviewed-By: Daniel Silverstone (on irc)
|
| |
|
|
|
|
|
|
| |
morphlib.app should be for application bringup and providing
command line options used by the library code.
Any external plugins that refer to morphlib.app.BuildCommand
will need fixing.
|
| |
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| |
|
|
|
|
|
| |
The artifact cache doesn't need to know the whole of an artifact object
to be able to retrieve something from the cache, just the basename.
This can be generated from the contents, or just saved itself.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
This way, cliapp will automatically print an error message, rather
than a stack trace.
|
| |
|
|
|
|
|
|
| |
Traversing morphologies is a little simpler now, the callback
for visit is enough to create a source pool
This still needs repository caches to be able to list_files()
before deducing a morphology will work.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
With this commit, the ArtifactResolver no longer computes the cache keys
when creating Artifact objects. This will have to happen as a
post-resolving step (e.g. prior to building or checking whether a local
or remote artifact cache has any of the resolved artifacts).
The CacheKeyComputer now takes an Artifact object and computes the cache
keys using its dependencies.
BuildGraph is no longer needed for the CacheKeyComputer unit tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new setting called 'cache-server' to morph. It is None
by default and should be set to the HTTP URL of a morph cache server
to be used.
The RemoteRepoCache object provides two methods: resolve_ref() and
cat_file(), both of which wrap the communication with the cache
server and return a SHA1 string and file contents, respectively.
The _create_source_pool() method now takes a local and an optional
remote repo cache and tries to do whatever is best to resolve
refs and load morphologies for the Source objects it creates.
|
| |
|
|
|
|
|
| |
This class implements the build order part of the old BuildGraph.
It takes a list of artifacts with dependencies, creates independent
build groups for these artifacts and provides an iterable interface
to traverse these groups and their artifacts.
|
| | |
|
| |
|
|
|
|
|
| |
This class takes a CacheKeyComputer and a SourcePool, analyses the
sources and their dependencies and creates a list of artifacts
(represented by Artifact objects) that would be created when
building sources in the pool.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
BuildEnvironment should contain all the information about whatever
settings affect the build. A subset of these will affect the cache
key, but CacheKeyComputer filters them itself.
BuildEnvironment has consumed the purpose of Morph.clean_env() because
the build environment is a good place to store this information.
The environment variables are decided based on the current settings.
The current environment can be accessed by the env attribute.
The tests are a little invasive because it reads environment variables
and most of the work is done in the constructor, but they should test
everything useful.
|
| | |
|
| |
|
|
|
|
| |
This class takes a SourcePool and resolves the dependencies and
dependents of all sources in this pool. It also performs basic (possibly
incorrect) cycle detection.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
An Artifact represents a thing that morph has built. An example would
be eglibc-runtime which morph may have built from the eglibc chunk
morphology. Another example would be a ready-to-use system image.
The LocalArtifactCache allows to store build artifacts in a local
directory. Users of this class can ask it whether it has a certain
artifact. They can also optain an I/O handle to read the artifact data
from.
In addition to just abstracting the way artifacts are stored,
LocalArtifactCache also allows to store and retrieve metadata for
(a) artifacts and (b) sources (the latter requires a cache key to
be provided to the LocalArtifactCache).
|
| |
|
|
|
| |
This class takes a source pool and computes a valid build order for
the sources in this pool, if possible.
|
| | |
|
| |
|
|
|
|
|
|
| |
This will eventually replace the Blob class we are currently using.
It is somewhat simpler, because the envisioned usage of Source will
be different from Blob: we'll be rewriting the build ordering class,
and the builder classes, in ways that simplify things, and thus a
simpler Source class will hopefully suffice.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
Jannis and I discussed a refactoring of most of morph's internals
to reduce coupling and increase cohesion. This is one of the results:
we'll want a couple of classes to manage locally cached git repositories.
This commit adds the LocalRepoCache class to manage a the git cache
directory. Later on, we'll add the CachedRepo class to represent
individual repositories.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
We don't do releases, so having the version number in the source is
misleading. If and when we start doing numbered releases, it's easy
to add back. However, YAGNI and all that.
|
| |
|
|
|
|
|
|
| |
This we a proof-of-concept, and we want to explore various options of
how to do this, so let's not keep the code in morph. The code also has
no tests, so it is already likely to have bit-rotted.
If we decide we want to, we can always resurrect it from git history.
|
| |
|
|
|
| |
This avoids problems with files with the right names but partial
content, if morph is killed in the middle of writing the file.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit introduces four new classes:
BuildController:
* takes an app instance and a tempdir
* allows to add BuildWorker objects
* provides a build() method that takes a set of blobs and
a build order that is then built by assigning work to the
build workers as needed
* the build() method takes care of polling the workers for
their state, moving them between busy and idle states
reliably, collect and print their output in a non-confusing
order, and makes sure to wait for all workers to finish
before processing the next group in the build order.
* at this point, when waiting for one or more workers to
become idle to assign them another blob to build, the
controller always picks the worker that has been idling
for the longest period of time. this can be changed later.
BuildWorker:
* base class for all worker classes
* takes a name and an app instance
* has a idle_since datetime property
* provides a build() method that takes a Blob object and builds
it in whatever way the subclasses implement it
* provides a check_complete(timeout) method that checks whether
the worker has finished building the blob yet or not
LocalBuildWorker:
* worker class for local builds that don't go through SSH
* it uses morphlib.execute.Execute to run morph in a child process
in build()
* at the moment, this class executes "./morph" instead of "morph"
as it assumes the user to run morph from its source tree. obviously,
this will have to be fixed later.
RemoteBuildWorker:
* doesn't implement anything yet, will be used for distributing
work to other machines running morph via SSH
Notes:
* At the moment, there is a degree of undesired redundancy when
building a stratum in a worker, as this will cause the worker to
rebuild all its dependencies. This will have to be fixed as it is
avoidable and wastes a lot of time and processing power.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
The "show-dependencies" command takes a series of build tuples and dumps
the resulting dependency graph (including strata and chunks at the
moment) to the standard output. It also dumps the resulting build order
which is a list of groups. These groups indicate which chunks and strata
can be built in parallel and are not dependent on each other.
|
| | |
|
| | |
|
| |
|
|
|
|
| |
The Stopwatch class does not have unit tests yet and the build times
stored in the cache for system images may be a little too fine-grained
at the moment.
|
| | |
|
| | |
|
| | |
|
