| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| | |
|
| |
|
|
|
|
|
|
| |
This code is an essential part of 'morph build'. It's quite complex and
really shouldn't be mixed in with the base Application class.
Given a dedicated class we can store some state in the object and avoid
functions with seven parameters, too.
|
| |
|
|
|
|
| |
Instead of leaving morph3 with a potentially confusing name, rename it
to `morphology` since it is now the only implementation of the
Morphology class.
|
| |
|
|
| |
This commit removes the now unneeded morph2 and its associated tests.
|
| |
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
| |
|
|
| |
This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This introduces a new artifactsplitrule module, which tries to provide a
nice abstraction over matching a sequence of things to a bunch of
outputs, to be used by both chunks splitting, for separating files out
into chunk artifacts, the stratum splitting, where chunks are
aggregated into stratum artifacts, and systems selecting the right
strata to go into the artifact.
|
| | |
|
| | |
|
| |
|
|
|
|
|
| |
This is an abstraction on top of SystemBranchDirectories, providing the
ability to add uncommitted changes to the temporary build branch, push
temporary build branches and retrieve the correct repository URI and ref
to build the system.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a LocalRefManager, which handles ref updates to local
repositories (i.e. your workspace).
It provides proxy methods for ref updates to a set of repositories.
If an exception occurs in the body of the context manager, the updates
will be rolled back to before the context manager was entered.
The purpose for using a LocalRefManager instead of making the changes to
the repositories directly, is to provide atomic updates to a set of refs
in a set of repositories, where all refs are updated, or none are.
This also adds a RemoteRefManager, which handles pushing branches to
remote repositories.
It provides a proxy push method, which will delete pushed branches, and
re-push deleted branches after the context manager exits.
Its purpose, instead of providing atomic updates to remote repositories,
is to provide temporary branches. This is because it is used to provide
temporary build branches. The difference between atomic update and
temporary push, is that the remote branches are deleted when the context
is left, rather than kept, as LocalRefManager does.
The RemoteRefManager currently cannot provide the same atomicity
guarantees as the LocalRefManager, so if there is a push between the
branch being created and the RemoteRefManager cleaning it up, that change
is lost without RemoteRefManager even knowing it existed.
Git 1.8.5 will add functionality to make this possible.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This represents the state of the index of a GitDirectory.
Methods that use the index are now used via the GitIndex class, rather
than using the default index, as previously used when the methods were
in GitDirectory.
GitIndex may be constructed with an alternative path, which can be used
to manipulate a git checkout without altering a developer's view of the
repository i.e. The working tree and default index.
This is needed for `morph build` and `morph deploy` to handle the build
without commit logic.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This provides access to the /baserock directory as if it were a dict,
abstracting away the details of how to get data out of it.
The abstraction is useful since it is easier to use than accessing
/baserock yourself, and allows the storage format to be changed more
easily.
Keys with / in may be supported in the future. since there have been
discussions about allowing morphologies to be placed in subdirectories.
Adding this support would require creating and removing directory
components when values are set and deleted respectively.
Iterating would require using os.walk instead of glob.iglob, since
python doesn't support ** in globs.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MorphologyFinder is a small wrapper on top of GitDirectory that
allows the inspection of morphologies in the repository.
Its purpose is to isolate the logic for reading morphologies into one
place.
It is used by passing a GitDirectory and optionally a ref to the
MorpholgyFinder constructor, then list_morphologies and read_morphology
may be used.
The ref is passed directly to the GitDirectory, so its semantics for
a ref of None or omitted are used. i.e. It uses the working tree.
Ref resolving is deferred until a morphology is listed or read, so
it will not raise an exception for an invalid ref until then.
|
| |
|
|
|
|
|
| |
Various parts of Morph need to change a set of morphologies at once,
particularly for petrification and unpetrification. This is easiest
done by loading all the morphologies into memory at once, and changing
them there, then saving again.
|
| |
|
|
|
|
|
|
|
| |
The old code is somewhat weird. The new code is meant to be cleaner and more
straightforward to understand and to use. For example, the old code has setting
of defaults in both the Morphology and MorphologyFactory classes. The new code
has a minimally simple Morphology class, and has all the logic to validate and
set defaults in the MorphologyLoader class. Further, the new code makes it
possible to load an invalid morphology, which will be useful later.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
ExtractedTarball is more or less the equivalent to MountableImage for
artifacts that are not mountable images. So in order to inspect root
file system tarballs, ExtractedTarball can be used, for disk images,
MountableImage can be used.
The morphlib.bins.call_in_artifact_directory() method combines these
two classes and provides a way to extract/mount an artifact and call
a callback with the temporary directory / mount point as its first
argument. Using this, a plugin that runs a command relative to an
artifact's root directory can be written easily.
|
| |
|
|
|
|
|
| |
This class would be pretty useful in other cases where a system image
needs to be mounted and inspected.
Also updates the Trebuchet plugin to use this class.
|
| | |
|
| | |
|
| |
|
|
|
| |
This can go away when we have made a release with yaml in it, and its
staging filler.
|
| |\ |
|
| | |
| |
| |
| |
| | |
Tests are currently broken, one because invalid JSON
can be valid YAML, and coverage is incomplete.
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When morph is built, it writes various version information from git
into morphlib.
When morphlib is loaded it attempts to read this version information.
if it cannot be found then it checks whether morphlib is being run from
inside a git checkout, if it is then it reads the information that way.
If it isn't in a git checkout then it raises an exception as builds made
in such a fashion are not reproducible.
The git version information retained is:
1. The output of git describe
This is a relatively human-friendly way of knowing a version and
gives a reasonably short output string.
This will end with `-unreproducible` if there were uncommitted changes.
2. The commit sha1, so the exact part of Morph's history can be found
3. The tree sha1, so if the branch has been rebased rather than
merged such that the commit is lost, you may still be able to find
it, though it requires a git-wizard to check it out
4. The branch of morph, so that it's easier to see if the
Further possible changes to increase reproducibility include:
1. Not allowing `python setup.py build` if there are uncommitted changes
2. Failing to run with uncommitted changes (recommended against since it
will just annoy developers who are making changes to morph, and make
them commit just to shut it up, then destroy the history later)
Requiring an extra flag to build in this case may work better.
3. Reading the uncommitted changes into a tree object and including
that would allow it to be recovered if the tree was later committed.
4. Checking whether the commit has been pushed upstream as well.
Too annoying to work.
|
| |
|
|
| |
Reviewed-By: Daniel Silverstone (on irc)
|
| |
|
|
|
|
|
|
| |
morphlib.app should be for application bringup and providing
command line options used by the library code.
Any external plugins that refer to morphlib.app.BuildCommand
will need fixing.
|
| |
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| |
|
|
|
|
|
| |
The artifact cache doesn't need to know the whole of an artifact object
to be able to retrieve something from the cache, just the basename.
This can be generated from the contents, or just saved itself.
|
| | |
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
| |
This way, cliapp will automatically print an error message, rather
than a stack trace.
|
| |
|
|
|
|
|
|
| |
Traversing morphologies is a little simpler now, the callback
for visit is enough to create a source pool
This still needs repository caches to be able to list_files()
before deducing a morphology will work.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
With this commit, the ArtifactResolver no longer computes the cache keys
when creating Artifact objects. This will have to happen as a
post-resolving step (e.g. prior to building or checking whether a local
or remote artifact cache has any of the resolved artifacts).
The CacheKeyComputer now takes an Artifact object and computes the cache
keys using its dependencies.
BuildGraph is no longer needed for the CacheKeyComputer unit tests.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new setting called 'cache-server' to morph. It is None
by default and should be set to the HTTP URL of a morph cache server
to be used.
The RemoteRepoCache object provides two methods: resolve_ref() and
cat_file(), both of which wrap the communication with the cache
server and return a SHA1 string and file contents, respectively.
The _create_source_pool() method now takes a local and an optional
remote repo cache and tries to do whatever is best to resolve
refs and load morphologies for the Source objects it creates.
|
| |
|
|
|
|
|
| |
This class implements the build order part of the old BuildGraph.
It takes a list of artifacts with dependencies, creates independent
build groups for these artifacts and provides an iterable interface
to traverse these groups and their artifacts.
|
| | |
|
| |
|
|
|
|
|
| |
This class takes a CacheKeyComputer and a SourcePool, analyses the
sources and their dependencies and creates a list of artifacts
(represented by Artifact objects) that would be created when
building sources in the pool.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
BuildEnvironment should contain all the information about whatever
settings affect the build. A subset of these will affect the cache
key, but CacheKeyComputer filters them itself.
BuildEnvironment has consumed the purpose of Morph.clean_env() because
the build environment is a good place to store this information.
The environment variables are decided based on the current settings.
The current environment can be accessed by the env attribute.
The tests are a little invasive because it reads environment variables
and most of the work is done in the constructor, but they should test
everything useful.
|
