| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
There's not really any reason you'd want to use the RemoteRepoCache
class except as a workaround for the slow speed of some LocalRepoCache
operations, so I can't see this ruining anyone's day.
The main reason for doing this is so we can simply the sourceresolver
code. One reason that the sourceresolver class is so hopelessly
complicated is that it right now has to use two incompatible interfaces
for Git repo caches.
I've taken the opportunity to detangle the RepoCache class from the
App class. Now all of the configuration for the RepoCache class is
passed into the constructor explicitly. This makes the class usable from
outside Morph:
resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[])
repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver)
Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
For a long time the CachedRepo class has basically been a wrapper around
the GitDir class, but with a few extra methods that don't really even
belong there.
It is now a tiny class in the localrepocache module which just keeps
track of a few extra attributes. All other functionality is provided
by the gitdir module.
This commit also removes the `git clone` approach for copying repos out
of the cache. The alternative approach implemented by
git.copy_repository() was slightly faster when I tested, so for now we
should use that everywhere. Longer term we should find out why this is
quicker than `git clone`, and fix Git itself to be fast.
Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
|
|
|
|
| |
Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
|
|
|
|
|
|
| |
This class was only used on the now removed branch-from-image command.
Change-Id: I75a0b4618b16a24bd9f3b3ea7b3a6228db723715
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a new 'Defaults' class to represent definitions defaults
The Python 'jsonschema' module is used to validate the contents of the
Defaults file. This module is already included in Baserock 'build' and
'devel' reference systems by way of the 'openstack-common' stratum.
This commit embeds a copy of the JSON-Schema schema for the DEFAULTS
file. I think the canonical location of this schema should be in the
reference definitions.git, for now. In future, the schemas should maybe
have their own repos. Either way, Morph should embed a copy for the time
being so that we are sure the schema matches how Morph expects to parse
the file.
Morph's automated tests are all updated to use definitions version 7.
I removed most of the tests for built-in build systems, because the
built-ins themselves are no longer part of Morph. Only the mechanism for
defining them needs to be tested now.
Change-Id: I65f8f1c967683ef605852bfae5c68518e53f9981
|
|
|
|
|
|
|
| |
I need to use this outside of the 'sourceresolver' module. Also, the
'sourceresolver' module is too big.
Change-Id: I523bcc9555193b7369768441b72f1059e6adde5c
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The intention is for this class to take over the from the Workspace and
SystemBranch classes. It allows Morph to load and parse definitions from
a Git repo, without requiring the user to run `morph checkout` or
`morph branch`: it can operate from any normal Git repository.
The class behaves differently when the Git repository is inside a Morph
system-branch checkout made with `morph branch` or `morph checkout`, to
avoid changing things under the feet of people who are used to those
commands.
Change-Id: I52a898efb9f6fb7f7e94c65b9ed38516bd51f49d
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The `morph anchor`, `morph build-morphology` and a potential `morph
diff` command would all benefit from having a unified way to parse the
argv for the systems it must operate on, especially in the case of the
potential `morph diff`, which needs to be able to handle being given two
sets of systems.
`morph anchor` may make use of it now by passing the list of systems to
the Source resolver, but `morph build-morphology` would have to iterate
over the systems and graph each independently.
Change-Id: I91ab4764ffca3aa16f144f89f68f37cc21b9f643
|
|
|
|
|
|
|
|
| |
Also add support to allow building compatible architectures on armv7
machines, as per the rationale in the comment in
_validate_architecture().
Change-Id: Ie0fe4002523f1f92f576bac8b654d4ea6fad5cf2
|
|
|
|
| |
Change-Id: I9344b9b80a6ec008715559390b63c9003f34bf90
|
|
|
|
| |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
|
|
|
|
| |
There's no need for this stuff to be in a separate class. This allows
integrating it with the caching in the SourceResolver class.
|
| |
|
| |
|
|
|
|
|
|
|
|
| |
This code is an essential part of 'morph build'. It's quite complex and
really shouldn't be mixed in with the base Application class.
Given a dedicated class we can store some state in the object and avoid
functions with seven parameters, too.
|
|
|
|
|
|
| |
Instead of leaving morph3 with a potentially confusing name, rename it
to `morphology` since it is now the only implementation of the
Morphology class.
|
|
|
|
| |
This commit removes the now unneeded morph2 and its associated tests.
|
|
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
|
|
|
| |
This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
|
| |
|
| |
|
|
|
|
|
|
|
|
|
| |
This introduces a new artifactsplitrule module, which tries to provide a
nice abstraction over matching a sequence of things to a bunch of
outputs, to be used by both chunks splitting, for separating files out
into chunk artifacts, the stratum splitting, where chunks are
aggregated into stratum artifacts, and systems selecting the right
strata to go into the artifact.
|
| |
|
| |
|
|
|
|
|
|
|
| |
This is an abstraction on top of SystemBranchDirectories, providing the
ability to add uncommitted changes to the temporary build branch, push
temporary build branches and retrieve the correct repository URI and ref
to build the system.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This adds a LocalRefManager, which handles ref updates to local
repositories (i.e. your workspace).
It provides proxy methods for ref updates to a set of repositories.
If an exception occurs in the body of the context manager, the updates
will be rolled back to before the context manager was entered.
The purpose for using a LocalRefManager instead of making the changes to
the repositories directly, is to provide atomic updates to a set of refs
in a set of repositories, where all refs are updated, or none are.
This also adds a RemoteRefManager, which handles pushing branches to
remote repositories.
It provides a proxy push method, which will delete pushed branches, and
re-push deleted branches after the context manager exits.
Its purpose, instead of providing atomic updates to remote repositories,
is to provide temporary branches. This is because it is used to provide
temporary build branches. The difference between atomic update and
temporary push, is that the remote branches are deleted when the context
is left, rather than kept, as LocalRefManager does.
The RemoteRefManager currently cannot provide the same atomicity
guarantees as the LocalRefManager, so if there is a push between the
branch being created and the RemoteRefManager cleaning it up, that change
is lost without RemoteRefManager even knowing it existed.
Git 1.8.5 will add functionality to make this possible.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This represents the state of the index of a GitDirectory.
Methods that use the index are now used via the GitIndex class, rather
than using the default index, as previously used when the methods were
in GitDirectory.
GitIndex may be constructed with an alternative path, which can be used
to manipulate a git checkout without altering a developer's view of the
repository i.e. The working tree and default index.
This is needed for `morph build` and `morph deploy` to handle the build
without commit logic.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This provides access to the /baserock directory as if it were a dict,
abstracting away the details of how to get data out of it.
The abstraction is useful since it is easier to use than accessing
/baserock yourself, and allows the storage format to be changed more
easily.
Keys with / in may be supported in the future. since there have been
discussions about allowing morphologies to be placed in subdirectories.
Adding this support would require creating and removing directory
components when values are set and deleted respectively.
Iterating would require using os.walk instead of glob.iglob, since
python doesn't support ** in globs.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
MorphologyFinder is a small wrapper on top of GitDirectory that
allows the inspection of morphologies in the repository.
Its purpose is to isolate the logic for reading morphologies into one
place.
It is used by passing a GitDirectory and optionally a ref to the
MorpholgyFinder constructor, then list_morphologies and read_morphology
may be used.
The ref is passed directly to the GitDirectory, so its semantics for
a ref of None or omitted are used. i.e. It uses the working tree.
Ref resolving is deferred until a morphology is listed or read, so
it will not raise an exception for an invalid ref until then.
|
|
|
|
|
|
|
| |
Various parts of Morph need to change a set of morphologies at once,
particularly for petrification and unpetrification. This is easiest
done by loading all the morphologies into memory at once, and changing
them there, then saving again.
|
|
|
|
|
|
|
|
|
| |
The old code is somewhat weird. The new code is meant to be cleaner and more
straightforward to understand and to use. For example, the old code has setting
of defaults in both the Morphology and MorphologyFactory classes. The new code
has a minimally simple Morphology class, and has all the logic to validate and
set defaults in the MorphologyLoader class. Further, the new code makes it
possible to load an invalid morphology, which will be useful later.
|
| |
|
| |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
ExtractedTarball is more or less the equivalent to MountableImage for
artifacts that are not mountable images. So in order to inspect root
file system tarballs, ExtractedTarball can be used, for disk images,
MountableImage can be used.
The morphlib.bins.call_in_artifact_directory() method combines these
two classes and provides a way to extract/mount an artifact and call
a callback with the temporary directory / mount point as its first
argument. Using this, a plugin that runs a command relative to an
artifact's root directory can be written easily.
|
|
|
|
|
|
|
| |
This class would be pretty useful in other cases where a system image
needs to be mounted and inspected.
Also updates the Trebuchet plugin to use this class.
|
| |
|
| |
|
|
|
|
|
| |
This can go away when we have made a release with yaml in it, and its
staging filler.
|
|\ |
|
| |
| |
| |
| |
| | |
Tests are currently broken, one because invalid JSON
can be valid YAML, and coverage is incomplete.
|
|/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When morph is built, it writes various version information from git
into morphlib.
When morphlib is loaded it attempts to read this version information.
if it cannot be found then it checks whether morphlib is being run from
inside a git checkout, if it is then it reads the information that way.
If it isn't in a git checkout then it raises an exception as builds made
in such a fashion are not reproducible.
The git version information retained is:
1. The output of git describe
This is a relatively human-friendly way of knowing a version and
gives a reasonably short output string.
This will end with `-unreproducible` if there were uncommitted changes.
2. The commit sha1, so the exact part of Morph's history can be found
3. The tree sha1, so if the branch has been rebased rather than
merged such that the commit is lost, you may still be able to find
it, though it requires a git-wizard to check it out
4. The branch of morph, so that it's easier to see if the
Further possible changes to increase reproducibility include:
1. Not allowing `python setup.py build` if there are uncommitted changes
2. Failing to run with uncommitted changes (recommended against since it
will just annoy developers who are making changes to morph, and make
them commit just to shut it up, then destroy the history later)
Requiring an extra flag to build in this case may work better.
3. Reading the uncommitted changes into a tree object and including
that would allow it to be recovered if the tree was later committed.
4. Checking whether the commit has been pushed upstream as well.
Too annoying to work.
|
|
|
|
| |
Reviewed-By: Daniel Silverstone (on irc)
|
|
|
|
|
|
|
|
| |
morphlib.app should be for application bringup and providing
command line options used by the library code.
Any external plugins that refer to morphlib.app.BuildCommand
will need fixing.
|
|
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
|
|
|
|
|
|
| |
The artifact cache doesn't need to know the whole of an artifact object
to be able to retrieve something from the cache, just the basename.
This can be generated from the contents, or just saved itself.
|
| |
|