| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
| |
This hasn't been used in a long time to my knowledge, its API completely
misses the point, and its implementation relies on old APIs that need to
change.
Until we discover we need it, and work out what it should look like, I
think the best thing to do is get rid of it.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
`morph merge` only worked for a small subset of cases, and has been left
to bit-rot, since we don't use it.
`morph tag` is just a `git tag` when we have petrified definitions
repository. We don't use it, nor do we need it, so it can go away rather
than take up valuable development time fixing it when requirements
change.
`old-foo` have all been superceded by newer versions and are no-longer
used.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
The copy-artifacts and list-artifacts commands are mainly useful for
making releases. As part of the release process we copy artifacts for
the entire build graph of the release to the artifact cache on
trove.baserock.org, to provide Baserock users with ready-built
artifacts.
This part of the release process is now automated, and the automation
require the list-artifacts command to function as a 'plumbing' command.
The copy-artifacts command is no longer required. It can be replaced
with:
morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET
The previous version of this plugin looked in the system artifact's
metadata for the list of artifacts. This is flawed as the final system
does not necessarily contain every build dependency. The new version of
the plugin calculates the build graph from source, using the same
process as the 'buildcommand' module. It also required looking in
Morph's artifact cache for the system artifact file to analyse.
|
| |
|
|
|
|
|
|
|
| |
Add a module to morphlib that can list all write
and configuration extensions either in morph itself
or the morphology repository.
The module also contains methods to find an extension
filename from the name and type.
|
| |\
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Conflicts:
morphlib/plugins/deploy_plugin.py
without-test-modules
Reviewed by:
Richard Maw
Lars Wirzenius
|
| | | |
|
| |/
|
|
|
|
|
|
|
|
|
| |
Add a plugin to implement both `morph push` and `morph pull`.
These commands are wrappers around the corresponding git commands
push and pull, which also implement the functionality of pushing
and pulling large files provided by git-fat.
For example, running `morph pull` will pull any commits from the
remote branch not on your local branch, and then pull any large
files from the separate git-fat/rsync store on the Trove.
|
| |
|
|
| |
This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
|
| | |
|
| |
|
|
|
|
|
|
|
| |
This introduces a new artifactsplitrule module, which tries to provide a
nice abstraction over matching a sequence of things to a bunch of
outputs, to be used by both chunks splitting, for separating files out
into chunk artifacts, the stratum splitting, where chunks are
aggregated into stratum artifacts, and systems selecting the right
strata to go into the artifact.
|
| |
|
|
|
|
|
| |
This is an abstraction on top of SystemBranchDirectories, providing the
ability to add uncommitted changes to the temporary build branch, push
temporary build branches and retrieve the correct repository URI and ref
to build the system.
|
| |\
| |
| |
| | |
Reviewed-by: Lars Wirzenius
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This will list all the paths generated by the walker generator function
that aren't in the specified set.
It removes directories from those returned by the walker, since with
os.walk(topdown=True) this culls the search space.
In the set of provided paths and the set of returned paths, if a directory
is given, then its contents are virtually part of the set.
This oddly specific behaviour is because invert_paths is to be used
with linux-user-chroot to mark subtrees as read-only, when it only
has a set of paths it wants to keep writable.
It takes a walker, rather than being given a path and using os.walk, so
that it is a pure function, so is easier to unit test.
|
| |/ |
|
| |
|
|
|
|
|
| |
Move "morph init" into a plugin, and remove it from the previously
existing plugin. This keeps all the old, tricky code in the old
plugin, and moves new, clean code into a new plugin. Eventually the
old plugin can be removed, since it'll be empty.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Cross-bootstrap is a way to build baserock on an architecture that
does not currently have Baserock. It can be used by `morph
cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact
that can be used as a root filesystem with a basic build environment
with a script named `native-bootstrap` which will build and install
every chunk in the system.
If done with a devel system, this will give you a suitable environment
for building a proper Baserock system.
This does not currently provide a kernel for the target architecture.
Apart from adding the cross-bootstrap plugin, it also makes the
following changes:
* Moves the lit of valid_archs into morphlib (instead of locally-scoped
in MorphologyFactory)
* BuildCommand takes an extra argument, build_env
* split BuildCommand's get_artifact_object into create_source_pool and
resolve_artifacts (plus changes things that use get_artifact_object to
use the new way)
* setup_mounts finds out whether to do so by whether build_mode is
'staging', instead of by whether the setting 'staging-chroot' is true.
* Makes ChunkBuilder's get_sources use the
morphlib.builder2.extract_sources() method, and moved
set_mtime_recursively into morphlib.builder2, since it's not currently
used anywhere else.
* moved ChunkBuilder's get_commands into the Morphology class (plus
changes to anything that used get_commands)
|
| |
|
|
|
|
| |
In addition, when we start using tbdiff for upgrades, all of this will
need to be re-designed and re-implemented anyway. The current plugin
is dead code. Note that the test suite had already been disabled.
|
| |
|
|
|
|
|
|
|
|
| |
This removes staging areas and extracted chunks from --tempdir.
Then asks the local artifact cache what artifacts it
has and how old they are, removing all sources older than
--cachedir-artifact-delete-older-than, and may delete other sources that
are younger than --cachedir-artifact-keep-younger-than if it still needs
to make space.
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
| |
This adds a `run-in-artifact` command which allows another command
to be run in a system.
There is also a `content-manifest` command which gives a
manifest of the artifacts, which commits they were built from,
and if possible, a version.
This adds a morphlib.bins.call_in_artifact_directory() method
to run a command inside an artifact and to generate a manifest.
|
| |
|
|
|
|
|
|
|
|
|
|
|
| |
ExtractedTarball is more or less the equivalent to MountableImage for
artifacts that are not mountable images. So in order to inspect root
file system tarballs, ExtractedTarball can be used, for disk images,
MountableImage can be used.
The morphlib.bins.call_in_artifact_directory() method combines these
two classes and provides a way to extract/mount an artifact and call
a callback with the temporary directory / mount point as its first
argument. Using this, a plugin that runs a command relative to an
artifact's root directory can be written easily.
|
| |
|
|
|
|
|
| |
This class would be pretty useful in other cases where a system image
needs to be mounted and inspected.
Also updates the Trebuchet plugin to use this class.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
| |
This adds a new optional field to system morphologies:
"configuration-extensions".
The deployment plugin relies heavily on code from the branch and
merge plugin. This needs to be eventually fixed by refactoring
the codebase so that the shared code is in morphlib and not in
plugins. However, doing that is beyond the scope of adding a
deployment plugin.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When morph is built, it writes various version information from git
into morphlib.
When morphlib is loaded it attempts to read this version information.
if it cannot be found then it checks whether morphlib is being run from
inside a git checkout, if it is then it reads the information that way.
If it isn't in a git checkout then it raises an exception as builds made
in such a fashion are not reproducible.
The git version information retained is:
1. The output of git describe
This is a relatively human-friendly way of knowing a version and
gives a reasonably short output string.
This will end with `-unreproducible` if there were uncommitted changes.
2. The commit sha1, so the exact part of Morph's history can be found
3. The tree sha1, so if the branch has been rebased rather than
merged such that the commit is lost, you may still be able to find
it, though it requires a git-wizard to check it out
4. The branch of morph, so that it's easier to see if the
Further possible changes to increase reproducibility include:
1. Not allowing `python setup.py build` if there are uncommitted changes
2. Failing to run with uncommitted changes (recommended against since it
will just annoy developers who are making changes to morph, and make
them commit just to shut it up, then destroy the history later)
Requiring an extra flag to build in this case may work better.
3. Reading the uncommitted changes into a tree object and including
that would allow it to be recovered if the tree was later committed.
4. Checking whether the commit has been pushed upstream as well.
Too annoying to work.
|
| | |
|
| |
|
|
|
|
|
| |
This adds a disk system image kind which does everything the syslinux-disk
one does, but without syslinux. It deliberately carries stubbed bootloader
operations so that we can later make syslinux-disk inherit from this one
and override it.
|
| |
|
|
|
| |
It's such a small amount of code, it's possibly not worth it,
but now all commands are in plugins.
|
| |
|
|
|
|
|
|
| |
morphlib.app should be for application bringup and providing
command line options used by the library code.
Any external plugins that refer to morphlib.app.BuildCommand
will need fixing.
|
| |
|
|
|
| |
deduce_mine_directory is duplicated in the plugin and morphlib.app
until all the commands are moved into the plugin.
|
| |
|
|
|
|
|
|
| |
The plugin is called trebuchet, so that if any other functionality
is required from morph it has a place to go.
make-patch generates a trebuchet patch, so it should go in the
trebuchet plugin.
|
| |
|
|
| |
This also publicizes cache_repo_and_submodules and traverse_morphs
|
| | |
|
| |
|
|
|
| |
Also, rename the syslinux-disk builder plugin file to make it clear
it's a system builder plugin.
|
| | |
|
| |
|
|
| |
Mea culpa. I managed to make changes without running ./check properly.
|
| |
|
|
|
|
|
| |
The artifact cache doesn't need to know the whole of an artifact object
to be able to retrieve something from the cache, just the basename.
This can be generated from the contents, or just saved itself.
|
| | |
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
| |
The helper class, Factory, has unit tests, which is why it's currently
separate. It may later get integrated with BlobBuilder, or the other
way around.
Classes that don't have unit tests are marked out of coverage.
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This commit introduces four new classes:
BuildController:
* takes an app instance and a tempdir
* allows to add BuildWorker objects
* provides a build() method that takes a set of blobs and
a build order that is then built by assigning work to the
build workers as needed
* the build() method takes care of polling the workers for
their state, moving them between busy and idle states
reliably, collect and print their output in a non-confusing
order, and makes sure to wait for all workers to finish
before processing the next group in the build order.
* at this point, when waiting for one or more workers to
become idle to assign them another blob to build, the
controller always picks the worker that has been idling
for the longest period of time. this can be changed later.
BuildWorker:
* base class for all worker classes
* takes a name and an app instance
* has a idle_since datetime property
* provides a build() method that takes a Blob object and builds
it in whatever way the subclasses implement it
* provides a check_complete(timeout) method that checks whether
the worker has finished building the blob yet or not
LocalBuildWorker:
* worker class for local builds that don't go through SSH
* it uses morphlib.execute.Execute to run morph in a child process
in build()
* at the moment, this class executes "./morph" instead of "morph"
as it assumes the user to run morph from its source tree. obviously,
this will have to be fixed later.
RemoteBuildWorker:
* doesn't implement anything yet, will be used for distributing
work to other machines running morph via SSH
Notes:
* At the moment, there is a degree of undesired redundancy when
building a stratum in a worker, as this will cause the worker to
rebuild all its dependencies. This will have to be fixed as it is
avoidable and wastes a lot of time and processing power.
|
| | |
|
| | |
|
| | |
|
| |
|
