path: root/without-test-modules
Commit message (Collapse)AuthorAgeFilesLines
* Remove update-gits commandRichard Maw2014-07-221-1/+0
| | | | | | | | | This hasn't been used in a long time to my knowledge, its API completely misses the point, and its implementation relies on old APIs that need to change. Until we discover we need it, and work out what it should look like, I think the best thing to do is get rid of it.
* Exorcise some old and unused commandsRichard Maw2014-07-071-1/+0
| | | | | | | | | | | | | `morph merge` only worked for a small subset of cases, and has been left to bit-rot, since we don't use it. `morph tag` is just a `git tag` when we have petrified definitions repository. We don't use it, nor do we need it, so it can go away rather than take up valuable development time fixing it when requirements change. `old-foo` have all been superceded by newer versions and are no-longer used.
* Rewrite copy_artifacts plugin as list_artifactsSam Thursfield2014-06-051-1/+1
| | | | | | | | | | | | | | | | | | | | | | The copy-artifacts and list-artifacts commands are mainly useful for making releases. As part of the release process we copy artifacts for the entire build graph of the release to the artifact cache on, to provide Baserock users with ready-built artifacts. This part of the release process is now automated, and the automation require the list-artifacts command to function as a 'plumbing' command. The copy-artifacts command is no longer required. It can be replaced with: morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET The previous version of this plugin looked in the system artifact's metadata for the list of artifacts. This is flawed as the final system does not necessarily contain every build dependency. The new version of the plugin calculates the build graph from source, using the same process as the 'buildcommand' module. It also required looking in Morph's artifact cache for the system artifact file to analyse.
* Add utilities for listing and finding extensions.Mark Doffman2014-03-311-0/+1
| | | | | | | | | Add a module to morphlib that can list all write and configuration extensions either in morph itself or the morphology repository. The module also contains methods to find an extension filename from the name and type.
* Merge branch 'baserock/richardipsum/merge_distbuild'Richard Ipsum2014-03-271-0/+1
|\ | | | | | | | | | | | | | | | | | | Conflicts: morphlib/plugins/ without-test-modules Reviewed by: Richard Maw Lars Wirzenius
| * Don't test distbuild pluginRichard Ipsum2014-03-211-0/+1
| |
* | Implement morph push and morph pulladamcoldrick/ingest-binaries-v5-rebaseAdam Coldrick2014-03-201-0/+2
|/ | | | | | | | | | | Add a plugin to implement both `morph push` and `morph pull`. These commands are wrappers around the corresponding git commands push and pull, which also implement the functionality of pushing and pulling large files provided by git-fat. For example, running `morph pull` will pull any commits from the remote branch not on your local branch, and then pull any large files from the separate git-fat/rsync store on the Trove.
* Revert "Add utilities for listing and finding extensions."Mark Doffman2014-02-211-1/+0
| | | | This reverts commit ab0a83a09a93ca33aa402d9c4d3b916a48a1a882.
* Add utilities for listing and finding extensions.Mark Doffman2014-02-211-0/+1
* Add split rules to sourcesRichard Maw2014-01-161-0/+1
| | | | | | | | | This introduces a new artifactsplitrule module, which tries to provide a nice abstraction over matching a sequence of things to a bunch of outputs, to be used by both chunks splitting, for separating files out into chunk artifacts, the stratum splitting, where chunks are aggregated into stratum artifacts, and systems selecting the right strata to go into the artifact.
* morphlib: Add BuildBranch abstractionRichard Maw2013-11-291-0/+2
| | | | | | | This is an abstraction on top of SystemBranchDirectories, providing the ability to add uncommitted changes to the temporary build branch, push temporary build branches and retrieve the correct repository URI and ref to build the system.
* Merge branch 'baserock/richardmaw/S8563/bootstrap-rootfs-protection'Richard Maw2013-08-231-1/+0
|\ | | | | | | Reviewed-by: Lars Wirzenius
| * fsutils: add invert_paths functionRichard Maw2013-08-151-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This will list all the paths generated by the walker generator function that aren't in the specified set. It removes directories from those returned by the walker, since with os.walk(topdown=True) this culls the search space. In the set of provided paths and the set of returned paths, if a directory is given, then its contents are virtually part of the set. This oddly specific behaviour is because invert_paths is to be used with linux-user-chroot to mark subtrees as read-only, when it only has a set of paths it wants to keep writable. It takes a walker, rather than being given a path and using os.walk, so that it is a pure function, so is easier to unit test.
* | Add "morph print-architecture" subcommandLars Wirzenius2013-08-151-0/+1
* Re-implement "morph init" using the new Workspace classLars Wirzenius2013-07-311-0/+1
| | | | | | | Move "morph init" into a plugin, and remove it from the previously existing plugin. This keeps all the old, tricky code in the old plugin, and moves new, clean code into a new plugin. Eventually the old plugin can be removed, since it'll be empty.
* Add morph cross-bootstrapSam Thursfield2013-07-301-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cross-bootstrap is a way to build baserock on an architecture that does not currently have Baserock. It can be used by `morph cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact that can be used as a root filesystem with a basic build environment with a script named `native-bootstrap` which will build and install every chunk in the system. If done with a devel system, this will give you a suitable environment for building a proper Baserock system. This does not currently provide a kernel for the target architecture. Apart from adding the cross-bootstrap plugin, it also makes the following changes: * Moves the lit of valid_archs into morphlib (instead of locally-scoped in MorphologyFactory) * BuildCommand takes an extra argument, build_env * split BuildCommand's get_artifact_object into create_source_pool and resolve_artifacts (plus changes things that use get_artifact_object to use the new way) * setup_mounts finds out whether to do so by whether build_mode is 'staging', instead of by whether the setting 'staging-chroot' is true. * Makes ChunkBuilder's get_sources use the morphlib.builder2.extract_sources() method, and moved set_mtime_recursively into morphlib.builder2, since it's not currently used anywhere else. * moved ChunkBuilder's get_commands into the Morphology class (plus changes to anything that used get_commands)
* Remove "morph make-patch" since it doesn't workLars Wirzenius2013-07-051-1/+0
| | | | | | In addition, when we start using tbdiff for upgrades, all of this will need to be re-designed and re-implemented anyway. The current plugin is dead code. Note that the test suite had already been disabled.
* plugins: add morph gc subcommandRichard Maw2013-06-071-0/+1
| | | | | | | | | | This removes staging areas and extracted chunks from --tempdir. Then asks the local artifact cache what artifacts it has and how old they are, removing all sources older than --cachedir-artifact-delete-older-than, and may delete other sources that are younger than --cachedir-artifact-keep-younger-than if it still needs to make space.
* Add the trovectl plugin to the morph plugins directory.Tiago Gomes2013-05-231-0/+1
* Acknowledge the lack of test coverageJonathan Maw2013-03-141-0/+1
* Add image inspection pluginJannis Pohlmann2013-02-211-0/+1
| | | | | | | | | | | | This adds a `run-in-artifact` command which allows another command to be run in a system. There is also a `content-manifest` command which gives a manifest of the artifacts, which commits they were built from, and if possible, a version. This adds a morphlib.bins.call_in_artifact_directory() method to run a command inside an artifact and to generate a manifest.
* Add ExtractedTarball class and method to extract/mount an artifactJannis Pohlmann2013-02-211-0/+1
| | | | | | | | | | | | | ExtractedTarball is more or less the equivalent to MountableImage for artifacts that are not mountable images. So in order to inspect root file system tarballs, ExtractedTarball can be used, for disk images, MountableImage can be used. The morphlib.bins.call_in_artifact_directory() method combines these two classes and provides a way to extract/mount an artifact and call a callback with the temporary directory / mount point as its first argument. Using this, a plugin that runs a command relative to an artifact's root directory can be written easily.
* Move MountableImage class into morphlibJannis Pohlmann2013-02-211-0/+1
| | | | | | | This class would be pretty useful in other cases where a system image needs to be mounted and inspected. Also updates the Trebuchet plugin to use this class.
* Add morphlib module for common write extension codeLars Wirzenius2013-02-061-0/+1
* Add deployment pluginLars Wirzenius2013-02-061-1/+2
| | | | | | | | | | | This adds a new optional field to system morphologies: "configuration-extensions". The deployment plugin relies heavily on code from the branch and merge plugin. This needs to be eventually fixed by refactoring the codebase so that the shared code is in morphlib and not in plugins. However, doing that is beyond the scope of adding a deployment plugin.
* Make morph get its version from git.Richard Maw2013-01-281-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When morph is built, it writes various version information from git into morphlib. When morphlib is loaded it attempts to read this version information. if it cannot be found then it checks whether morphlib is being run from inside a git checkout, if it is then it reads the information that way. If it isn't in a git checkout then it raises an exception as builds made in such a fashion are not reproducible. The git version information retained is: 1. The output of git describe This is a relatively human-friendly way of knowing a version and gives a reasonably short output string. This will end with `-unreproducible` if there were uncommitted changes. 2. The commit sha1, so the exact part of Morph's history can be found 3. The tree sha1, so if the branch has been rebased rather than merged such that the commit is lost, you may still be able to find it, though it requires a git-wizard to check it out 4. The branch of morph, so that it's easier to see if the Further possible changes to increase reproducibility include: 1. Not allowing `python build` if there are uncommitted changes 2. Failing to run with uncommitted changes (recommended against since it will just annoy developers who are making changes to morph, and make them commit just to shut it up, then destroy the history later) Requiring an extra flag to build in this case may work better. 3. Reading the uncommitted changes into a tree object and including that would allow it to be recovered if the tree was later committed. 4. Checking whether the commit has been pushed upstream as well. Too annoying to work.
* Add plugin to without-test-modules listLars Wirzenius2012-11-071-0/+1
* Initial non-syslinux disk image kindDaniel Silverstone2012-08-141-0/+1
| | | | | | | This adds a disk system image kind which does everything the syslinux-disk one does, but without syslinux. It deliberately carries stubbed bootloader operations so that we can later make syslinux-disk inherit from this one and override it.
* Move cmd_build to a separate pluginRichard Maw2012-08-011-0/+1
| | | | | It's such a small amount of code, it's possibly not worth it, but now all commands are in plugins.
* Move BuildCommand from app into its own moduleRichard Maw2012-08-011-0/+1
| | | | | | | | should be for application bringup and providing command line options used by the library code. Any external plugins that refer to will need fixing.
* Move init and minedir to branch_and_merge pluginRichard Maw2012-08-011-0/+1
| | | | | deduce_mine_directory is duplicated in the plugin and until all the commands are moved into the plugin.
* Move command make-patch to Trebuchet pluginRichard Maw2012-08-011-0/+1
| | | | | | | | The plugin is called trebuchet, so that if any other functionality is required from morph it has a place to go. make-patch generates a trebuchet patch, so it should go in the trebuchet plugin.
* morph: move update-gits to pluginRichard Maw2012-08-011-0/+1
| | | | This also publicizes cache_repo_and_submodules and traverse_morphs
* move show-dependencies command into a pluginRichard Maw2012-08-011-0/+1
* Start a tarball systembuilderLars Wirzenius2012-07-261-1/+3
| | | | | Also, rename the syslinux-disk builder plugin file to make it clear it's a system builder plugin.
* Move syslinux-disk system building into pluginLars Wirzenius2012-07-261-0/+1
* Fix test suiteLars Wirzenius2012-07-121-0/+1
| | | | Mea culpa. I managed to make changes without running ./check properly.
* morphlib: add artifact cache reference classRichard Maw2012-06-131-0/+1
| | | | | | | The artifact cache doesn't need to know the whole of an artifact object to be able to retrieve something from the cache, just the basename. This can be generated from the contents, or just saved itself.
* Mark plugin to not be unit testedLars Wirzenius2012-06-111-0/+1
* Add a preliminary sample pluginLars Wirzenius2012-06-111-0/+1
* morph: add make_patch commandRichard Maw2012-03-231-0/+1
* Refactor SystemBuilder and add helper classLars Wirzenius2012-03-021-1/+0
| | | | | | | | The helper class, Factory, has unit tests, which is why it's currently separate. It may later get integrated with BlobBuilder, or the other way around. Classes that don't have unit tests are marked out of coverage.
* Add poor man's unit tests for controller and workers.Jannis Pohlmann2012-01-251-2/+0
* Add controller, worker classes and a new "build-distributed" command.Jannis Pohlmann2012-01-231-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit introduces four new classes: BuildController: * takes an app instance and a tempdir * allows to add BuildWorker objects * provides a build() method that takes a set of blobs and a build order that is then built by assigning work to the build workers as needed * the build() method takes care of polling the workers for their state, moving them between busy and idle states reliably, collect and print their output in a non-confusing order, and makes sure to wait for all workers to finish before processing the next group in the build order. * at this point, when waiting for one or more workers to become idle to assign them another blob to build, the controller always picks the worker that has been idling for the longest period of time. this can be changed later. BuildWorker: * base class for all worker classes * takes a name and an app instance * has a idle_since datetime property * provides a build() method that takes a Blob object and builds it in whatever way the subclasses implement it * provides a check_complete(timeout) method that checks whether the worker has finished building the blob yet or not LocalBuildWorker: * worker class for local builds that don't go through SSH * it uses morphlib.execute.Execute to run morph in a child process in build() * at the moment, this class executes "./morph" instead of "morph" as it assumes the user to run morph from its source tree. obviously, this will have to be fixed later. RemoteBuildWorker: * doesn't implement anything yet, will be used for distributing work to other machines running morph via SSH Notes: * At the moment, there is a degree of undesired redundancy when building a stratum in a worker, as this will cause the worker to rebuild all its dependencies. This will have to be fixed as it is avoidable and wastes a lot of time and processing power.
* Add to without-test-modules.Jannis Pohlmann2012-01-191-0/+1
* add tester to without-test-modulesLars Wirzenius2011-12-091-0/+1
* Start a git module for abstracting away git operations.Lars Wirzenius2011-10-171-0/+1
* Initial import.Lars Wirzenius2011-09-291-0/+2