summaryrefslogtreecommitdiff
path: root/morphlib/plugins
Commit message (Collapse)AuthorAgeFilesLines
* Use sanitise_morphology_path in serialise_artifactRichard Ipsum2014-07-311-1/+1
|
* Fix list-artifacts post chunks-in-definitionsbaserock/richardmaw/bugfix/fix-list-artifactsRichard Maw2014-07-281-1/+1
| | | | | | | The chunks in definitions change changed the api of create_source_pool, and because list artifacts is not covered by any tests, and the create_source_pool method was confused with a function of the same name that did not need changing, it failed to be fixed.
* Remove update-gits commandRichard Maw2014-07-221-83/+0
| | | | | | | | | This hasn't been used in a long time to my knowledge, its API completely misses the point, and its implementation relies on old APIs that need to change. Until we discover we need it, and work out what it should look like, I think the best thing to do is get rid of it.
* Tidy deploy_plugin a littleRichard Maw2014-07-221-5/+5
|
* Fix JSON file handling to be binary filename safeLars Wirzenius2014-07-151-1/+2
|\ | | | | | | | | Reviewed-by: Lars Wirzenius Reviewed-by: Pedro Alvarez
| * Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-111-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* | Fix bug in list-artifacts pluginRichard Ipsum2014-07-111-1/+1
|/
* Use exact filenames to refer to morphology filesRichard Maw2014-07-105-27/+28
| | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
* Make MorphologyFinder use file pathsRichard Maw2014-07-071-3/+2
| | | | | | | | | | | | | | | | | | | | We want to use file paths to locate morphologies now, so the old model of get a list of names and hand it those back to get the contents doesn't really make sense any more. This abstraction initially came about as one idea I had for moving morphologies out of the source tree was to put them in something like git notes, where it's possible to look up information for one commit in another ref in the repository, at which point this abstraction would have been flexible enough to handle that as well as in the However, moving the chunk morphologies into the definitions repository has other benefits too, so it makes more sense to be honest about using filenames in the API. It remains as a single point where we can put the logic for knowing which files in a repository look like morphologies, but if we need to remove any further functionality, it should be replaced by a single function.
* Exorcise some old and unused commandsRichard Maw2014-07-072-2442/+536
| | | | | | | | | | | | | `morph merge` only worked for a small subset of cases, and has been left to bit-rot, since we don't use it. `morph tag` is just a `git tag` when we have petrified definitions repository. We don't use it, nor do we need it, so it can go away rather than take up valuable development time fixing it when requirements change. `old-foo` have all been superceded by newer versions and are no-longer used.
* Remove check on checkout/branch that there are systemsRichard Maw2014-07-071-22/+0
| | | | | | | This was the wrong response to the problem of accidentally checking-out morph when trying to check out morphs. Now that it's called definitions, this is irrelevent, and aborting a checkout when this check fails is the wrong thing to do.
* Remove code made vestigial from the `morph edit` rewriteRichard Maw2014-07-071-64/+0
| | | | | | This was originally used by `morph edit`, but since we removed the need to run `morph edit system stratum` and could shorten it to `morph edit chunk`, this function is no longer used.
* Update the help text for morph deployAdam Coldrick2014-06-181-1/+5
|
* Allow the user to specify deployments in a clusterAdam Coldrick2014-06-181-8/+43
| | | | | | | | Instead of taking the name of a cluster morphology and zero or more parameters for overriding the cluster morphology, morph deploy should take the name of a cluster morphology and the names of zero or more system deployments that are defined in the cluster morphology. If no deployment names are given then all deployments are deployed.
* Merge remote-tracking branch 'origin/sam/distbuild-gc-fix'Sam Thursfield2014-06-161-0/+3
|\ | | | | | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Don't delete subdirectories of tempdir in gc pluginSam Thursfield2014-06-131-0/+3
| | | | | | | | | | | | There is code that assumes these exist in at least one place: StagingArea.abort(). That code should be fixed, but we should also stop deleting them every time we run 'morph gc'.
* | Merge branch 'sam/remote-artifact-cache-failure'Sam Thursfield2014-06-101-12/+2
|\ \ | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * | Improve robustness when fetching artifacts from remote artifact cacheSam Thursfield2014-06-031-12/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Previously Morph would check if an artifact is present in the remote artifact cache, then fetch the necessary files. If an error occured during fetching, it would raise an error and abort. Instead, we should just try and fetch the files and if anything fails, move on to building locally. This avoids the situation where an error in the remote cache makes local building impossible, which we experienced recently.
* | | Morph deploy should never update git reposPaul Sherwood2014-06-101-0/+1
| |/ |/| | | | | | | | | | | | | The deploy plugin is relying on existing code which respects the 'no-git-update' setting and updates gits if it is not set. Since deploy can only work for built systems anyway, we can force this True for deployments, to avoid wasting the user/caller's time.
* | Fix up before mergeSam Thursfield2014-06-091-36/+43
| | | | | | | | | | | | | | - All tests now pass - The odd case of chunks with the same name but different repo URLs now correctly informs the user of the multiple checkouts that were done. - Tidyups
* | Add messages to help user know what is happeningPaul Sherwood2014-06-081-0/+16
| |
* | Simplify morph edit syntax and logicPaul Sherwood2014-06-081-124/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since we removed ref: fields from the system morph files, some of the logic in morph edit is no longer needed. In particular, running morph edit <system> <stratum> is a no-op. This is because the version of <system> and <stratum> are now implicit from the context of what we are doing. In other words we're always working with the current version of <system> and <stratum> from the system branch we are in. Because of the complexity of morph's logic, we didn't notice this when dropping the ref: fields, and we missed the opportunity to simplify our logic for 'morph edit' This patch aims to provide the simplest possible implementation of morph edit: morph edit <chunk> It checks all strata for instances of <chunk>, and does what morph edit should do for the instances it finds. A later patch can add warnings to help users deal with situations where <chunk> is not found, or is found more than once. Also since this changes the syntax of morph, it breaks many of our tests. Later patches will address that.
* | Make morph gc clear out deployments in tmpdirPaul Sherwood2014-06-071-1/+6
| | | | | | | | | | | | | | | | Generally deployment temp dirs are removed by morph during deployment, but in some cases deployment dirs may not be cleared up, for example if morph gets a SIGKILL or something unexpected happens that causes morph to terminate without having a chance to cleanup.
* | Rewrite copy_artifacts plugin as list_artifactsSam Thursfield2014-06-052-142/+124
|/ | | | | | | | | | | | | | | | | | | | | | The copy-artifacts and list-artifacts commands are mainly useful for making releases. As part of the release process we copy artifacts for the entire build graph of the release to the artifact cache on trove.baserock.org, to provide Baserock users with ready-built artifacts. This part of the release process is now automated, and the automation require the list-artifacts command to function as a 'plumbing' command. The copy-artifacts command is no longer required. It can be replaced with: morph list-artifacts --quiet REPO REF MORPH | rsync --files-from=- $TARGET The previous version of this plugin looked in the system artifact's metadata for the list of artifacts. This is flawed as the final system does not necessarily contain every build dependency. The new version of the plugin calculates the build graph from source, using the same process as the 'buildcommand' module. It also required looking in Morph's artifact cache for the system artifact file to analyse.
* Add distbuild-morphology command.Sam Thursfield2014-05-141-0/+23
| | | | | | | | | This is useful to build releases using distbuild. It avoids having the SHA1 fields in the artifact metadata files pointing to commits that exist only on temporary build branches. It also avoids file:// URLs in the repo fields. Note that the repo URL still points to the Trove used by the distbuild network, rather than being an upstream URL pointing to git.baserock.org.
* distbuild: Label state machine transition tablesSam Thursfield2014-04-141-0/+1
| | | | Makes it easier to see what they mean at a glance.
* distbuild: Stop workers blowing their entire cache away every timeDaniel Silverstone2014-04-141-6/+14
|
* Merge branch 'baserock/adamcoldrick/git-fat-handle-whitespace-rebase'Adam Coldrick2014-04-031-1/+21
|\ | | | | | | | | | | | | | | Signed-off by: Adam Coldrick Reviewed by: Daniel Silverstone <daniel.silverstone@codethink.co.uk> Richard Maw <richard.maw@codethink.co.uk>
| * Escape special characters and handle whitespace in gitattributesbaserock/adamcoldrick/git-fat-handle-whitespace-rebaseAdam Coldrick2014-04-021-1/+21
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Since git's parsing of .gitattributes can't deal with spaces, or even with quotes, replace any whitespace with [[:space:]]. Also, make sure to escape any glob metacharacters to avoid accidental pattern matches. Note: This approach could cause a file "foo<tab>bar" to be treated as a binary if a file "foo bar" has already been added with morph add-binary. However this is as good of a workaround as I can find. Signed-off by: Adam Coldrick
* | Add utilities for listing and finding extensions.Mark Doffman2014-03-311-43/+16
|/ | | | | | | | | Add a module to morphlib that can list all write and configuration extensions either in morph itself or the morphology repository. The module also contains methods to find an extension filename from the name and type.
* Merge branch 'baserock/richardipsum/merge_distbuild'Richard Ipsum2014-03-273-9/+337
|\ | | | | | | | | | | | | | | | | | | Conflicts: morphlib/plugins/deploy_plugin.py without-test-modules Reviewed by: Richard Maw Lars Wirzenius
| * UI change: distbuild has its own subcommandRichard Ipsum2014-03-261-7/+39
| |
| * Remove Initiator and InitiatorBuildCommandRichard Ipsum2014-03-261-49/+0
| |
| * Fix deployRichard Ipsum2014-03-261-2/+3
| | | | | | | | | | | | | | | | | | tarfile's open needs the file-like object to have a tell() method, objects returned from sockets don't have this method. So instead we fetch the artifact from the remote and cache it locally
| * Make InitiatorBuildCommand construct BuildCommandRichard Ipsum2014-03-261-0/+1
| | | | | | | | | | | | | | The constructor for BuildCommand sets up the caches, we want the caches to be set up for distbuild too so that we can deploy a system (system artifact will be fetched from the artifact cache)
| * Add the distbuild pluginRichard Ipsum2014-03-211-0/+343
| |
* | Implement morph push and morph pulladamcoldrick/ingest-binaries-v5-rebaseAdam Coldrick2014-03-201-0/+93
| | | | | | | | | | | | | | | | | | | | | | Add a plugin to implement both `morph push` and `morph pull`. These commands are wrappers around the corresponding git commands push and pull, which also implement the functionality of pushing and pulling large files provided by git-fat. For example, running `morph pull` will pull any commits from the remote branch not on your local branch, and then pull any large files from the separate git-fat/rsync store on the Trove.
* | Implement morph add-binary using git-fat to store large filesAdam Coldrick2014-03-201-0/+110
| | | | | | | | | | | | | | Add a plugin which implements the morph add-binary command. This command is used to add large files to a git repository. It sets up the files needed to use git-fat, and then runs `git add` with git-fat initiated.
* | Make existing morph commands use git-fatAdam Coldrick2014-03-201-0/+10
| | | | | | | | | | | | When cloning a repository, the files stored using git-fat need to be pulled. This situation occurs in `morph branch`, `morph edit`, and `morph checkout`.
* | deploy: Set status prefix to show which deployment the status is forRichard Maw2014-03-141-53/+69
| |
* | deploy: Make extension output display which it isRichard Maw2014-03-141-1/+2
| | | | | | | | This also makes it obey status prefix
* | Add the ability to do nested deploymentsRichard Maw2014-03-141-6/+18
| |
* | Move deploy logic into multiple methodsRichard Maw2014-03-141-14/+28
| | | | | | | | | | Check is now separate from setup, which is now separate from running the commands.
* | Move tempdir creation out of the loopRichard Maw2014-03-141-21/+27
| | | | | | | | | | | | We don't need to remove the whole thing every time, and for nested deployments, we want to keep the directories around, since there will be multiple deploys happening in it concurrently.
* | Patch buildcommand once, rather that once per systemRichard Maw2014-03-141-3/+5
| |
* | Remove ugly loading of old b&m pluginRichard Maw2014-03-141-10/+0
| |
* | Raise correct error on `morph checkout|branch` of repo with no morphsSam Thursfield2014-03-071-2/+2
| |
* | deploy: Record deployment information in deployed systemSam Thursfield2014-03-041-0/+54
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This commit introduces a new requirement: USERS MUST NOT HAVE SENSITIVE DATA IN THEIR ENVIRONMENT. Otherwise it will be leaked into the system. Note that configuration fields with 'PASSWORD' in their name are stripped before writing the /baserock/deployment.meta file, so the OpenStack OS_PASSWORD field is not leaked. We want this so that we can run hooks at upgrade-time in the future. These hooks might need to know how the system was configured and what releaseuu it was. I'm not quite sure how we will define 'release' yet, but by using `git tag` and `git describe` we are able to textually label a time period in the history of the system's source code. We already have the specific SHA1 of definitions.git stored in the system metadata, so this should give us enough to be able to implement specific hooks that work around any awkward upgrade complications we encounter in the future.
* | deploy: Add optional 'check' extensionsSam Thursfield2014-03-041-1/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | A write extension will have various kinds of sanity checks to do before actually performing the write. The current architecture of 'morph deploy' means that several minutes pass between the user starting the command and the write extension actually executing. It would be rage-inducing watching `morph deploy` spend 3 minutes unpacking a system only to then abort due to a silly error such as forgetting the --upgrade switch. Therefore it's better for now to split the sanity checks out into separate extensions that can be run as soon as possible and abort if the write extension is not going to be able to operate. For now this will just be used to validate usage of the --upgrade flag but in future checking connectivity to remote servers and the like should be done here too.
* | deploy: Add new --upgrade optionSam Thursfield2014-03-041-0/+7
| |
View Lorry Controller Status