summaryrefslogtreecommitdiff
path: root/morphlib/app.py
Commit message (Collapse)AuthorAgeFilesLines
* build: Allow passing in the name of the original ref to create_source_pool()Sam Thursfield2014-10-241-3/+9
| | | | | This means that we can force the building of a specific commit without losing the original branch name in the metadata of the resulting system.
* Merge branch 'baserock/richardmaw-os/tidy-build-logic-v7'Richard Maw2014-09-221-3/+5
|\ | | | | | | | | | | Reviewed-by: Lars Wirzenius (+2 to misc fixups) Reviewed-by: Sam Thursfield (+1 to per-source building) Reviewed-by: Paul Sherwood (+1 to per-source building)
| * Create multiple sources per stratum morphologyRichard Maw2014-09-191-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Building per-artifact results in undesirable behaviour, as multiple artifacts are produced for every chunk build. It therefore makes more sense to build per-source. This implies that actually, the model of one source per morphology is wrong and we should move the dependencies into the source. Unlike chunks however, where every chunk artifact has the same dependencies, stratum artifacts can have different dependencies. So before we can move the dependencies into the Source, we need to have as many Sources as Stratum Artifacts.
* | Prevent cliapp from logging env. variables with 'PASSWORD' in their nameSam Thursfield2014-09-191-0/+4
| | | | | | | | | | The upstream cliapp project is not interested in this functionality right now.
* | Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-191-1/+1
|/ | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* Cut BuildBranch out of morphlib.extensionsRichard Maw2014-08-121-3/+2
| | | | | | | | | | | This was previously used just so it could get the right repo and ref to read the file out of. However, there was a subtle bug in this behaviour, as if we had not previously used morph build in that branch, it would attempt to read the extensions from a branch which didn't exist. So now it reads it from the working tree, which always exists.
* Rewrite traverse_morphs to be able to load chunk morphologies from definitionsRichard Maw2014-07-221-32/+53
| | | | | | | | | | | | | | It will now load the morphology from the definitions repository if the "morph" field is present in the chunk spec. Rather than adapting the loop to fit yet-more changing circumstances, it has been partially rewritten, so there is one loop for loading systems and strata from definitions.git, another for chunks from definitions.git, and a third for chunks in the source repository. This is tidier than attempting to fit the logic in the main loop, as it makes it easier to remove afterwards when we no longer need to load chunk morphologies from the source repository.
* Merge remote-tracking branch ↵Richard Maw2014-07-111-11/+17
|\ | | | | | | | | | | | | 'origin/baserock/richardmaw/S11284/morphologies-by-path-v4' Reviewed-by: Sam Thursfield Reviewed-by: Lars Wirzenius
| * Use exact filenames to refer to morphology filesRichard Maw2014-07-101-11/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
* | Fix Morph failing to update some cached git reposSam Thursfield2014-07-071-11/+2
|/ | | | | | | | | | | | | | | | | | | | | | | I was getting the following error when running the 'do-release.py' script: ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux The commit that it wants did actually exist in git.baserock.org and the logs show that Morph hadn't done a `git remote update` to get the commit locally. This turned out to be because it'd had already looked up a different ref in linux.git. It hadn't needed to run 'git remote update', but it *had* added linux.git to a "don't need to update this repo again" list. Oops! I've moved the code in question to the cachedrepo module so that the repo object keeps that of whether it should be updated. The bug is now fixed. As a side effect this patch fixes the spurious 'Updating repo file:///...' messages, which were printed even though repos with file:/// URLs are never updated.
* Only updated cached git repos when necessarySam Thursfield2014-06-181-3/+11
| | | | | | | | | | When the given ref points to a specific commit, and it's already available in the locally cached copy of repo, there's no need to update the repo. If the ref points to a branch or tag, and the user didn't pass --no-git-update, the locally cached copy of the repo will still be updated. This should speed up many Morph commands.
* Add --build-log-on-stdout flagRichard Ipsum2014-05-141-0/+3
|
* Add write and configuration extensions to help.Mark Doffman2014-03-311-32/+45
| | | | | | Add a command 'help-extensions' to list all extensions. Add the ability to find help on an extension by calling 'morph help [extension name]'.
* Give a useful error when attempting to build a cluster morphSam Thursfield2014-03-121-1/+4
| | | | | | | | | | | | | Previously this resulted in a confusing traceback. This is a quick fix. I'd rather insert the error in the buildcommand module instead, but that code assumes the source being built is a system in several places before it actually checks the kind. Those would all need to be changed, or the code would need to reworked to call _validate_root_kind() much earlier. The Application.traverse_morphs() method is rather ugly anyway, so I'm happy to add further ugliness to it for the time being.
* Make '--help' and help subcommand the same.Mark Doffman2014-03-041-16/+52
| | | | | '--help' when used with a subcommand will show the subcommand help. Do not reflow the help text by using a custom formatter.
* Revert "Make '--help' and help subcommand the same."Mark Doffman2014-02-211-53/+16
| | | | This reverts commit a72c8dca6965d1ac239e4f0102f08fbf7fe59ac7.
* Revert "Add write and configuration extensions to help."Mark Doffman2014-02-211-51/+38
| | | | This reverts commit 329b81419be20e7b1f2651a47030186216044eec.
* Add write and configuration extensions to help.baserock/markdoffman/s10382/add-help-option-v2Mark Doffman2014-02-211-38/+51
| | | | | | | Add a command 'help-extensions' to list all extensions. Add the ability to find help on an extension by calling 'morph help [extension name]'. This will then call the extension with the '--help' option to obtain help text.
* Make '--help' and help subcommand the same.Mark Doffman2014-02-211-16/+53
| | | | | '--help' when used with a subcommand will show the subcommand help. Do not reflow the help text by using a custom formatter.
* Allow omitting repo and ref when referring to strataRichard Maw2014-02-131-5/+5
|
* Add --artifact-cache-server and --git-resolve-cache-serverLars Wirzenius2013-10-291-0/+14
|
* Merge branch 'danielfirth/RT193'Daniel Firth2013-10-111-1/+1
|\ | | | | | | | | Reviewed by: Lars Wirzenius Reviewed by: Pedro Alvarezwq
| * Added chunkname prefix to some elements of the debug log.Dan Firth2013-10-111-1/+1
| |
* | Morph now validates the 'tarball-server' option as a urlDaniel Firth2013-10-101-0/+14
|/
* Allow building with null repo/refRichard Maw2013-09-261-2/+6
|
* Avoid formatting "morph help foo" outputLars Wirzenius2013-09-241-0/+25
| | | | | | | | The cliapp formatting doesn't understand Markdown. Adding Markdown support in Morph is more work than there is time for right now, but we'll do it later. This quick hack just means we output the raw Markdown rather than something that is incomprehensible due to ruined formatting.
* Remove --staging-chroot, --staging-filler optionsLars Wirzenius2013-09-171-25/+0
| | | | | This also removes the long-obsolete code to install staging fillers in the staging area. We've not allowed users to do that for ages now.
* Set build-ref-prefix to be based on trove-id if the setting existsTiago Gomes2013-07-311-3/+9
|
* Don't show git config messages for 'morph branch' in verbose modeRichard Dale2013-06-251-3/+10
|
* Remove SystemKindBuilderFactoryLars Wirzenius2013-06-201-2/+0
| | | | | | We now only support one system-kind, so there's no need for the factory stuff, and at this point it only serves to obfuscate and complexify.
* Shorten create_if_not_exists logicRichard Maw2013-06-061-11/+7
| | | | | | | | I find the loop easier to read since there's less redundancy. Also less code to maintain. Note that tmpdir is not explicitly created, since makedirs of one of its subdirectories will handle that.
* process_args: Move directory creation to after dump configRichard Maw2013-06-061-4/+4
| | | | | | I put my directory creation logic there, since it's an unintended side-effect to create the directories if all you want to do is dump the config.
* Merge branch 'baserock/tiagogomes/tmpdir' of ↵Tiago Gomes2013-06-051-10/+18
|\ | | | | | | | | | | | | | | | | | | | | | | git://git.baserock.org/baserock/baserock/morph I had fixed an conflict and change to use morph_tmp instead of morph as default temp dir. Reviewed by Lars Wirzenius Conflicts: morphlib/app.py
| * Change the structure of the temporary directory used by morphTiago Gomes2013-06-051-5/+16
| | | | | | | | | | | | Now, inside the temporary directory we will have the following subdirectories: chunks, staging, failed and deployments. The failed directory will contain the staging areas of failed builds.
* | S7904: add new config options for minimum sizesRichard Maw2013-06-051-0/+28
| |
* | S7904: Move cache settings to storage optionsRichard Maw2013-06-051-10/+14
| |
* | Create cachedir and tempdir if they do not existRichard Maw2013-06-051-0/+6
|/ | | | | | | | | | | This has been a thorn in my side for many a deployment. A nascent cache directory will cause the size checks to fail. We could get complicated and try the parent directories, or create the directories in the size check, but the former is not worth the complication and the latter is the wrong place to put it, so do it during process_args, so the directories exist before subcommands are run, so they don't have to bother with it themselves.
* Merge branch 'liw/kill-bare-except'Lars Wirzenius2013-05-281-2/+2
|\ | | | | | | | | | | | | Reviewed-by: Richard Maw I, Lars, note that we have an old version of CoverageTestRunner in Baserock. The new version hides the spurious logging messages.
| * Stop using bare except: statementsLars Wirzenius2013-04-161-2/+2
| | | | | | | | | | | | | | | | | | It is almost never a good idea to catch all exceptions, and then do nothing about them. This patch logs all caught exceptions so that the user has some possibilty to debug what is happening. Also, make ./check check for bare excepts and fail the test suite if it finds anything.
* | Remove executable permissions from *.py files that have themLars Wirzenius2013-05-281-0/+0
| |
* | Merge branch 'ps/to-trove-id'Lars Wirzenius2013-05-271-3/+3
|\ \ | |/ |/| | | Changed Paul's patch so that --trove-prefix still works as an alias.
| * rename trove-prefix and trove_prefix to trove_idPaul Sherwood2013-04-131-3/+3
|/
* Fix condition for showing the staging-filler warningLars Wirzenius2013-03-201-1/+1
| | | | | | | If there is no staging-filler setting in a configuration file, it defaults to the empty string, not None. Reviewed-by: Jonathan Maw over the shoulder
* Warn if staging fillers are usedSam Thursfield2013-03-201-1/+5
| | | | | The 'master' branch of baserock:baserock/morphs no longer requires a staging filler.
* Add 'prefix' property for chunks within strataSam Thursfield2013-03-131-5/+0
| | | | | | | | | Morph no longer supports setting the prefix using the --prefix argument / setting. This was only used in tests and during bootstrap. If a chunk build-depends on a chunk within a stratum which has a custom prefix, that prefix is appended to the PATH in the build environment.
* Set environment variables defining target for build-essentialSam Thursfield2013-03-131-18/+0
| | | | | | In the future we will allow this to be modified to provide a cross-bootstrap mode, but for now we use the same target as the host compiler.
* Remove features used by old bootstrap methodSam Thursfield2013-03-131-24/+32
| | | | | | That means that bootstrapping Baserock is currently not possible with this branch of Morph, but there's no reason it cannot be bootstrapped using an older version of Morph instead.
* Change morph to reduce loogign of the environmentRic Holland2013-03-081-2/+3
| | | | | | | | Added function log_dict_diff to identify and record changes in dicts to the debug log This new function was then implemented in app.py to log changes in the environment
* Fix settings['tempdir'] not being setJonathan Maw2013-03-011-0/+5
|
* Make BuildCommand report current and total build stepsLars Wirzenius2013-02-211-1/+7
| | | | | | This adds a [Build 1/12765] to the output of the building of each artifact. This makes it easier to see how much work there might still be remaining.
View Lorry Controller Status