summaryrefslogtreecommitdiff
path: root/morphlib/app.py
Commit message (Collapse)AuthorAgeFilesLines
* Allow the deployment of individual chunks/strata from systemsAdam Coldrick2015-04-301-0/+4
| | | | | | | | | | | | | This commit allows the specification of one or more strata/chunks in a deployment entry in a cluster morphology to deploy instead of the full system if --partial is set. These are listed in a 'partial-deploy-components' field in each deployment definition. The components must be in the system, and this only works for tarball or sysroot deployments. It SHOULD NOT be used when deploying production systems, as it has a number of limitations. Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
* morph help: Expand upon the importance of the first trove-idRichard Maw2015-04-301-1/+3
| | | | Change-Id: I048687305054f08b0d8e01daa1366170b3eec323
* Use --verbose/-v to show build output, --debug/-d for morph debugJavier Jardón2015-04-221-6/+10
| | | | | | | | | | | Its going to be more likely that a baserock user is more interesed in the build log than actually the debug output from morph. And also the intuitive option would be to use -v for this. --verbose/-v: show build output --debug/-d: show morph debug output Change-Id: I1fb99034dc8680a5f168f6306724663aea33ebc5
* More robust parsing of plugin pathsBob Mottram2015-04-141-6/+8
| | | | | | | Previously it was possible to have some strange values for pluginmgr.locations if the same path was appeared twice Change-Id: I3ec257e48e85c3fd30759c3dcc2064f0b151ec45
* Make warning messages more user friendly.Michael Drake2015-04-011-0/+7
| | | | | | | | Printing the file and line number of the warning's origin makes the warning itself harder to spot, and can make it appear as if something has actually gone wrong. Change-Id: Ie77357d9a061d31e234e6e434521d2d274409773
* Reduce the number of 'new environment variable' logs on startupSam Thursfield2015-03-311-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This reduces noise on the console when using `morph build --verbose`. Before (example from my current chroot): $ morph build systems/devel-system-x86_64-generic.morph --verbose 2015-03-31 09:25:08 new environment variable SCHROOT_UID = "0" 2015-03-31 09:25:08 new environment variable SCHROOT_CHROOT_NAME = "baserock-2015-03-24-ostree" 2015-03-31 09:25:08 new environment variable LOGNAME = "root" 2015-03-31 09:25:08 new environment variable XDG_SEAT = "seat0" 2015-03-31 09:25:08 new environment variable PATH = "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" 2015-03-31 09:25:08 new environment variable XDG_VTNR = "1" 2015-03-31 09:25:08 new environment variable HOME = "/root" 2015-03-31 09:25:08 new environment variable SCHROOT_ALIAS_NAME = "baserock-2015-03-24-ostree" 2015-03-31 09:25:08 new environment variable TERM = "xterm" 2015-03-31 09:25:08 new environment variable SHELL = "/bin/bash" 2015-03-31 09:25:08 new environment variable SHLVL = "1" 2015-03-31 09:25:08 new environment variable SCHROOT_USER = "root" 2015-03-31 09:25:08 new environment variable EDITOR = "vim" 2015-03-31 09:25:08 new environment variable SCHROOT_GID = "0" 2015-03-31 09:25:08 new environment variable SCHROOT_GROUP = "root" 2015-03-31 09:25:08 new environment variable GIT_NO_REPLACE_OBJECTS = "1" 2015-03-31 09:25:08 new environment variable SCHROOT_COMMAND = "/usr/bin/env SSH_AUTH_SOCK=/tmp/tmp.eVw95CcpjS/auth-sock /bin/bash -l" 2015-03-31 09:25:08 new environment variable PYTHONPATH = "/src/morph/" 2015-03-31 09:25:08 new environment variable SSH_AUTH_SOCK = "/tmp/tmp.eVw95CcpjS/auth-sock" 2015-03-31 09:25:08 new environment variable SCHROOT_SESSION_ID = "baserock-2015-03-24-ostree-f3c36e20-c3db-4404-8808-e805ca0c1ac4" 2015-03-31 09:25:08 new environment variable MANPAGER = "less -R" 2015-03-31 09:25:08 new environment variable XDG_SESSION_ID = "2" 2015-03-31 09:25:08 new environment variable _ = "/src/morph/morph" 2015-03-31 09:25:08 new environment variable OLDPWD = "/src/morph" 2015-03-31 09:25:08 new environment variable PWD = "/src/ws/master/baserock/baserock/definitions" 2015-03-31 09:25:08 new environment variable USER = "root" 2015-03-31 09:25:08 Starting build 8f80a816dfe44f6c8352a4242b84bf48 2015-03-31 09:25:08 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 09:25:08 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f6137060cd0> 2015-03-31 09:25:15 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph ... After: $ morph build systems/devel-system-x86_64-generic.morph --verbose 2015-03-31 09:23:38 new environment variable GIT_NO_REPLACE_OBJECTS = "1" 2015-03-31 09:23:38 Starting build acfdf9a405384e1ea4e1b62cd1bce27f 2015-03-31 09:23:38 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 09:23:38 Adding uncommitted changes in /src/ws/master/baserock/baserock/definitions to <morphlib.gitdir.GitDirectory object at 0x7f8fcbe86bd0> 2015-03-31 09:23:45 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph ... Change-Id: I20bbdf35b280ccd305bc7e4a70b5f3de7679a368
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Only update Git submodules in cache when necessarySam Thursfield2015-02-101-20/+0
| | | | | | | | | | | | | | This saves a duplicate `git remote update origin` that was being run as part of each chunk build. For any repos that have submodules, it also avoids updating repos if the SHA1 we need to build is already present locally. As well as speeding up builds slightly, this means Morph can now build without being connected to a network, as long as the local Git cache all of the necessary repos and commits in the build, without needing the '--no-git-update' option. The code is also now in a more logical place than before.
* Make temporary build branches optionalSam Thursfield2015-01-061-0/+8
| | | | | | | | | | | | | | | Not everyone is a fan of the `morph build` magic that collects up your changes and puts them in a temporary branch. Now you can disable it by setting 'local-changes=ignore' in your morph.conf file. This speeds up `morph build` and `morph deploy` by 5-10 seconds on my machine. I looked an option to make `morph build` warn if there are uncommitted changes. I found that with a cold cache, it takes about 5 seconds on my machine to verify that there are no uncommitted changes to a checkout of definitions.git. That defeats the main purpose of this patch for me, so I didn't include the option.
* Improve logic for displaying command execution to userSam Thursfield2015-01-021-16/+24
| | | | | | | This avoids writing each command to the log file twice, as we did previously. Also, the user-visible message is now only constructed if we are definitely going to write it to the screen (a tiny optimisation). Hopefully the logic is clearer now too.
* Fix logging and display of commandlines being executed by MorphSam Thursfield2015-01-021-6/+7
| | | | | | | | | | | | | | | | | | | | We were attempting to display commandlines with correct shell-escaping, but the logic was super broken so users would end up seeing this sort of thing instead: 2015-01-02 10:25:42 # g | i | t | | r | e | v | - | p | a | r | s | e | | - | - | v | e | r | i | f | y | | ' | 9 | 8 | f | e | a | 8 | 7 | b | 7 | 2 | 7 | 2 | 5 | 3 | e | 7 | f | f | 8 | 1 | 0 | 5 | 4 | 3 | 4 | c | 9 | e | a | 9 | 0 | 2 | b | b | 6 | a | 6 | f | 7 | e | ^ | { | c | o | m | m | i | t | } | ' Commandlines should now display as intended, more like this: 2015-01-02 10:57:17 # git rev-parse --verify '9df9643842e4b4d8ece710fe6105f32fa38a0d22^{commit}' This broken logic was introduced as a post-review fixup in merge commit c57952ef44a0f1f161441970fcf2f27a39b0de7c.
* Merge branch 'sam/improve-command-failure-errors'Sam Thursfield2014-12-181-11/+16
|\ | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk> Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
| * Tiny optimisation of app.runcmd code pathSam Thursfield2014-12-091-9/+8
| |
| * Give less scary error messages when a containerised command failsSam Thursfield2014-12-091-2/+7
|/ | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This affects errors encountered at build time and at system-integration time. New errors look like this: ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'} Error output: + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted Previously the error message would have been this: Command failed: unshare --mount -- sh -ec. mount --make-rprivate / root="$1" shift while true; do case "$1" in --) shift break ;; *) mount_point="$1" mount_type="$2" mount_source="$3" shift 3 path="$root/$mount_point" mount -t "$mount_type" "$mount_source" "$path" ;; esac done exec "$@" - /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000 + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
* Fix `morph help-extensions` returning no itemsSam Thursfield2014-12-051-1/+1
| | | | | We were passing a nonsensical value for the 'kind' parameter so it would always return an empty list.
* Move create_source_pool code into new 'sourceresolver' moduleSam Thursfield2014-11-101-131/+0
| | | | | | | | This code is an essential part of 'morph build'. It's quite complex and really shouldn't be mixed in with the base Application class. Given a dedicated class we can store some state in the object and avoid functions with seven parameters, too.
* Remove one place where we pass the Application instance aroundSam Thursfield2014-11-061-2/+2
| | | | | | The MorphologyFactory class only uses the status() function of the morphlib.Application instance that it gets passed. So make it require only a status callback.
* build: Allow passing in the name of the original ref to create_source_pool()Sam Thursfield2014-10-241-3/+9
| | | | | This means that we can force the building of a specific commit without losing the original branch name in the metadata of the resulting system.
* Merge branch 'baserock/richardmaw-os/tidy-build-logic-v7'Richard Maw2014-09-221-3/+5
|\ | | | | | | | | | | Reviewed-by: Lars Wirzenius (+2 to misc fixups) Reviewed-by: Sam Thursfield (+1 to per-source building) Reviewed-by: Paul Sherwood (+1 to per-source building)
| * Create multiple sources per stratum morphologyRichard Maw2014-09-191-3/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Building per-artifact results in undesirable behaviour, as multiple artifacts are produced for every chunk build. It therefore makes more sense to build per-source. This implies that actually, the model of one source per morphology is wrong and we should move the dependencies into the source. Unlike chunks however, where every chunk artifact has the same dependencies, stratum artifacts can have different dependencies. So before we can move the dependencies into the Source, we need to have as many Sources as Stratum Artifacts.
* | Prevent cliapp from logging env. variables with 'PASSWORD' in their nameSam Thursfield2014-09-191-0/+4
| | | | | | | | | | The upstream cliapp project is not interested in this functionality right now.
* | Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-191-1/+1
|/ | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* Cut BuildBranch out of morphlib.extensionsRichard Maw2014-08-121-3/+2
| | | | | | | | | | | This was previously used just so it could get the right repo and ref to read the file out of. However, there was a subtle bug in this behaviour, as if we had not previously used morph build in that branch, it would attempt to read the extensions from a branch which didn't exist. So now it reads it from the working tree, which always exists.
* Rewrite traverse_morphs to be able to load chunk morphologies from definitionsRichard Maw2014-07-221-32/+53
| | | | | | | | | | | | | | It will now load the morphology from the definitions repository if the "morph" field is present in the chunk spec. Rather than adapting the loop to fit yet-more changing circumstances, it has been partially rewritten, so there is one loop for loading systems and strata from definitions.git, another for chunks from definitions.git, and a third for chunks in the source repository. This is tidier than attempting to fit the logic in the main loop, as it makes it easier to remove afterwards when we no longer need to load chunk morphologies from the source repository.
* Merge remote-tracking branch ↵Richard Maw2014-07-111-11/+17
|\ | | | | | | | | | | | | 'origin/baserock/richardmaw/S11284/morphologies-by-path-v4' Reviewed-by: Sam Thursfield Reviewed-by: Lars Wirzenius
| * Use exact filenames to refer to morphology filesRichard Maw2014-07-101-11/+17
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
* | Fix Morph failing to update some cached git reposSam Thursfield2014-07-071-11/+2
|/ | | | | | | | | | | | | | | | | | | | | | | I was getting the following error when running the 'do-release.py' script: ERROR:root:Command failed: morph --quiet --trove-host=git.baserock.org list-artifacts baserock:baserock/definitions sam/auto-release minimal-system-x86_32-generic ERROR: Ref d67a0e110187abd560a1de63fa172894a52839d5 is an invalid reference for repo git://git.baserock.org/delta/linux The commit that it wants did actually exist in git.baserock.org and the logs show that Morph hadn't done a `git remote update` to get the commit locally. This turned out to be because it'd had already looked up a different ref in linux.git. It hadn't needed to run 'git remote update', but it *had* added linux.git to a "don't need to update this repo again" list. Oops! I've moved the code in question to the cachedrepo module so that the repo object keeps that of whether it should be updated. The bug is now fixed. As a side effect this patch fixes the spurious 'Updating repo file:///...' messages, which were printed even though repos with file:/// URLs are never updated.
* Only updated cached git repos when necessarySam Thursfield2014-06-181-3/+11
| | | | | | | | | | When the given ref points to a specific commit, and it's already available in the locally cached copy of repo, there's no need to update the repo. If the ref points to a branch or tag, and the user didn't pass --no-git-update, the locally cached copy of the repo will still be updated. This should speed up many Morph commands.
* Add --build-log-on-stdout flagRichard Ipsum2014-05-141-0/+3
|
* Add write and configuration extensions to help.Mark Doffman2014-03-311-32/+45
| | | | | | Add a command 'help-extensions' to list all extensions. Add the ability to find help on an extension by calling 'morph help [extension name]'.
* Give a useful error when attempting to build a cluster morphSam Thursfield2014-03-121-1/+4
| | | | | | | | | | | | | Previously this resulted in a confusing traceback. This is a quick fix. I'd rather insert the error in the buildcommand module instead, but that code assumes the source being built is a system in several places before it actually checks the kind. Those would all need to be changed, or the code would need to reworked to call _validate_root_kind() much earlier. The Application.traverse_morphs() method is rather ugly anyway, so I'm happy to add further ugliness to it for the time being.
* Make '--help' and help subcommand the same.Mark Doffman2014-03-041-16/+52
| | | | | '--help' when used with a subcommand will show the subcommand help. Do not reflow the help text by using a custom formatter.
* Revert "Make '--help' and help subcommand the same."Mark Doffman2014-02-211-53/+16
| | | | This reverts commit a72c8dca6965d1ac239e4f0102f08fbf7fe59ac7.
* Revert "Add write and configuration extensions to help."Mark Doffman2014-02-211-51/+38
| | | | This reverts commit 329b81419be20e7b1f2651a47030186216044eec.
* Add write and configuration extensions to help.baserock/markdoffman/s10382/add-help-option-v2Mark Doffman2014-02-211-38/+51
| | | | | | | Add a command 'help-extensions' to list all extensions. Add the ability to find help on an extension by calling 'morph help [extension name]'. This will then call the extension with the '--help' option to obtain help text.
* Make '--help' and help subcommand the same.Mark Doffman2014-02-211-16/+53
| | | | | '--help' when used with a subcommand will show the subcommand help. Do not reflow the help text by using a custom formatter.
* Allow omitting repo and ref when referring to strataRichard Maw2014-02-131-5/+5
|
* Add --artifact-cache-server and --git-resolve-cache-serverLars Wirzenius2013-10-291-0/+14
|
* Merge branch 'danielfirth/RT193'Daniel Firth2013-10-111-1/+1
|\ | | | | | | | | Reviewed by: Lars Wirzenius Reviewed by: Pedro Alvarezwq
| * Added chunkname prefix to some elements of the debug log.Dan Firth2013-10-111-1/+1
| |
* | Morph now validates the 'tarball-server' option as a urlDaniel Firth2013-10-101-0/+14
|/
* Allow building with null repo/refRichard Maw2013-09-261-2/+6
|
* Avoid formatting "morph help foo" outputLars Wirzenius2013-09-241-0/+25
| | | | | | | | The cliapp formatting doesn't understand Markdown. Adding Markdown support in Morph is more work than there is time for right now, but we'll do it later. This quick hack just means we output the raw Markdown rather than something that is incomprehensible due to ruined formatting.
* Remove --staging-chroot, --staging-filler optionsLars Wirzenius2013-09-171-25/+0
| | | | | This also removes the long-obsolete code to install staging fillers in the staging area. We've not allowed users to do that for ages now.
* Set build-ref-prefix to be based on trove-id if the setting existsTiago Gomes2013-07-311-3/+9
|
* Don't show git config messages for 'morph branch' in verbose modeRichard Dale2013-06-251-3/+10
|
* Remove SystemKindBuilderFactoryLars Wirzenius2013-06-201-2/+0
| | | | | | We now only support one system-kind, so there's no need for the factory stuff, and at this point it only serves to obfuscate and complexify.
* Shorten create_if_not_exists logicRichard Maw2013-06-061-11/+7
| | | | | | | | I find the loop easier to read since there's less redundancy. Also less code to maintain. Note that tmpdir is not explicitly created, since makedirs of one of its subdirectories will handle that.
* process_args: Move directory creation to after dump configRichard Maw2013-06-061-4/+4
| | | | | | I put my directory creation logic there, since it's an unintended side-effect to create the directories if all you want to do is dump the config.
* Merge branch 'baserock/tiagogomes/tmpdir' of ↵Tiago Gomes2013-06-051-10/+18
|\ | | | | | | | | | | | | | | | | | | | | | | git://git.baserock.org/baserock/baserock/morph I had fixed an conflict and change to use morph_tmp instead of morph as default temp dir. Reviewed by Lars Wirzenius Conflicts: morphlib/app.py
View Lorry Controller Status