summaryrefslogtreecommitdiff
path: root/morphlib/plugins/deploy_plugin.py
Commit message (Collapse)AuthorAgeFilesLines
* deploy: Fix an unlikely crash at deploy-timeSam Thursfield2016-02-191-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | For some reason I had a deployment .morph file that specified: deploy-defaults: This resulted in the deploy_defaults variable having value None, which lead to a crash further down: 2016-02-19 12:03:52 Deciding on task order Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 574, in upgrade self.deploy(args) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 370, in deploy env_vars, deployments) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 403, in deploy_cluster parent_location='') File "/src/morph/morphlib/plugins/deploy_plugin.py", line 462, in deploy_system env_vars, deployment_filter, parent_location) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 483, in deploy_system_with_source_pool system_id, env_vars, deploy_defaults, deploy_params) File "/src/morph/morphlib/plugins/deploy_plugin.py", line 59, in configuration_for_system deploy_params.items() + AttributeError: 'NoneType' object has no attribute 'items' This change makes the code work as expected. Change-Id: I667f4142667be31797cac7c8994d35a404119cca
* Prepare the removal of workspace and system branches codeTiago Gomes2015-11-251-1/+1
| | | | Change-Id: Id470c7a77a47c89118a5d9d0d23b2206d8a839e4
* Determine whether to include_local_changes from app.settingsSam Thursfield2015-06-221-3/+1
| | | | | | | Previously callers to DefinitionsRepoWithApp.source_pool() could override the user's value, but that's not actually used anywhere. Change-Id: I43b53e7b9fc937886c8e6e95947e5e2b6776d085
* Use DefinitionsRepo class in deploy pluginSam Thursfield2015-06-161-74/+64
| | | | | | | | | | This allows you to run `morph deploy` from any Git checkout of definitions.git. This also changes `morph help-extensions` to use the DefinitionsRepo class, as the code overlaps. Change-Id: I64c48f59c8ec5aebc7169f3b4b4abbb759bd0b9a
* Set PYTHONPATH when running deployment extensionsAdam Coldrick2015-06-121-0/+8
| | | | | | | | | | | | | After moving deployment extensions into the definitions repository, but they need access to shared code, so deployment extensions need to import code which lives in definitions. However morph runs a copy of them in a temporary directory. Version 5 will allow deployment extensions to depend only on code which is also in definitions and the Python standard library, though they may also include code from elsewhere as described in PYTHONPATH. Change-Id: I998f01830656a5118bba1e579f649ec98f5f82c6
* morphlib: Use new temp_dir helper context managerRichard Maw2015-06-051-20/+7
| | | | Change-Id: Ie4e024a63f2ab1c7ea66f2cbedaef99c9adf5e69
* deploy: Don't imply --upgrade in subsystemsRichard Maw2015-05-271-2/+5
| | | | | | | | | | | --upgrade being passed on to subsystem deployments broke attempting to perfom an ssh-rsync upgrade with a system that had a sysroot in it. Therefore it is appropriate to prevent --upgrade being inherited like this, as it is seldom what is wanted, and if needed, it can be set in the cluster, or on the command-line separately. Change-Id: I59d261ded26df2808ee83cb862bdc247542d1327
* morph deploy: Allow overriding UPGRADE in definition or on command lineRichard Maw2015-05-271-4/+11
| | | | | | | | | | | | | | | | | | | Concievably, you may want to force a cluster to behave as an upgrade regardless of whether --upgrade is set or not. This would allow two use cases that are not currently possible: 1. To force a cluster to be upgrade-only, which would make our instructions for the upgrade-devel.morph cluster simpler. 2. To allow a cluster to deploy then immediately upgrade. One potential use for this would be to create a disk image with multiple versions on it. This also works as a work-around for --upgrade implying UPGRADE=yes for subsystem deployments, but that should be fixed separately. Change-Id: I2ec4b880fc9bce37adee9df67696f088c76650f4
* Ignore --upgrade flag rather than exitingRichard Ipsum2015-05-271-4/+0
| | | | | | | | Specifying --upgrade with the upgrade command is redundant, but it's not a contradiction, so we can safely ignore --upgrade if it happens to be specified with the upgrade command. Change-Id: I94213f3e45ce58a3c91f7be0ccca63b0eec9561c
* Give exact filename when an artifact is not found in the cache.Sam Thursfield2015-05-191-2/+3
| | | | | | | | | | | | | | | | | | | | | Old message: ERROR: Deployment failed as mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. New message: ERROR: Deployment failed as 45b8053511473de1220e894f589428c94c7d1240d11c0154a5c368bb7b7dedc2.system.mason-system-x86_64-generic-rootfs is not present in the artifact cache. Please ensure that mason-system-x86_64-generic-rootfs is built before deployment, and the artifact-cache-server (http://cache.baserock.org:8080/) is the correct one. Change-Id: Idd9e92f1c842d428795886a3a7c0026cc42f54b1
* Fix prototype of create_source_pool()Sam Thursfield2015-05-141-1/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This fixes an issue in `morph list-artifacts` and probably other commands: ERROR: Command failed: morph list-artifacts --quiet file:///ws/master/git.baserock.org/baserock/baserock/definitions HEAD systems/build-system-armv7lhf-jetson.morph systems/weston-system-armv7lhf-jetson.morph systems/genivi-baseline-system-armv7lhf-jetson.morph Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/usr/lib/python2.7/site-packages/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 67, in list_artifacts repo, ref, system_filename) File "/usr/lib/python2.7/site-packages/morphlib/plugins/list_artifacts_plugin.py", line 91, in list_artifacts_for_system status_cb=self.app.status) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 641, in create_source_pool definitions_original_ref=original_ref) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 583, in traverse_morphs definitions_absref, definitions_tree, visit) File "/usr/lib/python2.7/site-packages/morphlib/sourceresolver.py", line 408, in _process_definitions_with_children definitions_queue = collections.deque(system_filenames) TypeError: 'NoneType' object is not iterable The issue is that the sourceresolver.create_source_pool() function did not work correctly if passed a single filename. This was a regression from commit 4cc75039a78bd8aef9ef464bc0eb6c3ff16809d0. In order to have a logical function prototype, the create_source_pool() function now only takes a list of systems, instead of taking either a single system or a list. Change-Id: I8f5ec8859a7e26d8107ed1c268fe925818d080a5
* Move duplicate fix_chunk_build_mode function to a common locationAdam Coldrick2015-05-121-26/+2
| | | | Change-Id: I11b4dbeb50d67068701f269ef6ac7cfbd89f6aed
* Allow the deployment of individual chunks/strata from systemsAdam Coldrick2015-04-301-40/+215
| | | | | | | | | | | | | This commit allows the specification of one or more strata/chunks in a deployment entry in a cluster morphology to deploy instead of the full system if --partial is set. These are listed in a 'partial-deploy-components' field in each deployment definition. The components must be in the system, and this only works for tarball or sysroot deployments. It SHOULD NOT be used when deploying production systems, as it has a number of limitations. Change-Id: I04ac58af57216335d9257f6620d09f18f61ea714
* Improve 'Deploy failed as system is not yet build' error.Sam Thursfield2015-04-301-4/+5
| | | | | | | | | | If a user had the wrong setting for artifact-cache-server then they could build a system with 'morph distbuild', then be told it wasn't built by 'morph deploy'. The logs do not show any info about which cache server was used, so it looks like a Morph bug. This change should make such a configuration problem much easier to spot. Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
* deploy: Clean up `morph deploy` documentation a bitSam Thursfield2015-04-211-133/+30
| | | | | | | | | | | | Remove documentation specific to certain .configure and .write extensions, and point the user to `morph help-extensions` and `morph help xxx.write` instead. Don't mention the deprecated nfsboot.write extension. Other small cleanups. Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
* deploy: Deploy and upgrade systems from the same 'cluster' definitionSam Thursfield2015-04-211-17/+83
| | | | | | | | | | | | | | | | | | | This patch adds two fields to deployment (cluster) .morph files: 'upgrade-type' and 'upgrade-location'. The `morph deploy` command ignores these. The `morph upgrade` command will honour them if present, instead of the existing 'type' and 'location' fields. If they are not present, `morph upgrade` will give a warning, and will use the 'type' and 'location' fields as before. This avoids the need to edit the deployment .morph file after deploying a system. Small detail: the 'type' and 'location' variables are no longer removed from the environment that is passed to the .configure and .write extensions. This shouldn't affect anything. Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
* deploy: Factor out a bit of code into its own functionSam Thursfield2015-04-211-10/+37
| | | | | | | This behaviour is complex, it shouldn't be hidden away in a larger function. Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
* % escape all % charactersRichard Ipsum2015-03-271-1/+1
|
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Treat path given to deploy cmd as relative to cwdRichard Ipsum2015-01-231-2/+4
|
* Make temporary build branches optionalSam Thursfield2015-01-061-18/+33
| | | | | | | | | | | | | | | Not everyone is a fan of the `morph build` magic that collects up your changes and puts them in a temporary branch. Now you can disable it by setting 'local-changes=ignore' in your morph.conf file. This speeds up `morph build` and `morph deploy` by 5-10 seconds on my machine. I looked an option to make `morph build` warn if there are uncommitted changes. I found that with a cold cache, it takes about 5 seconds on my machine to verify that there are no uncommitted changes to a checkout of definitions.git. That defeats the main purpose of this patch for me, so I didn't include the option.
* deploy: Note that some of the extensions listed live in definitions.gitSam Thursfield2014-12-091-2/+5
| | | | | | | | The user should be aware of this because if they aren't building baserock:baserock/definitions or a repo forked from it, those extensions won't be available. Also fix some long lines that I seem to have failed to commit already.
* Add missing types to the docstring in the DeployPlugin.deploy() functionPete Fotheringham2014-12-051-1/+27
|
* Remove unicode charactersPedro Alvarez2014-12-011-2/+4
|
* Action review commentsPete Fotheringham2014-12-011-0/+23
| | | | | - Document different ways of calling parameters - Allowed values for boolean parameters
* Allow `morph deploy` to cache missing Git repos if it needs toSam Thursfield2014-11-111-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | This reverts commit c63f4e61f69820c71c2d8e9b96ce9bdec0d476a1. This change was introduced back when the Baserock system definitions used named Git refs rather than exact SHA1s. Morph would thus need to update each repo involved in a build and resolve each ref before it could build. Now that the system definitions use SHA1s, the 'updating gits' phase is much less slow. The problem with forcing 'no-git-update' is that it makes `morph deploy` depend on the remote repo cache. In the case that all repos involved in the build are avaiable in the remote repo cache (Trove), there is no need for them to have been cached locally just to construct a build graph. But if the network connection to the Trove is unreliable Morph will need to fall back to the local repo cache. Unless Morph has done a full local build of the system being built, it might not have every repo cached locally. Often, cached artifacts from the Trove or distbuild may be used instead of locally building everything. In this case the user sees `morph deploy` failing with errors such as: Repository upstream:binutils-redhat is not cached yet
* build: Resolve the ref being built in the user's local definitions.gitSam Thursfield2014-10-241-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | Most usefully, this patch means that Morph no longer updates its cached copy of definitions.git every time you run `morph build`. Also, it prevents confusion in the following situation. Imagine I have run: morph checkout baserock:baserock/definitions master I then wait a while, during which time someone pushes to 'master' in the definitions.git repo that I cloned from. Now I run: cd master morph build systems/whatever.morph Which commit does it build, the local head of 'master' or the remote head of 'master'? The answer, both before and after this patch, is that it builds the local version of master. But previously, this only happened because of the magic that we have to detect local changes. With this patch, the local change detection could be disabled and `morph build` would still build what the user had checked out as 'master' locally, not whatever 'master' pointed to in the remote repo.
* Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-191-2/+1
| | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* deploy: Allow extensions to write to Morph log fileRichard Maw2014-09-011-23/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is achieved by passing it the write end of a pipe, so that the extension has somewhere to write debug messages without clobbering either stdout or stderr. Previously deployment extensions could only display status messages on stdout, which is good for telling the user what is happening but is not useful when trying to do post-mortem debugging, when more information is usually required. This uses the asyncore asynchronous event framework, which is rather specific to sockets, but can be made to work with file descriptors, and has the advantage that it's part of the python standard library. It relies on polling file descriptors, but there's a trick with pipes to use them as a notification that a process has exited: 1. Ensure the write-end of the pipe is only open in the process you want to know when it exits 2. Make sure the pipe's FDs are set in non-blocking read/write mode 3. Call select or poll on the read-end of the file descriptor 4. If select/poll says you can read from that FD, but you get an EOF, then the process has closed it, and if you know it doesn't do that in normal operation, then the process has terminated. It took a few weird hacks to get the async_chat module to unregister its event handlers on EOF, but the result is an event loop that is asleep until the kernel tells it that it has to do something.
* Add `morph upgrade` command, deprecate `morph deploy --upgrade`Sam Thursfield2014-09-011-4/+26
| | | | | | The arguments to `morph deploy` can get quite long, any way we can make it shorter and clearer is useful. We can also avoid having the strange --no-upgrade flag in future.
* Clarify that multiple images can be deployed at once by `morph deploy`Sam Thursfield2014-09-011-1/+1
|
* Avoid creating and pushing temporary build branches when they aren't necessary.Richard Maw2014-08-121-32/+9
| | | | | | | | | | | | | | | | | | | | | Sorry about the big lump, I can split it into a nicer set of changes, but they didn't naturally emerge in a nice series. This creates a pushed_build_branch context manager, to eliminate code duplication between build and deploy. Rather than the build branch being constructed knowing whether it needs to push the branch, it infers that from the state of the repositories, and whether a local build would be possible. If there are no uncommitted changes and all local branches are pushed, then it doesn't create temporary branches or push them, and instead uses what it already has. It will currently create and use temporary build branches even for chunks that have no local changes, but it's pretty cheap, and doesn't require re-working the build-ref injection code to check whether there are local changes.
* Cut BuildBranch out of morphlib.extensionsRichard Maw2014-08-121-2/+1
| | | | | | | | | | | This was previously used just so it could get the right repo and ref to read the file out of. However, there was a subtle bug in this behaviour, as if we had not previously used morph build in that branch, it would attempt to read the extensions from a branch which didn't exist. So now it reads it from the working tree, which always exists.
* Turn BuildBranch methods into regular functionsRichard Maw2014-08-121-5/+9
| | | | | | | | | | | | | | | | | | | | | | | | Previously they were generator functions, which yielded interesting context at interesting times so that the caller could respond by printing status messages. The only benefits this had over callbacks were: 1. 1 fewer function scope to worry about. I don't have data on the amount of memory used for a function scope vs a generator, but it could be less cognitive load for determining which variables are defined in the callback's body. 2. It is possible to yield in the caller, so you could make that into a coroutine too, however this wasn't required in this case, as the yielded value was intended to be informational. The downsides to this are: 1. It's a rather peculiar construct, so can be difficult to understand what's going on, and the implications, which led to 2. If you call the function, but don't use the iterator it returned, then it won't do anything, which is very confusing indeed, if you're not used to how generator functions work.
* Tidy deploy_plugin a littleRichard Maw2014-07-221-5/+5
|
* Make our use of json binary path safebaserock/richardmaw/bugfix/unicode-safe-jsonRichard Maw2014-07-111-1/+2
| | | | | | | | | | | | | | | | | | | | | json only accepts unicode. Various APIs such as file paths and environment variables allow binary data, so we need to support this properly. This patch changes every[1] use of json.load or json.dump to escape non-unicode data strings. This appears exactly as it used to if the input was valid unicode, if it isn't it will insert \xabcd escapes in the place of non-unicode data. When loading back in, if json.load is told to unescape it with `encoding='unicode-escape'` then it will convert it back correctly. This change was primarily to support file paths that weren't valid unicode, where this would choke and die. Now it works, but any tools that parsed the metadata need to unescape the paths. [1]: The interface to the remote repo cache uses json data, but I haven't changes its json.load calls to unescape the data, since the repo caches haven't been made to escape the data.
* Use exact filenames to refer to morphology filesRichard Maw2014-07-101-6/+4
| | | | | | | | | | | | | | | | | | | Rather than repeatedly stripping and appending an optional .morph extension morphology names, instead always use the file path of the morphology relative to the definitions repository. This is an inversion of the previous logic, which would strip the .morph extension and use the "name" internally. The exception to this rule of always using the filename, is that `morph edit CHUNK` uses the name of the morphology as-defined in the stratum. This is based off Adam Coldrick's inital patch, but this version will allow the old style of providing the "name" by converting it into a path if it does not have either a / or a . in it. An unfortunate consequence of this change is that the show-dependencies command's output changed, so the test needed updating.
* Make MorphologyFinder use file pathsRichard Maw2014-07-071-3/+2
| | | | | | | | | | | | | | | | | | | | We want to use file paths to locate morphologies now, so the old model of get a list of names and hand it those back to get the contents doesn't really make sense any more. This abstraction initially came about as one idea I had for moving morphologies out of the source tree was to put them in something like git notes, where it's possible to look up information for one commit in another ref in the repository, at which point this abstraction would have been flexible enough to handle that as well as in the However, moving the chunk morphologies into the definitions repository has other benefits too, so it makes more sense to be honest about using filenames in the API. It remains as a single point where we can put the logic for knowing which files in a repository look like morphologies, but if we need to remove any further functionality, it should be replaced by a single function.
* Update the help text for morph deployAdam Coldrick2014-06-181-1/+5
|
* Allow the user to specify deployments in a clusterAdam Coldrick2014-06-181-8/+43
| | | | | | | | Instead of taking the name of a cluster morphology and zero or more parameters for overriding the cluster morphology, morph deploy should take the name of a cluster morphology and the names of zero or more system deployments that are defined in the cluster morphology. If no deployment names are given then all deployments are deployed.
* Morph deploy should never update git reposPaul Sherwood2014-06-101-0/+1
| | | | | | | | The deploy plugin is relying on existing code which respects the 'no-git-update' setting and updates gits if it is not set. Since deploy can only work for built systems anyway, we can force this True for deployments, to avoid wasting the user/caller's time.
* Add utilities for listing and finding extensions.Mark Doffman2014-03-311-43/+16
| | | | | | | | | Add a module to morphlib that can list all write and configuration extensions either in morph itself or the morphology repository. The module also contains methods to find an extension filename from the name and type.
* Merge branch 'baserock/richardipsum/merge_distbuild'Richard Ipsum2014-03-271-2/+3
|\ | | | | | | | | | | | | | | | | | | Conflicts: morphlib/plugins/deploy_plugin.py without-test-modules Reviewed by: Richard Maw Lars Wirzenius
| * Fix deployRichard Ipsum2014-03-261-2/+3
| | | | | | | | | | | | | | | | | | tarfile's open needs the file-like object to have a tell() method, objects returned from sockets don't have this method. So instead we fetch the artifact from the remote and cache it locally
* | deploy: Set status prefix to show which deployment the status is forRichard Maw2014-03-141-53/+69
| |
* | deploy: Make extension output display which it isRichard Maw2014-03-141-1/+2
| | | | | | | | This also makes it obey status prefix
* | Add the ability to do nested deploymentsRichard Maw2014-03-141-6/+18
| |
* | Move deploy logic into multiple methodsRichard Maw2014-03-141-14/+28
| | | | | | | | | | Check is now separate from setup, which is now separate from running the commands.
* | Move tempdir creation out of the loopRichard Maw2014-03-141-21/+27
| | | | | | | | | | | | We don't need to remove the whole thing every time, and for nested deployments, we want to keep the directories around, since there will be multiple deploys happening in it concurrently.
* | Patch buildcommand once, rather that once per systemRichard Maw2014-03-141-3/+5
| |
View Lorry Controller Status