summaryrefslogtreecommitdiff
path: root/morphlib/plugins
Commit message (Collapse)AuthorAgeFilesLines
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-1615-40/+25
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Use python3 compatible notation for octal constantsJavier Jardón2015-03-131-3/+3
| | | | Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
* print: use function instead the statement oneJavier Jardón2015-03-131-1/+2
| | | | Change-Id: I7c6f618d5d19e03d906798a6f799d74ac55b0d09
* Merge branch 'sam/distbuild-build-logs'Sam Thursfield2015-03-111-1/+0
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * distbuild: Create a new directory to store build logs for each build.Sam Thursfield2015-02-181-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | It gets messy having hundreds of build-step-xx.log files in the current directory, and if two builds are run in parallel from the same directory the logs for a given chunk will be mixed together in one file. Now, a new directory named build-0, build-1, build-2 etc is created for each new build. If the user passes --initiator-step-output-dir the logs will be placed in that directory, instead. This behaviour is the same as before.
* | Add an LRU cache for resolved tree refsSam Thursfield2015-02-181-1/+2
|/ | | | | | | | | | | | This uses the PyLRU module, from: <https://pypi.python.org/pypi/pylru/1.0.6>. Python 3.2 and newer provide a built-in LRU cache, but this is specifically for in-memory use. See <http://bugs.python.org/issue17528>. Git commits are immutable, so caching information about their contents is fairly easy and trouble-free. There's no danger of the cache becoming stale.
* Fix copyright yearsSam Thursfield2015-02-111-1/+1
|
* distbuild: Refuse to start controller if there are no workers listedSam Thursfield2015-02-111-0/+5
| | | | | | | | This is another situation where builds could hang forever if the server is misconfigured. Longer term, workers should be able to come and go dynamically without needing to reconfigure and restart the controller process.
* Treat path given to deploy cmd as relative to cwdRichard Ipsum2015-01-231-2/+4
|
* Treat path given to build cmd as relative to cwdRichard Ipsum2015-01-231-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently the build commands treat the system argument as a path relative to the root repo. This means that regardless of your working directory you must run morph build systems/foo-system.morph This behaviour can be confusing, for example when your working directory is $systembranch/definitions/systems you might expect to be able to run morph build foo-system.morph especially since most shells would tab-complete the filename for you. At the moment running the above command from $systembranch/definitions/systems would result in an error, because morph would look for $systembranch/definitions/foo-system.morph rather than $systembranch/definitions/systems/foo-system.morph This behaviour also means you can't give the morph build commands an absolute path to a system morph. This patch changes the treatment of the system arg so that it is interpreted relative to the current working directory.
* Merge branch 'sam/optional-temporary-build-branches'Sam Thursfield2015-01-192-26/+87
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
| * Make temporary build branches optionalSam Thursfield2015-01-062-26/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Not everyone is a fan of the `morph build` magic that collects up your changes and puts them in a temporary branch. Now you can disable it by setting 'local-changes=ignore' in your morph.conf file. This speeds up `morph build` and `morph deploy` by 5-10 seconds on my machine. I looked an option to make `morph build` warn if there are uncommitted changes. I found that with a cold cache, it takes about 5 seconds on my machine to verify that there are no uncommitted changes to a checkout of definitions.git. That defeats the main purpose of this patch for me, so I didn't include the option.
* | Merge branch 'sam/cached-repo-cleanup'Sam Thursfield2015-01-122-4/+4
|\ \ | |/ |/| | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-032-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
* | Rename builder2 module to builderSam Thursfield2014-12-191-2/+2
| |
* | deploy: Note that some of the extensions listed live in definitions.gitSam Thursfield2014-12-091-2/+5
| | | | | | | | | | | | | | | | The user should be aware of this because if they aren't building baserock:baserock/definitions or a repo forked from it, those extensions won't be available. Also fix some long lines that I seem to have failed to commit already.
* | Add missing types to the docstring in the DeployPlugin.deploy() functionPete Fotheringham2014-12-051-1/+27
|/
* Remove unicode charactersPedro Alvarez2014-12-011-2/+4
|
* Action review commentsPete Fotheringham2014-12-011-0/+23
| | | | | - Document different ways of calling parameters - Allowed values for boolean parameters
* Fix list-artifacts commandSam Thursfield2014-11-141-1/+1
| | | | | Silly mistake caused this to be broken, and there are no tests for this command inside the morph.git source tree.
* Allow `morph deploy` to cache missing Git repos if it needs toSam Thursfield2014-11-111-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | This reverts commit c63f4e61f69820c71c2d8e9b96ce9bdec0d476a1. This change was introduced back when the Baserock system definitions used named Git refs rather than exact SHA1s. Morph would thus need to update each repo involved in a build and resolve each ref before it could build. Now that the system definitions use SHA1s, the 'updating gits' phase is much less slow. The problem with forcing 'no-git-update' is that it makes `morph deploy` depend on the remote repo cache. In the case that all repos involved in the build are avaiable in the remote repo cache (Trove), there is no need for them to have been cached locally just to construct a build graph. But if the network connection to the Trove is unreliable Morph will need to fall back to the local repo cache. Unless Morph has done a full local build of the system being built, it might not have every repo cached locally. Often, cached artifacts from the Trove or distbuild may be used instead of locally building everything. In this case the user sees `morph deploy` failing with errors such as: Repository upstream:binutils-redhat is not cached yet
* Move create_source_pool code into new 'sourceresolver' moduleSam Thursfield2014-11-101-2/+4
| | | | | | | | This code is an essential part of 'morph build'. It's quite complex and really shouldn't be mixed in with the base Application class. Given a dedicated class we can store some state in the object and avoid functions with seven parameters, too.
* Only search for repository root in GitDirectory constructor if told toSam Thursfield2014-11-052-3/+3
| | | | | | | | | | | | | | | The GitDirectory() constructor, if passed a 'dirname' that doesn't contain a '.git' subdirectory, can search upwards to find the real root of the repository. This is used by the `add-binary`, `push`, and `pull` commands. This causes very confusing behaviour in the case that 'dirname' points to a directory that should be a Git repository, but isn't, and that directory is a path inside the working tree of another Git repository. Rather than raising an error, in this case the GitDirectory class would perform operations on a different repository to the one the caller expected. This 'search_for_root' behaviour is now opt-in, to avoid confusion.
* Fix lines longer than 79 charactersSam Thursfield2014-10-291-2/+2
|
* Merge remote-tracking branch 'origin/sam/build-fix-3'Sam Thursfield2014-10-291-6/+15
|\ | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * Fix distbuild to allow passing a commit instead of a named ref to be builtSam Thursfield2014-10-271-6/+15
|/ | | | | | | | | | | The recent changes to the BuildCommand.build() function caused distbuild to break, because I didn't make the same change to the InitiatorBuildCommand.build() function but did change how it was called. This commit adds the ability to have optional fields in distbuild messages. This is used to add an optional 'original_ref' field, which will get passed to `morph serialise-artifact` by new distbuild controllers, and will be ignored by older ones.
* build: Resolve the ref being built in the user's local definitions.gitSam Thursfield2014-10-242-4/+5
| | | | | | | | | | | | | | | | | | | | | | | | | | Most usefully, this patch means that Morph no longer updates its cached copy of definitions.git every time you run `morph build`. Also, it prevents confusion in the following situation. Imagine I have run: morph checkout baserock:baserock/definitions master I then wait a while, during which time someone pushes to 'master' in the definitions.git repo that I cloned from. Now I run: cd master morph build systems/whatever.morph Which commit does it build, the local head of 'master' or the remote head of 'master'? The answer, both before and after this patch, is that it builds the local version of master. But previously, this only happened because of the magic that we have to detect local changes. With this patch, the local change detection could be disabled and `morph build` would still build what the user had checked out as 'master' locally, not whatever 'master' pointed to in the remote repo.
* build: Restructure how BuildCommand.build() is calledSam Thursfield2014-10-241-3/+5
| | | | | | | | | Rather than take a list of triplets to build, the BuildCommand.build() function now takes a single repo/ref/morph triplet. Iterating through multiple sets of triplets is now done in the build plugin. There are a couple of cosmetic changes to the status output at the start and end of a build as a result.
* Fix synopsis of `morph edit` commandSam Thursfield2014-10-221-1/+1
| | | | | Previously it was incorrect in the `morph help` output and in Morph's man page.
* Fix the `morph list-artifacts` commandSam Thursfield2014-10-101-6/+7
| | | | It was broken by the per-source building changes.
* Allow distbuilds to choose where to put logsRichard Maw2014-10-081-0/+5
|
* Fix issues with distbuild caused by moving to building per-sourceRichard Maw2014-10-082-3/+5
|
* Allow ephemeral ports for distbuild servicesRichard Maw2014-10-021-1/+1
|
* distbuild: allow daemons to bind to ephemeral portsRichard Maw2014-10-011-2/+22
| | | | | | | | | | | | | You can bind to an ephemeral port by passing 0 as the port number. To work out which port you actually got, you need to call getsockname(). To facilitate being able to spawn multiple copies of the daemons for testing environments, you can pass a -file option, which will make the daemon write which port it actually bound to. If this path is a fifo, reading from it in the spawner process will allow synchronisation of only spawning services that require that port to be ready after it is.
* Remove overlap detection logicRichard Maw2014-10-011-32/+0
| | | | | | | I've rarely needed to use it, and on those rare occasions, it would have been easy enough to calculate it. Let's get rid of this step, and save everyone some time in future.
* Remove run-in-artifact subcommandRichard Maw2014-10-011-1/+1
| | | | It's easy enough to deploy the image.
* Merge branch 'baserock/richardmaw-os/tidy-build-logic-v7'Richard Maw2014-09-222-72/+57
|\ | | | | | | | | | | Reviewed-by: Lars Wirzenius (+2 to misc fixups) Reviewed-by: Sam Thursfield (+1 to per-source building) Reviewed-by: Paul Sherwood (+1 to per-source building)
| * Fix show-dependencies plugin to use source deps.Richard Maw2014-09-191-2/+2
| |
| * Build per-source rather than per-artifactRichard Maw2014-09-191-66/+55
| |
| * Remove Morphology.builds_artifactsRichard Maw2014-09-191-4/+0
| | | | | | | | | | This was used before the Artifact splitting code landed to determine which artifacts should be produced.
| * Rename BuildCommand.get_sources to fetch_sourcesRichard Maw2014-09-191-1/+1
| | | | | | | | | | There's other methods called get_sources in other modules, and fetch_sources explains more about what it does in the context.
* | Don't log environment variables with 'PASSWORD' in their name.Sam Thursfield2014-09-191-2/+1
|/ | | | | | | | | | | | | | | | | | | This involved rewriting the util.log_dict_diff() function. It has been renamed to log_environment_changes() to better reflect its purpose. It no longer logs both the old and new values in the event of an environment variable changing. It now just logs the new value. This makes the code simpler and seems like it should not be a big problem. Some projects recommend passing credentials through the environment. OpenStack does this, for example, see: <http://docs.openstack.org/user-guide/content/cli_openrc.html> It's unlikely that users would be happy about applications saving these passwords in log files all over their system. I do not recommend ever storing valuable passwords in the environment.
* Fix `morph branch-from-image` to work for deployed imagesbaserock/richardmaw/bugfix/branch-from-deployed-imageRichard Maw2014-09-111-1/+2
| | | | | | It broke when we added /baserock/deployment.meta. We didn't notice this because our test suite was looking at the artifact produced by morph build, and listed on the terminal.
* deploy: Allow extensions to write to Morph log fileRichard Maw2014-09-011-23/+30
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This is achieved by passing it the write end of a pipe, so that the extension has somewhere to write debug messages without clobbering either stdout or stderr. Previously deployment extensions could only display status messages on stdout, which is good for telling the user what is happening but is not useful when trying to do post-mortem debugging, when more information is usually required. This uses the asyncore asynchronous event framework, which is rather specific to sockets, but can be made to work with file descriptors, and has the advantage that it's part of the python standard library. It relies on polling file descriptors, but there's a trick with pipes to use them as a notification that a process has exited: 1. Ensure the write-end of the pipe is only open in the process you want to know when it exits 2. Make sure the pipe's FDs are set in non-blocking read/write mode 3. Call select or poll on the read-end of the file descriptor 4. If select/poll says you can read from that FD, but you get an EOF, then the process has closed it, and if you know it doesn't do that in normal operation, then the process has terminated. It took a few weird hacks to get the async_chat module to unregister its event handlers on EOF, but the result is an event loop that is asleep until the kernel tells it that it has to do something.
* Add `morph upgrade` command, deprecate `morph deploy --upgrade`Sam Thursfield2014-09-011-4/+26
| | | | | | The arguments to `morph deploy` can get quite long, any way we can make it shorter and clearer is useful. We can also avoid having the strange --no-upgrade flag in future.
* Clarify that multiple images can be deployed at once by `morph deploy`Sam Thursfield2014-09-011-1/+1
|
* morphloader: Get commands when loading morphologyAdam Coldrick2014-08-141-1/+1
| | | | | | Rather than having a `get_commands` method to obtain missing commands from the build system when they are needed, get the commands when loading a morphology.
* show-dependencies: Use sanitise_morphology_path instead of stripping `.morph`Adam Coldrick2014-08-141-2/+2
|
* Merge branch 'baserock/richardmaw/misc-fixups'baserock/richardmaw/tmpRichard Maw2014-08-121-104/+15
|\ | | | | | | | | Reviewed-by: Daniel Silverstone Reviewed-by: Pedro Alvarez
| * Fix `morph edit` when repo has the same ref as system branchRichard Maw2014-08-121-3/+11
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | There was a check in it to see whether it needed to do the git branch and git checkout based on whether the name of the branch matched that in the morphology. This had a couple of problems: 1. Now that we aren't always building from HEAD, we need to be able to roll its commit back, so using the existing branch isn't always the best idea. 2. It only checks the "ref" field, not "unpetrify-ref", so even though we clone the right ref in there, it's checking the commit id against the system branch name, so would always try to re-create the branch, and fail when it already exists. So now, we remove the original ref and re-create it with our preferred HEAD. A better solution might be to change the clone logic to not automatically checkout HEAD, and instead require an explicit branch then checkout, but the initial clone logic is shared with build, and I didn't feel like tracking down all the different places that it was used.
View Lorry Controller Status