summaryrefslogtreecommitdiff
path: root/morphlib/plugins
Commit message (Collapse)AuthorAgeFilesLines
...
* morphlib: Add command to get build-log for a given chunk and system fileLauren Perry2015-04-301-0/+90
| | | | Change-Id: I09e9b17ef2e0fb94dbf5a96dca91062d64433add
* Add a CSV manifest generator.Adam Coldrick2015-04-302-2/+338
| | | | | | | | | | | | | | | | This adds a `morph generate-manifest-csv` command which creates a CSV manifest for a system containing license information, upstream URL and a guess at the version of each component in a system. The existing `morph generate-manifest` is also renamed to `morph generate-manifest-genivi` to improve the distinction between the two commands which generate a manifest. The basis of this commit was done by Michael Drake <michael.drake@codethink.co.uk>. Change-Id: I712de11eaa6e1564c5bae5f5f396e5e77484972f
* Check number of args given to build commandsRichard Ipsum2015-04-301-5/+38
| | | | Change-Id: I501632c4d7acde06e391df3752275a3afc334ba6
* Improve 'Deploy failed as system is not yet build' error.Sam Thursfield2015-04-301-4/+5
| | | | | | | | | | If a user had the wrong setting for artifact-cache-server then they could build a system with 'morph distbuild', then be told it wasn't built by 'morph deploy'. The logs do not show any info about which cache server was used, so it looks like a Morph bug. This change should make such a configuration problem much easier to spot. Change-Id: I932ca3c5caa46990f9bcd77d75b3862f49e0dfb8
* distbuild: Add distbuild status commandLauren Perry2015-04-291-0/+47
| | | | | | | | | Adds a command to get the status of all recently ran distbuilds for a given server (e.g. Running, Finished, Failed, Cancelled), so as to tell if a build running via distbuild-start has finished or otherwise exited without going through the server's log files Change-Id: I5ce9fe54ae7b1bd8fe3e0d629f615042be8827ed
* distbuild: Add distbuild start and cancel functionalityLauren Perry2015-04-292-1/+72
| | | | | | | | | | | Add command for distbuild-start to build_plugin in morphlib, and create a boolean parameter to inform the initiator whether to disconnect the controller and leave the build running remotely. Add distbuild-cancel command to parse currently-running distbuild build-request IDs and cancel the one matching the given argument Change-Id: I458a5767bb768ceb2b4d8876adf1c86075d452bd
* get-repo: Delete partially cloned repo on errorsSam Thursfield2015-04-281-2/+9
| | | | | | | | | If a `morph get-repo` command hits an error or is interrupted while the 'git clone' operation is in progress, the partial clone is left around. When the user tries to run the command again, they hit an error saying the directory already exists. Change-Id: Id4bb57f2136049ede0ea900e7e359255617427ec
* Add a `morph get-repo` command to clone repositories from the cacheAdam Coldrick2015-04-271-0/+149
| | | | Change-Id: I0c175649bd98067ea69d471ebe6880aac5a25598
* Fix regression that caused original_ref to be lost in some buildsSam Thursfield2015-04-271-2/+3
| | | | | | | | | | Commit bd788c7219d8b ("Implement partial builds") caused the original_ref metadata to be lost in some cases. This meant that the metadata for built systems would have original_ref set to a SHA1 hex string, instead of the original named ref that the system was built from. Change-Id: Id31654ad60b411206ccc7ee70b46f005baa209ef
* morphlib: Add example use to morph certify and fix spelling errorLauren Perry2015-04-271-1/+6
| | | | Change-Id: I3c4b4f687703ab665841a3527044d0f669135513
* Calculate the build graph in worker-buildAdam Coldrick2015-04-241-5/+25
| | | | | | | Calculate the build graph and find the artifact which is referred to by the ArtifactReference which was deserialised. Change-Id: Icbef90932979fc1a18daf0dedac16f4429f56349
* distbuild: Don't check if there is free disk space on the initiatorSam Thursfield2015-04-221-6/+7
| | | | | | | This check is intended for local builds, not distbuilds. Much less disk space is needed to distbuild. Change-Id: Iaff77805119129cbf6584de418ef97c7f1b82ad8
* deploy: Clean up `morph deploy` documentation a bitSam Thursfield2015-04-211-133/+30
| | | | | | | | | | | | Remove documentation specific to certain .configure and .write extensions, and point the user to `morph help-extensions` and `morph help xxx.write` instead. Don't mention the deprecated nfsboot.write extension. Other small cleanups. Change-Id: Ie7ed030c71ade4876aafb3d8a43e864a28ddc279
* deploy: Deploy and upgrade systems from the same 'cluster' definitionSam Thursfield2015-04-211-17/+83
| | | | | | | | | | | | | | | | | | | This patch adds two fields to deployment (cluster) .morph files: 'upgrade-type' and 'upgrade-location'. The `morph deploy` command ignores these. The `morph upgrade` command will honour them if present, instead of the existing 'type' and 'location' fields. If they are not present, `morph upgrade` will give a warning, and will use the 'type' and 'location' fields as before. This avoids the need to edit the deployment .morph file after deploying a system. Small detail: the 'type' and 'location' variables are no longer removed from the environment that is passed to the .configure and .write extensions. This shouldn't affect anything. Change-Id: Id2a4e4f229b8adebdb57eded2049ac113a82a4be
* deploy: Factor out a bit of code into its own functionSam Thursfield2015-04-211-10/+37
| | | | | | | This behaviour is complex, it shouldn't be hidden away in a larger function. Change-Id: I953c9477e0210d395b97f5f8219eaebe4dbcd272
* distbuild: Add distbuild-list-jobs functionLauren Perry2015-04-171-0/+42
| | | | | | | | | | Add InitiatorListJobs class and list-jobs message template, add distbuild-list-jobs to morph commandlist, send running job information back to initiator, split out handling of build request and list-jobs messages to separate functions and change generating a random integer to UUID for message identification Change-Id: Id02604f2c1201dbc10f6bbd7f501b8ce1ce0deae
* Implement partial distbuildsAdam Coldrick2015-04-021-4/+20
| | | | | | | | | | | | | | In addition to partial builds we also want to be able to do partial distbuilds, and distbuild uses a different codepath. This commit updates the distbuild code to know what to do if a partial build is requested. It only builds up to the latest chunk/stratum that was requested, and displays where to find the artifacts for each of the chunks/strata requested upon completion of the build. The usage is the same as for local builds. Change-Id: I0537f74e2e65c7aefe5e71795f17999e2415fce5
* Add `morph certify` plugin to check for build reproducibility.Michael Drake2015-04-011-0/+140
| | | | | | | | | | | Currently does three checks: 1. Checks that all chunks in given system(s) have sha1 refs. 2. Checks that all sha1 refs are anchored. Unanchored refs can be removed on `git gc`. 3. Checks that all chunk repos are on the trove-host. Change-Id: Iaf105b1614a45616684e68a08f28b8529d4321fa
* Implement partial buildsAdam Coldrick2015-04-011-20/+88
| | | | | | | | | | | | | | | | | | This commit allows you to build only a single chunk or stratum and its dependencies, instead of a whole system. This will be useful for developers who want to quickly check the build commands of a chunk for example. You can give more than one chunk or stratum to the command, and the build will run up to the latest one, building the dependencies of both. An example of the usage: morph build systems/devel-system-x86_64-generic.morph \ strata/build-essential.morph strata/core/curl.morph This will only build up to curl in the core stratum. Change-Id: Id55b8335fc61e8fdf847f5e383cfedd13d07e6d2
* build: Clearer messages when creating temporary build branchesSam Thursfield2015-03-311-1/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | The temporary build branch (--local-changes=include) feature can be convenient, but it is also slow, and causes the resulting system to be 'unreproducible' (built from temporary commits). Before: $ morph build systems/devel-system-x86_64-generic.morph 2015-03-31 10:47:18 Starting build 53546647f92a4283bf96e33ea93e767e 2015-03-31 10:47:18 Collecting morphologies involved in building systems/devel-system-x86_64-generic.morph from master 2015-03-31 10:47:25 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph 2015-03-31 10:47:25 Deciding on task order ... After: $ morph build systems/devel-system-x86_64-generic.morph 2015-03-31 10:46:21 Looking for uncommitted changes (pass --local-changes=ignore to skip) 2015-03-31 10:46:21 Creating temporary branch in /src/ws/master/baserock/baserock/definitions named refs/heads/baserock/builds/f0b21fe240b244edb7e4142b6e201658/8df11f234ab24d22a9616ce911542332 2015-03-31 10:46:28 Building file:///src/ws/master/baserock/baserock/definitions 7735ec20d13490058a3d7758f79471637ce1ae71 systems/devel-system-x86_64-generic.morph 2015-03-31 10:46:28 Deciding on task order ... Change-Id: I21a3e65c29adf0b07f743289c6b3f0f4dddc34be
* Set PATH to fixed valueKejia Hu (Terry)2015-03-301-1/+1
| | | | | | | In the native build phase, building of cmake will fail if the order of pathes in $PATH is twisted. So we set it to fixed to avoid problem. Change-Id: I6b4d3ed3d5331c2db4f5f3324207fabb4310985c
* Add plugin to get details of given chunk name.Michael Drake2015-03-301-0/+79
| | | | | | | | | | | | | | | | | | | | | Usefully, this provides the resolved repo URL, which may be aliased in the actual defintions. This makes it easier to use a workflow that avoids `morph edit`. Resolved repo URLs can be git cloned. It also provides other info about the chunk. In the case of multiple strata containing the same chunk name, the tool lists them all, saying which stratum they come from. Example output: # morph get-chunk-details libpng In stratum graphics-common: Chunk: libpng Repo: git://git.baserock.org/delta/libpng Ref: 259fb7761d747655c607efcec7a12ff1f3c24561 Change-Id: I6b540e38d9521f3f473b51c9031e508ddcb458ee
* % escape all % charactersRichard Ipsum2015-03-271-1/+1
|
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-162-3/+3
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Merge "Use the modern way of the GPL copyright header: URL instead real address"Javier Jardón2015-03-1615-40/+25
|\
| * Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-1615-41/+26
| | | | | | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* | Use python3 compatible notation for octal constantsJavier Jardón2015-03-131-3/+3
| | | | | | | | Change-Id: I771c3de9cecda7a503f4d36ae5d9fabc040892e4
* | print: use function instead the statement oneJavier Jardón2015-03-131-1/+2
|/ | | | Change-Id: I7c6f618d5d19e03d906798a6f799d74ac55b0d09
* Merge branch 'sam/distbuild-build-logs'Sam Thursfield2015-03-111-1/+0
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * distbuild: Create a new directory to store build logs for each build.Sam Thursfield2015-02-181-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | It gets messy having hundreds of build-step-xx.log files in the current directory, and if two builds are run in parallel from the same directory the logs for a given chunk will be mixed together in one file. Now, a new directory named build-0, build-1, build-2 etc is created for each new build. If the user passes --initiator-step-output-dir the logs will be placed in that directory, instead. This behaviour is the same as before.
* | Add an LRU cache for resolved tree refsSam Thursfield2015-02-181-1/+2
|/ | | | | | | | | | | | This uses the PyLRU module, from: <https://pypi.python.org/pypi/pylru/1.0.6>. Python 3.2 and newer provide a built-in LRU cache, but this is specifically for in-memory use. See <http://bugs.python.org/issue17528>. Git commits are immutable, so caching information about their contents is fairly easy and trouble-free. There's no danger of the cache becoming stale.
* Fix copyright yearsSam Thursfield2015-02-111-1/+1
|
* distbuild: Refuse to start controller if there are no workers listedSam Thursfield2015-02-111-0/+5
| | | | | | | | This is another situation where builds could hang forever if the server is misconfigured. Longer term, workers should be able to come and go dynamically without needing to reconfigure and restart the controller process.
* Treat path given to deploy cmd as relative to cwdRichard Ipsum2015-01-231-2/+4
|
* Treat path given to build cmd as relative to cwdRichard Ipsum2015-01-231-2/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Currently the build commands treat the system argument as a path relative to the root repo. This means that regardless of your working directory you must run morph build systems/foo-system.morph This behaviour can be confusing, for example when your working directory is $systembranch/definitions/systems you might expect to be able to run morph build foo-system.morph especially since most shells would tab-complete the filename for you. At the moment running the above command from $systembranch/definitions/systems would result in an error, because morph would look for $systembranch/definitions/foo-system.morph rather than $systembranch/definitions/systems/foo-system.morph This behaviour also means you can't give the morph build commands an absolute path to a system morph. This patch changes the treatment of the system arg so that it is interpreted relative to the current working directory.
* Merge branch 'sam/optional-temporary-build-branches'Sam Thursfield2015-01-192-26/+87
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
| * Make temporary build branches optionalSam Thursfield2015-01-062-26/+87
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Not everyone is a fan of the `morph build` magic that collects up your changes and puts them in a temporary branch. Now you can disable it by setting 'local-changes=ignore' in your morph.conf file. This speeds up `morph build` and `morph deploy` by 5-10 seconds on my machine. I looked an option to make `morph build` warn if there are uncommitted changes. I found that with a cold cache, it takes about 5 seconds on my machine to verify that there are no uncommitted changes to a checkout of definitions.git. That defeats the main purpose of this patch for me, so I didn't include the option.
* | Merge branch 'sam/cached-repo-cleanup'Sam Thursfield2015-01-122-4/+4
|\ \ | |/ |/| | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-032-4/+4
| | | | | | | | | | | | | | | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
* | Rename builder2 module to builderSam Thursfield2014-12-191-2/+2
| |
* | deploy: Note that some of the extensions listed live in definitions.gitSam Thursfield2014-12-091-2/+5
| | | | | | | | | | | | | | | | The user should be aware of this because if they aren't building baserock:baserock/definitions or a repo forked from it, those extensions won't be available. Also fix some long lines that I seem to have failed to commit already.
* | Add missing types to the docstring in the DeployPlugin.deploy() functionPete Fotheringham2014-12-051-1/+27
|/
* Remove unicode charactersPedro Alvarez2014-12-011-2/+4
|
* Action review commentsPete Fotheringham2014-12-011-0/+23
| | | | | - Document different ways of calling parameters - Allowed values for boolean parameters
* Fix list-artifacts commandSam Thursfield2014-11-141-1/+1
| | | | | Silly mistake caused this to be broken, and there are no tests for this command inside the morph.git source tree.
* Allow `morph deploy` to cache missing Git repos if it needs toSam Thursfield2014-11-111-1/+0
| | | | | | | | | | | | | | | | | | | | | | | | This reverts commit c63f4e61f69820c71c2d8e9b96ce9bdec0d476a1. This change was introduced back when the Baserock system definitions used named Git refs rather than exact SHA1s. Morph would thus need to update each repo involved in a build and resolve each ref before it could build. Now that the system definitions use SHA1s, the 'updating gits' phase is much less slow. The problem with forcing 'no-git-update' is that it makes `morph deploy` depend on the remote repo cache. In the case that all repos involved in the build are avaiable in the remote repo cache (Trove), there is no need for them to have been cached locally just to construct a build graph. But if the network connection to the Trove is unreliable Morph will need to fall back to the local repo cache. Unless Morph has done a full local build of the system being built, it might not have every repo cached locally. Often, cached artifacts from the Trove or distbuild may be used instead of locally building everything. In this case the user sees `morph deploy` failing with errors such as: Repository upstream:binutils-redhat is not cached yet
* Move create_source_pool code into new 'sourceresolver' moduleSam Thursfield2014-11-101-2/+4
| | | | | | | | This code is an essential part of 'morph build'. It's quite complex and really shouldn't be mixed in with the base Application class. Given a dedicated class we can store some state in the object and avoid functions with seven parameters, too.
* Only search for repository root in GitDirectory constructor if told toSam Thursfield2014-11-052-3/+3
| | | | | | | | | | | | | | | The GitDirectory() constructor, if passed a 'dirname' that doesn't contain a '.git' subdirectory, can search upwards to find the real root of the repository. This is used by the `add-binary`, `push`, and `pull` commands. This causes very confusing behaviour in the case that 'dirname' points to a directory that should be a Git repository, but isn't, and that directory is a path inside the working tree of another Git repository. Rather than raising an error, in this case the GitDirectory class would perform operations on a different repository to the one the caller expected. This 'search_for_root' behaviour is now opt-in, to avoid confusion.
* Fix lines longer than 79 charactersSam Thursfield2014-10-291-2/+2
|
* Merge remote-tracking branch 'origin/sam/build-fix-3'Sam Thursfield2014-10-291-6/+15
|\ | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
View Lorry Controller Status