summaryrefslogtreecommitdiff
path: root/morphlib/sourceresolver.py
Commit message (Collapse)AuthorAgeFilesLines
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Fix call to status() in commit 364455436aded34ce6f843dfa68a32c14aaf01eaSam Thursfield2015-03-161-1/+2
| | | | I was meant to do this at merge time but didn't.
* Minimise repo cache fetchesRichard Ipsum2015-03-071-21/+49
| | | | | | | If we know the build system for a chunk is cached then we also know that it doesn't have a chunk morph, so we don't need to look for one. (We only cache the build systems of chunks that don't have chunk morphs)
* Fix bug in build system cache accessRichard Ipsum2015-03-071-3/+11
| | | | | | | | | This fixes a bug that causes morph to run build system detection even though the build system has already been cached. The cache is accessed before the key has been computed (absref isn't known until after we've called resolve_ref) so it always misses.
* morphlib/sourceresolver.py: Check and parse VERSION filejjardon/versioning_supportJavier Jardón2015-03-031-0/+25
| | | | | | | | | | | Only fail if: - VERSION file exists - and its a yaml file - and its a dict - and has the key 'version' - and the contents of the key 'version' is an int - and that int is in the list of non_supported_versions (empty at the moment)
* morphlib/sourceresolver.py: Remove not used functionsJavier Jardón2015-02-271-16/+0
| | | | _get_morphology_from_definitions() and _get_morphology_from_repo()
* morphlib/sourceresolver.py: Do not duplicate what is already done in ↵Javier Jardón2015-02-271-8/+3
| | | | load_from_string()
* morphlib/sourceresolver.py: Add _get_file_contents()Javier Jardón2015-02-271-12/+23
| | | | And make _get_morphology() use it
* morphlib/sourceresolver.py: Add _get_file_contents_from_definitions()Javier Jardón2015-02-271-1/+10
| | | | And make _get_morphology_from_definitions() use it
* morphlib/sourceresolver.py: Add _get_file_contents_from_repo()Javier Jardón2015-02-271-5/+11
| | | | And make _get_morphology_from_repo() use it
* sourceresolver: Never assume that a given ref is present locallySam Thursfield2015-02-201-6/+8
| | | | | | | | | | | | | Checking that a given ref exists using `git rev-parse --verify 1234^{commit}` is a reasonably quick operation. As a rough guide, 1000 invocations took 1.6 seconds on my PC. The code is too fragile and hard to reason about if we assume that one function has been called before another so the repo will already be up to date. This should fix any spurious InvalidRefError exceptions that the build graph speedups branch has introduced.
* sourceresolver: Factor out 'cache repo locally' code into a functionSam Thursfield2015-02-201-25/+19
| | | | | | | Also, move the repo.update() call into the 'fetch from tarball' code path in the localrepocache module -- there's no need to update straight after doing a `git clone`, but we do need to do one if we got the repo from a `git archive` tarball that may be out of date.
* sourceresolver: Simplify some code pathsSam Thursfield2015-02-201-33/+35
| | | | | It turns out that always looking for the chunk morph in the definitions repo allows us to make the code a little less nasty. Bonus!
* sourceresolver: Fix InvalidRefError when local git cache is out of dateSam Thursfield2015-02-201-7/+11
| | | | | | | | | | | | | Code in sourceresolver.py assumed that resolve_ref() would be called for the chunk repo before get_morphology() was called, so the repo would always be up to date. This wasn't actually true. If your local repo cache was out of date, you might see the following sort of error: InvalidRefError: Git directory /src/cache/gits/git___git_baserock_org_delta_usbutils has no commit at ref c37f146eb2c6642c600f1b025a6d56996b0697ff^{tree}.
* sourceresolver: Always try to read chunk morph files from definitionsSam Thursfield2015-02-201-29/+40
| | | | | | | | | | | | | | | | | | | | | This fixes the 'Morph ignores the chunk morph file I added to definitions' regression introduced in the recent build-graph speedups branch. The new 'detected-chunk-buildsystems' cache associates a (chunk repo, chunk ref) pair with the auto-detected build system. When calculating the build graph, if the is an auto-detected build system known already, that will be used instead of looking for a chunk morphology. There's a flaw here: if the user added or changed the chunk morphology in the definitions repo, the *chunk* ref won't necessarily have changed -- so Morph will ignore the user's changes, if it had already cached an autodetected buildsystem for that repo/ref pair. This doesn't cost much in terms of speed because we're just reading a file from disk. We can still avoid looking in the chunk repo for a chunk morph, because the chunk ref would be different if something had changed in the chunk repo.
* sourceresolver: Fix LsTreeError raised when constructing build graphSam Thursfield2015-02-201-7/+19
| | | | | | | | | | | | | | | | If you had a repo that was not in your local or remote cache in your definitions, Morph might raise LsTreeError and abort. The code was assuming the repo would already be cached in the local cache by the time it got to detecting build systems, but that's not always true -- if the (reponame, ref) pair was already in the 'resolved_trees' cache but the repo had been deleted from the local Git cache, to name one possibility. Also, the remoterepocache.ls_tree operation can fail if the repo is not hosted in the Trove, so we shouldn't abort the program in that case. Finally, this patch removes an unused variable.
* Add tests for sourceresolverAdam Coldrick2015-02-181-10/+10
| | | | | | | | | This only adds tests for the bits which were moved from morphologyfactory into sourceresolver, namely detection of build systems and the '_get_morphology()' function. These are just the morphologyfactory tests reworked slightly to work properly with the modified API.
* Read files from a local clone of definitions where possibleSam Thursfield2015-02-181-1/+30
| | | | | | | | | | | | | | | | | Most morphologies involved in a build are in the definitions repo these days. Currently we read each of them using `git cat-file`, which is slow. It's quicker to check out all the files in one go to a temporary directory and then read them from there. With the current workflow users often have definitions.git checked out on disk. It seems strange to not just read the files from there. There are two reasons why I don't want to do that yet: - there are commands which don't run inside a system branch, which would be broken if we expected to always be in a system branch - there may be local changes in the checked-out repo, and it takes around 5 seconds on each build to check if there aren't any local changes. It actually seems faster to just check out a known clean version from the cache.
* Split up traverse_morphs to improve readabilitySam Thursfield2015-02-181-61/+79
|
* Add an LRU cache for detected build-systemsSam Thursfield2015-02-181-19/+101
| | | | | | | This will speed up builds of chunks which don't have a chunk morph. It won't have much (if any) effect on the speed of the first build, but subsequent builds will be much faster as we won't have to query the git cache.
* Add an LRU cache for resolved tree refsSam Thursfield2015-02-181-29/+72
| | | | | | | | | | | | This uses the PyLRU module, from: <https://pypi.python.org/pypi/pylru/1.0.6>. Python 3.2 and newer provide a built-in LRU cache, but this is specifically for in-memory use. See <http://bugs.python.org/issue17528>. Git commits are immutable, so caching information about their contents is fairly easy and trouble-free. There's no danger of the cache becoming stale.
* Add a cache manager utilising pylru and pickleAdam Coldrick2015-02-181-2/+52
|
* Move MorphologyFactory into SourceResolverSam Thursfield2015-02-181-9/+69
| | | | | There's no need for this stuff to be in a separate class. This allows integrating it with the caching in the SourceResolver class.
* sourceresolver: Add comments and factor out common functionSam Thursfield2015-02-181-13/+15
|
* Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-031-2/+4
| | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
* Separate resolution of ref->commit and commit->treeSam Thursfield2014-12-021-7/+15
|
* Add docstring to SourceResolver class.Sam Thursfield2014-11-101-1/+22
|
* Set the 'update' flag in the SourceResolver constructorSam Thursfield2014-11-101-12/+14
|
* Move create_source_pool code into new 'sourceresolver' moduleSam Thursfield2014-11-101-0/+176
This code is an essential part of 'morph build'. It's quite complex and really shouldn't be mixed in with the base Application class. Given a dedicated class we can store some state in the object and avoid functions with seven parameters, too.
View Lorry Controller Status