summaryrefslogtreecommitdiff
path: root/morphlib/localrepocache.py
Commit message (Collapse)AuthorAgeFilesLines
* Unify local and remote repo cache modulesSam Thursfield2016-03-161-357/+0
| | | | | | | | | | | | | | | | | | | | | There's not really any reason you'd want to use the RemoteRepoCache class except as a workaround for the slow speed of some LocalRepoCache operations, so I can't see this ruining anyone's day. The main reason for doing this is so we can simply the sourceresolver code. One reason that the sourceresolver class is so hopelessly complicated is that it right now has to use two incompatible interfaces for Git repo caches. I've taken the opportunity to detangle the RepoCache class from the App class. Now all of the configuration for the RepoCache class is passed into the constructor explicitly. This makes the class usable from outside Morph: resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[]) repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver) Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
* Get rid of the CachedRepo class (almost)Sam Thursfield2016-03-031-10/+44
| | | | | | | | | | | | | | | | | | For a long time the CachedRepo class has basically been a wrapper around the GitDir class, but with a few extra methods that don't really even belong there. It is now a tiny class in the localrepocache module which just keeps track of a few extra attributes. All other functionality is provided by the gitdir module. This commit also removes the `git clone` approach for copying repos out of the cache. The alternative approach implemented by git.copy_repository() was slightly faster when I tested, so for now we should use that everywhere. Longer term we should find out why this is quicker than `git clone`, and fix Git itself to be fast. Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
* Hide get_repo() and cache_repo() functions, always use get_updated_repo()Sam Thursfield2016-03-031-9/+9
| | | | | | | | | | This allows us to simplify a couple of places. I cannot think of a single situation where'd you want to get the cached copy of a repo, but not update it. Think about it -- the repo might be *years* behind the upstream remote. Change-Id: I60340c7fb33e7bfe871ad30c0a9322a7202548e2
* Extract downloaded tarball as current userRichard Ipsum2015-07-201-1/+1
| | | | | | | | | | | | | | | | | | tar will preserve ownership and permissions by default if running as the superuser, whereas git doesn't track ownership, the owner is set to the user running the clone. This is a reproducibility hole since the resulting repo may differ depending on whether the repo was obtained from a tarball or via a git clone, but morph considers the repos to be equivalent. Running tar with --no-same-owner ensures the files written by tar will be owned by the user extracting the archive. This only fixes half the problem, lorry may also need to be modified to ensure files in the tarball have the same mode as those in the git repo. Change-Id: I849d054bc4574f9c0bbcfd608914306e52b7a22e
* localrepocache: Count freshly cloned repositories as updatedRichard Maw2015-06-171-1/+4
| | | | | | | This fixes requires_update_for_ref when it is given a non-fixed ref and it is retrieved from a call to get_updated_repo. Change-Id: Icbcf2e4ec45bd559c6bcb450ceb4a8861b56f654
* Fix crash in handling of recursive submodulesSam Thursfield2015-05-141-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It turns out I completely broke Morph's handling of recursive submodules in commit 0855c357e74e6dd7a. We didn't notice because nothing in the reference systems actually uses recursive submodules. However, building the baserock/tlsa/mason2 branch of definitions.git triggered the bug due to the delta:python-packages/gitpython chunk, which contains delta:python-packages/gitdb which in turn contains delta:python-packages/smmap. The error is this: Traceback (most recent call last): File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 290, in process_args cliapp.Application.process_args(self, args) File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build self._build(source_pool, filename, component_names=component_names) File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build bc.build_in_order(component) File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order self.cache_or_build_source(s, build_env) File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source self.build_source(source, build_env) File "/src/morph/morphlib/buildcommand.py", line 336, in build_source self.fetch_sources(source) File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources self.lrc.ensure_submodules(source.repo, source.sha1) File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules if (submod.url, submod.commit) not in done: AttributeError: 'tuple' object has no attribute 'url' This commit fixes the breakage and the baserock/tlsa/mason2 branch of definitions.git now builds again. Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
* LRC: Make get_updated_repo handle multiple refsRichard Maw2015-05-111-16/+27
| | | | | | | | | | Passing a single ref is still accepted, but if you have multiple refs you need to check from the same repository, it is more appropriate to do it in one call to get_updated_repo, as otherwise there will be unnecessary output about it not needing to be updated in multiple places. Change-Id: I194d7c0e3e84c4917518ba37672b508505c71b8e
* Show progress of downloads when --verbose is passed, not --debugSam Thursfield2015-05-061-1/+1
| | | | | | | This commit undoes behaviour changes from commit aa6dfcbb70c03dfeb3f9af02. Change-Id: Ie677fb9c4e6bcd6edeba2cdd87f4f6125dcae7a4
* Use --verbose/-v to show build output, --debug/-d for morph debugJavier Jardón2015-04-221-2/+2
| | | | | | | | | | | Its going to be more likely that a baserock user is more interesed in the build log than actually the debug output from morph. And also the intuitive option would be to use -v for this. --verbose/-v: show build output --debug/-d: show morph debug output Change-Id: I1fb99034dc8680a5f168f6306724663aea33ebc5
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-3/+3
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-2/+1
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Merge branch 'sam/fewer-submodule-updates'Sam Thursfield2015-03-031-13/+63
|\ | | | | | | | | Reviewed-By: Javier Jardón <javier.jardon@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * Only update Git submodules in cache when necessarySam Thursfield2015-02-101-14/+64
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This saves a duplicate `git remote update origin` that was being run as part of each chunk build. For any repos that have submodules, it also avoids updating repos if the SHA1 we need to build is already present locally. As well as speeding up builds slightly, this means Morph can now build without being connected to a network, as long as the local Git cache all of the necessary repos and commits in the build, without needing the '--no-git-update' option. The code is also now in a more logical place than before.
* | sourceresolver: Factor out 'cache repo locally' code into a functionSam Thursfield2015-02-201-2/+4
|/ | | | | | | Also, move the repo.update() call into the 'fetch from tarball' code path in the localrepocache module -- there's no need to update straight after doing a `git clone`, but we do need to do one if we got the repo from a `git archive` tarball that may be out of date.
* Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-031-2/+6
| | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
* Echo stderr of subcommands that do network IO when --verbose is usedSam Thursfield2014-10-291-5/+21
| | | | | | | | | | | | Morph can appear to hang in situations where it is actually waiting on a slow network operation. This change gives users a way to see the output of the subcommands that are doing the network IO (either 'wget', 'git clone' or 'git remote update'). The status information goes onto stderr, because that is where the subcommands write it. Morph tends to put its status output on stdout, but (a) some commands are machine-parsed, such as `serialise-artifact` and (b) it's tricky to get Git to put status output on stdout.
* Don't say 'Updating git repository xxx' unless we actually areSam Thursfield2014-10-291-1/+2
| | | | | | | This fixes an issue where various branch-and-merge commands appeared to be updating Git repos even when --no-git-update was specified. The flag was actually honoured but the message made it seem as if it was being ignored.
* Fix confusing messagebaserock/ps/message-fixPaul Sherwood2014-10-271-1/+1
|
* Prevent git-replace refs affecting git operationsRichard Maw2014-08-211-2/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We assumed that the sha1 of the tree of the commit of the ref we care about was sufficient to cache, but `git replace` means that you need to know the state of other branches too, since anything in refs/replace can completely change what the tree you check-out is. This behaviour can be disabled globally by setting GIT_NO_REPLACE_OBJECTS, so we're going to do that. If we need to integrate a project that uses git-replace to change the contents of their git trees then we could support that by: if any(refs/replace/*): potentially_replacable_objects = [ `git rev-parse HEAD`, `git rev-parse HEAD^{commit}`, `git rev-parse HEAD^{tree}`] potentially_replacable_objects.extend( `git ls-tree -r HEAD | awk '{print $3}'`) # NOTE: doesn't handle submodules, and you'd need to expand this # set whenever you process a replacement for object in refs/replace/*: if basename(object) not in potentially_replacable_objects: continue cache_key['replacements'][basename(object)] = `git rev-parse $object` If we were to support this would need to traverse the tree anyway, doing replacements, so we may as well use libgit to do the checkout anyway, and list which replacements were used. However, since the expected use-case of `git replace` is as a better way to do history grafting, we're unlikely to need it, as it would only have any effect if it replaced the commit we were using with a different one. Rubber-stamped-by: Daniel Silverstone
* Refactored localrepocacheDaniel Firth2013-12-201-96/+34
|
* Add LocalRepoCache.get_updated_repo methodLars Wirzenius2013-08-061-1/+13
| | | | | This is a helper to avoid having the same code in other places. Have it in the one place it belongs.
* Avoid caching repos that are on the local machine anywaySam Thursfield2012-11-121-3/+7
| | | | | Currently the message is still displayed "Updating xxx" but no update is actually done.
* Use tarballs instead of bundlesDaniel Silverstone2012-09-251-24/+40
| | | | | | | This makes a non-backward-compatible change to morph which switches it to using tarballs instead of bundles when initialising cached git repositories. This is faster because it doesn't require index-pack --fix-thin operations on the machine running morph.
* Refactor _fetch() and log bundle fetches.Daniel Silverstone2012-09-211-11/+18
| | | | | | Log (at chatty) when we start and finish bundle downloads. If we fail for any reason, log that reason since otherwise it gets swallowed if the plain git clone succeeds.
* Rework git caches to be bare mirrors of the repos.Daniel Silverstone2012-09-141-2/+2
| | | | | | | | This reworks the code for managing and using the git caches in morph to treat the caches as bare repositories which are mirrors of where we clone from. In addition we correctly prune the branches during updates, so that we don't end up accumulating pointless branches over and over. This is even more important with branch-and-merge generating temporary build refs for things.
* python scripts: pep8ize codebaseRichard Maw2012-08-011-47/+46
| | | | | | | | | This was done with the aid of the pep8 script, available by running `easy_install pep8`. It may be worth making this part of ./check, but that will require putting pep8 into the development tools stratum. This should be easy, given pep8 has no external dependencies.
* morph: remove dead code and replace Execute with app.runcmdRichard Maw2012-05-301-6/+9
|
* Change LocalRepoCache to use the new RepoAliasResolver.Jannis Pohlmann2012-05-041-64/+6
|
* Fix LocalRepoCache to not abort on %s-less repo patternsLars Wirzenius2012-05-041-3/+17
| | | | Also, add debugging messages for debugging prefix expansion.
* Remove use of baseurls in LocalRepoCacheLars Wirzenius2012-05-041-33/+23
| | | | The tests, they still fail.
* Implement repo-alias expansionLars Wirzenius2012-05-041-3/+42
| | | | | | | | Changed the delimieter between push and pull patterns to be #, since = is reasonably frequent in URLs, and # should never be necessary in git urls. Oh my how many tests now fail.
* Add tests for repo-alias expansionLars Wirzenius2012-05-041-0/+6
|
* Get rid of the old internal morph APIsLars Wirzenius2012-05-021-5/+18
|
* Add morphlib.Error and make local repo cache exceptions be based on itLars Wirzenius2012-04-251-2/+3
| | | | | This way, cliapp will automatically print an error message, rather than a stack trace.
* Fix LocalRepoCache tests.Jannis Pohlmann2012-04-191-1/+1
|
* Clone repos with no checkout, use correct gits cache dir in cmd_build.Jannis Pohlmann2012-04-191-2/+2
|
* Remember repo objects in LocalRepoCache to always return the same ones.Jannis Pohlmann2012-04-171-3/+10
|
* localrepocache: return repo after cloningRichard Maw2012-04-111-5/+14
|
* localrepocache: fixes to remote set-urlRichard Maw2012-04-111-3/+6
| | | | | | It needed shutil, but that exception was being missed. _git needs to be able to handle a different current directory for remote set-url to work
* localrepocache: fix remote origin url to upstreamRichard Maw2012-04-111-0/+1
| | | | | | When cloning from a bundle it makes the origin remote url point to the bundle. This is not ideal as we want to update these repositories later from upstream.
* localrepocache: fix bundle pathsRichard Maw2012-04-111-1/+1
| | | | They must end with .bndl as that is the format lorry generates
* localrepocache: return error descriptionRichard Maw2012-04-111-22/+43
| | | | | | | | Also change _fetch to throw its exception as since it is more pythonic to throw exceptions than return exit status, other error handling needed to be caught, so it hsa been kept consistent. This required the tests to throw an exception as well
* localrepocache: check every base-url before cloneRichard Maw2012-04-111-5/+7
| | | | | It should check that it doesn't already have the repository cached under a different base-url before cloning, likewise with bundles.
* Create cache directory when needed in LocalRepoCacheLars Wirzenius2012-04-101-0/+12
|
* Fix unit test failures: mark out-of-coverage, and use wrapper to allow testsLars Wirzenius2012-04-101-6/+5
| | | | | | | | The self._exists method needs to be used instead of raw os.path.exists, so that unit tests can override it easily. Also, a tiny clarification so that the caller of _clone_with_bundle decides whether it's OK to try to download a bundle.
* Various small fixes to make the new update-gits work again.Jannis Pohlmann2012-04-101-15/+23
|
* Use the real CachedRepo class in LocalRepoCacheLars Wirzenius2012-04-091-9/+1
|
* Remove downloaded bundle after it has been clonedLars Wirzenius2012-04-091-0/+10
|
* Add bundle support to LocalRepoCacheLars Wirzenius2012-04-091-2/+47
|
* Add a LocalRepoCache classLars Wirzenius2012-04-091-0/+144
Jannis and I discussed a refactoring of most of morph's internals to reduce coupling and increase cohesion. This is one of the results: we'll want a couple of classes to manage locally cached git repositories. This commit adds the LocalRepoCache class to manage a the git cache directory. Later on, we'll add the CachedRepo class to represent individual repositories.
View Lorry Controller Status