summaryrefslogtreecommitdiff
path: root/morphlib/git.py
Commit message (Collapse)AuthorAgeFilesLines
* gitdir: Re-introduce clone_into functions needed for the import toolHEADmasterChristopher Phang2016-09-261-0/+6
| | | | | | | | The clone_into functions were deprecated from commit 014a029ade9a045a839ca86c35690b218098ea33, but functions in baserock-import still depend on these functions. Change-Id: I7362e1ae70bf86677b0019248ca8538912507b51
* morphlib/git: Fix update_submodulesPedro Alvarez2016-04-061-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | It seems to be broken since d58d8e8f7a4ec03ff14021a4515c8283dad52573 and it's making `morph get-repo` fail with: /src/definitions # morph get-repo ansible 2016-04-06 11:36:31 Loading in all morphologies 2016-04-06 11:36:32 Checking out ref 0a7124541247cc613352054f4bc0c3e116e0b657 of ansible in ansible stratum Traceback (most recent call last): File "cliapp/app.py", line 190, in _run self.process_args(args) File "/src/morph/morphlib/app.py", line 299, in process_args cliapp.Application.process_args(self, args) File "cliapp/app.py", line 539, in process_args method(args[1:]) File "/src/morph/morphlib/plugins/get_repo_plugin.py", line 142, in get_repo chunk_dirname = checkout_chunk(morph, chunk) File "/src/morph/morphlib/plugins/get_repo_plugin.py", line 110, in checkout_chunk ref or chunk_spec['ref']) File "/src/morph/morphlib/plugins/get_repo_plugin.py", line 60, in _clone_repo gd.update_submodules(self.app) File "/src/morph/morphlib/gitdir.py", line 527, in update_submodules morphlib.git.update_submodules(app, self.dirname) File "/src/morph/morphlib/git.py", line 150, in update_submodules submodules.load() File "/src/morph/morphlib/git.py", line 69, in load content = self._read_gitmodules_file() File "/src/morph/morphlib/git.py", line 82, in _read_gitmodules_file ignore_fail=True) File "/src/morph/morphlib/git.py", line 306, in gitcmd return runcmd(cmdline, **kwargs) TypeError: 'str' object is not callable Change-Id: I8d08eb71a2b0a773bc4322ed46222bea0adbbee1
* morphlib/git: Ignore not valid submodulesPedro Alvarez2016-04-041-1/+8
| | | | | | | | | | | | Some repositories may have submodules listed in .gitmodules that are not pointing to a commit object in the tree. This was causing the following error: ERROR: /srv/distbuild/gits/git___git_foo_bar_git:c36652c7dd9f1687c87ce04ad3d4af2beb52dc25:.gitmodules: No commit object found for submodule "codegen" Instead of raising that error, capture it and ignore the submodule. Change-Id: Iba0428feea7605f797dc78e3e250f3d66c003670
* Add support for definitions version 8Pedro Alvarez2016-03-291-36/+11
| | | | | | | | | This code is a rework from changes done by: - Tiago Gomes <tiago.gomes@codethink.co.uk> https://storyboard.baserock.org/#!/story/86 Change-Id: I3475c2bcb648a272fee33bc878a521f79d4e6581
* Unify local and remote repo cache modulesSam Thursfield2016-03-161-4/+5
| | | | | | | | | | | | | | | | | | | | | There's not really any reason you'd want to use the RemoteRepoCache class except as a workaround for the slow speed of some LocalRepoCache operations, so I can't see this ruining anyone's day. The main reason for doing this is so we can simply the sourceresolver code. One reason that the sourceresolver class is so hopelessly complicated is that it right now has to use two incompatible interfaces for Git repo caches. I've taken the opportunity to detangle the RepoCache class from the App class. Now all of the configuration for the RepoCache class is passed into the constructor explicitly. This makes the class usable from outside Morph: resolver = morphlib.repoaliasresolver.RepoAliasResolver(aliases=[]) repo_cache = morphlib.repocache.RepoCache('/src/cache/gits', resolver) Change-Id: I596c81d7645b67504c88e555172a8c238f4f8a66
* Get rid of the CachedRepo class (almost)Sam Thursfield2016-03-031-16/+5
| | | | | | | | | | | | | | | | | | For a long time the CachedRepo class has basically been a wrapper around the GitDir class, but with a few extra methods that don't really even belong there. It is now a tiny class in the localrepocache module which just keeps track of a few extra attributes. All other functionality is provided by the gitdir module. This commit also removes the `git clone` approach for copying repos out of the cache. The alternative approach implemented by git.copy_repository() was slightly faster when I tested, so for now we should use that everywhere. Longer term we should find out why this is quicker than `git clone`, and fix Git itself to be fast. Change-Id: I1686ab43253d44c3903d9a0bad8bb75528e9cf75
* Ensure Git clones in a chunk build directory have correct ownershipSam Thursfield2015-10-071-0/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When preparing the build directory for a chunk, Morph clones the Git repo being built (and any submodules) into the staging area. Instead of using `git clone --no-hardlinks`, the morphlib.git.copy_repository() function uses 'cp -a' plus some faffing to achieve the same thing a bit faster. The 'cp -a' command will preserve the ownership and permissions of the cached copy. These may not match up with which user is doing the build. I found that files in my Git cache were owned by UID 1002. This caused chunks that needed to access the .git directory at build time to fail with strange errors. Worse, it would trigger a bug in Git[1] that leads to a fork bomb, which would cause Linux to freeze up completely. This occured even though I was building as `root`, because of the way `linux-user-chroot` drops certain privileges: presumably, setting SECBIT_NOROOT leads to CAP_DAC_OVERRIDE being unset. To avoid this bug, the code now ensures the copied .git repos are owned by the user and group who ran `morph`. Another way to fix this would be to change the morphlib.gits.copy_repository() function to use `git clone --no-hardlinks`. This is what YBD does. I found that there is a slight speed benefit to using the current code ... these are results of cloning repos from the cache with the 2 methods: With 'cp -a' plus a chown: upstream:bison: 4.54 seconds average (10 results) upstream:ybd: 0.13 seconds average (10 results) upstream:linux: 40.51 seconds average (10 results) With 'git clone --hardlinks': upstream:bison: 6.23 seconds average (10 results) upstream:ybd: 0.11 seconds average (10 results) upstream:linux: 43.36 seconds average (10 results) Test code is: <https://gist.github.com/ssssam/833e0ef8d04fb1fb6ff3>. Ideally we would fix `git clone --no-hardlinks` to be faster, but we may as well keep the existing code for the time being. [1]: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=750687 Change-Id: Ieea87322ea7b7f62975b9480f877755665656217
* Use python3 compatible notation for catching exceptionsJavier Jardón2015-03-161-2/+2
| | | | Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
* Use the modern way of the GPL copyright header: URL instead real addressJavier Jardón2015-03-161-3/+2
| | | | Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
* Remove unused code from morphlib.gitSam Thursfield2014-12-031-24/+0
| | | | | There is still code here that duplicates stuff the GitDirectory class should be doing, I think.
* Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-031-9/+0
| | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
* Check for commit object in submodule pathbaserock/franred/check-for-commit-object-in-submodule-pathFrancisco Redondo Marchena2014-11-041-1/+1
| | | | | Fix checking for commit object in submodules which their path and name are not the same.
* Echo stderr of subcommands that do network IO when --verbose is usedSam Thursfield2014-10-291-0/+11
| | | | | | | | | | | | Morph can appear to hang in situations where it is actually waiting on a slow network operation. This change gives users a way to see the output of the subcommands that are doing the network IO (either 'wget', 'git clone' or 'git remote update'). The status information goes onto stderr, because that is where the subcommands write it. Morph tends to put its status output on stdout, but (a) some commands are machine-parsed, such as `serialise-artifact` and (b) it's tricky to get Git to put status output on stdout.
* Fix `morph edit` for non-file URIsRichard Maw2014-08-261-1/+1
| | | | | | | | | The clone_into function is non-functional when you pass it a sha1 ref. If you have a file:// URI then this doesn't get used, which is how it slipped past the tests. Reviewed-by: Lars Wirzenius +2
* Prevent git-replace refs affecting git operationsRichard Maw2014-08-211-30/+41
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We assumed that the sha1 of the tree of the commit of the ref we care about was sufficient to cache, but `git replace` means that you need to know the state of other branches too, since anything in refs/replace can completely change what the tree you check-out is. This behaviour can be disabled globally by setting GIT_NO_REPLACE_OBJECTS, so we're going to do that. If we need to integrate a project that uses git-replace to change the contents of their git trees then we could support that by: if any(refs/replace/*): potentially_replacable_objects = [ `git rev-parse HEAD`, `git rev-parse HEAD^{commit}`, `git rev-parse HEAD^{tree}`] potentially_replacable_objects.extend( `git ls-tree -r HEAD | awk '{print $3}'`) # NOTE: doesn't handle submodules, and you'd need to expand this # set whenever you process a replacement for object in refs/replace/*: if basename(object) not in potentially_replacable_objects: continue cache_key['replacements'][basename(object)] = `git rev-parse $object` If we were to support this would need to traverse the tree anyway, doing replacements, so we may as well use libgit to do the checkout anyway, and list which replacements were used. However, since the expected use-case of `git replace` is as a better way to do history grafting, we're unlikely to need it, as it would only have any effect if it replaced the commit we were using with a different one. Rubber-stamped-by: Daniel Silverstone
* Make existing morph commands use git-fatAdam Coldrick2014-03-201-1/+9
| | | | | | When cloning a repository, the files stored using git-fat need to be pulled. This situation occurs in `morph branch`, `morph edit`, and `morph checkout`.
* morphlib.git: Refactor is_valid_sha1Richard Maw2013-09-051-2/+2
| | | | | | | It now uses a generator expression instead of a list, which makes the operation lazy. It also uses string.hexdigits instead of redefining it.
* Use Submodules class to read submodule configRichard Maw2013-08-231-6/+6
| | | | | | | | | | Previously it would use `git config --get-regex`, which while more terse, is a little arcane and the regular expression did not account for slashes or dashes in the option names, and the names usually correspond to a file path. Now we use the Submodules class, which validates that the listed commit is available.
* Add morph cross-bootstrapSam Thursfield2013-07-301-0/+2
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Cross-bootstrap is a way to build baserock on an architecture that does not currently have Baserock. It can be used by `morph cross-bootstrap <ARCH> <REPO> <REF> <MORPH>`, and will build an artifact that can be used as a root filesystem with a basic build environment with a script named `native-bootstrap` which will build and install every chunk in the system. If done with a devel system, this will give you a suitable environment for building a proper Baserock system. This does not currently provide a kernel for the target architecture. Apart from adding the cross-bootstrap plugin, it also makes the following changes: * Moves the lit of valid_archs into morphlib (instead of locally-scoped in MorphologyFactory) * BuildCommand takes an extra argument, build_env * split BuildCommand's get_artifact_object into create_source_pool and resolve_artifacts (plus changes things that use get_artifact_object to use the new way) * setup_mounts finds out whether to do so by whether build_mode is 'staging', instead of by whether the setting 'staging-chroot' is true. * Makes ChunkBuilder's get_sources use the morphlib.builder2.extract_sources() method, and moved set_mtime_recursively into morphlib.builder2, since it's not currently used anywhere else. * moved ChunkBuilder's get_commands into the Morphology class (plus changes to anything that used get_commands)
* Fix lines overflowing the 80 character limitRichard Dale2013-06-251-1/+2
|
* Don't show git config messages for 'morph branch' in verbose modeRichard Dale2013-06-251-1/+1
|
* Changed the error message in log when morph fails to find gitmodulesRic Holland2013-03-201-1/+1
| | | | morph.log will now show an INFO message rther than a ERROR
* B&M: Share error message formatting codeRichard Maw2013-03-041-21/+44
| | | | | | | This makes the warning messages include how to fix the problem. morphlib.git.check_config_set will return the values of the keys which were set, so it can be used to get git config.
* B&M: Check git config before some subcommandsRichard Maw2013-03-011-0/+17
| | | | | | | | | Some subcommands use git to create commits, in which case user config needs to be set. Others imply commits may be created, e.g. by cloning a repository, at which point it is useful to have a reminder that the configuration needs to be set.
* git: clone then checkout instead of clone -bRichard Maw2013-02-111-1/+5
| | | | | | | | | Newer versions of git have changed the behaviour when cloning with --branch=$sha1. It used to be nearly synonamous with clone then checkout. The clone then checkout logic is different to the clone with branch logic, since they have different behaviour with regard to the branches that are checked out in the workspace.
* Merge remote-tracking branch 'remotes/origin/samthursfield/ambiguous-refs' ↵Jannis Pohlmann2012-12-131-1/+1
|\ | | | | | | into baserock/merge-queue
| * Always use `git rev-parse --verify` to resolve refsSam Thursfield2012-12-131-1/+1
| | | | | | | | | | | | | | | | | | | | | | | | | | Previously some code used `git show-ref`, which is wrong -- given two refs named 'alpha/master' and 'master', `git show-ref master` will return both, sorted alphabetically. This can lead to build failures, etc. due to refs resolving to the wrong SHAs. We should also use `git rev-parse --verify` to verify SHA1s, which we previously did with `git rev-list`. Finally, `git rev-parse --verify` is more than twice as fast as `git show-ref`.
* | Add an initial implementation of "morph tag"Jannis Pohlmann2012-12-131-0/+13
|/ | | | | | | | | | | | | | | | | | | | | | | | | | In order to make releases and freeze system branches entirely, we need to be able to 100% petrify a system branch (that is, resolve ALL refs into SHA1s) and tag this state to be able to check it out again later. This is essentially what "morph tag" does. It takes a tag name and an arbitrary amount of arguments to "git tag", petrifies all morphologies of the current system branch behind the scenes, creates a dangling commit and attaches an annotated tag to it. Petrifying in this case means that all refs used for chunks are resolved into commit SHA1s. For stratum and system morphologies, the refs are replaced by the name of the tag that's being created. The "tag" command also supports tagging when stratum morphologies are spread across multiple repositories. In this case, it will include all statum morphologies from other repos in the tag commi in the branch root repo. The references to these morphologies are updated so that they point to the branch root repo and the tag being created. This commit also adds a few tests for "morph tag" to verify that all this works.
* Avoid caching repos that are on the local machine anywaySam Thursfield2012-11-121-1/+6
| | | | | Currently the message is still displayed "Updating xxx" but no update is actually done.
* Handle submodules in 'morph edit'Sam Thursfield2012-10-031-0/+22
| | | | | | | | | | | | | | | | We often have .gitmodules edited to contain a URI such as upstream:gnulib, so that we can transparently mirror these in different locations. It would be nice to set up git url.insteadOf rules to expand these for the submodules, but 'git submodule update' uses 'git clone' to fetch them, which will not take into account the configuration of the parent repository. Instead, we set up the submodules automatically and rewrite the URLs directly in the configuration. The user will need to recreate their system branch checkouts if their URL configuration changes, or update the URLs manually, but that should not happen often.
* Use origin/ref for all refs other than the system branch ref itselfSam Thursfield2012-10-031-5/+4
| | | | | | | | | | The rationale here is that inside a checkout of a system branch, the user should only be committing to the refs for that system branch, because those are the only ones that 'morph merge' will look at. This removes a problem where we could be confused by a ref with a name that would be sorted before 'master' in the result of 'git show-ref'.
* Extract is_valid_sha1() to morphlib.gitSam Thursfield2012-10-031-0/+6
|
* morph merge: Don't make a merge commit if only the refs changedSam Thursfield2012-09-141-0/+11
| | | | 'git commit' fails if there are no changes in any case.
* Rework git caches to be bare mirrors of the repos.Daniel Silverstone2012-09-141-2/+49
| | | | | | | | This reworks the code for managing and using the git caches in morph to treat the caches as bare repositories which are mirrors of where we clone from. In addition we correctly prune the branches during updates, so that we don't end up accumulating pointless branches over and over. This is even more important with branch-and-merge generating temporary build refs for things.
* build: Fail if user hasn't set user.name (or GIT_AUTHOR_NAME)Sam Thursfield2012-09-131-0/+12
| | | | | | Users in a VM may be running as 'root' so inferring from the username is often useless. Since build branches are pushed to a server it is useful to enforce useful onwership info.
* Git rid of morphlib.git methods that are not neededSam Thursfield2012-09-101-8/+0
|
* morph branch: Fail if branch already exists in the system repoSam Thursfield2012-09-101-0/+8
|
* Rewrite 'merge' commandSam Thursfield2012-09-101-1/+0
| | | | | | 'merge' now traverses every system in the branch root (baserock:morphs) and merges any chunk or stratum that was changed with 'morph edit'. It also takes care of updating the refs in the target branch.
* python scripts: pep8ize codebaseRichard Maw2012-08-011-8/+11
| | | | | | | | | This was done with the aid of the pep8 script, available by running `easy_install pep8`. It may be worth making this part of ./check, but that will require putting pep8 into the development tools stratum. This should be easy, given pep8 has no external dependencies.
* Remove optional msg argument from Submodule classLars Wirzenius2012-06-251-3/+2
| | | | | Nobody was using this, so it is obviously useless. Instead, log at the warning level.
* morph: remove dead code and replace Execute with app.runcmdRichard Maw2012-05-301-133/+20
|
* morphlib.git: fix submodule format stringRichard Maw2012-04-131-1/+1
|
* Remove final use of Treeish in submodules codeLars Wirzenius2012-04-121-3/+3
|
* Remove Treeish usage in submodule codeLars Wirzenius2012-04-121-24/+26
| | | | | This makes it possible to use the Submodule and Submodules classes in the refactored code that gets rid of Treeish completely.
* Make copy_repository get a repo instead of treeishLars Wirzenius2012-04-121-2/+2
| | | | This helps refactoring morph to get rid of treeishes.
* builder: ensure working directory matches commitRichard Maw2012-04-111-0/+7
| | | | | | | | When making a copy of the repository to build from, builder copies the .git directory then checks out the ref it wants. However sometimes this doesn't add the files to the working directory, possibly because the files it is missing weren't changed in the difference between what the .git thought HEAD was and what is now checked out.
* Refactor SystemBuilder and add helper classLars Wirzenius2012-03-021-1/+1
| | | | | | | | The helper class, Factory, has unit tests, which is why it's currently separate. It may later get integrated with BlobBuilder, or the other way around. Classes that don't have unit tests are marked out of coverage.
* Use "cp -a" and "git checkout" to unpack sources in the build tree.Jannis Pohlmann2012-02-221-11/+16
| | | | | | | | | This is done to keep the git repository intact with history and all that. cp + checkout turns out to be faster than a regular "git clone". In order to avoid roundtrips to the internet whenever a chunk tries to do something with submodules, we force the submodule URLs to point to our locally cached repos.
* Handle empty/short lines when validating submodulesLars Wirzenius2012-02-171-1/+1
|
* Switch to caching repos without checking them out.Jannis Pohlmann2012-02-141-5/+6
| | | | | | | | | | | | | This is almost as good as --bare. But unlike --bare, it properly sets up the 'origin' remote and creates the 'refs/remotes/origin/...' branches. When cloning with --bare, and you end up with "refs/heads/master", not "/refs/remotes/origin/master". You can re-add the remote 'origin', still no "refs/remotes/origin/master". Only after you update the remote (after re-adding it), the bare clone has "refs/remotes/origin/master"... and a "refs/heads/master" that is out of sync. An ugly mix, basically. So maybe this approach is better.
View Lorry Controller Status