| Commit message (Collapse) | Author | Age | Files | Lines |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
tar will preserve ownership and permissions by default
if running as the superuser, whereas git doesn't track ownership,
the owner is set to the user running the clone.
This is a reproducibility hole since the resulting repo may differ
depending on whether the repo was obtained from a tarball or via a git
clone, but morph considers the repos to be equivalent.
Running tar with --no-same-owner ensures the files
written by tar will be owned by the user extracting the archive.
This only fixes half the problem,
lorry may also need to be modified to ensure files in the
tarball have the same mode as those in the git repo.
Change-Id: I849d054bc4574f9c0bbcfd608914306e52b7a22e
|
|
|
|
|
|
|
| |
This fixes requires_update_for_ref when it is given a non-fixed ref and
it is retrieved from a call to get_updated_repo.
Change-Id: Icbcf2e4ec45bd559c6bcb450ceb4a8861b56f654
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
It turns out I completely broke Morph's handling of recursive submodules
in commit 0855c357e74e6dd7a.
We didn't notice because nothing in the reference systems actually uses
recursive submodules. However, building the baserock/tlsa/mason2 branch
of definitions.git triggered the bug due to the
delta:python-packages/gitpython chunk, which contains
delta:python-packages/gitdb which in turn contains
delta:python-packages/smmap.
The error is this:
Traceback (most recent call last):
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 190, in _run
self.process_args(args)
File "/src/morph/morphlib/app.py", line 290, in process_args
cliapp.Application.process_args(self, args)
File "/usr/lib/python2.7/site-packages/cliapp/app.py", line 539, in process_args
method(args[1:])
File "/src/morph/morphlib/plugins/build_plugin.py", line 291, in build
self._build(source_pool, filename, component_names=component_names)
File "/src/morph/morphlib/plugins/build_plugin.py", line 316, in _build
bc.build_in_order(component)
File "/src/morph/morphlib/buildcommand.py", line 296, in build_in_order
self.cache_or_build_source(s, build_env)
File "/src/morph/morphlib/buildcommand.py", line 316, in cache_or_build_source
self.build_source(source, build_env)
File "/src/morph/morphlib/buildcommand.py", line 336, in build_source
self.fetch_sources(source)
File "/src/morph/morphlib/buildcommand.py", line 400, in fetch_sources
self.lrc.ensure_submodules(source.repo, source.sha1)
File "/src/morph/morphlib/localrepocache.py", line 308, in ensure_submodules
if (submod.url, submod.commit) not in done:
AttributeError: 'tuple' object has no attribute 'url'
This commit fixes the breakage and the baserock/tlsa/mason2 branch of
definitions.git now builds again.
Change-Id: Id24ac40f4670a60655e84953bcfd84d8f77e1da9
|
|
|
|
|
|
|
|
|
|
| |
Passing a single ref is still accepted, but if you have multiple refs
you need to check from the same repository, it is more appropriate to do
it in one call to get_updated_repo, as otherwise there will be
unnecessary output about it not needing to be updated in multiple
places.
Change-Id: I194d7c0e3e84c4917518ba37672b508505c71b8e
|
|
|
|
|
|
|
| |
This commit undoes behaviour changes from commit
aa6dfcbb70c03dfeb3f9af02.
Change-Id: Ie677fb9c4e6bcd6edeba2cdd87f4f6125dcae7a4
|
|
|
|
|
|
|
|
|
|
|
| |
Its going to be more likely that a baserock user is more interesed
in the build log than actually the debug output from morph.
And also the intuitive option would be to use -v for this.
--verbose/-v: show build output
--debug/-d: show morph debug output
Change-Id: I1fb99034dc8680a5f168f6306724663aea33ebc5
|
|
|
|
| |
Change-Id: Ibda7a938cd16e35517a531140f39ef4664d85c72
|
|
|
|
| |
Change-Id: I992dc0c1d40f563ade56a833162d409b02be90a0
|
|\
| |
| |
| |
| | |
Reviewed-By: Javier Jardón <javier.jardon@codethink.co.uk>
Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
|
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This saves a duplicate `git remote update origin` that was being run as
part of each chunk build. For any repos that have submodules, it also
avoids updating repos if the SHA1 we need to build is already present
locally.
As well as speeding up builds slightly, this means Morph can now build
without being connected to a network, as long as the local Git cache
all of the necessary repos and commits in the build, without needing the
'--no-git-update' option.
The code is also now in a more logical place than before.
|
|/
|
|
|
|
|
| |
Also, move the repo.update() call into the 'fetch from tarball' code
path in the localrepocache module -- there's no need to update straight
after doing a `git clone`, but we do need to do one if we got the repo
from a `git archive` tarball that may be out of date.
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This consolidates a bunch of code paths that were previously duplicated.
This also changes the API for local cached repos to match the
function names GitDirectory uses. Note that the remote repo cache still
uses the old names, and should be fixed when time permits.
Some unit tests that use the CachedRepo module required a bit of
inelegant monkey-patching in order that they continue to work. A better
way to do this would be with the 'mock' library (which would need to be
added to Baserock 'build' and 'devel' systems before we could use it).
|
|
|
|
|
|
|
|
|
|
|
|
| |
Morph can appear to hang in situations where it is actually waiting on a
slow network operation. This change gives users a way to see the output
of the subcommands that are doing the network IO (either 'wget', 'git
clone' or 'git remote update').
The status information goes onto stderr, because that is where the
subcommands write it. Morph tends to put its status output on stdout,
but (a) some commands are machine-parsed, such as `serialise-artifact`
and (b) it's tricky to get Git to put status output on stdout.
|
|
|
|
|
|
|
| |
This fixes an issue where various branch-and-merge commands appeared to
be updating Git repos even when --no-git-update was specified. The flag
was actually honoured but the message made it seem as if it was being
ignored.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
We assumed that the sha1 of the tree of the commit of the ref we care
about was sufficient to cache, but `git replace` means that you need to
know the state of other branches too, since anything in refs/replace can
completely change what the tree you check-out is.
This behaviour can be disabled globally by setting
GIT_NO_REPLACE_OBJECTS, so we're going to do that.
If we need to integrate a project that uses git-replace to change the
contents of their git trees then we could support that by:
if any(refs/replace/*):
potentially_replacable_objects = [
`git rev-parse HEAD`,
`git rev-parse HEAD^{commit}`,
`git rev-parse HEAD^{tree}`]
potentially_replacable_objects.extend(
`git ls-tree -r HEAD | awk '{print $3}'`)
# NOTE: doesn't handle submodules, and you'd need to expand this
# set whenever you process a replacement
for object in refs/replace/*:
if basename(object) not in potentially_replacable_objects: continue
cache_key['replacements'][basename(object)] = `git rev-parse $object`
If we were to support this would need to traverse the tree anyway, doing
replacements, so we may as well use libgit to do the checkout anyway,
and list which replacements were used.
However, since the expected use-case of `git replace` is as a better way
to do history grafting, we're unlikely to need it, as it would only have
any effect if it replaced the commit we were using with a different one.
Rubber-stamped-by: Daniel Silverstone
|
| |
|
|
|
|
|
| |
This is a helper to avoid having the same code in other places. Have
it in the one place it belongs.
|
|
|
|
|
| |
Currently the message is still displayed "Updating xxx" but no
update is actually done.
|
|
|
|
|
|
|
| |
This makes a non-backward-compatible change to morph which switches it to using
tarballs instead of bundles when initialising cached git repositories. This is
faster because it doesn't require index-pack --fix-thin operations on the
machine running morph.
|
|
|
|
|
|
| |
Log (at chatty) when we start and finish bundle downloads. If we fail for any
reason, log that reason since otherwise it gets swallowed if the plain git
clone succeeds.
|
|
|
|
|
|
|
|
| |
This reworks the code for managing and using the git caches in morph to treat
the caches as bare repositories which are mirrors of where we clone from. In
addition we correctly prune the branches during updates, so that we don't end
up accumulating pointless branches over and over. This is even more important
with branch-and-merge generating temporary build refs for things.
|
|
|
|
|
|
|
|
|
| |
This was done with the aid of the pep8 script, available by running
`easy_install pep8`.
It may be worth making this part of ./check, but that will require
putting pep8 into the development tools stratum.
This should be easy, given pep8 has no external dependencies.
|
| |
|
| |
|
|
|
|
| |
Also, add debugging messages for debugging prefix expansion.
|
|
|
|
| |
The tests, they still fail.
|
|
|
|
|
|
|
|
| |
Changed the delimieter between push and pull patterns to be #,
since = is reasonably frequent in URLs, and # should never be
necessary in git urls.
Oh my how many tests now fail.
|
| |
|
| |
|
|
|
|
|
| |
This way, cliapp will automatically print an error message, rather
than a stack trace.
|
| |
|
| |
|
| |
|
| |
|
|
|
|
|
|
| |
It needed shutil, but that exception was being missed.
_git needs to be able to handle a different current directory for
remote set-url to work
|
|
|
|
|
|
| |
When cloning from a bundle it makes the origin remote url point to the
bundle. This is not ideal as we want to update these repositories later
from upstream.
|
|
|
|
| |
They must end with .bndl as that is the format lorry generates
|
|
|
|
|
|
|
|
| |
Also change _fetch to throw its exception as since it is more pythonic
to throw exceptions than return exit status, other error handling needed
to be caught, so it hsa been kept consistent.
This required the tests to throw an exception as well
|
|
|
|
|
| |
It should check that it doesn't already have the repository cached
under a different base-url before cloning, likewise with bundles.
|
| |
|
|
|
|
|
|
|
|
| |
The self._exists method needs to be used instead of raw os.path.exists,
so that unit tests can override it easily.
Also, a tiny clarification so that the caller of _clone_with_bundle
decides whether it's OK to try to download a bundle.
|
| |
|
| |
|
| |
|
| |
|
|
Jannis and I discussed a refactoring of most of morph's internals
to reduce coupling and increase cohesion. This is one of the results:
we'll want a couple of classes to manage locally cached git repositories.
This commit adds the LocalRepoCache class to manage a the git cache
directory. Later on, we'll add the CachedRepo class to represent
individual repositories.
|