summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* Resolve stratum and chunk refs in batchesjannispohlmann/resolve-refs-and-morphologies-in-batchesJannis Pohlmann2013-01-072-7/+98
| | | | | | | | | | | | | | This uses the functionality introduced in the previous commit to reduce the number of HTTP requests made during builds and generally while creating source pools. I noticed a weird thing in resolve_ref() while I wrote the corresponding batch function resolve_refs(): unless I'm misreading the code the repository caching at the end of resolve_ref() clones repositories from the remote repo cache that already failed to resolve the needed ref remotely. How would they be useful after cloning? Anyway, resolve_refs() is implemented in the same way but I added a FIXME note.
* Add RemoteRepoCache.resolve_refs() method with testsJannis Pohlmann2013-01-073-1/+124
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This method is for batch-resolving of refs into SHA1s using the remote repo cache. It takes a list of (repo, ref) tuples and returns dictionary that maps these tuples to dictionaries that look as follow in successful cases { 'repo': <original repo>, 'ref': <original ref>, 'repo-url': <expanded full repo URL>, 'sha1': <resolved SHA1>, 'tree': <tree SHA1 of the commit>, } and like this in negative cases: { 'repo': <original repo>, 'ref': <original ref>, 'repo-url': <expanded full repo URL>, 'error': <error string>, } The mapping may seem redundant but it might be useful when looking up the results. If tuples are passed in they will be available at the caller's end already.
* Merge remote-tracking branch 'origin/jonathanmaw/ccache-local-fix'Lars Wirzenius2013-01-032-5/+16
|\ | | | | | | Fixed a copyright year to make ./check happy.
| * Make ccache use the same directory for repos that are clonesJonathan Maw2013-01-031-4/+15
| | | | | | | | | | | | | | | | | | This patch makes morph store its ccaches by the url's basename instead of using the whole url. This fixes the problem of morph choking when it tries to mount a directory with a colon in its path, and also makes all clones of one repository use the same cache (probably). This will cause projects with the same name to use the same cache, but that is not likely to cause serious problems.
* | Merge remote-tracking branch 'origin/samthursfield/stable-glob'Lars Wirzenius2013-01-031-2/+2
|\ \ | |/ |/|
| * Fix sporadic test failures caused by the unstable ordering of glob.glob()Sam Thursfield2013-01-031-2/+2
|/ | | | | | | | glob.glob() can return results in any order. In practice it varies at least according to the file system being used. This means that test results may be different. To avoid this, we should always use sorted(glob.iglob()) instead, so that the code always behaves in the same way given the same inputs.
* Merge branch 'liw/silence-log-messages-during-unit-tests'Sam Thursfield2012-12-182-6/+8
|\
| * Silence log messages during unit testsLars Wirzenius2012-12-182-6/+8
| |
* | Merge branch 'samthursfield/stratum-build-depends'Sam Thursfield2012-12-187-73/+285
|\ \ | |/ |/| | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Add a stratum build-dependency to all as-root testsSam Thursfield2012-12-182-0/+13
| |
| * morph build: Update repo and ref in stratum build-depends fieldsSam Thursfield2012-12-182-0/+44
| |
| * morph edit: Update build-depends that refer to the edited stratumSam Thursfield2012-12-183-73/+209
| | | | | | | | | | | | | | The code took some refactoring. The core functionality is now all inside one function with make_available() separate, as this is used other places. The code is still far from perfect, but will hopefully be rewritten to use the new abstractions of system branches etc. soon
| * Validate stratum build-depends against system contentsSam Thursfield2012-12-171-0/+19
|/ | | | | This ensures that we don't try to build the same stratum from two different refs.
* Fix default setting for compiler cache dir; update testsJannis Pohlmann2012-12-172-12/+7
| | | | Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
* Add support for ccacheJonathan Maw2012-12-173-17/+30
| | | | | | | | | | | | | | | | | | | This is set up so that each individual project repository has its own ccache, all under one defined directory. The top-level ccache directory is added as the setting 'compiler-cache-dir', and defaults to $cachedir/ccache. When a build is performed, this will bind-mount a project's ccache into the /tmp/ccache of the staging-area and set up the environment variables so that ccache will be used (if appropriate executables are installed to /usr/lib/ccache in the staging-area). In addition, this removes code for ccache-remotedir, as it is unrelated to this implementation of ccache. Reviewed-by: Jannis Pohlmann <jannis.pohlmann@codethink.co.uk> Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
* Drop committer info from git tag environment; update tag test outputsJannis Pohlmann2012-12-178-28/+30
| | | | | | | | | | | The committer information in the environment used to run git in morph tag is not needed. In morph build it makes sense as morph commits without the user knowing. With morph tag, it's the user that decides to create the commit and tag. There is something weird going on, where morph tag may end up generating commits with different SHA1s on different machines. The full log output in the morph tag tests might help investigate what happens.
* Merge branch 'jannispohlmann/cache-resolved-refs-and-morphologies'Jannis Pohlmann2012-12-141-4/+18
|\ | | | | | | Reviewed-by: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| * Cache resolved refs and morphologies when traversing morphsJannis Pohlmann2012-12-131-4/+18
| | | | | | | | | | | | | | This cuts down on the number of HTTP requests made while computing th build order (> create source pool > traverse morphs). In an example system I was building this patch reduced the requests from 501 to 195, reducing the time to building the first chunk from 123s to 18s.
* | Merge remote-tracking branch 'remotes/origin/samthursfield/ambiguous-refs' ↵Jannis Pohlmann2012-12-137-56/+79
|\ \ | | | | | | | | | into baserock/merge-queue
| * | morph branch: Check if the ref already exists before anything elseSam Thursfield2012-12-133-6/+20
| | | | | | | | | | | | | | | This was done to ensure tests.branching/branch-fails-if-branch-exists always passes, but also seems like the right approach in general.
| * | Add a test to catch any future use of git show-refSam Thursfield2012-12-131-0/+32
| | |
| * | Always use `git rev-parse --verify` to resolve refsSam Thursfield2012-12-136-50/+27
| |/ | | | | | | | | | | | | | | | | | | | | | | | | Previously some code used `git show-ref`, which is wrong -- given two refs named 'alpha/master' and 'master', `git show-ref master` will return both, sorted alphabetically. This can lead to build failures, etc. due to refs resolving to the wrong SHAs. We should also use `git rev-parse --verify` to verify SHA1s, which we previously did with `git rev-list`. Finally, `git rev-parse --verify` is more than twice as fast as `git show-ref`.
* | Fix morph tag usage of git commit-tree; fix detecting existing tagsJannis Pohlmann2012-12-131-12/+23
| |
* | Fix bashism in morph tag test scriptsJannis Pohlmann2012-12-134-4/+4
| |
* | Merge remote-tracking branch 'origin/jannispohlmann/morph-tag-v2'Sam Thursfield2012-12-1311-0/+746
|\ \ | |/ |/|
| * Add an initial implementation of "morph tag"Jannis Pohlmann2012-12-1311-0/+746
|/ | | | | | | | | | | | | | | | | | | | | | | | | | In order to make releases and freeze system branches entirely, we need to be able to 100% petrify a system branch (that is, resolve ALL refs into SHA1s) and tag this state to be able to check it out again later. This is essentially what "morph tag" does. It takes a tag name and an arbitrary amount of arguments to "git tag", petrifies all morphologies of the current system branch behind the scenes, creates a dangling commit and attaches an annotated tag to it. Petrifying in this case means that all refs used for chunks are resolved into commit SHA1s. For stratum and system morphologies, the refs are replaced by the name of the tag that's being created. The "tag" command also supports tagging when stratum morphologies are spread across multiple repositories. In this case, it will include all statum morphologies from other repos in the tag commi in the branch root repo. The references to these morphologies are updated so that they point to the branch root repo and the tag being created. This commit also adds a few tests for "morph tag" to verify that all this works.
* Merge branch 'jonathanmaw/find-artifacts-rebase' of ↵Richard Maw2012-12-101-0/+123
|\ | | | | | | ssh://trove.baserock.org/baserock/baserock/morph
| * Add find-artifacts scriptJonathan Maw2012-12-101-0/+123
| | | | | | | | | | This script reads a built baserock system and finds all the chunks used to build it in the artifact cache
* | Merge branch 'jonathanmaw/cache-key-less-rigorous'Sam Thursfield2012-12-072-0/+49
|\ \
| * | Avoid rebuilds when unimportant morphology contents changeJonathan Maw2012-11-292-0/+49
| | | | | | | | | | | | | | | | | | | | | | | | This changes the cache key generation so that it will ignore unimportant fields of the morphology, e.g. description, build-depends, chunks. description is unimportant because it does not affect building, and build-depends/chunks are unimportant because they are already considered
* | | Merge branch 'baserock/richardmaw/build-less-artifacts'Richard Maw2012-12-032-4/+19
|\ \ \ | |_|/ |/| | | | | Rubber stamped because the usual reviewers are on holiday.
| * | Test source pool having duplicates addedRichard Maw2012-12-031-0/+9
| | | | | | | | | | | | | | | The lookup method will only find the first entry, but previously would allow later entries to be found by iterating.
| * | Ensure sourcepool does not contain duplicatesRichard Maw2012-12-031-2/+3
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Before this change, if a stratum had dependencies listed, then morph would add that stratum to the list of things to build multiple times, resulting in more things being built. This was not a major problem, since it would just result in more cache lookups than necessary, but lookups are more expensive on remote artifact caches, and anything attempting to find which artifact would be the result of a build would get confused by there being many top-level artifacts with no dependencies.
| * | Amend test to not add the same source to the pool twiceRichard Maw2012-12-031-2/+7
|/ / | | | | | | | | | | | | | | The test relied on undesired behaviour, since it would create the same source, since the sha1 is not used in the sourcepool's key. Now it creates a new source pool and replaces the source it is being compared to.
* | Revert "Merge branch 'liw/hardlink-staging-area-pre-rebase'"Jannis Pohlmann2012-11-303-89/+33
|/ | | | | | | | | | | | | | This reverts commit c859d70d86423a52bc7053abf64e9ca21f62a487, reversing changes made to 3085a672d1e8b5177be33f0463385de72a0ef5bf. Unfortunately, hardlinking and linux-user-chroot both break builds in various ways. Hardlinking breaks the bootstrap process by creating symlinks like /usr/libexec that can then not be overwritten with real files by install scripts from morphologies like gcc. linux-user-chroot caused problems by breaking privileged operations such as chgrp and CAP_SETFCAP. As a consequence, chunks like util-linux and libcap can no longer be built.
* Merge branch 'liw/hardlink-staging-area-pre-rebase'Lars Wirzenius2012-11-293-33/+89
|\ | | | | | | | | | | | | | | This is based on, and incorporates, Joe Burmeister's changes to fill staging areas using hardlinking, for speed reasons. I have reformatted the code a bit, and fixed it to pass the test suite, and fixed a bug in how commands are run in the staging area chroot (preserving the environment).
| * Change method comments to docstringsLars Wirzenius2012-11-291-3/+6
| | | | | | | | This also makes coverage.py be silent on Debian squeeze.
| * Use the builder runcmd, to get build environment rightLars Wirzenius2012-11-291-6/+4
| |
| * Reformat code, plus adapt for non-staging-area buildsLars Wirzenius2012-11-292-16/+22
| |
| * Refactor and fix to make it pass (more of) the test suiteLars Wirzenius2012-11-292-10/+29
| |
| * Hardlinked chroot done with linux-user-chrootJoe Burmeister2012-11-292-34/+64
|/ | | | | | | | | | | | | | The patch gives two things. Improves morph build time by reusing decompressed files of chunks/stage-fillers with hardlinks from the chroot. Rather than decompressing each time into each chroot. Original: real 5h 17m 47s Hardlink: real 2h 52m 27s It uses linux-user-chroot to create the chroot and make all but the basics readonly.
* Fix line longer than 79 charsSam Thursfield2012-11-261-1/+1
|
* Fix another baserock: URLLars Wirzenius2012-11-211-1/+1
|
* Fix URLs in example in READMELars Wirzenius2012-11-211-2/+2
|
* Merge branch 'master' of git://git.baserock.org/baserock/baserock/morphLars Wirzenius2012-11-211-1/+2
|\
| * Merge branch 'samthursfield/remove-gzip-file-header'Sam Thursfield2012-11-161-1/+2
| |\ | | | | | | | | | Reviewed-By: Lars Wirzenius <lars.wirzenius@codethink.co.uk>
| | * Don't store original filename in compressed image filesSam Thursfield2012-11-161-1/+2
| |/ | | | | | | | | | | | | gzip files can store the name and date of the uncompressed original, and this is displayed in some GUI archive managers, or when gunzip -N is used. In our case the original filename is something like 'tmpXsgaY15', which is useless information and serves only to confuse.
* | Fix baserock: aliased URLs to point at new locationLars Wirzenius2012-11-211-6/+6
|/
* Merge branch 'samthursfield/handle-target-disk-full' of ↵Richard Maw2012-11-155-25/+101
|\ | | | | | | | | | | | | | | | | | | | | git://git.baserock.org/baserock/morph A couple of nasties are fixed in here. Previously when the disk was full, Morph logged a backtrace into morph.log and then tried to unmount the disk image, but as there were still open file handles inside it the unmount would fail and the user would end up with a backtrace for a failed unmount and a stuck loopback device that they would need to fix manually.
| * Cleanly handle out of disk space on target filesystem.Sam Thursfield2012-11-154-1/+60
| | | | | | | | Includes new test.
View Lorry Controller Status