summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* TEMP: import pylru from morphlibbaserock/adamcoldrick/build-graph-speedups-with-pylruAdam Coldrick2015-01-273-1/+537
| | | | | This should be installed as a chunk, but for now drop it into the morph source tree.
* Show a message when the build is startingbaserock/adamcoldrick/build-graph-speedupsAdam Coldrick2015-01-231-1/+1
|
* Add tests for sourceresolverAdam Coldrick2015-01-233-14/+341
| | | | | | | | | This only adds tests for the bits which were moved from morphologyfactory into sourceresolver, namely detection of build systems and the '_get_morphology()' function. These are just the morphologyfactory tests reworked slightly to work properly with the modified API.
* Read files from a local clone of definitions where possibleSam Thursfield2015-01-231-1/+30
| | | | | | | | | | | | | | | | | Most morphologies involved in a build are in the definitions repo these days. Currently we read each of them using `git cat-file`, which is slow. It's quicker to check out all the files in one go to a temporary directory and then read them from there. With the current workflow users often have definitions.git checked out on disk. It seems strange to not just read the files from there. There are two reasons why I don't want to do that yet: - there are commands which don't run inside a system branch, which would be broken if we expected to always be in a system branch - there may be local changes in the checked-out repo, and it takes around 5 seconds on each build to check if there aren't any local changes. It actually seems faster to just check out a known clean version from the cache.
* Add a mechanism for extracting all files from a given commit to a dirSam Thursfield2015-01-234-6/+89
| | | | | | This is nice because it's fast. We don't have to copy all the Git history along with it like we do with a clone. And it doesn't touch any files in the cached repo.
* Split up traverse_morphs to improve readabilitySam Thursfield2015-01-231-61/+79
|
* Add an LRU cache for detected build-systemsSam Thursfield2015-01-231-21/+101
| | | | | | | This will speed up builds of chunks which don't have a chunk morph. It won't have much (if any) effect on the speed of the first build, but subsequent builds will be much faster as we won't have to query the artifact cache.
* Add an LRU cache for resolved tree refsSam Thursfield2015-01-233-28/+73
| | | | | | | | | | | | This uses the PyLRU module, from: <https://pypi.python.org/pypi/pylru/1.0.6>. Python 3.2 and newer provide a built-in LRU cache, but this is specifically for in-memory use. See <http://bugs.python.org/issue17528>. Git commits are immutable, so caching information about their contents is fairly easy and trouble-free. There's no danger of the cache becoming stale.
* Add a cache manager utilising pylru and pickleAdam Coldrick2015-01-231-0/+51
|
* Move MorphologyFactory into SourceResolverSam Thursfield2015-01-234-381/+69
| | | | | There's no need for this stuff to be in a separate class. This allows integrating it with the caching in the SourceResolver class.
* sourceresolver: Add comments and factor out common functionSam Thursfield2015-01-231-13/+15
|
* Convert the empty-stratum test into a yarnAdam Coldrick2015-01-225-38/+35
| | | | | | | We will use MorphologyLoader.load_from_file() to load morphologies from a temporary file, so the output will be unpredictable. That will break this test, so we may as well convert it into a yarn to fix it.
* Fix some yarns to shut down their git servers when finishedAdam Coldrick2015-01-223-0/+3
| | | | | | This commit fixes an issue where running morph's yarn tests would leave behind a number of scripts/git-daemon-wrap processes due to some tests not stopping their git server properly.
* Merge branch 'baserock/jmalk/cross-bootstrap-aarch64-v2'Sam Thursfield2015-01-212-5/+5
|\ | | | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Rob Kendrick <rob.kendrick@codethink.co.uk> Reviewed-By: Paul Sherwood <paul.sherwood@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk> Reviewed-By: Zara Zaimeche <zara.zaimeche@codethink.co.uk>
| * Add armv8l64 and armv8b64 to list of valid architecturesJosh Malkinson2015-01-212-5/+5
|/
* Fix line > 79 characters that was breaking ./checkSam Thursfield2015-01-191-1/+2
|
* Merge branch 'sam/optional-temporary-build-branches'Sam Thursfield2015-01-193-26/+95
|\ | | | | | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
| * Make temporary build branches optionalSam Thursfield2015-01-063-26/+95
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | Not everyone is a fan of the `morph build` magic that collects up your changes and puts them in a temporary branch. Now you can disable it by setting 'local-changes=ignore' in your morph.conf file. This speeds up `morph build` and `morph deploy` by 5-10 seconds on my machine. I looked an option to make `morph build` warn if there are uncommitted changes. I found that with a cold cache, it takes about 5 seconds on my machine to verify that there are no uncommitted changes to a checkout of definitions.git. That defeats the main purpose of this patch for me, so I didn't include the option.
* | Update CopyrightRichard Ipsum2015-01-191-1/+1
| |
* | Correct resolution of error_message_for_containerised_commandline method.Craig Griffiths2015-01-161-1/+1
| |
* | Fix copyright yearsSam Thursfield2015-01-122-2/+2
| |
* | Merge branch 'sam/cached-repo-cleanup'Sam Thursfield2015-01-1217-318/+214
|\ \ | |/ |/| | | | | Reviewed-By: Adam Coldrick <adam.coldrick@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk>
| * Remove unused code from morphlib.gitSam Thursfield2014-12-031-24/+0
| | | | | | | | | | There is still code here that duplicates stuff the GitDirectory class should be doing, I think.
| * Rework CachedRepo to use the GitDirectory class where possibleSam Thursfield2014-12-0317-287/+199
| | | | | | | | | | | | | | | | | | | | | | | | | | This consolidates a bunch of code paths that were previously duplicated. This also changes the API for local cached repos to match the function names GitDirectory uses. Note that the remote repo cache still uses the old names, and should be fixed when time permits. Some unit tests that use the CachedRepo module required a bit of inelegant monkey-patching in order that they continue to work. A better way to do this would be with the 'mock' library (which would need to be added to Baserock 'build' and 'devel' systems before we could use it).
| * Separate resolution of ref->commit and commit->treeSam Thursfield2014-12-021-7/+15
| |
* | Merge remote-tracking branch 'origin/sam/rename-builder2'Sam Thursfield2015-01-055-10/+10
|\ \ | | | | | | | | | | | | Reviewed-By: Pedro Alvarez <pedro.alvarez@codethink.co.uk> Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk>
| * | Rename builder2 module to builderSam Thursfield2014-12-195-9/+9
| | |
* | | Merge remote-tracking branch 'origin/baserock/tiagogomes/sysroot-checks'Sam Thursfield2015-01-022-3/+31
|\ \ \ | | | | | | | | | | | | | | | | Reviewed-By: Paul Sherwood <paul.sherwood@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | | Add some checks to the sysroot deployment extensionTiago Gomes2015-01-022-3/+31
| | | | | | | | | | | | | | | | | | | | | | | | Ensure that a) the deployment directory must not exist b) the extension can not be used to upgrade a system
* | | | Merge branch 'sam/fix-command-logging'Sam Thursfield2015-01-021-16/+25
|\ \ \ \ | |/ / / |/| | | | | | | | | | | Reviewed-By: Paul Sherwood <paul.sherwood@codethink.co.uk> Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
| * | | Improve logic for displaying command execution to userSam Thursfield2015-01-021-16/+24
| | | | | | | | | | | | | | | | | | | | | | | | | | | | This avoids writing each command to the log file twice, as we did previously. Also, the user-visible message is now only constructed if we are definitely going to write it to the screen (a tiny optimisation). Hopefully the logic is clearer now too.
| * | | Fix logging and display of commandlines being executed by MorphSam Thursfield2015-01-021-6/+7
|/ / / | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | We were attempting to display commandlines with correct shell-escaping, but the logic was super broken so users would end up seeing this sort of thing instead: 2015-01-02 10:25:42 # g | i | t | | r | e | v | - | p | a | r | s | e | | - | - | v | e | r | i | f | y | | ' | 9 | 8 | f | e | a | 8 | 7 | b | 7 | 2 | 7 | 2 | 5 | 3 | e | 7 | f | f | 8 | 1 | 0 | 5 | 4 | 3 | 4 | c | 9 | e | a | 9 | 0 | 2 | b | b | 6 | a | 6 | f | 7 | e | ^ | { | c | o | m | m | i | t | } | ' Commandlines should now display as intended, more like this: 2015-01-02 10:57:17 # git rev-parse --verify '9df9643842e4b4d8ece710fe6105f32fa38a0d22^{commit}' This broken logic was introduced as a post-review fixup in merge commit c57952ef44a0f1f161441970fcf2f27a39b0de7c.
* | | Merge branch 'emmet/armv8'Sam Thursfield2014-12-192-5/+19
|\ \ \ | | | | | | | | | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | | Support armv8l, armv8b, aarch64, and aarch64bEmmet Hikory2014-12-192-5/+19
|/ / / | | | | | | | | | | | | | | | Also add support to allow building compatible architectures on armv8 machines, as per the rationale in the comment in _validate_architecture().
* | | Merge branch 'baserock/richardmaw/bugfix/stagingarea-mounts-inside-destdir-v2'Sam Thursfield2014-12-191-3/+10
|\ \ \ | |/ / |/| | | | | Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | Don't mount /dev/shm in bootstrap mode.Sam Thursfield2014-12-191-3/+4
| | | | | | | | | | | | | | | | | | There's no chroot used in bootstrap mode, so the host's /dev/shm can be used, and nothing in the world will be looking to /src/tmp/staging/tmpXXX/dev/shm for anything.
| * | stagingarea: Mount things inside dirnamebaserock/richardmaw/bugfix/stagingarea-mounts-inside-destdir-v2Richard Maw2014-12-121-1/+6
| | |
* | | Merge remote-tracking branch 'petefoth/petefoth/generic-params2'Sam Thursfield2014-12-186-15/+125
|\ \ \ | | | | | | | | | | | | | | | | Reviewed-By: James Thomas <james.thomas@codethink.co.uk> Reviewed-By: Sam Thursfield <sam.thursfield@codethink.co.uk>
| * | | Add 'do not use' warnings to nfsboot write extensionPete Fotheringham2014-12-152-3/+19
| | | |
| * | | Document KERNEL_ARGS write extension parameterPete Fotheringham2014-12-154-0/+33
| | | |
| * | | Whitespace removalPete Fotheringham2014-12-151-8/+8
| | | |
| * | | Document BOOTLOADER_INSTALL and BOOTLOADER_CONFIG_FORMAT write extension ↵Pete Fotheringham2014-12-154-3/+50
| | | | | | | | | | | | | | | | parameters
| * | | Document DTB_PATH write extension parameterPete Fotheringham2014-12-153-1/+15
| |/ /
* | | Merge branch 'sam/improve-command-failure-errors'Sam Thursfield2014-12-184-35/+76
|\ \ \ | | | | | | | | | | | | | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Richard Maw <richard.maw@codethink.co.uk> Reviewed-By: Daniel Silverstone <daniel.silverstone@codethink.co.uk> Reviewed-By: Mike Smith <mike.smith@codethink.co.uk>
| * | | Tiny optimisation of app.runcmd code pathSam Thursfield2014-12-091-9/+8
| | | |
| * | | Give a useful path to failed staging areasSam Thursfield2014-12-091-4/+7
| | | | | | | | | | | | | | | | | | | | | | | | Morph tells the user that an error occurred in the staging area, then moves the staging area somewhere else. Giving the old path rather than the new path is pretty annoying.
| * | | Remove duplicate 'import' statementSam Thursfield2014-12-091-1/+0
| | | |
| * | | Give less scary error messages when a containerised command failsSam Thursfield2014-12-094-22/+61
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This affects errors encountered at build time and at system-integration time. New errors look like this: ERROR: Command failed: baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000: Containerisation settings: {'mounts': (('dev/shm', 'tmpfs', 'none'), ('tmp', 'tmpfs', 'none')), 'mount_proc': True, 'root': '/var/tmp/staging/tmp1YQ2yN/minimal-system-x86_64-generic.inst'} Error output: + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted Previously the error message would have been this: Command failed: unshare --mount -- sh -ec. mount --make-rprivate / root="$1" shift while true; do case "$1" in --) shift break ;; *) mount_point="$1" mount_type="$2" mount_source="$3" shift 3 path="$root/$mount_point" mount -t "$mount_type" "$mount_source" "$path" ;; esac done exec "$@" - /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ dev/shm tmpfs none tmp tmpfs none -- linux-user-chroot --chdir . --mount-proc proc /var/tmp/staging/tmppeA1Iw/gerrit-x86_64.inst/ baserock/system-integration/02-install-gerrit-gerrit-installation-binaries-misc-0000 + install -D /usr/share/gerrit/gerrit-2.9.war /home/gerrit2/gerrit/gerrit-2.9.war -o gerrit2 -g gerrit2 -m 644 install: can't change ownership of /home/gerrit2/gerrit/gerrit-2.9.war: Operation not permitted
* | | | Remove trailing \Sam Thursfield2014-12-181-1/+1
| |/ / |/| | | | | | | | I meant to do this as part of the previous merge.
* | | Merge branch 'sam/fix-openstack-credentials-check'Sam Thursfield2014-12-101-4/+11
|\ \ \ | | | | | | | | | | | | | | | | Reviewed-By: Richard Ipsum <richard.ipsum@codethink.co.uk> Reviewed-By: Francisco Redondo Marchena <francisco.marchena@codethink.co.uk>