summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* element.py: Only cache on element error or successbschubert/dont-cache-errors-from-host-toolsBenjamin Schubert2018-11-191-18/+5
| | | | | | Reverse the way we were handling caching by only caching when we know it's a problem with the build or the build is a success in order not to cache transient errors.
* Don't cache sandbox errorsBenjamin Schubert2018-11-193-90/+147
| | | | | | | | | | Sandbox errors (like missing host tools) are dependent on the host system and rarely on what is actually done. It is therefore better to not cache them as they are subject to change between two runs. Also add test to ensure sandbox failure are not cached
* Merge branch 'tpollard/494' into 'master'Jürg Billeter2018-11-1712-62/+393
|\ | | | | | | | | Don't pull artifact buildtrees by default See merge request BuildStream/buildstream!786
| * Don't pull artifact build trees by default.tpollard/494Tom Pollard2018-11-177-60/+378
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | The addition of cached build trees being included in element artifacts has led to mostly redundant download overheads when pulling from a remote artifact server. As such the default behaviour of pull shouldn't fetch the build tree object if available. element.py: extend relevant pull logic for specified subdir consideration and ensure push logic does not lead to partial artifact pushes. Change assumption that the buildtree can be extracted if artifact is cached. __cached_buildtree() and __pull_directories() helpers added. _artifactcache/: artifactcache.py & cascache.py inclusion of helper functions for subdir artifact checking & fetching, fetch logic extended to only pull required artifact directories. extract & checkout updated to handle full/partial operation. tests/: addition of integration test pullbuildtrees.py, buildtrees.py adapted cli options, testutils/artifactshare.py has_artifact changed to return artifact digest on true condition.
| * Add cli main and user config option for 'pull-buildtrees' context.Tom Pollard2018-11-175-2/+15
|/ | | | | | | | | | | | _context.py: Add pull_buildtrees global user context, the default of which is set to False via the addition of pull-buildtrees to userconfig.yaml cache group. _frontend/app.py & cli.py: Add --pull-buildtrees as a bst main option, which when passed will override the default or user defined context for pull_buildtrees. tests/completions/completions.py: Update for the added flag.
* Merge branch 'aevri/unit_tests' into 'master'Jürg Billeter2018-11-151-0/+18
|\ | | | | | | | | contributing: add guidance on unit tests See merge request BuildStream/buildstream!943
| * contributing: add guidance on unit testsAngelos Evripiotis2018-11-151-0/+18
|/ | | | | Decrease uncertainty around whether unit tests are welcome in the project or not.
* Merge branch 'richardmaw/distinguish-sandboxing-build-fail' into 'master'richardmaw-codethink2018-11-148-42/+136
|\ | | | | | | | | | | | | Distinguish between bubblewrap sandboxing failure and command failure Closes #286 See merge request BuildStream/buildstream!868
| * tests/integration/sandbox-bwrap.py: Test distinguishing sandbox exit code ↵Richard Maw2018-11-144-1/+58
| | | | | | | | from command
| * tests/testutils/site.py: Check for bwrap supporting --json-status-fdRichard Maw2018-11-141-1/+3
| |
| * buildstream/sandbox/_sandboxbwrap.py: Distinguish sandbox failure from ↵Richard Maw2018-11-142-20/+52
| | | | | | | | | | | | | | | | | | | | | | command failure If `bwrap` fails to set up the sandbox and start the payload command it won't write an exit-code in --json-status-fd, so we can report if it was a sandboxing failure if we don't get exit-code status and a payload command failure if we do and it's non-zero. Closes https://gitlab.com/BuildStream/buildstream/issues/286
| * _platform/linux.py: Move get_bwrap_version into _site.pyRichard Maw2018-11-142-20/+23
|/ | | | | It's inconvenient to have to create a Linux platform to parse the bwrap version and we want to get the version in a consistent manner.
* Merge branch 'valentindavid/integration-tests-aarch64' into 'master'Javier Jardón2018-11-1412-17/+72
|\ | | | | | | | | | | | | Run tests on aarch64 Closes #369 and #755 See merge request BuildStream/buildstream!948
| * Test aarch64 only overnightValentin David2018-11-141-3/+11
| |
| * Add support for aarch64 in a testValentin David2018-11-141-1/+1
| |
| * Disable tests on example on other architectures than x86_64Valentin David2018-11-146-6/+32
| |
| * Disable cachekey tests on other architectures than x86_64Valentin David2018-11-142-1/+5
| |
| * Run tests on aarch64Valentin David2018-11-143-7/+24
|/ | | | Fixes #755
* Merge branch 'element-path_not_validated' into 'master'Jürg Billeter2018-11-123-10/+33
|\ | | | | | | | | Element path not validated before use See merge request BuildStream/buildstream!937
| * tests/frontend: Add test for invalid element-pathelement-path_not_validatedPhillip Smyth2018-11-122-0/+17
| |
| * _project.py: Validate nodes early in Project._loadPhillip Smyth2018-11-121-10/+16
|/ | | | | | _project.py: Added validate_nodes() helper function to prevent duplicate lists element-path was being used before node validation resulting in uncaught errors
* Merge branch 'bschubert/fix-silence-stopiteration-pep-0479' into 'master'richardmaw-codethink2018-11-121-20/+22
|\ | | | | | | | | source.py: don't let StopIteration propagate to silence() contextmanager See merge request BuildStream/buildstream!945
| * source.py: don't let StopIteration propagate to silence() contextmanagerbschubert/fix-silence-stopiteration-pep-0479Benjamin Schubert2018-11-091-20/+22
|/ | | | | | As per PEP 0479 (https://www.python.org/dev/peps/pep-0479/), StopIteration thrown in context managers are not valid starting from Python 3.7.
* Merge branch 'jmac/cas_to_cas_v2' into 'master'Jim MacArthur2018-11-092-94/+652
|\ | | | | | | | | | | | | Direct CAS-to-CAS import Closes #574 See merge request BuildStream/buildstream!911
| * virtual_directory_import.py: New testjmac/cas_to_cas_v2Jim MacArthur2018-11-091-0/+271
| |
| * _casbaseddirectory: Fix order of list_modified_pathsJim MacArthur2018-11-091-9/+26
| | | | | | | | This will now return paths in the same order as list_relative_paths.
| * _casbaseddirectory.py: Enable direct CAS-to-CAS import.Jim MacArthur2018-11-091-85/+355
|/ | | | Fixes issue #574.
* Merge branch 'danielsilverstone-ct/plugin-destroy-debug-to-stderr' into 'master'Valentin David2018-11-091-1/+2
|\ | | | | | | | | plugin.py: Redirect DEBUG from `__del__` to `sys.stderr` See merge request BuildStream/buildstream!944
| * plugin.py: Redirect DEBUG from `__del__` to `sys.stderr`danielsilverstone-ct/plugin-destroy-debug-to-stderrDaniel Silverstone2018-11-091-1/+2
|/ | | | | | | | | Previously, debug messages from `Plugin.__del__()` went to stdout which was a problem for the test suite in extremely rare and hard to predict circumstances. This corrects that by sending the message to stderr where all the rest of the messages tend to go. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Merge branch 'jennis/quota_declaration_fix' into 'master'Javier Jardón2018-11-083-7/+53
|\ | | | | | | | | | | | | Add local cache expiry documentation and fix misleading error message when specifying a percentage cache quota Closes #700 See merge request BuildStream/buildstream!939
| * artifactcache.py: Fix misleading error message when using % cache quotajennis/quota_declaration_fixJames Ennis2018-11-082-7/+15
| | | | | | | | | | Due to the changed Exception message, this patch also changes the test_parse_size_over_1024T test in misc.py
| * using_config.rst: Add documentation to showing how to impose quotas on the ↵James Ennis2018-11-081-0/+38
|/ | | | | | local cache This patch partially resolves #700
* Merge branch 'aevri/contributing_gitlab' into 'master'Angelos Evripiotis2018-11-081-1/+23
|\ | | | | | | | | Add more to GitLab-relevant parts of contributing See merge request BuildStream/buildstream!935
| * contributing: emphasise the 'why' in commitsAngelos Evripiotis2018-11-081-0/+10
| | | | | | | | | | | | | | | | | | | | | | Explain why it's useful to mention the decisions made for a change and provide links for background, e.g. issue numbers. My general experience of git histories is that folks don't do enough explaining, so it's great we have a contributing section for it. I've found it's easier for folks to stick to rules when they know the practical reasons for them, so provide some.
| * contributing: non-WIP should always be landableAngelos Evripiotis2018-11-081-0/+6
| | | | | | | | | | | | | | | | | | | | | | | | | | | | I was recently surprised that we don't prefer addressing review comments in 'fixup!' commits. Coming from GitHub, I've found that fixup commits make it easier for reviewers to see what has changed since their last review. The idea is to use '--autosquash' before landing to clean up the history again. It's a pleasant surprise that it's easy to keep track of what changed in merge-requests between pushes, so we can always keep the history clean. Document this, so folks like me can see the light sooner.
| * contributing: WIP again to be kind to reviewersAngelos Evripiotis2018-11-081-1/+7
|/ | | | | | | | As someone coming from GitHub to GitLab, I was pleasantly surprised by the 'filter by WIP status' option. To make sure we get the most out of it, add a guideline to keep the filter clean, to reduce reviewer burden.
* Merge branch 'valentindavid/fix-script-install-root' into 'master'Valentin David2018-11-086-2/+88
|\ | | | | | | | | | | | | Fix bug with root mounted as non-artifact in script plugin. Closes #756 and #749 See merge request BuildStream/buildstream!941
| * Fix bug with root mounted as non-artifact in script plugin.Valentin David2018-11-086-2/+88
|/ | | | The issue was introduced by 6ccfab0b1b25990e406446d5cbe5aee83a5e158a.
* Merge branch 'bschubert/tests-no-chroot-linux' into 'master'Benjamin Schubert2018-11-088-49/+134
|\ | | | | | | | | | | | | test behavior when there is no sandbox on Linux Closes #696 and #736 See merge request BuildStream/buildstream!921
| * Test that helpful messages are raised when missing dependenciesBenjamin Schubert2018-11-085-1/+97
| | | | | | | | | | This adds a `reason` to the SandboxEror thrown in sandboxdummy to be able to understand where the error comes from
| * conftest.py: Ensure platform is not maintained between testsBenjamin Schubert2018-11-081-0/+7
| | | | | | | | | | | | | | | | | | | | | | | | This removes the `_instance` on the platform object that we use for caching and not recreating the object everytime at the start of every test. This is to ensure our tests share the least amount of state. The performance penalty is from 5 to 10% accross the whole test suite. The readings were done 5 times for each before and after the change and on the same computer.
| * Move bwrap checks in platform/linux.pyBenjamin Schubert2018-11-082-48/+30
|/ | | | | | | | Remove the bwraps checks from _site.py and put them in platform.linux which is the only place where they are run. This allows the removal of a double level of caching, making reasoning about tests easier
* Merge branch 'valentindavid/fix-recursive-strip-debug' into 'master'Valentin David2018-11-084-3/+8
|\ | | | | | | | | | | | | Fix infinite recursion in default strip debug command Closes #645 See merge request BuildStream/buildstream!918
| * Fix infinite recursion in default strip debug commandValentin David2018-11-084-3/+8
|/ | | | Fixes #645.
* Merge branch 'valentindavid/script-artifact-corruption' into 'master'richardmaw-codethink2018-11-076-7/+70
|\ | | | | | | | | | | | | Fix cache corruption by scripts when layout and integration commands are used Closes #749 See merge request BuildStream/buildstream!934
| * Fix cache corruption by scripts when layout and integration commands are usedValentin David2018-11-076-7/+70
|/ | | | | | | | Root directory was marked as a non-artifact mount, so not using SafeHardLink. However integration commands executed with write access to the root directory. Fixes #749
* Merge branch 'jmac/make_writable_combination' into 'master'Jim MacArthur2018-11-071-10/+3
|\ | | | | | | | | | | | | Use _force_rmtree instead of custom code. Closes #735 See merge request BuildStream/buildstream!927
| * element.py: Use _force_rmtree instead of custom code.jmac/make_writable_combinationJim MacArthur2018-11-071-10/+3
|/ | | | | There are arguments for and against using onerror - see issue #735 for details.
* Merge branch 'jennis/docs_patch' into 'master'richardmaw-codethink2018-11-073-67/+81
|\ | | | | | | | | Update our instructions/examples regarding how we declare remote caches See merge request BuildStream/buildstream!932
| * using_configuring_artifact_server.rst: Remove examples and redirect to user ↵James Ennis2018-11-071-47/+7
| | | | | | | | config/project config