summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* tests/frontend/workspace.py: Add cache key checks to test_buildjuerg/workspace-cache-keysJürg Billeter2018-02-271-0/+2
| | | | Regression test for #273
* Determine as early as possible whether a build is pendingJürg Billeter2018-02-272-4/+41
| | | | | | | | | Workspaced sources are considered unstable if a build is pending as the build will modify the contents of the workspace. Determine as early as possible if a build is pending to be able to discard unstable cache keys. Fixes #273
* tests/frontend/pull.py: Do not use a workspace for the non-strict testJürg Billeter2018-02-271-2/+3
| | | | | The cache status of reverse dependencies of workspaced elements cannot always be determined upfront.
* element.py: Remove unused _get_strict_cache_key() methodjuerg/cache-key-handlingJürg Billeter2018-02-271-12/+0
| | | | Cache key handling is now contained within the Element class.
* element.py: Add _can_query_cache() methodJürg Billeter2018-02-273-4/+13
| | | | | Keep the knowledge about which cache key is required for cache queries within the Element class.
* Remove unused _KeyStrength importsJürg Billeter2018-02-273-5/+0
|
* _artifactcache: Add key parameter to pull() methodJürg Billeter2018-02-273-34/+61
| | | | Contain cache key logic in Element class.
* _artifactcache: Add keys parameter to push() methodJürg Billeter2018-02-273-14/+16
| | | | Contain cache key logic in Element class.
* _artifactcache: Add key parameter to push_needed() methodJürg Billeter2018-02-273-7/+10
| | | | Contain cache key logic in Element class.
* _artifactcache: Add keys parameter to commit() methodJürg Billeter2018-02-274-26/+24
| | | | Contain cache key logic in Element class.
* _ostree.py: Use list of refs as commit parameterJürg Billeter2018-02-272-11/+7
|
* _artifactcache: Add key parameter to extract() methodJürg Billeter2018-02-274-22/+22
| | | | Contain cache key logic in Element class.
* _artifactcache: Add key parameter to contains() methodJürg Billeter2018-02-274-43/+14
| | | | Contain cache key logic in Element class.
* _artifactcache/artifactcache.py: Add pull() to abstract base classJürg Billeter2018-02-271-0/+12
|
* _artifactcache/artifactcache.py: Add push() to abstract base classJürg Billeter2018-02-271-0/+17
|
* _artifactcache/tarcache.py: Remove unused remove() methodJürg Billeter2018-02-271-14/+0
|
* element.py: Remove unused strength parameter from _cached()Jürg Billeter2018-02-261-22/+4
| | | | | | Remove unused strength parameter from _cached() and _remotely_cached(), which makes them simple accessor functions as `__cached` is equivalent to `__strong_cached` with a strict build plan.
* tests/frontend/push.py: Test that we don't push stuff that we just pulledSam Thursfield2018-02-262-0/+67
|
* tests/frontend: Summarize the existing push and pull testsSam Thursfield2018-02-262-0/+27
| | | | | These tests are too long to understand at a glance so a summary is needed of each.
* Don't push artifacts to remote caches that already contain themSam Thursfield2018-02-263-8/+64
| | | | | | | | | | | | | | | | | | | | | This adds two separate checks to avoid pushing artifacts unnecessarily. First, in the OSTreeCache.push() method we now first obtain the list of remotes that contain the refs that we are about to push. We then avoid pushing to any remote that already contains that ref. Note that the internal ref map is read once on process startup, so if multiple `bst` processes are pushing to a cache they might still both push the same artifact. There is an existing issue for this: https://gitlab.com/BuildStream/buildstream/issues/179 Secondly the Element._skip_push() method now checks if all remote caches configured for pushing already have a given artifact, and will skip the push job altogether if they do. The first check would already mean that no pushes would actually happen, but without the second check the user would still see Push jobs being created for every artifact which would be quite misleading.
* _artifactcache/ostreecache.py: Rework so that all remotes for a ref are trackedSam Thursfield2018-02-261-58/+81
| | | | | | | | | | This commit adds a couple of simple classes in order to reduce the number of different dictionaries tracking the state of the remotes within the OSTreeCache object. It also extends the internal ref map to remember all the remotes that store a given ref, not just the highest priority ref, and modifies the remote_contains_key() method to expose that data.
* _artifactcache/artifactcache.py: Fix name mismatch in doc commentSam Thursfield2018-02-261-1/+1
|
* doc/source/format.rst: Document format version for junction dependency attributeTristan Van Berkom2018-02-261-0/+4
| | | | | | The junction attribute was merged fairly recently in the 1.1 series without bumping the project format version, this documents the junctions addition to be added since the closest version bump where support exists.
* _downloadablefilesource.py: Add ETag supportjuerg/etagJürg Billeter2018-02-251-7/+24
| | | | | | This avoids unnecessary downloads when tracking `tar` and `zip` sources. Fixes #62
* MANIFEST.in: remove files that are no longer in this repoChandan Singh2018-02-241-2/+0
| | | | | We don't have `Dockerfile` and `Dockerfile-build.sh` anymore in this repository so remove them from `MANIFEST.in`.
* tests/integration/shell.py: Added tests for inheriting environment variablesTristan Van Berkom2018-02-252-3/+44
|
* tests/integration/shell.py: Dont use shlex, use command vectorsTristan Van Berkom2018-02-251-9/+15
| | | | | | | | | Using shlex messes with the argv we want to pass to a shell, this was not working for the newly added test which passes "${var}" strings as arguments to the shell. Also, removed a redundant line to explicitly build something in the 'no_shell' test; that is taken care of by the helper.
* testutils/runcli.py: Print something helpful if BuildStream fails to exit ↵Tristan Van Berkom2018-02-251-0/+1
| | | | | | | | properly When testing a recent patch, I had missed a return of the exit code in `bst shell` resulting in the CLI exiting with `None`; this patch makes the fixture more helpful, where previously it just printed `None`
* doc/source/projectconf.rst: Added documentation about customizing the shellTristan Van Berkom2018-02-252-0/+57
|
* element.py: Use project shell configuration when launching shells.Tristan Van Berkom2018-02-251-9/+10
| | | | | | | Use the toplevel project configuration to decide: o What command to run for an interactive shell o Which environment variables to inherit from host environment when not running an isolated shell
* _context.py: Added _get_toplevel_project() convenience accessorTristan Van Berkom2018-02-251-0/+11
|
* _project.py: Parse the new 'shell' sectionTristan Van Berkom2018-02-251-3/+15
| | | | | | | | | | | | | | | | | | | | | This new section informs BuildStream how to launch interactive shells for the specific project. These new options are part of the effort to make applications work better inside a `bst shell` environment, issue #223 Initial options include: o Default shell command to run for an interactive shell, this is used for interactive debugging sessions, and also if no command was specified in `bst shell` o List of environment variables to inherit from user environment, this is useful to propagate some host stuff into the environment for debugging; and only considered when it is not an "isolated" shell. These changes also bump the BST_FORMAT_VERSION
* _frontend: Added App.shell() to call instead of directly invoking the ↵Tristan Van Berkom2018-02-242-2/+20
| | | | | | | | Element._shell() And use this place to format a custom prompt for PS1, here we have click and we use the ANSI colors in PS1 only if colors are enabled.
* element.py: Element._shell() now allows a custom prompt for PS1Tristan Van Berkom2018-02-241-2/+6
|
* sandbox/_sandboxbwrap.py: Use --die-with-parentJürg Billeter2018-02-242-0/+26
| | | | | | | This ensures subprocesses are cleaned up when the bwrap parent dies. This is available since bubblewrap 0.1.8. We skip the option if the host bwrap does not support it.
* _artifactcache/ostreecache.py: Make enable_push a keyword argumentJürg Billeter2018-02-242-2/+2
| | | | Boolean positional arguments can be confusing.
* sandbox/_sandboxbwrap.py: Unshare IPC namespace in build sandboxJürg Billeter2018-02-241-0/+1
| | | | Isolate sandbox processes from System V IPC.
* sandbox/_sandboxbwrap.py: Unshare UTS namespace in build sandboxJürg Billeter2018-02-241-0/+1
| | | | | Set hostname to buildstream in build sandbox to reduce host contamination.
* bst shell: Added --isolate option, disabled by default.tristan/isolate-shellTristan Van Berkom2018-02-233-17/+19
| | | | | | | | | | | | | | | | | | This changes a few things: o How the shell environment and sandbox modes are setup is not decided by whether a `--build` sandbox was staged or not o In interactive build debugging mode, an isolated build sandbox is used o The bst shell command now has the option to control it o All invocations of bst shell now use a read-only rootfs, avoiding FUSE layer and file descriptor limits (this can be made optional later if needed) This is a part of issue #223
* Inherit user id and group id for bst shellJürg Billeter2018-02-223-2/+13
| | | | | | This allows D-Bus access. Fixes #227
* _ostree.py: Disable OSTree's minimum disk space check in our repossam/disable-ostree-free-space-checkSam Thursfield2018-02-221-0/+7
| | | | | | | | | | | | | | | | This interacts badly with our builds, usually if the check does fire it's at the point that we are trying to commit a completed build (which may have taken a long time to complete) into the cache. Even though there might be enough space available to store the build, the error causes BuildStream to consider it "failed" and forces the user to start again. This aims to fix https://gitlab.com/BuildStream/buildstream/issues/216 although there is some evidence that it does not completely do so. BuildStream currently has no minimum disk space check, and no way to automatically clean up the cache. There is an open issue for this: https://gitlab.com/BuildStream/buildstream/issues/135
* doc/source/projectconf.rst: Moving docs for `fail-on-overlaps`Tristan Van Berkom2018-02-211-17/+19
| | | | | | This is not a part of the documentation on how external plugins are loaded and referenced; lets put it in the main general section instead.
* tests/integration/base/generate-base.sh: Typo fix in commentSam Thursfield2018-02-211-1/+1
|
* tests/integration: Use a minimal custom base sysrootSam Thursfield2018-02-219-29/+83
| | | | | | | | | | | | | | | We have been using the Freedesktop SDK binaries to provide a base system to run the integration tests. This works OK but it weighs in at 985MB of content, and there is no simple way to customize it so that we only download the bits we actually need. This commit changes the tests to use a custom sysroot based on the Alpine Linux distribution. The sysroot is 155MB unpacked, and packs down to a 27MB .tar.xz. This speeds up the integration tests significantly as we greatly reduce the amount of network traffic required and the amount of data that gets copied around when creating the staging area.
* tests/integration: Do not use C++ compiler in workspace-mount testJürg Billeter2018-02-213-8/+8
| | | | | | | This means that the integration test suite no longer requires a C++ compiler, and in practice this can allow us to avoid depending on the GNU C/C++ compilers altogether and instead use the vastly smaller Tiny C Compiler.
* tests/integration: Use a simpler CMake test projectSam Thursfield2018-02-215-31/+17
| | | | | | | | | | | The 'step7' tutorial was fine, but it requires a C++ compiler to build. I have replaced it with a simpler testcase which is based on the existing amhello testcase. This means that the integration test suite no longer requires a C++ compiler, and in practice this can allow us to avoid depending on the GNU C/C++ compilers altogether and instead use the vastly smaller Tiny C Compiler.
* doc: Add 'sandboxing' sectionSam Thursfield2018-02-203-0/+204
| | | | | This attempts to document and specify our current sandboxing functionality.
* Create workspace directory after checking for potential issuesChandan Singh2018-02-191-14/+15
| | | | | | | | | | At present, in case that `bst workspace open` for reasons such as sources not being tracked or if tracking fails, BuildStream will leave behind an empty directort corresponding to the workspace. Restructure code so that workspace directory is only created after other checks have passed. Fixes #200.
* doc: Update information on how plugin configuration is composedJonathan Maw2018-02-191-19/+34
| | | | | | | | It previously neglected to describe how sources were composed, and we've changed how sources are composed, recently. In addition, it was slightly confusing to have the project conf be the title of two different stages of composition.
* tests: Add a git test that project overrides workJonathan Maw2018-02-194-0/+52
|