summaryrefslogtreecommitdiff
Commit message (Collapse)AuthorAgeFilesLines
* _sandboxremote.py: Make storage-service optional with remote cachejuerg/remote-cacheJürg Billeter2020-12-043-64/+82
|
* _context.py: Add support for remote-cache configurationJürg Billeter2020-12-041-1/+10
|
* _cas: Add support for remote cacheJürg Billeter2020-12-045-35/+127
|
* Use CASCache.open()Jürg Billeter2020-12-045-8/+8
|
* cascache.py: Add open() methodJürg Billeter2020-12-041-0/+18
|
* cascache.py: Generalize remote_missing_blobs() into missing_blobs()Jürg Billeter2020-12-043-35/+16
| | | | | The implementation can be reused to replace `local_missing_blobs()` and simplify `contains_files()`.
* cascache.py: Optimize _fetch_tree() using add_objects()Jürg Billeter2020-12-041-5/+6
|
* cascache.py: Add add_objects() methodJürg Billeter2020-12-041-16/+48
| | | | | This allows adding multiple objects in a single batch, avoiding extra gRPC round trips to buildbox-casd.
* cascache.py: Remove unused parameters from add_object()Jürg Billeter2020-12-043-10/+4
|
* cascache.py: Remove _ensure_blob() methodJürg Billeter2020-12-041-24/+2
| | | | | It's only used by `_fetch_tree()` and can be replaced by a single additional line.
* cascache.py: Also fetch file blobs in _fetch_directory()Jürg Billeter2020-12-044-22/+7
| | | | | This eliminates code duplication in `ArtifactCache`, `SourceCache` and `ElementSourcesCache`.
* cascache.py: Reimplement _fetch_directory() with FetchTree()Jürg Billeter2020-12-041-55/+17
| | | | This simplifies the code, delegating the logic to buildbox-casd.
* Move _reachable_refs_dir() method from cascache.py to testutilsJürg Billeter2020-12-042-33/+20
| | | | It's not used outside testutils.
* _sandboxremote.py: Drop unused output_files_required parameterJürg Billeter2020-12-041-22/+7
|
* Always require artifact filesJürg Billeter2020-12-045-141/+1
| | | | | | | | In line with the previous commit, also require artifact files for build dependencies with remote execution. Local artifact files will become optional with the upcoming remote cache support.
* Drop pull-artifact-files config option for remote executionJürg Billeter2020-12-044-34/+9
| | | | | | | | | | | | | | | With `pull-artifact-files` disabled, artifact file blobs are completely ignored in cache checks. While this allows remote execution builds without downloading build outputs, it can result in build failures when blobs expire on the server side. It also does not download missing blobs on demand. I.e. `bst artifact checkout` and other commands don't work with `pull-artifact-files` disabled. This commit removes the `pull-artifact-files` config option due to the above issues. It will be replaced by remote cache support, which only downloads blobs as needed but still considers all file blobs in cache checks for reliable builds. It will also support download of missing blobs on demand.
* element.py: Move get_logs() to internal APIJürg Billeter2020-12-043-12/+11
|
* Call _initialize_state() in Element._new_from_load_element()Jürg Billeter2020-12-049-51/+2
| | | | | With the cache queries moved to job threads, `_initialize_state()` is fairly lightweight and can be called earlier.
* Move artifact cache query to pull jobJürg Billeter2020-12-044-103/+85
|
* Always schedule pull jobJürg Billeter2020-12-043-58/+64
| | | | | This is preparation to perform artifact cache query as part of the same job as artifact pulling.
* _artifact.py: Add cached and strong_key parameters to set_cached()Jürg Billeter2020-12-041-4/+14
|
* Move source cache query to fetch jobJürg Billeter2020-12-0411-29/+90
|
* _stream.py: Add query_cache() methodJürg Billeter2020-12-042-0/+20
| | | | This is preparation for explicit cache query.
* _elementsources.py: Make cache query explicitJürg Billeter2020-12-041-5/+16
| | | | | | Cache query can be fairly expensive as it checks the presence of all blobs. Make this more explicit with a `query_cache()` method, instead of implicitly querying the cache on the first call to `cached()`.
* _artifact.py: Make cache query explicitJürg Billeter2020-12-043-22/+25
| | | | | | | Cache query can be fairly expensive as it checks the presence of all artifact blobs. Make this more explicit with a `query_cache()` method, instead of implicitly querying the cache on the first call to `cached()`.
* _pipeline.py: Drop the optimization for cached elements in the plannerJürg Billeter2020-12-041-17/+9
| | | | | | | | | The overhead of planning already cached elements and unneeded build-only dependencies should be fairly small as unneeded jobs can still be skipped. This optimization was also already disabled for non-strict build plans with a remote artifact cache. This change is necessary in preparation for parallelizing cache queries.
* fetchqueue.py: Don't skip elements with a cached failureJürg Billeter2020-12-041-1/+1
| | | | | | The build queue requires the sources to be available for all elements where `_cached_success()` returns `False`. This includes elements with a cached failure.
* tests/internals/pluginloading: Add missing get_ref() to FooSourceJürg Billeter2020-12-041-0/+3
|
* tests/frontend/push.py: Allow pushing of dependenciesJürg Billeter2020-12-041-4/+4
| | | | | | The assertions in `test_push_after_pull` are too strict. Pushing dependencies to the second (empty) artifact server should not cause a test failure.
* Merge branch 'bschubert/no-multiprocessing' into 'master'Benjamin Schubert2020-12-0428-612/+697
|\ | | | | | | | | | | | | Rework the scheduler to use threads instead of processes Closes #911, #93, and #810 See merge request BuildStream/buildstream!1982
| * scheduler.py: Reconnect signal handlers soonerbschubert/no-multiprocessingBenjamin Schubert2020-12-041-1/+1
| | | | | | | | | | This reduces a race condition where a sigint received shortly after restarting the scheduler would cause the schedulert to crash.
| * utils.py: remove 'is_single_threaded' method, we use it only in testsBenjamin Schubert2020-12-042-40/+23
| | | | | | | | | | This moves it to tests with a simplified usage, since we don't use it anywhere else
| * scheduler.py: Use threads instead of processes for jobsBenjamin Schubert2020-12-0425-492/+434
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | This changes how the scheduler works and adapts all the code that needs adapting in order to be able to run in threads instead of in subprocesses, which helps with Windows support, and will allow some simplifications in the main pipeline. This addresses the following issues: * Fix #810: All CAS calls are now made in the master process, and thus share the same connection to the cas server * Fix #93: We don't start as many child processes anymore, so the risk of starving the machine are way less * Fix #911: We now use `forkserver` for starting processes. We also don't use subprocesses for jobs so we should be starting less subprocesses And the following highlevel changes where made: * cascache.py: Run the CasCacheUsageMonitor in a thread instead of a subprocess. * casdprocessmanager.py: Ensure start and stop of the process are thread safe. * job.py: Run the child in a thread instead of a process, adapt how we stop a thread, since we ca't use signals anymore. * _multiprocessing.py: Not needed anymore, we are not using `fork()`. * scheduler.py: Run the scheduler with a threadpool, to run the child jobs in. Also adapt how our signal handling is done, since we are not receiving signals from our children anymore, and can't kill them the same way. * sandbox: Stop using blocking signals to wait on the process, and use timeouts all the time. * messenger.py: Use a thread-local context for the handler, to allow for multiple parameters in the same process. * _remote.py: Ensure the start of the connection is thread safe * _signal.py: Allow blocking entering in the signal's context managers by setting an event. This is to ensure no thread runs long-running code while we asked the scheduler to pause. This also ensures all the signal handlers is thread safe. * source.py: Change check around saving the source's ref. We are now running in the same process, and thus the ref will already have been changed.
| * plugin.py: Add a helper to run blocking processes in subprocessesBenjamin Schubert2020-12-012-48/+200
| | | | | | | | | | | | | | | | | | | | | | | | This ensures that we can cleanly cleanup processes and threads on termination of BuildStream. Plugins should use this helper whenever there is a risk of them being blocked on a syscall for an indefinite amount of time * downloadablefilesource.py: Use this new helper to do the actual download, which would prevent the process from completely blocking if we have a badly behaving upstream
| * downloadablefilesource.py: Reset the file opener between every testBenjamin Schubert2020-12-012-1/+7
| | | | | | | | | | | | | | This is required when we run this in the main process, with the threaded scheduler rework. Otherwise the state is kept between tests
| * element.py: Stop setting a deterministic umask for staging sourcesBenjamin Schubert2020-12-012-24/+2
| | | | | | | | | | | | | | | | | | This does not behaves as we would expect, as it is not always consistent, and doesn't have any impact in most cases. We should revisit our handling of permissions and umasks separately, in the meantime, this is required in order to fix building with a threaded scheduler, as it would otherwise introduce concurrency errors
| * _signals.py: allow calling signal handler from non-main threadsBenjamin Schubert2020-12-012-8/+18
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | * This modifies the signal terminator so that it can be called from any thread. This checks that either: - The signal handler is already in place - Or the caller is in the main thread, allowing to set the signal handler. This also removes the exact callback that was added instead of removing the last one, and fixes the `suspend_handler` to do the same. This is required, as we don't know which interleaving of calls will be done, and we can't guarantee that the last one is the right one to remove
| * utils.py: Don't block on the call's `communicate` callBenjamin Schubert2020-12-011-1/+15
|/ | | | | | This ensures that, if we were to receive signals or other things while we are on this blocking call, we would be able to process them instead of waiting for the end of the process
* Merge branch 'tristan/version-error-fix' into 'master'bst-marge-bot2020-11-303-1/+5
|\ | | | | | | | | Fix missing cert tests to expect the error for the correct reason See merge request BuildStream/buildstream!2105
| * tests/sandboxes/remote-exec-config.py: Fixed generated project to include ↵Tristan van Berkom2020-11-301-1/+3
| | | | | | | | | | | | | | | | min-version This test was broken as it was failing for the wrong reason, even though in both cases it was a missing yaml key. Fix the test to fail due to it being missing the required cert specified in the cache config.
| * tests/sourcecache/config.py: Fixed generated project to include min-versionTristan van Berkom2020-11-301-0/+1
| | | | | | | | | | | | This test was broken as it was failing for the wrong reason, even though in both cases it was a missing yaml key. Fix the test to fail due to it being missing the required cert specified in the cache config.
| * tests/artifactcache/config.py: Fixed generated project to include min-versionTristan van Berkom2020-11-301-0/+1
|/ | | | | | This test was broken as it was failing for the wrong reason, even though in both cases it was a missing yaml key. Fix the test to fail due to it being missing the required cert specified in the cache config.
* Merge branch 'chandan/setup-py-version-warn' into 'master'bst-marge-bot2020-11-241-2/+15
|\ | | | | | | | | | | | | setup.py: Ensure we have a version number Closes #1383 See merge request BuildStream/buildstream!2101
| * setup.py: Ensure we have a version numberchandan/setup-py-version-warnChandan Singh2020-11-241-2/+15
|/ | | | | | | | BuildStream requires a version number at runtime. But, builds fine if I can't find any tags. So, make it an error at build time if we don't have a valid version number. Fixes #1383.
* Merge branch 'juerg/ci' into 'master'Jürg Billeter2020-11-241-0/+10
|\ | | | | | | | | tests/frontend/push.py: Skip expiry test without subsecond mtime See merge request BuildStream/buildstream!2104
| * tests/frontend/push.py: Skip expiry test without subsecond mtimeJürg Billeter2020-11-241-0/+10
|/ | | | | Skip an artifact expiry test in the case we don't have subsecond mtime precision.
* Merge branch 'tristan/fix-glob-handling' into 'master'Tristan Van Berkom2020-11-2016-69/+291
|\ | | | | | | | | | | | | Fix glob handling in the CLI Closes #959 See merge request BuildStream/buildstream!2102
| * tests/frontend/show.py: Added test for glob pattern matches.tristan/fix-glob-handlingTristan van Berkom2020-11-191-0/+46
| |
| * tests/frontend/artifact_show.py: Added tests for glob pattern matches.Tristan van Berkom2020-11-199-0/+121
| | | | | | | | | | | | This tests a few glob patterns through `bst artifact show` and also asserts that globs which match both elements and artifacts will produce an error.
| * _stream.py: Use consistent glob handling with utils.glob()Tristan van Berkom2020-11-192-3/+8
| | | | | | | | | | | | | | | | | | | | | | | | | | We should not have a different globbing behavior than split rules for the command line. This should also make artifact globbing slightly more performant, as the regular expression under the hood need not be recompiled for each file being checked. This commit also updates tests/frontend/artifact_list_contents.py to use a double star `**` (globstar syntax) in order to match path separators as well as all other characters in the list contents command.
View Lorry Controller Status