summaryrefslogtreecommitdiff
path: root/src/buildstream
Commit message (Collapse)AuthorAgeFilesLines
* testing/.../site: windows-friendly HAVE_OLD_GITaevri/oldgitAngelos Evripiotis2019-10-151-1/+3
|
* workspace.py: Do not close gRPC channelsJürg Billeter2019-10-152-5/+0
| | | | This is now handled in Context.prepare_fork().
* _remote.py: Do not use subprocess to check remoteJürg Billeter2019-10-151-37/+6
| | | | This is no longer required as gRPC connections are closed before fork.
* _context.py: Replace is_fork_allowed() with prepare_fork()Jürg Billeter2019-10-152-13/+10
|
* scheduler.py: Call is_fork_allowed() right before spawning jobsJürg Billeter2019-10-151-2/+7
| | | | | gRPC channels might be opened after the scheduler has already been started. Make sure channels are closed right before spawning jobs.
* _basecache.py: Add close_grpc_channels() methodJürg Billeter2019-10-151-3/+10
|
* cascache.py: Rename close_channel() to close_grpc_channels()Jürg Billeter2019-10-153-5/+5
| | | | This aligns the method name with has_open_grpc_channels().
* cascache.py: Reset _casd_cas in close_channel()Jürg Billeter2019-10-151-0/+1
|
* _remote.py: Reset _initialized in close()Jürg Billeter2019-10-151-0/+2
|
* _sourcecache.py: Reset source_service in SourceRemote.close()Jürg Billeter2019-10-151-0/+4
|
* _artifactcache.py: Reset artifact_service in ArtifactRemote.close()Jürg Billeter2019-10-151-0/+4
|
* win32: _platform/win32: add support for win32Angelos Evripiotis2019-10-142-0/+63
| | | | | | Copy the approach of 'Darwin' and provide a SandboxDummy. This enables us to run 'bst workspace list' on Windows.
* Remove XXX comment about missing progressTristan Maat2019-10-101-2/+9
| | | | | | | | | | | | | This should be safe now - this particular point turned out to be involved in loading dependencies of junction elements, rather than anything in their projects. This meant that, yes, we were missing progress, however junction elements are not allowed to have dependencies in the first place, so we simply short-circuit their load and avoid the problem altogether. We also added more explicit progress opt-outs, since it's far too easy to end up with spurious Nones.
* testutils/context.py: Mock tasks instead of accepting NonesTristan Maat2019-10-102-5/+10
| | | | | | | | | | | | To ensure that we only disable element loading task progress reporting for very specific code paths, we need to teach the test suite to be a bit smarter. For this reason we now mock a _Task object and return it in our mock context's relevant method invocations. Other code paths that deliberately invoke the loader without task reporting now mark their loads with NO_PROGRESS.
* loader.py: Avoid loading deps of junction metaelementsTristan Maat2019-10-101-1/+19
| | | | | | | | | | | | | By avoiding this, loading metaelements of junctions becomes cheap even for junctions with erroneous dependencies, and we can ignore their task reporting. Task reporting for junction metaelement loading is confusing, since the junction element itself will never be part of the pipeline, so we'd rather not have this show up as an actual loaded element. Elements loaded from the junction are loaded separately, therefore this does not affect their progress display.
* _sourcecache: Fallback to fetch source when remote has missing blobsBenjamin Schubert2019-10-101-0/+5
| | | | | | If a remote has some missing blobs for a source, we should not fail abruptly but instead continue to the next remote, and, in the worst case, fetch the source again.
* _fuse/mount.py: Monitor the fuse process while waiting for the mountbschubert/fuse-permissionsBenjamin Schubert2019-10-091-14/+35
| | | | | | | | | | | In some cases, users might not have permissions to use fuse, or fuse might crash. This was previously leading to a hanged process and, with chance an error message on the UI, which could be overwritten. This ensures we are explicitely monitoring the fuse process while waiting and adds better reporting of the fuse errors.
* _artifactcache.py: Don't push artifact blobs when no files are presentBenjamin Schubert2019-10-082-2/+4
| | | | | | | | | | Previously, if an artifact proto had no files at all in it, we would fail at pushing it, making BuildStream crash. When no files are part of an artifact proto, we can short-circuit the call and avoid pushing them unecessarily. - Add a test to ensure this doesn't come back.
* _scheduler.py: Listen for buildbox-casd failure and terminateBenjamin Schubert2019-10-083-2/+44
| | | | | | | This adds a listener on the scheduler's event loop to ensure that the buildbox-casd process stays alive during the run. If that fails, terminate all running processes, we know they can't succeed anyways and exit accordingly.
* Defer committing workspace files to cachetraveltissues/1159Darius Makovsky2019-10-082-9/+10
| | | | | | | | | Remove XFAIL mark from test_workspace_visible and remove the explicit SourceCache.commit() in the workspace source plugin. Allow buildstream to handle the commit logic. Add handling for non-cached workspace sources in `source.Source._generate_keys()`.
* cli.py: no fcntl on WindowsAngelos Evripiotis2019-10-081-9/+18
| | | | | | | | | | | | | Work around the fact that we can't import 'fcntl' on Windows, and confine the workaround to as small a scope as we can. This enables us to run at least these commands on Windows: bst help bst init We can't run any commands that require a Platform object though, which is most commands.
* cascache.py: add a 'log_level' parameter and use it to run buildbox-casdBenjamin Schubert2019-10-073-4/+23
| | | | | | | | | | This changes how we instantiate the CASCache by reusing the log level parameters from BuildStream and forward them to buildbox-casd. By default, buildbox-casd will now have '--log-level warning', and --verbose will enable 'info' and --debug will enable 'trace'. This way, we can easily tweak buildbox-casd's verbosity
* element.py: remove call to _source_cached()traveltissues/benchmarkDarius Makovsky2019-10-051-1/+0
| | | | | | | Remove call to Element._source_cached() in _calculate_cache_key and do not recalculate workspace keys. tests: shell browsing of workspaces is currently broken
* element.py: remove concept of key instabilityDarius Makovsky2019-10-051-56/+9
| | | | | | | Workspace keys are determined by contained files at time of opening and are not forcibly reset tests: workspaces have keys when they're opened and are not recalculated
* cascache.py: Block SIGINT in the CasUsageMonitor processBenjamin Schubert2019-10-041-2/+5
| | | | | | We don't want this process to be killed if someones CTRL+C BuildStream and the continues the build. We can therefore just ignore SIGINT there.
* cascache.py: Block SIGINT in the buildbox-casd processBenjamin Schubert2019-10-041-3/+6
| | | | | | We don't want SIGINT to be forwarded to the buildbox-casd process, otherwise it would get killed if someone CTRL+C the BuildStream process and then continues the build, which would make everything fail.
* _scheduler/jobs/job.py: sort importsAngelos Evripiotis2019-10-041-4/+4
|
* cascache.py: Show path to log file when buildbox-casd doesn't exit normallyBenjamin Schubert2019-10-041-2/+12
|
* cascache.py: Save casd logs in a file for retrievalBenjamin Schubert2019-10-041-1/+32
| | | | | Save all casd logs in a log file under its cas/ directory, and keep only the last 10 of them.
* cascache.py: Send message in case of unclean termination of buildbox-casdBenjamin Schubert2019-10-021-2/+27
| | | | | | | This adds messages in the various mis-termination of Buildbox-casd, to notify users that something might have gone wrong there. It also adds a few tests to validate the various behaviors.
* cascache.py: Extract buildbox-casd termination in its own methodBenjamin Schubert2019-10-021-17/+28
|
* Do not check for casd in bash completionTristan Maat2019-10-022-3/+9
| | | | | | | | | This caused stacktraces when a user didn't have buildbox-casd installed, even though it wasn't required for completion, even for artifacts. It does add a slightly ugly parameter to Context, long-term it might make sense to create a special slimmed-down context for completion.
* _casbaseddirectory.py: Use CASCache.import_directory()juerg/capture-treeJürg Billeter2019-10-021-56/+13
|
* cascache.py: Add import_directory() methodJürg Billeter2019-10-021-0/+35
| | | | | This allows import of a whole directory tree with a single gRPC call to buildbox-casd CaptureTree() instead of importing files one at a time.
* _context.py: Remove duplicated log_message_lines assignTom Pollard2019-09-251-1/+0
|
* import.py: assemble the element via `stage_sources`Darius Makovsky2019-09-241-2/+1
|
* loader.py: Remove workspace handling in element collectionDarius Makovsky2019-09-241-5/+1
|
* Remove workspace mounting for sandboxesDarius Makovsky2019-09-244-41/+12
| | | | | | | | Workspaces will be handled via the workspace source plugin methods. This does not currently support reflecting build artifacts in the open workspace. tests: Mark incremental workspace builds as xfail (strict)
* element.py: always check sources for cachingDarius Makovsky2019-09-241-15/+4
| | | | | | | | | | | | | | | | | | | | | | Check that sources are cached even if they are workspaced and do not reset workspace cache data partially reverts !1470 closes #1088 element.py: remove workspaces in cache key calc Using the workspace source plugin, workspaces should now be handled like sources for the purpose of calculating element cache keys. partially reverts !1470 works towards #1073 Since the source keys are now calculated using the unique keys of the workspace source, this change will break external tracking for open workspaces. In future attempting to track an open workspace might raise a SourceError. The test is rewritten to close the workspace before tracking.
* Support loading 'workspace'-kind sourcesDarius Makovsky2019-09-242-7/+43
| | | | | Add the workspace-kind source to the sources manually to prevent a user incorrectly loading the source plugin.
* workspace.py: add workspace source pluginDarius Makovsky2019-09-241-0/+150
| | | | | | | | | | | | | | | | | | | | | | | | | The `workspace.init_workspace()` call should wrap `source._init_workspace` for held sources to support those sources not publishing `BST_VIRTUAL_DIRECTORY` This object owns a directory digest attribute used inplace of the source ref. `track` and `fetch` become noop methods and the workspace is imported into the CAS in the call to `get_unique_key` which also sets the digest attribute and owns that Directory object. The directory is referenced during stage to import directly to the virtual directory object. Importing is expected to be expensive and will be optimised in future. When the unique key is retrieved for the workspace source it will also be commited to the sourcecache. The logic for this source is still a slight variant on other sources since it cannot itself be expected to be in the cache when it's opened. In the source preflight method the preflights of the held sources must be called.
* cascache.py: Create `CASCache.close_channel()`Darius Makovsky2019-09-241-5/+11
|
* _remote.py: Remove unused bytestreamJürg Billeter2019-09-191-5/+0
|
* casremote.py: Add missing copyright headerJürg Billeter2019-09-191-0/+17
|
* casremote.py: Do not directly communicate with CAS serverJürg Billeter2019-09-191-68/+0
| | | | | | | | All communication with CAS servers should be proxied through buildbox-casd to allow connection sharing among job subprocesses. It is no longer needed to query the server capabilities as buildbox-casd already does that, so we can simply remove the code from BuildStream.
* cascache.py: Use casd in remote_missing_blobs()Jürg Billeter2019-09-191-2/+11
| | | | Remote CAS servers should not be contacted directly.
* cascache.py: Create CAS stub for buildbox-casd channelJürg Billeter2019-09-191-2/+21
|
* casremote.py: Drop Status requestJürg Billeter2019-09-191-18/+0
| | | | | The push status check is implemented by BuildStream artifact and source services. It's not part of the CAS protocol.
* _sourcecache.py: Move capabilities check to _check() and extend itJürg Billeter2019-09-191-10/+22
| | | | | | | It's sufficient to check the capabilities once per bst session. This avoids the extra round trip in remote.init(). This also adds a check for allow_updates for push remotes.
* _artifactcache.py: Move capabilities check to _check() and extend itJürg Billeter2019-09-191-9/+21
| | | | | | | It's sufficient to check the capabilities once per bst session. This avoids the extra round trip in remote.init(). This also adds a check for allow_updates for push remotes.
View Lorry Controller Status