summaryrefslogtreecommitdiff
path: root/buildstream/element.py
Commit message (Collapse)AuthorAgeFilesLines
...
* element.py: Support command batching for integration commandsJürg Billeter2018-11-271-8/+7
|
* element.py: Defer workspace.prepared setting in case of pending commandsJürg Billeter2018-11-271-1/+6
|
* element.py: Cache artifact also on SandboxCommandErrorJürg Billeter2018-11-271-2/+2
|
* element.py: Pass element object to sandboxJürg Billeter2018-11-271-0/+2
| | | | This allows the sandbox to output messages using the plugin id.
* element.py: Use SandboxFlags.NONE instead of 0Jürg Billeter2018-11-271-1/+2
|
* element.py: Only cache on element error or successBenjamin Schubert2018-11-191-19/+6
| | | | | | Reverse the way we were handling caching by only caching when we know it's a problem with the build or the build is a success in order not to cache transient errors.
* Don't cache sandbox errorsBenjamin Schubert2018-11-191-89/+103
| | | | | | | | | | Sandbox errors (like missing host tools) are dependent on the host system and rarely on what is actually done. It is therefore better to not cache them as they are subject to change between two runs. Also add test to ensure sandbox failure are not cached
* Don't pull artifact build trees by default.tpollard/494Tom Pollard2018-11-171-37/+125
| | | | | | | | | | | | | | | | | | | | | | The addition of cached build trees being included in element artifacts has led to mostly redundant download overheads when pulling from a remote artifact server. As such the default behaviour of pull shouldn't fetch the build tree object if available. element.py: extend relevant pull logic for specified subdir consideration and ensure push logic does not lead to partial artifact pushes. Change assumption that the buildtree can be extracted if artifact is cached. __cached_buildtree() and __pull_directories() helpers added. _artifactcache/: artifactcache.py & cascache.py inclusion of helper functions for subdir artifact checking & fetching, fetch logic extended to only pull required artifact directories. extract & checkout updated to handle full/partial operation. tests/: addition of integration test pullbuildtrees.py, buildtrees.py adapted cli options, testutils/artifactshare.py has_artifact changed to return artifact digest on true condition.
* element.py: Use _force_rmtree instead of custom code.jmac/make_writable_combinationJim MacArthur2018-11-071-10/+3
| | | | | There are arguments for and against using onerror - see issue #735 for details.
* element.py: Do not include type name of artifact cache in cache keyJürg Billeter2018-11-051-1/+1
| | | | | | | The artifact cache backend does not affect build outputs and we anyway no longer have pluggable artifact cache backends. This hardcodes CASCache instead of removing the entry completely to avoid cache key changes.
* element.py: Remove unused artifacts parameter from _new_from_meta()Jürg Billeter2018-11-051-4/+3
|
* _project.py: Remove unused artifacts parameter from create_element()Jürg Billeter2018-11-051-1/+1
|
* element.py: Remove artifacts parameter from constructorJürg Billeter2018-11-051-2/+2
| | | | Get the artifact cache from the context instead.
* element.py: Pass bare_directory also to SandboxRemote()juerg/remote-executionJürg Billeter2018-11-031-0/+1
| | | | Fixes: 825963b5 ("element: Make "--sysroot" take a bare directory")
* element: Make "--sysroot" take a bare directoryJonathan Maw2018-10-301-3/+8
| | | | | | | | | | | | | | | | | | i.e. instead of taking a directory that must contain "root" and "scratch", and treating "root" as the root, use the directory directly. In element.py: * __sandbox takes the `bare_sandbox` arg, to pass into the sandbox's constructor In sandbox.py: * If bare_sandbox, `_root` is the passed-in directory, and `__scratch` is None. * Trying to use `__scratch` when bare_sandbox is True is a bug. In _mount.py: * Don't get the value of `__scratch` if it's not needed. This is part of #539
* Element: Use cached buildtree in build shells and failure shellsJonathan Maw2018-10-301-4/+8
| | | | | | | | | | | | | | | | | | | | | | | | This includes changes in app.py: * Interactive failure shell no longer uses the failed build sysroot, defaulting to the cached build tree. Changes in element.py are: * Errors caused by building don't store the failed build sysroot, instead storing that a sandbox can be created to debug the error. * When staging sources, will stage the element's cached build tree if it exists. Changes in _exceptions.py: * BstError.sandbox is now a flag of whether a sandbox can be opened up to debug the error. Changes in widget.py: * Don't try to print any information about the sandbox. Changes in _message.py: * Fix documentation so Message.sandbox is not a directory any more. This is part of #539
* element.py: Always clean up the rootdirJonathan Maw2018-10-301-2/+2
| | | | | | | | | | | | | We shouldn't need it to persist now that we cache failed build dirs. This change breaks the test `tests/integration/shell.py::test_sysroot_workspace_visible`. I can no longer see a use-case for this test. AIUI, it tested that the failed build sysroot stored in the builddir has the workspace's files in, despite the workspace being unmounted. I believe this behaviour is made redundant by cached buildtrees. This fixes part of #539
* various: Move _sentinel from utils.py to _yaml.pyDaniel Silverstone2018-10-261-1/+1
| | | | | | | | | The `_sentinel` in `utils.py` was used only for yaml node tracking. As such, simplify matters by removing it from `utils.py` and move it to `_yaml.py` which means that we no longer need to try and avoid a circular import issue by means of runtime importing. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* element.py: Silence pylint warning about abstract methodDaniel Silverstone2018-10-251-1/+1
| | | | | | | | Unfortunately pylint can't understand that we'll always use subclasses of Element and as such complains of a no-return function (`assemble()`) having its return value used. This quietens that warning. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* element.py: Simplify some conditions with `in (foo, bar)`Daniel Silverstone2018-10-251-2/+2
| | | | | | | Where we have conditions of the form `var == foo or var == bar` it can be simplified to `var in (foo, bar)` which pylint prefers. Signed-off-by: Daniel Silverstone <daniel.silverstone@codethink.co.uk>
* Fix python warnings: Use collections.abc instead collectionsjjardon/warnings_collectionsJavier Jardón2018-10-181-1/+2
| | | | | | | | Since python 3.3, collections has been moved to collections.abc module. For backwards compatibility, they continue to be visible in this module through Python 3.7. Subsequently, they will be removed entirely. See https://docs.python.org/3/library/collections.html
* Fix spelling of it's and itsRichard Dale2018-10-161-4/+4
|
* Rename element_enums.py -> types.pyTristan Van Berkom2018-10-121-2/+2
| | | | | | This will be the place to store low level data types used throughout the core, for now this includes public and private types.
* element.py: Prepare local sandbox for bst checkout and bst shellJürg Billeter2018-10-051-5/+8
| | | | A remote execution sandbox is not suitable for these commands.
* element.py: Do not save workspace on failed buildJosh Smith2018-10-041-4/+6
| | | | | This fixes #634, BuildStream no longer saves workspace configuration on a failed build.
* element.py: Put the temporary staging directory in artifactdirjmac/temporaries-inside-cachedirJim MacArthur2018-10-031-4/+4
|
* element.py: Validate sandbox configJürg Billeter2018-09-271-4/+18
|
* element.py: Add __use_remote_execution() helper methodJürg Billeter2018-09-271-3/+9
|
* element.py/_assemble(): Call get_virtual_directory later.jmac/stop-caching-vdirsJim MacArthur2018-09-251-3/+1
|
* element.py: Fix cache check in non-strict modeJürg Billeter2018-09-241-1/+1
| | | | | | | | | The behavior of __assert_cached() was changed when the keystrength parameter was introduced. This restores the previous behavior. Fixes #607. Fixes: 3d308894 ("element.py: Cache the result of checking whether...")
* element.py: Remove early warning and check directory is not NoneJim MacArthur2018-09-211-8/+5
| | | | | | The check for directory should have always been here. The early warning test doesn't work for `bst checkout` because caches haven't been configured at the point of the test, so I've had to remove it.
* cascache.py: Move push/pull messaging to cascacheJosh Smith2018-09-191-4/+0
| | | | | | | | Pulled/Pushed messages will no longer be produced from within element.py, instead they will be produced during CasCache push() and pull() appropriately. Message consistency has also been improved.
* element.py: Remove redundant timed_activityJosh Smith2018-09-191-9/+8
| | | | | This removes the timed_activity for an element _push action. This is unnecessary as the job is already being timed elsewhere.
* _artifactcache/artifactcache.py: Changes to mark_required_elements()Tristan Van Berkom2018-09-141-7/+12
| | | | | | | | | This was previously append_required_artifacts(), which presumed that we knew at startup time what the cache keys of all elements in the loaded pipeline would be. This fixes unexpected deletions of required artifacts when dynamic tracking is enabled with `bst build --track-all target.bst`
* element: Remove __cached fieldRichard Maw2018-09-131-4/+1
| | | | | This can get out of sync with other two cache states, and we can do without it.
* element: validate configuration variablesTiago Gomes2018-09-101-1/+9
| | | | Ensure that protected variables are not being redefined by the user.
* element.py: Remove _get_artifact_cache() accessor.Tristan Van Berkom2018-09-101-10/+0
| | | | | | | | | | | | The artifact cache is anyway a singleton, the Element itself has an internal handle on the artifact cache because it is needed very frequently; but an artifact cache is not element specific and should not be looked up by surrounding code on a per element basis. Updated _scheduler/queues/queue.py, _scheduler/queues/buildqueue.py and _scheduler/jobs/elementjob.py to get the artifact cache directly from the Platform
* element.py: Remove _get_artifact_size()Tristan Van Berkom2018-09-101-18/+6
| | | | | | | | There is no justification to hold onto this state here. Instead, just make `Element._assemble()` return the size of the artifact it cached, and localize handling of that return value in the BuildQueue implementation where the value is observed.
* element.py: Adding missing API documenting commentTristan Van Berkom2018-09-101-0/+7
|
* element.py: Switch to SandboxRemote if config option is setJim MacArthur2018-09-071-1/+27
| | | | https://gitlab.com/BuildStream/buildstream/issues/454
* element.py: Get the updated virtual directory after runningJim MacArthur2018-09-071-0/+2
| | | | | | | | | Executing run() on a sandbox can now replace the virtual directory, since remote execution returns a potentially different directory rather than an update to the existing one. Call get_virtual_directory() again after running to accout for this. https://gitlab.com/BuildStream/buildstream/issues/454
* Add "remote-execution" project configuration optionJim MacArthur2018-09-071-0/+6
| | | | | | This just adds one option, "remote-execution/url". Affects multiple files. https://gitlab.com/BuildStream/buildstream/issues/454
* plugin.py: Added _configure() and _get_configuring() private APIsTristan Van Berkom2018-09-021-1/+1
| | | | | | | | | Keeps track of whether the plugin is currently being configured. Adjusted Element and Source classes to call _configure() in place of calling configure() directly. This is a part of #620
* element.py: Do not attempt to pull workspaced elementsjuerg/workspaced-dependenciesJürg Billeter2018-08-281-0/+4
|
* element.py: Schedule assemble for key of workspaced elementsJürg Billeter2018-08-281-7/+15
| | | | | | | | | For uncached workspaced elements, assemble is required even just to calculate the cache key. As dynamic scheduling relies on cache keys, schedule assemble for uncached workspace elements even if they have not been marked as required yet. Fixes #461.
* element.py: chmod 777 directories if delete fails.Jim MacArthur2018-08-241-1/+18
| | | | | | Staging may produce directories with less than 'rwx' permissions which will cause tempfile to throw an exception on deleting the temporary directory if not changed.
* Allow source plugins to access previous sourcesChandan Singh2018-08-151-2/+8
| | | | | | | | | | | | | Source plugin implementations can now specify that they need access to previously staged sources by specifying `BST_REQUIRES_PREVIOUS_SOURCES_TRACK` and/or `BST_REQUIRES_PREVIOUS_SOURCES_FETCH`, corresponding to access at `track` and `fetch` times respectively. Fixes #381. Replaces !505. For relevant discussion, see this discussion: https://gitlab.com/BuildStream/buildstream/merge_requests/505#note_83780747
* Move KeyStrength enum out to element_enums.pyJim MacArthur2018-08-151-36/+3
|
* _project.py: Add fatal-warnings configuration itemJosh Smith2018-08-151-18/+8
| | | | | | | | | | | | | | | | | | | | This allows for users to configure fatal-warnings to be either a list of warnings. This commit deprecates the use of fail-on-overlap within project.conf, this will now use the fatal-warnings configuration item. element.py: Cache key calculation now takes into account all of the fatal-warnings tests: This modifys the tests/frontend/overlaps.py tests to support the new fatal-warnings configuration. Backwards compatibility is also tested for `fail-on-overlap` _versions.py: BST_FORMAT_VERSION bumped to 15 for fatal-warnings BST_CORE_ARTIFACT_VERSION bumpted to 5 for fatal-warnings Fixes: #526
* Implementing relative workspacesrelative_workspacesPhillip Smyth2018-08-101-3/+4
| | | | | | | This fixes #191 A note has been added to NEWS explaining backwards compatibility issues
View Lorry Controller Status