summaryrefslogtreecommitdiff
path: root/src
Commit message (Collapse)AuthorAgeFilesLines
...
* _messenger.py: Use kwargs in timed_activity() and simple_task()Tristan van Berkom2020-10-271-20/+19
| | | | | This allows more flexible usage of the APIs, to allow passing all of the Message() constructor arguments through kwargs.
* element.py: Remove Element._get_brief_display_key()Tristan van Berkom2020-10-271-14/+2
| | | | | Now that we have _DisplayKey(), no need for this extra accessor, it's not helping anything to have it anymore.
* _artifactcache.py: Use Element._get_display_key()Tristan van Berkom2020-10-271-9/+5
| | | | | Instead of using Element._get_brief_display_key(), no need for that second accessor.
* _elementsources.py: Fix copy/paste error in API doc commentTristan van Berkom2020-10-271-1/+1
|
* Adding _DisplayKey typeTristan van Berkom2020-10-276-22/+39
| | | | | | | | | | | | | | | | | | | Instead of passing around untyped tuples for cache keys, lets have a clearly typed object for this. This makes for more readable code, and additionally corrects the data model statement of intent that some cache keys should be displayed as "dim", instead informing the frontend about whether the cache key is "strict" or not, allowing the frontend to decide how to display a strict or non-strict key. This patch does the following: * types.py: Add _DisplayKey * element.py: Return a _DisplayKey from Element._get_display_key() * Other sources: Updated to use the display key object
* _frontend/cli.py: Drop support for bst shell --use-buildtree=tryJürg Billeter2020-10-272-22/+11
| | | | | `--use-buildtree` is now a boolean option for `bst shell`. If no buildtree is available, an error is raised.
* _stream.py: Simplify code in shell()Jürg Billeter2020-10-271-87/+31
| | | | | | Use single scheduler run for pulling dependencies and buildtree. Deduplicate cache checks and corresponding warning/error messages with and without pulling enabled.
* element.py: Add optional scope parameter to _set_required()Jürg Billeter2020-10-271-5/+8
|
* Move shell element loading logic from Cli to StreamJürg Billeter2020-10-272-66/+70
|
* _stream.py: Use callback for shell promptJürg Billeter2020-10-273-14/+15
|
* _frontend/cli.py: Make bst shell --use-buildtree imply --buildJürg Billeter2020-10-271-0/+4
|
* _frontend/cli.py: Drop support for bst shell --use-buildtree=askJürg Billeter2020-10-271-27/+2
|
* workspace.py: Don't use the same directory for everythingbschubert/simplify-workspace-testBenjamin Schubert2020-10-261-6/+3
|
* workspace.py: Simplify invocation of the cli commandBenjamin Schubert2020-10-261-5/+1
|
* workspace.py: Remove some abstractions and write the whole test in placeBenjamin Schubert2020-10-261-46/+31
| | | | | | This is to remove some hoops that are currently used, that do not fit well with most of the style in the code. It also removes indirections and make the test easier to understand
* workspace.py: Remove arguments that are unused during the testBenjamin Schubert2020-10-261-31/+15
|
* workspace.py: Simplify test by only allowing opening a single workspaceBenjamin Schubert2020-10-261-52/+10
| | | | | This test seems to be copy pasted from our tests around workspaces and doesn't need to be so complicated
* scheduler.py: Invoke the ticker callback at the end of run()juerg/scheduler-tickerJürg Billeter2020-10-261-0/+3
| | | | This allows the frontend to render pending messages.
* src/buildstream/testing/_utils/site.py: Adding have_subsecond_mtime()Tristan van Berkom2020-10-251-0/+32
| | | | | | | A utility for tests to check if subsecond precision mtime is supported at a given filesystem path (it will depend on underlying filesystem, so we cannot have a simple HAVE_SUBSECOND_MTIME variable to check, because we don't know where tests will operate at this stage).
* element.py: Remove unused import of OrderedDict()tristan/remove-unused-importTristan van Berkom2020-10-061-1/+0
|
* _loader/loader.py: Track link elements and resolve overridesTristan van Berkom2020-10-061-4/+190
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | | When an override is specified, it might be specified using a link element and that link element might not be otherwise used thus far to load anything. This patch makes the following changes to the loader: * Add Loader._resolve_link() A new function which updates a projects internal cache of link elements and targets. The link cache is updated any time a link element is resolved, also parent project link caches are updated to have knowledge of the subproject links. * Add Loader._expand_link() A function to expand links in user provided paths, so that we can transform these to "real paths", i.e. project relative paths wich any link elements substituted. * Use Loader._expand_link() when matching a user specified override from a junction to a real path. * Add Loader._shallow_load_overrides() This function simply ensures we perform a shallow load on any elements specified as keys in the junction override dictionary, in case they are links. Shallow loading these elements does not cause them to inadvertently load unrequired subprojects, nor can it cause any new elements to be included in the build graph returned to the caller of the Loader code (as the Loader code only returns dependencies of the specified toplevels, even if other orthogonal elements happen to get loaded as a side effect). This is necessary to ensure we populate the link caches when a link is used to specify an override, but is otherwise unused. This fixes #1398
* element.py: Ensure element methods with side effects on others are not ↵bschubert/stricter-job-scheduler-separationBenjamin Schubert2020-10-051-0/+20
| | | | | | | called in jobs This adds some assertions in the code to ensure we are not calling methods with side effects on the graph of elements in jobs.
* element.py: Stop computing cache keys when caching, we should have themBenjamin Schubert2020-10-011-2/+4
| | | | | | | | | | This removes a call to `__update_cache_key_non_strict()` in the job for non strict builds. This call has an impact on other elements and should never be done beforehand. This however means we need to compute cache keys before, and we therefore need to re-compute the cache keys as an element become buildable
* Refactor: Lazily instantiate ProvenanceInformation objectstristan/lazy-provenanceTristan van Berkom2020-10-0120-167/+140
| | | | | | | | | | | | | As a rule, throughout the codebase we should not be using internal ProvenanceInformation objects in our APIs, but rather Node objects. This is because ProvenanceInformation is generated on the fly from a Node object, and it is needlessly expensive to instantiate one before it is absolutely needed. This patch unilaterally fixes the codebase to pass `provenance_node` Node objects around as arguments rather than `provenance` ProvenanceInformation objects.
* _stream.py: Pull missing artifacts in push()juerg/pushJürg Billeter2020-09-292-46/+8
| | | | | | | | | | | As per #819, BuildStream should pull missing artifacts by default. The previous behavior was to only pull missing buildtrees. A top-level `--no-pull` option can easily be supported in the future. This change makes it possible to use a single scheduler session (with concurrent pull and push jobs). This commit also simplifies the code as it removes the `sched_error_action` emulation, using the regular frontend code path instead.
* element.py: Add skip_uncached parameter to _skip_push()Jürg Billeter2020-09-292-8/+15
| | | | | | This allows proper error handling when pushing an uncached element should result in a failure (bst artifact push), not a skipped job (bst build).
* element.py: Handle uncached state in _push()Jürg Billeter2020-09-291-1/+7
| | | | | | This allows proper error handling when pushing an uncached element should result in a failure (bst artifact push), not a skipped job (bst build).
* element.py: Use Artifact.pull() methodJürg Billeter2020-09-281-58/+11
| | | | | | | This replaces `__pull_strong()` and `__pull_weak()` with calls to `Artifact.pull()`. Besides simplifying code in `element.py`, this removes the call to `_pull_done()` from the job process, which would cause issues with the upcoming thread-based scheduler.
* _artifact.py: Add pull() methodJürg Billeter2020-09-281-0/+26
|
* _artifact.py: Assert that proto is available after `set_cached()`Jürg Billeter2020-09-281-0/+1
|
* _artifactcache.py: link_key(): Update reference if it exists alreadyJürg Billeter2020-09-281-2/+5
| | | | | We want the weak reference to point to the most recent artifact with the same weak cache key, not the oldest.
* _loader/loadelement.pyx: Support multiple filenames in a dependencyTristan van Berkom2020-09-281-38/+132
| | | | | | | | | | | | | | | This allows declaring multiple dependencies with common properties in a mode convenient way, e.g.: depends: - type: build junction: subproject.bst filename: - foo.bst - bar.bst - baz.bst This was requested during this email thread: https://lists.apache.org/thread.html/r9d716f2734cae5a81731f9a46d73f944a7287ff168f6987e25597c6a%40%3Cdev.buildstream.apache.org%3E
* BuildElement: Fix logging regressiontristan/fix-logging-regressionTristan van Berkom2020-09-261-2/+3
| | | | | Staging activities are usually silent nested, this regressed when porting to recently changed APIs.
* ScriptElement: Fix logging regressionTristan van Berkom2020-09-261-2/+3
| | | | | Staging activities are usually silent nested, this regressed when porting to recently changed APIs.
* filter.py: Combine integration commands in assemble()Jürg Billeter2020-09-241-6/+17
| | | | | | | | | Plugins must not access public data of build dependencies in `integrate()` as the build dependencies are not guaranteed to be cached at that stage. This combines integration commands of the sole build dependency and the filter element itself at time of assembly instead of integration.
* _elementsources.py: Raise SkipJob() when tracking is unimplemented by all ↵Tristan van Berkom2020-09-241-0/+7
| | | | | | | | sources This ensures we get a SKIP message instead of a SUCCESS message when tracking an element where all of it's sources did not implement track(), which is the case for sources like `local`, `workspace` or `patch`.
* source.py: Remove BST_KEY_REQUIRES_STAGETristan van Berkom2020-09-244-90/+137
| | | | | | | | | | | | | | | | | | | | | | | | | | | | Refactored this to remove unneeded complexity in the code base, as described here: https://lists.apache.org/thread.html/r4b9517742433f07c79379ba5b67932cfe997c1e64965a9f1a2b613fc%40%3Cdev.buildstream.apache.org%3E Changes: * source.py: Added private Source._cache_directory() context manager We also move the assertion about nodes which are safe to write to a bit lower in Source._set_ref(), as this was unnecessarily early. When tracking a workspace, the ref will be none and will turn out to be none afterwards, it is not a problem that a workspace's node is a synthetic one, as tracking will never affect it. * local plugin: Implement get_unique_key() and stage() using the new context manager in order to optimize staging and cache key calculations here. * workspace plugin: Implement get_unique_key() and stage() using the new context manager in order to optimize staging and cache key calculations here. * trackqueue.py: No special casing with Source._is_trackable()
* __init__.py: Added missing import of Directory, this is public APITristan van Berkom2020-09-231-0/+1
| | | | | We don't allow importing symbols from sub-packages of BuildStream, so any public API must find it's way to the toplevel __init__.py.
* Sandbox: Remove Sandbox.set_output_directory()Tristan van Berkom2020-09-211-10/+0
| | | | | This was a dead codepath, not used by any sandbox implementation for any reason, and used to be called inconsistently by some elements.
* ArtifactElement: Don't call Sandbox.set_output_directory()Tristan van Berkom2020-09-211-3/+0
| | | | That API is useless fluff and does not have any affect on anything.
* Sandbox: Assert absolute paths where they are setTristan van Berkom2020-09-211-0/+4
|
* Element: Substitute variables on dependency configurationsTristan van Berkom2020-09-211-0/+6
| | | | This was missing in the initial implementation.
* element.py: Clarify weak and strict key descriptions in __update_cache_keys()tristan/fix-cache-key-commentsTristan van Berkom2020-09-211-9/+11
|
* buildelement.py: Implement "location" dependency configurationTristan van Berkom2020-09-181-11/+86
| | | | | | | This addresses the feature request to stage dependencies in sysroots from a couple years back: https://mail.gnome.org/archives/buildstream-list/2018-August/msg00009.html
* buildelement.py: Remove legacy command stepsTristan van Berkom2020-09-181-13/+1
| | | | | | | | | | | This was actually deadcode, since node.validate_keys() was called on the configure dictionary without the legacy command steps. If any element was using the legacy commands, they would have been met with a load time error anyway. This commit also updates the cache key test, since removing these legacy commands affects BuildElement internally in such a way as to affect the cache keys.
* ScriptElement: Porting to new APITristan van Berkom2020-09-183-117/+100
| | | | | | | | | | | | | | | | | | | | | | | | Instead of relying on Element.search(), use Element.configure_dependencies() to configure the layout. Summary of changes: * scriptelement.py: Change ScriptElement.layout_add() API to take an Element instead of an element name, this is now not optional (one cannot specify a `None` element). This is an API breaking change * plugins/elements/script.py: Implement Element.configure_dependencies() in order to call ScriptElement.layout_add(). This is a breaking YAML format change. * tests/integration: Script integration tests updated to use the new YAML format * tests/cachekey: Updated for `script` element changes
* element.py: Adding new configure_dependencies() public API methodTristan van Berkom2020-09-184-4/+122
| | | | | | | | | | | | | | | | | | | | | | | | | | | | | This patch implements the essentials of the proposal to extend the dependency attributes: https://lists.apache.org/thread.html/r850aeb270200daade44261f16dbad403bf762e553b89dcafa0848fe7%40%3Cdev.buildstream.apache.org%3E And essentially this will obsolete issue #931 by providing a more suitable replacement for Element.search(). Summary of changes: * _loader/loadelement.pyx: The Dependency object now loads the `config` node, and raises an error if the `config` node is specified on a runtime-only dependency. * element.py: Created the new Element.configure_dependencies() virtual method. If elements implement this method, then a list of all dependencies are collected at element construction time and given to the implementing (depending) element. If a build dependency has more than one `config` specified, then it will be given to the Element twice, and if there is no `config` specified, then the tuple will still be given to the element with a Null `config`. Elements are provided via a new DependencyConfiguration type.
* _loader/loadelement.pyx: Added `path` propertyTristan van Berkom2020-09-181-0/+11
| | | | | This helps simplify the following Element.configure_dependencies() implementing patch.
* _loader/loadelement.pyx: Merge duplicate dependencies.Tristan van Berkom2020-09-182-29/+26
| | | | | | | | | | | | | | | | | | When the same element is specified multiple times as a direct dependency, merge and accumulate the results into the already loaded dependency. * If the same element is a runtime and build dependency separately, it will be a single dependency of both runtime and build. * If either of the dependencies are `strict`, it will be a strict dependency. The build graph retains the invariant that an element only ever depends on another element once directly, only the YAML can express the same dependency differently more than once, and the results are accumulated. This consequently remoces LoadErrorReason.DUPLICATE_DEPENDENCY as this is no longer relevant.
* _loader: Added DependencyTypeTristan van Berkom2020-09-185-32/+54
| | | | | | This is a bit nicer than relying on strings in the Symbol enumeration, and allows for some bitwise operations so we can test for BUILD or RUNTIME.
View Lorry Controller Status