| Commit message (Collapse) | Author | Age | Files | Lines |
|---|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Instead of relying on Element.search(), use Element.configure_dependencies() to
configure the layout.
Summary of changes:
* scriptelement.py:
Change ScriptElement.layout_add() API to take an Element instead of an element name,
this is now not optional (one cannot specify a `None` element).
This is an API breaking change
* plugins/elements/script.py:
Implement Element.configure_dependencies() in order to call ScriptElement.layout_add().
This is a breaking YAML format change.
* tests/integration: Script integration tests updated to use the new YAML format
* tests/cachekey: Updated for `script` element changes
|
| | |
|
| | |
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
This patch implements the essentials of the proposal to extend the
dependency attributes:
https://lists.apache.org/thread.html/r850aeb270200daade44261f16dbad403bf762e553b89dcafa0848fe7%40%3Cdev.buildstream.apache.org%3E
And essentially this will obsolete issue #931 by providing a more
suitable replacement for Element.search().
Summary of changes:
* _loader/loadelement.pyx: The Dependency object now loads the `config` node,
and raises an error if the `config` node is specified on a runtime-only
dependency.
* element.py: Created the new Element.configure_dependencies() virtual method.
If elements implement this method, then a list of all dependencies are
collected at element construction time and given to the implementing
(depending) element.
If a build dependency has more than one `config` specified, then
it will be given to the Element twice, and if there is no `config`
specified, then the tuple will still be given to the element with
a Null `config`.
|
| |
|
|
|
|
| |
Test that when the same dependency is added as a build and runtime
dependency separately, they end up being the same dependency which
is both a build & runtime dependency in the loaded build graph.
|
| |
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
When the same element is specified multiple times as a direct dependency,
merge and accumulate the results into the already loaded dependency.
* If the same element is a runtime and build dependency separately, it
will be a single dependency of both runtime and build.
* If either of the dependencies are `strict`, it will be a strict
dependency.
The build graph retains the invariant that an element only ever depends
on another element once directly, only the YAML can express the same
dependency differently more than once, and the results are accumulated.
This consequently remoces LoadErrorReason.DUPLICATE_DEPENDENCY as this is
no longer relevant.
|
| |
|
|
| |
LoadErrorReason.DUPLICATE_DEPENDENCY
|
| |
|
|
|
|
| |
This is a bit nicer than relying on strings in the Symbol enumeration,
and allows for some bitwise operations so we can test for BUILD or
RUNTIME.
|
| |\
| |
| |
| |
| |
| |
| | |
element.py: Refactor overlap warnings.
Closes #1340
See merge request BuildStream/buildstream!2060
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
* Stop using Element.search() in order to match elements in the build
graph when collecting overlap warnings, which is error prone and
will produce incorrect results when encountering elements with the
same name across project boundaries.
Use Plugin._unique_id to match up elements instead.
* Print Element._get_full_name() in the warning outputs, which is more
accurate than element.name.
* General refactor of code to use more descriptive variable names,
improved comments, making the whole overlap code a bit more easy
to understand.
Consequently, this patch also proxies `_unique_id` through PluginProxy
as this is required by the overlap whitelist algorithm.
This fixes #1340
|
| |\
| |
| |
| |
| | |
Hide runtime dependencies from elements
See merge request BuildStream/buildstream!2052
|
| | | |
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This is a large breaking change, a summary of the changes are that:
* The Scope type is now private, since Element plugins do not have
the choice to view any other scopes.
* Element.dependencies() API change
Now it accepts a "selection" (sequence) of dependency elements, so
that Element.dependencies() can iterate over a collection of dependencies,
ensuring that we iterate over every element only once even when we
need to iterate over multiple element's dependencies.
The old API is moved to Element._dependencies() and still used internally.
* Element.stage_dependency_artifacts() API change
This gets the same treatment as Element.dependencies(), and the old
API is also preserved as Element._stage_dependency_artifacts(), so
that the CLI can stage things for `bst artifact checkout` and such.
* Element.search() API change
The Scope argument is removed, and the old API is preserved as
Element._search() temporarily, until we can remove this completely.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
This prepares the ground for policing the dependencies which are visible
to an Element plugin, such that plugins are only allowed to see the
elements in their Scope.BUILD scope, even if they call Element.dependencies()
on a dependency.
This commit does the following:
* Element.dependencies() is now a user facing frontend which yields
ElementProxy elements instead of Elements.
* Various core codepaths have been updated to call the internal
Element._dependencies() codepath which still returns Elements.
|
| | |
| |
| |
| |
| |
| | |
This will be returned to Element plugins in place of real plugins,
allowing the core to police what Element plugins are allowed to access
more strictly.
|
| | | |
|
| |/ |
|
| |\
| |
| |
| |
| | |
Add element-level source cache
See merge request BuildStream/buildstream!2044
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
The `directory` value determines where a source is staged within the
build root of an element, however, it does not directly affect
individual sources.
With this change the sources will individually be cached in CAS
independent of the value of `directory`. `ElementSources` will use the
value of `directory` when staging all element sources into the build
root.
This results in a cache key change as the `directory` value is moved
from the unique key of individual sources to the unique key of
`ElementSources`.
This is in preparation for #1274.
|
| | |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| |
| | |
Sources have been cached in CAS individually, except for sources that
transform other sources, which have been cached combined with all
previous sources of the element. This caching structure may be confusing
as sources are specified in the element as a list and this is not a good
fit for #1274 where we want to support caching individual sources in a
Remote Asset server with a BuildStream-independent URI (especially the
`directory` configuration would be problematic).
This replaces the combined caching of 'previous' sources with an
element-level source cache, which caches all sources of an element
staged together. Sources that don't depend on previous sources are still
cached individually.
This also makes it possible to add a list of all element sources to the
source proto used by the element-level source cache.
|
| | | |
|
| | | |
|
| |/ |
|
| |\
| |
| |
| |
| | |
_stream.py: In source_push() don't skip fetching for cached elements
See merge request BuildStream/buildstream!2054
|
| | |
| |
| |
| |
| | |
Test that source push succeeds if the source needs to be fetched even
if the artifact of the corresponding element is already cached.
|
| |/
|
|
|
|
| |
`skip_cached` skips elements with a cached artifact. However, for
`source_push()` we need the sources of an element and having a cached
artifact does not guarantee that the sources are cached, too.
|
| |\
| |
| |
| |
| | |
.gitlab-ci.yml: Remove the WSL tests, we're not using this anymore.
See merge request BuildStream/buildstream!2055
|
| |/
|
|
|
|
|
|
|
|
|
|
| |
Our code does nothing special for WSL, and modern versions of WSL are
implemented in such a way that we need not worry about this working
on WSL anymore.
As agreed on the mailing list, let's shut this down.
Mailing list link:
https://lists.apache.org/thread.html/rc49b6416980093ae9e73fda923e43ed668daa0c2a8d1bc94e52b774f%40%3Cdev.buildstream.apache.org%3E
|
| |\
| |
| |
| |
| | |
Remove unused Scope argument from artifact name related APIs.
See merge request BuildStream/buildstream!2051
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
| |
Additionally, this reverts terminology back to calling these "artifact names",
and not "artifact refs", which is a terminology which crept in from various
underlying implementations.
Summary of changes:
* _artifact.py:
- get_dependency_refs() renamed to get_dependency_artifact_names()
- get_dependency_artifact_names() loses the Scope argument
- Consequently, a huge and needless XXX comment is removed
* _artifactelement.py:
- _new_from_artifact_ref() renamed to _new_from_artifact_name()
- get_dependency_refs() renamed to get_dependency_artifact_names()
- get_dependency_artifact_names() loses the Scope argument
* _project.py:
- Now call _new_from_artifact_name()
- Removes a legacy XXX comment which is not particularly relevant
* element.py:
- __get_dependency_refs() renamed to __get_dependency_artifact_names()
- Adapt __get_last_build_artifact() to updated API names.
|
| |\
| |
| |
| |
| | |
Add some type informations to remove non-typechecked parts of the code
See merge request BuildStream/buildstream!2048
|
| | |
| |
| |
| | |
pytest now provides type annotations
|
| | |
| |
| |
| |
| | |
This does not contain all the possible needed annotations, but just
enough to have mypy pass.
|
| | | |
|
| | | |
|
| | |
| |
| |
| | |
This helps with type checking and give better feedback to mypy
|
| |/
|
|
| |
This enables type checking from mypy on the cython module
|
| |\
| |
| |
| |
| | |
scriptelement.py: Remove legacy manual creation of staging directory.
See merge request BuildStream/buildstream!2050
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
| |
This branch statement comes from a time before virtual directories
when we were using os.makedirs() to create the target directory, and
was then ported to use virtual directories for that purpose.
However, during the same migration, the Element.stage_artifact()
function was made to guarantee the existence of the staging root
location, with a call to:
vbasedir.descend( ... staging directory ..., create=True)
This makes the branch statement in scriptelement.py redundant.
|
| |\
| |
| |
| |
| | |
scriptelement.py: Simplify staging in the case no layout is provided
See merge request BuildStream/buildstream!2049
|
| |/ |
|
| |\
| |
| |
| |
| | |
downloadablefilesource: Remove deprecated methods in python3.8
See merge request BuildStream/buildstream!2046
|
| |/
|
|
|
|
| |
splituser and such are deprecated since python3.8
We should stop using them
|
| |\
| |
| |
| |
| | |
Add artifact cache testing with buildbarn Asset API implementation
See merge request BuildStream/buildstream!2037
|
| | |
| |
| |
| |
| | |
This new endpoint targets a buildbarn deployment with bb-storage
alongside bb-asset-hub for index storage.
|
| | | |
|
| | | |
|
| |/
|
|
|
| |
This enables for seperate index/storage artifact servers
to be configured by environment variables passed through tox.
|
| |\
| |
| |
| |
| | |
_elementsources.py: Fix source names in unique key
See merge request BuildStream/buildstream!2023
|
| |/
|
|
|
|
|
|
|
|
|
|
|
|
| |
The goal was to include the source plugin kind in the element cache key
as the unique key of a source may not be unique across different source
plugins. This is the source equivalent of the `element-plugin-name`
value in the element cache key.
However, `Source._get_source_name()` was the wrong method for this as
that also includes the key itself, which may not even be set yet.
This results in a cache key change.
Fixes: 3953bcc6 ("element.py: clobber sources with workspace")
|
