diff options
| author | Tristan Van Berkom <tristan.van.berkom@gmail.com> | 2020-09-19 09:48:24 +0000 |
|---|---|---|
| committer | Tristan Van Berkom <tristan.van.berkom@gmail.com> | 2020-09-19 09:48:24 +0000 |
| commit | a7e2c92885711336a6774792a9d160ea3fe335bf (patch) | |
| tree | a280a5e144e82f5248ea8cb3f034c34065437938 | |
| parent | 1656082576b8e4ab67cb16300ad75c60954cb420 (diff) | |
| parent | c30a1afff47397bb4f8ff2d84eab6a96ac0816a5 (diff) | |
| download | buildstream-a7e2c92885711336a6774792a9d160ea3fe335bf.tar.gz | |
Merge branch 'tristan/dependency-config' into 'master'
Implement Element.configure_dependencies()
See merge request BuildStream/buildstream!2032
66 files changed, 1472 insertions, 549 deletions
@@ -2,6 +2,15 @@ (unreleased) ============ +Format +------ + o The `script` element no longer has a `layout` configuration directly, and now + exposes a new `location` dependency configuration instead. + + o The BuildElement now has a new `location` dependency configuration, allowing + BuildElement plugins to also stage dependencies into custom locations in + the sandbox. + Core ---- @@ -11,6 +20,9 @@ Core - Element.search() Elements now can only ever see dependencies in their build scope. + * BREAKING CHANGE: Changed ScriptElement.layout_add() API to take Element instances + in place of Element names + ================== buildstream 1.93.5 ================== diff --git a/doc/source/format_declaring.rst b/doc/source/format_declaring.rst index 391591530..98b5f926e 100644 --- a/doc/source/format_declaring.rst +++ b/doc/source/format_declaring.rst @@ -419,12 +419,6 @@ Attributes: The :ref:`element name <format_element_names>` to depend on. -* ``type`` - - This attribute is used to express the :ref:`dependency type <format_dependencies_types>`. - This field is not permitted in :ref:`Build-Depends <format_build_depends>` or - :ref:`Runtime-Depends <format_runtime_depends>`. - * ``junction`` This attribute can be used to specify the junction portion of the :ref:`element name <format_element_names>` @@ -437,6 +431,12 @@ Attributes: In the case that a *junction* is specified, the ``filename`` attribute indicates an element in the *junctioned project*. +* ``type`` + + This attribute is used to express the :ref:`dependency type <format_dependencies_types>`. + This field is not permitted in :ref:`Build-Depends <format_build_depends>` or + :ref:`Runtime-Depends <format_runtime_depends>`. + * ``strict`` This attribute can be used to specify that this element should @@ -447,6 +447,42 @@ Attributes: verbatim in the output of the depending element, for instance when static linking is in use. +* ``config`` + + This attribute defines the custom :term:`dependency configuration <Dependency configuration>`, + which is supported by select :mod:`Element <buildstream.element>` implementations. + + Elements which support :term:`dependency configuration <Dependency configuration>` do so + by implementing the + :func:`Element.configure_dependencies() <buildstream.element.Element.configure_dependencies>` + abstract method. It is up to each element or abstract element class to + document what is supported in their :term:`dependency configuration <Dependency configuration>`. + + .. attention:: + + It is illegal to declare :term:`dependency configuration <Dependency configuration>` + on runtime dependencies, since runtime dependencies are not visible to the depending + element. + + +Redundant dependency declarations +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +It is permitted to declare dependencies multiple times on the same element in the same +element declaration, the result will be an inclusive OR of all configurations you have +expressed in the redundant dependencies on the same element. + +* If a dependency is defined once as a ``build`` dependency and once as a ``runtime`` + :ref:`dependency type <format_dependencies_types>`, then the resulting dependency + type will be ``all`` + +* If any of the redundantly declared dependencies are specified as ``strict``, then + the resulting dependency will be ``strict``. + +Declaring redundant dependencies on the same element can be interesting when you +need to specify multiple :term:`dependency configurations <Dependency configuration>` +for the same element. For example, one might want to stage the same dependency +in multiple locations in the build sandbox. + Cross-junction dependencies ~~~~~~~~~~~~~~~~~~~~~~~~~~~ diff --git a/doc/source/main_glossary.rst b/doc/source/main_glossary.rst index c97277532..aa0c6da1f 100644 --- a/doc/source/main_glossary.rst +++ b/doc/source/main_glossary.rst @@ -44,6 +44,15 @@ Glossary See :ref:`Dependencies document <format_dependencies>` for more details. + Dependency configuration + Additional custom YAML configuration which is used to define + an :term:`Element's <Element>` relationship with it's :term:`Dependency <Dependency>`. + + This is supported on limited :term:`Element <Element>` implementations, and + each :term:`Element <Element>` defines what configuration it supports. + + See the :ref:`dependency documentation <format_dependencies>` for details + on dependency configuration. Element An atom of a :term:`BuildStream project <Project>`. Projects consist of diff --git a/src/buildstream/__init__.py b/src/buildstream/__init__.py index 4d151873d..c6722023f 100644 --- a/src/buildstream/__init__.py +++ b/src/buildstream/__init__.py @@ -30,12 +30,12 @@ if "_BST_COMPLETION" not in os.environ: from .utils import UtilError, ProgramNotFoundError from .sandbox import Sandbox, SandboxFlags, SandboxCommandError - from .types import CoreWarnings + from .types import CoreWarnings, OverlapAction from .node import MappingNode, Node, ProvenanceInformation, ScalarNode, SequenceNode from .plugin import Plugin from .source import Source, SourceError, SourceFetcher from .downloadablefilesource import DownloadableFileSource - from .element import Element, ElementError + from .element import Element, ElementError, DependencyConfiguration from .buildelement import BuildElement from .scriptelement import ScriptElement diff --git a/src/buildstream/_elementproxy.py b/src/buildstream/_elementproxy.py index acb08ce8b..a7b1f09a0 100644 --- a/src/buildstream/_elementproxy.py +++ b/src/buildstream/_elementproxy.py @@ -18,7 +18,7 @@ # Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> from typing import TYPE_CHECKING, cast, Optional, Iterator, Dict, List, Sequence -from .types import _Scope +from .types import _Scope, OverlapAction from .utils import FileListResult from ._pluginproxy import PluginProxy @@ -96,13 +96,23 @@ class ElementProxy(PluginProxy): sandbox: "Sandbox", *, path: str = None, + action: str = OverlapAction.WARNING, include: Optional[List[str]] = None, exclude: Optional[List[str]] = None, orphans: bool = True ) -> FileListResult: - return cast("Element", self._plugin).stage_artifact( - sandbox, path=path, include=include, exclude=exclude, orphans=orphans - ) + + owner = cast("Element", self._owner) + element = cast("Element", self._plugin) + + assert owner._overlap_collector is not None, "Attempted to stage artifacts outside of Element.stage()" + + with owner._overlap_collector.session(action, path): + result = element._stage_artifact( + sandbox, path=path, action=action, include=include, exclude=exclude, orphans=orphans, owner=owner + ) + + return result def stage_dependency_artifacts( self, @@ -110,6 +120,7 @@ class ElementProxy(PluginProxy): selection: Sequence["Element"] = None, *, path: str = None, + action: str = OverlapAction.WARNING, include: Optional[List[str]] = None, exclude: Optional[List[str]] = None, orphans: bool = True @@ -120,7 +131,7 @@ class ElementProxy(PluginProxy): if selection is None: selection = [cast("Element", self._plugin)] cast("Element", self._owner).stage_dependency_artifacts( - sandbox, selection, path=path, include=include, exclude=exclude, orphans=orphans + sandbox, selection, path=path, action=action, include=include, exclude=exclude, orphans=orphans ) def integrate(self, sandbox: "Sandbox") -> None: @@ -154,3 +165,20 @@ class ElementProxy(PluginProxy): def _file_is_whitelisted(self, path): return cast("Element", self._plugin)._file_is_whitelisted(path) + + def _stage_artifact( + self, + sandbox: "Sandbox", + *, + path: str = None, + action: str = OverlapAction.WARNING, + include: Optional[List[str]] = None, + exclude: Optional[List[str]] = None, + orphans: bool = True, + owner: Optional["Element"] = None + ) -> FileListResult: + owner = cast("Element", self._owner) + element = cast("Element", self._plugin) + return element._stage_artifact( + sandbox, path=path, action=action, include=include, exclude=exclude, orphans=orphans, owner=owner + ) diff --git a/src/buildstream/_loader/__init__.py b/src/buildstream/_loader/__init__.py index a4be9cfe5..1eee7c941 100644 --- a/src/buildstream/_loader/__init__.py +++ b/src/buildstream/_loader/__init__.py @@ -19,6 +19,6 @@ from .types import Symbol from .metasource import MetaSource -from .loadelement import LoadElement, Dependency +from .loadelement import LoadElement, Dependency, DependencyType from .loadcontext import LoadContext from .loader import Loader diff --git a/src/buildstream/_loader/loadelement.pyi b/src/buildstream/_loader/loadelement.pyi index 67b14df8f..717792bd5 100644 --- a/src/buildstream/_loader/loadelement.pyi +++ b/src/buildstream/_loader/loadelement.pyi @@ -5,6 +5,7 @@ from ..node import Node, ProvenanceInformation def extract_depends_from_node(node: Node) -> List[Dependency]: ... class Dependency: ... +class DependencyType: ... class LoadElement: first_pass: bool diff --git a/src/buildstream/_loader/loadelement.pyx b/src/buildstream/_loader/loadelement.pyx index 01334d124..80b96cd2c 100644 --- a/src/buildstream/_loader/loadelement.pyx +++ b/src/buildstream/_loader/loadelement.pyx @@ -23,7 +23,6 @@ from pyroaring import BitMap, FrozenBitMap # pylint: disable=no-name-in-module from .._exceptions import LoadError from ..exceptions import LoadErrorReason -from ..element import Element from ..node cimport MappingNode, Node, ProvenanceInformation, ScalarNode, SequenceNode from .types import Symbol @@ -37,6 +36,22 @@ cdef int _next_synthetic_counter(): return _counter +# DependencyType +# +# A bitfield to represent dependency types +# +cpdef enum DependencyType: + + # A build dependency + BUILD = 0x001 + + # A runtime dependency + RUNTIME = 0x002 + + # Both build and runtime dependencies + ALL = 0x003 + + # Dependency(): # # Early stage data model for dependencies objects, the LoadElement has @@ -50,22 +65,24 @@ cdef int _next_synthetic_counter(): # # Args: # element (LoadElement): a LoadElement on which there is a dependency -# dep_type (str): the type of dependency this dependency link is +# dep_type (DependencyType): the type of dependency this dependency link is # cdef class Dependency: cdef readonly LoadElement element # The resolved LoadElement - cdef readonly str dep_type # The dependency type (runtime or build or both) + cdef readonly int dep_type # The dependency type (runtime or build or both) cdef readonly str name # The project local dependency name cdef readonly str junction # The junction path of the dependency name, if any cdef readonly bint strict # Whether this is a strict dependency cdef Node _node # The original node of the dependency + cdef readonly list config_nodes # The custom config nodes for Element.configure_dependencies() - def __cinit__(self, element=None, dep_type=None): + def __cinit__(self, LoadElement element = None, int dep_type = DependencyType.ALL): self.element = element self.dep_type = dep_type self.name = None self.junction = None self.strict = False + self.config_nodes = None self._node = None # provenance @@ -77,6 +94,17 @@ cdef class Dependency: def provenance(self): return self._node.get_provenance() + # path + # + # The path of the dependency represented as a single string, + # instead of junction and name being separate. + # + @property + def path(self): + if self.junction is not None: + return "{}:{}".format(self.junction, self.name) + return self.name + # set_element() # # Sets the resolved LoadElement @@ -98,41 +126,52 @@ cdef class Dependency: # # Args: # dep (Node): A node to load the dependency from - # default_dep_type (str): The default dependency type + # default_dep_type (DependencyType): The default dependency type # - cdef load(self, Node dep, str default_dep_type): - cdef str dep_type + cdef load(self, Node dep, int default_dep_type): + cdef str parsed_type + cdef MappingNode config_node self._node = dep self.element = None if type(dep) is ScalarNode: self.name = dep.as_str() - self.dep_type = default_dep_type + self.dep_type = default_dep_type or DependencyType.ALL self.junction = None self.strict = False elif type(dep) is MappingNode: if default_dep_type: - (<MappingNode> dep).validate_keys(['filename', 'junction', 'strict']) - dep_type = default_dep_type + (<MappingNode> dep).validate_keys([Symbol.FILENAME, Symbol.JUNCTION, Symbol.STRICT, Symbol.CONFIG]) + self.dep_type = default_dep_type else: - (<MappingNode> dep).validate_keys(['filename', 'type', 'junction', 'strict']) - - # Make type optional, for this we set it to None - dep_type = (<MappingNode> dep).get_str(<str> Symbol.TYPE, None) - if dep_type is None or dep_type == <str> Symbol.ALL: - dep_type = None - elif dep_type not in [Symbol.BUILD, Symbol.RUNTIME]: + (<MappingNode> dep).validate_keys([Symbol.FILENAME, Symbol.TYPE, Symbol.JUNCTION, Symbol.STRICT, Symbol.CONFIG]) + + # Resolve the DependencyType + parsed_type = (<MappingNode> dep).get_str(<str> Symbol.TYPE, None) + if parsed_type is None or parsed_type == <str> Symbol.ALL: + self.dep_type = DependencyType.ALL + elif parsed_type == <str> Symbol.BUILD: + self.dep_type = DependencyType.BUILD + elif parsed_type == <str> Symbol.RUNTIME: + self.dep_type = DependencyType.RUNTIME + else: provenance = dep.get_scalar(Symbol.TYPE).get_provenance() raise LoadError("{}: Dependency type '{}' is not 'build', 'runtime' or 'all'" - .format(provenance, dep_type), LoadErrorReason.INVALID_DATA) + .format(provenance, parsed_type), LoadErrorReason.INVALID_DATA) self.name = (<MappingNode> dep).get_str(<str> Symbol.FILENAME) - self.dep_type = dep_type self.junction = (<MappingNode> dep).get_str(<str> Symbol.JUNCTION, None) self.strict = (<MappingNode> dep).get_bool(<str> Symbol.STRICT, False) + config_node = (<MappingNode> dep).get_mapping(<str> Symbol.CONFIG, None) + if config_node: + if self.dep_type == DependencyType.RUNTIME: + raise LoadError("{}: Specifying 'config' for a runtime dependency is not allowed" + .format(config_node.get_provenance()), LoadErrorReason.INVALID_DATA) + self.config_nodes = [config_node] + # Here we disallow explicitly setting 'strict' to False. # # This is in order to keep the door open to allowing the project.conf @@ -152,7 +191,7 @@ cdef class Dependency: # Only build dependencies are allowed to be strict # - if self.strict and self.dep_type == Symbol.RUNTIME: + if self.strict and self.dep_type == DependencyType.RUNTIME: raise LoadError("{}: Runtime dependency {} specified as `strict`.".format(self.provenance, self.name), LoadErrorReason.INVALID_DATA, detail="Only dependencies required at build time may be declared `strict`.") @@ -169,6 +208,22 @@ cdef class Dependency: if not self.junction and ':' in self.name: self.junction, self.name = self.name.rsplit(':', maxsplit=1) + # merge() + # + # Merge the attributes of an existing dependency into this dependency + # + # Args: + # other (Dependency): The dependency to merge into this one + # + cdef merge(self, Dependency other): + self.dep_type = self.dep_type | other.dep_type + self.strict = self.strict or other.strict + + if self.config_nodes and other.config_nodes: + self.config_nodes.extend(other.config_nodes) + else: + self.config_nodes = self.config_nodes or other.config_nodes + # LoadElement(): # @@ -242,6 +297,9 @@ cdef class LoadElement: # store the link target and provenance # if self.kind == 'link': + # Avoid cyclic import here + from ..element import Element + element = Element._new_from_load_element(self) element._initialize_state() @@ -345,9 +403,9 @@ def _dependency_cmp(Dependency dep_a, Dependency dep_b): # If there are no inter element dependencies, place # runtime only dependencies last if dep_a.dep_type != dep_b.dep_type: - if dep_a.dep_type == Symbol.RUNTIME: + if dep_a.dep_type == DependencyType.RUNTIME: return 1 - elif dep_b.dep_type == Symbol.RUNTIME: + elif dep_b.dep_type == DependencyType.RUNTIME: return -1 # All things being equal, string comparison. @@ -424,13 +482,12 @@ def sort_dependencies(LoadElement element, set visited): # Args: # node (Node): A YAML loaded dictionary # key (str): the key on the Node corresponding to the dependency type -# default_dep_type (str): type to give to the dependency -# acc (list): a list in which to add the loaded dependencies -# rundeps (dict): a dictionary mapping dependency (junction, name) to dependency for runtime deps -# builddeps (dict): a dictionary mapping dependency (junction, name) to dependency for build deps +# default_dep_type (DependencyType): type to give to the dependency +# acc (dict): a dict in which to add the loaded dependencies # -cdef void _extract_depends_from_node(Node node, str key, str default_dep_type, list acc, dict rundeps, dict builddeps) except *: +cdef void _extract_depends_from_node(Node node, str key, int default_dep_type, dict acc) except *: cdef SequenceNode depends = node.get_sequence(key, []) + cdef Dependency existing_dep cdef Node dep_node cdef tuple deptup @@ -438,21 +495,13 @@ cdef void _extract_depends_from_node(Node node, str key, str default_dep_type, l dependency = Dependency() dependency.load(dep_node, default_dep_type) deptup = (dependency.junction, dependency.name) - if dependency.dep_type in [Symbol.BUILD, None]: - if deptup in builddeps: - raise LoadError("{}: Duplicate build dependency found at {}." - .format(dependency.provenance, builddeps[deptup].provenance), - LoadErrorReason.DUPLICATE_DEPENDENCY) - else: - builddeps[deptup] = dependency - if dependency.dep_type in [Symbol.RUNTIME, None]: - if deptup in rundeps: - raise LoadError("{}: Duplicate runtime dependency found at {}." - .format(dependency.provenance, rundeps[deptup].provenance), - LoadErrorReason.DUPLICATE_DEPENDENCY) - else: - rundeps[deptup] = dependency - acc.append(dependency) + + # Accumulate dependencies, merging any matching elements along the way + existing_dep = <Dependency> acc.get(deptup, None) + if existing_dep is not None: + existing_dep.merge(dependency) + else: + acc[deptup] = dependency # Now delete the field, we dont want it anymore node.safe_del(key) @@ -473,10 +522,8 @@ cdef void _extract_depends_from_node(Node node, str key, str default_dep_type, l # (list): a list of Dependency objects # def extract_depends_from_node(Node node): - cdef list acc = [] - cdef dict rundeps = {} - cdef dict builddeps = {} - _extract_depends_from_node(node, <str> Symbol.BUILD_DEPENDS, <str> Symbol.BUILD, acc, rundeps, builddeps) - _extract_depends_from_node(node, <str> Symbol.RUNTIME_DEPENDS, <str> Symbol.RUNTIME, acc, rundeps, builddeps) - _extract_depends_from_node(node, <str> Symbol.DEPENDS, None, acc, rundeps, builddeps) - return acc + cdef dict acc = {} + _extract_depends_from_node(node, <str> Symbol.BUILD_DEPENDS, <int> DependencyType.BUILD, acc) + _extract_depends_from_node(node, <str> Symbol.RUNTIME_DEPENDS, <int> DependencyType.RUNTIME, acc) + _extract_depends_from_node(node, <str> Symbol.DEPENDS, <int> 0, acc) + return [dep for dep in acc.values()] diff --git a/src/buildstream/_loader/loader.py b/src/buildstream/_loader/loader.py index 94ee9078b..6ebf89a05 100644 --- a/src/buildstream/_loader/loader.py +++ b/src/buildstream/_loader/loader.py @@ -31,7 +31,7 @@ from .._includes import Includes from ._loader import valid_chars_name from .types import Symbol from . import loadelement -from .loadelement import LoadElement, Dependency, extract_depends_from_node +from .loadelement import LoadElement, Dependency, DependencyType, extract_depends_from_node from ..types import CoreWarnings, _KeyStrength from .._message import Message, MessageType @@ -147,7 +147,7 @@ class Loader: # Pylint is not very happy with Cython and can't understand 'dependencies' is a list dummy_target.dependencies.extend( # pylint: disable=no-member - Dependency(element, Symbol.RUNTIME) for element in target_elements + Dependency(element, DependencyType.RUNTIME) for element in target_elements ) with PROFILER.profile(Topics.CIRCULAR_CHECK, "_".join(targets)): diff --git a/src/buildstream/_overlapcollector.py b/src/buildstream/_overlapcollector.py new file mode 100644 index 000000000..30ecfa32c --- /dev/null +++ b/src/buildstream/_overlapcollector.py @@ -0,0 +1,328 @@ +# +# Copyright (C) 2020 Codethink Limited +# +# This program is free software; you can redistribute it and/or +# modify it under the terms of the GNU Lesser General Public +# License as published by the Free Software Foundation; either +# version 2 of the License, or (at your option) any later version. +# +# This library is distributed in the hope that it will be useful, +# but WITHOUT ANY WARRANTY; without even the implied warranty of +# MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU +# Lesser General Public License for more details. +# +# You should have received a copy of the GNU Lesser General Public +# License along with this library. If not, see <http://www.gnu.org/licenses/>. +# +# Authors: +# Tristan Van Berkom <tristan.vanberkom@codethink.co.uk> + +import os +from contextlib import contextmanager +from typing import TYPE_CHECKING, Optional, List, Tuple +from .plugin import Plugin +from .types import CoreWarnings, OverlapAction +from .utils import FileListResult + +if TYPE_CHECKING: + from typing import Dict + + # pylint: disable=cyclic-import + from .element import Element + + # pylint: enable=cyclic-import + + +# OverlapCollector() +# +# Collects results of Element.stage_artifact() and saves +# them in order to raise a proper overlap error at the end +# of staging. +# +# Args: +# element (Element): The element for which we are staging artifacts +# +class OverlapCollector: + def __init__(self, element: "Element"): + + # The Element we are staging for, on which we'll issue warnings + self._element = element # type: Element + + # The list of sessions + self._sessions = [] # type: List[OverlapCollectorSession] + + # The active session, if any + self._session = None # type: Optional[OverlapCollectorSession] + + # session() + # + # Create a session for collecting overlaps, calls to OverlapCollector.collect_stage_result() + # are expected to always occur within the context of a session (this context manager). + # + # Upon exiting this context, warnings and/or errors will be issued for any overlaps + # which occurred either as a result of overlapping files within this session, or + # as a result of files staged during this session, overlapping with files staged in + # previous sessions in this OverlapCollector. + # + # Args: + # action (OverlapAction): The action to take for this overall session's overlaps with other sessions + # location (str): The Sandbox relative location this session was created for + # + @contextmanager + def session(self, action: str, location: Optional[str]): + assert self._session is None, "Stage session already started" + + if location is None: + location = "/" + + self._session = OverlapCollectorSession(self._element, action, location) + + # Run code body where staging results can be collected. + yield + + # Issue warnings for the current session, passing along previously completed sessions + self._session.warnings(self._sessions) + + # Store the newly ended session and end the session + self._sessions.append(self._session) + self._session = None + + # collect_stage_result() + # + # Collect and accumulate results of Element.stage_artifact() + # + # Args: + # element (Element): The name of the element staged + # result (FileListResult): The result of Element.stage_artifact() + # + def collect_stage_result(self, element: "Element", result: FileListResult): + assert self._session is not None, "Staging files outside of staging session" + + self._session.collect_stage_result(element, result) + + +# OverlapCollectorSession() +# +# Collect the results of a single session +# +# Args: +# element (Element): The element for which we are staging artifacts +# action (OverlapAction): The action to take for this overall session's overlaps with other sessions +# location (str): The Sandbox relative location this session was created for +# +class OverlapCollectorSession: + def __init__(self, element: "Element", action: str, location: str): + + # The Element we are staging for, on which we'll issue warnings + self._element = element # type: Element + + # The OverlapAction for this session + self._action = action # type: str + + # The Sandbox relative directory this session was created for + self._location = location # type: str + + # Dictionary of files which were ignored (See FileListResult()), keyed by element unique ID + self._ignored = {} # type: Dict[int, List[str]] + + # Dictionary of files which were staged, keyed by element unique ID + self._files_written = {} # type: Dict[int, List[str]] + + # Dictionary of element IDs which overlapped, keyed by the file they overlap on + self._overlaps = {} # type: Dict[str, List[int]] + + # collect_stage_result() + # + # Collect and accumulate results of Element.stage_artifact() + # + # Args: + # element (Element): The name of the element staged + # result (FileListResult): The result of Element.stage_artifact() + # + def collect_stage_result(self, element: "Element", result: FileListResult): + + for overwritten_file in result.overwritten: + + overlap_list = None + try: + overlap_list = self._overlaps[overwritten_file] + except KeyError: + + # Create a fresh list + # + self._overlaps[overwritten_file] = overlap_list = [] + + # Search files which were staged in this session, start the + # list off with the bottom most element + # + for element_id, staged_files in self._files_written.items(): + if overwritten_file in staged_files: + overlap_list.append(element_id) + break + + # Add the currently staged element to the overlap list, it might be + # the only element in the list if it overlaps with a file staged + # from a previous session. + # + overlap_list.append(element._unique_id) + + # Record written files and ignored files. + # + self._files_written[element._unique_id] = result.files_written + if result.ignored: + self._ignored[element._unique_id] = result.ignored + + # warnings() + # + # Issue any warnings as a batch as a result of staging artifacts, + # based on the results collected with collect_stage_result(). + # + # Args: + # sessions (list): List of previously completed sessions + # + def warnings(self, sessions: List["OverlapCollectorSession"]): + + # Collect a table of filenames which overlapped something from outside of this session. + # + external_overlaps = {} # type: Dict[str, int] + + # + # First issue the warnings for this session + # + if self._overlaps: + overlap_warning = False + detail = "Staged files overwrite existing files in staging area: {}\n".format(self._location) + for filename, element_ids in self._overlaps.items(): + + # If there is only one element in the overlap list, it means it has + # overlapped a file from a previous session. + # + # Ignore it and handle the warning below + # + if len(element_ids) == 1: + external_overlaps[filename] = element_ids[0] + continue + + # Filter whitelisted elements out of the list of overlapping elements + # + # Ignore the bottom-most element as it does not overlap anything. + # + overlapping_element_ids = element_ids[1:] + warning_elements = self._filter_whitelisted(filename, overlapping_element_ids) + + if warning_elements: + overlap_warning = True + + detail += self._overlap_detail(filename, warning_elements, element_ids) + + if overlap_warning: + self._element.warn( + "Non-whitelisted overlaps detected", detail=detail, warning_token=CoreWarnings.OVERLAPS + ) + + if self._ignored: + detail = "Not staging files which would replace non-empty directories in staging area: {}\n".format( + self._location + ) + for element_id, ignored_filenames in self._ignored.items(): + element = Plugin._lookup(element_id) + detail += "\nFrom {}:\n".format(element._get_full_name()) + detail += " " + " ".join( + ["{}\n".format(os.path.join(self._location, filename)) for filename in ignored_filenames] + ) + self._element.warn( + "Not staging files which would have replaced non-empty directories", + detail=detail, + warning_token=CoreWarnings.UNSTAGED_FILES, + ) + + if external_overlaps and self._action != OverlapAction.IGNORE: + detail = "Detected file overlaps while staging elements into: {}\n".format(self._location) + + # Find the session responsible for the overlap + # + for filename, element_id in external_overlaps.items(): + absolute_filename = os.path.join(self._location, filename) + overlapped_id, location = self._search_stage_element(absolute_filename, sessions) + element = Plugin._lookup(element_id) + overlapped = Plugin._lookup(overlapped_id) + detail += "{}: {} overlaps files previously staged by {} in: {}\n".format( + absolute_filename, element._get_full_name(), overlapped._get_full_name(), location + ) + + if self._action == OverlapAction.WARNING: + self._element.warn("Overlaps detected", detail=detail, warning_token=CoreWarnings.OVERLAPS) + else: + from .element import ElementError + + raise ElementError("Overlaps detected", detail=detail, reason="overlaps") + + # _search_stage_element() + # + # Search the sessions list for the element responsible for staging the given file + # + # Args: + # filename (str): The sandbox relative file which was overwritten + # sessions (List[OverlapCollectorSession]) + # + # Returns: + # element_id (int): The unique ID of the element responsible + # location (str): The sandbox relative staging location where element_id was staged + # + def _search_stage_element(self, filename: str, sessions: List["OverlapCollectorSession"]) -> Tuple[int, str]: + for session in reversed(sessions): + for element_id, staged_files in session._files_written.items(): + if any( + staged_file + for staged_file in staged_files + if os.path.join(session._location, staged_file) == filename + ): + return element_id, session._location + + assert False, "Could not find element responsible for staging: {}".format(filename) + + # Silence the linter with an unreachable return statement + return None, None + + # _filter_whitelisted() + # + # Args: + # filename (str): The staging session relative filename + # element_ids (List[int]): Ordered list of elements + # + # Returns: + # (List[Element]): The list of element objects which are not whitelisted + # + def _filter_whitelisted(self, filename: str, element_ids: List[int]): + overlap_elements = [] + + for element_id in element_ids: + element = Plugin._lookup(element_id) + if not element._file_is_whitelisted(filename): + overlap_elements.append(element) + + return overlap_elements + + # _overlap_detail() + # + # Get a string to describe overlaps on a filename + # + # Args: + # filename (str): The filename being overlapped + # overlap_elements (List[Element]): A list of Elements overlapping + # element_ids (List[int]): The ordered ID list of elements which staged this file + # + def _overlap_detail(self, filename, overlap_elements, element_ids): + filename = os.path.join(self._location, filename) + if overlap_elements: + overlap_element_names = [element._get_full_name() for element in overlap_elements] + overlap_order_elements = [Plugin._lookup(element_id) for element_id in element_ids] + overlap_order_names = [element._get_full_name() for element in overlap_order_elements] + return "{}: {} {} not permitted to overlap other elements, order {} \n".format( + filename, + " and ".join(overlap_element_names), + "is" if len(overlap_element_names) == 1 else "are", + " above ".join(reversed(overlap_order_names)), + ) + else: + return "" diff --git a/src/buildstream/buildelement.py b/src/buildstream/buildelement.py index a7900a25d..6020e67db 100644 --- a/src/buildstream/buildelement.py +++ b/src/buildstream/buildelement.py @@ -45,6 +45,41 @@ If you are targetting Linux, ones known to work are the ones used by the `project.conf <https://gitlab.com/freedesktop-sdk/freedesktop-sdk/blob/freedesktop-sdk-18.08.21/project.conf#L74>`_ +Location for staging dependencies +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ +The BuildElement supports the "location" :term:`dependency configuration <Dependency configuration>`, +which means you can use this configuration for any BuildElement class. + +The "location" configuration defines where the dependency will be staged in the +build sandbox. + +**Example:** + +Here is an example of how one might stage some dependencies into +an alternative location while staging some elements in the sandbox root. + +.. code:: yaml + + # Stage these build dependencies in /opt + # + build-depends: + - baseproject.bst:opt-dependencies.bst + config: + location: /opt + + # Stage these tools in "/" and require them as + # runtime dependencies. + depends: + - baseproject.bst:base-tools.bst + +.. note:: + + The order of dependencies specified is not significant. + + The staging locations will be sorted so that elements are staged in parent + directories before subdirectories. + + Location for running commands ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ The ``command-subdir`` variable sets where the build commands will be executed, @@ -140,18 +175,6 @@ from .element import Element from .sandbox import SandboxFlags -# This list is preserved because of an unfortunate situation, we -# need to remove these older commands which were secret and never -# documented, but without breaking the cache keys. -_legacy_command_steps = [ - "bootstrap-commands", - "configure-commands", - "build-commands", - "test-commands", - "install-commands", - "strip-commands", -] - _command_steps = ["configure-commands", "build-commands", "install-commands", "strip-commands"] @@ -171,9 +194,31 @@ class BuildElement(Element): self._command_subdir = self.get_variable("command-subdir") # pylint: disable=attribute-defined-outside-init - for command_name in _legacy_command_steps: + for command_name in _command_steps: self.__commands[command_name] = node.get_str_list(command_name, []) + def configure_dependencies(self, dependencies): + + self.__layout = {} # pylint: disable=attribute-defined-outside-init + + # FIXME: Currently this forcefully validates configurations + # for all BuildElement subclasses so they are unable to + # extend the configuration + + for dep in dependencies: + # Determine the location to stage each element, default is "/" + location = "/" + if dep.config: + dep.config.validate_keys(["location"]) + location = dep.config.get_str("location") + try: + element_list = self.__layout[location] + except KeyError: + element_list = [] + self.__layout[location] = element_list + + element_list.append((dep.element, dep.path)) + def preflight(self): pass @@ -193,6 +238,17 @@ class BuildElement(Element): if self.get_variable("notparallel"): dictionary["notparallel"] = True + # Specify the layout in the key, if any of the elements are not going to + # be staged in "/" + # + if any(location for location in self.__layout if location != "/"): + sorted_locations = sorted(self.__layout) + layout_key = { + location: [dependency_path for _, dependency_path in self.__layout[location]] + for location in sorted_locations + } + dictionary["layout"] = layout_key + return dictionary def configure_sandbox(self, sandbox): @@ -203,6 +259,10 @@ class BuildElement(Element): sandbox.mark_directory(build_root) sandbox.mark_directory(install_root) + # Mark the artifact directories in the layout + for location in self.__layout: + sandbox.mark_directory(location, artifact=True) + # Allow running all commands in a specified subdirectory if self._command_subdir: command_dir = os.path.join(build_root, self._command_subdir) @@ -210,23 +270,26 @@ class BuildElement(Element): command_dir = build_root sandbox.set_work_directory(command_dir) - # Tell sandbox which directory is preserved in the finished artifact - sandbox.set_output_directory(install_root) - # Setup environment sandbox.set_environment(self.get_environment()) def stage(self, sandbox): - # Stage deps in the sandbox root - with self.timed_activity("Staging dependencies", silent_nested=True): - self.stage_dependency_artifacts(sandbox) + # First stage it all + # + sorted_locations = sorted(self.__layout) + for location in sorted_locations: + element_list = [element for element, _ in self.__layout[location]] + self.stage_dependency_artifacts(sandbox, element_list, path=location) - # Run any integration commands provided by the dependencies - # once they are all staged and ready - with sandbox.batch(SandboxFlags.NONE, label="Integrating sandbox"): - for dep in self.dependencies(): - dep.integrate(sandbox) + # Now integrate any elements staged in the root + # + root_list = self.__layout.get("/", None) + if root_list: + element_list = [element for element, _ in root_list] + with sandbox.batch(SandboxFlags.NONE), self.timed_activity("Integrating sandbox", silent_nested=True): + for dep in self.dependencies(element_list): + dep.integrate(sandbox) # Stage sources in the build root self.stage_sources(sandbox, self.get_variable("build-root")) diff --git a/src/buildstream/element.py b/src/buildstream/element.py index 0aabe1be6..fd6e2da5c 100644 --- a/src/buildstream/element.py +++ b/src/buildstream/element.py @@ -103,20 +103,21 @@ from .plugin import Plugin from .sandbox import SandboxFlags, SandboxCommandError from .sandbox._config import SandboxConfig from .sandbox._sandboxremote import SandboxRemote -from .types import CoreWarnings, _Scope, _CacheBuildTrees, _KeyStrength +from .types import _Scope, _CacheBuildTrees, _KeyStrength, OverlapAction from ._artifact import Artifact from ._elementproxy import ElementProxy from ._elementsources import ElementSources -from ._loader import Symbol, MetaSource +from ._loader import Symbol, DependencyType, MetaSource +from ._overlapcollector import OverlapCollector from .storage.directory import Directory from .storage._filebaseddirectory import FileBasedDirectory from .storage.directory import VirtualDirectoryError if TYPE_CHECKING: + from typing import Tuple from .node import MappingNode, ScalarNode, SequenceNode from .types import SourceRef - from typing import Tuple # pylint: disable=cyclic-import from .sandbox import Sandbox @@ -148,6 +149,26 @@ class ElementError(BstError): self.collect = collect +class DependencyConfiguration: + """An object representing the configuration of a dependency + + This is used to provide dependency configurations for elements which implement + :func:`Element.configure_dependencies() <buildstream.element.Element.configure_dependencies>` + """ + + def __init__(self, element: "Element", path: str, config: Optional["MappingNode"]): + + self.element = element # type: Element + """The dependency Element""" + + self.path = path # type: str + """The path used to refer to this dependency""" + + self.config = config # type: Optional[MappingNode] + """The custom :term:`dependency configuration <Dependency configuration>`, or ``None`` + if no custom configuration was provided""" + + class Element(Plugin): """Element() @@ -235,6 +256,7 @@ class Element(Plugin): # Internal instance properties # self._depth = None # Depth of Element in its current dependency graph + self._overlap_collector = None # type: Optional[OverlapCollector] # # Private instance properties @@ -334,6 +356,46 @@ class Element(Plugin): ############################################################# # Abstract Methods # ############################################################# + def configure_dependencies(self, dependencies: List[DependencyConfiguration]) -> None: + """Configure the Element with regards to it's build dependencies + + Elements can use this method to parse custom configuration which define their + relationship to their build dependencies. + + If this method is implemented, then it will be called with all direct build dependencies + specified in their :ref:`element declaration <format_dependencies>` in a list. + + If the dependency was declared with custom configuration, it will be provided along + with the dependency element, otherwise `None` will be passed with dependencies which + do not have any additional configuration. + + If the user has specified the same build dependency multiple times with differing + configurations, then those build dependencies will be provided multiple times + in the ``dependencies`` list. + + Args: + dependencies (list): A list of :class:`DependencyConfiguration <buildstream.element.DependencyConfiguration>` + objects + + Raises: + :class:`.ElementError`: When the element raises an error + + The format of the :class:`MappingNode <buildstream.node.MappingNode>` provided as + :attr:`DependencyConfiguration.config <buildstream.element.DependencyConfiguration.config> + belongs to the implementing element, and as such the format should be documented by the plugin, + and the :func:`MappingNode.validate_keys() <buildstream.node.MappingNode.validate_keys>` + method should be called by the implementing plugin in order to validate it. + + .. note:: + + It is unnecessary to implement this method if the plugin does not support + any custom :term:`dependency configuration <Dependency configuration>`. + """ + # This method is not called on plugins which do not implement it, so it would + # be a bug if this accidentally gets called. + # + assert False, "Code should not be reached" + def configure_sandbox(self, sandbox: "Sandbox") -> None: """Configures the the sandbox for execution @@ -596,9 +658,10 @@ class Element(Plugin): sandbox: "Sandbox", *, path: str = None, + action: str = OverlapAction.WARNING, include: Optional[List[str]] = None, exclude: Optional[List[str]] = None, - orphans: bool = True + orphans: bool = True, ) -> FileListResult: """Stage this element's output artifact in the sandbox @@ -610,6 +673,7 @@ class Element(Plugin): Args: sandbox: The build sandbox path: An optional sandbox relative path + action (OverlapAction): The action to take when overlapping with previous invocations include: An optional list of domains to include files from exclude: An optional list of domains to exclude files from orphans: Whether to include files not spoken for by split domains @@ -626,38 +690,24 @@ class Element(Plugin): unless the existing directory in `dest` is not empty in which case the path will be reported in the return value. - **Example:** - - .. code:: python + .. attention:: - # Stage the dependencies for a build of 'self' - for dep in self.dependencies(): - dep.stage_artifact(sandbox) + When staging artifacts with their dependencies, use + :func:`Element.stage_dependency_artifacts() <buildstream.element.Element.stage_dependency_artifacts>` + instead. """ + assert self._overlap_collector is not None, "Attempted to stage artifacts outside of Element.stage()" - if not self._cached(): - detail = ( - "No artifacts have been cached yet for that element\n" - + "Try building the element first with `bst build`\n" - ) - raise ElementError("No artifacts to stage", detail=detail, reason="uncached-checkout-attempt") - - # Time to use the artifact, check once more that it's there - self.__assert_cached() - - self.status("Staging {}/{}".format(self.name, self._get_brief_display_key())) - # Disable type checking since we can't easily tell mypy that - # `self.__artifact` can't be None at this stage. - files_vdir = self.__artifact.get_files() # type: ignore - - # Hard link it into the staging area # - vbasedir = sandbox.get_virtual_directory() - vstagedir = vbasedir if path is None else vbasedir.descend(*path.lstrip(os.sep).split(os.sep), create=True) - - split_filter = self.__split_filter_func(include, exclude, orphans) - - result = vstagedir.import_files(files_vdir, filter_callback=split_filter, report_written=True, can_link=True) + # The public API can only be called on the implementing plugin itself. + # + # ElementProxy calls to stage_artifact() are routed directly to _stage_artifact(), + # and the ElementProxy takes care of starting and ending the OverlapCollector session. + # + with self._overlap_collector.session(action, path): + result = self._stage_artifact( + sandbox, path=path, action=action, include=include, exclude=exclude, orphans=orphans + ) return result @@ -667,9 +717,10 @@ class Element(Plugin): selection: Sequence["Element"] = None, *, path: str = None, + action: str = OverlapAction.WARNING, include: Optional[List[str]] = None, exclude: Optional[List[str]] = None, - orphans: bool = True + orphans: bool = True, ) -> None: """Stage element dependencies in scope @@ -684,23 +735,22 @@ class Element(Plugin): is called is used as the `selection`. Args: - sandbox: The build sandbox + sandbox (Sandbox): The build sandbox selection (Sequence[Element]): A list of dependencies to select, or None - path An optional sandbox relative path - include: An optional list of domains to include files from - exclude: An optional list of domains to exclude files from - orphans: Whether to include files not spoken for by split domains + path (str): An optional sandbox relative path + action (OverlapAction): The action to take when overlapping with previous invocations + include (List[str]): An optional list of domains to include files from + exclude (List[str]): An optional list of domains to exclude files from + orphans (bool): Whether to include files not spoken for by split domains Raises: (:class:`.ElementError`): if forbidden overlaps occur. """ - overlaps = _OverlapCollector(self) - - for dep in self.dependencies(selection): - result = dep.stage_artifact(sandbox, path=path, include=include, exclude=exclude, orphans=orphans) - overlaps.collect_stage_result(dep, result) + assert self._overlap_collector is not None, "Attempted to stage artifacts outside of Element.stage()" - overlaps.overlap_warnings() + with self._overlap_collector.session(action, path): + for dep in self.dependencies(selection): + dep._stage_artifact(sandbox, path=path, include=include, exclude=exclude, orphans=orphans, owner=self) def integrate(self, sandbox: "Sandbox") -> None: """Integrate currently staged filesystem against this artifact. @@ -923,6 +973,73 @@ class Element(Plugin): return None + # _stage_artifact() + # + # Stage this element's output artifact in the sandbox + # + # This will stage the files from the artifact to the sandbox at specified location. + # The files are selected for staging according to the `include`, `exclude` and `orphans` + # parameters; if `include` is not specified then all files spoken for by any domain + # are included unless explicitly excluded with an `exclude` domain. + # + # Args: + # sandbox: The build sandbox + # path: An optional sandbox relative path + # action (OverlapAction): The action to take when overlapping with previous invocations + # include: An optional list of domains to include files from + # exclude: An optional list of domains to exclude files from + # orphans: Whether to include files not spoken for by split domains + # owner: The session element currently running Element.stage() + # + # Raises: + # (:class:`.ElementError`): If the element has not yet produced an artifact. + # + # Returns: + # The result describing what happened while staging + # + def _stage_artifact( + self, + sandbox: "Sandbox", + *, + path: str = None, + action: str = OverlapAction.WARNING, + include: Optional[List[str]] = None, + exclude: Optional[List[str]] = None, + orphans: bool = True, + owner: Optional["Element"] = None, + ) -> FileListResult: + + owner = owner or self + assert owner._overlap_collector is not None, "Attempted to stage artifacts outside of Element.stage()" + + if not self._cached(): + detail = ( + "No artifacts have been cached yet for that element\n" + + "Try building the element first with `bst build`\n" + ) + raise ElementError("No artifacts to stage", detail=detail, reason="uncached-checkout-attempt") + + # Time to use the artifact, check once more that it's there + self.__assert_cached() + + self.status("Staging {}/{}".format(self.name, self._get_brief_display_key())) + # Disable type checking since we can't easily tell mypy that + # `self.__artifact` can't be None at this stage. + files_vdir = self.__artifact.get_files() # type: ignore + + # Hard link it into the staging area + # + vbasedir = sandbox.get_virtual_directory() + vstagedir = vbasedir if path is None else vbasedir.descend(*path.lstrip(os.sep).split(os.sep), create=True) + + split_filter = self.__split_filter_func(include, exclude, orphans) + + result = vstagedir.import_files(files_vdir, filter_callback=split_filter, report_written=True, can_link=True) + + owner._overlap_collector.collect_stage_result(self, result) + + return result + # _stage_dependency_artifacts() # # Stage element dependencies in scope, this is used for core @@ -943,13 +1060,9 @@ class Element(Plugin): # occur. # def _stage_dependency_artifacts(self, sandbox, scope, *, path=None, include=None, exclude=None, orphans=True): - overlaps = _OverlapCollector(self) - - for dep in self._dependencies(scope): - result = dep.stage_artifact(sandbox, path=path, include=include, exclude=exclude, orphans=orphans) - overlaps.collect_stage_result(dep, result) - - overlaps.overlap_warnings() + with self._overlap_collector.session(OverlapAction.WARNING, path): + for dep in self._dependencies(scope): + dep._stage_artifact(sandbox, path=path, include=include, exclude=exclude, orphans=orphans, owner=self) # _new_from_load_element(): # @@ -978,6 +1091,15 @@ class Element(Plugin): element = load_element.project.create_element(load_element) cls.__instantiated_elements[load_element] = element + # If the element implements configure_dependencies(), we will collect + # the dependency configurations for it, otherwise we will consider + # it an error to specify `config` on dependencies. + # + if element.configure_dependencies.__func__ is not Element.configure_dependencies: + custom_configurations = [] + else: + custom_configurations = None + # Load the sources from the LoadElement element.__load_sources(load_element) @@ -985,11 +1107,38 @@ class Element(Plugin): for dep in load_element.dependencies: dependency = Element._new_from_load_element(dep.element, task) - if dep.dep_type != "runtime": + if dep.dep_type & DependencyType.BUILD: element.__build_dependencies.append(dependency) dependency.__reverse_build_deps.add(element) - if dep.dep_type != "build": + # Configuration data is only collected for build dependencies, + # if configuration data is specified on a runtime dependency + # then the assertion will be raised by the LoadElement. + # + if custom_configurations is not None: + + # Create a proxy for the dependency + dep_proxy = cast("Element", ElementProxy(element, dependency)) + + # Class supports dependency configuration + if dep.config_nodes: + custom_configurations.extend( + [DependencyConfiguration(dep_proxy, dep.path, config) for config in dep.config_nodes] + ) + else: + custom_configurations.append(DependencyConfiguration(dep_proxy, dep.path, None)) + + elif dep.config_nodes: + # Class does not support dependency configuration + provenance = dep.config_nodes[0].get_provenance() + raise LoadError( + "{}: Custom dependency configuration is not supported by element plugin '{}'".format( + provenance, element.get_kind() + ), + LoadErrorReason.INVALID_DEPENDENCY_CONFIG, + ) + + if dep.dep_type & DependencyType.RUNTIME: element.__runtime_dependencies.append(dependency) dependency.__reverse_runtime_deps.add(element) @@ -1002,6 +1151,9 @@ class Element(Plugin): no_of_build_deps = len(element.__build_dependencies) element.__build_deps_uncached = no_of_build_deps + if custom_configurations is not None: + element.configure_dependencies(custom_configurations) + element.__preflight() if task: @@ -1315,10 +1467,10 @@ class Element(Plugin): # Stage what we need if shell and scope == _Scope.BUILD: - self.stage(sandbox) + self.__stage(sandbox) else: # Stage deps in the sandbox root - with self.timed_activity("Staging dependencies", silent_nested=True): + with self.timed_activity("Staging dependencies", silent_nested=True), self.__collect_overlaps(): self._stage_dependency_artifacts(sandbox, scope) # Run any integration commands provided by the dependencies @@ -1623,7 +1775,7 @@ class Element(Plugin): # Step 1 - Configure self.__configure_sandbox(sandbox) # Step 2 - Stage - self.stage(sandbox) + self.__stage(sandbox) try: if self.__batch_prepare_assemble: cm = sandbox.batch( @@ -2438,6 +2590,16 @@ class Element(Plugin): self.configure_sandbox(sandbox) + # __stage(): + # + # Internal method for calling public abstract stage() method. + # + def __stage(self, sandbox): + + # Enable the overlap collector during the staging process + with self.__collect_overlaps(): + self.stage(sandbox) + # __prepare(): # # Internal method for calling public abstract prepare() method. @@ -2544,6 +2706,20 @@ class Element(Plugin): def __use_remote_execution(self): return bool(self.__remote_execution_specs) + # __collect_overlaps(): + # + # A context manager for collecting overlap warnings and errors. + # + # Any places where code might call Element.stage_artifact() + # or Element.stage_dependency_artifacts() should be run in + # this context manager. + # + @contextmanager + def __collect_overlaps(self): + self._overlap_collector = OverlapCollector(self) + yield + self._overlap_collector = None + # __sandbox(): # # A context manager to prepare a Sandbox object at the specified directory, @@ -3117,112 +3293,6 @@ class Element(Plugin): self.__artifact._cache_key = self.__cache_key -# _OverlapCollector() -# -# Collects results of Element.stage_artifact() and saves -# them in order to raise a proper overlap error at the end -# of staging. -# -# Args: -# element (Element): The element for which we are staging artifacts -# -class _OverlapCollector: - def __init__(self, element): - self.element = element - - # Dictionary of files which were ignored (See FileListResult()), keyed by element unique ID - self.ignored = {} # type: Dict[int, List[str]] - - # Dictionary of files which were staged, keyed by element unique ID - self.files_written = {} # type: Dict[int, List[str]] - - # Dictionary of element IDs which overlapped, keyed by the file they overlap on - self.overlaps = {} # type: Dict[str, List[int]] - - # collect_stage_result() - # - # Collect and accumulate results of Element.stage_artifact() - # - # Args: - # element (Element): The name of the element staged - # result (FileListResult): The result of Element.stage_artifact() - # - def collect_stage_result(self, element: Element, result: FileListResult): - if result.overwritten: - for overwritten_file in result.overwritten: - # Completely new overwrite - if overwritten_file not in self.overlaps: - # Search for the element we've overwritten in self.written_files - for element_id, staged_files in self.files_written.items(): - if overwritten_file in staged_files: - self.overlaps[overwritten_file] = [element_id, element._unique_id] - break - # Record the new overwrite in the list - else: - self.overlaps[overwritten_file].append(element._unique_id) - - self.files_written[element._unique_id] = result.files_written - if result.ignored: - self.ignored[element._unique_id] = result.ignored - - # overlap_warnings() - # - # Issue any warnings as a batch as a result of staging artifacts, - # based on the results collected with collect_stage_result(). - # - def overlap_warnings(self): - if self.overlaps: - overlap_warning = False - warning_detail = "Staged files overwrite existing files in staging area:\n" - for filename, element_ids in self.overlaps.items(): - overlap_warning_elements = [] - # The bottom item overlaps nothing - overlapping_element_ids = element_ids[1:] - for element_id in overlapping_element_ids: - element = Plugin._lookup(element_id) - if not element._file_is_whitelisted(filename): - overlap_warning_elements.append(element) - overlap_warning = True - - warning_detail += self._overlap_error_detail(filename, overlap_warning_elements, element_ids) - - if overlap_warning: - self.element.warn( - "Non-whitelisted overlaps detected", detail=warning_detail, warning_token=CoreWarnings.OVERLAPS - ) - - if self.ignored: - detail = "Not staging files which would replace non-empty directories:\n" - for element_id, ignored_filenames in self.ignored.items(): - element = Plugin._lookup(element_id) - detail += "\nFrom {}:\n".format(element._get_full_name()) - detail += " " + " ".join(["/" + filename + "\n" for filename in ignored_filenames]) - self.element.warn("Ignored files", detail=detail) - - # _overlap_error_detail() - # - # Get a string to describe overlaps on a filename - # - # Args: - # filename (str): The filename being overlapped - # overlap_elements (List[Element]): A list of Elements overlapping - # element_ids (List[int]): The ordered ID list of elements which staged this file - # - def _overlap_error_detail(self, filename, overlap_elements, element_ids): - if overlap_elements: - overlap_element_names = [element._get_full_name() for element in overlap_elements] - overlap_order_elements = [Plugin._lookup(element_id) for element_id in element_ids] - overlap_order_names = [element._get_full_name() for element in overlap_order_elements] - return "/{}: {} {} not permitted to overlap other elements, order {} \n".format( - filename, - " and ".join(overlap_element_names), - "is" if len(overlap_element_names) == 1 else "are", - " above ".join(reversed(overlap_order_names)), - ) - else: - return "" - - # _get_normal_name(): # # Get the element name without path separators or diff --git a/src/buildstream/exceptions.py b/src/buildstream/exceptions.py index caf08ae57..4b9118978 100644 --- a/src/buildstream/exceptions.py +++ b/src/buildstream/exceptions.py @@ -137,8 +137,9 @@ class LoadErrorReason(Enum): PROTECTED_VARIABLE_REDEFINED = 23 """An attempt was made to set the value of a protected variable""" - DUPLICATE_DEPENDENCY = 24 - """A duplicate dependency was detected""" + INVALID_DEPENDENCY_CONFIG = 24 + """An attempt was made to specify dependency configuration on an element + which does not support custom dependency configuration""" LINK_FORBIDDEN_DEPENDENCIES = 25 """A link element declared dependencies""" diff --git a/src/buildstream/plugins/elements/compose.py b/src/buildstream/plugins/elements/compose.py index 808419675..0d49884f6 100644 --- a/src/buildstream/plugins/elements/compose.py +++ b/src/buildstream/plugins/elements/compose.py @@ -82,17 +82,15 @@ class ComposeElement(Element): pass def stage(self, sandbox): - pass - - def assemble(self, sandbox): - - require_split = self.include or self.exclude or not self.include_orphans # Stage deps in the sandbox root with self.timed_activity("Staging dependencies", silent_nested=True): self.stage_dependency_artifacts(sandbox) + def assemble(self, sandbox): manifest = set() + + require_split = self.include or self.exclude or not self.include_orphans if require_split: with self.timed_activity("Computing split", silent_nested=True): for dep in self.dependencies(): diff --git a/src/buildstream/plugins/elements/filter.py b/src/buildstream/plugins/elements/filter.py index c817ca46b..783079c06 100644 --- a/src/buildstream/plugins/elements/filter.py +++ b/src/buildstream/plugins/elements/filter.py @@ -219,9 +219,6 @@ class FilterElement(Element): pass def stage(self, sandbox): - pass - - def assemble(self, sandbox): with self.timed_activity("Staging artifact", silent_nested=True): for dep in self.dependencies(recurse=False): # Check that all the included/excluded domains exist @@ -250,6 +247,8 @@ class FilterElement(Element): raise ElementError("Unknown domains declared.", detail=detail) dep.stage_artifact(sandbox, include=self.include, exclude=self.exclude, orphans=self.include_orphans) + + def assemble(self, sandbox): return "" def _get_source_element(self): diff --git a/src/buildstream/plugins/elements/script.py b/src/buildstream/plugins/elements/script.py index 502212e10..6bcb02c81 100644 --- a/src/buildstream/plugins/elements/script.py +++ b/src/buildstream/plugins/elements/script.py @@ -45,19 +45,25 @@ class ScriptElement(buildstream.ScriptElement): BST_MIN_VERSION = "2.0" def configure(self, node): - for n in node.get_sequence("layout", []): - dst = n.get_str("destination") - elm = n.get_str("element", None) - self.layout_add(elm, dst) - - node.validate_keys(["commands", "root-read-only", "layout"]) + node.validate_keys(["commands", "root-read-only"]) self.add_commands("commands", node.get_str_list("commands")) - self.set_work_dir() self.set_install_root() self.set_root_read_only(node.get_bool("root-read-only", default=False)) + def configure_dependencies(self, dependencies): + for dep in dependencies: + + # Determine the location to stage each element, default is "/" + location = "/" + if dep.config: + dep.config.validate_keys(["location"]) + location = dep.config.get_str("location", location) + + # Add each element to the layout + self.layout_add(dep.element, dep.path, location) + # Plugin entry point def setup(): diff --git a/src/buildstream/plugins/elements/script.yaml b/src/buildstream/plugins/elements/script.yaml index b388378da..390f60355 100644 --- a/src/buildstream/plugins/elements/script.yaml +++ b/src/buildstream/plugins/elements/script.yaml @@ -1,3 +1,21 @@ +# The script element allows staging elements into specific locations +# via it's "location" dependency configuration +# +# For example, if you want to stage "foo-tools.bst" into the "/" of +# the sandbox at buildtime, and the "foo-system.bst" element into +# the %{build-root}, you can do so as follows: +# +# build-depends: +# - foo-tools.bst +# - filename: foo-system.bst +# config: +# location: "%{build-root}" +# +# Note: the default of the "location" parameter is "/", so it is not +# necessary to specify the location if you want to stage the +# element in "/" +# + # Common script element variables variables: # Defines the directory commands will be run from. @@ -10,16 +28,6 @@ config: # It is recommended to set root as read-only wherever possible. root-read-only: False - # Defines where to stage elements which are direct or indirect dependencies. - # By default, all direct dependencies are staged to '/'. - # This is also commonly used to take one element as an environment - # containing the tools used to operate on the other element. - # layout: - # - element: foo-tools.bst - # destination: / - # - element: foo-system.bst - # destination: %{build-root} - # List of commands to run in the sandbox. commands: [] diff --git a/src/buildstream/scriptelement.py b/src/buildstream/scriptelement.py index 5ecae998c..0a3a4d3dc 100644 --- a/src/buildstream/scriptelement.py +++ b/src/buildstream/scriptelement.py @@ -36,11 +36,11 @@ import os from collections import OrderedDict from typing import List, Optional, TYPE_CHECKING -from .element import Element, ElementError +from .element import Element from .sandbox import SandboxFlags if TYPE_CHECKING: - from typing import Dict + from typing import Dict, Tuple class ScriptElement(Element): @@ -48,7 +48,7 @@ class ScriptElement(Element): __cwd = "/" __root_read_only = False __commands = None # type: OrderedDict[str, List[str]] - __layout = [] # type: List[Dict[str, Optional[str]]] + __layout = {} # type: Dict[str, List[Tuple[Element, str]]] # The compose element's output is its dependencies, so # we must rebuild if the dependencies change even when @@ -75,8 +75,8 @@ class ScriptElement(Element): called from. Args: - work_dir: The working directory. If called without this argument - set, it'll default to the value of the variable ``cwd``. + work_dir: The working directory. If called without this argument + set, it'll default to the value of the variable ``cwd``. """ if work_dir is None: self.__cwd = self.get_variable("cwd") or "/" @@ -90,8 +90,8 @@ class ScriptElement(Element): once the commands have been run. Args: - install_root: The install root. If called without this argument - set, it'll default to the value of the variable ``install-root``. + install_root: The install root. If called without this argument + set, it'll default to the value of the variable ``install-root``. """ if install_root is None: self.__install_root = self.get_variable("install-root") or "/" @@ -108,47 +108,54 @@ class ScriptElement(Element): If this variable is not set, the default permission is read-write. Args: - root_read_only: Whether to mark the root filesystem as read-only. + root_read_only: Whether to mark the root filesystem as read-only. """ self.__root_read_only = root_read_only - def layout_add(self, element: Optional[str], destination: str) -> None: - """Adds an element-destination pair to the layout. + def layout_add(self, element: Element, dependency_path: str, location: str) -> None: + """Adds an element to the layout. Layout is a way of defining how dependencies should be added to the staging area for running commands. Args: - element: The name of the element to stage, or None. This may be any - element found in the dependencies, whether it is a direct - or indirect dependency. - destination: The path inside the staging area for where to - stage this element. If it is not "/", then integration - commands will not be run. + element (Element): The element to stage. + dependency_path (str): The element relative path to the dependency, usually obtained via + :attr:`the dependency configuration <buildstream.element.DependencyConfiguration.path>` + location (str): The path inside the staging area for where to + stage this element. If it is not "/", then integration + commands will not be run. If this function is never called, then the default behavior is to just stage the build dependencies of the element in question at the - sandbox root. Otherwise, the runtime dependencies of each specified - element will be staged in their specified destination directories. + sandbox root. Otherwise, the specified elements including their + runtime dependencies will be staged in their respective locations. .. note:: - The order of directories in the layout is significant as they - will be mounted into the sandbox. It is an error to specify a parent - directory which will shadow a directory already present in the layout. + The order of directories in the layout is not significant. - .. note:: + The paths in the layout will be sorted so that elements are staged in parent + directories before subdirectories. - In the case that no element is specified, a read-write directory will - be made available at the specified location. + The elements for each respective staging directory in the layout will be staged + in the predetermined deterministic staging order. """ # - # Even if this is an empty list by default, make sure that its + # Even if this is an empty dict by default, make sure that it is # instance data instead of appending stuff directly onto class data. # if not self.__layout: - self.__layout = [] - self.__layout.append({"element": element, "destination": destination}) + self.__layout = {} + + # Get or create the element list + try: + element_list = self.__layout[location] + except KeyError: + element_list = [] + self.__layout[location] = element_list + + element_list.append((element, dependency_path)) def add_commands(self, group_name: str, command_list: List[str]) -> None: """Adds a list of commands under the group-name. @@ -164,8 +171,8 @@ class ScriptElement(Element): :func:`~buildstream.element.Element.node_subst_list`) Args: - group_name (str): The name of the group of commands. - command_list (list): The list of commands to be run. + group_name (str): The name of the group of commands. + command_list (list): The list of commands to be run. """ if not self.__commands: self.__commands = OrderedDict() @@ -174,17 +181,20 @@ class ScriptElement(Element): ############################################################# # Abstract Method Implementations # ############################################################# - def preflight(self): - # The layout, if set, must make sense. - self.__validate_layout() + pass def get_unique_key(self): + sorted_locations = sorted(self.__layout) + layout_key = { + location: [dependency_path for _, dependency_path in self.__layout[location]] + for location in sorted_locations + } return { "commands": self.__commands, "cwd": self.__cwd, "install-root": self.__install_root, - "layout": self.__layout, + "layout": layout_key, "root-read-only": self.__root_read_only, } @@ -196,67 +206,46 @@ class ScriptElement(Element): # Setup environment sandbox.set_environment(self.get_environment()) - # Tell the sandbox to mount the install root - directories = {self.__install_root: False} - - # set the output directory - sandbox.set_output_directory(self.__install_root) + # Mark the install root + sandbox.mark_directory(self.__install_root, artifact=False) # Mark the artifact directories in the layout - for item in self.__layout: - destination = item["destination"] - was_artifact = directories.get(destination, False) - directories[destination] = item["element"] or was_artifact - - for directory, artifact in directories.items(): - # Root does not need to be marked as it is always mounted - # with artifact (unless explicitly marked non-artifact) - if directory != "/": - sandbox.mark_directory(directory, artifact=artifact) + for location in self.__layout: + sandbox.mark_directory(location, artifact=True) def stage(self, sandbox): - # Stage the elements, and run integration commands where appropriate. + # If self.layout_add() was never called, do the default staging of + # everything in "/" and run the integration commands if not self.__layout: - # if no layout set, stage all dependencies into the sandbox root with self.timed_activity("Staging dependencies", silent_nested=True): self.stage_dependency_artifacts(sandbox) - # Run any integration commands provided by the dependencies - # once they are all staged and ready with sandbox.batch(SandboxFlags.NONE, label="Integrating sandbox"): for dep in self.dependencies(): dep.integrate(sandbox) - else: - # If layout, follow its rules. - for item in self.__layout: - - # Skip layout members which dont stage an element - if not item["element"]: - continue - - element = self.search(item["element"]) - with self.timed_activity( - "Staging {} at {}".format(element.name, item["destination"]), silent_nested=True - ): - element.stage_dependency_artifacts(sandbox, [element], path=item["destination"]) - - with sandbox.batch(SandboxFlags.NONE): - for item in self.__layout: - - # Skip layout members which dont stage an element - if not item["element"]: - continue - - element = self.search(item["element"]) - - # Integration commands can only be run for elements staged to / - if item["destination"] == "/": - with self.timed_activity("Integrating {}".format(element.name), silent_nested=True): - for dep in element.dependencies(): - dep.integrate(sandbox) + else: + # First stage it all + # + sorted_locations = sorted(self.__layout) + + for location in sorted_locations: + element_list = [element for element, _ in self.__layout[location]] + self.stage_dependency_artifacts(sandbox, element_list, path=location) + + # Now integrate any elements staged in the root + # + root_list = self.__layout.get("/", None) + if root_list: + element_list = [element for element, _ in root_list] + with sandbox.batch(SandboxFlags.NONE), self.timed_activity("Integrating sandbox", silent_nested=True): + for dep in self.dependencies(element_list): + dep.integrate(sandbox) + + # Ensure the install root exists + # install_root_path_components = self.__install_root.lstrip(os.sep).split(os.sep) sandbox.get_virtual_directory().descend(*install_root_path_components, create=True) @@ -277,26 +266,6 @@ class ScriptElement(Element): # Return where the result can be collected from return self.__install_root - ############################################################# - # Private Local Methods # - ############################################################# - - def __validate_layout(self): - if self.__layout: - # Cannot proceeed if layout is used, but none are for "/" - root_defined = any([(entry["destination"] == "/") for entry in self.__layout]) - if not root_defined: - raise ElementError("{}: Using layout, but none are staged as '/'".format(self)) - - # Cannot proceed if layout specifies an element that isn't part - # of the dependencies. - for item in self.__layout: - if item["element"]: - if not self.search(item["element"]): - raise ElementError( - "{}: '{}' in layout not found in dependencies".format(self, item["element"]) - ) - def setup(): return ScriptElement diff --git a/src/buildstream/types.py b/src/buildstream/types.py index 3b1f7a4db..48dd5de6d 100644 --- a/src/buildstream/types.py +++ b/src/buildstream/types.py @@ -100,6 +100,12 @@ class CoreWarnings: which is not whitelisted. See :ref:`Overlap Whitelist <public_overlap_whitelist>` """ + UNSTAGED_FILES = "unstaged-files" + """ + This warning will be produced when a file cannot be staged. This can happen when + a file overlaps with a directory in the sandbox that is not empty. + """ + REF_NOT_IN_TRACK = "ref-not-in-track" """ This warning will be produced when a source is configured with a reference @@ -114,11 +120,51 @@ class CoreWarnings: BAD_CHARACTERS_IN_NAME = "bad-characters-in-name" """ - This warning will be produces when filename for a target contains invalid + This warning will be produced when a filename for a target contains invalid characters in its name. """ +class OverlapAction(FastEnum): + """OverlapAction() + + Defines what action to take when files staged into the sandbox overlap. + + .. note:: + + This only dictates what happens when functions such as + :func:`Element.stage_artifact() <buildstream.element.Element.stage_artifact>` and + :func:`Element.stage_dependency_artifacts() <buildstream.element.Element.stage_dependency_artifacts>` + are called multiple times in an Element's :func:`Element.stage() <buildstream.element.Element.stage>` + implementation, and the files staged from one function call result in overlapping files staged + from previous invocations. + + If multiple staged elements overlap eachother within a single call to + :func:`Element.stage_dependency_artifacts() <buildstream.element.Element.stage_dependency_artifacts>`, + then the :ref:`overlap whitelist <public_overlap_whitelist>` will be ovserved, and warnings will + be issued for overlapping files, which will be fatal warnings if + :attr:`CoreWarnings.OVERLAPS <buildstream.types.CoreWarnings.OVERLAPS>` is specified + as a :ref:`fatal warning <configurable_warnings>`. + """ + + ERROR = "error" + """ + It is an error to overlap previously staged files + """ + + WARNING = "warning" + """ + A warning will be issued for previously staged files, which will fatal if + :attr:`CoreWarnings.OVERLAPS <buildstream.types.CoreWarnings.OVERLAPS>` is specified + as a :ref:`fatal warning <configurable_warnings>` in the project. + """ + + IGNORE = "ignore" + """ + Overlapping files are acceptable, and do not cause any warning or error. + """ + + # _Scope(): # # Defines the scope of dependencies to include for a given element diff --git a/tests/cachekey/project/elements/build1.expected b/tests/cachekey/project/elements/build1.expected index 71113a494..369036a16 100644 --- a/tests/cachekey/project/elements/build1.expected +++ b/tests/cachekey/project/elements/build1.expected @@ -1 +1 @@ -99eb5ee7e7b93f1f9e4abfd2b392ed3b9746aeebc74f9239e5e0ca3ca444ab47
\ No newline at end of file +e47d9b32ba894f1d454b66d8d30d3d20226b54c327e8b6a3893c9c55ff02378c
\ No newline at end of file diff --git a/tests/cachekey/project/elements/build2.expected b/tests/cachekey/project/elements/build2.expected index 10d60050d..6e9dfd075 100644 --- a/tests/cachekey/project/elements/build2.expected +++ b/tests/cachekey/project/elements/build2.expected @@ -1 +1 @@ -9458b71a2054df6ca3d1ce9df2090649b1416a368e41ae40bfa0cc52f5efb05e
\ No newline at end of file +ffe535e1ac311448584d9f0b17f51bb85c8cc9bd3c081a411f9c3ea464f5fcd7
\ No newline at end of file diff --git a/tests/cachekey/project/elements/build3.bst b/tests/cachekey/project/elements/build3.bst new file mode 100644 index 000000000..e1422862f --- /dev/null +++ b/tests/cachekey/project/elements/build3.bst @@ -0,0 +1,13 @@ +# This tests that staging into an alternative +# prefix affects the cache key +# +kind: manual +sources: +- kind: local + path: files/local + +depends: +- elements/build1.bst +- filename: elements/build1.bst + config: + location: /opt diff --git a/tests/cachekey/project/elements/build3.expected b/tests/cachekey/project/elements/build3.expected new file mode 100644 index 000000000..7b3aabe9f --- /dev/null +++ b/tests/cachekey/project/elements/build3.expected @@ -0,0 +1 @@ +c8bae677ec6af58543e4b779a295c9f6bee0c1eface38d9be62a66c247a68ba5
\ No newline at end of file diff --git a/tests/cachekey/project/elements/script1.expected b/tests/cachekey/project/elements/script1.expected index 71be17a37..5d9799f1a 100644 --- a/tests/cachekey/project/elements/script1.expected +++ b/tests/cachekey/project/elements/script1.expected @@ -1 +1 @@ -38078af3714d6d5128669dc7b4d7e942a46ea4137c388d44a1f455b4cc1918d0
\ No newline at end of file +1220b2849910f60f864c2c325cda301809dcb9b730281862448508fbeae5fb91
\ No newline at end of file diff --git a/tests/cachekey/project/target.bst b/tests/cachekey/project/target.bst index cabf3f7bf..30f8660a6 100644 --- a/tests/cachekey/project/target.bst +++ b/tests/cachekey/project/target.bst @@ -22,6 +22,7 @@ depends: - sources/zip2.bst - elements/build1.bst - elements/build2.bst +- elements/build3.bst - elements/compose1.bst - elements/compose2.bst - elements/compose3.bst diff --git a/tests/cachekey/project/target.expected b/tests/cachekey/project/target.expected index 38d4abeef..9ea9991e9 100644 --- a/tests/cachekey/project/target.expected +++ b/tests/cachekey/project/target.expected @@ -1 +1 @@ -5bf8ea2d6c0900c226c8fd3dae980f0b92e40f3fc1887b64d31bf04b6210d763
\ No newline at end of file +cb3b1bbf3d8b7be1a0ee7305c1056e95225f84b2c2b5189cfee69a66dcbd0786
\ No newline at end of file diff --git a/tests/elements/filter/basic/element_plugins/dynamic.py b/tests/elements/filter/basic/element_plugins/dynamic.py index 16a600823..401c6b128 100644 --- a/tests/elements/filter/basic/element_plugins/dynamic.py +++ b/tests/elements/filter/basic/element_plugins/dynamic.py @@ -20,13 +20,10 @@ class DynamicElement(Element): pass def stage(self, sandbox): - pass - - def assemble(self, sandbox): with self.timed_activity("Staging artifact", silent_nested=True): - for dep in self.dependencies(): - dep.stage_artifact(sandbox) + self.stage_dependency_artifacts(sandbox) + def assemble(self, sandbox): bstdata = self.get_public_data("bst") bstdata["split-rules"] = self.split_rules self.set_public_data("bst", bstdata) diff --git a/tests/format/dependencies.py b/tests/format/dependencies.py index b1a684081..12eb19d3a 100644 --- a/tests/format/dependencies.py +++ b/tests/format/dependencies.py @@ -236,24 +236,48 @@ def test_no_recurse(cli, datafiles): @pytest.mark.datafiles(DATA_DIR) @pytest.mark.parametrize( - ("element", "asserts"), - [ - ("build-runtime", False), - ("build-build", True), - ("build-all", True), - ("runtime-runtime", True), - ("runtime-all", True), - ("all-all", True), - ], + "target", ["merge-separate-lists.bst", "merge-single-list.bst",], ids=["separate-lists", "single-list"], ) -def test_duplicate_deps(cli, datafiles, element, asserts): +def test_merge(cli, datafiles, target): + project = os.path.join(str(datafiles), "dependencies2") + + # Test both build and run scopes, showing that the two dependencies + # have been merged and the run-build.bst is both a runtime and build + # time dependency, and is not loaded twice into the build graph. + # + element_list = cli.get_pipeline(project, [target], scope="build") + assert element_list == ["run-build.bst"] + + element_list = cli.get_pipeline(project, [target], scope="run") + assert element_list == ["run-build.bst", target] + + +@pytest.mark.datafiles(DATA_DIR) +def test_config_unsupported(cli, datafiles): project = os.path.join(str(datafiles), "dependencies3") - result = cli.run(project=project, args=["show", "{}.bst".format(element)]) + result = cli.run(project=project, args=["show", "unsupported.bst"]) + result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DEPENDENCY_CONFIG) + - if asserts: - result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.DUPLICATE_DEPENDENCY) - assert "[line 10 column 2]" in result.stderr - assert "[line 8 column 2]" in result.stderr - else: - result.assert_success() +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.parametrize( + "target,number", [("supported1.bst", 1), ("supported2.bst", 2),], ids=["one", "two"], +) +def test_config_supported(cli, datafiles, target, number): + project = os.path.join(str(datafiles), "dependencies3") + + result = cli.run(project=project, args=["show", target]) + result.assert_success() + + assert "TEST PLUGIN FOUND {} ENABLED DEPENDENCIES".format(number) in result.stderr + + +@pytest.mark.datafiles(DATA_DIR) +def test_config_runtime_error(cli, datafiles): + project = os.path.join(str(datafiles), "dependencies3") + + # Test that it is considered an error to specify `config` on runtime-only dependencies + # + result = cli.run(project=project, args=["show", "runtime-error.bst"]) + result.assert_main_error(ErrorDomain.LOAD, LoadErrorReason.INVALID_DATA) diff --git a/tests/format/dependencies2/merge-separate-lists.bst b/tests/format/dependencies2/merge-separate-lists.bst new file mode 100644 index 000000000..5be04d7a1 --- /dev/null +++ b/tests/format/dependencies2/merge-separate-lists.bst @@ -0,0 +1,8 @@ +kind: autotools +description: Depend on the same element twice, as a build and as a runtime dependency + +build-depends: +- run-build.bst + +runtime-depends: +- run-build.bst diff --git a/tests/format/dependencies2/merge-single-list.bst b/tests/format/dependencies2/merge-single-list.bst new file mode 100644 index 000000000..4fd46e695 --- /dev/null +++ b/tests/format/dependencies2/merge-single-list.bst @@ -0,0 +1,8 @@ +kind: autotools +description: Depend on the same element twice, as a build and as a runtime dependency + +depends: +- filename: run-build.bst + type: runtime +- filename: run-build.bst + type: build diff --git a/tests/format/dependencies3/all-all.bst b/tests/format/dependencies3/all-all.bst deleted file mode 100644 index 98122472d..000000000 --- a/tests/format/dependencies3/all-all.bst +++ /dev/null @@ -1,11 +0,0 @@ -kind: import - -sources: -- kind: local - path: all-all.bst - -depends: -- filename: dep.bst - type: all -- filename: dep.bst - type: all diff --git a/tests/format/dependencies3/build-all.bst b/tests/format/dependencies3/build-all.bst deleted file mode 100644 index 4c66524e7..000000000 --- a/tests/format/dependencies3/build-all.bst +++ /dev/null @@ -1,11 +0,0 @@ -kind: import - -sources: -- kind: local - path: all-all.bst - -depends: -- filename: dep.bst - type: build -- filename: dep.bst - type: all diff --git a/tests/format/dependencies3/build-build.bst b/tests/format/dependencies3/build-build.bst deleted file mode 100644 index 2a813b3ab..000000000 --- a/tests/format/dependencies3/build-build.bst +++ /dev/null @@ -1,11 +0,0 @@ -kind: import - -sources: -- kind: local - path: all-all.bst - -depends: -- filename: dep.bst - type: build -- filename: dep.bst - type: build diff --git a/tests/format/dependencies3/build-runtime.bst b/tests/format/dependencies3/build-runtime.bst deleted file mode 100644 index f740736d8..000000000 --- a/tests/format/dependencies3/build-runtime.bst +++ /dev/null @@ -1,11 +0,0 @@ -kind: import - -sources: -- kind: local - path: all-all.bst - -depends: -- filename: dep.bst - type: build -- filename: dep.bst - type: runtime diff --git a/tests/format/dependencies3/elements/dep.bst b/tests/format/dependencies3/elements/dep.bst new file mode 100644 index 000000000..9e5cf96b6 --- /dev/null +++ b/tests/format/dependencies3/elements/dep.bst @@ -0,0 +1,2 @@ +kind: manual +description: Some kinda element diff --git a/tests/format/dependencies3/elements/runtime-error.bst b/tests/format/dependencies3/elements/runtime-error.bst new file mode 100644 index 000000000..948997aa9 --- /dev/null +++ b/tests/format/dependencies3/elements/runtime-error.bst @@ -0,0 +1,6 @@ +kind: configsupported + +runtime-depends: +- filename: dep.bst + config: + enabled: true diff --git a/tests/format/dependencies3/elements/supported1.bst b/tests/format/dependencies3/elements/supported1.bst new file mode 100644 index 000000000..528475ab0 --- /dev/null +++ b/tests/format/dependencies3/elements/supported1.bst @@ -0,0 +1,6 @@ +kind: configsupported + +depends: +- filename: dep.bst + config: + enabled: true diff --git a/tests/format/dependencies3/elements/supported2.bst b/tests/format/dependencies3/elements/supported2.bst new file mode 100644 index 000000000..041ef08c1 --- /dev/null +++ b/tests/format/dependencies3/elements/supported2.bst @@ -0,0 +1,9 @@ +kind: configsupported + +depends: +- filename: dep.bst + config: + enabled: true +- filename: dep.bst + config: + enabled: true diff --git a/tests/format/dependencies3/elements/unsupported.bst b/tests/format/dependencies3/elements/unsupported.bst new file mode 100644 index 000000000..eca090018 --- /dev/null +++ b/tests/format/dependencies3/elements/unsupported.bst @@ -0,0 +1,6 @@ +kind: configunsupported + +depends: +- filename: dep.bst + config: + enabled: true diff --git a/tests/format/dependencies3/plugins/configsupported.py b/tests/format/dependencies3/plugins/configsupported.py new file mode 100644 index 000000000..06f85f7a3 --- /dev/null +++ b/tests/format/dependencies3/plugins/configsupported.py @@ -0,0 +1,29 @@ +from buildstream import Element + + +class ConfigSupported(Element): + BST_MIN_VERSION = "2.0" + + def configure(self, node): + pass + + def configure_dependencies(self, dependencies): + self.configs = [] + + for dep in dependencies: + if dep.config: + dep.config.validate_keys(["enabled"]) + self.configs.append(dep) + + self.info("TEST PLUGIN FOUND {} ENABLED DEPENDENCIES".format(len(self.configs))) + + def preflight(self): + pass + + def get_unique_key(self): + return {} + + +# Plugin entry point +def setup(): + return ConfigSupported diff --git a/tests/format/dependencies3/plugins/configunsupported.py b/tests/format/dependencies3/plugins/configunsupported.py new file mode 100644 index 000000000..9dcaca1ee --- /dev/null +++ b/tests/format/dependencies3/plugins/configunsupported.py @@ -0,0 +1,19 @@ +from buildstream import Element + + +class ConfigUnsupported(Element): + BST_MIN_VERSION = "2.0" + + def configure(self, node): + pass + + def preflight(self): + pass + + def get_unique_key(self): + return {} + + +# Plugin entry point +def setup(): + return ConfigUnsupported diff --git a/tests/format/dependencies3/project.conf b/tests/format/dependencies3/project.conf index 8b361b03d..c1b99eb12 100644 --- a/tests/format/dependencies3/project.conf +++ b/tests/format/dependencies3/project.conf @@ -1,2 +1,11 @@ -name: dup-dup-checker +# Basic project +name: test min-version: 2.0 +element-path: elements + +plugins: +- origin: local + path: plugins + elements: + - configsupported + - configunsupported diff --git a/tests/format/dependencies3/runtime-all.bst b/tests/format/dependencies3/runtime-all.bst deleted file mode 100644 index c08594623..000000000 --- a/tests/format/dependencies3/runtime-all.bst +++ /dev/null @@ -1,11 +0,0 @@ -kind: import - -sources: -- kind: local - path: all-all.bst - -depends: -- filename: dep.bst - type: runtime -- filename: dep.bst - type: all diff --git a/tests/format/dependencies3/runtime-runtime.bst b/tests/format/dependencies3/runtime-runtime.bst deleted file mode 100644 index d01181f9b..000000000 --- a/tests/format/dependencies3/runtime-runtime.bst +++ /dev/null @@ -1,11 +0,0 @@ -kind: import - -sources: -- kind: local - path: all-all.bst - -depends: -- filename: dep.bst - type: runtime -- filename: dep.bst - type: runtime diff --git a/tests/frontend/overlaps.py b/tests/frontend/overlaps.py index 28bf8a7f5..7a9925e48 100644 --- a/tests/frontend/overlaps.py +++ b/tests/frontend/overlaps.py @@ -6,75 +6,87 @@ import pytest from buildstream.testing.runcli import cli # pylint: disable=unused-import from buildstream.exceptions import ErrorDomain, LoadErrorReason from buildstream import _yaml -from buildstream.plugin import CoreWarnings +from buildstream import CoreWarnings, OverlapAction from tests.testutils import generate_junction # Project directory DATA_DIR = os.path.join(os.path.dirname(os.path.realpath(__file__)), "overlaps") -def gen_project(project_dir, fail_on_overlap, use_fatal_warnings=True, project_name="test"): +def gen_project(project_dir, fatal_warnings, *, project_name="test", use_plugin=False): template = {"name": project_name, "min-version": "2.0"} - if use_fatal_warnings: - template["fatal-warnings"] = [CoreWarnings.OVERLAPS] if fail_on_overlap else [] - else: - template["fail-on-overlap"] = fail_on_overlap + template["fatal-warnings"] = [CoreWarnings.OVERLAPS, CoreWarnings.UNSTAGED_FILES] if fatal_warnings else [] + if use_plugin: + template["plugins"] = [{"origin": "local", "path": "plugins", "elements": ["overlap"]}] projectfile = os.path.join(project_dir, "project.conf") _yaml.roundtrip_dump(template, projectfile) @pytest.mark.datafiles(DATA_DIR) -@pytest.mark.parametrize("use_fatal_warnings", [True, False]) -def test_overlaps(cli, datafiles, use_fatal_warnings): +@pytest.mark.parametrize("error", [False, True], ids=["warning", "error"]) +def test_unstaged_files(cli, datafiles, error): project_dir = str(datafiles) - gen_project(project_dir, False, use_fatal_warnings) - result = cli.run(project=project_dir, silent=True, args=["build", "collect.bst"]) - result.assert_success() + gen_project(project_dir, error) + result = cli.run(project=project_dir, silent=True, args=["build", "unstaged.bst"]) + if error: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.UNSTAGED_FILES) + else: + result.assert_success() + assert "WARNING [unstaged-files]" in result.stderr @pytest.mark.datafiles(DATA_DIR) -@pytest.mark.parametrize("use_fatal_warnings", [True, False]) -def test_overlaps_error(cli, datafiles, use_fatal_warnings): +@pytest.mark.parametrize("error", [False, True], ids=["warning", "error"]) +def test_overlaps(cli, datafiles, error): project_dir = str(datafiles) - gen_project(project_dir, True, use_fatal_warnings) + gen_project(project_dir, error) result = cli.run(project=project_dir, silent=True, args=["build", "collect.bst"]) - result.assert_main_error(ErrorDomain.STREAM, None) - result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.OVERLAPS) - - -@pytest.mark.datafiles(DATA_DIR) -def test_overlaps_whitelist(cli, datafiles): - project_dir = str(datafiles) - gen_project(project_dir, True) - result = cli.run(project=project_dir, silent=True, args=["build", "collect-whitelisted.bst"]) - result.assert_success() + if error: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.OVERLAPS) + else: + result.assert_success() + assert "WARNING [overlaps]" in result.stderr +# +# When the overlap is whitelisted, there is no warning or error. +# +# Still test this in fatal/nonfatal warning modes +# @pytest.mark.datafiles(DATA_DIR) -def test_overlaps_whitelist_ignored(cli, datafiles): +@pytest.mark.parametrize("error", [False, True], ids=["warning", "error"]) +def test_overlaps_whitelisted(cli, datafiles, error): project_dir = str(datafiles) - gen_project(project_dir, False) + gen_project(project_dir, error) result = cli.run(project=project_dir, silent=True, args=["build", "collect-whitelisted.bst"]) result.assert_success() + assert "WARNING [overlaps]" not in result.stderr @pytest.mark.datafiles(DATA_DIR) -def test_overlaps_whitelist_on_overlapper(cli, datafiles): +@pytest.mark.parametrize("error", [False, True], ids=["warning", "error"]) +def test_overlaps_whitelist_on_overlapper(cli, datafiles, error): # Tests that the overlapping element is responsible for whitelisting, # i.e. that if A overlaps B overlaps C, and the B->C overlap is permitted, # it'll still fail because A doesn't permit overlaps. project_dir = str(datafiles) - gen_project(project_dir, True) + gen_project(project_dir, error) result = cli.run(project=project_dir, silent=True, args=["build", "collect-partially-whitelisted.bst"]) - result.assert_main_error(ErrorDomain.STREAM, None) - result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.OVERLAPS) + if error: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.OVERLAPS) + else: + result.assert_success() + assert "WARNING [overlaps]" in result.stderr @pytest.mark.datafiles(DATA_DIR) def test_overlaps_whitelist_undefined_variable(cli, datafiles): project_dir = str(datafiles) gen_project(project_dir, False) - result = cli.run(project=project_dir, silent=True, args=["build", "whitelist-undefined.bst"]) + result = cli.run(project=project_dir, silent=True, args=["show", "whitelist-undefined.bst"]) # Assert that we get the expected undefined variable error, # and that it has the provenance we expect from whitelist-undefined.bst @@ -84,12 +96,11 @@ def test_overlaps_whitelist_undefined_variable(cli, datafiles): @pytest.mark.datafiles(DATA_DIR) -@pytest.mark.parametrize("use_fatal_warnings", [True, False]) -def test_overlaps_script(cli, datafiles, use_fatal_warnings): +def test_overlaps_script(cli, datafiles): # Test overlaps with script element to test # Element.stage_dependency_artifacts() with Scope.RUN project_dir = str(datafiles) - gen_project(project_dir, False, use_fatal_warnings) + gen_project(project_dir, False) result = cli.run(project=project_dir, silent=True, args=["build", "script.bst"]) result.assert_success() @@ -119,3 +130,68 @@ def test_overlap_subproject(cli, tmpdir, datafiles, project_policy, subproject_p else: result.assert_success() assert "WARNING [overlaps]" in result.stderr + + +# Test unstaged-files warnings when staging to an alternative location than "/" +# +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.parametrize("error", [False, True], ids=["warning", "error"]) +def test_unstaged_files_relocated(cli, datafiles, error): + project_dir = str(datafiles) + gen_project(project_dir, error, use_plugin=True) + result = cli.run(project=project_dir, silent=True, args=["build", "relocated-unstaged.bst"]) + if error: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.UNSTAGED_FILES) + else: + result.assert_success() + assert "WARNING [unstaged-files]" in result.stderr + + +# Test overlap warnings when staging to an alternative location than "/" +# +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.parametrize("error", [False, True], ids=["warning", "error"]) +def test_overlaps_relocated(cli, datafiles, error): + project_dir = str(datafiles) + gen_project(project_dir, error, use_plugin=True) + result = cli.run(project=project_dir, silent=True, args=["build", "relocated.bst"]) + if error: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.OVERLAPS) + else: + result.assert_success() + assert "WARNING [overlaps]" in result.stderr + + +# Test overlap warnings as a result of multiple calls to Element.stage_dependency_artifacts() +# +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.parametrize( + "target,action,error", + [ + ("multistage-overlap-ignore.bst", OverlapAction.IGNORE, False), + ("multistage-overlap.bst", OverlapAction.WARNING, False), + ("multistage-overlap.bst", OverlapAction.WARNING, True), + ("multistage-overlap-error.bst", OverlapAction.ERROR, True), + ], + ids=["ignore", "warn-warning", "warn-error", "error"], +) +def test_overlaps_multistage(cli, datafiles, target, action, error): + project_dir = str(datafiles) + gen_project(project_dir, error, use_plugin=True) + result = cli.run(project=project_dir, silent=True, args=["build", target]) + + if action == OverlapAction.WARNING: + if error: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.PLUGIN, CoreWarnings.OVERLAPS) + else: + result.assert_success() + assert "WARNING [overlaps]" in result.stderr + elif action == OverlapAction.IGNORE: + result.assert_success() + assert "WARNING [overlaps]" not in result.stderr + elif action == OverlapAction.ERROR: + result.assert_main_error(ErrorDomain.STREAM, None) + result.assert_task_error(ErrorDomain.ELEMENT, "overlaps") diff --git a/tests/frontend/overlaps/directory-file.bst b/tests/frontend/overlaps/directory-file.bst new file mode 100644 index 000000000..ab3e98a62 --- /dev/null +++ b/tests/frontend/overlaps/directory-file.bst @@ -0,0 +1,9 @@ +kind: import +config: + source: / + target: / +depends: +- with-directory.bst +sources: +- kind: local + path: "directory-file" diff --git a/tests/frontend/overlaps/directory-file/directory-file b/tests/frontend/overlaps/directory-file/directory-file new file mode 100644 index 000000000..f73f3093f --- /dev/null +++ b/tests/frontend/overlaps/directory-file/directory-file @@ -0,0 +1 @@ +file diff --git a/tests/frontend/overlaps/multistage-overlap-error.bst b/tests/frontend/overlaps/multistage-overlap-error.bst new file mode 100644 index 000000000..5b06c671b --- /dev/null +++ b/tests/frontend/overlaps/multistage-overlap-error.bst @@ -0,0 +1,12 @@ +kind: overlap + +build-depends: +- filename: subdir-a.bst + config: + location: / +- filename: c.bst + config: + location: /opt + +config: + action: error diff --git a/tests/frontend/overlaps/multistage-overlap-ignore.bst b/tests/frontend/overlaps/multistage-overlap-ignore.bst new file mode 100644 index 000000000..d40ae9a8f --- /dev/null +++ b/tests/frontend/overlaps/multistage-overlap-ignore.bst @@ -0,0 +1,12 @@ +kind: overlap + +build-depends: +- filename: subdir-a.bst + config: + location: / +- filename: c.bst + config: + location: /opt + +config: + action: ignore diff --git a/tests/frontend/overlaps/multistage-overlap.bst b/tests/frontend/overlaps/multistage-overlap.bst new file mode 100644 index 000000000..bf8984d5a --- /dev/null +++ b/tests/frontend/overlaps/multistage-overlap.bst @@ -0,0 +1,12 @@ +kind: overlap + +build-depends: +- filename: subdir-a.bst + config: + location: / +- filename: c.bst + config: + location: /opt + +config: + action: warning diff --git a/tests/frontend/overlaps/plugins/overlap.py b/tests/frontend/overlaps/plugins/overlap.py new file mode 100644 index 000000000..b1d8642a2 --- /dev/null +++ b/tests/frontend/overlaps/plugins/overlap.py @@ -0,0 +1,56 @@ +from buildstream import Element, OverlapAction + + +# A testing element to test the behavior of staging overlapping files +# +class OverlapElement(Element): + + BST_MIN_VERSION = "2.0" + + def configure(self, node): + node.validate_keys(["action"]) + self.overlap_action = node.get_enum("action", OverlapAction) + + def configure_dependencies(self, dependencies): + self.layout = {} + + for dep in dependencies: + location = "/" + if dep.config: + dep.config.validate_keys(["location"]) + location = dep.config.get_str("location") + try: + element_list = self.layout[location] + except KeyError: + element_list = [] + self.layout[location] = element_list + + element_list.append((dep.element, dep.path)) + + def preflight(self): + pass + + def get_unique_key(self): + sorted_locations = sorted(self.layout) + layout_key = { + location: [dependency_path for _, dependency_path in self.layout[location]] + for location in sorted_locations + } + return {"action": str(self.overlap_action), "layout": layout_key} + + def configure_sandbox(self, sandbox): + for location in self.layout: + sandbox.mark_directory(location, artifact=True) + + def stage(self, sandbox): + sorted_locations = sorted(self.layout) + for location in sorted_locations: + element_list = [element for element, _ in self.layout[location]] + self.stage_dependency_artifacts(sandbox, element_list, path=location, action=self.overlap_action) + + def assemble(self, sandbox): + return "/" + + +def setup(): + return OverlapElement diff --git a/tests/frontend/overlaps/relocated-unstaged.bst b/tests/frontend/overlaps/relocated-unstaged.bst new file mode 100644 index 000000000..65592a3dc --- /dev/null +++ b/tests/frontend/overlaps/relocated-unstaged.bst @@ -0,0 +1,9 @@ +kind: overlap + +build-depends: +- filename: directory-file.bst + config: + location: /opt + +config: + action: warning diff --git a/tests/frontend/overlaps/relocated.bst b/tests/frontend/overlaps/relocated.bst new file mode 100644 index 000000000..a7aea4f1f --- /dev/null +++ b/tests/frontend/overlaps/relocated.bst @@ -0,0 +1,15 @@ +kind: overlap + +build-depends: +- filename: a.bst + config: + location: /opt +- filename: b.bst + config: + location: /opt +- filename: c.bst + config: + location: /opt + +config: + action: warning diff --git a/tests/frontend/overlaps/subdir-a.bst b/tests/frontend/overlaps/subdir-a.bst new file mode 100644 index 000000000..b4da5b851 --- /dev/null +++ b/tests/frontend/overlaps/subdir-a.bst @@ -0,0 +1,7 @@ +kind: import +config: + source: / + target: /opt +sources: +- kind: local + path: "a" diff --git a/tests/frontend/overlaps/unstaged.bst b/tests/frontend/overlaps/unstaged.bst new file mode 100644 index 000000000..974bc3fe4 --- /dev/null +++ b/tests/frontend/overlaps/unstaged.bst @@ -0,0 +1,4 @@ +kind: compose + +build-depends: +- directory-file.bst diff --git a/tests/frontend/overlaps/with-directory.bst b/tests/frontend/overlaps/with-directory.bst new file mode 100644 index 000000000..39632580e --- /dev/null +++ b/tests/frontend/overlaps/with-directory.bst @@ -0,0 +1,7 @@ +kind: import +config: + source: / + target: / +sources: +- kind: local + path: "with-directory" diff --git a/tests/frontend/overlaps/with-directory/directory-file/file b/tests/frontend/overlaps/with-directory/directory-file/file new file mode 100644 index 000000000..f73f3093f --- /dev/null +++ b/tests/frontend/overlaps/with-directory/directory-file/file @@ -0,0 +1 @@ +file diff --git a/tests/integration/manual.py b/tests/integration/manual.py index defc2503c..22b87fb9f 100644 --- a/tests/integration/manual.py +++ b/tests/integration/manual.py @@ -204,3 +204,22 @@ def test_manual_command_subdir(cli, datafiles): result.assert_success() with open(os.path.join(checkout, "hello")) as f: assert f.read() == "hello from subdir\n" + + +# Test staging artifacts into subdirectories +@pytest.mark.datafiles(DATA_DIR) +@pytest.mark.skipif(not HAVE_SANDBOX, reason="Only available with a functioning sandbox") +def test_manual_stage_custom(cli, datafiles): + project = str(datafiles) + checkout = os.path.join(cli.directory, "checkout") + + # Verify that the element builds, and has the correct expected output. + result = cli.run(project=project, args=["build", "manual/manual-stage-custom.bst"]) + result.assert_success() + result = cli.run( + project=project, args=["artifact", "checkout", "manual/manual-stage-custom.bst", "--directory", checkout] + ) + result.assert_success() + + with open(os.path.join(checkout, "test.txt")) as f: + assert f.read() == "This is another test\n" diff --git a/tests/format/dependencies3/dep.bst b/tests/integration/project/elements/manual/import-file.bst index f4f9f6862..195841951 100644 --- a/tests/format/dependencies3/dep.bst +++ b/tests/integration/project/elements/manual/import-file.bst @@ -1,4 +1,5 @@ kind: import + sources: - kind: local - path: project.conf + path: files/import-source diff --git a/tests/integration/project/elements/manual/manual-stage-custom.bst b/tests/integration/project/elements/manual/manual-stage-custom.bst new file mode 100644 index 000000000..abd29c88a --- /dev/null +++ b/tests/integration/project/elements/manual/manual-stage-custom.bst @@ -0,0 +1,13 @@ +kind: manual + +depends: +- base.bst + +build-depends: +- filename: manual/import-file.bst + config: + location: /flying-ponies + +config: + install-commands: + - cp /flying-ponies/subdir/test.txt %{install-root} diff --git a/tests/integration/project/elements/script/corruption-2.bst b/tests/integration/project/elements/script/corruption-2.bst index 39c4f2c23..75c5e92d0 100644 --- a/tests/integration/project/elements/script/corruption-2.bst +++ b/tests/integration/project/elements/script/corruption-2.bst @@ -1,10 +1,8 @@ kind: script -depends: -- filename: base.bst - type: build -- filename: script/corruption-image.bst - type: build +build-depends: +- base.bst +- script/corruption-image.bst config: commands: diff --git a/tests/integration/project/elements/script/corruption.bst b/tests/integration/project/elements/script/corruption.bst index 037d4daca..841a2dd69 100644 --- a/tests/integration/project/elements/script/corruption.bst +++ b/tests/integration/project/elements/script/corruption.bst @@ -1,21 +1,9 @@ kind: script -depends: -- filename: base.bst - type: build -- filename: script/corruption-image.bst - type: build -- filename: script/corruption-integration.bst - type: build +build-depends: +- base.bst +- script/corruption-image.bst +- script/corruption-integration.bst variables: install-root: "/" - -config: - layout: - - element: base.bst - destination: "/" - - element: script/corruption-image.bst - destination: "/" - - element: script/corruption-integration.bst - destination: "/" diff --git a/tests/integration/project/elements/script/marked-tmpdir.bst b/tests/integration/project/elements/script/marked-tmpdir.bst index 506cdd5f4..7abbd3261 100644 --- a/tests/integration/project/elements/script/marked-tmpdir.bst +++ b/tests/integration/project/elements/script/marked-tmpdir.bst @@ -1,8 +1,7 @@ kind: compose -depends: -- filename: base.bst - type: build +build-depends: +- base.bst public: bst: diff --git a/tests/integration/project/elements/script/script-layout.bst b/tests/integration/project/elements/script/script-layout.bst index 11ca353e3..f19b27e52 100644 --- a/tests/integration/project/elements/script/script-layout.bst +++ b/tests/integration/project/elements/script/script-layout.bst @@ -5,19 +5,12 @@ variables: install-root: /buildstream/nstall build-root: /buildstream/uild -depends: - - filename: base.bst - type: build +build-depends: + - base.bst - filename: script/script.bst - type: build + config: + location: /buildstream/uild config: - layout: - - element: base.bst - destination: / - - - element: script/script.bst - destination: /buildstream/uild - commands: - - "cp %{build-root}/test %{install-root}" + - "cp %{build-root}/test %{install-root}" diff --git a/tests/integration/project/elements/script/script.bst b/tests/integration/project/elements/script/script.bst index ffca23ab7..3f3eb55c7 100644 --- a/tests/integration/project/elements/script/script.bst +++ b/tests/integration/project/elements/script/script.bst @@ -1,9 +1,8 @@ kind: script description: Script test -depends: - - filename: base.bst - type: build +build-depends: +- base.bst config: commands: diff --git a/tests/integration/project/elements/script/tmpdir.bst b/tests/integration/project/elements/script/tmpdir.bst index 685a694ea..5fbf55dc9 100644 --- a/tests/integration/project/elements/script/tmpdir.bst +++ b/tests/integration/project/elements/script/tmpdir.bst @@ -1,8 +1,7 @@ kind: script -depends: -- filename: script/no-tmpdir.bst - type: build +build-depends: +- script/no-tmpdir.bst config: commands: |